Skip to content

feat: expose aggregate job metrics at GET /metrics - #71

Merged
cristiangirlea merged 1 commit into
mainfrom
feat/worker-metrics
Oct 21, 2026
Merged

cristiangirlea merged 1 commit into
mainfrom
feat/worker-metrics

Conversation

@cristiangirlea

Copy link
Copy Markdown
Owner

Fixes #67.

What

GET /metrics on the worker answers in the Prometheus text exposition format (text/plain; version=0.0.4; charset=utf-8), written with the standard library and behind the same bearer token as every other route. It exposes jobs by task and terminal status, accepted submissions, 429 refusals, queue depth, running jobs, pool size and readiness, the concurrency and capacity settings, per-task histograms of job duration (submission to terminal status) and queue time (submission to running) with fixed buckets, and bridge_info. Label values come only from fixed sets (the task allowlist, the four terminal statuses, capacity, the backend name); no job ID, idempotency key, input, result, error message, model path or token can reach the body. The numbers live in memory and reset when the worker restarts; a job read back from jobs.sqlite counts nothing.

Design

  • JobStore keeps the counters and per-bucket histogram counts and updates them under its lock: a _submitted_at monotonic stamp is set when a job is accepted, queue time is observed on the child's started message, duration in _finish, which every terminal path goes through. The body of submit moved into _accept so one try/except CapacityError counts every 429 (full store, shutdown, a job-table write the database refuses) exactly once.
  • snapshot() copies plain values under the lock; metrics.render() formats them in the request thread without it, so a scrape never waits for a job.
  • tasks.py gained a TEST_TASKS constant (used by validate and the store) so the test-task series exist on a worker started with BRIDGE_TEST_TASKS=1.
  • The PHP client and the MCP server are unchanged.

Files

  • worker/ai_bridge/metrics.py (new): buckets, observe, label escaping, number formatting, one renderer per family kind, render.
  • worker/ai_bridge/jobs.py: the aggregates, _accept, the stamps, snapshot().
  • worker/ai_bridge/server.py: the route; _reply and the new _text share _send (same headers as before).
  • worker/ai_bridge/tasks.py: TEST_TASKS.
  • tests/python/test_jobs.py, tests/python/test_server.py: snapshot tests (every lifecycle case, queue depth, pool gauges after a crash, the 429 counter, histogram consistency, the leak grep, a scrape during a running job, formatting outside the lock), a strict parser with its own negative cases, renderer tests, the HTTP cases (401, 404).
  • tests/integration.py: reads /metrics before and after its jobs, 401 with a wrong token, counters moved, no document text in the body.
  • .github/workflows/model-smoke.yml: metrics.py on paths:.
  • docs/protocol.md (route table and a Metrics section), docs/security.md, README.md, CHANGELOG.md.

Checks

The full command block of AGENTS.md, run as one command on this head (2322b46) in a worktree, compose's default image PHP 8.5:

  • --profile test config: ok
  • python-tests: Ran 211 tests, OK
  • php-tests: PASS: 147 PHP contract checks
  • php -l over src, examples, tests/php: no syntax errors
  • php-integration: PASS: 184 total PHP checks including real HTTP failure cases
  • integration: PASS: 12 concurrent isolated jobs through a reused FrankenPHP worker; top_k narrowing; embedding; redaction; invalid and missing-job cases; worker metrics
  • mcp-fetcher, mcp-tests: Ran 125 tests, OK
  • composer-install: PhpAiBridge\Client autoloaded from vendor/ (archive mode left to CI: a worktree uses the path repository)
  • php-fetcher, php-analysis: [OK] No errors
  • php-psr18-tests: PASS: 177 PHP contract checks

PHP 8.2, 8.3 and 8.4 are left to CI. main is this branch's base commit (nothing to merge).

Review

Independent review on another model family (codex, read-only) of the diff against main: no defect to fix, nothing outside the branch, no missing test against the criteria, security nothing open. Security read of the diff: no secret, no client-supplied string reaches the body, authentication identical to the other routes.

Not verified here: a scrape by a real Prometheus server (the format is checked by the strict parser in the tests); model-smoke.yml was not run.

The worker answers GET /metrics in the Prometheus text format behind the
service token: jobs by task and terminal status, submissions, capacity
refusals, queue depth, running jobs, pool size and readiness, the
concurrency and capacity settings, and per-task histograms of job and
queue durations. Label values come from fixed sets and no job content
reaches the body; the numbers live in memory and reset on restart.
Documented in docs/protocol.md and docs/security.md.
@cristiangirlea
cristiangirlea merged commit 4cb64af into main Oct 21, 2026
7 checks passed
@cristiangirlea
cristiangirlea deleted the feat/worker-metrics branch October 21, 2026 15:43
@cristiangirlea cristiangirlea mentioned this pull request Oct 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: GET /metrics on the worker in Prometheus text format, aggregate only

1 participant