Repository navigation
feat: expose aggregate job metrics at GET /metrics - #71
Merged
Merged
Conversation
The worker answers GET /metrics in the Prometheus text format behind the service token: jobs by task and terminal status, submissions, capacity refusals, queue depth, running jobs, pool size and readiness, the concurrency and capacity settings, and per-task histograms of job and queue durations. Label values come from fixed sets and no job content reaches the body; the numbers live in memory and reset on restart. Documented in docs/protocol.md and docs/security.md.
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #67.
What
GET /metricson the worker answers in the Prometheus text exposition format (text/plain; version=0.0.4; charset=utf-8), written with the standard library and behind the same bearer token as every other route. It exposes jobs by task and terminal status, accepted submissions, 429 refusals, queue depth, running jobs, pool size and readiness, the concurrency and capacity settings, per-task histograms of job duration (submission to terminal status) and queue time (submission to running) with fixed buckets, andbridge_info. Label values come only from fixed sets (the task allowlist, the four terminal statuses,capacity, the backend name); no job ID, idempotency key, input, result, error message, model path or token can reach the body. The numbers live in memory and reset when the worker restarts; a job read back fromjobs.sqlitecounts nothing.Design
JobStorekeeps the counters and per-bucket histogram counts and updates them under its lock: a_submitted_atmonotonic stamp is set when a job is accepted, queue time is observed on the child'sstartedmessage, duration in_finish, which every terminal path goes through. The body ofsubmitmoved into_acceptso onetry/except CapacityErrorcounts every 429 (full store, shutdown, a job-table write the database refuses) exactly once.snapshot()copies plain values under the lock;metrics.render()formats them in the request thread without it, so a scrape never waits for a job.tasks.pygained aTEST_TASKSconstant (used byvalidateand the store) so the test-task series exist on a worker started withBRIDGE_TEST_TASKS=1.Files
worker/ai_bridge/metrics.py(new): buckets,observe, label escaping, number formatting, one renderer per family kind,render.worker/ai_bridge/jobs.py: the aggregates,_accept, the stamps,snapshot().worker/ai_bridge/server.py: the route;_replyand the new_textshare_send(same headers as before).worker/ai_bridge/tasks.py:TEST_TASKS.tests/python/test_jobs.py,tests/python/test_server.py: snapshot tests (every lifecycle case, queue depth, pool gauges after a crash, the 429 counter, histogram consistency, the leak grep, a scrape during a running job, formatting outside the lock), a strict parser with its own negative cases, renderer tests, the HTTP cases (401, 404).tests/integration.py: reads/metricsbefore and after its jobs, 401 with a wrong token, counters moved, no document text in the body..github/workflows/model-smoke.yml:metrics.pyonpaths:.docs/protocol.md(route table and a Metrics section),docs/security.md,README.md,CHANGELOG.md.Checks
The full command block of
AGENTS.md, run as one command on this head (2322b46) in a worktree, compose's default image PHP 8.5:--profile test config: okpython-tests: Ran 211 tests, OKphp-tests: PASS: 147 PHP contract checksphp -loversrc,examples,tests/php: no syntax errorsphp-integration: PASS: 184 total PHP checks including real HTTP failure casesintegration: PASS: 12 concurrent isolated jobs through a reused FrankenPHP worker; top_k narrowing; embedding; redaction; invalid and missing-job cases; worker metricsmcp-fetcher,mcp-tests: Ran 125 tests, OKcomposer-install:PhpAiBridge\Clientautoloaded fromvendor/(archive mode left to CI: a worktree uses the path repository)php-fetcher,php-analysis: [OK] No errorsphp-psr18-tests: PASS: 177 PHP contract checksPHP 8.2, 8.3 and 8.4 are left to CI.
mainis this branch's base commit (nothing to merge).Review
Independent review on another model family (codex, read-only) of the diff against main: no defect to fix, nothing outside the branch, no missing test against the criteria, security nothing open. Security read of the diff: no secret, no client-supplied string reaches the body, authentication identical to the other routes.
Not verified here: a scrape by a real Prometheus server (the format is checked by the strict parser in the tests);
model-smoke.ymlwas not run.