feat: send Retry-After on 429 and expose it to the PHP client and the MCP server

CHANGELOG.md

@@ -8,6 +8,7 @@ All notable changes to this project are documented in this file. The format foll
 
 ### Added
 
+- A 429 from the worker carries `Retry-After` in whole seconds: when the first retained job expires, or 1 when every slot holds an unfinished job. The PHP client exposes it as `BridgeException::$retryAfterSeconds` (and refuses a response header block over 16 KiB as `invalid_response`) without retrying by itself; the MCP server waits for it within a call's budget instead of guessing.
 - `BRIDGE_STATE_DIR`: the worker keeps its job table (never inputs) in `jobs.sqlite` there, so retained results survive a restart and unfinished jobs are reported as `failed` with `service_restarted` instead of disappearing; unset, the table stays in memory as before. Compose's `worker` keeps it on a tmpfs.
 - An optional `idempotency_key` on `POST /v1/jobs` (1–128 printable ASCII characters): a repeat while its job is retained returns that job with 202 instead of submitting again. The PHP client's `submit`, `submitRerank`, `submitEmbed` and `submitRedact` take it as `?string $idempotencyKey` and, when one is sent, accept any valid status in the response.
 

docs/integration.md

@@ -178,10 +178,28 @@ PHP objects reused by a long-lived worker must not retain per-user inputs, authe
 | `transport_error` during submission | Response unknown; a job may have been created | Resubmit with the same `idempotencyKey` while the job would still be retained; never retry without one |
 | `http_error`, status 401 | Service credentials rejected | Fix server-side configuration, do not forward service secrets |
 | `http_error`, status 400/413 | Invalid or oversized request | Reject/correct input |
-| `http_error`, status 429 | Total queued/running/retained capacity exhausted | Apply backpressure; completed retained jobs also occupy capacity |
+| `http_error`, status 429 | Total queued/running/retained capacity exhausted; no job was created | Apply backpressure; resubmit no sooner than `retryAfterSeconds` (below). Completed retained jobs also occupy capacity |
 | `invalid_response` | Peer violated the client contract | Treat as an integration failure, not a valid task result |
 | Terminal `failed` or `timed_out` | Task did not succeed | Report a safe failure; retries require an application policy |
 
+When the worker is full, `submit()` throws a `BridgeException` with `httpStatus` 429 and, from the worker's `Retry-After` header, `retryAfterSeconds`: when capacity is expected back. The client never retries by itself. Unlike a `transport_error`, a 429 means no job exists, so scheduling the submission again is safe; how is the application's choice, for instance a delayed message on its own queue:
+
+```php
+try {
+    $job = $client->submitRerank($query, $documents);
+} catch (\PhpAiBridge\BridgeException $error) {
+    if ($error->httpStatus !== 429) {
+        throw $error;
+    }
+    // Null when the header was missing or malformed (a proxy answered, say): fall back to your own delay.
+    $delaySeconds = $error->retryAfterSeconds ?? 5;
+    $this->queue->dispatchLater(new SubmitRerank($applicationJobId), $delaySeconds);
+    return new Response('', 503, ['Retry-After' => (string) $delaySeconds]);
+}
+```
+
+`dispatchLater()` and `SubmitRerank` stand for your framework's delayed dispatch (Symfony Messenger's `DelayStamp`, Laravel's `->delay()`); cap the attempts, since a worker kept full by other callers can refuse again.
+
 The current default limits are two warm task processes, 128 total job slots and 300 seconds of terminal retention. These are settings, not measured throughput or a production SLA. The model profile keeps one task process, which loads each model on its first job of that kind and keeps it.
 
 ## Before shipping a framework integration

docs/protocol.md

@@ -31,10 +31,12 @@ Submit `{"task":"redact","input":{"text":"...","entities":["PER"],"min_score":0.
 
 Failures use generic error objects with `code` and `message`. Task errors include `task_failed`, `worker_crashed`, `worker_start_failed`, `deadline_exceeded` and `service_restarted`. Internal exception text is not sent to clients. HTTP contract failures return 400, missing auth 401, missing routes/jobs 404, a connection idle for five seconds while its body is read 408 (a stall while reading headers closes the connection without a response), oversized bodies 413 and capacity exhaustion 429.
 
+Every 429 carries `Retry-After` in whole delay seconds (an integer of at least 1, never an HTTP date): when every slot is taken and some jobs have finished, the time until the first of them leaves retention, rounded up; when every slot holds an unfinished job, the worker is shutting down or the job table cannot be written, 1. A 429 created no job, so resubmitting after that delay cannot duplicate work. The PHP client exposes the value as `BridgeException::$retryAfterSeconds` and the MCP server waits for it within a call's budget; both read one header of 1 to 6 digits and ignore anything else, a repeated header included.
+
 The default capacity is 128 total queued, running and retained jobs, concurrency is 2, and terminal retention is 300 seconds. A full result cache rejects submissions until entries expire; it does not silently evict a result early. Configure these with `BRIDGE_CAPACITY`, `BRIDGE_CONCURRENCY`, and `BRIDGE_RETENTION_SECONDS`. The worker keeps as many child processes as its concurrency and replaces a child after `BRIDGE_WORKER_MAX_JOBS` jobs (default 1000, between 1 and 1000000), as well as after a cancellation, a deadline or a crash. A numeric setting that does not parse stops the worker at start with a message naming its variable; one outside its range is refused before any child starts.
 
 With `BRIDGE_STATE_DIR` set, the job table (never a job's input) is written through to SQLite at `<BRIDGE_STATE_DIR>/jobs.sqlite` on every status change; unset, it stays in memory and a restart loses every job. When the worker starts on an existing file, a job still `queued` or `running` there becomes `failed` with error `service_restarted` (its input was never stored, so it is not replayed; a shutdown reports such jobs as `cancelled` to whoever polls the stopping process but leaves them unfinished in the file), a finished job whose retention has not run out by the wall clock is served by GET as before until it does, and an expired one is deleted. A 404 for a well-formed ID therefore means the job expired, was never submitted to this instance, or was lost to a restart without a state directory; the worker cannot tell these apart.
 
-The PHP client uses a fresh cURL handle per call, refuses redirects, ignores proxy environment variables, caps response bytes, and verifies that GET/cancel responses match the requested ID. Configure its request timeout separately from job execution and local waiting deadlines. There are no transparent submission retries. The MCP server in `mcp_server/` is a second, independent client with the same checks; unlike the PHP client it cancels a job whose local wait expires, because no later request will collect it.
+The PHP client uses a fresh cURL handle per call, refuses redirects, ignores proxy environment variables, caps response bytes, and verifies that GET/cancel responses match the requested ID. Configure its request timeout separately from job execution and local waiting deadlines. There are no transparent submission retries, a 429 included. The MCP server in `mcp_server/` is a second, independent client with the same checks; unlike the PHP client it cancels a job whose local wait expires, because no later request will collect it.
 
 Adding a task requires trusted source changes to `worker/ai_bridge/tasks.py`: an explicit name, input validation, bounded output and execution logic. Add a PHP result type where useful, plus successful, malformed-input and lifecycle tests. Task imports, file paths and command strings must not come from request data.

docs/testing.md

@@ -48,9 +48,9 @@ PHP branch support was verified locally on 2026-09-17 on PHP 8.2.33, 8.3.33, 8.4
 
 | Layer | Scope and counting |
 | --- | --- |
-| Python worker | 184 tests: lifecycle, validation and HTTP, the job table in SQLite (a store reopened on its file fails unfinished jobs as `service_restarted`, serves retained results, deletes expired rows from the file and never holds an input; a submission the table refuses is a 429, a later write it refuses leaves the job served, and a key whose row outlived a failed delete runs again) and idempotency keys, process crashes, cancellation, timeouts, retention, allocation/exit races, the warm pool (that successive jobs share a child in which no live container or frame holds a reference to a finished job's input or result, after a success, a failure and real task payloads, with the detector checked against a planted needle; that a cancelled, timed-out or crashed child is replaced; that a child is recycled after its share of jobs; that reads, submissions and cancellations answer at once while a child is being started, that a child whose start ends after the store closed does not outlive it, and that a job assigned to or waiting for a live child does not fail when no replacement can start; that a job whose child died before taking it is tried once more on another, only once, that a job sent but not yet taken can still be cancelled or time out, and that a job whose child crashed after taking it is not; that a child, idle or running a job, ignores the SIGINT a terminal's Ctrl+C sends the whole process group; that the service token is absent from a child's environment and from its environment block when the service started it; that a message for an earlier job lands nowhere; that a child that never reports ready is replaced and that children dying at start are not restarted in a loop; and that a cancellation does not hold up reads), redaction rules including IPv6, lower-case IBANs and email addresses in any script and their speed on hostile input, NER aggregation, name joining, the second title-cased reading, the uncased model's reading of the whole text and the merge driven by stubs, the fetcher's pins and the model directories an ONNX worker checks for before it starts, batched cross-encoder scoring, the NER measurement's sample and scoring, the workflows' runners, action pins and checkout credentials, both Claude Code hooks (the post-edit rules, and the pre-call push guard through its own cases), progress bounds, the service's reading of its BRIDGE_* settings (defaults, every variable, malformed and out-of-range values, a state directory that cannot hold the table), and the model-name table every result reports from |
-| MCP server | 119 tests, offline with hash-pinned wheels: the protocol client against an in-process worker, every tool through the SDK's in-memory client with test tasks enabled, the entry point as a subprocess, result validators on synthetic data, the index builder (chunking, the format and each corruption it refuses, building against the worker, caching, its command line) `bridge_search` (the model gate, path confinement, reloads, rerank on and off), the reachability of every answer in the LLM evaluation, that a default search sees only part of its corpus and that its two-step question takes two steps, that the demo worker reaches none of the semantic evaluation's answers, that a tool waits out a full worker, before each of a search's jobs and within one budget, and says so when it stays full, the evaluation runner's command, isolation checks, scoring and per-backend launcher and index check without starting `claude`, a check that the server's model names match the worker's, that every path parameter says where it resolves and every position in a result says what it counts, and a check that every timing claim in a tool description is a measured median |
-| PHP contracts + HTTP | 168 checks per PHP version; this **includes** the 133 contract checks, not 168 + 133 |
+| Python worker | 189 tests: lifecycle, validation and HTTP, the job table in SQLite (a store reopened on its file fails unfinished jobs as `service_restarted`, serves retained results, deletes expired rows from the file and never holds an input; a submission the table refuses is a 429, a later write it refuses leaves the job served, and a key whose row outlived a failed delete runs again) and idempotency keys, process crashes, cancellation, timeouts, retention, the Retry-After a full or closing store gives and its header on every 429, allocation/exit races, the warm pool (that successive jobs share a child in which no live container or frame holds a reference to a finished job's input or result, after a success, a failure and real task payloads, with the detector checked against a planted needle; that a cancelled, timed-out or crashed child is replaced; that a child is recycled after its share of jobs; that reads, submissions and cancellations answer at once while a child is being started, that a child whose start ends after the store closed does not outlive it, and that a job assigned to or waiting for a live child does not fail when no replacement can start; that a job whose child died before taking it is tried once more on another, only once, that a job sent but not yet taken can still be cancelled or time out, and that a job whose child crashed after taking it is not; that a child, idle or running a job, ignores the SIGINT a terminal's Ctrl+C sends the whole process group; that the service token is absent from a child's environment and from its environment block when the service started it; that a message for an earlier job lands nowhere; that a child that never reports ready is replaced and that children dying at start are not restarted in a loop; and that a cancellation does not hold up reads), redaction rules including IPv6, lower-case IBANs and email addresses in any script and their speed on hostile input, NER aggregation, name joining, the second title-cased reading, the uncased model's reading of the whole text and the merge driven by stubs, the fetcher's pins and the model directories an ONNX worker checks for before it starts, batched cross-encoder scoring, the NER measurement's sample and scoring, the workflows' runners, action pins and checkout credentials, both Claude Code hooks (the post-edit rules, and the pre-call push guard through its own cases), progress bounds, the service's reading of its BRIDGE_* settings (defaults, every variable, malformed and out-of-range values, a state directory that cannot hold the table), and the model-name table every result reports from |
+| MCP server | 125 tests, offline with hash-pinned wheels: the protocol client against an in-process worker, every tool through the SDK's in-memory client with test tasks enabled, the entry point as a subprocess, result validators on synthetic data, the index builder (chunking, the format and each corruption it refuses, building against the worker, caching, its command line) `bridge_search` (the model gate, path confinement, reloads, rerank on and off), the reachability of every answer in the LLM evaluation, that a default search sees only part of its corpus and that its two-step question takes two steps, that the demo worker reaches none of the semantic evaluation's answers, that a tool waits out a full worker for its Retry-After within the call's budget or with its own backoff without one, before each of a search's jobs and within one budget, and says so when it stays full, the evaluation runner's command, isolation checks, scoring and per-backend launcher and index check without starting `claude`, a check that the server's model names match the worker's, that every path parameter says where it resolves and every position in a result says what it counts, and a check that every timing claim in a tool description is a measured median |
+| PHP contracts + HTTP | 178 checks per PHP version, a full fault-worker's Retry-After read through the client among them; this **includes** the 141 contract checks (Retry-After from a loopback peer, present, absent, malformed and repeated, and a header block over 16 KiB refused), not 178 + 141 |
 | FrankenPHP | 12 jobs submitted by six concurrent test threads through a reused PHP worker, a 512-document `top_k` request, embedding and redaction round trips, and invalid-input, missing-job and multibyte boundary checks |
 | PHP syntax | All 13 PHP source/example/test files |
 | Consumer install | `composer validate --strict` on the library, then a throwaway consumer project in tmpfs, with Packagist disabled and the platform pinned to PHP 8.2.0, installs its `git archive` zip from an `artifact` repository (or from a path repository in a git worktree whose git directory is outside the mount; the archive is built from the committed `HEAD`, so an uncommitted change is not what it tests), checks that `vendor/` holds `src/Client.php` and `docs/` and not `worker/`, `tests/` or `docker/`, and autoloads `PhpAiBridge\Client`; runs once, in the MCP server job, with no network |

mcp_server/bridge_mcp/protocol.py

@@ -11,14 +11,17 @@
 STATUSES = TERMINAL | {"queued", "running"}
 JOB_ID = re.compile(r"[a-f0-9]{32}")
 ERROR_CODE = re.compile(r"[a-z_]{1,40}")
+# Delay seconds only, as the PHP client reads them; an HTTP date would need the two clocks to agree.
+RETRY_AFTER = re.compile(r"[ \t]*([0-9]{1,6})[ \t]*")
 
 
 class BridgeError(Exception):
     """A protocol, transport or task failure. Messages never contain the token or worker internals."""
 
-    def __init__(self, message: str, code: str, status: int | None = None, job_id: str | None = None):
+    def __init__(self, message: str, code: str, status: int | None = None, job_id: str | None = None,
+                 retry_after: int | None = None):
         super().__init__(message)
-        self.code, self.status, self.job_id = code, status, job_id
+        self.code, self.status, self.job_id, self.retry_after = code, status, job_id, retry_after
 
 
 class Bridge:
@@ -135,6 +138,9 @@ def _request(self, method: str, path: str, body: dict | None = None) -> dict:
             connection.request(method, path, data, headers)
             response = connection.getresponse()
             status, content_type = response.status, response.getheader("Content-Type") or ""
+            # A repeated Retry-After is a conflict the caller cannot trust, so it counts as absent.
+            values = (response.headers.get_all("Retry-After") or []) if status == 429 else []
+            retry_after = RETRY_AFTER.fullmatch(values[0]) if len(values) == 1 else None
             raw = response.read(MAX_BYTES + 1)
         except (OSError, http.client.HTTPException) as error:
             raise BridgeError("Bridge transport failed: " + type(error).__name__, "transport_error") from None
@@ -154,7 +160,8 @@ def _request(self, method: str, path: str, body: dict | None = None) -> dict:
             # free text, and a proxy's HTML error page contributes nothing but its status.
             code = decoded.get("error", {}).get("code") if isinstance(decoded, dict) and isinstance(decoded.get("error"), dict) else None
             code = code if isinstance(code, str) and ERROR_CODE.fullmatch(code) else "http_error"
-            raise BridgeError(f"Bridge returned HTTP {status}: {code}", code, status)
+            raise BridgeError(f"Bridge returned HTTP {status}: {code}", code, status,
+                              retry_after=int(retry_after[1]) if retry_after else None)
         if not is_json:
             raise BridgeError("Expected a JSON response", "invalid_response", status)
         if decoded is None: