Problem
The idempotency store treats a processing record as permanent until a matching request calls complete(), but complete() is fire-and-forget and there is no lease or TTL on in-flight records. In middleware.ts:
// app/backend/src/idempotency/middleware.ts
res.send = (body: any) => {
// Fire and forget cache save (log on failure)
store.complete(key, recordStatus, status, bodyString).catch(err => console.error(...));
return originalSend(body);
};And in store.ts, tryAcquire inserts status='processing' and never times it out; the only expiry is cleanup(maxAgeHours) which deletes by created_at with no regard for status.
Consequences, distinctly:
- A crash orphans the key. If the process dies after
tryAcquire inserts processing but before complete runs, the row stays processing forever. Every retry with the same Idempotency-Key returns 409 AlreadyProcessingError indefinitely (until a manual cleanup with a large window deletes it), permanently stranding a claim submission — the exact class of operation idempotency exists to protect.
- Replay-after-success races to 409. Because
complete is not awaited, a fast retry that arrives immediately after the first response can still read processing and return 409 instead of the cached response, so the "same key ⇒ same result" guarantee is violated under concurrency.
- The record has no state-machine recovery. There is no
failed-terminal distinction for a crashed in-flight request; a caller cannot distinguish "still running" from "abandoned", so safe retry policy is impossible.
Root cause
The store models three statuses (processing/succeeded/failed) but no notion of a lease/ownership window or an abandoned terminal state, and the middleware treats complete as best-effort rather than as part of the request lifecycle.
Why this is architecturally hard
- This is a concurrency-control problem, not a timeout. A correct fix needs a lease/heartbeat on the
processing record (e.g. lease_expires_at refreshed while the handler runs) plus a transition rule for abandoned leases; naively deleting processing rows older than N seconds can delete a still-running long request and allow double-execution — the opposite of idempotency.
- The response-caching path is coupled to
res.send. Caching must be awaited or moved to an onFinished/interceptor that runs before the connection is released, so a replay can never observe processing after the first response is committed.
- Two storage backends.
store.ts targets Postgres via Pool, but the module must keep working with whatever IdempotencyStore the app boots (and its tests); the lease protocol must be expressed in the store interface, not in one SQL dialect.
- Claim submission is the trust boundary. The key protects exactly-once claim submissions to the on-chain adapter; getting the lease wrong risks either stranding claims (current bug) or double-submitting them (a naive fix).
Proposed design
Add a lease_expires_at (or processing_until) column set on acquire and refreshed via a heartbeat while the handler runs; treat expired-lease processing rows as abandoned (transition to a terminal failed/abandoned state and allow re-acquire). Await complete (or use a response-finish hook) so a succeeded result is visible before the client's connection closes. A table of the target semantics:
| State |
On acquire (existing key) |
On expiry |
On complete |
processing (lease live) |
409 |
n/a |
→ succeeded/failed |
processing (lease expired) |
re-acquire with new lease |
→ abandoned |
n/a |
succeeded/failed |
replay cached response |
n/a |
no-op |
Acceptance criteria
Service
Tests
Out of scope
Redis ZSET rate limiting and webhook HMAC are separate, already-tracked concerns.
Getting started
Files: app/backend/src/idempotency/store.ts, app/backend/src/idempotency/middleware.ts, app/backend/src/idempotency/error.ts, and the idempotency specs under app/backend/src/idempotency/.
Good first files to read: store.ts (the tryAcquire/complete/cleanup contract) and middleware.ts (the fire-and-forget complete and 409 paths).
Problem
The idempotency store treats a
processingrecord as permanent until a matching request callscomplete(), butcomplete()is fire-and-forget and there is no lease or TTL on in-flight records. Inmiddleware.ts:And in
store.ts,tryAcquireinsertsstatus='processing'and never times it out; the only expiry iscleanup(maxAgeHours)which deletes bycreated_atwith no regard for status.Consequences, distinctly:
tryAcquireinsertsprocessingbut beforecompleteruns, the row staysprocessingforever. Every retry with the sameIdempotency-Keyreturns 409AlreadyProcessingErrorindefinitely (until a manualcleanupwith a large window deletes it), permanently stranding a claim submission — the exact class of operation idempotency exists to protect.completeis not awaited, a fast retry that arrives immediately after the first response can still readprocessingand return 409 instead of the cached response, so the "same key ⇒ same result" guarantee is violated under concurrency.failed-terminal distinction for a crashed in-flight request; a caller cannot distinguish "still running" from "abandoned", so safe retry policy is impossible.Root cause
The store models three statuses (
processing/succeeded/failed) but no notion of a lease/ownership window or anabandonedterminal state, and the middleware treatscompleteas best-effort rather than as part of the request lifecycle.Why this is architecturally hard
processingrecord (e.g.lease_expires_atrefreshed while the handler runs) plus a transition rule for abandoned leases; naively deletingprocessingrows older than N seconds can delete a still-running long request and allow double-execution — the opposite of idempotency.res.send. Caching must be awaited or moved to anonFinished/interceptor that runs before the connection is released, so a replay can never observeprocessingafter the first response is committed.store.tstargets Postgres viaPool, but the module must keep working with whateverIdempotencyStorethe app boots (and its tests); the lease protocol must be expressed in the store interface, not in one SQL dialect.Proposed design
Add a
lease_expires_at(orprocessing_until) column set on acquire and refreshed via a heartbeat while the handler runs; treat expired-leaseprocessingrows as abandoned (transition to a terminalfailed/abandonedstate and allow re-acquire). Awaitcomplete(or use a response-finish hook) so a succeeded result is visible before the client's connection closes. A table of the target semantics:processing(lease live)succeeded/failedprocessing(lease expired)abandonedsucceeded/failedAcceptance criteria
Service
Tests
Out of scope
Redis ZSET rate limiting and webhook HMAC are separate, already-tracked concerns.
Getting started
Files:
app/backend/src/idempotency/store.ts,app/backend/src/idempotency/middleware.ts,app/backend/src/idempotency/error.ts, and the idempotency specs underapp/backend/src/idempotency/.Good first files to read:
store.ts(thetryAcquire/complete/cleanupcontract) andmiddleware.ts(the fire-and-forgetcompleteand 409 paths).