fix(sandbox): let a reconnecting worker adopt its run, not restart it

[pedrofrxncx]

The bug

Runs keep restarting mid-job and losing work. Root cause is a protocol gap.

The daemon exposes exactly two run verbs:

POST   /_sandbox/dispatch        ← carries the full harness input
DELETE /_sandbox/runs/{runId}    ← cancel

There is no attach verb. So a replacement worker has no way to say "I am the new reader for run X, stream me from seq 199" — the only thing it can do is POST /dispatch again, and the daemon has to guess whether that means reconnect or a competing turn. It guessed from whether the incumbent's connection was still open, which describes a socket, not a worker.

What actually happens

2m4s   Scaled down replica set deco-studio-worker-59cfcffd68 from 2 to 1
2m4s   Killing pod/deco-studio-worker-59cfcffd68-fmkzl
119s   SuccessfulRescale  New size: 2; reason: Current number of replicas below Spec.MinReplicas

KEDA scales in the worker holding a run (then immediately back up, because 1 is below minReplicas). DBOS recovers the workflow and re-dispatches within a second or two — faster than the dying pod's TCP close is observed by the daemon. The daemon sees a live client, takes the takeover branch, SIGKILLs a healthy harness and re-runs the turn:

run ended  chunks=197 elapsedMs=295253 error="superseded"
[decopilot] attempt superseded by a newer dispatch; leaving the terminal to it
[hostedHarness] ... fromSeq: 199

There was never a second writer. Only a slow-closing one.

#7006 made the daemon able to reattach, but only once the run is already marked detached — so it does not fire when the reconnect wins the race against the socket close, which is the common case.

The change

A dispatch for a run that still looks live now waits supersedeGrace (5s) for the incumbent to hang up, and reattaches if it does. Waiting costs a reconnect nothing — it is about to stream anyway.

Preserved:

• A genuine double-dispatch still supersedes, just that much later. The "two agents, one worktree" guarantee (2026-08-07) is unchanged.
• A run that finishes inside the grace is collected, not re-run — the re-dispatch gets its buffered frames and terminal.
• The wait is lock-free, so the registry is re-read afterwards: a run that ended or was replaced during it is no longer ours to cancel.

Tests

Four, one per branch: reconnect-during-grace reattaches and does not cancel the harness; a genuinely-attached incumbent is still superseded; a run finishing inside the grace is collected; plus the existing takeover suite, with the grace shortened so the suite stays fast.

go test ./... and go test -race ./internal/dispatch/ pass; go vet clean.

This is a mitigation, not the fix

The protocol should carry an explicit attach — GET /_sandbox/runs/{runId}/stream?fromSeq=N — so a reconnect can never be read as a competing turn and POST /dispatch goes back to meaning "start this turn" only. Two things to settle first: the frame buffer currently spans only from the moment of detach, so an attach arriving later would still gap, and ingestRun/JetStream may be the better replay source since it already holds every chunk by seq.

The trigger, fixed too

keda-hpa-deco-studio-worker reported 0/5 (avg) while a run was in flight: the metrics-api triggers count ENQUEUED work only, and a hosted run is dequeued the instant it starts, then streams for minutes with nothing behind it. KEDA saw an idle worker and scaled in the pod holding a live DBOS workflow.

/dbos-queue-depth/:queueName now reports in_flight (PENDING) alongside queue_length (ENQUEUED). The two numbers mean different things for scaling and are wired to separate triggers: backlog scales up, in-flight sets a replica floor (ceil(in_flight / workerConcurrency)) so running work can never be scaled out from under itself.

in_flight is a floor while DBOS recovery is healthy — per-pod PENDING is capped at workerConcurrency by the dequeue, so it can never ask for more replicas than are already busy. It is not unconditional: a PENDING row survives its executor's death until recovery re-enqueues it, so a rollout briefly counts dead pods' rows too. That is bounded by a 30-minute dequeuedAfter window (above the 10-minute run-liveness ceiling, so it never clips a real run) rather than left to pin a replica forever. Both queries pass loadInput: false — they count rows, and a hosted-harness input is the whole run request.

The chart half is decocms/deco-apps-cd#479, and it must land only after this is fully rolled out: KEDA resolves valueLocation with gjson, an absent path errors rather than reading 0, and the HPA then freezes at its current replica count.

An e2e (packages/e2e/tests/dbos-queue-depth.spec.ts) pins both field names: they are a wire contract with a ScaledObject in another repo, where a rename would be silent.

Follow-up commit: the two edge paths

The grace's wait is lock-free, so its result described a pointer that could already be stale by the time it returned. Two ways that went wrong:

• A run that finished with its client still reading has delivered its terminal, and release retains nothing (no buffer to hand on). The dispatch behind it adopted that drained entry — zero frames, no terminal — and the thread never settled.
• An incumbent replaced during the wait was displaced on the strength of the old entry's state. An already-detached replacement — another reconnect — got SIGKILLed: the exact case the grace exists to prevent, one race narrower.

Both are now impossible by construction. waitForDetach reports nothing; the decision is read from the registry under the lock afterwards. In finishedRuns → collect. Gone from the map → its terminal was delivered, so this dispatch is a new turn. Still there → detached ? reattach : supersede. Two tests, each verified red against the previous commit.

Third commit: the holes the grace itself opened

A review pass on the two commits above, and this is the one that matters:

A stop pressed during the grace restarted the run. DELETE is exactly what ends the wait — cancelling closes the run's done — so the waiting dispatch woke, found the run gone, and started it. A stop button that relaunches the turn it just stopped, with that harness holding the checkout for a full run. The handler's tombstone check ran once at the door; the grace put five seconds between it and the decision. The tombstone is now re-read inside the post-wait critical section and resolves to a 410. Covered at both tiers — a Go unit test and a black-box daemon-e2e one that gets 200 (restarted) without the fix and 410 with it.

release was not atomic. It called markFinished() (which clears detached) before taking reg.mu, so a dispatch could observe a run that had finished but was still listed in activeRuns, read it as "attached", take it over, and discard a terminal that had already been produced. All of release now runs under one lock, close(done) included.

Plus: the takeover log reported the grace constant instead of the measured wait; one test passed via the wrong branch and never checked the successor got a terminal; another swapped a replacement into the registry before detaching it, so a poll landing between the two would see a live second writer.

---

Summary by cubic

Fixes runs restarting mid-job and losing work when KEDA scales in the worker holding them. The daemon has no attach verb, so a reconnect and a competing turn arrive as the same POST /dispatch; a dispatch for a live run now waits up to 5s for the incumbent to hang up and reattaches instead of taking over. The queue-depth endpoint also now reports in-flight work, so KEDA stops scaling in a worker that is merely busy rather than backed up.

Daemon reattach

• A genuine double-dispatch still supersedes, just 5s later.
• A run that finishes during the wait is collected, not re-run.
• The wait's decision is re-read from the registry afterwards, so a replaced or drained entry can't be adopted by mistake.
• A stop pressed during the wait resolves to a tombstone instead of restarting the run it just cancelled.

KEDA trigger

• /dbos-queue-depth/:queue now returns queue_length and in_flight (PENDING count).
• in_flight counts only runs dequeued within the last 30 minutes, so a stuck PENDING row ages out of the replica floor instead of pinning capacity forever.
• The decocms/deco-apps-cd ScaledObject needs the new in_flight trigger — deploy the chart side first, or the scaler errors on the missing field.

This is a mitigation: the protocol should gain an explicit attach verb so a reconnect can never look like a competing turn.

^{Written for commit d22a425. Summary will update on new commits.}