Skip to content

📨 fix: Deliver Lane Cleanup and Quarantine Receipts When a Worker Run Stops - #402

Merged
danny-avila merged 1 commit into
mainfrom
danny-avila/worker-ownership-delivery
Nov 17, 2026
Merged

danny-avila merged 1 commit into
mainfrom
danny-avila/worker-ownership-delivery

Conversation

@danny-avila

Copy link
Copy Markdown
Collaborator

Summary

Follow-up to #401. That PR stopped the Code API from waiting on a lost cleanup acknowledgement after a rejected lane settlement. This PR fixes the worker so the acknowledgement, or a quarantine receipt, is not lost in the first place. It also covers fulfilled results, which #401 does not.

The breakage

A linked-worktree lane (and, the same way, any root under concurrent workspace slots) holds a server-side fence for each assignment. After a settlement, Code API releases that fence only when the worker either acknowledges local cleanup (workspace-cleanup) or reports the lane as quarantined (quarantine). Until then, the lane refuses new work, and so does the checkout above it (WORKSPACE_QUARANTINED).

On the worker, both of those calls were bound to the concurrent run's abort signal. If the run stopped after a settlement had committed but before the call went out, reportWorkspaceOwnership gave up without sending anything. The fallback quarantine receipt then gave up the same way. On reconnect, Code API offers only unsettled assignments again, so nothing ever re-sent the call. The fence stayed, with no TTL, until an operator reset it.

The trigger

Any failure that stops the concurrent run does this to every slot. The most likely one is a single failed registration heartbeat: inside runConcurrent the heartbeat had no retry and stopped all slots on its first transient error.

On the hosted bridge, an execute_command on a lane was rejected by the worker. One second later a sibling lane's in-flight command was quarantined. That pattern matches a run stop followed by the worker's one-second reconnect. The rejected lane's fence then held its plain assignment id for hours, the checkout above it refused every request, and a request queued behind it on the same lane failed only when its slot reservation expired 65 s later.

The fix (packages/code/src/worker.ts)

  • Cleanup acknowledgement: once the settlement has committed, it is sent with its existing bounded retry (3 attempts) and no longer depends on the run's abort signal.
  • Quarantine receipt: sent the same way, even when the run is stopping. If delivery is still uncertain (transport failure or 5xx), the receipt is kept in memory. The next run delivers it right after registration, before any slot leases. A definite refusal (non-retryable 4xx) drops it.
  • Heartbeat: a concurrent run now absorbs one transient registration heartbeat failure, retrying after the existing 100 ms delay, before it stops every slot. Terminal failures (401, 403, WORKER_FENCED, WORKER_QUARANTINED) still stop immediately. The two callers that already retried transient failures without limit keep that behavior (Number.POSITIVE_INFINITY).

Uncertain outcomes still quarantine. An interrupted command still quarantines its local guard and is never settled as fulfilled. What changes is that its quarantine receipt now actually reaches Code API when the run is stopping, instead of leaving a plain fence behind.

Compatibility

There is no protocol change and no opt-in flag. These are the same endpoints and bodies, which every Code API since concurrent workspace slots already accepts. A late workspace-cleanup after #401 is idempotent. A late quarantine is still checked against the assignment's receipt and the workspace's reset epoch, so it cannot fence a root that was reset after the assignment.

Trade-offs

  • A stopping worker may now wait for these bounded deliveries: up to 3 attempts of the control transport timeout (10 s by default) per call.
  • Kept receipts live in memory. A process restart loses them, and the lane then stays fenced until reset, as before.

Testing

  • New packages/code/src/worker-receipts.test.ts drives the real BridgeWorker.run against a fixture bridge that records every control call:
    • A committed read_file and a committed execute_command result still send workspace-cleanup when the run stops right after the settlement.
    • An interrupted command still quarantines its local guard, and its quarantine receipt is delivered.
    • A receipt lost to a transport outage while stopping is delivered first on the next run, and only once.
    • One failed heartbeat no longer aborts an in-flight assignment; two consecutive failures still stop every slot.
  • 5 of the 6 tests fail on main. The sixth, "two consecutive heartbeat failures still stop every slot", passes on main too and guards against the retry becoming unbounded. All 6 pass with this change; I ran the file three times.
  • packages/code npm test: 1000 pass, 9 fail. The same 9 fail on main locally (programmatic executor and storage-permission tests that depend on the environment).
  • Service tests that run the real worker (src/bridge/concurrent-worker.test.ts, src/bridge/instructions.test.ts): 8 pass.
  • tsc -p packages/code/tsconfig.json --noEmit is clean.

…tops

A concurrent worker run that stopped after a lane's settlement committed
dropped the cleanup acknowledgement and the quarantine receipt, because both
were bound to the run's abort signal. Code API offers only unsettled work on
reconnect, so nothing re-sent them and the lane's fence stayed until reset,
also blocking its parent checkout. One failed registration heartbeat was
enough to stop every slot this way.

The cleanup acknowledgement for a committed settlement and the quarantine
receipt are now delivered with their bounded retries regardless of the run
stopping. A quarantine receipt whose delivery stays uncertain is kept and
delivered after the next registration, before any slot leases. A concurrent
run absorbs one transient heartbeat failure before stopping its slots.
Uncertain outcomes still quarantine; the receipt now also reaches Code API.
@danny-avila

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-17T00:07:20.636506Z eee9e31 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. 👍

Reviewed commit: eee9e31a79

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@danny-avila

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. 🚀

Reviewed commit: eee9e31a79

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@danny-avila

Copy link
Copy Markdown
Collaborator Author

@codex review — final review. Please check that receipts for already-settled assignments survive a stopping run without weakening quarantine of uncertain outcomes, and that the one-retry heartbeat stays bounded.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Delightful!

Reviewed commit: eee9e31a79

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@danny-avila
danny-avila merged commit c0f98af into main Nov 17, 2026
31 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant