Repository navigation
fix: Release a Rejected Lane's Fence Without Its Cleanup Acknowledgement - #401
Conversation
A worker rejects only work that left the root unchanged, yet a slot-scoped fence was released only after the worker also acknowledged local cleanup. When a worker's concurrent run stopped between a committed rejection and that acknowledgement, nothing ever re-sent it: the lane kept a plain fence with no TTL, and its parent checkout refused every request as quarantined. The result commit now releases the fence for a rejected settlement, still by compare-and-delete on the assignment's own id, so a quarantine receipt or a newer owner is never released. Fulfilled results keep waiting for cleanup, and interrupted or reported-uncertain outcomes still quarantine.
|
@codex review |
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
Codex Review: Didn't find any major issues. Can't wait for the next one! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
|
@codex review |
|
Codex Review: Didn't find any major issues. Keep it up! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
|
@codex review — final review. Please check that a rejected lane settlement releases only its own fence (never a quarantine marker or a newer owner) and that uncertain outcomes still quarantine. |
|
Codex Review: Didn't find any major issues. Nice work! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
What broke
A linked-worktree lane on a paired worker could stay fenced forever after the worker rejected an assignment. The lane's fence key kept the plain assignment id with no TTL, so:
WORKSPACE_QUARANTINED, andWORKSPACE_QUARANTINED("Bridge workspace already has incomplete stateful work") until the key was deleted by hand.Seen in production: a new lane was rejected with
INVALID_REQUEST("No linked worktree named … in .worktrees"), and the parent checkout was blocked for over four hours.Trigger
For a slot-scoped (lane) fence, the result commit released the fence only once the worker had also acknowledged local cleanup (
localCleanup). If the worker's concurrent run stopped after the rejection was committed but before that acknowledgement was sent, nothing re-sent it: on reconnect the server re-offers only unsettled assignments. The fence then outlived everything that could clear it. A sibling lane that was in flight at the same moment was quarantined in the same way.The same rejection on another lane a few seconds earlier released normally, because its cleanup acknowledgement landed before any abort.
Fix
A worker rejects only work that left the root unchanged, so a rejected settlement no longer waits for the cleanup acknowledgement. The result-commit script now deletes the fence when the settlement is
rejected, still by compare-and-delete on the assignment's own id:After a rejection, the server may admit the next request on that lane before the worker has cleared its local guard. The worker already makes that request wait for the earlier one, as it does for non-lane workspaces.
Tests
service/src/workspace-tools/requests.test.ts(runs against real Redis in CI):execute_commandandlist_filesare queued on a new lane, and the first is rejected with no cleanup acknowledgement. The fence is released, the queued request gets the slot, a late acknowledgement leaves the new owner's fence alone, and the parent checkout runs. This test fails before the fix.WORKSPACE_QUARANTINED.concurrent-store.test.ts: "committed results retain the root fence until cleanup" now uses a fulfilled result, the case it was written for.Results:
src/bridge/plusrequests.test.ts, 337 pass, 0 fail.main(job-cancellation-commit, twotool-input-signaturejq tests).Not in this PR
packages/code/src/worker.tsas a follow-up.