Skip to content

fix: Release a Rejected Lane's Fence Without Its Cleanup Acknowledgement - #401

Merged
danny-avila merged 1 commit into
mainfrom
danny-avila/lane-reject-fence
Nov 16, 2026
Merged

danny-avila merged 1 commit into
mainfrom
danny-avila/lane-reject-fence

Conversation

@danny-avila

Copy link
Copy Markdown
Collaborator

What broke

A linked-worktree lane on a paired worker could stay fenced forever after the worker rejected an assignment. The lane's fence key kept the plain assignment id with no TTL, so:

  • every later request on that lane failed WORKSPACE_QUARANTINED, and
  • because a checkout refuses to start while any indexed lane fence exists, every request on the parent checkout failed WORKSPACE_QUARANTINED ("Bridge workspace already has incomplete stateful work") until the key was deleted by hand.

Seen in production: a new lane was rejected with INVALID_REQUEST ("No linked worktree named … in .worktrees"), and the parent checkout was blocked for over four hours.

Trigger

For a slot-scoped (lane) fence, the result commit released the fence only once the worker had also acknowledged local cleanup (localCleanup). If the worker's concurrent run stopped after the rejection was committed but before that acknowledgement was sent, nothing re-sent it: on reconnect the server re-offers only unsettled assignments. The fence then outlived everything that could clear it. A sibling lane that was in flight at the same moment was quarantined in the same way.

The same rejection on another lane a few seconds earlier released normally, because its cleanup acknowledgement landed before any abort.

Fix

A worker rejects only work that left the root unchanged, so a rejected settlement no longer waits for the cleanup acknowledgement. The result-commit script now deletes the fence when the settlement is rejected, still by compare-and-delete on the assignment's own id:

Outcome Fence
rejected released at result commit (new)
fulfilled released once cleanup is acknowledged (unchanged)
quarantine receipt (before or after the rejection) kept; the receipt value never matches the id (unchanged)
interrupted / never settled kept (unchanged)
lost result commit still throws and quarantines (unchanged)

After a rejection, the server may admit the next request on that lane before the worker has cleared its local guard. The worker already makes that request wait for the earlier one, as it does for non-lane workspaces.

Tests

service/src/workspace-tools/requests.test.ts (runs against real Redis in CI):

  • Production shape: execute_command and list_files are queued on a new lane, and the first is rejected with no cleanup acknowledgement. The fence is released, the queued request gets the slot, a late acknowledgement leaves the new owner's fence alone, and the parent checkout runs. This test fails before the fix.
  • Still quarantined: (a) rejected, then a quarantine receipt arrives; (b) interrupted, never settled. In both, the parent checkout still fails WORKSPACE_QUARANTINED.

concurrent-store.test.ts: "committed results retain the root fence until cleanup" now uses a fulfilled result, the case it was written for.

Results:

  • Real Redis 7.2: src/bridge/ plus requests.test.ts, 337 pass, 0 fail.
  • Full suite: 1269 pass, 5 fail. The same 5 fail on main (job-cancellation-commit, two tool-input-signature jq tests).
  • Typecheck and lint: no new errors in the changed files.

Not in this PR

  • A worker that stops right after a fulfilled result can leave a lane fence the same way.
  • The root cause is on the worker: it should still send the cleanup acknowledgement when its run aborts, and retry a failed registration heartbeat once instead of stopping every slot. That belongs in packages/code/src/worker.ts as a follow-up.

A worker rejects only work that left the root unchanged, yet a slot-scoped
fence was released only after the worker also acknowledged local cleanup.
When a worker's concurrent run stopped between a committed rejection and
that acknowledgement, nothing ever re-sent it: the lane kept a plain fence
with no TTL, and its parent checkout refused every request as quarantined.

The result commit now releases the fence for a rejected settlement, still by
compare-and-delete on the assignment's own id, so a quarantine receipt or a
newer owner is never released. Fulfilled results keep waiting for cleanup,
and interrupted or reported-uncertain outcomes still quarantine.
@danny-avila

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-16T23:44:11.983914Z 61bdc99 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Can't wait for the next one!

Reviewed commit: 61bdc99706

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@danny-avila

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Keep it up!

Reviewed commit: 61bdc99706

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@danny-avila

Copy link
Copy Markdown
Collaborator Author

@codex review — final review. Please check that a rejected lane settlement releases only its own fence (never a quarantine marker or a newer owner) and that uncertain outcomes still quarantine.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Nice work!

Reviewed commit: 61bdc99706

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant