Repository navigation
🧵 feat: Run Background Commands Without Holding Their Worktree - #408
Conversation
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
Codex Review: Didn't find any major issues. You're on a roll. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
|
Codex Review: Didn't find any major issues. You're on a roll. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
|
@codex review Final review of head 7b71312: flag only blocking issues. Please look hardest at the invariants rather than style: (1) slot accounting and fence release for detached keys in |
|
Codex Review: Didn't find any major issues. Bravo. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
|
@codex review Final review of head a2faea1: flag only blocking issues. Since the last review this branch merged |
|
Codex Review: Didn't find any major issues. Another round soon, please! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
Summary
A background shell command no longer holds its worktree while it runs. A request marked
detached: truetakes a machine slot but neither waits for nor blocks the other operations of the same worktree.The limitation
Every workspace operation is admitted under its worktree's isolation key, one at a time. That is correct for an edit or a short command, but a background command (a build, a test suite, a dev server) holds the key for its whole run. Everything else the same conversation wants to do in that worktree queues behind it: the next read, the next edit, a subagent's command.
On one paired machine over two weeks, workspace requests spent 51.7 h queued against 42.8 h running. About two thirds of the long waits were for the request's own worktree, and 73% of that time was behind a command already in progress there (median blocker 48 s). Raising the slot count from 4 to 8 would have removed at most 8% of the wait, because the wait is for the key and not for a slot.
What changes
execute_commandacceptsdetached: true(onlytrue, and never withenvironmentAction). A detached command:\0detached-command\0<assignmentId>. It conflicts with nothing, so it does not wait for its worktree and nothing waits for it. It still takes one of the worker's slots.slots - 1slots of a worker; the next one queues until one finishes.<marker>.detached/<assignmentId>.json. A marker left by an earlier worker incarnation, or by an assignment that is no longer running, makes the worktree unavailable exactly as an ordinary stale marker does.The caller decides which commands are detached. LibreChat marks only commands the model submitted as background calls (LibreChat-AI/LibreChat, separate PR). A detached command and ordinary operations share the working tree unserialized, which is the point; the tool description on the LibreChat side tells the model not to background a command that later commands depend on.
How it works
Negotiation. The worker advertises
workspaceTools.commandFeatures: ['detached']. Code API answers registration withsupportedWorkspaceCommandFeatures, the worker re-registers with the intersection, and the worker status endpoint reports what was registered, so LibreChat sendsdetachedonly to a worker that runs it. There is no protocol version bump: an old worker never advertises the feature, and an old Code API never offers it.Admission (
slots.ts). The scheduler testsbusy[key]first and applies the cap only to a detached key:A detached key never becomes a waiting checkout, so it cannot hold back a lane or the checkout above one.
Fences (
store.ts). An ordinary assignment's fence has no TTL, because the worktree must stay closed until its owner releases it. A detached command's fence is keyed by assignment, so nothing else can ever own it; it and itsquarantined:receipt expire with the assignment outcome. A detached request is refused up front when its worktree, or the checkout above it, is already quarantined.Worker occupancy (
worker.ts). The detached assignment is tracked under the detached key for "one operation per key", and under the worktree key for quarantine: an uncertain outcome quarantines the worktree. A lost cleanup acknowledgement alone does not, because the command had already settled and its marker was cleared.Defaults and opt-outs
On by default at every layer; each has an opt-out.
CODEAPI_BRIDGE_DETACHED_COMMANDS=false--no-detached-commandsorLIBRECHAT_CODE_DETACHED_COMMANDS=falseCompatibility
detachedrequest for it is refused instead of being run as an ordinary command.detached. Nothing sends it until the worker status reports the feature, which itself requires a replica with this change to have answered registration.Trade-offs
.detachedmarker directories are added to the sandbox's protected paths for every native-srt worker.Testing
New tests:
service/src/bridge/detached.test.ts(live Redis): a detached command overlaps its lane and its parent checkout; detached commands never take the last slot; a detached command's quarantine receipt expires and leaves its lane usable; a detached request is refused while its worktree or the checkout above it is quarantined, when the worker does not advertise the feature with spare slots, and on a replica that opted out.service/src/bridge/slots.test.ts,router.test.ts,service/src/workspace-tools/requests.test.ts: scheduler ordering with an admitted detached request still queued, the registration offer and its opt-out, durable request admission ids.packages/code/src/worker-detached.test.ts: drives the realBridgeWorker— a detached command runs beside an ordinary command on the same worktree, an interrupted one quarantines the worktree, a lost cleanup acknowledgement alone does not, and an unadvertised feature is rejected.packages/code/src/workspace-guards.test.ts,native-pool.test.ts,protocol.test.ts,workspace-cli.test.ts,linked-worktrees.test.ts,workspace-instances.test.ts: marker ownership across incarnations, the dedicated executor and its teardown, request and capability validation, the CLI opt-out.Runs on this head:
packages/code:tsc --noEmitclean.npm test: 1,058 tests, 1,027 pass, 22 skipped, 9 fail. The same 9 fail onmainlocally (programmatic executor and storage-permission tests that depend on the environment).service:bun test ./src/bridge ./src/workspace-toolsagainst a live Redis 7: 433 pass, 0 fail across 26 files.bun run buildsucceeds.tests/compose-bridge-config.cjs(needs Docker), the macOS storage jobs, and the live-sandbox and packed-worker smoke jobs. Nothing here has run against a deployed Code API and LibreChat yet.