Skip to content

🧵 feat: Run Background Commands Without Holding Their Worktree - #408

Merged
danny-avila merged 2 commits into
mainfrom
danny-avila/detached-commands
Nov 17, 2026
Merged

danny-avila merged 2 commits into
mainfrom
danny-avila/detached-commands

Conversation

@danny-avila

Copy link
Copy Markdown
Collaborator

Summary

A background shell command no longer holds its worktree while it runs. A request marked detached: true takes a machine slot but neither waits for nor blocks the other operations of the same worktree.

The limitation

Every workspace operation is admitted under its worktree's isolation key, one at a time. That is correct for an edit or a short command, but a background command (a build, a test suite, a dev server) holds the key for its whole run. Everything else the same conversation wants to do in that worktree queues behind it: the next read, the next edit, a subagent's command.

On one paired machine over two weeks, workspace requests spent 51.7 h queued against 42.8 h running. About two thirds of the long waits were for the request's own worktree, and 73% of that time was behind a command already in progress there (median blocker 48 s). Raising the slot count from 4 to 8 would have removed at most 8% of the wait, because the wait is for the key and not for a slot.

What changes

execute_command accepts detached: true (only true, and never with environmentAction). A detached command:

  • Is scheduled on its own key, \0detached-command\0<assignmentId>. It conflicts with nothing, so it does not wait for its worktree and nothing waits for it. It still takes one of the worker's slots.
  • Cannot starve ordinary work. Detached commands hold at most slots - 1 slots of a worker; the next one queues until one finishes.
  • Runs in its own executor on the worker, with its own sandbox process and scratch, separate from the worktree's ordinary executor.
  • Still quarantines its worktree if the worker dies mid-command. It arms its own durable marker, <marker>.detached/<assignmentId>.json. A marker left by an earlier worker incarnation, or by an assignment that is no longer running, makes the worktree unavailable exactly as an ordinary stale marker does.

The caller decides which commands are detached. LibreChat marks only commands the model submitted as background calls (LibreChat-AI/LibreChat, separate PR). A detached command and ordinary operations share the working tree unserialized, which is the point; the tool description on the LibreChat side tells the model not to background a command that later commands depend on.

How it works

packages/code/src/
├── protocol.ts          # `detached` on the request, `commandFeatures`, workspaceDetachedCommandKey()
├── worker.ts            # occupancy under the detached key, quarantine checks under the worktree key
├── native-pool.ts       # executeDetached(): one dedicated executor per detached command
├── workspace-guards.ts  # per-assignment marker beside the worktree's marker
└── storage.ts           # marker files, per-path serialisation

service/src/bridge/
├── slots.ts             # Lua: a detached key conflicts only with the detached cap
├── store.ts             # admission id, fence with TTL, capability gate
└── router.ts            # offers `supportedWorkspaceCommandFeatures`

Negotiation. The worker advertises workspaceTools.commandFeatures: ['detached']. Code API answers registration with supportedWorkspaceCommandFeatures, the worker re-registers with the intersection, and the worker status endpoint reports what was registered, so LibreChat sends detached only to a worker that runs it. There is no protocol version bump: an old worker never advertises the feature, and an old Code API never offers it.

Admission (slots.ts). The scheduler tests busy[key] first and applies the cap only to a detached key:

if busy[key] then return true end
if isDetached(key) then return detachedHeld >= capacity - 1 end

A detached key never becomes a waiting checkout, so it cannot hold back a lane or the checkout above one.

Fences (store.ts). An ordinary assignment's fence has no TTL, because the worktree must stay closed until its owner releases it. A detached command's fence is keyed by assignment, so nothing else can ever own it; it and its quarantined: receipt expire with the assignment outcome. A detached request is refused up front when its worktree, or the checkout above it, is already quarantined.

Worker occupancy (worker.ts). The detached assignment is tracked under the detached key for "one operation per key", and under the worktree key for quarantine: an uncertain outcome quarantines the worktree. A lost cleanup acknowledgement alone does not, because the command had already settled and its marker was cleared.

Defaults and opt-outs

On by default at every layer; each has an opt-out.

Layer Opt-out Also off when
Code API CODEAPI_BRIDGE_DETACHED_COMMANDS=false the bridge is configured with a single workspace slot
Worker --no-detached-commands or LIBRECHAT_CODE_DETACHED_COMMANDS=false the executor pool is off or the worker has one slot

Compatibility

  • Old worker, new Code API: the worker does not advertise the feature; a detached request for it is refused instead of being run as an ordinary command.
  • New worker, old Code API: the feature is not offered, so the worker drops it on re-registration.
  • Mixed Code API replicas: a replica without this change rejects the new request field. Deploy Code API everywhere before any caller sends detached. Nothing sends it until the worker status reports the feature, which itself requires a replica with this change to have answered registration.

Trade-offs

  • While a detached command runs, Code API holds no fence on its worktree. After an ambiguous failure the worker quarantines the worktree locally at once; Code API learns of it from the next ordinary operation's receipt.
  • The executor pool is sized to the slot count, so starting a detached command may close an idle executor for another worktree, which pays start-up again on its next command.
  • .detached marker directories are added to the sandbox's protected paths for every native-srt worker.
  • Marker operations for one path are serialised in-process; a hung filesystem call on that path would stall the others for it.

Testing

New tests:

  • service/src/bridge/detached.test.ts (live Redis): a detached command overlaps its lane and its parent checkout; detached commands never take the last slot; a detached command's quarantine receipt expires and leaves its lane usable; a detached request is refused while its worktree or the checkout above it is quarantined, when the worker does not advertise the feature with spare slots, and on a replica that opted out.
  • service/src/bridge/slots.test.ts, router.test.ts, service/src/workspace-tools/requests.test.ts: scheduler ordering with an admitted detached request still queued, the registration offer and its opt-out, durable request admission ids.
  • packages/code/src/worker-detached.test.ts: drives the real BridgeWorker — a detached command runs beside an ordinary command on the same worktree, an interrupted one quarantines the worktree, a lost cleanup acknowledgement alone does not, and an unadvertised feature is rejected.
  • packages/code/src/workspace-guards.test.ts, native-pool.test.ts, protocol.test.ts, workspace-cli.test.ts, linked-worktrees.test.ts, workspace-instances.test.ts: marker ownership across incarnations, the dedicated executor and its teardown, request and capability validation, the CLI opt-out.

Runs on this head:

  • packages/code: tsc --noEmit clean. npm test: 1,058 tests, 1,027 pass, 22 skipped, 9 fail. The same 9 fail on main locally (programmatic executor and storage-permission tests that depend on the environment).
  • service: bun test ./src/bridge ./src/workspace-tools against a live Redis 7: 433 pass, 0 fail across 26 files. bun run build succeeds.
  • Not run locally: tests/compose-bridge-config.cjs (needs Docker), the macOS storage jobs, and the live-sandbox and packed-worker smoke jobs. Nothing here has run against a deployed Code API and LibreChat yet.

@danny-avila

Copy link
Copy Markdown
Collaborator Author

@codex review

Head: 7b71312

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-17T12:39:29.188159Z a2faea1 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. You're on a roll.

Reviewed commit: 7b71312afb

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@danny-avila

Copy link
Copy Markdown
Collaborator Author

@codex review

Head: 7b71312

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. You're on a roll.

Reviewed commit: 7b71312afb

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@danny-avila

Copy link
Copy Markdown
Collaborator Author

@codex review

Final review of head 7b71312: flag only blocking issues. Please look hardest at the invariants rather than style: (1) slot accounting and fence release for detached keys in service/src/bridge/slots.ts and service/src/bridge/store.ts, including a worker that disappears mid-command; (2) the worker quarantining a worktree after an uncertain detached outcome, and never treating a stale .detached marker as available (packages/code/src/worker.ts, workspace-guards.ts, storage.ts); (3) mixed-version behaviour between Code API replicas and workers during a rolling deploy.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Bravo.

Reviewed commit: 7b71312afb

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@danny-avila

Copy link
Copy Markdown
Collaborator Author

@codex review

Final review of head a2faea1: flag only blocking issues. Since the last review this branch merged main (#406, #407). The only conflict was in service/src/workspace-tools/requests.test.ts, where both sides added tests and both are kept. Please check that #407's fence-epoch increment in enqueueForActiveIncarnation (service/src/bridge/store.ts) is correct for a detached command, whose fence and epoch keys are per assignment.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Another round soon, please!

Reviewed commit: a2faea1f16

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@danny-avila
danny-avila merged commit c7bf37d into main Nov 17, 2026
31 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant