Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions docker-compose.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ services:
- CODEAPI_BRIDGE_DYNAMIC_WORKERS=${CODEAPI_BRIDGE_DYNAMIC_WORKERS:-true}
- CODEAPI_BRIDGE_MAX_WORKSPACE_LEASE_SLOTS=${CODEAPI_BRIDGE_MAX_WORKSPACE_LEASE_SLOTS:-1}
- CODEAPI_BRIDGE_LANE_GIT=${CODEAPI_BRIDGE_LANE_GIT:-false}
- CODEAPI_BRIDGE_DETACHED_COMMANDS=${CODEAPI_BRIDGE_DETACHED_COMMANDS:-true}
- CODEAPI_BRIDGE_IMPORT_FILE_MAX_BYTES=${CODEAPI_BRIDGE_IMPORT_FILE_MAX_BYTES:-104857600}
- CODEAPI_BRIDGE_WORKER_ID=${CODEAPI_BRIDGE_WORKER_ID:-}
- CODEAPI_BRIDGE_RECOVERY_SERVER_ID=${CODEAPI_BRIDGE_RECOVERY_SERVER_ID:-}
Expand Down Expand Up @@ -72,6 +73,7 @@ services:
- CODEAPI_BRIDGE_DYNAMIC_WORKERS=${CODEAPI_BRIDGE_DYNAMIC_WORKERS:-true}
- CODEAPI_BRIDGE_MAX_WORKSPACE_LEASE_SLOTS=${CODEAPI_BRIDGE_MAX_WORKSPACE_LEASE_SLOTS:-1}
- CODEAPI_BRIDGE_LANE_GIT=${CODEAPI_BRIDGE_LANE_GIT:-false}
- CODEAPI_BRIDGE_DETACHED_COMMANDS=${CODEAPI_BRIDGE_DETACHED_COMMANDS:-true}
- CODEAPI_BRIDGE_WORKER_ID=${CODEAPI_BRIDGE_WORKER_ID:-}
- CODEAPI_BRIDGE_RECOVERY_SERVER_ID=${CODEAPI_BRIDGE_RECOVERY_SERVER_ID:-}
- CODEAPI_BRIDGE_ENROLLMENT_TTL_SECONDS=${CODEAPI_BRIDGE_ENROLLMENT_TTL_SECONDS:-0}
Expand Down
7 changes: 7 additions & 0 deletions docs/remote-bridge/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -366,6 +366,13 @@ execution.
(`worktree: <name>`) nests beneath its checkout's key: sibling lanes run
concurrently, while a lane and its checkout exclude each other, and a lane
cannot start while its checkout is quarantined.
- An `execute_command` request marked `detached` takes a slot under a key of
its own instead of its worktree's, so other operations there are admitted
while it runs and it is admitted while they run. Detached commands may hold
at most one slot fewer than the negotiated count. Code API offers the feature
only above one slot (`CODEAPI_BRIDGE_DETACHED_COMMANDS=false` opts out) and
refuses the field for a worker that does not advertise
`commandFeatures: ['detached']`.
- Workspace tool admission waits for capacity up to 30 seconds without a
`X-LibreChat-Workspace-Queue-Wait-Ms` header. A caller can advertise a longer
per-request allowance, bounded by five minutes and any server queue ceiling.
Expand Down
50 changes: 48 additions & 2 deletions packages/code/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -866,7 +866,8 @@ librechat-code run \

Slots are per machine, not a fleet-wide execution limit. A busy machine does not
consume another machine's slots. Requests for the same root remain serialized,
including commands started through background tools. Independent checkouts can
except a command the caller marks detached (see
[Detached commands](#detached-commands)). Independent checkouts can
use different slots; selecting subdirectories beneath one registered parent root
does not create separate scheduling boundaries.

Expand Down Expand Up @@ -1080,6 +1081,51 @@ retain the selected root's sandbox boundary, not a shared parent-directory grant
The `LIBRECHAT_CODE_WORKSPACE_QUARANTINE_FILE` single-file override is rejected
when multiple roots are configured; unset it to use separate root-derived markers.

#### Detached commands

A long test or build started in the background would otherwise hold its root,
checkout or lane for its whole run, and every later read, edit or command there
would queue behind it. A caller can instead mark an `execute_command` request
`detached: true`. A detached command still takes one slot, but it does not hold
its worktree: other operations in that worktree, its checkout and its lanes are
admitted and run while it runs, and it is admitted while they run. Its timeout,
output, cancellation, settlement and result are those of any other command.

Nothing serializes a detached command with the work beside it, so the caller is
responsible for choosing commands that tolerate concurrent edits, such as a test
run or a build, rather than a formatter or a `git` operation that rewrites the
tree.

The feature is on by default wherever it can hold: Code API offers it when
`CODEAPI_BRIDGE_MAX_WORKSPACE_LEASE_SLOTS` is above 1, and a worker advertises
`commandFeatures: ['detached']` when it runs native-srt commands with
`--workspace-lease-slots` above 1 and Code API offered it. LibreChat reads the
advertisement from the worker status and sends `detached` only then; a request
that carries it for a worker that does not advertise it is refused, never run
serialized. Set `CODEAPI_BRIDGE_DETACHED_COMMANDS=false` on every Code API
replica, or start the worker with `--no-detached-commands`
(`LIBRECHAT_CODE_DETACHED_COMMANDS=false`), to opt out. An older Code API never
offers the feature and an older worker never advertises it, so either side can
be upgraded first.

- Detached commands may hold at most one slot fewer than the negotiated count,
so one slot always remains for other work. Further detached commands wait
without holding back the requests queued behind them.
- Each detached command runs in its own SRT executor for the same root, with
the root's sandbox policy, created for the command and closed after it. It
counts toward the executor bound, and it never shares the root's own executor,
scratch directory or credentials.
- Each detached command arms its own durable marker in a `.detached` directory
beside its worktree's quarantine marker, and clears it after settlement. If the
worker stops while one runs, or one ends ambiguously, the marker stays and the
worktree is quarantined exactly as for any other uncertain mutation. A detached
command also refuses to start in a quarantined worktree.
`clear-workspace-quarantine` removes these markers together with the worktree's
own, and `--reset-workspace-quarantine` refuses while any remain.
- A detached command has no server-side fence on its worktree. Its own fence, and
the receipt left if it is quarantined, expire with the request's outcome instead
of waiting for a reset.

`LIBRECHAT_CODE_WORKSPACE_LEASE_SLOTS` is the equivalent worker setting. Both
ceilings must be integers from 1 to 8; the lower ceiling wins. An older Code API
without the negotiation receipt keeps the worker on the serial protocol. Deploy
Expand All @@ -1088,7 +1134,7 @@ change while work is active fails closed; stop and drain the worker before
changing it.

Different roots can run concurrently; requests targeting the **same root remain
serialized**, even across chats or agents. This is root-level exclusion, not
serialized**, even across chats or agents, apart from detached commands. This is root-level exclusion, not
file-level locking. Assign separate project/worktree roots for independent work.
The admission queue remains bounded at 32 requests per worker. An idle SRT process
cache is bounded by the local slot setting and evicts only idle executors. Runtime
Expand Down
27 changes: 19 additions & 8 deletions packages/code/src/cli.ts
Original file line number Diff line number Diff line change
Expand Up @@ -31,9 +31,10 @@ import {
defaultBridgeIdentityPath,
defaultWorkspaceQuarantinePath,
defaultWorkspacePath,
detachedWorkspaceMutationQuarantineDirectory,
ensurePrivateWorkspaceDirectory,
loadBridgeIdentity,
loadWorkspaceMutationQuarantine,
loadWorkspaceMutationQuarantines,
saveBridgeIdentity,
saveWorkspaceMutationQuarantine,
prepareEnvironmentPreparationDirectory,
Expand All @@ -47,7 +48,7 @@ import {
} from './runtime.js';
import { RuntimeWorkspaceCommandSandbox } from './workspace-runtime.js';
import { NativeProcessWorkspaceCommandSandbox } from './native-process.js';
import { NativeWorkspaceCommandPool } from './native-pool.js';
import { NativeWorkspaceCommandPool, detachedCommandsEnabled } from './native-pool.js';
import { GitWorktreeWorkspaceTools, internalWorkspaceId } from './workspace-instances.js';
import { LaneGitWorkspaceTools, ownsGitMetadata, ownsLinkedWorktreeMetadata } from './lane-git.js';
import { LINKED_WORKTREE_DIRECTORY, LinkedWorktreeWorkspaceTools, verifyLinkedWorktree } from './linked-worktrees.js';
Expand Down Expand Up @@ -856,6 +857,10 @@ async function run(
}),
]),
);
const rootQuarantineStatePaths = [...rootQuarantinePaths.values()].flatMap((path) => [
path,
detachedWorkspaceMutationQuarantineDirectory(path),
]);
for (const path of setupProtectedPaths) {
const namespace = await lstat(path).catch((error: NodeJS.ErrnoException) => {
if (error.code === 'ENOENT') return undefined;
Expand Down Expand Up @@ -885,13 +890,13 @@ async function run(
})));
await assertEnvironmentResourceIsolation(environments.flatMap(environment => environment.resources ?? []), roots.map(root => root.root), [
identityPath, ...setupProtectedPaths, preparationDirectory, github.privateKeyPath,
...environments.map(environment => environment.path), ...rootQuarantinePaths.values(),
...environments.map(environment => environment.path), ...rootQuarantineStatePaths,
].filter((path): path is string => path != null));
await assertEnvironmentDefinitionsOutsideRoots(resourceRoots, roots);
const snapshotStores = environments.flatMap(environment => environment.snapshotStore ? [environment.snapshotStore] : []);
await assertEnvironmentResourceIsolation(snapshotStores.map(store => ({ kind: 'npm-cache' as const, path: store.store, access: 'read-only' as const })), roots.map(root => root.root), [
identityPath, ...setupProtectedPaths, preparationDirectory, github.privateKeyPath, ...environments.map(environment => environment.path),
...rootQuarantinePaths.values(), ...environments.flatMap(environment => (environment.resources ?? []).map(resource => resource.path)),
...rootQuarantineStatePaths, ...environments.flatMap(environment => (environment.resources ?? []).map(resource => resource.path)),
].filter((path): path is string => path != null));
await assertEnvironmentDefinitionsOutsideRoots(snapshotStores.map(store => ({ path: store.store, sourceParents: store.controlPaths,
definition: { name: 'dependency-store', root: store.store }, fingerprint: '' })), roots);
Expand Down Expand Up @@ -1139,7 +1144,7 @@ async function run(
...(environments.some(environment => environment.definition.setup?.reuse) ? [preparationDirectory] : []),
...snapshotStores.map(store => store.store),
...environments.map(environment => environment.path),
...rootQuarantinePaths.values(),
...rootQuarantineStatePaths,
github.privateKeyPath,
].filter((path): path is string => path != null),
allowedDomains: commandAllowedDomains,
Expand Down Expand Up @@ -1204,13 +1209,20 @@ async function run(
)
: new NativeProcessWorkspaceCommandSandbox(nativeOptionsForWorkspace(roots[0].id))
: undefined;
const detachedCommands = detachedCommandsEnabled({
optOut: args.includes('--no-detached-commands'),
enabled: process.env.LIBRECHAT_CODE_DETACHED_COMMANDS,
pooled: nativeCommandSandbox instanceof NativeWorkspaceCommandPool,
workspaceLeaseSlots,
});
if (allowWorkspaceCommands && workspaceTools) {
workspaceTools = new SandboxWorkspaceTools({
workspaceTools,
commandWorkspaces: roots.map(root => root.id),
...(nativeProgrammaticEnabled
? { programmaticLanguages: ['bash'] }
: {}),
...(detachedCommands ? { commandFeatures: ['detached'] } : {}),
commandSandbox:
nativeCommandSandbox ??
new RuntimeWorkspaceCommandSandbox({
Expand Down Expand Up @@ -1408,7 +1420,7 @@ async function run(
workerId,
workspaceRoot: join(source.root, LINKED_WORKTREE_DIRECTORY, worktree),
});
return (await loadWorkspaceMutationQuarantine(path)) != null;
return (await loadWorkspaceMutationQuarantines(path)).length > 0;
},
log(level, message) {
if (level === 'debug' && !debugLogs) return;
Expand Down Expand Up @@ -1633,8 +1645,7 @@ async function run(
roots.length === 1
? {
async assertAvailable() {
const record =
await loadWorkspaceMutationQuarantine(
const [record] = await loadWorkspaceMutationQuarantines(
mutationQuarantinePath,
);
if (record != null) {
Expand Down
66 changes: 65 additions & 1 deletion packages/code/src/linked-worktrees.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -20,11 +20,12 @@ import {
linkedWorktreeWorkspaceId,
verifyLinkedWorktree,
} from './linked-worktrees.js';
import { NativeWorkspaceCommandPool as CommandPool } from './native-pool.js';
import { LocalWorkspaceTools, WorkspaceToolError } from './workspace.js';

import type { NativeWorkspaceCommandPool } from './native-pool.js';
import type { NativeProcessSandboxOptions } from './native-process.js';
import type { WorkspaceToolRequest } from './protocol.js';
import type { WorkspaceExecuteCommandRequest, WorkspaceToolRequest } from './protocol.js';

const execFileAsync = promisify(execFile);

Expand Down Expand Up @@ -345,3 +346,66 @@ test('a removed lane releases its command root and credential route', async (t)
);
assert.deepEqual(released, [join(root, '.worktrees', 'task-a')]);
});

test('a detached lane command runs in its own executor while the lane keeps working', async (t) => {
const { parent, root } = await checkout();
t.after(() => rm(parent, { recursive: true, force: true }));
const executors: Array<{ root: string; requests: WorkspaceExecuteCommandRequest[]; closed: boolean }> = [];
let releaseBackground!: () => void;
const backgroundFinished = new Promise<void>((resolve) => { releaseBackground = resolve; });
let backgroundStarted!: () => void;
const backgroundRunning = new Promise<void>((resolve) => { backgroundStarted = resolve; });
const pool = new CommandPool(new Map([['repo', { workspaceRoot: root }]]), 2, (options) => {
const executor = { root: options.workspaceRoot, requests: [] as WorkspaceExecuteCommandRequest[], closed: false };
executors.push(executor);
return {
async prepare() {},
async close() {
executor.closed = true;
},
async execute(request) {
executor.requests.push(request);
if (request.command === 'npm test') {
backgroundStarted();
await backgroundFinished;
}
return { ...request, stdout: request.command, stderr: '', exitCode: 0, timedOut: false, truncated: false };
},
};
});
const tools = await laneTools(root, pool);
const command = {
protocolVersion: 1 as const,
operation: 'execute_command' as const,
workspaceId: 'repo',
worktree: 'task-a',
};
const id = linkedWorktreeWorkspaceId('repo', 'task-a');
const laneRoot = join(root, '.worktrees', 'task-a');

const background = tools.execute({ ...command, command: 'npm test', detached: true });
await backgroundRunning;
const ordinary = await tools.execute({ ...command, command: 'git status' });
assert.equal(ordinary.workspaceId, 'repo');
assert.deepEqual(await tools.whileIdle('repo', 'task-a', async () => 'retired'), { ran: false });
assert.deepEqual(executors, [
{
root: laneRoot,
requests: [{ protocolVersion: 1, operation: 'execute_command', workspaceId: id, command: 'npm test' }],
closed: false,
},
{
root: laneRoot,
requests: [{ protocolVersion: 1, operation: 'execute_command', workspaceId: id, command: 'git status' }],
closed: false,
},
]);

releaseBackground();
const result = await background;
assert.equal(result.workspaceId, 'repo');
assert.equal(result.operation === 'execute_command' && result.stdout, 'npm test');
assert.deepEqual(await tools.whileIdle('repo', 'task-a', async () => 'retired'), { ran: true, value: 'retired' });
assert.deepEqual(executors.map((executor) => executor.closed), [true, true]);
await pool.close();
});
Loading
Loading