Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 30 additions & 9 deletions .ai/contexts/ipc-bridge.md
Original file line number Diff line number Diff line change
Expand Up @@ -987,22 +987,43 @@ through its spawn port and a real Git bash; it skips when Git for Windows is
unavailable. It does not exercise Electron or cover every MSYS tool.
Before its first `ready`, a failed singleton settles requests false and permits
up to three startup attempts per app session (#1140), each with the same 15 s
deadline. A later assignment (or explicit `start`) retries only after a backoff
of 1 s after the first failure and 2 s after the second; there is no background
retry timer. Concurrent tabs share one attempt and its readiness promise.
Close, release and observation cannot restart a failed helper. The fallback
deadline. A later assignment (or explicit `start`) retries only after a monotonic
backoff of 1 s after the first failure and 2 s after the second. An unreferenced
timer also retries when that window expires while tracked live tabs are waiting
(#1156). Concurrent tabs share one attempt and its readiness promise.
Close, release and observation cannot restart a failed helper, even after the
backoff expires with no tracked tabs waiting. The fallback
warning is emitted once per app session, while retry attempts, their failures
and budget exhaustion are logged separately. Events from failed attempts are
ignored, including delayed ready, exit and stream errors.

Failed assignments are not replayed: tabs created during failure/backoff keep
plain ConPTY close, and new tabs can acquire jobs after recovery. Retrofitting
old tabs would not capture children that already escaped and would complicate
PID/lifecycle ownership. Once a helper has been ready, its failure remains
Tracked tabs whose initial assignment fails during startup/backoff retain their
original PID and spawn timestamp. Registration and retry retention check the
same live, unchanged, unreleased PTY even while an evidence stop is waiting for
its initial assignment. If that stop returns `IDENTITY` without killing the PTY,
the tab remains eligible for recovery and a later evidence stop can use its job.
The temporary `closing` guard is checked when readiness dispatches the late
assignment, not when registering or retaining the waiting tab.
On readiness, each receives at most one late
assignment, only while its PTY is still live, unchanged, open and unreleased.
The helper removes the pending recovery before sending it; a failed late
assignment is never retried. Closing or releasing cancels pending recovery.
The late assignment promise updates the same PTY record as the initial one,
including evidence close and natural-exit observation. Plain ConPTY close stays
available while recovery is pending or has failed.
Children spawned before late assignment are not captured; only children spawned
after the root joins the job inherit membership and are ended on tab close.
Once a helper has been ready, its failure remains
terminal for this session: a new process cannot inherit its existing job
handles. This change only recovers startup failures, not lost jobs. Tests stub
the spawn port and clock, including timeout-then-success through trackWindowsPty,
permanent timeout/spawn failure, the retry cap and stale-attempt events.
`windows-job-recovery` combines the real PTY tracking and helper state machines
with a stubbed child process, independent monotonic/wall clocks and controlled
timers. It covers automatic and explicit recovery, lifetime guards, one late
attempt, job close, evidence close racing a startup failure, and backwards
wall-clock steps. The helper tests also check cleanup after backoff expiry with
no waiting tabs; removing the non-assignment failure guard fails those checks.
EOF/finally
only closes handles, with no membership scan or termination. Release also just
closes the handle; there is no limit flag to clear and no release-time flag API
Expand Down Expand Up @@ -1173,7 +1194,7 @@ the assigned processes alive because no job is armed. Pure-Node native parent-ex
tests exercise this property; they are not an Electron exit measurement.

Resident PowerShell with P/Invoke may be flagged by EDR/AV or ended by the user.
There is no restart policy. No EDR product was tested.
After readiness there is no restart policy. No EDR product was tested.

Tests load shipped production code: pty-ops-jobs covers close ordering, grace
deadline, exclusions, detach and deferred assignment; windows-job-helper covers
Expand Down
2 changes: 1 addition & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ What changes for you in each release of Switchboard. How to write an entry: [doc

### Fixed
- After restarting the app or Windows, the sidebar shows the projects as they were at exit, even if the last project load finished just before you closed the app or signed out. (#1136)
- On Windows, a slow start no longer disables detached-process cleanup for every later tab: Switchboard retries up to twice for newly opened tabs. Tabs opened while cleanup is unavailable keep the existing close behavior. (#1140)
- On Windows, a slow start no longer leaves tabs without process cleanup: tabs already open, restored ones included, are covered once cleanup starts, and programs they launch afterwards end when the tab closes. Programs launched before that can keep running. (#1140, #1156)
- Claude and Codex sessions in `.work-files/worktrees/<name>`, and Codex sessions in `.worktrees`, `.claude-worktrees` and `.claude/worktrees`, appear under their existing repository, including nested worktrees and sessions already indexed. Repository settings, archive and hidden state follow the group, and project deletion removes the session metadata it previews for both runtimes. Archiving a repository covers its worktrees' sessions; a new session in a worktree reopens the repository. Resume and fork use the recorded worktree directory and refuse to launch if it no longer exists; Claude sessions in a deleted `.claude/worktrees` directory are refused instead of starting in the repository. (#1137)
- A conversation running as a background job (`claude --bg`) no longer slips into a project's collapsed "Older" rows: while its job is alive it stays visible and sorts with the running sessions, and the Running filter lists it, whatever the date of its last indexed message. (#1146)

Expand Down
9 changes: 8 additions & 1 deletion src/main/terminals/pty-ops.js
Original file line number Diff line number Diff line change
Expand Up @@ -22,8 +22,15 @@ function trackWindowsPty(pty, { platform = process.platform, spawnTime = Date.no
} else if (!record.closing) void releaseWindowsPty(pty);
});
const current = () => !record.exited && !record.released && pty._agent?.exitCode === undefined;
const assign = at => record.assignment = jobs.assign(record.id, pty.pid, at, current)
const rememberAssignment = assignment => record.assignment = assignment
.then(assigned => { record.assigned = assigned; return assigned; });
const assign = at => {
const pid = pty.pid;
const samePty = () => current() && pty.pid === pid;
return rememberAssignment(jobs.assign(record.id, pid, at, samePty, {
current: () => samePty() && !record.closing, onLateAssignment: rememberAssignment,
}));
};
if (pty.pid > 0) return assign(spawnTime);
return new Promise(resolve => {
const socket = pty._agent?.outSocket;
Expand Down
40 changes: 36 additions & 4 deletions src/main/terminals/windows-job-helper.js
Original file line number Diff line number Diff line change
Expand Up @@ -3,23 +3,39 @@
const { spawn } = require('node:child_process');
const fs = require('node:fs');
const path = require('node:path');
const { performance } = require('node:perf_hooks');
const ERROR_MEMBER_SKIPPED = 299;
const MAX_START_ATTEMPTS = 3;

function createWindowsJobHelper({ spawn: spawnChild = spawn, warn = () => {}, startupMs = 15000,
requestMs = 3000, protectedPids = () => [] } = {}) {
let child, started = false, ready = false, failed = false, buffer = '', startupTimer, earlyCloseWarned = false;
let attempts = 0, retryAt = 0, unavailableWarned = false;
let retryTimer;
const lateAssignments = new Map();
const pending = new Map();
const queued = new Map();
const waiting = new Map();
let resolveReady;
let readiness;

function scheduleRetry() {
clearTimeout(retryTimer);
for (const [id, entry] of lateAssignments) if (!entry.current()) lateAssignments.delete(id);
if (ready || attempts >= MAX_START_ATTEMPTS) { lateAssignments.clear(); return; }
if (!failed || !lateAssignments.size) return;
retryTimer = setTimeout(() => {
retryTimer = undefined;
for (const [id, entry] of lateAssignments) if (!entry.current()) lateAssignments.delete(id);
if (lateAssignments.size) { start(); if (failed) scheduleRetry(); }
}, Math.max(0, retryAt - performance.now()));
retryTimer.unref?.();
}

function fail(reason) {
if (failed) return;
failed = true;
retryAt = Date.now() + 1000 * 2 ** (attempts - 1);
retryAt = performance.now() + 1000 * 2 ** (attempts - 1);
clearTimeout(startupTimer);
resolveReady(false);
for (const entry of pending.values()) { clearTimeout(entry.timer); entry.resolve(false); }
Expand All @@ -32,6 +48,7 @@ function createWindowsJobHelper({ spawn: spawnChild = spawn, warn = () => {}, st
} catch {}
unavailableWarned = true;
try { child?.kill(); } catch {}
scheduleRetry();
}

function settle(entry, value) {
Expand Down Expand Up @@ -68,6 +85,11 @@ function createWindowsJobHelper({ spawn: spawnChild = spawn, warn = () => {}, st
ready = true; clearTimeout(startupTimer); resolveReady(true);
for (const entry of queued.values()) send(entry);
queued.clear();
clearTimeout(retryTimer);
for (const [id, entry] of lateAssignments) {
lateAssignments.delete(id);
if (entry.lateCurrent()) entry.onLateAssignment(request('assign', id, entry.args, entry.lateCurrent));
}
continue;
}
const match = /^(assigned|terminated|released|observed|error) ([\w-]+)(?: (\d+))?$/.exec(line);
Expand All @@ -84,9 +106,10 @@ function createWindowsJobHelper({ spawn: spawnChild = spawn, warn = () => {}, st

function start() {
if (failed) {
if (ready || attempts >= MAX_START_ATTEMPTS || Date.now() < retryAt) return Promise.resolve(false);
if (ready || attempts >= MAX_START_ATTEMPTS || performance.now() < retryAt) return Promise.resolve(false);
} else if (started) return readiness;
started = true;
clearTimeout(retryTimer);
failed = false;
buffer = '';
const attempt = ++attempts;
Expand Down Expand Up @@ -165,6 +188,8 @@ function createWindowsJobHelper({ spawn: spawnChild = spawn, warn = () => {}, st
}

function finish(op, id, args = '') {
lateAssignments.delete(id);
if (!lateAssignments.size) clearTimeout(retryTimer);
if (op === 'terminate' && !ready && !failed && !earlyCloseWarned) {
earlyCloseWarned = true;
try { warn('[pty] Tab closed before Windows job helper ready; using plain ConPTY close; hidden processes may remain'); } catch {}
Expand Down Expand Up @@ -241,9 +266,16 @@ function createWindowsJobHelper({ spawn: spawnChild = spawn, warn = () => {}, st
if (!ready || failed || pending.has(id)) return Promise.resolve(null);
return request('observe', id).then(value => value && typeof value === 'object' ? value : null);
},
assign(id, pid, spawnTime, current = () => true) {
assign(id, pid, spawnTime, current = () => true, { onLateAssignment, current: lateCurrent = current } = {}) {
if (!Number.isSafeInteger(pid) || pid <= 0 || pid === process.pid || !Number.isSafeInteger(spawnTime)) return Promise.resolve(false);
return request('assign', id, ` ${pid} ${spawnTime}`, current);
const args = ` ${pid} ${spawnTime}`;
return request('assign', id, args, current).then(assigned => {
if (!assigned && failed && !ready && attempts < MAX_START_ATTEMPTS && onLateAssignment && current()) {
lateAssignments.set(id, { args, current, lateCurrent, onLateAssignment });
scheduleRetry();
}
return assigned;
});
},
terminate,
terminateWithEvidence,
Expand Down
26 changes: 23 additions & 3 deletions test/windows-job-helper.test.js
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ const assert = require('node:assert/strict');
const fs = require('node:fs');
const path = require('node:path');
const { EventEmitter } = require('node:events');
const { performance } = require('node:perf_hooks');

for (const mode of ['nonempty', 'roster never settles', 'already expired']) {
test(`A4c2afix evidence ${mode} preserves other jobs and releases its handle`, async t => {
Expand Down Expand Up @@ -60,6 +61,7 @@ function fixture(t, options = {}) {

test('a timed-out start followed by a successful retry gives the next tabs jobs', async t => {
t.mock.timers.enable({ apis: ['setTimeout', 'Date'] });
t.mock.method(performance, 'now', () => Date.now());
const { trackWindowsPty } = require('../src/main/terminals/pty-ops');
const children = [], writes = [], warnings = [];
const { createWindowsJobHelper } = require('../src/main/terminals/windows-job-helper');
Expand All @@ -82,14 +84,14 @@ test('a timed-out start followed by a successful retry gives the next tabs jobs'
t.mock.timers.tick(999);
assert.equal(await helper.assign('during-backoff', 898, 100), false);
assert.equal(children.length, 1, 'tab creation during backoff cannot spawn');
t.mock.timers.tick(1);
assert.equal(await helper.terminate('old-tab'), false, 'close must not trigger a retry');
assert.equal(await helper.release('old-tab'), false);
assert.equal(await helper.observe('old-tab'), null);
assert.equal(children.length, 1, 'cleanup and status must not trigger a retry');
t.mock.timers.tick(1);
const next = trackWindowsPty(tab(899), { jobs: helper, platform: 'win32', spawnTime: 100 });
const concurrent = trackWindowsPty(tab(900), { jobs: helper, platform: 'win32', spawnTime: 100 });
assert.equal(children.length, 2, 'the next tab must retry helper startup');
assert.equal(children.length, 2, 'waiting tabs and new tabs share one retry');
const retryReady = helper.start();
assert.notEqual(retryReady, firstReady, 'retry has its own readiness promise');
children[0].stdout.emit('data', 'ready\ninvalid\n');
Expand All @@ -100,7 +102,7 @@ test('a timed-out start followed by a successful retry gives the next tabs jobs'
assert.deepEqual(writes, [], 'old helper events cannot flush the retry queue');
children[1].stdout.emit('data', 'ready\n');
assert.equal(await retryReady, true);
assert.equal(writes.length, 2, 'only new tabs are assigned; failed assignments are not replayed');
assert.equal(writes.length, 3, 'new tabs and the failed live tab are assigned');
for (const line of writes) children[1].stdout.emit('data', `assigned ${line.split(' ')[1]}\n`);
assert.equal(await next, true, 'next tab receives a job after recovery');
assert.equal(await concurrent, true);
Expand All @@ -109,9 +111,27 @@ test('a timed-out start followed by a successful retry gives the next tabs jobs'
assert.equal(warnings.some(line => line.includes('stale warning')), false);
});

for (const operation of ['terminate', 'release', 'observe']) {
test(`1156: ${operation} cannot restart a failed helper after backoff without waiting tabs`, async t => {
t.mock.timers.enable({ apis: ['setTimeout', 'Date'] });
t.mock.method(performance, 'now', () => Date.now());
const f = fixture(t);
const assignment = f.helper.assign('untracked', 897, 100);
t.mock.timers.tick(5000);
assert.equal(await assignment, false);
t.mock.timers.tick(999);
assert.equal(f.invocations.length, 1);
t.mock.timers.tick(1);
const result = f.helper[operation]('untracked');
assert.equal(f.invocations.length, 1, `${operation} must not restart after backoff expires`);
assert.equal(await result, operation === 'observe' ? null : false);
});
}

for (const failure of ['timeout', 'spawn error']) {
test(`permanently failing helper stops after three starts (${failure})`, async t => {
t.mock.timers.enable({ apis: ['setTimeout', 'Date'] });
t.mock.method(performance, 'now', () => Date.now());
let attempts = 0;
const f = fixture(t, { spawn() {
attempts++;
Expand Down
Loading
Loading