[rush-daemon] Arbitrate environment restarts and bound client restart retries

[Sean Larkin (TheLarkInn)]

Summary

Concurrent rush-client requests whose environments differ (e.g. AGENT=1..4) each asked the daemon for a tier-2 process restart. The daemon granted every one, so environments ping-ponged across successors; each client's single restart retry saw retryAfterRestart again and exited 1 (The successor requested another restart; the single safe retry was exhausted.), and the workspace was left with no daemon. This PR makes the daemon arbitrate restarts and makes the client retry a bounded number of times, then fall back in-process.

Root cause

• libraries/rush-client-core/src/executeWithDaemonRestart.ts allowed exactly one retry; any second retryAfterRestart threw startupFailed (a hard exit 1, since launchClient.ts calls it outside the connect fallback).
• WorkspaceRequestLifecycle#prepareAsync: a request that classified as Restart immediately took the workspace transition and queued for the exclusive gate lease. The RequestScheduler is FIFO, so every later request, including ones whose env matched the running process, queued behind it and was then rejected with RestartPendingBeforeExecution. That let one mismatched request preempt all other queued work, and every successor repeated the same thing for the next env.

Fix

Daemon (@rushstack/rush-daemon)

• New WorkspaceRestartArbiter: each dispatched request (except cancellable graph watch observers) holds a ticket. A request that needs a restart releases its gate lease and waits until no other ticket is still being served, i.e. every in-flight or queued request that this process can serve has drained. Only then does it take the exclusive transition and restart. Restart candidates are admitted one at a time. The others wake, see restartPending, and get the typed pre-execution retryAfterRestart, so started work is never replayed.
• The wait uses the request's admission budget (RequestAdmissionController.waitForRestartDrainAsync): it honors noWait, the remaining waitTimeoutMs and abort, and reports failures through the existing admissionErrorCode path. An abandoned waiter never triggers a restart.

Client (@rushstack/rush-client-core)

• executeWithDaemonRestartAsync follows up to 6 successors. The first retry is immediate and later ones use jittered exponential backoff (50 ms up to 1 s). Each successor's ownership is attested before the next hand-off, and all of this stays inside the admission deadline.
• When retries or the deadline run out, it returns a new fallback outcome (reason: 'restartRetriesExhausted') instead of throwing. launchClient already handles fallback by printing the message and running in-process Rush, so apps/rush-cli-client is unchanged. This keeps the PR clear of FIX-startup-wedge ([rush] rush-client-core: a failed or slow daemon startup leaves a durable .starting reservation that wedges the workspace (every command waits ~16 s, daemon start refuses); an invalid RUSH_* env value triggers it and hides the real error #6050).

Tests

• rush-client-core connectOrStartDaemon.test.ts: new restart-twice fixture mode. A second restart request is retried and succeeds with a decreasing admission budget. restart-always now returns the bounded restartRetriesExhausted fallback where it used to throw.
• rush-daemon WorkspaceRestartArbitration.test.ts (real host, real native builds, real successor): while a matching build is held in flight, a mismatched-env build arrives, and then another matching build arrives. Both matching builds complete (exit 0, no retryAfterRestart). The mismatched request resolves last with retryAfterRestart, and the successor is running.
• rush-daemon WorkspaceRestartArbiter.test.ts: unit coverage for drain ordering, one-at-a-time admission, and no-wait/timeout/abort accounting.

Linux validation (WSL Ubuntu-24.04, node 22.23.2)

Synthetic workspace: mkws-synth <ws> --projects 12 --shape wide --sleep-ms 5000. Warm the daemon, dirty the sources, then run env _=fixed AGENT=<n> rush-client build --only p0<n> for n=1..4 at the same time (script: /workspaces/rushd-lab/fixes/restart-arbitration-repro.sh).

• Before (main): 4/4 exit 1 at 6.7 s with single safe retry was exhausted; daemon status afterwards: could not connect (no daemon).
• After (default 30 s queue timeout, host load average ~70 on 32 cores): the envs serialize and nothing fails because of another client's env. 3/4 exit 0 (13.8 s, 25.3 s, 36.9 s). The 4th env was still queued behind three ~11 s env generations and hit the admission deadline: daemon admission failed (wait-timeout) at 31.1 s. A daemon is left running.
• After with RUSH_DAEMON_QUEUE_TIMEOUT_SECONDS=120: 4/4 exit 0 (13.2 s, 24.1 s, 34.4 s, 44.3 s), one successor per env. A daemon is left running.

Note: with the 30 s default, a request can hit the wait-timeout after N environments ahead of it have each built and restarted. This is the same class as #6051 / #6067 (a wait for progress charged against the default deadline). Once #6067 lands, the restart-drain wait can follow its waitTimeoutIsDefault rule. #6047 removes the common volatile-var trigger, so most envs will no longer need a restart at all.

rush build --to @rushstack/rush-client-core --to @rushstack/rush-daemon --to @rushstack/rush-cli-client (including lint) passes; rush test --only @rushstack/rush-client-core --only @rushstack/rush-daemon passes.

Compatibility with #6067

The restart-drain wait reports its own admission errors and doesn't call RequestAdmissionController's private error mapping. So this PR is independent of #6067's signature changes and composes with them. Verified on Linux: pull/6079/head merged with pull/6067/head merges cleanly, rush build --to @rushstack/rush-daemon --to @rushstack/rush-cli-client (including lint) passes, and rush test --only @rushstack/rush-daemon --only @rushstack/rush-client-core passes.

Follow-ups (optional)

• Environment fingerprinting of volatile vars (PWD, _, SHLVL) is the most common trigger; see [rush] rushd: request environment fingerprint covers every env var, so per-shell variables (PWD, OLDPWD, SHLVL, _, TERM, WSL_INTEROP, ...) restart the daemon and discard the warm graph #6047.
• Apply [rush-daemon] Don't let the default queue timeout fail late compatible builds #6067's waitTimeoutIsDefault rule to the restart-drain wait once it lands.
• Queued requests after a source edit (seen by the agent-reporter session): the client-side error single safe retry was exhausted no longer exists. Those requests now retry up to 6 successors and then fall back in-process, and restarts are serialized. If that edit changes the daemon's own runtime (for example a self-hosted rushstack workspace whose build rewrites rush-daemon/rush-lib output), each such build still costs one restart. Making the runtime fingerprint ignore outputs the build itself rewrites would be a separate fix.
• A successor could adopt/serve all waiting requests whose env matches it in one batch. Today each env still costs one restart.

Fixes #6057

This came out of the automated rushd Linux performance/behavior analysis ("Rushd Hive"; board thread #74, confirmations #98 and #102).

[Copilot]

Copilot review overview

🟡 Changes recommended

Restart deadlines can be exceeded, graph-control restarts bypass arbitration, and public retry documentation remains inconsistent.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 2 · 1

Open (3)

• Enforce admission deadline across retry backoff and reconnection · New
• Route graph-control restarts through restart arbitration · New
• Update protocol and client docs for bounded multi-retry behavior · New

What changed in this PR

Adds daemon restart arbitration and bounded client retries to prevent concurrent environment changes from causing restart loops.

Changes:

• Serializes daemon restart candidates after active compatible requests drain.
• Retries successor daemons with backoff, then falls back in-process.
• Adds unit and integration coverage for arbitration and retries.

File	Description
libraries/​rush-daemon/​src/​WorkspaceRestartArbiter.ts	Implements restart ticket arbitration.
libraries/​rush-daemon/​src/​WorkspaceRequestLifecycle.ts	Integrates arbitration into request preparation.
libraries/​rush-daemon/​src/​WorkspaceRequestAdmission.ts	Applies admission budgets to drain waits.
libraries/​rush-daemon/​src/​test/​WorkspaceRestartArbitration.test.ts	Tests end-to-end restart ordering.
libraries/​rush-daemon/​src/​test/​WorkspaceRestartArbiter.test.ts	Tests arbiter ordering and cancellation.
libraries/​rush-client-core/​src/​test/​fixtures/​daemon.ts	Adds repeated-restart fixture modes.
libraries/​rush-client-core/​src/​test/​connectOrStartDaemon.test.ts	Tests bounded successor retries.
libraries/​rush-client-core/​src/​executeWithDaemonRestart.ts	Implements retries, backoff, and fallback.
libraries/​rush-client-core/​src/​DaemonClient.ts	Adds the restart-exhaustion fallback reason.
common/​reviews/​api/​rush-client-core.api.md	Updates the public API report.
common/​changes/​@rushstack/​rush-daemon/​fix-restart-arbitration_2026-09-24-01-45.json	Records the daemon patch.
common/​changes/​@rushstack/​rush-client-core/​fix-restart-arbitration_2026-09-24-01-45.json	Records the client-core patch.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.