[rush-daemon] Don't let the default queue timeout fail late compatible builds - #6067
Merged
Sean Larkin (TheLarkInn) merged 2 commits intoSep 24, 2026
Merged
Sean Larkin (TheLarkInn) merged 2 commits into
Sean Larkin (TheLarkInn) merged 2 commits into
Conversation
…e builds A SHARED-BUILD request that arrives after the current batch has closed waits on the graph-execution gate. That wait was charged to the same admission deadline (default 30 s), so every late build failed with wait-timeout while another build ran longer than 30 s. A client-default timeout now bounds workspace admission only; explicit --no-wait/--wait-timeout/RUSH_DAEMON_QUEUE_TIMEOUT_SECONDS still bound the entire wait. Admission failure messages now explain what the request waited for and how to wait longer. Fixes #6051 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Sean Larkin (TheLarkInn)
requested review from
Bharat Middha (bmiddha),
David Michon (dmichon-msft) and
Jonathon Anthony (jxanthony)
as code owners
September 24, 2026 02:05
This was referenced Sep 24, 2026
Mo Jazayeri (mojaza)
approved these changes
Sep 24, 2026
Sean Larkin (TheLarkInn)
deleted the
thelarkinn-fix-rushd-late-build-queue-timeout
branch
September 24, 2026 18:00
Contributor
There was a problem hiding this comment.
Copilot review overview
🟢 Approval recommended
The implementation preserves explicit deadlines and cancellation while comprehensively testing the corrected default behavior.
Review effort: Balanced
Findings: None
What changed in this PR
Fixes #6051 by preventing the default daemon queue timeout from failing compatible builds waiting behind an active build.
Changes:
- Distinguishes default and explicitly configured admission timeouts.
- Removes the default deadline from compatible graph-execution waits while preserving cancellation and explicit limits.
- Improves admission failure messages, documentation, and test coverage.
| File | Description |
|---|---|
libraries/rush-daemon/src/WorkspaceRequestAdmission.ts |
Implements graph-specific timeout semantics. |
libraries/rush-daemon/src/test/PhasedRequestBatching.test.ts |
Tests default and explicit late-build timeouts. |
libraries/rush-daemon/src/PhasedRequestRouter.ts |
Uses graph-execution admission handling. |
libraries/rush-daemon/README.md |
Documents deadline behavior. |
libraries/rush-daemon-protocol/src/test/RequestAdmission.test.ts |
Tests the new admission field. |
libraries/rush-daemon-protocol/src/RequestEnvelopeValidation.ts |
Validates the new wire field. |
libraries/rush-daemon-protocol/src/DaemonRequestAdmission.ts |
Defines waitTimeoutIsDefault. |
common/reviews/api/rush-daemon-protocol.api.md |
Updates the API report. |
common/changes/@rushstack/rush-daemon/fix-rushd-late-build-queue-timeout_2026-09-24-01-45.json |
Records the daemon patch. |
common/changes/@rushstack/rush-daemon-protocol/fix-rushd-late-build-queue-timeout_2026-09-24-01-45.json |
Records the protocol patch. |
common/changes/@rushstack/rush-cli-client/fix-rushd-late-build-queue-timeout_2026-09-24-01-45.json |
Records the client patch. |
apps/rush-cli-client/src/test/ClientAdmissionControls.test.ts |
Tests configured admission and messages. |
apps/rush-cli-client/src/launchClient.ts |
Marks only built-in defaults and formats failures. |
apps/rush-cli-client/src/ClientAdmissionControls.ts |
Adds admission conversion and message helpers. |
apps/rush-cli-client/README.md |
Documents client timeout semantics. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
With the daemon on, if a
rush-client buildarrives while another build has been running for more than 30 seconds, it failed withdaemon admission failed (wait-timeout)and exit code 1. This happened for an identical target, a subset, or a disjoint--only. After this change, with default settings, the late build waits for the running build and then runs or skips as usual. An explicit wait limit is still enforced. A timeout message now says what the request was waiting for and how to wait longer.Root cause
PhasedRequestRoutercloses a batch (#acceptingCurrentBatch = false) right after it reconciles invalidations. A compatibleSHARED-BUILDrequest that arrives after that point waits on the per-graph execution gate:admissionController.acquireAsync(#graphExecutionScheduler).WorkspaceRequestAdmission.ts).rush-clientalways sentwaitTimeoutMs = queueTimeoutSeconds * 1000(default 30 s). Any build that took longer than 30 s therefore failed every late build.Fix
Deadline semantics (documented in both READMEs):
--no-wait,--wait-timeout,RUSH_DAEMON_QUEUE_TIMEOUT_SECONDS, ordaemon.queueTimeoutSecondsinrush.json): unchanged. One absolute deadline covers both workspace admission and the graph-execution gate.SHARED-BUILDrequest waiting on the graph-execution gate is queued only behind running compatible shared builds. That is progress, not contention, so this wait has no deadline, and the request runs in the next batch. Ctrl+C still cancels it.Changes:
@rushstack/rush-daemon-protocol: new optional admission fieldwaitTimeoutIsDefault, validated and added to the API report. Older daemons ignore it and keep the previous behavior.@rushstack/rush-cli-client: setswaitTimeoutIsDefault: trueonly when the timeout comes from the built-in default. Admission failures now print, for example:rush-client: daemon admission failed (wait-timeout): timed out after 5s waiting for another daemon request in this workspace to finish (a command that needs exclusive access, or a running build). To wait longer, use --wait-timeout <seconds> or set RUSH_DAEMON_QUEUE_TIMEOUT_SECONDS.--no-waitfailures get a similar explanation.@rushstack/rush-daemon: newRequestAdmissionController.acquireGraphExecutionAsync(), which drops the deadline for aSHARED-BUILDgraph wait only when the timeout is the client default. The daemon-side timeout error names what the request was waiting for and how to wait longer.Optional follow-ups, not included here: joining an iteration that is already running (a subset request could subscribe to the in-flight iteration), and head-of-line blocking, where a small target waits for the largest (board #38). Queue feedback for non-TTY clients is tracked separately in #6052.
Tests
rush-daemon(PhasedRequestBatching.test.ts):SHARED-BUILDrequest with a default timeout (20 ms) is still waiting 100 ms into a running compatible batch, then completes successfully and runs its operation.wait-timeout, and the holder still succeeds.rush-daemon-protocol: validation ofwaitTimeoutIsDefault.rush-cli-client:getConfiguredAdmission(default vs. explicit) andformatAdmissionFailure.Linux validation (WSL Ubuntu 24.04, Node 22.23.2)
rush build --to @rushstack/rush-cli-client --to @rushstack/rush-daemonpassed, including lint and API Extractor.rush test --onlypassed forrush-daemon-protocol,rush-daemon, andrush-cli-client.Repro: a synthetic workspace with 16 projects at 1.5 s each, where
p16sleeps 40 s and its source is edited before each round. A holderrush-client buildstarts at t0; at +5 s,build --only p03andbuild --wait-timeout 5 --only p03start concurrently. Times are seconds from t0.buildbuild --only p03(default timeout)daemon admission failed (wait-timeout).build --wait-timeout 5 --only p03daemon admission failed (wait-timeout).--wait-timeout/RUSH_DAEMON_QUEUE_TIMEOUT_SECONDSremedyFixes #6051
This came out of the automated
rushdLinux performance/behavior analysis.