Skip to content

[rush-daemon] Don't let the default queue timeout fail late compatible builds - #6067

Merged
Sean Larkin (TheLarkInn) merged 2 commits into
mainfrom
thelarkinn-fix-rushd-late-build-queue-timeout
Sep 24, 2026
Merged

Sean Larkin (TheLarkInn) merged 2 commits into
mainfrom
thelarkinn-fix-rushd-late-build-queue-timeout

Conversation

@TheLarkInn

@TheLarkInn Sean Larkin (TheLarkInn) commented Sep 24, 2026 •

Copy link
Copy Markdown
Member

Summary

With the daemon on, if a rush-client build arrives while another build has been running for more than 30 seconds, it failed with daemon admission failed (wait-timeout) and exit code 1. This happened for an identical target, a subset, or a disjoint --only. After this change, with default settings, the late build waits for the running build and then runs or skips as usual. An explicit wait limit is still enforced. A timeout message now says what the request was waiting for and how to wait longer.

Root cause

  • PhasedRequestRouter closes a batch (#acceptingCurrentBatch = false) right after it reconciles invalidations. A compatible SHARED-BUILD request that arrives after that point waits on the per-graph execution gate: admissionController.acquireAsync(#graphExecutionScheduler).
  • That wait used the remaining time of the same absolute admission deadline as workspace admission (WorkspaceRequestAdmission.ts).
  • rush-client always sent waitTimeoutMs = queueTimeoutSeconds * 1000 (default 30 s). Any build that took longer than 30 s therefore failed every late build.

Fix

Deadline semantics (documented in both READMEs):

  • Explicit limit (--no-wait, --wait-timeout, RUSH_DAEMON_QUEUE_TIMEOUT_SECONDS, or daemon.queueTimeoutSeconds in rush.json): unchanged. One absolute deadline covers both workspace admission and the graph-execution gate.
  • Built-in 30 s default: applies to workspace admission only, i.e. waiting behind a command that needs exclusive access. A SHARED-BUILD request waiting on the graph-execution gate is queued only behind running compatible shared builds. That is progress, not contention, so this wait has no deadline, and the request runs in the next batch. Ctrl+C still cancels it.

Changes:

  • @rushstack/rush-daemon-protocol: new optional admission field waitTimeoutIsDefault, validated and added to the API report. Older daemons ignore it and keep the previous behavior.
  • @rushstack/rush-cli-client: sets waitTimeoutIsDefault: true only when the timeout comes from the built-in default. Admission failures now print, for example: rush-client: daemon admission failed (wait-timeout): timed out after 5s waiting for another daemon request in this workspace to finish (a command that needs exclusive access, or a running build). To wait longer, use --wait-timeout <seconds> or set RUSH_DAEMON_QUEUE_TIMEOUT_SECONDS. --no-wait failures get a similar explanation.
  • @rushstack/rush-daemon: new RequestAdmissionController.acquireGraphExecutionAsync(), which drops the deadline for a SHARED-BUILD graph wait only when the timeout is the client default. The daemon-side timeout error names what the request was waiting for and how to wait longer.

Optional follow-ups, not included here: joining an iteration that is already running (a subset request could subscribe to the in-flight iteration), and head-of-line blocking, where a small target waits for the largest (board #38). Queue feedback for non-TTY clients is tracked separately in #6052.

Tests

  • rush-daemon (PhasedRequestBatching.test.ts):
    • A late SHARED-BUILD request with a default timeout (20 ms) is still waiting 100 ms into a running compatible batch, then completes successfully and runs its operation.
    • The same request with an explicit 20 ms timeout fails with wait-timeout, and the holder still succeeds.
  • rush-daemon-protocol: validation of waitTimeoutIsDefault.
  • rush-cli-client: getConfiguredAdmission (default vs. explicit) and formatAdmissionFailure.

Linux validation (WSL Ubuntu 24.04, Node 22.23.2)

rush build --to @rushstack/rush-cli-client --to @rushstack/rush-daemon passed, including lint and API Extractor. rush test --only passed for rush-daemon-protocol, rush-daemon, and rush-cli-client.

Repro: a synthetic workspace with 16 projects at 1.5 s each, where p16 sleeps 40 s and its source is edited before each round. A holder rush-client build starts at t0; at +5 s, build --only p03 and build --wait-timeout 5 --only p03 start concurrently. Times are seconds from t0.

client before (main) after (this PR)
holder build rc 0, 48.2 s rc 0, 46.8 s
late build --only p03 (default timeout) rc 1, 36.1 s, daemon admission failed (wait-timeout). rc 0, 47.3 s
late build --wait-timeout 5 --only p03 rc 1, 11.3 s, daemon admission failed (wait-timeout). rc 1, 11.2 s, new message naming the wait and the --wait-timeout / RUSH_DAEMON_QUEUE_TIMEOUT_SECONDS remedy

Fixes #6051


This came out of the automated rushd Linux performance/behavior analysis.

…e builds

A SHARED-BUILD request that arrives after the current batch has closed waits on the graph-execution gate. That wait was charged to the same admission deadline (default 30 s), so every late build failed with wait-timeout while another build ran longer than 30 s. A client-default timeout now bounds workspace admission only; explicit --no-wait/--wait-timeout/RUSH_DAEMON_QUEUE_TIMEOUT_SECONDS still bound the entire wait. Admission failure messages now explain what the request waited for and how to wait longer.

Fixes #6051

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@TheLarkInn
Sean Larkin (TheLarkInn) merged commit 98bf216 into main Sep 24, 2026
11 checks passed
@TheLarkInn
Sean Larkin (TheLarkInn) deleted the thelarkinn-fix-rushd-late-build-queue-timeout branch September 24, 2026 18:00

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟢 Approval recommended

The implementation preserves explicit deadlines and cancellation while comprehensively testing the corrected default behavior.

Review effort: Balanced
Findings: None

What changed in this PR

Fixes #6051 by preventing the default daemon queue timeout from failing compatible builds waiting behind an active build.

Changes:

  • Distinguishes default and explicitly configured admission timeouts.
  • Removes the default deadline from compatible graph-execution waits while preserving cancellation and explicit limits.
  • Improves admission failure messages, documentation, and test coverage.
File Description
libraries/​rush-daemon/​src/​WorkspaceRequestAdmission.ts Implements graph-specific timeout semantics.
libraries/​rush-daemon/​src/​test/​PhasedRequestBatching.test.ts Tests default and explicit late-build timeouts.
libraries/​rush-daemon/​src/​PhasedRequestRouter.ts Uses graph-execution admission handling.
libraries/​rush-daemon/​README.md Documents deadline behavior.
libraries/​rush-daemon-protocol/​src/​test/​RequestAdmission.test.ts Tests the new admission field.
libraries/​rush-daemon-protocol/​src/​RequestEnvelopeValidation.ts Validates the new wire field.
libraries/​rush-daemon-protocol/​src/​DaemonRequestAdmission.ts Defines waitTimeoutIsDefault.
common/​reviews/​api/​rush-daemon-protocol.api.md Updates the API report.
common/​changes/​@rushstack/​rush-daemon/​fix-rushd-late-build-queue-timeout_2026-09-24-01-45.json Records the daemon patch.
common/​changes/​@rushstack/​rush-daemon-protocol/​fix-rushd-late-build-queue-timeout_2026-09-24-01-45.json Records the protocol patch.
common/​changes/​@rushstack/​rush-cli-client/​fix-rushd-late-build-queue-timeout_2026-09-24-01-45.json Records the client patch.
apps/​rush-cli-client/​src/​test/​ClientAdmissionControls.test.ts Tests configured admission and messages.
apps/​rush-cli-client/​src/​launchClient.ts Marks only built-in defaults and formats failures.
apps/​rush-cli-client/​src/​ClientAdmissionControls.ts Adds admission conversion and message helpers.
apps/​rush-cli-client/​README.md Documents client timeout semantics.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Status: Closed

3 participants