Skip to content

[rush-daemon] Stop --verbose / --parallelism / --timeline from reloading the warm graph - #6063

Merged
Sean Larkin (TheLarkInn) merged 3 commits into
mainfrom
thelarkinn-fix-rushd-flag-reload
Sep 24, 2026
Merged

Sean Larkin (TheLarkInn) merged 3 commits into
mainfrom
thelarkinn-fix-rushd-flag-reload

Conversation

@TheLarkInn

@TheLarkInn Sean Larkin (TheLarkInn) commented Sep 24, 2026 •

Copy link
Copy Markdown
Member

Summary

The Rush daemon pins its warm operation graph to the first request's non-selection parameter identity. That identity included presentation and scheduling flags such as --verbose, -p/--parallelism and --timeline. So build --verbose after a warm build caused a tier-1 reload that threw away every retained result: every operation was restored from the build cache again, and switching back to plain build reloaded a second time.

With this change, these flags apply per request. After a warm build, build --verbose, build -p 2 and build --timeline keep the same generation and run 0 operations, and the flags still take effect.

Root cause

  • PhasedScriptAction.getEngineParameterIdentity() (rush-lib) built the identity from actionName, the global parameters and all non-selection action parameters. That included --verbose, --parallelism and --timeline.
  • WorkspaceRequestLifecycle (rush-daemon) treats any identity mismatch as a tier-1 reload. The reload creates a new resolver and graph with no retained results.
  • None of these flags is baked into the graph:
    • OperationGraph.quietMode and OperationGraph.parallelism are read on every iteration, and watch mode already changes them at runtime.
    • In engine mode, --timeline only adds ConsoleTimelinePlugin, which writes to a no-op presentation terminal.

Fix

  • rush-lib: getEngineParameterIdentity() now excludes --verbose, --parallelism and --timeline. Graph-affecting custom parameters, global parameters and the command name remain in the identity. A new PhasedCommandEngine.requestSettings exposes the request-scoped values as IPhasedCommandEngineRequestSettings { quietMode, parallelism } (@alpha).
  • rush-daemon:
    • ProductionDaemonRequestResolver returns requestSettings on the resolved phased request.
    • PhasedRequestRouter applies quietMode and parallelism to the shared graph just before it schedules each iteration.
    • The router now coalesces only requests whose settings are equal, so a -p 1 request never runs inside a batch that uses the default parallelism.
    • Resolvers that do not supply settings behave as before.

rebuild <-> build: not addressed here (follow-up)

A rebuild graph is not just a build graph with some operations invalidated. It is created with isIncrementalBuildAllowed = false, and that setting is baked in when the graph is constructed:

  • CacheableOperationPlugin sets isCacheReadAllowed per operation.
  • ShellOperationRunnerPlugin uses no :incremental command.
  • LegacySkipPlugin sets allowSkip.
  • DaemonIpcOperationRunnerPlugin is skipped.

Running rebuild on a warm build graph would need a per-iteration "non-incremental" mode across these plugins and runners. Folding the command name into the build identity would be incorrect: the rebuild would restore from cache instead of executing. So this PR leaves rebuild behavior unchanged. It still causes one tier-1 reload, and the next build causes another (see steps 9–10 below). Flag variants such as rebuild --verbose no longer re-key the graph.

Suggested follow-ups:

  1. Add a per-iteration non-incremental mode (cache reads off, initial command, no persistent IPC runner). Then rebuild --to X can simply invalidate the selected operations on the shared build graph.
  2. As defense in depth, carry retained results across a tier-1 reload for operations whose own and config state hashes are unchanged.

Tests

  • libraries/rush-lib/src/api/test/PhasedCommandEngineParameterIdentity.test.ts:
    • --verbose/-v, --parallelism/-p, --timeline and selection parameters leave the identity unchanged.
    • A graph-affecting custom flag (--production) and rebuild still change it.
    • requestSettings reports verbosity and parallelism, including 50%.
  • libraries/rush-daemon/src/test/WorkspaceRequestScopedParameters.test.ts:
    • Runs build → build → build --verbose → build → build -p 1 --timeline → build against a real daemon host and native graph.
    • Each step keeps the same generation and the same graph object, reports lastReloadTier = Reuse, and executes nothing.
    • graph.quietMode and graph.parallelism follow each request.
    • After a source edit, build --verbose -p 1 rebuilds a and b on the same generation.
  • In a Linux lab clone:
    • rush build --to @rushstack/rush-daemon --to @rushstack/rush-cli-client (includes lint and API Extractor; the API reports match the committed ones exactly) passes.
    • rush-lib identity test: 3/3 passed.
    • rush-daemon WorkspaceRequestScopedParameters|PhasedRequest*|ProductionDaemon*|WorkspaceReloadTier*: 71/71 passed.

Linux validation

Setup: WSL Ubuntu-24.04, node 22, synthetic mkws-synth --projects 12 --sleep-ms 300, one bash process per series. The machine was heavily loaded (load average about 90), so wall times are noisy; the generation, tier and operation counts are deterministic.

# command before: gen / tier / executed / cache-restored after: gen / tier / executed / cache-restored
1 build (cold) 2 / 1 / 0 / 12 2 / 1 / 0 / 12
2 build 2 / 0 / 0 / 0 2 / 0 / 0 / 0
3 build --verbose 3 / 1 / 0 / 12 2 / 0 / 0 / 0
4 build 4 / 1 / 0 / 12 2 / 0 / 0 / 0
5 build -p 2 5 / 1 / 0 / 12 2 / 0 / 0 / 0
6 build 6 / 1 / 0 / 12 2 / 0 / 0 / 0
7 build --timeline 7 / 1 / 0 / 12 2 / 0 / 0 / 0
8 build 8 / 1 / 0 / 12 2 / 0 / 0 / 0
9 rebuild --to p02 9 / 1 / 2 / 0 3 / 1 / 2 / 0
10 build 10 / 1 / 0 / 12 4 / 1 / 0 / 12 (rebuild follow-up)
11 build 10 / 0 / 0 / 0 4 / 0 / 0 / 0

Fixes #6048


This came out of the automated rushd Linux performance and behavior analysis ("Rushd Hive", board BUG #48; prototype by A01).

…loading the warm graph

Presentation and scheduling flags (--verbose, --parallelism, --timeline) are no longer part of the engine parameter identity, so they no longer force a tier-1 reload that discards all retained results. rushd applies quietMode/parallelism to the shared graph before each iteration and only coalesces requests with equal settings.

Fixes #6048

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Fixes #6048

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

The new concurrent batching compatibility behavior lacks a test covering requests with different settings.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 1 Low severity

Open (1)
What changed in this PR

This PR keeps Rush daemon operation graphs warm when request-scoped presentation or scheduling flags change.

Changes:

  • Excludes verbosity, parallelism, timeline, and selection flags from graph identity.
  • Applies request settings per iteration and restricts batching to matching settings.
  • Adds identity and warm-graph integration coverage.
File Description
libraries/​rush-lib/​src/​index.ts Exports request settings type.
libraries/​rush-lib/​src/​cli/​scriptActions/​PhasedScriptAction.ts Separates request-scoped flags from graph identity.
libraries/​rush-lib/​src/​api/​test/​PhasedCommandEngineParameterIdentity.test.ts Tests identity and settings parsing.
libraries/​rush-lib/​src/​api/​PhasedCommandEngine.ts Exposes parsed request settings.
libraries/​rush-daemon/​src/​test/​WorkspaceRequestScopedParameters.test.ts Tests sequential warm-graph reuse.
libraries/​rush-daemon/​src/​ProductionDaemonRequestResolver.ts Propagates request settings.
libraries/​rush-daemon/​src/​PhasedRequestRouter.ts Applies settings and gates request coalescing.
libraries/​rush-daemon/​src/​DaemonRequestDispatcher.ts Passes settings to the router.
common/​reviews/​api/​rush-lib.api.md Updates rush-lib API report.
common/​reviews/​api/​rush-daemon.api.md Updates rush-daemon API report.
common/​changes/​@rushstack/​rush-daemon/​flag-reload_2026-09-24.json Records daemon patch.
common/​changes/​@microsoft/​rush/​flag-reload_2026-09-24.json Records Rush patch.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread libraries/rush-daemon/src/PhasedRequestRouter.ts
…gether

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@TheLarkInn
Sean Larkin (TheLarkInn) merged commit 7aecafa into main Sep 24, 2026
10 checks passed
@TheLarkInn
Sean Larkin (TheLarkInn) deleted the thelarkinn-fix-rushd-flag-reload branch September 24, 2026 21:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Status: Closed

Development

Successfully merging this pull request may close these issues.

[rush] rushd: --verbose, -p/--parallelism and build/rebuild alternation trigger a tier-1 graph reload that discards all warm results

3 participants