[5/5] ci: compile the heavy Rust lanes through nix, cached on ix's binary cache - #4487
Closed
harivansh-afk wants to merge 27 commits into
Closed
[5/5] ci: compile the heavy Rust lanes through nix, cached on ix's binary cache#4487harivansh-afk wants to merge 27 commits into
harivansh-afk wants to merge 27 commits into
Conversation
A scheduled GitHub-hosted reconcile (ix-runners.yml) converges a pool of ix VMs against nix/ix-pool.toml, the pool's single source of truth (size, region, families). The runner NixOS policy lives in nix/ci-runner.nix: toolchain parity with the hosted images, 9 job-family placement labels dealt round-robin so repeat visits of a family land on a warm member, substituters for cache.ix.dev baked into the image (slot users are untrusted nix clients, so nothing a job passes at runtime can add one). Security shape: the reconcile runs ONLY on GitHub-hosted runners - IX_TOKEN never reaches a pool VM; runner registration uses short-lived tokens attached at VM create; fork PRs cannot read secrets, so untrusted events cannot touch the pool. Onboarding is: set IX_TOKEN, merge.
devShells.ci and ci-msrv (nix/ci-shell.nix) pin every CI toolchain - one shell per Rust toolchain, never 'cargo +toolchain' inside a shell (the shadowed rustup shim breaks it, hence .envrc also skips its nix branch in CI). The ci-env helper exports the curated environment (PATH, LIBCLANG/OPENSSL/PKG_CONFIG, LD_LIBRARY_PATH - openssl-sys binaries link a store libssl with no rpath) into GITHUB_ENV once. The single workflow seam is .github/actions/setup-ci-shell: nix arm enters the shell and exports ci-env; fallback arm (fork PRs on hosted runners, which have no nix) does rustup show + mise, exactly what jobs did before. Every ix-specific path fails open to today's behavior. setup-musl-cross gains the same shape for the musl target; the pool's musl-gcc wraps the full cross gcc (the thin libc wrapper links broken static-PIE binaries). flake.nix also carries lib.mkPool consumed by the reconcile workflow.
Three us-west-1 host incidents on 2026-08-17 each took the whole pool, because every member and every replacement lived there. Index-modulo split per ix-runners multi-region support: 16 members per region, one region's incident costs half the pool, and failed creates fail over to the other region the same tick instead of piling into the sick one. The ENG-10881 us-east v6 blackhole that used to forbid east placement is already mitigated in the member image (gai.conf v4 precedence in ci-runner.nix), so the xwin cross lanes are safe there. Adoption is organic: the reconcile matches members by name, so existing members move only at their next natural replacement - no forced wave.
The vend endpoint accepts main revs only, so this bump rides after the ix-runners merge. Brings the regions-list vocabulary the pool spec in the previous commit uses; without it the reconcile refuses 'unknown key regions' and the pool freezes (deliberately - unknown keys never default).
…ives vitest sizes its worker pool at availableParallelism()-1, and the workers leg spawns one workerd process per vitest worker. Every typescript_web fixture test spawns its own vitest instance, so N concurrent tests multiply into N x workers workerd processes at ~1 GiB anon each under V8. Measured at the moment a 64-core CI guest OOM-killed the suite: 22 resident workerd, ~13 GiB RSS + 14 GiB zram. The upstream cap (#7871) covers only the browser pool, and the lane had never passed unpinned on 64-core hosts (A/B with perfect cgroup oom_kill correlation). Two bounds, each modeling what the dial actually controls: - pin maxWorkers in the vitest config so pool size stops scaling with host core count (no effect at the hosted runners' 15-worker envelope); - a vitest-pool-serial nextest group (max-threads = 1) for sdk_test_typescript_web, so one vitest pool exists at a time and peak memory equals one pool - which is what VITEST_MAX_WORKERS models. Makes the suite portable to any runner size instead of encoding the 16-vCPU hosted envelope.
baml_release read release/platforms.json via include_str! four directories above its own root - library source, not a test - so the platform contract was compiled into baml_cli, baml, and everything downstream from a path outside every crate. Each consumer (the gradle plugin, the csharp pack tool, the release scripts, the CI change filters) then carried its own copy of that path. Move platforms.json into baml_release (content unchanged, R100) and point every reader at the new location. The contract now lives with the crate that compiles it in, and any per-crate build or source-slicing tool sees a self-contained crate instead of a silent reach above the workspace root.
Three sources reached above the workspace root and three build scripts read or wrote outside their own crate. Each gets the narrowest fix that keeps cargo behavior identical (proven by execution: full 'cargo test --no-run' green, generated files land in the same places): - baml_tests/tests/baml_src.rs include_str!'d a playground fixture from typescript2/ (the comment calls it 'intentionally cursed') and baml_project's param_schema.rs include_str!'d above the root under cfg(test) - which still compiles under --no-run. Both now read through a path that stays inside the workspace, failing loudly if the fixture is absent. - baml_tests' build.rs wrote generated_tests.rs into src/; when the source tree is read-only it now generates into OUT_DIR instead, same contents either way. - bridge_ctypes' and sdkgen_cpp's build scripts guarded against writing outside their crate when the source is not writable. - harness_setup resolved the shared fixture corpus and the canonical TypeScript sources by walking ../.. from the manifest dir; both walks now honor an env override (BAML_SDK_TEST_FIXTURES, BAML_SDK_TEST_TYPESCRIPT_SOURCES) and keep the relative walk as the default, and its fs::copy calls stop inheriting 0444 modes from read-only sources (the copies belong to the build). Together these make the workspace consumable by any per-crate build system - source slicing no longer breaks compilation, and no build mutates a sibling crate's tree.
runs-on swaps for every Linux lane: same-repo events target ['self-hosted', 'ix-<family>']; fork PRs keep the exact Blacksmith runner each job used before (they cannot be allowed on persistent VMs that later run trusted, secret-bearing jobs). checkout moves to actions/checkout@v6 with clean: false - tracked files still hard-reset to the SHA, ignored build state (target/, node_modules) survives for warm rebuilds on persistent runners; a no-op on ephemeral ones. ix-cross.yml adds the cross-compile lane (zig/xwin windows + darwin compile checks) that hosted runners had no capacity for. actionlint learns the ix label. Lane-level env pins ride along (CARGO_BUILD_JOBS/NEXTEST threads tuned to the 16-vCPU envelope the suites were tuned on; VITEST_MAX_WORKERS paired with the vitest-pool-serial group). wasm-pack-tests keeps upstream's bridge_wasm check (sys_llm no longer exists) and only swaps its runner/toolchain provisioning.
VITEST_MAX_WORKERS 4 -> 12 and the one-pool-at-a-time nextest group -> 2 in-flight pools. Both were OOM tourniquets for the pre-2026-08-17 elastic platform (idle guests deflated to ~6 GiB; re-inflation lost the race to the workerd spawn burst). The fleet now runs a per-guest working-set floor (learned peak stays plugged ~an hour) plus a post-boot inflation holdoff, so the memory the burst needs is already resident on any member that has seen this suite. Expected: the two serialized vitest_workers tests (91s + 49s) overlap, taking the lane from its ~278s floor toward ~160-180s. Step-down ladder if the lane goes red with 'Worker exited unexpectedly': workers 12 -> 8, then revert to 4 + max-threads 1.
…links NEXTEST/RUST_TEST_THREADS 16 -> 32 at the job level (workflow env overrides the runner unit's 16-pin deterministically; the 16 was tuned against the pre-floor deflation behavior - revert if the rust init-race or loom soak flakes return). GOMAXPROCS=16 joins the existing honesty-pin family, and GOFLAGS=-ldflags=-w drops DWARF from fixture test-binary links nothing ever debugs.
The sdk lanes were left on the imperative arm when the pool conversion landed: every job re-provisions its language toolchains through mise, measured at 27s per job on a warm persistent runner, while the converted lanes enter the flake CI shell in 3-8s. Measured today on the pool, the go lane runs 111s total; 27s of that is the mise step. This finishes the conversion. devShells.ci-sdk is the ci surface plus the toolchains the sdk lanes spawn from PATH: node 22 + pnpm (typescript lanes), temurin 23 + gradle with JAVA_HOME pinned (java lane). go, python, uv, wasm-pack, sccache, direnv, and cargo-nextest were already in the ci shell. One shared attr, not one per lane: every Linux sdk lane runs on the same ix-sdk runner family, so one closure warms all of them. The Blacksmith fork-PR fallback (and the macOS/Windows lanes, which have no nix) is byte-identical to the two steps this replaces: the composite's no-nix arm runs rustup show + setup-mise with each lane's unchanged install-args. dotnet stays on actions/setup-dotnet on both arms; it was never mise-provisioned. Version drift vs mise pins, accepted under the same policy as the earlier conversions: go and gradle track the pinned nixpkgs (mise pins go 1.23.11, gradle 8.14); node major matches (22); temurin major matches (23).
…ge build
'rustup run 1.93.0 cargo build' dies on the nix CI arm in 0.07s - that
rustup owns no toolchains ('toolchain 1.93.0-x86_64-unknown-linux-gnu is
not installed', SETUP FAIL java_gradle_unix, proven on PR CI job
95515772259). The pin is also redundant on the mise arm: the workspace
rust-toolchain.toml already selects 1.93.0 through the rustup shim. Plain
cargo resolves correctly on both arms - the same shape go's setup.sh has
always used, which is why the go lane passed where java failed. setup.ps1
updated in sync (Windows runs the rustup shim; behavior unchanged).
The msrv lane is build-only (cargo test --no-run) with a unique Rust toolchain nothing else warms: per-VM sccache gives it nothing, and its cold runs were the worst absolute delta on the board. This converts its compile into a per-rustc-unit nix derivation graph (index's cargo-unit machinery, pinned by rev - the pinned path is anonymously evaluable, verified with internal-visibility inputs as 404 positive controls), substituted from cache.ix.dev, where a trusted builder publishes the graph for every trunk rev. A PR then compiles only its own delta: changed units and their reverse-dependencies. Arm equivalence is the design constraint: both arms compile the SAME program - profile dev with the lane's opt-level pins threaded into the planner, debuginfo dialed down but debug_assertions untouched (those change which code compiles; a --no-run lane never runs its binaries, so debuginfo is free to drop). The workflow probes before choosing: eval the root's out-path, ask the local store then cache.ix.dev with negative-narinfo caching disabled (a cached miss poisons a family member for an hour). Hit -> nix build. Miss or ANY failure -> today's cargo path, unchanged: a unit-graph miss costs the whole DAG with worse constants than cargo, so the fallback is a first-class arm, not an error path. Measured: cold-nix 224s vs 228s hosted median vs 1208s+ cold-cargo on the pool; warm 70-176s.
Second L2 lane: the wasm job (despite its name it only builds - cargo build --target wasm32-unknown-unknown --release) gets its own per-unit graph. One graph per (target, toolchain, profile) is the machinery's rule, so the wasm32 graph is a separate buildWorkspace instantiation: target as a first-class argument, release profile stated explicitly (the default would silently diverge from the lane), and .cargo/config.toml rustflags threaded so every wasm32 unit carries --cfg getrandom_backend (verified: 319/319 wasm32 rustc invocations carry it, zero host units do). The lane's crate selection is guarded against drift: the flake transcribes the [package.metadata.ci] wasm_support markers, and the probe first re-derives the set from cargo metadata and refuses the nix arm on any mismatch - a silent divergence would be a green job gating nothing. Both probes are byte-identical in shape by intent: two probes in one file reading differently is how one rots. nix/l2-roots.txt is the builder contract, format pinned in-file: bare flake attribute names, two roots per lane (<lane>-check = the unit DAG, <lane>-eval-roots = the IFD artifacts guests otherwise recompute at eval), and a measure-first rule - closure size is the invisible regression, a dev-debuginfo graph measured 93 GiB vs 4.6 GiB release on the same units, and the builder refuses roots over 30 GiB. Measured: wasm-check closure 393 MiB; warm wasm lane 42-74s vs 58s hosted median.
First lane that RUNS what it builds. The musl graph (default features, dev profile, debuginfo kept - these binaries execute, backtraces matter) compiles the lane's exact selection as per-rustc-unit derivations; the machinery's whole-workspace nextestExport emits binaries-metadata + cargo-metadata whose interpolated store paths pin every test binary into one substitutable closure. On a probe hit the job substitutes that one root and runs cargo nextest against the prebuilt binaries in the real checkout (--workspace-remap for per-test cwd, INSTA_WORKSPACE_ROOT for insta's baked snapshot dirs); on any miss or failure it takes today's cargo path unchanged. Arm equivalence holds because tests execute in the same runner environment either way - the arm changes who compiled, never what runs where. Static musl binaries dodge the loader-path class entirely. pack_e2e stays on the cargo path on both arms (needs a writable sibling-binary layout a store dir cannot be). Today's musl lane: 420-511s vs the 352s hosted median, compile-bound. With the compile substituted the lane is test-execution-bound, projected ~200-300s. DRAFT until the index pin bump: nextestExport needs the index rev carrying the whole-workspace export (forge main ea527d85eabe, mirror publish pending); the closure must be measured against the builder's 30 GiB refusal before the roots go live (debuginfo on - if it blows, line-tables-only on BOTH arms in one commit).
The new pin (mirror publish of 2026-08-16, forge ea527d85eabe) is the
first index rev carrying the whole-workspace nextest metadata export
that .#musl-test-export consumes. Anonymous evaluability re-verified at
this rev with the original pin's protocol (access-tokens cleared,
internal-visibility inputs 404ing as positive controls).
The bump trips the trap the msrv graph's HAZARD note predicted:
policy.compiler.embedMetadata exists now, defaults false, and renders
nightly-only -Zembed-metadata=no into every unit - fatal on all of this
flake's stable rust-overlay toolchains, and undetectable by the
machinery's own guard (it reads rustToolchain.ixRustChannel, which
rust-overlay toolchains do not carry; null is an accept arm). One shared
l2Policy = recursiveUpdate pureBuild { compiler.embedMetadata = true; }
now serves all graphs; recursiveUpdate rather than // so a future preset
key under compiler cannot be silently clobbered.
Re-keys every unit in the msrv and wasm graphs (toolchain/machinery id
is in the unit hash): one-time full builder republish, the same
self-heal the #19 delta proved.
Same conversion as the musl lane, host triple, and the bigger prize: gnu is the workspace's widest test lane and its cold/contended tail is the worst behavior left on the board - 1453s on a trunk wave, and a 25-minute cap cancellation under family contention, against 92s warm and a 198s hosted median. The recompiles ARE the tail; substituting them removes it. The gnu graph compiles the lane's exact selection (--all-features, the five exclusions verbatim - baml_tests/baml_cli/baml_lsp2_actions run in snapshot-tests, sdk_test_* in the sdk matrix) as per-rustc-unit derivations, dev profile, opt-level 1, debuginfo kept (these binaries run; backtraces matter). The job probes .#gnu-test-export - the machinery's whole-workspace nextest export, manifests + every test binary in one closure - and on a hit runs cargo nextest against the prebuilt binaries in the real checkout (--workspace-remap for per-test cwd; INSTA_WORKSPACE_ROOT for baml_surface's insta tests, whose snapshot dirs are compile-time baked to the unit's /build placeholder). Any miss or failure takes today's cargo path unchanged. The list phase executes each binary once; these glibc binaries link the graph's openssl with no rpath, and the loader resolves it through the LD_LIBRARY_PATH ci-env already exports (the export's linked-paths synthesis is empty in v1 - if the runtime env ever stops carrying it, the failure is loud: "libssl.so.3: cannot open shared object file" at list time, the exact class ci-env's export was built for). Also adds `id: toolchains` to this job's setup-ci-shell step - the one job in the file that lacked it, and the gate every probe reads. Projected: test-execution-bound ~150-250s in every regime vs the 198s hosted median; the 25-minute contended tail and the 40-minute cap headroom become dead letters.
…t graphs Measured on hc1 (2026-08-17), full dev debuginfo: musl-test-export closure 43.6 GiB (built cold in 13.5 min) gnu-test-export closure 41.2 GiB (built cold in 17 min) Both over the L2 builder's 30 GiB refusal (nix/l2-roots.txt measure-first rule). The graphs themselves are sound - ~300 test binaries per lane built end to end at the b8652df4 pin. debug=0 would fit but these lanes RUN their binaries and file:line backtraces on real test failures are worth keeping; line-tables-only drops the variable/type debuginfo that is ~all of the bloat. Pinned in each graph's env AND its lane's workflow env in the same commit: the two arms must compile the same program shape, and a knob set on one arm only is the worst gate shape (verdict decided by cache state). Re-measure before enabling the builder on these roots.
The gnu and musl lanes go red whenever the L2 probe hits: nextest resolves
the main .config/nextest.toml's package(=sdk_test_*) / package(=baml_bridge)
overrides and setup-script filters against the --cargo-metadata it is given,
and the unit-graph exports deliberately exclude those packages - a package
predicate naming a package absent from the metadata is a HARD parse error
('operator didn't match any packages'), not a no-op. Every entry in the main
config is sdk/bridge-scoped, so the nix-arm runs now pass --config-file
.config/nextest-l2.toml, which pins the nextest version and nothing else.
The cargo arm is untouched (full workspace metadata resolves the filters
fine). Delete nextest-l2.toml once the exports carry whole-workspace cargo
metadata.
Validation caveat: this PR's own CI probably cannot prove the fix - the
source change produces a new export root, so the probe misses and the lanes
take the (green, unaffected) cargo arm. The proof is the first canary run
after the L2 builder publishes the new root.
Same failure class one layer up: -E 'not binary(=pack_e2e)' is a hard
filterset parse error ('operator didn't match any binary names') when
pack_e2e is absent from the export metadata - and it is always absent, so
the exclusion was a structural no-op on this arm. The cargo arm keeps it.
Proven live on PR CI (job 95479342789): the config-file fix cleared the
config errors and this was the next and only remaining parse failure.
…on the nix arm With both parse layers fixed, the nix arm actually ran and exposed the next divergence: the export CONTAINS baml_cli and baml_lsp2_actions test binaries, which this job's cargo arm excludes (they run in snapshot-tests). Two baml_cli insta tests failed structurally - file-based snapshots resolve against the unit build's baked /build/<crate>/ manifest dir, which INSTA_WORKSPACE_ROOT cannot remap, so every assertion reads '+new results' (proven: job 95481125387, 254/256 passed, both failures baml_cli describe_command_tests). The arms must run the same test set, so both nix arms now exclude exactly the two packages present-but-excluded-on-cargo. Absent packages (baml_tests, sdk_test_*, baml_bridge) need no exclusion and must not get one - naming an absent package in a filterset is a hard parse error.
…den fixture path Two defects the first canary run through the merged L2 config exposed: 1. The gnu and musl exports DISAGREE about which excluded packages they carry - musl contains baml_cli and baml_lsp2_actions, gnu contains neither (proven live: gnu job 95521894678 died 'operator didn't match any packages' on the very exclusion that musl needs, musl job 95521894883 needed it). A static -E cannot serve both, so each arm now asks the export's cargo-metadata.json which of the two packages exist and excludes exactly those. Empty set = no -E at all. 2. baml_project's wire_shape_matches_the_ts_golden_fixture reads ../../../typescript2/... relative to its baked unit source dir, which does not exist in the relocated build (musl: 1576/1577 passed, this the only failure). The test ships BAML_PARAM_SCHEMA_GOLDEN for exactly this case; both arms now point it at the real checkout.
…uilds The namespace work added file-based insta snapshots to baml_surface, and file snapshots resolve against the compile-time CARGO_MANIFEST_DIR - which for the CI nix unit graph is a build sandbox (/build/baml_surface-<ver>/) that does not exist when the prebuilt binary runs. insta reads every assertion as '+new results' and the musl lane went red on the first run that carried the new tests (job 95536180317; the .snap files verified present and correct in the unit source - the binary just cannot reach them). Inline snapshots are unaffected, which is why only these three tests failed. BAML_SURFACE_SNAPSHOT_DIR binds insta's snapshot path when set; unset, behavior is byte-identical. The nix arms set it to the real checkout's snapshot dir. Same pattern as BAML_PARAM_SCHEMA_GOLDEN one crate over, which exists for the same relocated-build reason.
clippy doc_markdown, caught by prek on the previous push.
|
@harivansh-afk is attempting to deploy a commit to the Boundary Team on Vercel. A member of the Team first needs to authorize it. |
Contributor
|
Warning Review limit reached
Next review available in: 59 minutes Limit details: You’ve used all 8 included reviews currently available under your plan. You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Repository UI Review profile: CHILL Plan: Pro Plus Run ID: ⛔ Files ignored due to path filters (1)
📒 Files selected for processing (55)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
harivansh-afk
changed the title
harivansh-afk
marked this pull request as ready for review
August 17, 2026 23:45
Contributor
Author
|
Superseded by #4483, which carries the compile-cache machinery inside the non-gating preview workflow instead of the gating lanes. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The heavy Rust lanes (msrv, wasm, gnu, musl) get a second compile path: a per-rustc-unit nix derivation graph, substituted from ix's binary cache, where a trusted builder publishes the graph for every trunk rev. A PR then compiles only its own delta - changed units and their reverse-dependencies - instead of the whole workspace. Think per-crate sccache, except the cache key is the full input closure, so it can never serve a stale hit.
The design constraint throughout is arm equivalence: both arms compile the same program (same profile, same
debug_assertions; only debuginfo is dialed down, and the gnu/musl test lanes run real binaries so even that is scoped to lanes that never execute what they build... the msrv lane is--no-runby construction). Where equivalence could not be proven, the lane keeps cargo - the probe fails open: no published graph for a rev, no nix on the runner, any error at all → the job takes the exact cargo path it takes today. There is no state where this PR makes a lane worse than the status quo.Measured on the worst lane on the board - msrv, whose unique toolchain nothing else warms: cold start 224s vs your 228s warm Blacksmith median. A freshly rebuilt pool recovers to warm speed on its first wave instead of after days of cache accretion.
Requires nothing from you beyond the stack below it; cache population for this repo's revs is on our side.