feat(runner): microVM boot canary startup preflight (RIG-3148) - #847
Open
rigel-mintaka wants to merge 3 commits into
Open
feat(runner): microVM boot canary startup preflight (RIG-3148)#847rigel-mintaka wants to merge 3 commits into
rigel-mintaka wants to merge 3 commits into
Conversation
|
Merging to
After your PR is submitted to the merge queue, this comment will be automatically updated with its status. If the PR fails, failure details will also be posted here |
|
Compass engineering docs preview: https://compass-runner-rig-3148-boot.compass-eng-docs.pages.dev Deployed from |
Add the dynamic half of the microVM backend's startup preflight: a boot
canary that really boots a throwaway VM through the backend's own verbs
(Create->Start->Exec->Remove), proving the whole chain -- KVM, vsock,
image, guest supervisor, exec gate -- not just binary presence.
- `CanaryReport{BootLatency, GuestRSSBytes}` and
`(*MicroVMRuntime).BootCanary(ctx)` in microvm_preflight.go. The canary
owns the VM's whole lifetime inside the call, so it derives a bounded
ctx (caller deadline as-is when present, else a 90s bound) and severs
teardown from it so a timed-out boot still tears down. Remove and the
throwaway-workspace cleanup are joined into the return, never discarded.
Boot latency is the Start wall time; guest PSS is best-effort telemetry
(a read error leaves it at 0, logged, never fatal -- the gate is the
boot chain).
- Extend the runner startup gate: a `canaryBooter` probe interface
(distinct single-method probe, not a widened microVMPreflighter) and a
`runMicroVMPreflight` helper that runs VerifyMicroVMSupport, then -- only
once it passes -- BootCanary, logging the report. A microVM engine that
cannot canary is a fail-closed startup error naming the type, never a
silent skip.
- Hermetic seam-faked suite over the launch/client seams (no KVM), plus a
KVM-gated TestBootCanary riding the existing microVM CI leg. A one-line
cross-reference distinguishes the no-boot enablement smoke test from
this real boot canary.
Refs RIG-3148
Co-authored-by: Matt Wilkinson <matt@rigel.build>
… discipline (RIG-3148) Additive review-fix commit on the W3 boot-canary PR, closing the one gating finding plus three lows from the review round: - medium (test-adequacy): the PSS-read-error fail-open branch — the one fail-open seam on an otherwise fail-closed startup gate (record §(e)/OQ-10) — had no hermetic coverage, so a regression flipping it to fatal (spurious Runner-startup refusal on a host with an unreadable smaps_rollup) would pass every test. Added a `pssErr` knob to canaryFakeVM (threaded through the launch recorder) and TestBootCanaryPSSErrorNonFatal, which pins that a PSS read error leaves BootCanary succeeding with GuestRSSBytes == 0 and the session torn down. Mutation-verified RED on flip-to-fatal. - low (lock discipline): BootCanary read `session.vm` outside `m.mu`, the one production reader deviating from the microvmSession invariant. Now reads the handle under the lock, matching Stop (microvm_lifecycle.go). - low (doc): tightened the canaryTeardownGrace comment to state it bounds Remove only if/when Remove honors its ctx deadline — today Remove is deadline-agnostic — so the constant no longer implies an enforced ceiling it lacks. - low (test symmetry): TestBootCanaryNonZeroExitFails now carries the temp-leak and wasShutdown assertions its sibling failure-path tests carry. No behavior change on the happy path; the under-lock read is a visibility/ invariant tightening with no functional effect (single-owner canary session). Refs RIG-3148 Co-authored-by: Matt Wilkinson <matt@rigel.build>
…RIG-3148) Round-r1 review fold for the V5 W3 boot canary. All four findings are test-only; `microvm_preflight.go` is unchanged. ### Nonce round-trip (was the gating medium) The nonce-mismatch branch — exec returns exit 0 but stdout does not carry the minted nonce — was uncovered: `canaryFakeClient.Exec` always echoes the command back, so the fake structurally could not produce exit-0-with-wrong-stdout. That branch is the "exec gate" leg of the whole-chain claim (record §(e)): exit 0 alone only proves a call returned; the nonce is the sole assertion the guest ran our command and returned our data. Added a `stdout *string` override knob (mirroring the r0 `pssErr` knob) and `TestBootCanaryNonceMismatchFails`. Mutation-verified: disabling the nonce check goes RED, restored GREEN. ### Deadline guard (was low, tautological test) `TestBootCanaryHonorsCallerDeadline` could not fail on the bug it named — a 5s caller bound is clamped by `context.WithTimeout` regardless of the `if _, ok := ctx.Deadline(); !ok` guard, so a re-deriving implementation passes it. Added `TestBootCanaryHonorsLongerCallerDeadline` with a 10-minute caller deadline (the only case the guard actually protects: a longer caller deadline must pass through, not be clamped to the 90s canary bound). Mutation-verified: removing the guard goes RED for the new test while the old 5s test stays GREEN. ### Partial PSS (was low, half-covered contract) The real `VM.PSS()` returns a partial map alongside a joined error as its normal shape; `TestBootCanaryPSSErrorNonFatal` drove `pss=nil` so the partial-data-plus-error combination was undefended. Added `TestBootCanaryPartialPSSStillReported`. Mutation-verified: discarding the partial map inside the error branch goes RED. ### Path convention (was low) `boot_canary_microvm_test.go` built `<RunRoot>/microvm` by string concatenation beside the production `filepath.Join`; switched to `filepath.Join`. Gates green: gofmt, linux + `-tags microvm` builds, hermetic suite (11 tests), golangci-lint `--build-tags microvm` (0 issues), nilaway (exit 0). The remaining r1 low (threading a signal-cancellable ctx into the startup preflight callsite) is a control-flow-ordering judgment call and is filed separately for Matt. Spec-impact: none. Refs RIG-3148 Co-authored-by: Matt Wilkinson <matt@rigel.build>
rigel-mintaka
force-pushed
the
compass-runner/rig-3148-boot-canary
branch
from
September 3, 2026 05:19
5be9b07 to
ee6a3a6
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add the dynamic half of the microVM backend's startup preflight: a boot
canary that really boots a throwaway VM through the backend's own verbs
(Create->Start->Exec->Remove), proving the whole chain -- KVM, vsock,
image, guest supervisor, exec gate -- not just binary presence.
CanaryReport{BootLatency, GuestRSSBytes}and(*MicroVMRuntime).BootCanary(ctx)in microvm_preflight.go. The canaryowns the VM's whole lifetime inside the call, so it derives a bounded
ctx (caller deadline as-is when present, else a 90s bound) and severs
teardown from it so a timed-out boot still tears down. Remove and the
throwaway-workspace cleanup are joined into the return, never discarded.
Boot latency is the Start wall time; guest PSS is best-effort telemetry
(a read error leaves it at 0, logged, never fatal -- the gate is the
boot chain).
canaryBooterprobe interface(distinct single-method probe, not a widened microVMPreflighter) and a
runMicroVMPreflighthelper that runs VerifyMicroVMSupport, then -- onlyonce it passes -- BootCanary, logging the report. A microVM engine that
cannot canary is a fail-closed startup error naming the type, never a
silent skip.
KVM-gated TestBootCanary riding the existing microVM CI leg. A one-line
cross-reference distinguishes the no-boot enablement smoke test from
this real boot canary.
Refs RIG-3148
Co-authored-by: Matt Wilkinson matt@rigel.build