Nested-test CI gaps after #630: x86 L2-over-restored-L1, vsock-integrity runner capacity, remaining re-enables

[ejc3]

Tracking issue for everything gated or deferred while landing #663 (the #630 zero-flake work). The goal of #663 was green nested tests on the dev box and CI; these are the items that did not make the bar for CI and why.

1. x86: nested KVM (VMX) does not survive an L1 snapshot restore

The five L2 tests (test_nested_l2_fuse, _nfs, _network_fuse, _network_nfs, _with_large_files) and test_nested_run_fcvm_inside_vm are gated to aarch64.

Evidence (run 27323861536):

• Host-Root-x64-SnapshotDisabled: all five L2 tests pass (61-81s each) — x86 L2 nesting works on the runners.
• Host-Root-x64-SnapshotEnabled: all five fail with the inner fcvm's L2 PUT /actions timing out after 30s. The only nested test that passed was the one that created the L1 cache (its L1 was paused/resumed in-process; VMX state intact). Every later L1 was restored from cache into a fresh firecracker — and its VMX state is gone, so the L2 can never start.
• ARM survived this exact class only via the NV2 firecracker fork (counter offset) + the miss→restore convergence ([meta] Stabilize and re-enable ALL flaky / disabled e2e tests #630). x86 has no equivalent.

Fix directions: (a) teach the x86 path that nested-profile L1s are not snapshot-compatible (profile-level no-snapshot), or (b) x86 equivalent of the restore-coherence work. Option (a) is small and honest; a tested --no-snapshot per-arch variant of the test launches exists in the #663 history if useful.

2. test_vsock_integrity_nested exceeds CI runner capacity

Gated with #[ignore]. Passes reliably on a 64-core Graviton3 dev box (592s, 680s — full 2.5GB / 4480-transfer corruption sweep, "No corruption detected") but fails at its 840s in-test budget on the small CI runners (run 27323861536, arm64-SnapshotEnabled, both tries). One L2 chain saturates those instances (load ~2 from a single L1+L2).

Fix directions: larger runner class for the nested matrix, a reduced-transfer CI variant (e.g. 1/4 of the sweep), or keep it as a dev-box/manual gate. Related wall-clock lever: #660 (nested concurrency re-measure post-#630).

3. Still-ignored tests from the original #630 set

• L3/L4 chains + podman-load-over-FUSE: Re-measure and re-enable the expensive nested tests (L3/L4, podman-load-over-fuse) post-#630 #660 (re-measure post-fix; old timings were measured under the now-fixed storm pathology).
• test_nested_l2_with_benchmarks: manual benchmarking only, by design.

4. Open forensic threads

• Restored-clone restore stall (117s, deterministic) + dead serial console after cache-restore — forensics from #630 round 4 #662: dead serial console on every restore of a specific cached snapshot + the (since-fixed-by-removal) 117s restore stall — diagnostic blindness worth fixing before the next investigation in this area.
• test_clone_port_forward_stress_rootless: first clone's pre-storm curl RSTs (health-ready vs proxy-path-ready race; reproduces on main) #661: the pasta addr_seen poisoning. Root-caused to pasta splicing inbound flows to the last-overheard source address; upstream patch v2 submitted to passt-dev (validated 6/6 locally vs 1-in-3 failure). fcvm currently green because the test tolerates the retry; once the pasta fix ships, pin the runner/dev pasta version.
• One unexplained local observation (dev box, post-fix(#630): zero-flake nested NV2 tests — timer coherence, restore convergence, NFS-over-restore, FICLONE truncation #663 state): an L1 restored from an hours-old converge cache hung mid-podman load through the FUSE map. Suspected stale-cache + mixed-binary contamination (not reproduced on a clean build yet); if it recurs on clean state it belongs with Restored-clone restore stall (117s, deterministic) + dead serial console after cache-restore — forensics from #630 round 4 #662's restore-coherence questions.

CI policy meanwhile

The nested matrix that gates PRs = the aarch64 L2 set (proven green across four consecutive full-suite rounds on dev hardware and passing on arm64 CI). x86 keeps L1-level nested coverage (test_kvm_available_in_vm).

[ejc3]

Update: test_nested_l2_with_large_files is now SnapshotDisabled-only in CI (perf, #686)

Separate from the x86-VMX-restore gap above, the arm64 side had a cost problem: in the Host-Root-arm64-SnapshotEnabled job, test_nested_l2_with_large_files was the single longest test by ~17× — it ran twice (cold populate ~225s + warm restore ~842s ≈ 18 min), serialized at the tail.

Profiling showed the 32MB FUSE-over-FUSE copy is only ~20s; the cost is a 4GB nested-VM snapshot create→restore whose UFFD page faults traverse double Stage-2 translation. That's pure snapshot-lifecycle overhead and adds nothing to what the test guards — FUSE-over-FUSE data integrity (the #630 DSB cache-coherency fix), a live-traffic property of the always-on kernel patch, independent of snapshots.

Change (in main, PR #686): run it only in the SnapshotDisabled job (where it already runs, ~225s) and exclude it from SnapshotEnabled. Mechanism: the Makefile CI_NESTED_FILTER adds & not test(=test_nested_l2_with_large_files) only when FCVM_NO_SNAPSHOT is empty (SnapshotEnabled). Snapshot/restore correctness stays covered by test_startup_snapshot_* and the CH/clone roundtrips.

Net: arm64 SnapshotEnabled long pole ~49 → ~31 min (overall pole becomes x64 SnapshotEnabled ~34 min, which has no nested tests to trim).

The other two CI nested tests (test_nested_run_fcvm_inside_vm, test_nested_l2_nfs) still run in both modes. The deeper "nested snapshot restore is ~3.7× slower than cold boot" anomaly that makes this test expensive is still unaddressed and is the real follow-up lever.

[ejc3]

Status against main (dfd7b3b, 2026-08-18).

Still open:

• Item 1 (x86 VMX over restored L1): unchanged. The five L2 tests and test_nested_run_fcvm_inside_vm are still aarch64-only (tests/test_kvm.rs:250, 827-908), and no x86 profile-level no-snapshot landed; snapshot_cache_opt_out (src/commands/podman/mod.rs:326) has no nested-profile entry and nv2_profile remains ARM-only. Option (a) is still the small honest fix: an x86 nesting-profile predicate feeding snapshot_cache_opt_out, plus ungating the tests, pinned by a unit test next to every_snapshot_cache_opt_out_trigger_stands_alone.
• Item 2: test_vsock_integrity_nested is still #[ignore] pointing here (tests/test_vsock_integrity.rs:30). Nothing changed on runner class; a reduced-sweep CI variant remains the cheapest path.
• Item 3: L3/L4 chains and podman-load-over-FUSE are still ignored (tests/test_kvm.rs:926-976, 1924), benchmarks test still manual by design.

Resolved since filing:

• The test_clone_port_forward_stress_rootless: first clone's pre-storm curl RSTs (health-ready vs proxy-path-ready race; reproduces on main) #661 pasta thread's follow-up (pin the pasta version once the fix ships) is done: fcvm now builds pasta from a pinned upstream commit with the addr_seen patch embedded (src/setup/pasta.rs, bc86db3), so runners and dev boxes run exactly the proposed fix.
• The ci(#61): provision Cloud Hypervisor in CI + trim the arm64 SnapshotEnabled long pole #686 large-files trim described in the comment above is in place (Makefile CI_NESTED_FILTER, lines 98-111).

Keeping this open for items 1-3.