Re-measure and re-enable the expensive nested tests (L3/L4, podman-load-over-fuse) post-#630

[ejc3]

What

Five nested tests remain #[ignore]d after #630 re-enabled the main suite (the "#630-B" bucket):

Test	Current ignore reason	Old timing claim
test_nested_l3_network_fuse	L3 FUSE-over-FUSE ×2: slow	container startup > 10 min
test_nested_l3_network_nfs	L3 via NFS: slow	untested at L3
test_nested_l3	3-hop FUSE chain	"FUSE mount init alone takes 10+ min"
test_nested_l4	4 levels	worse than L3
test_podman_load_over_fuse	podman load in nested VM	"389s in CI (75x slower than L1)"

Why the old timing data is suspect

Every one of these timing claims predates the #630 root-cause fixes and was measured under the pathology. The #630 investigation proved the dominant cost in this family was not FUSE/gzip/NV2-inherent overhead but:

1. Timer-storm starvation of snapshot-restored/resumed NV2 guests (25–300× slowdown) — fixed in the firecracker fork via KVM_ARM_SET_COUNTER_OFFSET.
2. Firecracker event-loop starvation after pause→save→resume — bypassed by converging the snapshot-miss path on the restore flow.

Concrete recalibration from the fixed system: podman load of the same ~100 MB archive inside an L1 measured 7.8–8.0s healthy vs >8 minutes under the storm — the very same "389s in CI" class this bucket's ignore reasons cite. test_nested_l2_fuse now passes in ~3 min where it used to time out at 600s.

Plan

1. Re-measure first, on the post-[meta] Stabilize and re-enable ALL flaky / disabled e2e tests #630 stack (fork with counter-offset fix, converged miss path, stale-NFS-export self-heal, per-level iperf3 ports). One manual run each: make test-root FILTER=<name> IGNORED=1 STREAM=1. Expect L3 to be dramatically cheaper than the 10-min-FUSE-init claim; L4 unknown.
2. For any test that now fits a sane budget (≤ ~10 min): un-ignore it, place it in the nested-tests nextest group, set the budget from measured data + margin.
3. For any test still genuinely architecture-bound (multi-hop FUSE latency compounds per level): keep ignored with an accurate, post-[meta] Stabilize and re-enable ALL flaky / disabled e2e tests #630 reason and a measured number — the current reasons cite storm-era data and are misleading.
4. Run the re-enabled ones N times serialized for flake statistics before merging (same bar as [meta] Stabilize and re-enable ALL flaky / disabled e2e tests #630: zero flakes, no retry-passes).
5. Optional wall-time follow-up: validate nested-tests at max-threads = 2-3 (one full-suite datapoint at 3-concurrent exists from the [meta] Stabilize and re-enable ALL flaky / disabled e2e tests #630 round-2 validation run); keep serialized if any contention shows.

Dependencies / notes

• Run AFTER the [meta] Stabilize and re-enable ALL flaky / disabled e2e tests #630 PR lands (needs the fork + fcvm fixes).
• The host kernel 7.1 upgrade (Host kernel upgrade 6.18.3 → 7.1 (fallback 6.19.y): pick up upstream NV fixes, then re-evaluate multi-vCPU L2 #659) is complementary: 6.19+ nested-vgic work + 7.1 shadow-S2 debugfs help if L3/L4 surface interrupt or translation issues.
• Do not chase these on contended CI runners until they pass locally N times — that was the original sin of the January disables.

Refs: #630, #659

[ejc3]

Still valid on main as of dfd7b3b. All five tests remain ignored with the original reasons (tests/test_kvm.rs:926, 940, 959, 975, 1924) and the doc comments still carry the pre-#630 numbers ("FUSE mount init alone takes 10+ minutes", "389s in CI").

Updates since this was filed that change the plan slightly:

1. The old timings are now doubly stale. Restore the nested reflink, delete the copy fallback, guard sun_path #843 restored the nested reflink and deleted the copy fallback, so the L2+ disk prep that used to be a full copy inside the L1 is CoW again. Any re-measurement must be on post-Restore the nested reflink, delete the copy fallback, guard sun_path #843 main.
2. Plan step 2 is already wired. .config/nextest.toml has the nested-tests group (max-threads=1) and its override already matches test(/nested/) | test(/podman_load_over_fuse/) with a 1200s slow-timeout. Re-enabling is deleting the #[ignore] line and rewriting the stale comment with the measured number.
3. Plan step 5 is answered: PR fix(podman): cross-VM image-cache race; run a CI subset of nested NV2 tests #677 measured two concurrent nested tests at 3.7x slower plus a flake versus serialized. Keep max-threads=1.
4. CI scope: Makefile CI_ROOT_BASE runs only a kept trio of nested tests in CI, so un-ignored L3/L4 tests would run locally via make test-root, not in CI. That fits the issue's own warning about contended runners; I would not add them to the kept list.
5. The dependency note about the host kernel 7.1 (Host kernel upgrade 6.18.3 → 7.1 (fallback 6.19.y): pick up upstream NV fixes, then re-evaluate multi-vCPU L2 #659) is outdated; main pinned deployable kernels to 6.18.44 (55fc053).

What is left is exactly the measurement campaign: one manual run per test on an ARM64 NV2 metal box against current main, un-ignore whatever fits the 1200s budget, rewrite the reasons on whatever does not with the measured post-fix number, then the serialized zero-flake repeats before merge. Needs the metal box; nothing else blocks it.