Open work after the restore-latency round: remaining 136ms, eth0 assumptions, cache opt-outs, 7.x wedge attribution

[ejc3]

Everything I still have open after the restore-latency work landed (#802, #803, #804 merged; #805 open). Each item below was re-verified against main at 0594e96 today rather than carried forward from notes, and items already covered by an existing issue are cross-referenced instead of restated.

Restore path: the remaining 136ms

The guest restore ACK is now 136ms p50 / 195 p95 (from ~643ms), attributed per phase by the telemetry that rides every ACK. What is left, measured on Graviton3 at concurrency 1 across 141 restores:

phase	p50	p95
boundary reopen	66.1	96.1
gateway ARP	27.6	41.0
boundary verify	16.4	34.3
clock step	3.6	12.3
boundary destroy	0.9	1.4

• Reopen (66ms) is the last spawn cluster. It is one iptables-restore per family plus one ip -batch per family. The link up/down already moved to SIOCSIFFLAGS; the same treatment for route reinstatement (rtnetlink RTM_NEWROUTE) and for the gate teardown would remove the last four spawns on the path. Each spawn on a freshly restored clone costs roughly 15-20ms because its text and libraries fault back in through the memory backend, so this is where the remaining budget is.
• Gateway ARP (28ms) has no spawn left — it is already a native AF_PACKET frame with no reply wait. The cost is first-touch page faults in that code path, which suggests prefetching the boundary's working set into the clone rather than micro-optimising the send.
• Verify (16ms) is two iptables -S reads and is the safety argument itself, so it should only go away if the gate state can be proven some cheaper way (reading nftables directly, or carrying a gate fingerprint in the manifest and validating that).

Interface naming: eth0 is still assumed outside the snapshot boundary

a6b6af4f (#797) taught the snapshot boundary to discover the guest's external interface, because Cloud Hypervisor's PCI virtio-net is renamed to enp0s4 by udev while Firecracker's MMIO device keeps eth0. The rest of the codebase still hardcodes the name. Verified counts of "eth0" as a literal in code (not comments, not test fixtures) on main:

• fc-agent/src/network.rs — 8 sites: IPv6 add/del/show, the IPv6 default route, and the loopback DNAT publication

• src/network/{veth,routed,bridged}.rs, src/commands/podman/vm_config.rs, src/storage/snapshot.rs

• rootfs-config.toml — the 10-eth0.network systemd unit and its mmds.conf drop-in match on Name=eth0

• Reuse the boundary's external_interface() discovery (sysfs device symlink) for these paths, or thread the discovered name through once at boot. The systemd unit is the interesting one: it runs before anything of ours and matching on Name=eth0 is what makes the kernel ip= address survive the rename.

Snapshot cache: three opt-outs remain

AGENTS.md says cache everything and take no opt-out unless behaviourally required. snapshot_cache_opt_out (src/commands/podman/mod.rs:313) still has three:

• FCVM_BOOTPLAN=vsock — root-caused, small fix. snapshot_key() hashes the serialized FirecrackerConfig, which has no field for the boot-plan transport, so a vsock-built snapshot and an MMDS-built one collide on one key; the opt-out exists only to stop the former being restored under the latter. Adding a serialized transport field gives vsock-built entries their own namespace and lets the opt-out be deleted. Guard: that struct has #[serde(skip)] fields, so confirm the new field is not skipped, or the key will not change.
• Cloud Hypervisor — the podman run pre-start snapshot cache is simply not wired up for CH (explicit snapshot create/run already work). Wiring it is the real fix and is larger than the above. Related: [epic] Hypervisor-agnostic abstraction: support Firecracker AND Cloud Hypervisor (the two microVMs) behind a pluggable trait #632.
• rootfs_override — a disk-only clone cold-boots from a captured disk, so this looks behaviourally required. Re-verify against the principle rather than assuming.

Kernel: attribute the 7.x nested wedge

#751 repinned to 6.18.44 away from a wedging config; attribution between upstream 7.x and our FUSE remap patch is still undetermined. The previously banked CONTROL_VERDICT=RED is invalid and should not be reused: the 7.1.8-noremap inner run died at Reflink copy failed ... requires a kernel profile with this patch, a prerequisite failure that never reaches the virtio-blk read path under test. A positive control is banked (6.18.44 + all patches, inner run succeeded).

• Redesigned experiment: pre-create the L2 disk from the host onto /mnt/fcvm-btrfs so the host btrfs performs the reflink, then boot L2 inside an L1 on 7.1.8-with-patches versus 7.1.8-minus-remap and hammer virtio-blk reads (reads never call remap_file_range). Wedge on both ⇒ upstream 7.x; wedge only with our patch ⇒ ours. Related: Host kernel upgrade 6.18.3 → 7.1 (fallback 6.19.y): pick up upstream NV fixes, then re-evaluate multi-vCPU L2 #659, Kernel patch audit vs 7.0.14: NV2 crash has an upstream fix we lack; 4 patches can be dropped #770.

Stress beyond the gate

bench/restore-matrix.py (#805) covers backend × network × concurrency 1/4/16 × volumes × kill-mid-restore, and is clean at 33/33 cells. It is a closed-loop harness.

• Drive the open-loop scale harness (reqscale, from Salvage the unlanded work from the archived WIP branches #795) at arrival rates the closed-loop bench cannot produce, deconfounding FILE vs UFFD within a rate interval, plus sustained and fan-out phases. The goal is to surface defects, not to publish numbers: every failure gets root-caused rather than retried.
• A matrix cell that exercises the boundary repair path in vivo. Today only the unit tests cover verified_armed == false; the natural in-vivo shape is a --privileged workload mutating the gate from inside the guest while a clone restores, which is also the scenario that motivated asserting the link-down unconditionally.

CI hygiene

• Deduplicate the btrfs bootstrap. .github/workflows/kernels.yml sets up /mnt/fcvm-btrfs inline in two places; make setup-btrfs does the same thing and is what a developer runs. One of them should call the other so they cannot drift.

Already tracked elsewhere (listed so this issue is not read as the whole picture)

• PastaNetwork::cleanup returning Ok after a failed kill or PID-file removal — PastaNetwork::cleanup returns Ok after a failed kill or PID-file removal #800
• Re-enabling the five #[ignore]d NV2-flaky nested L3/L4 tests in tests/test_kvm.rs — Re-measure and re-enable the expensive nested tests (L3/L4, podman-load-over-fuse) post-#630 #660 (and Nested-test CI gaps after #630: x86 L2-over-restored-L1, vsock-integrity runner capacity, remaining re-enables #664 for the surrounding CI gaps)

[ejc3]

AMI follow-up from the 2026-08-12 Host-job failures (chown ~/.local/share/containers/storage/overlay/l: operation not permitted on every rootless build, #792/#805):

The default rootless podman store arrives pre-contaminated from the AMI. Timeline: Host jobs' localhost tests passed on Aug 10, failed deterministically from Aug 12 — bracketing 8a9c564 ("Create the runner AMI from the running builder instead of stopping it"). A live-snapshot AMI bakes whatever rootless-store state the builder accumulated (partial layers, foreign-uid files under a possibly different subuid map).

Mitigations landed on the #808 stack:

• FCVM_CONFIG_DIR replaces the XDG redirect, so CI tests no longer resolve the default store at all
• scripts/runner-podman-hygiene.sh removes the default store at every runner setup step (safe unconditionally: CI's storage.conf points the graphroot at btrfs, so any content there is a dropping by definition), logging the store's ownership tree before removal — the next contaminated runner's job log will contain the exact uid:gid mode path listing needed to confirm what the builder leaves behind

The remaining root-cause item for the AMI side: either exclude ~ubuntu/.local/share/containers from the builder snapshot, or clean it in build-ami.sh before the image is created.

[ejc3]

The '7.x wedge attribution' item is resolved: the restore-band bimodality was probe-grid aliasing (2ms probe → unimodal 40.9ms, FG run), the small real tail is fsync-on-workingset-persist (code-inferred), and the separate 4K-minor hard wedge was root-caused to EEXIST-without-wake and fixed in #817 (0/160 validation). The 136ms restore decomposition, eth0 assumptions, and cache opt-outs remain open.

[ejc3]

Status refresh against main (dfd7b3b, 2026-08-18). Two items are done, one sub-item is obsolete, the rest stand.

Resolved:

• 7.x wedge attribution: closed above (2026-08-14 comment; uffd: wake waiters after an EEXIST'd UFFDIO_CONTINUE #817 fixed the EEXIST-without-wake wedge, the bimodality was probe-grid aliasing).
• Reopen spawn cluster, mostly: ef8741a removed the link-down and with it the route purge, so route capture/reinstatement no longer exists (snapshot_network.rs builds the manifest with routes: Vec::new() by construction). The rtnetlink RTM_NEWROUTE idea from this issue is obsolete, there are no routes to put back. Gate teardown now reuses the verify pass's -S parse (state_cache), so the reopen is down to two spawns, one iptables-restore per family. Those two could still become netlink/nftables writes if the reopen phase still dominates, but that needs a fresh phase measurement first since ef8741a changed the shape of the path.

Still open, verified on main:

• eth0 literals: fc-agent/src/network.rs (185, 224, 415, 440, 473, 496, 523, 868), src/network/routed.rs, src/commands/podman/vm_config.rs, and the Name=eth0 systemd unit plus mmds.conf drop-in in rootfs-config.toml:141-151.
• All three cache opt-outs remain (src/commands/podman/mod.rs:330-350). FirecrackerConfig still has no serialized boot-plan transport field, so the FCVM_BOOTPLAN=vsock fix from the issue body is still the small one. The #[serde(skip)] guard to watch is config.rs:31.
• reqscale exists (bench/chromium/reqscale.py) but the open-loop campaign has not been run; no records in the archive.
• The in-vivo repair-path cell is still missing. restore-matrix.py:294 asserts the repair path is NOT taken, which is the health check, not the tamper cell. Note the original motivation text ("asserting the link-down unconditionally") is stale: there is no link-down anymore, but the verified_armed == false reinstall path still exists and still has only unit coverage.
• kernels.yml still bootstraps btrfs inline twice (mkfs.btrfs at lines 335 and 560) instead of calling make setup-btrfs.

[ejc3]

Current-main audit (4c51dfb): the route-reinstatement proposal is obsolete because ef8741a removed link-down and route destruction. The old wedge-attribution item was already resolved by #817. rootfs_override remains a justified cold-boot requirement, not a cache bug.

Remaining items are guest-interface generalization after a CH reproduction, forced-vsock cache identity, CH automatic snapshot caching, one live tampered-boundary regression, and deduplicating kernel-workflow btrfs setup. The old 136 ms phase measurements cannot justify a present optimization without remeasurement. Optional performance experiments are not evidence of current runtime defects. Scope for those enhancements is being confirmed with the user; no blanket closure.