Kernel patch audit vs 7.0.14: NV2 crash has an upstream fix we lack; 4 patches can be dropped

[ejc3]

Audit of every out-of-tree kernel patch against Linux 7.0.14, triggered by the 2026-08-08 host hang on fcvm-metal-arm. Full report lives at /home/ubuntu/patch-audit-7.0.14.md on the ARM box.

Headline: the crash is a known upstream bug with an existing fix that 7.0.14 does not have. Four of our nine patches can be dropped. The two NV2 barriers have never been A/B'd and one of them looks misplaced.

---

1. The nested.c crash — root cause found, fix identified

What happened

On 2026-08-08, fcvm-metal-arm (then on 6.18.3-fcvm-8ee6c35df0e1) hung hard under Firecracker load. EC2 instance status check went impaired at 12:19 with 0% CPU; SSH accepted TCP then never completed. The serial console held:

kernel BUG at arch/arm64/kvm/nested.c:754!
Internal error: Oops - BUG: 00000000f2000800 [#1] SMP
pc : kvm_vcpu_put_hw_mmu+0x214/0x2c0
lr : kvm_arch_vcpu_put+0x120/0x138
CPU: 35 UID: 0 PID: 3934971 Comm: fc_vcpu 1
Hardware name: Amazon EC2 c7gd.metal
Call trace:
 kvm_vcpu_put_hw_mmu+0x214/0x2c0 (P)
 kvm_arch_vcpu_put+0x120/0x138
 kvm_sched_out+0x6c/0xc8
 __schedule+0x954/0x1660
 schedule+0x34/0x148
 xfer_to_guest_mode_handle_work+0x94/0x160
 kvm_arch_vcpu_ioctl_run+0xb4/0xac8
 kvm_vcpu_ioctl+0x1a0/0xc30
note: fc_vcpu 1[3934971] exited with irqs disabled

The failing assertion:

if (host_data_test_flag(L1_VNCR_MAPPED)) {
        BUG_ON(vcpu->arch.vncr_tlb->cpu != smp_processor_id());   /* <-- this one */
        BUG_ON(is_hyp_ctxt(vcpu));

The invariant and how it is violated

The VNCR pseudo-TLB holds the translation of L1's VNCR_EL2 page, mapped into a per-physical-CPU fixmap slot while L1 runs. Its state is split across four places: the per-CPU L1_VNCR_MAPPED flag, vncr_tlb->cpu, the fixmap PTE, and kvm->arch.vncr_map_count. The assertion says: if this CPU's flag is set, the pseudo-TLB must name this same CPU.

The violating sequence, verified in git:

1. KVM maps a VNCR translation on CPU N, sets vt->cpu = N, sets L1_VNCR_MAPPED, bumps vncr_map_count.
2. A VNCR_EL2-triggered fault calls kvm_translate_vncr().
3. That calls invalidate_vncr(vt), which marks the pseudo-TLB invalid and clears the host fixmap PTE — but does not clear L1_VNCR_MAPPED or decrement the count.
4. On success it sets vt->cpu = -1 and defers the remap. The per-CPU flag is now stale.
5. Any schedule-out before repair reaches kvm_sched_out() → kvm_vcpu_put_hw_mmu(), which sees the stale flag, compares -1 against the current CPU, and trips the first BUG_ON.

CPU migration is not required — an ordinary schedule-out on the same CPU is sufficient. The thread dies with IRQs disabled inside the scheduler path, which wedges that CPU and then the box. (The "exited with irqs disabled" note reflects the scheduler context, not guest behaviour.)

The upstream fix — and our exposure

5949004d7032767e8fde1e8c986a33f241b2a192 — KVM: arm64: nv: Fully update VNCR fixmap state in kvm_translate_vncr() — Oliver Upton, 2026-06-02, committed by Marc Zyngier 2026-06-03, Fixes: 069a05e535496 ("KVM: arm64: nv: Handle VNCR_EL2-triggered faults").

Its commit message describes our exact failure:

While the fixmap invalidation does clear the mapping from host stage-1, it does not clear the L1_VNCR_MAPPED flag. Depending on the state of the VNCR TLB at vcpu_put(), this could potentially precipitate a BUG_ON() if vt->cpu is reset.

It factors full teardown into this_cpu_reset_vncr_fixmap() and calls it from both kvm_vcpu_put_hw_mmu() and kvm_translate_vncr().

Independently verified against the stable tree:

• The fix is ABSENT from v7.0.14 (git merge-base --is-ancestor against the fetched v7.0.14 tag). It is in no tag through v7.1-rc7; it merged for 7.2 via 751d041a13bd (kvmarm-7.2).
• The vulnerable code is present in the 7.0.14 source at arch/arm64/kvm/nested.c:806.
• We are running 7.0.14-fcvm-cd6cd2b4b52e right now, so we are still exposed.

Two things this corrects about the initial reading: the assertion still existing at v7.1-rc7 did not mean "unfixed" (the fix postdates rc7), and the fix deliberately keeps the assertion — the invariant is correct; it removes the state transition that violated it.

Related fixes in the same kvmarm-7.2 NV-MMU series, worth reviewing together: 4b54e2374d1b (hold mmu_lock while initializing vncr_tlb), 4be6cbeb93d2 (NULL VNCR pseudo-TLB deref), 6bef47288ce1 (stage-1 block-mapping validity), d8839941df7d / 699a2cc7f608 (restart walks after races), 9f76b039a72d (release PFN when VNCR translation races an MMU notifier).

Reportability

Do not file this as a new upstream bug — it is already known and fixed. Backport 5949004d7032 instead. A stock-7.0.14 reproduction is not needed to explain our failure, but is appropriate if we request a stable backport, since our trace came from a patched kernel and stable maintainers deserve a clean story.

---

2. Patch verdicts

kernel/patches-x86/ contains only symlinks to the two shared FUSE patches, so rows 1–2 cover both arches; the rest are ARM64-only.

Patch	Purpose	Upstream 7.0.14 status	Verdict
0001-fuse-add-remap_file_range (shared)	FICLONE/FICLONERANGE over FUSE via private opcode 54	No equivalent; UAPI enum ends at 53	KEEP, verify sizing
0002-fuse-fix-utimensat-... (shared)	UTIME_NOW wire semantics + extra permission path	FATTR_MTIME_NOW bug still present; permission rule now generic	KEEP, trimmed
mmfr4-override.vm.patch	arm64.nv2 alias forcing MMFR4.NV_frac=2	KVM understands NV_frac, but no boot override exists	KEEP
nv2-vsock-cache-sync.patch	dsb(sy) on every NV-capable vCPU exit	No equivalent through kvmarm-7.2	INVESTIGATE (retain pending A/B)
nv2-vsock-rx-barrier.patch	dsb(sy) in ARM64 virtio-vsock RX worker	No equivalent through kvmarm-7.2	INVESTIGATE (strong drop candidate)
wfx-stopped-exit.patch	WFx by a stopped vCPU → userspace whole-VM shutdown	kvm_prepare_system_event() already does this properly	DROP
psci-debug-emulate-nested.patch	pr_debug() around nested trap forwarding	Debug only	DROP
psci-debug-handle-exit.patch	pr_debug() around SMC forwarding	Debug only	DROP
psci-debug-psci.patch	Three unconditional [KVM PSCI DEBUG] pr_info()	Debug only	DROP

Detail on the non-obvious ones

wfx-stopped-exit — drop. Upstream kvm_prepare_system_event() already stops all vCPUs and initialises the complete shutdown event. Ours conflates "a stopped vCPU executed WFI" with "the whole VM is shutting down", and can pre-empt nested WFx forwarding. Local commit e1c2ea9bf696 established that poweroff -f (not halt -f) is what invokes PSCI SYSTEM_OFF, which is the behaviour this was reaching for.

psci-debug-* — drop all three. Logging only, no correctness change. psci-debug-psci.patch uses unconditional pr_info(), and those [KVM PSCI DEBUG] lines are visible in the host journal today — including in the crash console capture. They should not be in a kernel run day to day; if kept for NV debugging they belong behind a debug config, not in the default patch set.

The two NV2 DSB barriers — the real open question. Neither has an upstream equivalent through kvmarm-7.2, and the substantial post-6.18 VNCR/TLB work does not subsume them, so they cannot simply be dropped as "fixed upstream". However:

• nv2-vsock-rx-barrier appears to be on the wrong side of the data path. The Linux virtio-vsock RX worker consumes VMM→guest buffers; the corruption we were chasing is L2-guest-TX data read by L1 Firecracker. That path already uses virtio_rmb()/DMA synchronisation. This is the strongest candidate for removal.
• nv2-vsock-cache-sync's supporting evidence (local 9a37c89109f0) is a 100 MiB pass recorded without a same-build unpatched control. It has never been A/B'd.

Both fire on every exit / every RX worker invocation, so they carry ongoing cost for evidence we do not actually have.

mmfr4-override.vm — keep. 7.0.14 and v7.1-rc7 teach KVM about NV_frac (88aea41b9bc5, 642c23ea8b45) but provide no MMFR4 boot override, and the virtual-EL2 ID-register visibility problem is not moot. Note it is VM-only: the host_kernel build_inputs glob excludes *.vm.patch, so it is not in the host kernel.

0002-fuse-fix-utimensat — keep, trimmed. The FATTR_MTIME_NOW hunk is still needed (7.0.14 still suppresses it under writeback, fs/fuse/dir.c:2027-2032). The extra ATTR_FORCE/permission hunk is redundant since upstream f2b20f6ee842 made ATTR_TOUCH checking generic.

---

3. Separate concern: the mmio.c force-flush proof of concept

Not in the patch directories, but present as a local branch commit (f4f42c8c52f9, rebased 9930100e6114): KVM: arm64: Force cache flush for NV2 guests (bypass FWB).

On every NV-capable nested MMIO write it walks vcpu->arch.hw_mmu->pgt across 0–64 MiB, clean-invalidating every mapped granule, bracketed by DSBs, before kvm_io_bus_write(). It does not hold kvm->mmu_lock or mark the walk shared, though normal stage-2 flush/walk mutation does. A block mapping can also push a leaf granule past the nominal 64 MiB range.

Independently of the crash, this can race page-table mutation and is very broad. It labels itself a proof of concept and should be quarantined — and must not be included in any stock reproduction or upstream report.

It was not in the crash kernel: 6.18.3-fcvm-8ee6c35df0e1's /boot/System.map (built with CONFIG_KALLSYMS_ALL=y, CONFIG_LTO_NONE=y) contains neither nv2_flush_walker nor kvm_nv2_flush_guest_dcache, while containing ordinary KVM MMIO symbols. The walker is passed as a callback so it could not vanish through inlining.

---

4. Caveats and one correction

• A claim in the audit that did not survive checking. The report asserted that the 6.18.3→7.0.14 rebase (Repin deployable kernels 6.18.3 -> 6.18.44 (7.x wedges nested guests) #751) dropped the >4 GiB FICLONE fix from local commit 559437b95b11. Checked directly by diffing only the added kernel lines of the pre-rebase and regenerated patch files: identical, no regression. The STATX_SIZE refresh and derive-length-from-request logic are both intact. The commit header/date/subject did change, which is the likely source of the false positive. Sizing behaviour is still worth an explicit 10 GiB clone test, but nothing was lost in the rebase.
• Negative search results are weak evidence. "No public report matching our trace" is not proof none exists; the load-bearing evidence here is the upstream commit itself.
• The audit was research-only — no tests were run, so the DSB verdicts are analysis, not measurement. That is exactly why the factorial matrix below matters.
• Provenance: produced by a deep Codex run over the fcvm tree, a full upstream git tree with tags through v7.1-rc7, and the fetched stable v7.0.14 tag; key claims (the fix commit's existence, content, and absence from 7.0.14; the >4 GiB claim above) were re-verified by hand.

---

5. Next steps, ordered by value

1. Backport 5949004d7032 to the 7.0.14 host kernel and stress it. Applies with a four-line offset. Repeat the Firecracker/nested load that produced the BUG; capture console, KVM tracepoints, vCPU migrations. Consider testing the full kvmarm-7.2 NV-MMU series as a separate variant. This is the one item that protects against a repeat of the 25-minute hang.
2. Run a 2×4 DSB factorial matrix isolating the exit DSB, the Linux RX DSB, and the Firecracker event-handler DSB, with GiB-scale bidirectional raw-vsock and FUSE-over-FUSE integrity stress, repeated, with exit/throughput cost measured. Expect to remove the RX DSB — but let the evidence decide.
3. Drop wfx-stopped-exit and the three psci-debug-* patches. Verify poweroff -f, CPU_OFF/hotplug, WFI idle, nested WFx forwarding, and absence of log spam.
4. Explicitly test >4 GiB FICLONE (10 GiB clone with immediate size/data verification) so the sizing behaviour is proven rather than assumed, and ensure the test cannot silently skip on an unsupported kernel.
5. Narrow mmfr4-override.vm: A/B an unpatched VM kernel on the real NV2 stack, recording raw ID registers, feature detection, KVM_CAP_ARM_EL2, and L2/L3 launch.
6. Trim 0002-fuse-fix-utimensat to the FATTR_MTIME_NOW hunk and restore a writeback-enabled test across the owner/non-owner × NOW/explicit matrix.
7. Quarantine the mmio.c force-flush PoC; audit its locking and memory-attribute assumptions separately.
8. Only then consider upstream/stable submissions — nominate 5949004d for affected stable trees with transparent stock evidence, and treat the private FUSE opcode and any demonstrated NV2 coherency fix as separate submissions.

---

Context: the 6.18.3→7.0.14 rebase is #751. The host now runs 7.0.14-fcvm-cd6cd2b4b52e. Host-side diagnostics were also added on the AWS side so a recurrence reboots in ~2 minutes with the console captured, instead of hanging silently.

[ejc3]

Status against main as of dfd7b3b (2026-08-18). The headline exposure is superseded but most of the audit's actions are still open, so keeping this open as the tracking issue.

What changed: main no longer runs 7.0.14 at all. Commit 55fc053 (PR #751) pinned every deployable profile, including the nested host kernel, to 6.18.44 after an A/B/A showed all probed 7.x kernels wedge nested guests, and tests/test_remap_patch_invariants.rs now refuses 7.0.14/7.1.7/7.1.8 by name. Item 4 is done: test_ficlone_cp_reflink_in_vm (tests/test_remap_file_range.rs:175) does a 5 GiB FICLONE with exact-size and past-4GiB data verification, and the u32 truncation hunk is pinned by test_remap_patch_invariants.rs. The mmio.c force-flush PoC never reached this repo; there are no hits for it on main.

What has not happened:

1. The VNCR fix (5949004d7032) is not backported and nobody has checked whether 6.18.44 stable contains it. The original crash was on 6.18.3, so moving from 7.0.14 to 6.18.44 does not by itself close the exposure. First step is a one-command check in a kernel checkout: does v6.18.44 have "KVM: arm64: nv: Fully update VNCR fixmap state in kvm_translate_vncr()" (equivalently, does arch/arm64/kvm/nested.c define this_cpu_reset_vncr_fixmap). If yes, close this item and add a version-floor assertion to the pin test. If no, add the backport to kernel/patches-arm64/ with a fixture-based application test like the one added for the rx barrier in 10db034.
2. wfx-stopped-exit.patch and the three psci-debug-*.patch files are all still in kernel/patches-arm64/. Dropping them needs a nested-ARM validation pass (poweroff -f, WFI idle, nested WFx forwarding) on a metal box.
3. Both NV2 DSB barriers are still applied and still unA/B'd. The rx barrier was re-anchored for the rebase (10db034) rather than dropped, so the factorial matrix from step 2 of the issue remains the deciding evidence.
4. 0002-fuse-fix-utimensat still carries both hunks; the ATTR_FORCE/inode_permission hunk the audit called redundant has not been trimmed.

Items 1, 2, and 3 all need an ARM metal box to validate, which is also the box where the crash reproduces, so they should be batched into one metal session: verify or backport the VNCR fix, drop the four patches, and run the DSB matrix against the same load.

[ejc3]

The unresolved VNCR backport question is now checked against the exact pinned stable source. Linux v6.18.44 arch/arm64/kvm/nested.c still calls invalidate_vncr(vt) in kvm_translate_vncr(), later resets vt->cpu to -1, and has no this_cpu_reset_vncr_fixmap helper. invalidate_vncr only clears validity and the fixmap PTE; it does not clear L1_VNCR_MAPPED or decrement vncr_map_count. The upstream 5949004d7032767e8fde1e8c986a33f241b2a192 change repairs precisely that state transition. Moving to 6.18.44 therefore did not include this fix.

Sources:

• https://raw.githubusercontent.com/gregkh/linux/v6.18.44/arch/arm64/kvm/nested.c
• torvalds/linux@5949004

Next scoped change is this backport plus deterministic state/application tests. Barrier A/B experiments and removal of debug/WFx patches remain separate; none are being bundled into the backport. No host reboot, shared-host crash reproduction, or upstream submission has been performed.

[ejc3]

The VNCR backport merged in #920. Hardware validation on September 8 still reproduces a separate unresolved nested-execution failure.

Tested reviewed fcvm head f8b9ece376988d822bfc2ef5261dfc7391b7898e on an isolated c8g.metal-48xl (Graviton4, 192 cores). Both host 6.18.44-fcvm-ec792071f6fc and nested guest vmlinux-nested-6.18.44-aarch64-0f3464ff2fe9.bin include the VNCR fix. Other kernel patches were retained. The host booted with kvm-arm.mode=nested numa=off arm64.nv2 and reported VHE+NV2 mode initialized successfully.

Test	Observed result
Nested KVM availability	Passed, 29.736s
test_nested_run_fcvm_inside_vm, cold at both levels	Passed, 22.692s; actual inner command completed with INNER_FCVM_EXIT_CODE=0
Same test with snapshots enabled, two runs	L1 snapshot creation and restore completed, then starting L2 stalled; both runs recorded L1 soft lockups and RCU starvation
Large-file/standard benchmark	Interrupted during image build by EC2 API termination; no completed performance result

The cold case is a comparison control, not a workaround that makes the snapshot-enabled failure acceptable. These are whole-test durations, not VM boot latencies. The two failing runs were explicitly stopped after diagnostic capture at 478.338s and 151.795s, not allowed to reach their natural timeout. No host VNCR BUG/Oops was observed in these short runs; that is not a hardware stress proof of its absence.

A 10-second profile of the second failure captured actual interrupted pc,lr registers. Of 1,967 samples (33,041,392,123 summed cycle periods):

• 77.90%: __kvm_tlb_flush_vmid_range -> exit_vmid_context.
• 14.19%: read-lock waiting in kvm_fault_lock.
• 5.07%: write-lock waiting in kvm_mmu_notifier_invalidate_range_start -> kvm_handle_hva_range, or reacquisition through __cond_resched_rwlock_write.

The dominant samples arrive when interrupts are restored after the TLB invalidation region. They attribute CPU cost to host KVM mapping invalidation/TLB flushing and its MMU lock, not to the following instruction itself. Ordinary ARM perf guest attribution was misleading here because KVM marks samples as guest while a vCPU is loaded. Exact host ELF and interrupt registers resolve that ambiguity. The trigger for excessive invalidation is still unproved; this does not implicate the VNCR change or justify removing barriers.

The test instance was API-terminated at 15:32:42 UTC. The checked-in pbox watchdog's 5%/30-minute idle policy can classify two busy cores out of 192 as idle, but the caller is not confirmed: audit/log/metric reads were denied. Both EBS volumes survived. A recovery box retrieved the logs, profiles, exact host ELF, installed image package, guest kernels and Firecracker binaries, then was terminated at 15:42:47 UTC. No IAM or watchdog policy was changed.

Evidence is preserved on the dev host at /home/ubuntu/.local/state/fcvm-validation/pbox920.68P7iq/ and on the retained pbox work disk under /mnt/work/fcvm-vncr-920.btzXap/. The task scratch disk retains the build/cache tree for reuse. perf-intr.data SHA256: a54ec7c643bb3f548edd79b3ceaaabd4cadedf6b8f613c1b860fc07e1b4b6efa.

Next proof: identify which memory operation initiates the host MMU-notifier invalidations and compare that path between cold and restored L1 execution. Restored nesting, throughput measurement, the barrier A/B work, and debug/WFx patch removal remain unfinished. Keeping this issue open. No upstream submission was made.