Observed 2026-08-28 on runner-i-066ffbdeef2c172d4 (c7gd.metal), job Host-Root-arm64-SnapshotEnabled for #867 head 55892ac (a PR that changes only scripts/cargo-target-link.sh and its tests). The cold pass was 1185/1185. In the warm pass the D-state watchdog fired at 11:52:33Z (about 3 minutes in) on firecracker-def pid 4128025 (fc_api and fc_vcpu siblings in S), again at 11:55:37Z, and the pass ended 11 failed / 4 timed out with VM timeout after 570 seconds, process did not exit within 60s on a SIGTERM'd baseline, and a firecracker that survived fcvm's SIGKILL. The post-job stray guard counted 161 stray process groups. Among the failures: a_clone_dies_when_its_memory_server_dies, concurrent_sigkill_reaps_clones_parked_on_unserved_faults, test_startup_snapshot_bridged, test_ficlone_cp_reflink_in_vm, test_restored_clone_reboot_comes_back_healthy.
Live on the runner at 12:55Z (read-only), 153 firecracker processes, eight or more in D for 47 to 58 minutes, all on the same stacks:
[<0>] __synchronize_srcu+0xfc/0x110
[<0>] synchronize_srcu+0x194/0x228
[<0>] mmu_notifier_unregister+0xc8/0x198
[<0>] kvm_destroy_vm+0x154/0x318
[<0>] kvm_device_release+0xf8/0x170
[<0>] __fput+0xe4/0x330
[<0>] fput_close_sync+0x4c/0x138
[<0>] __arm64_sys_close+0x44/0xa0
and, for a process in exit:
[<0>] __synchronize_srcu+0xfc/0x110
[<0>] synchronize_srcu+0x194/0x228
[<0>] __mmu_notifier_release+0xdc/0x2a8
[<0>] exit_mmap+0x40c/0x470
[<0>] __mmput+0x3c/0x1b8
[<0>] do_exit+0x28c/0x9e8
MemAvailable 100 GB, HugePages 512/512 free, so not memory. Seven userfaultfd descriptors were still held by serve processes.
Reading: an SRCU grace period on the mmu-notifier SRCU never completes, so every later VM teardown on the host blocks in D forever; the runner then fails every job it takes (it picked up #868's Container-arm64 at 12:49Z while wedged). A reader holding the SRCU read side indefinitely fits a vCPU parked in an unserved userfault after its memory server died, which is the case concurrent_sigkill_reaps_clones_parked_on_unserved_faults and a_clone_dies_when_its_memory_server_dies exist for, and both failed in this pass; #817 (EEXIST wake) closed one such hole. Either a hole remains on arm64 under the warm pass, or the reap ordering (pidfd edge -> SIGKILL -> uffd release) can still leave a faulting vCPU inside the SRCU read side.
Next: reproduce on an arm64 box with the two parking tests looped under the warm pass; capture /proc/<vcpu tid>/stack of the PARKED clone (the reader), not only the wedged destroyers; check whether the serve's uffd fd outlives the SIGKILLed clone. Runner recovery needs a reboot: D-state here is not killable.
Observed 2026-08-28 on runner-i-066ffbdeef2c172d4 (c7gd.metal), job Host-Root-arm64-SnapshotEnabled for #867 head 55892ac (a PR that changes only scripts/cargo-target-link.sh and its tests). The cold pass was 1185/1185. In the warm pass the D-state watchdog fired at 11:52:33Z (about 3 minutes in) on
firecracker-defpid 4128025 (fc_api and fc_vcpu siblings in S), again at 11:55:37Z, and the pass ended 11 failed / 4 timed out withVM timeout after 570 seconds,process did not exit within 60son a SIGTERM'd baseline, and a firecracker that survived fcvm's SIGKILL. The post-job stray guard counted 161 stray process groups. Among the failures:a_clone_dies_when_its_memory_server_dies,concurrent_sigkill_reaps_clones_parked_on_unserved_faults,test_startup_snapshot_bridged,test_ficlone_cp_reflink_in_vm,test_restored_clone_reboot_comes_back_healthy.Live on the runner at 12:55Z (read-only), 153 firecracker processes, eight or more in D for 47 to 58 minutes, all on the same stacks:
and, for a process in exit:
MemAvailable 100 GB, HugePages 512/512 free, so not memory. Seven userfaultfd descriptors were still held by serve processes.
Reading: an SRCU grace period on the mmu-notifier SRCU never completes, so every later VM teardown on the host blocks in D forever; the runner then fails every job it takes (it picked up #868's Container-arm64 at 12:49Z while wedged). A reader holding the SRCU read side indefinitely fits a vCPU parked in an unserved userfault after its memory server died, which is the case
concurrent_sigkill_reaps_clones_parked_on_unserved_faultsanda_clone_dies_when_its_memory_server_diesexist for, and both failed in this pass; #817 (EEXIST wake) closed one such hole. Either a hole remains on arm64 under the warm pass, or the reap ordering (pidfd edge -> SIGKILL -> uffd release) can still leave a faulting vCPU inside the SRCU read side.Next: reproduce on an arm64 box with the two parking tests looped under the warm pass; capture
/proc/<vcpu tid>/stackof the PARKED clone (the reader), not only the wedged destroyers; check whether the serve's uffd fd outlives the SIGKILLed clone. Runner recovery needs a reboot: D-state here is not killable.