Self-hosted runner agents die mid-job, presenting as test failures (3x in one evening)

[ejc3]

Three CI jobs failed on 2026-08-14/15 with no failing test. All three were runner-agent deaths mid-job. Each cost a 20-40 minute job and, because the failure looks like a code failure, an investigation.

Signature (identical in all three):

• The job log stops mid-stream while the job keeps running for another 15-20 minutes.
• The test-root step never receives a conclusion; every later step stays pending (.steps[].conclusion == null), including "Upload test logs" — so the test-logs-* artifact is MISSING for exactly the failed job while every sibling job in the same run uploaded one.
• No TIMEOUT line, and tests were completing normally right up to the last emitted line. CI's nextest profile terminates any test after 2x120s and reports it, so a hung test would have been named.
• GitHub eventually marks the job failure.

Evidence per occurrence (all on PR #818's head 2917e07, three different machines):

job	runner	duration	instance state after
Host-Root-x64-SnapshotDisabled	runner-i-0d74c773f1e998a00	17:20:53 -> 18:00:57	terminated (gone from EC2)
Host-Root-x64-SnapshotDisabled	runner-i-0f0956aaa6475b7c6	18:23:44 -> 18:43:46	terminated (gone from EC2)
Host-Root-arm64-SnapshotDisabled	runner-i-02b10d79333d05c37	23:58:35 -> 00:32:38	instance still running, agent status=offline

The third case is the clearest: the EC2 instance is alive and the GitHub runner agent is simply gone. SSH to it also times out from an allowlisted IP, so the box is not healthy either.

Confirmation it is not the code: the same commit, rerun on a fresh runner, passed 15/15 with every required job present, including the Host-Root-arm64-SnapshotDisabled job that had died.

Why it concentrates on SnapshotDisabled: that configuration cold-boots every VM, making it by far the longest and most VM-heavy job, so it has the widest window to be caught by whatever kills the agent (scale-down/lease reclaim, or the host wedging under load).

Why it matters beyond the lost time: the failure is indistinguishable from a code failure at a glance, and it stops the run before the diagnostic artifact is uploaded — so the evidence needed to tell the two apart is exactly what gets destroyed. This session spent a long investigation, and produced one wrong attribution to a PR, before the runner data settled it.

Suggested directions (autoscaler lives in ~/aws/runner-autoscale.tf):

• Do not reclaim/terminate an instance whose runner reports busy=true; if a lease must expire, drain rather than kill.
• Emit a distinguishable signal when the agent dies mid-job (a post-job check, or an annotation) so a dead agent never reads as a test failure.
• Consider uploading test logs incrementally, or on a trap, so a killed job still leaves its diagnostics.

[ejc3]

Status on current main (dfd7b3b): the detection half of this is done, the prevention half is not.

What landed since filing:

• PR Classify a runner death whose log blob survives but stops growing #840 (merged c26ef93) taught scripts/classify_ci_failure.py the exact shape reported here: a completed failed job with one in_progress step and only pending steps after it, plus either a missing log blob (404 BlobNotFound) or a blob that stopped growing 120s or more before GitHub concluded the job. On the live API those populations do not overlap: healthy jobs write their blob within 0-2s of completed_at, agent deaths showed 569-1149s. Both classify as infrastructure_silent (_has_silent_loss_shape, classify_ci_failure.py:276, verdicts at :351-363).
• The Summary job now runs a "Diagnose runner loss" step on any failed run (ci.yml:1050) and annotates the run page with "::error::: the runner agent died mid-job (). This is NOT a test failure." So a dead agent no longer reads as a code failure.
• claude.yml's classify-ci-failure job reruns the failed jobs once automatically when every root failure classifies as infrastructure, on attempt 1 only. The lost 20-40 minute job now retries itself without an investigation.
• All of it is pinned: tests/test_ci_infrastructure.py covers the missing-blob, stale-blob, fresh-blob (negative), explicit-shutdown, and never-rerun-attempt-2 cases against fixtures in tests/fixtures/ci-infrastructure/, and tests/test_ci_workflow_coverage.rs fails if the ci.yml step disappears or names a script that does not exist.

On the third suggestion (incremental or trap-based log upload): the repo went the other way. Evidence that must survive an agent death is routed to channels that survive it, the D-state watchdog prints to the live step log and fc-agent pushes guest vitals to the console at failure sites (PR #842). The "Upload test logs" steps are still single if: always() steps (ci.yml:592, 869, 1012), so the per-VM debug logs still die with the runner. The classifier no longer needs them to attribute the failure, but they are unavailable for deeper debugging of a killed job.

Keeping this open for the first suggestion, which is the actual prevention. Nothing has touched the autoscaler in ~/aws/runner-autoscale.tf, and two of the three instances here were terminated mid-job, which looks like reclaim of a busy runner. The fix belongs in the scale-down path: check the runner's busy flag before terminating, drain instead of kill, and terminate only after the job concludes or a hard cap expires. Until that lands, each occurrence costs one automatic rerun instead of a false investigation, but the deaths themselves continue.
Fourth occurrence, 2026-08-18, same signature: job 95807132826 (Host-Root-arm64-SnapshotDisabled again, PR #860 attempt 1) concluded failure with the test-root step frozen in_progress, every later step pending including the log upload, the job log stopping mid-stream at 17:46:14 with no failing test, and runner-i-0bb1333e10a8686f1 reading status=offline busy=true afterwards. The rerun on a fresh runner passed the full matrix. That is four for four on the longest VM-heavy jobs, and the diagnostic artifact was destroyed each time.

[ejc3]

The first branch of this taxonomy has a root cause now: #884.

Your table already splits the signature two ways, and they turn out to be different failures:

"terminated (gone from EC2)" — explained. Our own cleanup Lambda terminates runners at MAX_INSTANCE_AGE_HOURS = 12 (~/aws/runner-autoscale.tf:1002) regardless of whether the runner is executing a job; the code formats busy= into its log line and terminates anyway. EC2 records it as User initiated, and the spot request says instance-terminated-by-user, not -no-capacity. Measured on 2026-08-29:

i-09fff3a7d97fd4066  launched 18:37:02  terminated "User initiated (06:41:34)"  age 12.07h
i-02fefa9deeb59e9c8  launched 19:05:28  terminated "User initiated (07:06:34)"  age 12.02h

That also explains the shape you describe: the agent is killed with the job in flight, so the step never gets a conclusion, later steps stay pending, and the test-logs-* artifact is missing for exactly that job while siblings upload theirs.

"instance still running, agent offline, SSH times out" — still open. Hit again on 2026-08-29 (i-0311760fa164e1391, 3h old, well under the cap, StateTransitionReason empty, GitHub reporting status=offline, SSH timing out). Not the age cap and not a reclaim. That branch is presumably #871's wedge, but it has not been proven.

Practical consequence while #884 is open: a rerun is not neutral. It can land on another near-cap runner and die identically, which happened twice in a row on #883, 25 minutes apart. Before rerunning, check the reason and the age:

aws ec2 describe-instances --instance-ids <id> \
  --query 'Reservations[].Instances[].{r:StateTransitionReason,l:LaunchTime,s:State.Name}' --output text

All three causes render identically on GitHub as "The self-hosted runner lost communication with the server", which is why this keeps costing investigations.