Runner autoscaler kills runners mid-job at the 12h age cap, and the failure reads as an AWS spot reclaim

[ejc3]

Every CI failure on #867, #872, #873, #874, #880, #881 and #883 tonight was a job dying with no failed step and the annotation "The self-hosted runner lost communication with the server". I read those as AWS spot reclaims for most of the night. They are not. We are terminating our own runners while they are executing jobs.

Evidence

EC2 records the reason, and it is not a reclaim:

i-09fff3a7d97fd4066  launched 2026-08-28T18:37:02Z  terminated  "User initiated (2026-08-29 06:41:34 GMT)"   age 12.07h
i-02fefa9deeb59e9c8  launched 2026-08-28T19:05:28Z  terminated  "User initiated (2026-08-29 07:06:34 GMT)"   age 12.02h

describe-spot-instance-requests agrees: the recent statuses are instance-terminated-by-user, not instance-terminated-no-capacity or -by-price.

Both died within four minutes of turning 12 hours old, which is MAX_INSTANCE_AGE_HOURS in ~/aws/runner-autoscale.tf:1002. The cleanup Lambda runs every 5 minutes, so a runner is killed in the first sweep after it crosses the cap:

# Hard age cap: reap regardless of busy state (see MAX_INSTANCE_AGE_HOURS)
if age_hours > MAX_INSTANCE_AGE_HOURS:
    print(f'Terminating over-age: {instance_id} (age=... busy={runner_info.get("busy", False)})')
    if runner_info.get('id'):
        deregister_runner(runner_info['id'], pat)
    ec2.terminate_instances(InstanceIds=[instance_id])

The log line even formats busy= and then terminates anyway. A job assigned to a runner at 11h55m is killed five minutes in, and the only signal anyone downstream sees is "lost communication", which is indistinguishable from a genuine reclaim.

Cost

Six reruns tonight on a fleet where a full matrix is roughly 35 minutes. The reruns are not free either: a rerun assigned to another near-cap runner dies the same way, which is what happened to #883 twice in a row (i-09fff3a7d97fd4066 then i-02fefa9deeb59e9c8, 25 minutes apart).

Fix

Drain instead of kill. At the cap, deregister the runner so GitHub assigns it no new work, then terminate only once GitHub stops reporting it busy, with a bounded grace window (a full matrix job is about 35 minutes, so 60 to 90 minutes is enough) after which the hard kill still applies so a wedged host cannot live forever. That keeps the property the cap exists for and stops it destroying work in progress.

Two smaller things worth doing at the same time:

• Log the decision on both sides. Terminating over-age ... busy=True should be loud, because today it is the only trace that the failure was ours.
• Reconsider whether the cap needs to be this aggressive for a fleet whose jobs are long. It exists to bound the blast radius of a wedged host, and the drain window handles that.

Related

#871 tracked the D-state wedge that motivated the age cap. This is the other half: the cap that fixed the wedge now destroys healthy work.

[ejc3]

Prior art: #834 recorded this same signature on 2026-08-14/15 (three jobs, no failing test, missing test-logs artifact for exactly the failed job). Its table already separates the two sub-cases: instances gone from EC2, which this issue explains, and an instance still running with the agent offline and SSH timing out, which is a different failure and remains open.

[ejc3]

The drain implementation is merged in ejc3/aws#44 (5c2f6f550497), with lease/registration follow-ups in ejc3/aws#46 and ejc3/aws#49. The red-first regression is case_busy_runner_past_the_soft_cap_is_drained_not_killed. This changes the unconditional 12-hour termination to a bounded drain policy; it does not prove deployment.

Deployment verification on September 8 is blocked by this host role: lambda:GetFunctionConfiguration is denied for github-runner-cleanup in us-east-1. No IAM or infrastructure changes were made. Keep this issue open until the deployed policy is verified. The documented hard-ceiling behavior must not be described as unlimited job draining.