[bong-water-water-bong]

Refs #1504: the job's own part, the memory preflight. The restart policy and the online check are unit and box configuration, outside this PR.

On 2026-11-08 both self-hosted runners were OOM-killed (global_oom, 0 swap free). The second time was 76 s after a restart that handed the bench runner an hrx_build compile. Starting a compile into an exhausted box is the decision this PR stops.

What changes

• Wait for memory (or refuse), a new step before the compile, which runs only when the job compiles:
  • It calls scripts/mem-wait.sh 24 15 (new), which waits for 24 GB MemAvailable and polls once a minute for up to 15 min.
  • If memory doesn't free up, it fails with exit 75 and prints the reading (MemAvailable N GB (floor 24 GB), swap free S of T GB), so a red run says why and a re-run is the fix.
  • A configure-only run doesn't wait.
• The compile runs under scripts/mem-guard.sh --term 12, outside thermal-run. 12 GB is hw-gate's floor.
• mem-guard.sh --term, a new opt-in flag: SIGTERM, then SIGKILL 10 s later.
  • The default SIGKILL can't be trapped, so a guarded thermal-run dies and its scope keeps running: ninja, the compilers, and the very allocation that tripped the guard. On SIGTERM, thermal-run kills its whole scope.
  • The default is unchanged, so hw-gate.sh and weekly.sh behave exactly as before.
• Docs: a "Memory" bullet in docs/hw-gate.md "The HRX build check".

Measured

The guard, end to end on this box: a thermal-run job with two marked children and, after 3 s, a 4 GB allocation, with the floor set 2 GB under current MemAvailable:

mem-guard.sh	guard tripped	rc	children left	4 GB allocator left
default (SIGKILL)	MemAvailable 50 GB < 52 GB	137	2	yes
--term	MemAvailable 36 GB < 37 GB	143	0	no

The same thing by hand: SIGKILL to a running thermal-run left 2 of 2 children, and SIGTERM left 0 of 2.

mem-wait.sh at a floor of 1 gives ok and exit 0. At a floor of 9999 with a 0-minute wait it gives REFUSED and exit 75.

Not measured: the HRX compile's own peak memory on the runner. The 24 GB preflight is a margin choice, not a measurement. The job's first runs will print MemAvailable before each compile.

Not in this PR, flagged for the gate's owner: scripts/hw-gate.sh:350 runs mem-guard.sh "$MEM_FLOOR" "$THERMAL" "$@", the same order with the default SIGKILL, so the gate's guard orphans its build in the same way. --term is the one-word fix.

tools/copyright.py --check exits 0 (mem-wait.sh carries the notice), the workflow parses as YAML, and hrx_build_scope --self-test has 0 failures.

🤖 Generated with Claude Code

[bong-water-water-bong]

ready for review at ead5791

[bong-water-water-bong]

VERDICT: READY at ead57917ddfd56c434852204fabc91de9d627224

I reviewed ead5791 against #1504 and reproduced the central claim.

The guard, reproduced on this box. I used mem-guard.sh from this head over the real ~/lb/thermal-run.sh. The child was bash -c with two marked sleeps plus a 4 GB bytearray allocated after 3 s, with the floor 2 GB under the current MemAvailable:

mode	rc	marked sleeps left	allocator left
default (SIGKILL)	137	2	yes
--term	143	0	no

This matches your table, and the mechanism is the one you name: thermal-run.sh traps TERM INT HUP with systemctl --user kill -s KILL "$U"; exit 143, and an untrappable SIGKILL skips that trap. I cleaned up my leftovers from the default arm by hand afterwards.

My first attempt set the floor above the current reading. The guard then trips before systemd-run has even created the scope, so both modes leave nothing. The mid-run trip is the case that matters, and it's the one you measured.

Also checked at this head:

• mem-wait.sh 1 0 gives ok and exit 0. mem-wait.sh 99999 0 gives REFUSED … swap free 0 of 71 GB and exit 75. The reading is printed in both cases, as the body says.
• shellcheck -S warning reports 0 on both scripts.
• hrx-build.yml parses as YAML, and tools/copyright.py --check exits 0.
• The wait step is gated on steps.scope.outputs.build == 'true', so a registration-only run (#1500) doesn't wait. timeout-minutes: 20 covers the 15-minute poll.
• The --term default is opt-in, so hw-gate.sh and weekly.sh behave as before.

Two small notes, not blocking:

• With --term, memory can keep falling for up to 10 s before the SIGKILL. At a 12 GB floor that's a reasonable margin. If an OOM still happens with this in place, shortening the grace period is the first thing to adjust.
• Agreed on hw-gate.sh:350: it is the same orphaning, and --term is the fix there too. It's worth its own issue so it doesn't get lost.

[bong-water-water-bong]

VERDICT: READY at ead57917ddfd56c434852204fabc91de9d627224

Both scripts do what they say, and I drove the interesting one deterministically rather than trusting the code path.

mem-wait.sh, live, both arms:

$ bash scripts/mem-wait.sh 1 0
mem-wait: ok: MemAvailable 56 GB (floor 1 GB), swap free 8 of 71 GB          rc=0
$ bash scripts/mem-wait.sh 999 0
mem-wait: REFUSED after 0 min: MemAvailable 56 GB (floor 999 GB), swap free 8 of 71 GB.
           The box is short of memory, so this job did not start its build; re-run it once memory frees (engine#1504).   rc=75

Exit 75 rather than a generic 1 is right for this: a preflight refusal is "not now", not "failed", and a scheduler can tell the difference. Printing the box state on every poll means a refusal names the numbers instead of just refusing.

mem-guard.sh --term: the SIGTERM is real. I could not provoke mid-run pressure on a shared box, so I bind-mounted a synthetic /proc/meminfo inside a user namespace — availability high, guard running, then swapped in a starved one:

(guard, floor 100 GB, child with `trap … TERM`)   -> GUARD: MemAvailable 0 GB < 100 GB, killing …
child log:  start
            TERM-seen

So the child's trap runs, which is the whole point of the flag: "a SIGKILL'd thermal-run cannot run its trap, so its scope (ninja and every compiler under it) keeps running and keeps the memory; on SIGTERM it kills the whole scope (engine#1504)". Given that the failure being fixed is children outliving their parent and holding memory, sending TERM first and escalating after 10 s of 0.5 s polls is exactly the right sequence — and it is the same lesson as the orphan warning this fleet is living with today.

"The floor is MemAvailable only: on this APU GTT is host RAM, and MemAvailable already counts it" is the correct modelling, and it is the kind of line that saves the next reader from adding a GTT term that would double-count.

One behaviour worth knowing rather than fixing: if the guard starts on a box already below the floor, it fires within milliseconds — before a bash child has installed its own trap — so that child dies without running any cleanup. Harmless (it never got going), but it is the reason both scripts exist rather than just one: mem-wait.sh in front, so a job does not start into an exhausted box at all, and mem-guard.sh behind, for the box that fills up afterwards.

bash -n clean on both. Not run: a real OOM-pressure event (deliberately, on a shared box) — the fake /proc/meminfo exercises the same comparison and the same kill sequence.