tart_health_agent: fix three bugs that made guest probes collect nothing

[rcurranmoz]

Enabling tart_health_guests on 2026-08-17 produced guest_reachable: false on 26/26 slots with
no guest fields at all, so it was reverted the same day (ronin_puppet #1336 → #1338). Three
independent bugs, each verified against a live host.

1. The probe never reached the guest — Tcl ate it

GUEST_PROBE is multi-line and contains $(...), $4, [0-9a-f]. It was interpolated into a Tcl
double-quoted spawn argument, so Tcl substituted the $ and treated [ ] as command
substitution:

extra characters after close-quote
    while executing "spawn ssh ... "uptime | sed -n 's/.*up \([^,]*\)...

expect died before connecting. Now base64'd — alphanumeric plus +/= is special to neither Tcl nor
the remote shell, whatever the probe grows into later.

2. The password was never sent

expect { -re "(P|p)assword:" { send "admin\r"; exp_continue } timeout { } eof { } }

Measured against a real guest this printed the prompt and then sat until the 60 s timeout. The
multi-line glob form with an explicit expect eof returns all five keys. Side-by-side on
macmini-m4-236: glob → probe_ok=1 + disk 61; -re single-line → prompt, then nothing.

3. guest_up_s was an epoch, not a duration

sysctl -n kern.boottime prints:

{ sec = 1786987025, usec = 7795 } Mon Aug 17 17:17:05 2026

.*sec = is greedy, so it matched usec = and captured the microseconds. guest_up_s
became now - 7795 ≈ 1.79e9. Measured 1786979442 on a guest whose uptime said 4 mins; anchored
it reads 263.

Also: the pid lookup, broken by the vault rollout

pgrep -f "[t]art run --no-graphics $vm"

Enabling tart.inject_vault fleet-wide changed the command line to tart run --no-graphics --dir=vault:/Users/admin/.tart-vault/<vm>:ro <vm>, so --no-graphics and the VM name stopped being
adjacent, the pattern matched nothing, and tart_run_uptime_s was None on 25/26 slots. Now
anchored on the VM name at end-of-line, which is flag-agnostic.

Verified end to end

--dry-run against macmini-m4-236, from a staged copy (deployed tree untouched):

field	slot 1	slot 2
guest_reachable	True	True
guest_uptime_s	368	616
guest_disk_free_gib	61 (rolled)	45 (not rolled)
clock_skew_s	0	−6
tart_run_uptime_s	5760	10464

That disk column is the signal we actually wanted — it separates rolled from un-rolled slots.

⚠️ Do NOT re-enable tart_health_guests on the strength of this alone

Now that the data arrives, it shows Hangar's reboot-loop heuristic is wrong for this pool:

guest_uptime_s < tart_run_uptime_s * REBOOT_LOOP_RATIO   # once tart run is up > 1h

These guests are numberOfTasksToRun: 1 — they reboot after every task, so guest_uptime is one
task (600–2400 s) while tart run is days. Both slots above satisfy the condition and would
report CRIT "guest rebooting under a healthy tart run". The real loop was ~84 s.

A ratio cannot separate those. It needs an absolute floor, or better tasks-resolved-count, which
was 0 on every genuinely looping slot. That threshold lives server-side in hangar
(backend/app/api/tart_health.py) and is the remaining blocker — otherwise we trade 24 false warns
for ~24 false crits, which is worse.

🤖 Generated with Claude Code