validate: catch the host that passes every check and still eats every task

[rcurranmoz]

--quarantine-on-register holds a fresh host "pending validation" — and nothing validated anything. So the only way a host proved itself unfit was by failing real work. macmini-m4-242 destroyed 15 production tasks (mochitest, jsreftest, web-platform-tests) at ~43s each before anyone looked.

Its KVM presented 1280x1024@75Hz, and mozharness fatally halts a pre-test refresh-rate check at anything but 60Hz — so every task died before running a single test. Every other signal on that host was perfect: puppet green, sentinel present, worker up, disk fine, semaphores byte-identical to a working host.

reprovision validate <host>
reprovision batch <file> --action validate

Reading the display is the interesting part

Over a plain SSH login as admin, CoreGraphics has no window server and answers refresh=0.0, 0x0 — not an error, just zeros, which a naive check sails straight past. So ssh.display_mode() hops into the console user's session with launchctl asuser (the same technique the Safari automation uses) and treats all-zeros as unknown, never as fine.

Verified on a known-good/known-bad pair:

host	refresh	resolution
m4-242	75.00 Hz	1280x1024	failed 15/15 tasks
m4-243	60.00 Hz	1920x1080	healthy

Why this is not part of preflight

That's where I first proposed putting it, and it was wrong. preflight runs before the bootstrap, and cltbld does not exist until puppet creates it. There's no session to query, so a preflight version would silently pass exactly the fresh hosts it exists to catch.

Deliberate scoping

• Gates on refresh rate only, at 60Hz — the number mozharness itself enforces. validate should agree with what CI will decide, not invent a second standard.
• Resolution is reported, not gated — it varies legitimately across the fleet.
• A down worker warns rather than fails — these hosts reboot between tasks, so it can just mean we caught one mid-cycle.
• Exit 2 (skipped) when not yet bootstrapped, 1 when bootstrapped and unfit. Read-only throughout — safe on live workers.

216 pass, ruff check . clean on the pinned 0.15.19.

Ref: RELOPS-2515

🤖 Generated with Claude Code