[bong-water-water-bong]

Follow-up to the 10:54Z smoke FAIL in ~/briefs/DEMO-SMOKE.md, which I triaged from the run's own artefacts and found to be a reading failure, not a demo failure.

The incident, in three commands

$ tools/npu_serve_smoke.py --judge ~/.cache/scratch-orch/smoke-20261111T1030      # the run's real dir
rate         FAIL  0 of 0 replies carry no decode timing
first-token  FAIL  a reply carries no prompt_ms
content      FAIL  no replies                       ← three lines that read as a broken demo

$ cp …/serve.log /tmp/…/ ; cp …/npu.json /tmp/…/reply_0.json                       # the same bytes, renamed
$ tools/npu_serve_smoke.py --judge /tmp/…
route  ok  the serve log names the fast route
rate   ok  median 100.5 tok/s (floor 60.0)
first-token ok  median 279 ms (limit 600)
content ok  1 replies, all with content            RESULT: PASS

The demo was healthy the whole time — fast route line, 100.5 tok/s against a 60 floor, first token timings.prompt_ms = 279.458517 against a 600 limit, content "The capital of France is Paris.", and hold.log records "command exited 0 after 4s". The ad-hoc harness wrote npu.json; --judge looks for reply_0.json.

What this changes

"0 of 0 replies" is not a verdict about a run — it is a directory this reader cannot read. So rejudge() now refuses it by name:

serve       cannot judge: no replies in <dir>
            looked for serve.log, reply_0.json …, load.json; the .json files here are: npu.json
            if this is another harness's output, name its reply as reply_0.json and re-run --judge

rc 2, no verdict lines.

judge() is untouched and still FAILs on an empty reply list — that is the different case, a run that produced no reply, and it must keep failing. The distinction is the one #2039 drew between a skip and a failure, and the one DEMO-SMOKE.md itself makes in its "Reporting" section: a reading that cannot be trusted is worse than no reading.

Verified

$ tools/npu_serve_smoke.py --judge ~/.cache/scratch-orch/smoke-20261111T1030      → rc 2, message above
$ tools/npu_serve_smoke.py --judge <the renamed dir>                              → rc 0, RESULT: PASS
$ tools/npu_serve_smoke.py --self-test
npu_serve_smoke self-test: 16 judging cases, 3 runs against a stand-in serve and a TERM mid-request ok   rc 0

The self-test gains the arm: a directory holding serve.log + a npu.json (i.e. another harness's name) and no reply_*.json must return 2 — so the next person triaging a smoke FAIL is told they are looking at the wrong directory instead of after a broken demo.

Scope: one function plus one self-test arm in tools/npu_serve_smoke.py. No behaviour change on any run whose artefacts this tool wrote.

[github-actions]

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪

🧪 PR contains tests

🔒 No security concerns identified

⚡ Recommended focus areas for review Incorrect error handling for empty reply directories The rejudge function now correctly returns exit code 2 when a directory contains no reply_*.json files, instead of incorrectly reporting three FAIL verdicts. This prevents misinterpretation of a directory with no replies as a failed demo run, which was the case in the 2026-11-11 smoke test incident where an ad-hoc harness wrote npu.json instead of reply_0.json. def rejudge(args): d = os.path.expanduser(args.judge) with open(os.path.join(d, "serve.log"), errors="replace") as f: log_text = f.read() replies = [] i = 0 while os.path.exists(os.path.join(d, "reply_%d.json" % i)): with open(os.path.join(d, "reply_%d.json" % i)) as f: replies.append(json.load(f)) i += 1 # A directory with no replies is not a demo that failed - it is the wrong directory (or a harness that wrote # its reply somewhere else), and saying so is the difference between "the run failed" and "look again". The # 2026-11-11 10:54Z smoke was logged FAIL on exactly this: the ad-hoc harness wrote `npu.json`, `--judge` # looked for `reply_0.json`, and three FAIL lines read as a broken demo. The same bytes, renamed, give PASS. if not replies: found = sorted(f for f in os.listdir(d) if f.endswith(".json")) if os.path.isdir(d) else [] print("serve cannot judge: no replies in %s" % d) print(" looked for serve.log, reply_0.json …, load.json; the .json files here are: %s" % (", ".join(found) if found else "none")) print(" if this is another harness's output, name its reply as reply_0.json and re-run --judge") return 2

[bong-water-water-bong]

VERDICT at 8faab98: READY

"A directory with no replies says so" is the distinction this gate exists to make, applied to the gate itself. npu_serve_smoke is the check that the shipped binary's serve actually runs (#2119) — the thing the private NPU-phase checker could not see, which is why #2102 and #2106 were release blockers. A judge that reads a directory with no replies and exits quietly is the same defect one level up: it would report nothing rather than report that it could not tell.

The diagnostic is the part worth keeping:

if not replies:
    print("serve       cannot judge: no replies in %s" % d)
    print("            looked for serve.log, reply_0.json …, load.json; the .json files here are: %s" …)

Naming both what it expected and what is actually there turns "the gate says nothing" into a one-line answer — the difference between an operator re-running the smoke and an operator wondering whether the gate is broken. It is the same shape as #2019's "refuse by file and number" and #2141's suggested "name both commits": a check that cannot answer should say which inputs it looked for.

Non-zero rc on that path is the other half, and the more important one: "cannot judge" must not be exit 0 — a gate that cannot tell and exits successfully is worse than no gate, because it launders the unknown into a pass.

[bong-water-water-bong]

VERDICT at 8faab98: READY

Reviewed head 8faab98 (refs #2129 follow-up). This is the second verdict on this head.

• The change is confined to rejudge(). judge() and run() are untouched, so a live run that got no reply still goes through judge() on an empty list and FAILs (rc 1), as the new comment says it must. Only re-judging a directory with no reply_*.json is refused.
• By hand:
  • a directory with serve.log and only npu.json prints "cannot judge: no replies in …", names npu.json as the .json found, and exits 2;
  • the same serve.log with a real reply_0.json gives RESULT: PASS, rc 0.
• --self-test: "16 judging cases, 3 runs against a stand-in serve and a TERM mid-request ok". Mutant: with the new if not replies: guard disabled, the self-test fails (rc 1). I restored the file (git status clean).
• git merge-tree onto main: no conflicts. CI build and CodeQL green.

Nit: rc 2 is also what argparse returns for a usage error, so a caller can tell "cannot judge" from "bad arguments" only by the text. The hw-gate phase does not call --judge, so nothing depends on it today; a distinct code would make it unambiguous, should a script ever key on it.