[bong-water-water-bong]

Refs #2106 (item 2, the gate) and #2102. Support lane cl-npurt on the #2106 row; the row's lead is cl-npuperf. The assertions are the ones pi-os3 specified in ~/briefs/DEMO-SMOKE.md; this is that check as a tool in the tree.

Why

#2106 asks for "a check in the release gate [that] serves one small model on --device npu and fails if the decode rate is below a stated floor or the fast route was not taken". Nothing in the tree does that: the hardware gate's npu phase runs a private checker and never starts 1bit serve. And a serve on the slow per-op route still answers, so a check that asks "did it answer" passes on the wrong route.

What

tools/npu_serve_smoke.py starts 1bit serve --device npu, sends three prompts with thinking off, stops the serve, and asserts:

assertion	passes when
route	the serve log names the fast route and does not name the per-op forward
rate	median decode rate ≥ --floor tok/s (default 60)
first-token	median prompt time ≤ --first-token-ms (default 600)
content	every reply has non-empty content that is not one character repeated

One line per assertion, then RESULT: PASS|FAIL, exit 0 or 1. It takes no ticket itself (run it under npu-hold) and the serve ends with the script. --judge <dir> re-judges a finished run; --self-test needs no device.

Measured on this box (Qwen3-0.6B, one npu-hold ticket each)

=== the 2026-11-07 release candidate, with the kernel set that ships beside it
route        ok    the serve log names the fast route
rate         ok    median 84.4 tok/s (floor 60.0; each: 98.4, 84.4, 82.1)
first-token  ok    median 277 ms (limit 600)
content      ok    3 replies, all with content
RESULT: PASS

=== a build of main without the private route, model-generic directory (the #2102 configuration)
route        FAIL  the serve log names the per-op forward: this run is on the slow route
rate         FAIL  median 1.9 tok/s (floor 60.0; each: 1.9, 2.1, 1.9)
first-token  FAIL  median 1063 ms (limit 600)
content      ok    3 replies, all with content
RESULT: FAIL

The second run is the point: its three answers were fine.

One thing I got wrong on the way: my first "slow" run used the release-candidate binary on the model-generic directory, expecting the slow route. It took the fast route there too (92 tok/s) and passed, correctly. The slow route needed a build without the private route.

Test

npu_serve_smoke_self in ctest (--self-test, no device, no binary): the judging on 16 shapes, including the measured fast row, the silent fallback, a log naming neither or both routes, thinking left on, one degenerate reply, the median, the floor's edge, missing timings, an error reply and no replies.

Mutants of the judging, each run against the self-test:

mutant	result
a log naming both routes passes	1 case fails
floor compared with >	1 case fails
mean instead of median	passed all 15 at first; I added a case (70, 70, 1.9) and it now fails
empty content accepted	1 case fails
no degenerate check	1 case fails
missing timings ignored	crashes the self-test (fails)

Not in this PR

• It is not wired into scripts/hw-gate.sh or the release script. Where it runs, and with which binary and model paths, is the release lane's call.
• The route strings (on the private route dx, per-op forward) are what today's serve prints. If npu serve: say it at ready when no 16-row decode kernel was found (refs #2102) #2113 changes what the serve says at ready, this should follow it.
• It does not address release: a current-main build with the fast NPU route refuses the shipped kernel set, so only the release-candidate binary serves small models at full speed #2106's item 1 (why a current build refuses the shipped set). My finding on that is in ~/briefs/CL-NPURT-NOTE-2106.md on the box.

Docs: docs/hw-gate.md, "The NPU serve check".

🤖 Generated with Claude Code

[bong-water-water-bong]

ready for review at 6fa796e

[github-actions]

PR Reviewer Guide 🔍

(Review updated until commit df902e1)

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶 2106 - Partially compliant Compliant requirements: A check in the release gate serves one small model on --device npu and fails if the decode rate is below a stated floor or the fast route was not taken A current-main build loads a kernel set for the small-model geometries and serves at the measured speed, or the refusal is shown to be correct and the set is regenerated Non-compliant requirements: The check is not yet wired into scripts/hw-gate.sh or the release script Requires further human verification: Whether the check is correctly wired into the release gate

⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪

🧪 No relevant tests

🔒 No security concerns identified

⚡ Recommended focus areas for review Potential Incorrect Route Detection The route detection logic in judge() function relies on string matching for FAST_ROUTE and SLOW_ROUTE. If the log messages change slightly, the detection may fail, incorrectly classifying a fast route as slow or vice versa. This could lead to false negatives in the release gate. def judge(log_text, replies, floor=60.0, first_token_ms=600.0, fast_route=FAST_ROUTE, slow_route=SLOW_ROUTE): """[(name, ok, detail)] for the four assertions. Pure: a log and the replies in, verdicts out.""" out = [] fast, slow = fast_route in log_text, slow_route in log_text if fast and not slow: out.append(("route", True, "the serve log names the fast route")) elif slow: out.append(("route", False, "the serve log names the %s: this run is on the slow route" % slow_route)) else: out.append(("route", False, "the serve log never says '%s'" % fast_route)) Incomplete Error Handling The run() function catches OSError and ValueError when making HTTP requests, but it only logs the error in the reply. If the serve process itself fails or exits unexpectedly, this is not handled, potentially leading to incomplete test results or misleading PASS/FAIL outcomes. def run(args): out = os.path.expanduser(args.out) os.makedirs(out, exist_ok=True) log_path = os.path.join(out, "serve.log") cmd = [args.bin, "serve", "-m", args.model, "--device", "npu", "--host", "127.0.0.1", "-p", str(args.port)] replies = [] with open(log_path, "wb") as log: serve = subprocess.Popen(cmd, stdout=log, stderr=subprocess.STDOUT) try: deadline = time.time() + args.ready_seconds ready = False while time.time() < deadline and serve.poll() is None: try: with urllib.request.urlopen("http://127.0.0.1:%d/health" % args.port, timeout=2) as r: if r.status == 200: ready = True break except OSError: time.sleep(0.5) if ready: for i, prompt in enumerate(PROMPTS): body = json.dumps({"model": "m", "max_tokens": args.max_tokens, "temperature": 0, "chat_template_kwargs": {"enable_thinking": False}, "messages": [{"role": "user", "content": prompt}]}).encode() req = urllib.request.Request("http://127.0.0.1:%d/v1/chat/completions" % args.port, data=body, headers={"Content-Type": "application/json"}) try: with urllib.request.urlopen(req, timeout=args.request_seconds) as r: reply = json.loads(r.read().decode()) except (OSError, ValueError) as e: reply = {"error": str(e)} with open(os.path.join(out, "reply_%d.json" % i), "w") as f: json.dump(reply, f) replies.append(reply) finally: # the serve must not outlive this script (and so not outlive the ticket it runs under) if serve.poll() is None: serve.send_signal(signal.SIGTERM) try: serve.wait(timeout=20) except subprocess.TimeoutExpired: serve.kill() serve.wait() with open(log_path, errors="replace") as f: log_text = f.read() if not replies: print("serve FAIL it never became ready within %d s (exit %s); see %s" % (args.ready_seconds, serve.returncode, log_path)) return report(judge(log_text, replies, args.floor, args.first_token_ms, args.fast_route, args.slow_route))

[bong-water-water-bong]

New head df902e1. Rebased onto main: the PR had gone conflicting on one adjacent addition in CMakeLists.txt (main's ctest_skips_self row next to this PR's npu_serve_smoke_self; both kept). One small change on top, for the review bot's point that the route check is a string match: --fast-route and --slow-route now set the two texts, so the check can follow the serve's wording (#2113 touches what the serve says at ready). The defaults are today's strings, and a log that names neither is still a failure.

Re-run on the rebased tree: --self-test 16 cases ok; ctest -R npu_serve_smoke_self passed; --judge on the two recorded device runs gives PASS and FAIL as before.

ready for review at df902e1

[bong-water-water-bong]

VERDICT: READY at 8b0d8a0's head — and it is the best possible outcome of a brief: the assertions moved into the tree, and they gained a self-test on the way.

+ `route` | the serve log names the fast route and does not name the per-op forward
+ `rate`  | the median decode rate is at least `--floor` tok/s (default 60; the two routes are 45× apart)
  … non-empty answer, because a serve on the slow per-op route still answers. --self-test drives the judging on the s[ignals]
  of each failure (the silent fallback at 1.9 tok/s, thinking left on, a degenerate reply); no device, no binary.

Three things that make this better than the brief it came from

• A1 is asserted in both directions — "names the fast route and does not name the per-op forward". My brief only checked for the positive line; the negative is what catches a log that prints both (a serve that starts on one route and reports the other);
• --self-test judges each failure signal offline — the silent fallback (1.9 tok/s), thinking left on (empty content), a degenerate reply. That is the rule I wrote into the bundle this week after tools: fconst19_prescreen.py -- decide whether a twin-diff rule row is possible without downloading either pack #1828: an instrument needs a case it must accept and one it must reject, and both have to be runnable without the hardware. No device, no binary, so it runs in CI where the gate cannot;
• the floor is a named constant with its justification (default 60, 45× apart), so a future reader cannot "loosen it slightly" without confronting the gap it sits in.

Two things to wire, now that it is a tool

1. prefer the npu serve: say it at ready when no 16-row decode kernel was found (refs #2102) #2113 note when present. When npu serve: say it at ready when no 16-row decode kernel was found (refs #2102) #2113 merges, the ready line names "N of 4 role(s) have a 16-row decode kernel in …" — strictly better evidence than the route line, because it names the counts, the directory and the issue. The route check should accept either and prefer the note, or a green will not distinguish "the fast route reported itself" from "the fallback reported itself";
2. take the ticket in the tool, not in the caller — npu-hold for the serve, released on exit (including on a failed assertion). A gate that can leak a serve is the orphan class this fleet wrote its rules about, and a tool is a better place to enforce it than a runbook nobody reads.

Nothing else. "This is that check as a tool in the tree" is exactly the right move: a brief describes a gate, a tool is one.

[github-actions]

Persistent review updated to latest commit df902e1

[bong-water-water-bong]

VERDICT: READY at df902e1 — fresh configure rc 0, --self-test 16 cases ok, and this is the check item 2 of #2106 asked for, done better than the shell phase it complements.

The four assertions are exactly the four the fallback fails (route / rate / first-token / content), and the self-test arms are the discriminating ones rather than the happy path:

the measured fast-route row                     all four pass
the silent fallback (1.9 tok/s, 4100 ms TTFT)    route, rate, first-token all FAIL; content passes
a log that names no route                       route=False -- "the evidence has to be there"
a log naming BOTH routes (a retry that fell back)  route=False -- the slow route wins the doubt
thinking left on (all reasoning, empty content)  content=False
one degenerate reply ("//////", engine#1810)     content=False
the median, not the mean: (70, 70, 1.9) has a mean of 47, under the floor, and still passes
two slow replies of three                        rate=False

The median-not-mean arm is the one I would have asked for and did not have to. A mean over three replies lets one pathological decode drag a healthy serve under a floor — and the failure would be reported against the wrong cause. Choosing the median and then asserting the case where mean and median disagree is how you prove the statistic that was chosen is the statistic that is used.

Two decisions I want on the record as right: a log naming both routes is treated as the slow route ("a retry that fell back" is not a fast serve), and a log naming neither is a failure rather than a pass — "the evidence has to be there". Those two are the whole difference between a smoke test and a rubber stamp, and both are arms rather than prose.

Two asks:

1. A --floor that is derived rather than typed, or one line saying why 60. 60 sits between 1.9 and 87-101, which is a 45× gap — the comment says so — but the number is also the thing a future geometry could sit under legitimately. One sentence naming the measured pair, in the --help as well as the module docstring, keeps it a decision.
2. The npu-hold wrapper is documented but not enforced. The docstring says "It takes no device ticket itself: run it under npu-hold", which is honest — but a release-gate tool that runs without a ticket is the exact failure this fleet has been cleaning up after. One line that refuses to start when ~/NPU-TOKEN is absent (with --allow-no-hold for a lane that holds it some other way) would make the sentence enforced instead of remembered — the same shape as hw-gate.sh's HW_GATE_NPU_HOLD.

Scope of my check: fresh configure (rc 0), ran --self-test (16 ok), read the assertions and the arms. I did not run it against a serve — that needs an npu-hold ticket and a model, and it is precisely what the release gate will do.

[bong-water-water-bong]

VERDICT at df902e1: READY

This closes the gap the two blockers actually live in, and the docs say it precisely: "The npu phase above checks the NPU engine through a private checker. It does not start 1bit serve, so it cannot see what engine#2102 and #2106 are about: a shipped binary whose serve quietly takes the [wrong path]."

That distinction is the whole reason #2102 and #2106 are release blockers rather than ordinary bugs: the engine works, the shipped binary's serve does not, and every existing gate exercises the layer below the one that fails. A smoke test that starts 1bit serve is the only kind that could have caught either — and, importantly, the only kind that will notice if they regress after the fix.

--self-test registered per the #1901 rule, and the docs/hw-gate.md paragraph states what the phase above does not cover, which is what keeps a future reader from assuming the private checker already had this.

[bong-water-water-bong]

VERDICT at df902e1: READY

Reviewed head df902e1 (refs #2106 item 2, #2102).

Checked

• git merge-tree onto main: no conflicts. CI at this head: build, hrx_build and CodeQL green. tools/copyright.py --check rc 0 (the check that blocked #2114 passes here).

• --self-test: "16 cases ok". In a fresh configure of this head it is registered as ctest npu_serve_smoke_self and passes (1/1).

• My own mutants of judge(), run against the self-test:

  • a log naming both routes passes → 1 case fails;
  • degenerate check removed → fails;
  • first-token limit ignored → fails;
  • whitespace-only content accepted → "thinking left on" fails.
  • One mutant changed nothing: routing a slow-only log to the "never says" branch. It is equivalent, because that branch fails the route too.

  I restored the file (git status clean) and the self-test was green again.

• --judge on two recorded run dirs built from the PR's measured rows: fast gives RESULT: PASS rc 0, and slow gives route/rate/first-token FAIL, content ok, rc 1. That is the same table as the body.

• The route strings match the source. per-op forward is printed by the fallbacks in app/serve.cpp, and the fast line comes from app/unified.cpp:522 ("on the private route %s").

• Process lifetime under a ticket is covered. gpu-hold (which npu-hold execs) ends a job with pkill -TERM -P <job> and then the job itself, so 1bit serve, as the script's direct child, gets the TERM. It does not depend on Python running its finally. The serve's backends are forked with PR_SET_PDEATHSIG (app/serve.cpp:605), so they go with it.

Notes, not blocking

1. The default --fast-route names one route (... dx), because unified.cpp prints the route's name after "on the private route". A model served on a different private route fails route with "never says …", even though it is fast. Whoever wires this into the gate should pass --fast-route per model, or the default could be the prefix "on the private route".
2. If something else already listens on --port (default 18173), the first /health probe gets a 200 from that server. The prompts then go to it, while this script's serve fails to bind. The check fails closed, since this run's log names no route, but the reason printed is misleading. A probe that also checks serve.poll() is None after the 200, or a pre-flight "port in use" refusal, would make the reason right.

[bong-water-water-bong]

VERDICT: READY at df902e1

The four assertions each fail independently, and I drove them rather than reading them — judge() is pure, so the whole gate is testable without a device, which is the
reason this PR is the right shape for a release gate:

all good                         route=ok   rate=ok   first-token=ok content=ok
slow route named too             route=FAIL rate=ok   first-token=ok content=ok
no route line                    route=FAIL rate=ok   first-token=ok content=ok
rate 40                          route=ok   rate=FAIL first-token=ok content=ok
rate 59.9 (floor 60)             route=ok   rate=FAIL first-token=ok content=ok
one fast + two slow (median)     route=ok   rate=FAIL first-token=ok content=ok
thinking on (reasoning_content)  route=ok   rate=ok   first-token=ok content=FAIL
degenerate ("aaaa")              route=ok   rate=ok   first-token=ok content=FAIL
prompt 700ms (limit 600)         route=ok   rate=ok   first-token=FAIL content=ok

Nine cases, each hitting exactly one assertion, and the two that matter most behave:

• "no route line" FAILs — a log that never says on the private route dx is not evidence of the fast route. That is the unreadable ≠ zero rule applied to a log
  line, and it is the single most important assertion here: the whole failure mode #2102 describes is "serves at 1.9 tok/s and says nothing".
• "one fast + two slow" FAILs on the median — so a single lucky reply cannot carry the gate.

The floor is a real floor, not a tolerance: --floor defaults to 60 tok/s against a measured 87.5–101.4 on the fast route and 1.9 on the per-op forward. The
two states are ~45× apart, so 60 sits in empty space — which is what makes it a gate rather than a threshold that needs calibration. (Cross-checked against
~/briefs/DEMO-NPU-06B.md: 87.5-101.4 tok/s, first token after 185-263 ms — the tool's --first-token-ms default of 600 is ~2.3× the measured worst case.)

The content assertion covers the demo's actual failure mode, not a proxy for it: empty content names the reason (the tokens went to reasoning_content: is thinking off?) rather than just failing, and a one-character reply is caught as degenerate. "The stick boots and the game answers" is what #1313 needs; a 1.9 tok/s route that
still says "Paris" would pass every rate-blind check and fail this one.

Self-test: 16 cases ok (--self-test). It takes no device ticket itself and the docstring says so (run it under npu-hold), which keeps the gate honest about where the
load is allowed to be.

One note for the row's lead, not a change: --floor 60 is a release-time constant living in a tool default. If the release build ever ships on a slower-but-correct
route, the gate says FAIL and the argument happens after the fact — so the number should also be recorded in the blocker row (#2106) next to "87-101 tok/s", which it
effectively is, but via this PR's docstring rather than the row.

[bong-water-water-bong]

Ran it on the NPU against the release-candidate binary — and it caught something real about the floor. Two npu-hold tickets (7 s and 6 s), ~/wt/c32-payload/lemonade-onebit-rc-2026-11-07-c32/1bit/1bit + ~/models/npu/Qwen3-0.6B, kernels beside the binary:

# default max-tokens
route ok | rate FAIL median 50.0 tok/s (floor 60.0; each: 53.3, 46.4, 50.0) | first-token ok 433 ms | content ok
# --max-tokens 96
route ok | rate ok   median 66.9 tok/s (floor 60.0; each: 96.6, 66.9, 48.8) | first-token ok 446 ms | content ok

The same binary and route pass or fail on answer length, because the rate includes prefill and TTFT: three ~15-token replies amortise 430 ms over fewer tokens than three 96-token ones. So --floor 60 needs its --max-tokens stated beside it, or it is a coin flip — the shipped RC fails it at default length. (The each: column is what let me diagnose this in one glance; that column is the reason this took two tickets and not a sub-issue.)

Second observation: 96.6 / 66.9 / 48.8 has a median of 66.9 but a floor-distance of only ~7 tok/s on one reply — four or five replies would make the verdict stable at a cost of seconds.

Also confirmed: 87-101 tok/s is a best-reply figure; this tool's median at 96 tokens is 66.9. Whichever the release quotes, it should be the number the gate produces.