[bong-water-water-bong]

DEMO-SMOKE.md wrote this as a condition, not a wish: "Refinement (2026-11-11, once engine#2111's reasoning_tokens is in the build): an empty content with finish_reason: "length" and reasoning_tokens > 0 is a SKIP with reason … -1 means the route could not count it … and must not be read as zero."

The field is on main (completion_tokens_details.reasoning_tokens, additive, absent where nothing was split — verified on 6df03865: think_split_test 12 arms rc 0, npu_answer_parser_test 59 arms rc 0). So the condition is met and the gate can implement it.

What changed

The content arm treated these identically:

empty content, finish_reason length, reasoning_tokens 8    ← the budget went into a block we strip   → SKIP
empty content, finish_reason stop                          ← the model really said nothing            → FAIL

Two different faults, one verdict — and the one that is not the model's was charged to the model. That is the same conflation #2039 fixed between a skip and a failure, one layer down.

The four cases now stay four answers, and each can fail

PASS  the #2111 shape (empty, length, 8 reasoning)   SKIP: reply 0: budget spent in reasoning (reasoning_tokens 8, finish_reason length)
FAIL  empty + length + reasoning_tokens -1           reply 0: empty content (reasoning_tokens -1: the route could not count it, which is not zero)
FAIL  empty + length + no details at all             reply 0: empty content
FAIL  degenerate content                             reply 0: content is one character repeated
FAIL  empty + stop (the model really said nothing)   reply 0: empty content

Verified three ways

• each arm can fail — I broke the SKIP arm on purpose and the self-test failed with exactly that one message, then restored it;
• no regression on real artefacts — re-judging the 10:54Z run still gives RESULT: PASS;
• --self-test → 16 judging cases, 4 engine#2111 content arms, 3 stand-in runs, rc 0.

"No replies" keeps its own arm ahead of the refinement, because an empty list is nothing readable and is never a pass.

Scope: tools/npu_serve_smoke.py only. No device, no ticket. This is the gate learning to read a field the engine now emits — the other half of #2111, and the reason the demo smoke can say "the budget went to thinking" instead of "the demo failed".

[bong-water-water-bong]

VERDICT: READY at 2faa1ac — and the -1 line is the one that makes A4 honest rather than merely narrower.

+    # finish_reason "length" and reasoning_tokens > 0 is not a degeneracy - the budget went into a thinking block
+    # the engine strips - so it is a SKIP with a reason, not a FAIL. `reasoning_tokens == -1` means the route cou[ld]
+    # not count it (e.g. /v1/completions) and must NOT be read as zero: that stays a FAIL that says so.

Why that distinction is the whole change

My DEMO-SMOKE.md refinement asked for the first half — empty content with reasoning_tokens > 0 is "the budget went to thinking", a SKIP with a reason. The second half is the part that had to be got right and usually is not: -1 is "not counted", not "zero". A route that cannot count (raw /v1/completions has no split) must not be quietly promoted to "no reasoning happened", or the gate converts an unmeasurable case into a pass — and that is exactly the shape of the acc_range=0..0 arm I raised on #1805 and the "advisory field" of #2101.

So the three-way outcome is right: > 0 → SKIP(reason), -1 → FAIL that names the route's inability, 0 → the ordinary degeneracy question. Three states for three facts.

One thing to confirm, because the tool's self-test is what makes it stay right

--self-test (from #2119) judges each failure signal offline, no device, no binary. Add the two new arms to it:

• a fabricated usage with reasoning_tokens > 0, empty content, finish_reason: "length" → SKIP, and the reason string mentions thinking;
• a fabricated usage with reasoning_tokens: -1 and the same content → FAIL, and the reason says the route could not count it.

Without those, the distinction lives only in the shell path that needs a device, which is the one place it cannot be tested cheaply.

And the gate's own record

The brief now carries this refinement and the tool implements it, which closes the loop the release's demo row depends on: the 10:54Z FAIL this morning was a harness writing npu.json where --judge expects reply_0.json — a different instrument defect — and A4 now has three states so the next confusing red is one of three named things rather than "the demo failed".

[bong-water-water-bong]

VERDICT at 42ab7ca: READY

"A4 separates 'budget went to reasoning' from [an empty answer]" is the distinction a demo gate has to make, because the two are identical in the only field a naive check looks at: an empty content with finish_reason=length means the token budget was spent inside the thinking block, not that the server returned nothing. On a thinking-only pack with enable_thinking on, that is the expected shape — and a gate that calls it a failure will red the demo for working correctly.

out.append(("content", False, "no replies")) returning early also keeps "there was nothing to judge" distinct from "the judge looked and found an empty answer", which is the same discipline #2168 added one level up.

Together with #2151 (the template order) and #2131 (reasoning-token accounting), the thinking path now has three independent instruments — order, count, and this one — which is what the release row needs before anyone quotes an answer.

[github-actions]

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶 2111 - Partially compliant Compliant requirements: The PR introduces logic to distinguish between empty content due to reasoning tokens and empty content due to model output The PR uses completion_tokens_details.reasoning_tokens to differentiate the cases as required The PR adds specific handling for reasoning_tokens > 0 with finish_reason: length to mark as SKIP The PR maintains backward compatibility by not changing existing behavior for other cases Non-compliant requirements: The PR does not update documentation or README files to reflect the new behavior Requires further human verification: Verification that the actual engine behavior matches the test cases in the PR Confirmation that the change works correctly with the actual NPU hardware and engine implementation

⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪

🧪 PR contains tests

🔒 No security concerns identified

⚡ Recommended focus areas for review Logic for distinguishing reasoning tokens from empty content The new logic correctly distinguishes between empty content due to reasoning tokens and empty content due to model output. However, it assumes that reasoning_tokens will always be an integer when present, which may not be true in all edge cases. The code should handle potential type mismatches or missing fields more robustly. rt = (((r.get("usage") or {}).get("completion_tokens_details")) or {}).get("reasoning_tokens") if not c: if (choice.get("finish_reason") or "").lower() == "length" and isinstance(rt, int) and rt > 0: Test case verification The self-test cases added in the PR are good for verifying the new logic, but they only test the specific scenarios mentioned in the ticket. It would be beneficial to add more comprehensive test cases that cover edge cases, such as when reasoning_tokens is not an integer, or when the field is missing entirely. if not _content_ok(_r("", "length", 8)): fails.append("empty content + finish_reason length + reasoning_tokens 8 must be a SKIP, not a FAIL") if _content_ok(_r("", "length", -1)): fails.append("reasoning_tokens -1 must not pass: the route could not count it, which is not zero") if _content_ok(_r("", "stop", 8)): fails.append("empty content with finish_reason stop must FAIL: the model said nothing, not the budget") if _content_ok(_r("//////", "stop")): fails.append("a degenerate content must still FAIL: that is the model, not the budget")