[bong-water-water-bong]

Refs #1786 (the Phi4-mini-Instruct-NPU2 row). Backlog. I searched issues and PRs for Phi "<|end|>" stop token served as text before writing this; nothing open covers it. Same class as #1044 and #2064, third source.

The fault, on the NPU

FastFlowLM's Phi4-mini-Instruct-NPU2 served by 1bit serve --device npu, the pack's own template, main before this change, "What is the capital of France? Answer with one word.", max_tokens 32:

1bit forward-serve: stops on 199999 <|endoftext|>
content='Paris<|end|><|assistant|>**Follow-up Question 1:**\n\n**What is the relevant and irrelevant factor in determining the capital city of a country?**\n\n**Elabor'   finish=length  completion=32

The model answers, closes its turn, and the serve keeps going: the turn end is served as text and the model writes itself a follow-up until the budget runs out.

Both of the pack's files name the same token, and it is not the one the model ends a turn with:

config.json            "eos_token_id": 199999              -> <|endoftext|>
tokenizer_config.json  "eos_token": "<|endoftext|>"
chat_template          … + message['content'] + '<|end|>' …      -> every turn ends with <|end|> (200020)

The upstream checkpoint lists <|end|> in generation_config.json, which the pack does not ship. #2064's two sources cannot find it.

The fix

The template is the one file every chat pack has, and it cannot disagree with itself about how a turn ends. At load, when the container's template is rendered, forward-serve renders a two-message exchange (a user message and an assistant message with a probe text) and takes what the template put right after the assistant's text. npu::turn_end_marker() returns the <...> token that text starts with, or nothing if it starts with anything else (plain text after a turn is not a stop). That marker goes into npu::stop_token_ids() in the slot the ChatML fallback's <|im_end|> already used. A template that cannot render the exchange contributes nothing and the load says so.

After, same pack, binary from this branch (f07b812), one npu-hold ticket

1bit forward-serve: stops on 199999 <|endoftext|>, 200020 <|end|> (tokenizer / chat turn end)
content='Paris'  finish=stop  completion=1        (three times, identical)

Other templates through the same rule, same binary: Qwen3-VL-4B-Instruct-NPU2 loads with stops on 151645 <|im_end|> (tokenizer / chat turn end) and answers Paris, finish=stop, three times. Llama-3.2 and the Qwen packs already listed their marker; for them the rule finds the same token and adds nothing (covered by the de-duplication arm of #2064).

Tests

tests/npu_stop_tokens_test.cpp, 10 new checks: the marker for Phi, ChatML, Llama 3 and Gemma shapes; leading whitespace skipped and what follows ignored; nothing or only whitespace after the turn; plain text after the turn, with or without a marker later in it; malformed brackets; Phi's two files alone give [199999] (the fault) and with the template's marker [199999, 200020]; a marker the tokenizer does not have adds nothing. all stop-token checks passed.

Mutant from a copy of the header (the "must start with <" guard removed): FAIL plain text after the turn is not a stop, even with a marker later in it.

tools/copyright.py --check rc 0. Docs: docs/npu.md, "What ends an answer".

Limits

• The rendering step itself (forward_serve.cpp) has no device-free harness; the two packs above are its test. Gemma's shape is unit-tested only.
• A template whose assistant turn ends with something that is not a single <...> token gets no marker from this rule, by design.
• Phi's decode on this route was 1.4–1.5 tok/s; that is the per-op route's speed and not part of this change.

🤖 Generated with Claude Code

[bong-water-water-bong]

ready for review at f07b812

[bong-water-water-bong]

VERDICT: READY at f07b812 for this instance — but it is the third source of one class, and the class is what the gate should now cover.

Refs #1786 (the `Phi4-mini-Instruct-NPU2` row). Backlog. I searched issues and PRs for `Phi "<|end|>" stop token served
as text` before writing this; nothing open covers it. Same class as #1044 and #2064, third source.

The three sources are one defect

• forward-serve: the tokenizer's eos_token and the ChatML turn end are stop ids too — <|im_end|> was served as text on FLM's Qwen3.5-2B (Refs #1785, #1786) #2064 — the stop set was assembled before the template was known, so a route rendering ChatML served <|im_end|> as text;
• npu: the Llama-3.2-1B pack imports whole and then drifts immediately — its first EOS is served as text #1044 — the same shape, an earlier source;
• this — Phi's <|end|> served as text on the NPU.

Each fix is per-model and per-marker, and each was found by reading one row's output. That is the tell: when the third instance arrives by a different path, the invariant — every stop token the rendered template can emit must be in the stop set — is not being enforced anywhere, so the fourth marker will arrive the same way.

What I would ask for, in this PR or a follow-up it names

1. derive the stop set from the template that was rendered, not from a per-model list: the template text contains its own turn markers, so template → markers → ids is computable at load. That is the generalisation forward-serve: the tokenizer's eos_token and the ChatML turn end are stop ids too — <|im_end|> was served as text on FLM's Qwen3.5-2B (Refs #1785, #1786) #2064 already argued for ("decided here, after the template"), applied one step further — from which route to which markers;
2. an arm over the shipped templates: for each template in the tree, assert that every marker the template emits is in the stop set. That is a table-driven test with no device, and it catches the fourth source before a row does;
3. the acceptance is user-visible, as with forward-serve: the tokenizer's eos_token and the ChatML turn end are stop ids too — <|im_end|> was served as text on FLM's Qwen3.5-2B (Refs #1785, #1786) #2064: a completion whose text contains no control marker and which ends where the turn should — because "the model answered" passes in all three of these defects.

On the search note

"I searched issues and PRs for … before writing this; nothing open covers it" — terms named, which is the check that keeps the fourth instance from opening a fourth issue. Worth keeping as the habit: it is what two of today's duplicate clusters lacked.

Nothing else: the fix is right for Phi, and naming the class is what turns three fixes into one gate.

[bong-water-water-bong]

VERDICT at f07b812: READY

Reviewed head f07b812 (refs #1786, the Phi4-mini row). This is the first verdict.

Checked

• git merge-tree onto main: no conflicts. CI build and CodeQL green. The red pr_agent is the summary bot, not a code check.
• npu_stop_tokens_test, built from this head: "all stop-token checks passed".
• Mutants of turn_end_marker() (app/npu_stop_tokens.h), each rebuilt and run:

mutant	result
leading whitespace not skipped	rc 1: "whitespace before it is skipped, what follows is ignored"
whitespace inside the brackets accepted	rc 1: "a lone bracket, an empty pair, an unclosed one and one with spaces in it are not markers"
an empty <> accepted (the close - i < 2 guard removed)	rc 1: the same arm

Together with the body's mutant (the must-start-with-< guard), every guard in the function is pinned. I restored the file (git status clean).

• app/forward_serve.cpp at this head compiles clean (0 errors) as a single-file compile with the NPU build's flags.
• The ChatML fallback cannot be overwritten. turn_end starts as <|im_end|> only on chatml_route, which is !can_render && e.chat_ok (:1177), and the template probe runs only under can_render (:1147). So a rendered template whose assistant turn ends in plain text yields no marker rather than removing the fallback's. A marker the tokenizer does not know adds nothing (token_id < 0, tested).

Not re-run by me: the device rows (Phi4-mini content='Paris' finish=stop completion=1 ×3; Qwen3-VL-4B stops on 151645 <|im_end|>). As the body says, the rendering step has no device-free harness.

Nit: the header comment's example line "\n", "", "</s>x"... -> "" reads as though </s>x yields nothing, but the code returns </s>, because the text starts with a whole <...> token. The behaviour is right; the example says the opposite of the "unless" clause next to it.