npu: bonsai2-27b-NPU serves, loads its decode family, and returns empty completions on every prompt #1998

Activity

1. bong-water-water-bong commented on Nov 10, 2026

   

   bong-water-water-bong

   on Nov 10, 2026

   CollaboratorAuthor

   More actions

   Claiming this, and offering a hypothesis that would make it a non-bug rather than a defect — because I have produced this exact symptom before and it was not a failure.

   The precedent

   On #1346 one of my arms returned an empty content string with finish_reason: length at max_tokens 24, and I initially read it as a broken serve. It was a thinking model: the
   tokens were in reasoning_content, and the 24-token cap ended the response while it was still reasoning. Four arms and a fleet sweep later, that arm is recorded as clean. The lesson I
   wrote down then was: "empty content + reasoning_content + finish_reason: length is a thinking model, not a failure."

   This issue's reproduction is greedy at max_tokens 24 on a 27B — the two conditions that make that pattern most likely. So before it becomes a row of its own, the check is cheap
   and it distinguishes the two cases completely:

   1. usage.completion_tokens — if it is > 0 while content is empty, the model generated; the harness is reading the wrong field;
   2. finish_reason — length with completion_tokens == 24 means it was still going, i.e. truncation, not silence; stop with 0 tokens is a real failure;
   3. reasoning_content (or message.reasoning) in the same JSON — a thinking model's output lives there;
   4. and the decisive control: max_tokens 256 on the same prompt. A thinking 27B will answer, eventually, with the same content the 24-token run truncated. If it produces nothing
      at 256 too, the diagnosis changes entirely.

   Why the 0.5B control does not settle it

   "the same harness against q05-cal (0.5B) returns Paris and 2 in 23 s, so the harness and the HTTP path are fine" — that is a good control for the route, and it does rule out the
   transport. It does not rule out the model's own behaviour, because a 0.5B and a 27B do not reason for the same number of tokens before emitting an answer. That is exactly the asymmetry
   #1346 hit: a small model answered immediately at the same cap, the large one did not.

   What I would run

   The issue already has everything but the output fields: bash ~/wt/a2b-window11.sh <binary> /tmp/out.txt then look at /tmp/out.txt for finish_reason, usage and reasoning_content,
   plus one replay at max_tokens 256. That needs an npu-hold window (the sealed kernel set is at ~/models/npu/bonsai2-27b-NPU/kernels), which is why I am writing this rather than running
   it — this box is at 155 G and the queue has other lanes' tickets.

   If it turns out completion_tokens == 24 and finish_reason == length, this should close as "not a defect: a thinking model under a 24-token cap", with the harness taught to print the
   field it is reading — the same correction I had to make for my own #1346 arms.
2. bong-water-water-bong commented on Nov 10, 2026

   

   bong-water-water-bong

   on Nov 10, 2026

   CollaboratorAuthor

   More actions

   Reproduced, and the completion is not empty: the tokens are in reasoning_content (lane cl-npurt).

   Same pack, same pristine-main binary the report used (~/scratch-packs/a2b-main-arm/build-npu/1bit), same serve line and request shape as a2b-window11.sh, one npu-hold ticket (197 s). I printed the raw response instead of choices[0].message.content:

   == POST /v1/chat/completions  {"messages":[{"role":"user","content":"The capital of France is"}],"max_tokens":8,"temperature":0,"top_k":1}
   {"choices":[{"finish_reason":"length","index":0,"message":{"content":"","reasoning_content":"\nThe user is asking a simple","role":"assistant"}}],
    "timings":{"predicted_ms":68003.1,"prompt_ms":17848.4},"usage":{"completion_tokens":8,"prompt_tokens":13,...}}
   
   == POST /v1/completions  {"prompt":"The capital of France is","max_tokens":4,"temperature":0}
   {"choices":[{"finish_reason":"length","index":0,"text":" Paris.\nThe"}], ... "usage":{"completion_tokens":4,"prompt_tokens":5,...}}
   

   So:

   • The forward is producing sensible tokens. On the chat route this model (a thinking model) opens with its reasoning, the serve puts that in reasoning_content, and finish_reason is length: all of the token budget went to reasoning before any answer text. The harness reads only message.content, which is "". With max_tokens 24 the same thing happens: 24 reasoning tokens, no content yet.
   • The raw-completion route answers Paris on the same serve. The same pack gave Paris.\nThe capital of in eight runs across two builds in the npu: the loader materialises the token embedding table in RAM (0.74 GiB I8 -> 2.96 GiB resident); map it read-only and read one row per token #1784 decode row two hours earlier (main 8f10bfb and d805b90^).
   • It is not an EOS at the template boundary (the suggested first check): finish_reason is length, with 8 of 8 tokens generated.
   • The 0.5B control answers in content because it does not open with a reasoning block.

   For the A2b identity row, either compare reasoning_content + content together, or use /v1/completions, or raise max_tokens (at roughly 7–8 s per token on this pack that is expensive). The two arms being byte-identical in what they returned is still a valid identity result; it was just comparing empty strings.

   One thing I noticed in the serve log and did not chase: that binary prints this build has no chat-template renderer (ONEBIT_JINJA=OFF), so the container's template is NOT used and the chat route falls back to ChatML. That is a property of how that arm was configured, not of the pack.

   Closing as not a fault in the engine or the container. Reopen if a run with enough budget to finish the reasoning still returns no content.
3. bong-water-water-bong

   closed this as not plannedon Nov 10, 2026
4. bong-water-water-bong

   added a commit that references this issue on Nov 11, 2026

   npu serve: usage says how many completion tokens were reasoning (Refs #…

   ...

   539a77f