[bong-water-water-bong]

What

FastFlowLM's Qwen2.5-3B-Instruct-NPU2 loads on the per-op forward but answers noise. On the known-answer prompt it gives UNS �leon�leon�ucz …. Refs #1786; host-side only, the NPU is off.

Cause. The forward adds the q/k/v projection biases only when attention_bias is on, and it took that from config.json alone. The comment says a "tensor-level probe" decides it when the key is absent, but no such probe existed. Qwen2 configs never state the key, yet every Qwen2 layer has the three biases. Our own importers write attention_bias: true; FastFlowLM's pack does not. So on this pack the biases were loaded and never added.

Fix. The rule is now the one q/k normalisation already follows (engine#1788):

• If config.json does not state attention_bias and a layer carries a q, k or v bias, the biases are applied. The startup line says so: attention_bias=1 <- the container: layer 0 carries q/k/v projection biases.
• If config.json states attention_bias: false and the container carries biases, the pack is refused by name instead of guessed.
• A stated true is left as stated.

Evidence (host, 2026-11-12)

	known-answer prompt, the pack's own chat template
before (main)	UNS �leon�leon�ucz ZA … (68 tokens of noise)
after	`Paris<

Blast radius. I scanned every q4nx container on the box: 24 carry q/k/v biases. Only two lack the key:

• this pack;
• flm-whisper. Whisper runs on its own audio path (audio_ops.h), not this forward.

The other 22 already state attention_bias: true, so nothing changes for them.

Tests

• New batch_check_state_qwen2 (CI arm). It fabricates a two-layer Qwen2 with q/k/v biases and no attention_bias key, then compares the forward's sequential logits with a numpy Qwen2 forward on the container's own weights. This uses the same reference mechanism the lfm3 arm got in npu: the per-op forward reads LFM3, so LFM3-1.7B-NPU2 loads and answers (Refs #1786) #2254.
  • after: corr 1.0000000, max |d| 6.6e-06 of 9.6;
  • the binary without the fix fails it (corr 0.444, argmax 200 vs 7), while still passing batched == sequential.
• q4nx_qk_norm_test: four probe checks covering absent+biases → on, absent+none → off, stated false+biases → refusal, and stated true → unchanged.
• Full ctest: 268/268, device tests excluded. Every batch_check_state_ci.py arm (qwen3, lfm2, lfm3, qwen3_5, qwen2) was run by hand: 0 FAIL. tools/copyright.py --check rc 0.

Not in this PR: a kernel set or device row for this pack. Both wait for the NPU.

🤖 Generated with Claude Code

[bong-water-water-bong]

ready for review at 32dcc97

[github-actions]

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶 1786 - Partially compliant Compliant requirements: it imports with zero gaps and its self-check passes against a named reference it serves on --device npu it answers a known-answer prompt with the model's own chat template, 3 identical requests returning identical text Non-compliant requirements: decode tok/s and time-to-first-token measured under npu-hold the row names the kernel set and the engine commit that produced it Requires further human verification: decode tok/s and time-to-first-token measured under npu-hold the row names the kernel set and the engine commit that produced it

⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪

🧪 PR contains tests

🔒 No security concerns identified

⚡ Recommended focus areas for review Attention bias handling The fix correctly addresses the issue where Qwen2's attention biases were not being applied due to the absence of the attention_bias key in config.json. The code now checks for the presence of q/k/v projection biases in the container and applies them when the key is absent, which fixes the noise output for Qwen2.5-3B-Instruct-NPU2. This is a correctness fix for tensor handling and bias application in the forward pass. // engine#1786: attention_bias the same way. Qwen2's config.json never states it and every layer carries q/k/v // biases; with the key absent they were loaded and never added, so Qwen2.5-3B-Instruct-NPU2 answered noise. if (mw_ && mw_->config.num_layers > 0) { std::vector<unsigned char> has_b(size_t(mw_->config.num_layers), 0); auto present = [](const TensorDesc& t) { return t.name[0] && t.data_size > 0; }; for (int l = 0; l < mw_->config.num_layers; l++) has_b[size_t(l)] = present(mw_->layers[l].q_proj_bias) || present(mw_->layers[l].k_proj_bias) || present(mw_->layers[l].v_proj_bias); const int by = q4nx_semantics_attention_bias_from_tensors(&sem_, has_b.data(), mw_->config.num_layers); if (by == -2) { err_ = "config.json says attention_bias is false and model.q4nx carries q/k/v projection biases; " "serving it either way would be a guess"; return false; } if (by >= 0) fprintf(stderr, " [semantics] attention_bias=1 <- the container: layer %d carries q/k/v projection " "biases (config.json does not state attention_bias)\n", by); } Attention bias semantics The new function q4nx_semantics_attention_bias_from_tensors correctly implements the logic for determining attention bias from tensor presence. It ensures that when attention_bias is not specified in the config but biases are present in the container, the bias is enabled. It also correctly refuses models where the config explicitly states attention_bias: false but biases are present, preventing incorrect behavior. int q4nx_semantics_attention_bias_from_tensors(Q4nxSemantics* s, const unsigned char* has_bias, int layers) { if (!s || !has_bias || layers <= 0 || s->attention_bias) return -1; for (int l = 0; l < layers; l++) if (has_bias[l]) { if (s->attention_bias_stated) return -2; s->attention_bias = 1; return l; } return -1; }

[bong-water-water-bong]

VERDICT at 32dcc97: READY

Reviewed at 32dcc9782281c3acb8a8fae4f0fd6d31993f67dd. It is on main (0 behind) and merges clean. The diff is 7 files, +150/-7.

Checked

• The rule is narrow. q4nx_semantics_attention_bias_from_tensors acts only when attention_bias is off and a layer carries a q, k or v bias.
  • A key that is absent turns the biases on.
  • A stated key returns -2, and init refuses by name.
  • get_num parses JSON true and false, so a stated boolean false counts as stated and reaches the refusal.
• The forward is safe on partial biases. q_bias_, k_bias_ and v_bias_ are always allocated zero-filled (q4nx_forward.cpp:1832), so a layer with only some of the three adds zeros for the missing ones. Nothing reads out of bounds.
• Tests. q4nx_qk_norm_test reports ok. Every batch_check_state_ci.py arm passes with NPU_INFER_HOST_GEMM=1: qwen2, lfm3, lfm2, qwen3_5 and the default. On qwen2: corr 1.0000000 (needs 0.99999), max |d| 6.62e-06 of 9.6, argmax 7 vs 7. tools/copyright.py --check returns rc 0.
• Blast radius, re-scanned. I read every model.q4nx header on the box and found 25 containers carrying q/k/v biases. 3 lack the key: the Qwen2.5-3B pack and two copies of Whisper (~/.config/flm/models/Whisper-V3-Turbo-NPU2 and scratch-packs/flm-whisper). The body counts 24 and 2 because it missed the second Whisper copy. Both copies run on the audio path, so the conclusion holds.

My mutants (each restored, and the tests pass again)

mutant	result
the probe in init replaced by by = -1 (main's behaviour)	qwen2 arm FAIL: corr 0.4443442, argmax 200 vs 7, which is the body's 0.444
the attention_bias_stated refusal dropped	q4nx_qk_norm FAIL: "config.json says attention_bias=false and the container carries biases: a disagreement (-2, refused)"

The known-answer row on the real pack (noise before, Paris<|im_end|> after) is the author's; I did not re-run it. The kernel set and the device row for this pack are owed once the NPU is back on.

CI build at this head: completed success, "100% tests passed, 0 tests failed out of 268". batch_check_state_qwen2 is test #251 there and passed.