[bong-water-water-bong]

What

FastFlowLM's Gemma4-12B-IT-NPU2 is a dense Gemma4, with no per-layer embedding. The per-op forward failed on it at init, with an empty error message. Three changes take its text tower to a correct answer on the host. Refs #1786; host-side only, the NPU is off.

1. No per-layer embedding. load_norms read post_layernorm, the PLE block's norm, for every Gemma4, and this model has none. It is now read only when ple_dim_ > 0. The bf16 lookups also name the missing tensor instead of returning false silently.

2. The wide chunks. Every I8 tensor is declared [rows/16, K/256, bytes]:

   • bytes is 2304 (Q4NX_CHUNK_Q4W) or 4352 (Q4NX_CHUNK_Q8W, the tied head);
   • a chunk is 16 rows × 256 columns;
   • it starts with 128 bf16 scales at g*16 + r (one per row and 32-column group, symmetric, no zero-point);
   • then 4-bit two's-complement codes at c*8 + r/2 (even row in the low nibble), or int8 codes at c*16 + r.

   A Q4W tensor has exactly the byte count of a 4608-byte Q4S grid, and that is how it was read. Layer 0 then went non-finite at its first attention. declared_row_bytes now reads a 3-D tensor's last dimension, and dequant_any routes by it.

3. attention_k_eq_v. The global layers have no v_proj: their values are the k projection, taken before k_norm and then given Gemma4's scale-free v norm. This is our llama.cpp fork's rule (src/models/gemma4.cpp: Vcur = wv ? wv x : Kcur). The forward fills those v rows from k. A missing v_proj without the key is refused by name.

How the layout was settled

The reference is independent weights, not a guess: Google's own gemma-4-12B-it-qat-q4_0-gguf.

tensor (GGUF name)	corr with our Q4W reading	rel-rms
blk.0.attn_q / attn_k / attn_output / ffn_down	0.99856 / 0.99856 / 0.99867 / 0.99861	0.052–0.055
blk.5.attn_q / attn_k (global layer)	0.99878 / 0.99877	0.050
blk.47.ffn_gate	0.99875	0.050

• The 5% rel-rms is FastFlowLM's own re-quantization against Google's. It is the same in every tensor, and about half the chunk scales match Google's bit-for-bit after bf16 rounding.
• Alternative orders score far lower: row-major codes −0.04, GGUF block order −0.06, offset-8 codes −0.47, a row-major scale slot 0.08.
• The Q8W head reproduces the pack's own bf16 embedding at corr 1.00000; the next-best layout scores 0.77.

Evidence (host, 2026-11-12)

prompt (the pack's own template, thinking off)	before (main)	after
capital of France	init failed: (empty)	`<
why is the sky blue	—	`<

The model writes the template's own empty-thought prefix, then answers. Gemma4-E2B, the PLE model, still answers Paris<turn|> on the host.

Tests

• npu_dequant_bf16_test → wide_chunks:
  • both kinds over a 2 × 2 grid with signed scales: every element is checked, and pad columns are zeroed;
  • refusals: one byte short, rows off the 16-row grid, a non-wide chunk size;
  • the Q4S byte-count collision itself.
• q4nx_gemma3_semantics_test: Gemma4-12B's keys (attention_k_eq_v, no PLE, global head 512 × 1 kv head), and a Gemma4 without the key.
• Full ctest: 268/268, device tests excluded. Every batch_check_state_ci.py arm (qwen3, lfm2, lfm3, qwen3_5, qwen2, qwen2vl) passes. tools/copyright.py --check rc 0. Re-run at this head after the rebase onto npu: Qwen2.5-VL's contiguous "mrope" is a position layout, so its text tower loads and answers (Refs #1786) #2261. The Paris answer at this head gives identical tokens.

Not in this PR: the vision and audio towers, a kernel set, and a device row.

🤖 Generated with Claude Code

[bong-water-water-bong]

ready for review at 16b9d8a

[github-actions]

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶 1786 - Partially compliant Compliant requirements: it imports with zero gaps and its self-check passes against a named reference it answers a known-answer prompt with the model's own chat template, 3 identical requests returning identical text Non-compliant requirements: it serves on --device npu decode tok/s and time-to-first-token measured under npu-hold the row names the kernel set and the engine commit that produced it Requires further human verification: it serves on --device npu decode tok/s and time-to-first-token measured under npu-hold the row names the kernel set and the engine commit that produced it

⏱️ Estimated effort to review: 4 🔵🔵🔵🔵⚪

🧪 PR contains tests

🔒 No security concerns identified

⚡ Recommended focus areas for review Incorrect tensor shape handling for wide chunks The code changes introduce handling for wide chunks with shape [rows/16, K/256, bytes], but the logic for determining the chunk size based on the tensor's last dimension may not correctly handle all cases, especially when the tensor dimensions are not strictly multiples of 16 and 256. This could lead to incorrect dequantization if the tensor shape doesn't align properly with the expected chunk structure. if (d && d->ndim == 3 && (d->shape[2] == Q4NX_CHUNK_Q4W || d->shape[2] == Q4NX_CHUNK_Q8W)) return d->shape[2]; return (d && d->ndim == 2) ? d->shape[1] : 0; Potential silent failure in tensor loading The error handling in load_norms() was improved to provide more informative error messages, but there's still a potential for silent failures if the tensor loading logic is not robust enough to catch all edge cases, especially when dealing with missing or malformed tensors in the model. if (!td || n < 0) { err_ = std::string("model.q4nx has no ") + nm; return false; } const uint16_t* p = bf16_data(mw_, td, (size_t)n); if (!p) { err_ = std::string(nm) + " is not bf16 of " + std::to_string(n) + " values"; return false; } Incorrect handling of attention_k_eq_v The code introduces handling for attention_k_eq_v, but the logic for checking if v_proj is absent and then copying from k_proj may not be robust enough. If the model configuration is not correctly parsed or if there are inconsistencies in the tensor names, this could lead to incorrect behavior or runtime errors. const bool v_absent = lw->v_proj_weight.name[0] == '\0' || lw->v_proj_weight.data_size == 0; if (v_absent && sem_.attention_k_eq_v) { // attention_k_eq_v: v is the k projection (normalised scale-free in attention(), as Gemma4's v is) std::copy(wk_.begin(), wk_.begin() + (ptrdiff_t)((size_t)nkv * hd * kpad_), wv_.begin()); } else if (v_absent) { err_ = "model.layers." + std::to_string(l) + " has no self_attn.v_proj.weight and config.json does not " "declare attention_k_eq_v"; return false; } else if (!deq(&lw->v_proj_weight, wv_.data(), nkv * hd, kpad_)) { return false; }

[bong-water-water-bong]

VERDICT at 16b9d8a: READY

Reviewed at 16b9d8ae8b4a93f0870eae229bea5304ce50280b. It is 3 behind main and merges clean. The diff is 8 files, +170/-4.

Checked

• The wide-chunk reader matches the layout it documents.
  • Scales are at (c/32)*16 + r.
  • Q4W codes are at c*8 + r/2, even row in the low nibble, two's complement: 256 columns × 8 bytes = 2048 B.
  • Q8W codes are at c*16 + r: 4096 B.
  • q4nx_dequant_tensor_wide refuses a wrong byte count, rows off the 16-row grid, and any other chunk size, and it zeroes the pad columns.
• Routing. declared_row_bytes returns the last dimension only for a 3-D tensor whose chunk is 2304 or 4352. Every 2-D tensor reads as before.
• attention_k_eq_v. The fill copies k's weight rows (nkv * hd * kpad_) into wv_, so v is Wk·x before k_norm. Attention then applies Gemma4's scale-free v norm, which is the fork's Vcur = wv ? wv x : Kcur. A missing v_proj without the key is refused by name.
• PLE guard. post_layernorm is read only when ple_dim_ > 0. The rd() lambda now names the missing tensor instead of leaving init failed: empty.
• Tests. I built and ran them in an ONEBIT_NPU host tree:
  • npu_dequant_bf16_test: all cases ok, including the new wide arms;
  • q4nx_gemma3_semantics_test: ok;
  • every batch_check_state_ci.py arm (qwen2, qwen2vl, lfm3, lfm2, qwen3_5, default): rc 0;
  • tools/copyright.py --check: rc 0.

My mutants on the wide reader (each restored, and the test passes again)

mutant	npu_dequant_bf16_test
nibble order swapped (odd row in the low nibble)	FAIL, 2 checks
scale slot row-major (r*8 + c/32)	FAIL, 3 checks
offset-8 codes (v - 8)	FAIL, 2 checks

These cover the alternative orders the body ruled out against Google's QAT GGUF. I did not re-run that correlation: neither the Gemma4-12B pack's weights nor the GGUF are on this box (only FLM's xclbin dirs). The 0.9986 figures and the Paris and sky answers are the author's.

Notes (not blocking)

• The forward-level rules have no CI test yet. If the wv_ copy is dropped, v is zero on every global layer. If the PLE guard is reverted, this model fails at init again. Only the real-pack run would show either. A fabricated dense-Gemma4 arm in batch_check_state_ci.py (global layers without v_proj, attention_k_eq_v: true, no PLE), checked against a numpy reference as the lfm3, qwen2 and qwen2vl arms are, would pin both.
• The key is detected with strstr on two spellings. "attention_k_eq_v" : true (a space before the colon) or a pretty-printed nested text_config with other whitespace would read as off, and the load would then refuse by name rather than misread, which is safe. get_num already handles true. If the key can sit under text_config, a nested lookup would be more robust than either.
• Owed on the device, NPU off: a kernel set for the 16×256 wide shapes, and the device row.

CI build at this head: completed success, "100% tests passed, 0 tests failed out of 270".

[bong-water-water-bong]

VERDICT: READY -- the byte-count collision is handled the right way round, and it is asserted rather than described

Three changes, and the second is the one that would have been easy to get dangerously wrong.

The collision is real and the discriminator is the declaration, not the size. A Q4W tensor's byte count equals a 4608-byte Q4S grid's, so a size test cannot tell them apart -- the old code read the 12B's tensors as Q4S and layer 0 went non-finite at its first attention. The fix reads the declared shape:

if (d && d->ndim == 3 && (d->shape[2] == Q4NX_CHUNK_Q4W || d->shape[2] == Q4NX_CHUNK_Q8W)) return d->shape[2];

and the geometry is then validated rather than trusted: if (nbytes != ntr * ntc * chunk_bytes) return -1;. A mis-declared pack is refused, not misread.

What makes this reviewable without the pack is the test, and it goes further than I would have asked:

• the synthetic arms check the mapping itself -- (code, two's complement, c*8 + r/2, even row low) * scale (g*16 + r), row-major grid -- against google's QAT Q4_0 GGUF, so the byte layout is anchored to something outside this tree;
• wide: another chunk size is not accepted (4608 -> -1) is the collision guard, pinned;
• wide: one byte short is refused and wide: rows off the 16-row grid are refused are the shape validations, pinned;
• wide: the columns past the container's K are written 0 pins the padding rule;
• and the last line of the test states the collision as an assertion -- (4096/16)*(3840/256)*Q4W == (4096/32)*(3840/256)*4608 -- so the reason the discriminant must be the declared shape is in the tree, not just in this description.

The other two changes are the right kind of small. post_layernorm is read only when ple_dim_ > 0, so a dense Gemma4 with no PLE block stops looking for a tensor it does not have; and the bf16 lookups now name the missing tensor instead of returning false into an empty error message -- a load failure with no message is a defect of its own, independent of this model, and fixing it is what made the 12B diagnosable at all.

One observation, not an ask: the discriminant itself (ndim == 3 && shape[2] in {2304, 4352}) has no direct arm, because exercising it needs a real ggml_tensor declaration rather than the function's own arguments. The function-level refusal arms cover the same ground from below, so this is a note for whoever next touches the reader: the 3-D-with-4608 case (a legitimately declared Q4S grid) is the one input that would prove the branch is not over-eager, and it costs one tensor in an existing test.