[bong-water-water-bong]

Refs #1786.

Stacked on #2242. This branch is #2242's head de3a5d4 plus one commit, 5b69816. Until #2242 merges, the diff against main includes #2242's commits. The commit this PR adds is git diff de3a5d4 5b69816: 5 files, +143/−11. Gemma3-270M also needs #2242's gemma3 rules to answer, which is why the branch is stacked rather than separate. I will rebase onto main once #2242 merges.

What it fixes

Gemma3-270M-NPU2 fails at load on main and on #2242:

step failed: MLP dequant model.layers.0.self_attn.q_proj.weight: data_size 409600 at geometry 1024x768 = 96 tiles
(chunk 4266 B is not a kind, and no padded grid fits): the container and config.json disagree

Its hidden size, 640, is off the 5,120-byte tile's 256-column grid. FastFlowLM stores every I8 tensor of this pack (127 of 127) as 1,280-byte quarter tiles:

16 rows x 128 columns, 4 groups of 32
scales      [64] bf16 @[0,128)     index g*16 + r
zero-points [64] bf16 @[128,256)   index g*16 + r
codes            @[256,1280)       byte c*8 + r/2, even row = low nibble
grid (rows/16) x (K/128), row-major

That is the 5,120-byte tile's reading with one half and four groups. Both kinds cost 0.625 bytes a weight, so a K % 256 == 0 tensor (this pack's down_proj, K = 2048) has the same byte count either way. What names the kind is the declared shape: [chunks, 1280] here, [tiles, 5120] in every other pack (Medgemma-4B: 239 of 239). dequant_any() takes the declared row width and reads quarter tiles into the forward's padded K, writing the pad columns as 0.

How the reading was settled (host only)

The first four guesses were all 32 x 64 chunks, and all of them were wrong (correlation about 0.07). Then I searched chunk shape × grid order × scale order × code order against an independent GGUF of the same model: blk.0.attn_v.weight of unsloth's gemma-3-270m-it-UD-IQ2_M.gguf is Q5_0, which gguf-py dequantizes.

v_proj vs GGUF attn_v (Q5_0), correlation
0.9962  16x128, grid row-major, scales g*16+r, codes c*8+r/2     <- this reading
0.8390  16x128, scales r*4+g
0.0939  32x64 (the closest of the first guesses)

A second, unrelated check is the pack's own tied head. lm_head dequantised this way is the bf16 embed_tokens divided by √640 (the pack stores the embedding pre-scaled; ratio 0.03955 against 1/√640 = 0.03953). Over the first 4096 rows: correlation 0.9963, |dequant − k·embedding| / |scale| mean 0.235, max 0.567.

Before and after, on the real pack (host GEMM, scratch probe over init_host/step)

#2242's head:  init/step fails (above)
this head:     prefill 21 tokens 37.9s   out: 50429 107 106   -> 'Paris\n<end_of_turn>'
               "Explain in three sentences why the sky is blue." (40 tokens) ->
               'The sky is blue because of a phenomenon called Rayleigh scattering. This is because blue light has
                shorter wavelengths than other colors, and these shorter wavelengths are scattered by air molecules in
                the atmosphere. This scattering causes'

This is a correctness path, not a serving row: the NPU is off, and no kernel set for this pack's shapes was built or run.

Checks

• npu_dequant_bf16_test gets six new q4h checks: every element of a 2 × 2 grid against the formula, the pad columns written 0, and refusals for one byte short, a buffer narrower than K, and rows off the 16-row grid.
• ctest -j 6 --timeout 200 -E "npu_rt_shim|npu_dmabuf_import" at 5b69816: 262 of 262.
• tools/copyright.py --check rc 0.

🤖 Generated with Claude Code

[bong-water-water-bong]

ready for review at 5b69816 (stacked on #2242; the commit under review is 5b69816 alone)

[github-actions]

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶 1786 - Partially compliant Compliant requirements: Every FastFlowLM/*-NPU2 model source imports and serves on --device npu It imports with zero gaps and its self-check passes against a named reference It serves on --device npu It answers a known-answer prompt with the model's own chat template, 3 identical requests returning identical text decode tok/s and time-to-first-token measured under npu-hold Non-compliant requirements: The row names the kernel set and the engine commit that produced it Requires further human verification: The row names the kernel set and the engine commit that produced it

⏱️ Estimated effort to review: 4 🔵🔵🔵🔵⚪

🧪 PR contains tests

🔒 No security concerns identified

⚡ Recommended focus areas for review Incorrect handling of quarter-tile tensors The PR introduces support for quarter-tile tensors (Q4NX_CHUNK_Q4H) but the logic for detecting and handling these tensors might be flawed. Specifically, the declared_row_bytes function and its usage in dequant_any could lead to incorrect tensor kind detection if the declared shape does not accurately reflect the actual tensor layout. This could cause incorrect dequantization of tensors, leading to incorrect model outputs. static int64_t declared_row_bytes(const TensorDesc* d) { return (d && d->ndim == 2) ? d->shape[1] : 0; } static int dequant_any(const void* data, size_t nbytes, int rows, int cols, float* out, int64_t decl_row_bytes = 0) { if (!data || !out || rows <= 0 || cols <= 0) return -1; if (decl_row_bytes == Q4NX_CHUNK_Q4H) { // Quarter tiles on the container's own K (Gemma3-270M: 640), read into the forward's padded width. if (rows % Q4NX_Q4H_ROWS) return -1; const size_t chunks = nbytes / Q4NX_CHUNK_Q4H, row_tiles = (size_t)(rows / Q4NX_Q4H_ROWS); if (chunks % row_tiles) return -1; return q4nx_dequant_tensor_q4h((const uint8_t*)data, nbytes, rows, (int)(chunks / row_tiles) * Q4NX_Q4H_COLS, cols, out); } Potential BOS token handling inconsistency The PR adds logic to read BOS token information from tokenizer_config.json when config.json does not provide it. However, the logic might not be robust enough to handle all edge cases, such as when tokenizer_config.json is missing or malformed, potentially leading to incorrect BOS token handling. // have (linear_attention / SSM) and interleaved/multimodal RoPE are // refused. Sliding-window layers and per-layer-type RoPE are executed: // a sliding window is a mask and per-layer-type RoPE is a per-layer // selection, both host-side, so Gemma4 needs no new kernel. const int n_linear = q4nx_semantics_linear_layers(&sem_); // GDN (qwen3_5 linear_attention) IS implemented: the fused q|k|v and the // z projection are derived from the config's linear_* dims and run through // the same full-ELF GEMM path; conv/alpha/beta/recurrence are host-side. // Text-only mRoPE reduces to the ordinary partial-RoPE angles (all three // position components are equal), so it is executed too. Refuse only when // the linear geometry is underivable from the config. lin_kd_ = lin_vd_ = lin_nkh_ = lin_nvh_ = lin_khd_ = lin_vhd_ = 0; if (n_linear > 0) { lin_nkh_ = q4nx_config_int(cfg_path.c_str(), "linear_num_key_heads", 0); lin_nvh_ = q4nx_config_int(cfg_path.c_str(), "linear_num_value_heads", 0); lin_khd_ = q4nx_config_int(cfg_path.c_str(), "linear_key_head_dim", 0); Gemma3-specific logic might be incomplete The PR adds support for Gemma3-specific configurations, including sliding window patterns and local RoPE bases. However, the logic for determining layer types and handling sliding layers might not cover all possible configurations or edge cases, potentially leading to incorrect model behavior for some Gemma3 variants. /* Gemma3 (gemma3, gemma3_text, FastFlowLM's gemma3_text_only): the sandwich norms, a local RoPE base for the sliding * layers and the 5:1 sliding pattern. gemma3n is another architecture (AltUp, laurel) and is not this. */ const int is_gemma3 = (starts_with(model_type, "gemma3") && !starts_with(model_type, "gemma3n")) || (starts_with(arch, "Gemma3") && !starts_with(arch, "Gemma3n")); const int is_qwen3 = starts_with(model_type, "qwen3") || starts_with(arch, "Qwen3"); const int is_phi = starts_with(model_type, "phi") || starts_with(arch, "Phi"); const int is_qwen35 = starts_with(model_type, "qwen3_5") || starts_with(arch, "Qwen3_5"); /* RMSNorm convention: gemma2 scales by (1 + w); gemma4 by w. Gemma3 is (1 + w) in HF, but the containers this * forward reads (FastFlowLM's gemma3 NPU2 packs) store the norm weights with the 1 already added, as llama.cpp's * GGUF does: on Medgemma-4B-NPU2 every norm kind (input, post-attention, pre/post feedforward, q, final) equals * the GGUF's to bf16 rounding, and adding 1 again answered the known-answer prompt "*,,,,,," where the plain * form answers "Paris" (engine#1786). */ s->rmsnorm_offset = (is_gemma && !is_gemma4 && !is_gemma3) ? 1 : 0;

[bong-water-water-bong]

VERDICT at 5b69816: READY

The quarter tile is the right unit to generalise, and it is what makes the smaller Gemma3 load: a 1,280-byte tile is a quarter of the 5,120-byte one, so a model whose hidden size divides into quarters rather than wholes now has a decoded path instead of a refusal. q4nx_dequant.{h,c} with npu_dequant_bf16_test.c keeps the arithmetic in one place with its own arm, which is what the other tile widths already do.

This supersedes #2242 — I verified rather than assumed: both branches carry an identical 41-line change to q4nx_semantics.c (the gemma3 text decoders), and #2248 adds the dequant files on top. #2242's other files (tools/ci/dupdef_check.py, tools/hrx_f16_narrowing_check.py) are not its own work — they are #2246 and #2231 arriving through a main merge (merged 03:15Z and 22:53Z), which is why a three-dot diff shows them. There are no conflict markers between the two branches, but sharing 11 of 13 files means merging both would double-apply the semantics change.

So: ready #2248, and #2242 should close unless it carries something the diff does not show.

[bong-water-water-bong]

VERDICT: READY at 5b69816 — the record of four wrong guesses is worth as much as the fifth.

The stacking is stated in the form a reviewer can use

This branch is #2242's head `de3a5d4` plus one commit, `5b69816`.
Until #2242 merges, the diff against `main` includes #2242's commits.
The commit this PR adds is `git diff de3a5d4 5b69816`: 5 files, +143/−11.

Base named, the contaminating range named, and the exact command that isolates this PR's own change given. That is the cleanest form of a stacked declaration I have seen in this queue, and it removes the diffstat trap (#2035, and my own near-miss on #2228) before it can happen.

Four wrong guesses, published

"The first four guesses were all 32 x 64 chunks, and all of them were wrong (correlation about 0.07). Then I search[ed]…"

0.07 is the signature of a wrong reading — near-zero correlation against the GGUF tensor, which is the same instrument #2063 used to prove its q/k/v order. Reporting the four failures is what makes the search that followed credible, and what stops the next person from starting at 32×64.

Two independent checks, which is the right number

1. v_proj vs GGUF attn_v (Q5_0) — correlation against the pack's own tensor, so the oracle is external to the reader;
2. the pack's own tied head — lm_head dequantised this way is the bf16 embed_tokens: a structural equality that no amount of tuning can fake.

A single check on a new reader can pass for the wrong reason; two that fail independently cannot.

Two asks

1. say whether the search was exhaustive or bounded — "then I searched" is the right shape; naming the space (which readings were tried: 5,120 / 1,280 / both × half/four groups) tells a reader whether the answer is the only one that fits or the first one that fit;
2. keep the 0.625 bytes/weight observation — "both kinds cost 0.625 bytes a weight" is the arithmetic that makes the two candidate readings distinguishable by size rather than only by correlation, and it is the cheapest check in the PR.

[bong-water-water-bong]

VERDICT: READY at 5b69816 — fresh configure rc 0, npu_dequant_bf16_test runs 102 arms, all cases ok, including the six new quarter-tile ones.

The distinction this adds is the one that cannot be inferred from size, and the comment says so explicitly: a quarter tile is 16 rows × 128 columns in 1,280 bytes, the 5,120-byte tile's reading with one half and four groups — "both cost 0.625 bytes a weight, so the byte count cannot tell them apart and the declared row width is what names the kind". That is the same lesson as #2203's 4,608-byte tile and #1999's blocked_fits: a length-based decoder that grows a branch must have a declared discriminator, and here the container's shape row ([chunks, 1280] vs [tiles, 5120]) is it.

The arms cover the four ways a new tile goes wrong, and two are the ones I would have asked for:

q4h: every element is code (at c*8 + r/2) * scale + zero (at g*16 + r), grid row-major   <- the index arithmetic
q4h: the columns past the container's K are written 0                                    <- padding is zeroed, not left stale
q4h: one byte short is refused                                                           <- length check
q4h: a buffer narrower than K is refused                                                 <- geometry check

the columns past the container's K are written 0 matters because K = 640 is not on the 256-column grid — that is why this tile exists — so a padded column left stale would feed garbage into a real MAC. And the pair one byte short / narrower than K are the two refusals that keep a mis-declared tensor from being read as a valid one.

One ask: an arm that the same buffer read as a 5,120-byte tile fails or differs, i.e. that the kind is decided by the declared row width and not by whichever branch is tried first. The comment asserts the byte count cannot distinguish them; an arm showing the wrong-kind read is rejected (or produces different values) is what keeps a future if reordering from silently switching kinds — the exact failure #2203's bf16 did not hijack it arm guards one tile over.

Scope of my check: fresh configure rc 0, built and ran npu_dequant_bf16_test (102 arms, all pass). No pack: Gemma3-270M is not on this box, so the served-answer claim is the PR's own; the tile's arithmetic and its refusals are what the arms prove.