[bong-water-water-bong]

Fixes #1889. Refs #1786.

This body describes head 676a8f6 (rewritten after the review at adb81c4, which rightly said the old body described an older head).

What is in it (18 files)

1. npu: Hunyuan-7B answers on the forward route: use_qk_norm is read, tiles are q,k,v, RoPE base and norm order follow the model (Fixes #1889) #1976, brought onto main. npu: Hunyuan-7B answers on the forward route: use_qk_norm is read, tiles are q,k,v, RoPE base and norm order follow the model (Fixes #1889) #1976 is READY at a43aea3 and conflicts with main; its merge-up Merge main into #1976: keep both new test functions in npu_pack_import_test.cpp (Refs #1889) #2130 is READY and has npu: Hunyuan-7B answers on the forward route: use_qk_norm is read, tiles are q,k,v, RoPE base and norm order follow the model (Fixes #1889) #1976's branch as its base, which the queue does not merge. This branch is origin/main + Merge main into #1976: keep both new test functions in npu_pack_import_test.cpp (Refs #1889) #2130's head 34cac62 + one conflict (q4nx_semantics.c: keep lfm2 in the q/k-norm name table from npu: LFM2.5-1.2B answers on the forward route — conv layers run, the symmetric tile kind is read, q/k norm on #2170 and the use_qk_norm key). The Hunyuan work is cl-npupack-a's and cl-1077's. If this lands, npu: Hunyuan-7B answers on the forward route: use_qk_norm is read, tiles are q,k,v, RoPE base and norm order follow the model (Fixes #1889) #1976 and Merge main into #1976: keep both new test functions in npu_pack_import_test.cpp (Refs #1889) #2130 are superseded.
2. The missing-q-gain refusal is scoped to layers with softmax attention. npu: Hunyuan-7B answers on the forward route: use_qk_norm is read, tiles are q,k,v, RoPE base and norm order follow the model (Fixes #1889) #1976 refuses a container whose layer has no q_norm while q/k norm is on. Applied to every layer, it refuses packs that load on main:
   • every LFM2 pack (layer 0 is a short-convolution layer): found by me on a device run of the first merge;
   • every qwen3_5 hybrid (layer 0 is a linear-attention layer, 18 of Qwen3.5-2B's 24 have no self_attn.q_norm): found by the reviewer at adb81c4, after I had exempted conv layers only.
     The rule is now q4nx_qk_gain_missing(qk_norm, layer_has_softmax_attention, gain_present), called with !is_conv_layer(l) && !gdn_l, where gdn_l is the test the step bodies use to send a layer to gdn_layer().
3. Load rows CI can run. tests/batch_check_state_ci.py fabricates a container at test time; it now also writes an LFM2 hybrid (batch_check_state_lfm2, full comparison) and a qwen3_5 hybrid (batch_check_state_qwen35_load, load only). The suite was green with each of the two refusals above in it.
4. From npu: the 4,608-byte symmetric chunk is read, so Hy-MT2-1.8B-NPU2 serves on the forward route #2203 (closed, folded in under the lane cap): the 4,608-byte symmetric chunk, Q4NX_CHUNK_Q4S, so Hy-MT2-1.8B-NPU2 loads. Its proof is the tie test against the pack's own bf16 embedding (mean 0.244, max 1.000 of a step; the eleven other readings are 3 to 8 steps off) and main's decoder returning -1 on the pack's real tiles; details in npu: the 4,608-byte symmetric chunk is read, so Hy-MT2-1.8B-NPU2 serves on the forward route #2203's body.

The refusal, before and after, on real containers

FastFlowLM Qwen3.5-2B-NPU2 (ONEBIT_TEST_Q4NX=…/q35-2b gdn_batch_order_test, host GEMM)
  at adb81c4   SKIP: init_host failed … layer 0 has no self_attn.q_norm.weight …                       rc 77
  at a7bbf94   PASS  4 tokens, 993280 logits: 18 GDN layer pass(es) … matched … bit for bit            rc 0
               batch_check_state_test on the same pack: ALL PASS, last chunk rel-L1 0.00e+00           rc 0

LFM2.5-1.2B-NPU2
  at b6df520   forward init failed: … layer 0 has no self_attn.q_norm.weight …
  at a7bbf94   loads; five answers identical to its #2170 device run

Hunyuan-7B imported before the gains fix (~/scratch-packs/w3b-pack/hunyuan: use_qk_norm true, 0 q_norm tensors)
  at 03145fa   forward init failed: … layer 0 has no self_attn.q_norm.weight …   (the refusal still fires where it should)

On the device, at a7bbf94 (the code of this head; 676a8f6 merges main on top)

Binary checked before each ticket (nm build/1bit | grep -c q4nx_dequant_tile_q4s = 1).

Qwen3.5-2B-NPU2     known answer: 3 answers, identical=True, content is Paris=True      decode 2.60 / 2.36 tok/s
Hy-MT2-1.8B-NPU2    against FastFlowLM 1.0.12's runtime, same model.q4nx: identical answers 4 of 5   (1 of 5 on main's semantics for this family)
LFM2.5-1.2B-NPU2    5 of 5 answers identical to its #2170 run

One thing seen and not explained: in the first ticket at this code the Qwen3.5 row ran, and then the Hy-MT2 and LFM2.5 servers started in the same ticket exited after their first ELF loaded, with nothing in their logs. Each came up normally in the next two tickets with the same binary (the rows above are from those). It looks like the device rather than this change, but I have not shown that.

Tests

• q4nx_qk_norm_test: the four cases of the rule.
• batch_check_state_ci.py: lfm2 ALL PASS (batched = sequential, rel-L1 0); qwen3_5 PASS (load only); the original qwen3 arm unchanged.
• Rebuilt by name and run: npu_dequant_bf16 (8 q4s checks), npu_design_grid, q4nx_layer_kinds, lfm2_shortconv, npu_rope_llama3, npu_pack_import, npu_moe_container, gdn_batch_order.
• ctest -j 6 after a fresh configure: 239 of 241; the two are npu_rt_shim and npu_dmabuf_import, which do not link in my build directory.
• That nothing of npu: Hunyuan-7B answers on the forward route: use_qk_norm is read, tiles are q,k,v, RoPE base and norm order follow the model (Fixes #1889) #1976 was lost: git merge-tree of main + Merge main into #1976: keep both new test functions in npu_pack_import_test.cpp (Refs #1889) #2130 conflicts only in q4nx_semantics.c (the reviewer checked the same), and no add_test name on main is missing here.

Limits, stated

• The fabricated qwen3_5 row is load only, and not by choice. On that fabricated model the full comparison fails its last check: the batched last chunk's logits against the sequential reference, rel-L1 0.71 with the same argmax (0.45 with two GDN layers and no attention layer, 0 with attention layers only). FastFlowLM's real Qwen3.5-2B passes the same binary with rel-L1 0, and NPU_GDN_CHECK reports both of the GDN path's internal comparisons identical on the fabricated model. Whether the fabrication or the batched path is at fault is not established; the row claims the load and says so in the script's header. This is a finding inside npu-pack: use_qk_norm is never consulted, so Hunyuan-7B imports with its q/k norm omitted and answers gibberish #1889/npu: every FastFlowLM/*-NPU2 model source imports and serves on --device npu (FLM 1.0.12 model sources, 59 repos) #1786's scope that I have not resolved.
• The two new add_test rows sit in the block CI configures (ONEBIT_NPU_HOST AND NOT ONEBIT_NPU). My build is the other configuration, so I ran the script by hand; CI will be the first to run the registrations.
• npu: Hunyuan-7B answers on the forward route: use_qk_norm is read, tiles are q,k,v, RoPE base and norm order follow the model (Fixes #1889) #1976's own Hunyuan-7B completion check was not rerun here (a 6.7 GB import; free space is under the floor).

🤖 Generated with Claude Code

[bong-water-water-bong]

ready for review at b6df520

[github-actions]

PR Reviewer Guide 🔍

(Review updated until commit adb81c4)

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶 1889 - Partially compliant Compliant requirements: Read use_qk_norm from the pack's config.json When the config says true and the layout arithmetic did not find the gains, refuse by name When it does find them, set p.knorm_off / p.qnorm_off from the measured section instead of the fixed kNormOff Non-compliant requirements: None Requires further human verification: None 1786 - Partially compliant Compliant requirements: None Non-compliant requirements: The specific requirements for each model in the FLM 1.0.12 parity work are not fully verified in this PR's scope Requires further human verification: Verification of each FastFlowLM/*-NPU2 model's import, serving, and performance metrics Confirmation that all listed models (59 repos) are properly handled by the changes

⏱️ Estimated effort to review: 4 🔵🔵🔵🔵⚪

🧪 PR contains tests

🔒 No security concerns identified

⚡ Recommended focus areas for review QK Norm Order Logic The logic for determining the order of q/k normalization and RoPE application has been updated to correctly handle Hunyuan models. However, the change introduces a conditional execution path that may not be thoroughly tested for all model types, especially those that do not explicitly declare use_qk_norm. If a model type is not properly recognized, it could default to the wrong normalization order. // per-head RMSNorm on q and k, only for architectures that have it. A KV-shared // layer (Gemma4) computed no k at all, so its k is never normalized or used. // engine#1889: WHERE it runs is per family. Qwen3 and Gemma normalise and then rotate; Hunyuan's graph rotates // and then normalises (q_proj -> RotaryEmbedding -> q_norm -> attention), so for it the same code runs after the // RoPE block below. The RMS itself is rotation-invariant; the per-dimension gain is not, so the order matters // wherever a rotary pair's two gains differ. auto qk_normalise = [&]() { std::vector<float> t(HD); for (int h = 0; h < NH; h++) { rmsnorm(&q[(size_t)h * HD], q_norm_[cur_layer_].data(), HD, t.data(), "q_norm"); memcpy(&q[(size_t)h * HD], t.data(), sizeof(float) * HD); } if (append_kv) { for (int h = 0; h < NKV; h++) { rmsnorm(&k[(size_t)h * HD], k_norm_[cur_layer_].data(), HD, t.data(), "k_norm"); memcpy(&k[(size_t)h * HD], t.data(), sizeof(float) * HD); } } }; if (sem_.qk_norm && !sem_.qk_norm_after_rope) qk_normalise(); // Gemma4 normalises v with a SCALE-FREE RMSNorm (its v_norm has no weight). Attention Tile Order The attention tile order logic has been updated to correctly handle Hunyuan models, which have a different projection order (q, k, v) compared to other models (k, q, v). This change is crucial for correct tensor layout interpretation. However, the new logic relies on attn_q_first flag which must be correctly set for all models to avoid incorrect tensor interpretation. // o_proj's K is q_proj's output width). struct Slot { const char* proj; int rows; int64_t base; int k; int first; }; const int64_t attn_off = attn_payload_off(H); // engine#1889: k, q, v on every family but the Hunyuan block, which is q, k, v (Plan::attn_q_first). const AttnFirstTiles ft = attn_first_tiles(n_kv, n_q, p.attn_q_first); const Slot slots[] = {{"k_proj", kv_out, attn_off, int(H), ft.k}, {"q_proj", q_out, attn_off, int(H), ft.q}, {"v_proj", kv_out, attn_off, int(H), ft.v}, {"o_proj", int(H), p.o_off, p.o_k, 0}}; for (int L = 0; L < p.dims.layers; ++L) {

---

⚠️ Review coverage: The following files were not included in this review because of the token budget:

• docs/npu.md

✅ Resolved findings

npu/forward/src/q4nx_forward.cpp:4880-4882

RoPE Base Calculation

The RoPE base calculation logic has been updated to correctly handle Hunyuan models with alpha parameter. This is important for correct attention computation. However, the new logic introduces a dependency on the alpha parameter, which must be correctly parsed and validated to prevent incorrect base calculations.

[bong-water-water-bong]

VERDICT: READY at b6df520 — and the "unchanged" claim is true in the way that matters, which I checked rather than assumed.

The deadlock it breaks is real

Fixes #1889. This is #1976, unchanged, brought onto main. #1976 is READY at a43aea3 and cannot merge: it conflicts with main.
Its merge-up, #2130, is READY too and cannot merge either, because its base is #1976's branch

Two READY PRs, neither mergeable, and the second blocked only by the first — that is a queue that cannot drain by itself. Re-applying onto main is the right resolution, provided the re-application carries the reviewed content rather than a re-type.

What I verified about "unchanged"

Byte comparison against #1976's head cannot settle this (main has moved underneath), so I checked why the files differ:

npu/forward/include/q4nx_rope.h            identical to #1976
CMakeLists.txt                             43 commits on main since #1976
npu/forward/src/q4nx_forward.cpp           17
npu/forward/src/q4nx_semantics.c            2
npu/forward/include/q4nx_semantics.h        1

So the difference is adaptation to main's own movement, file by file, and q4nx_rope.h — the header the qk-norm rule lives in — is untouched. That is what "unchanged" should mean, and it is worth saying in the body exactly like that: "unchanged except the four files main has moved since, adapted". A reviewer can then check four files instead of thirteen, and the claim stops depending on trust.

Two things to keep

• the Fixes #1889 and the "brings npu: Hunyuan-7B answers on the forward route: use_qk_norm is read, tiles are q,k,v, RoPE base and norm order follow the model (Fixes #1889) #1976 onto main" sentences together — one says what the issue is, the other says why this PR exists at all when a READY PR already covers it; without the second, a reviewer thinks it is a duplicate;
• Merge main into #1976: keep both new test functions in npu_pack_import_test.cpp (Refs #1889) #2130's fate, named: if it is a merge-up of the same work, it should close against this one, or the second re-application recreates the queue this PR is draining.

Nothing else: the content is the reviewed content, the base is current, and the deadlock has an exit.

[bong-water-water-bong]

Head moved to 03145fa, before any verdict. The merge as I first pushed it (b6df520) broke a pack that serves on main, and the body's "all I wrote is the merge" is no longer true: there is now one real change of mine.

What broke. #1976 refuses a container whose layer has no q_norm when q/k norm is on (a Hunyuan container imported without its gains would otherwise normalise q to zero). Main has since gained the LFM2 packs (#2170), where q/k norm is on and ten of sixteen layers, layer 0 among them, are short-convolution layers with no attention and therefore no q_norm. Each PR is right alone; together:

$ 1bit serve -m LFM2.5-1.2B-NPU2 --device npu          # at b6df520, inside an npu-hold ticket
1bit serve: forward init failed: config.json says q and k are normalised per head (use_qk_norm when stated, else
model_type/architectures) and layer 0 has no self_attn.q_norm.weight: normalising with an absent gain zeroes q, so
this container would answer wrong text (engine#1889); re-import the pack with a build that carries the gains

No test caught it: the suite was 239 of 241 at b6df520 with this defect in it. I found it because I ran the LFM2 pack on the merged binary to check my conflict resolution.

The fix. The rule is about attention layers. It is now q4nx_qk_gain_missing(qk_norm, layer_has_attention, q_gain_present) in q4nx_semantics.h, called with !is_conv_layer(l); four checks in q4nx_qk_norm_test pin it (an attention layer without its gain is still refused; a layer with no attention is not).

After, same pack, same ticket shape, at the fixed code:

LFM2.5 answers identical to the #2170 device run, per prompt: [True, True, True, True, True]

ctest -j 6: 239 of 241 (the same two targets that do not link in my build directory). git diff --stat origin/main: 11 files, 344 insertions, 13 deletions; the file list is still #1976's eleven.

What I still have not done: rerun #1976's own Hunyuan-7B check on this head, and a Hunyuan container without gains to see the refusal still fire end to end (the unit check covers the rule, not the load).

ready for review at 03145fa

[github-actions]

Persistent review updated to latest commit 03145fa

[bong-water-water-bong]

One of the two things I listed as not done at 03145fa, now measured on that head (no change to the code, the head does not move):

The refusal still fires end to end. ~/scratch-packs/w3b-pack/hunyuan is a Hunyuan-7B container imported on 11-10, before the importer carried the gains: config.json says use_qk_norm: true, and of its 291 tensors none is a q_norm or k_norm. Device-free (NPU_INFER_HOST_GEMM=1, opens no device):

$ NPU_INFER_HOST_GEMM=1 1bit serve -m ~/scratch-packs/w3b-pack/hunyuan --device npu --port 18578        exit 1
  [semantics] qk_norm=1  <- use_qk_norm when stated, else model_type/architectures
  [semantics] qk_norm order: RoPE, then norm  <- model_type/architectures (hunyuan* normalises after RoPE)
1bit serve: forward init failed: config.json says q and k are normalised per head (…) and layer 0 has no
self_attn.q_norm.weight: normalising with an absent gain zeroes q, so this container would answer wrong text
(engine#1889); re-import the pack with a build that carries the gains

So on this head: an attention layer without its gain is refused (this container), a layer with no attention is not (LFM2.5, five answers identical to its #2170 run), and the unit test pins both.

Still not done: #1976's Hunyuan-7B completion check on this head. It needs a fresh import of the 7B pack (6.7 GB) and free space on this box is 174G, under the 175G floor, so I am not writing it.

[bong-water-water-bong]

VERDICT: READY at 03145fa — fresh configure rc 0, and the three suites this touches all pass: npu_pack_import_test (full set), q4nx_qk_norm_test 21 arms, npu_rope_llama3_test (all llama3 rope checks passed).

Bringing a READY-but-unmergeable PR onto main is the right move and the body says why plainly: #1976 is READY at a43aea3, conflicts with main; its merge-up #2130 is READY too but sits on #1976's branch and the queue only merges main-based work. So the fix is to carry the change, not to keep a chain of PRs that cannot land.

The conflict is a one-line semantic collision, and the resolved form is the one I would have picked: #2170 added lfm2 to the q/k-norm name table on the line that #1976 follows with the use_qk_norm key — i.e. a family list and a switch that both answer "does this layer carry q/k gains". Keeping the family entry and the switch means LFM2's conv layers take the "not a refusal" path, which is exactly the arm that exists:

q/k norm on, a layer with no attention and no gain: not a refusal (LFM2's conv layers)
q/k norm on, an attention layer with its gain: fine          q/k norm off: no gain is expected anywhere

That middle arm is the one that matters after this merge: without it, the family entry makes a conv-only layer look like an attention layer missing its gains, and the refusal would fire on a healthy pack.

One ask: a docs/npu.md line naming which families the name table covers now (qwen3*, gemma*, lfm2, plus anything #1976's use_qk_norm key overrides). The conflict was precisely a list-and-switch interaction, so the table is the place a reader will next need to know whether a new family needs an entry, a declared key, or both.

Scope of my check: fresh configure rc 0; built and ran npu_pack_import_test, q4nx_qk_norm_test (21 arms) and npu_rope_llama3_test (all pass). No device and no Hunyuan pack on this box: the served behaviour #1889 reports is the PR's own.

[bong-water-water-bong]

Head moved to c066ffc, still before any verdict: the test that was missing. At 03145fa I wrote "no test caught it"; this is the one that does.

tests/batch_check_state_ci.py already fabricates a two-layer qwen3 container at test time and runs batch_check_state_test on it in host-GEMM mode (no device, no model on disk). With a second argument, lfm2, it now writes an LFM2-shaped hybrid instead: three layers (short convolution, attention with q/k norms, short convolution), model_type: lfm2, norm_eps, no layer_types. Registered as batch_check_state_lfm2 beside the existing test, in the same CI-only block. It is the only container with conv layers that CI loads through the forward.

Before (the test binary built at b6df520, the merge without the fix), new test:

$ python3 tests/batch_check_state_ci.py build/batch_check_state_test lfm2
SKIP: init_host failed for …/q4nx/model.q4nx (config.json says q and k are normalised per head (…) and layer 0 has no
self_attn.q_norm.weight: normalising with an absent gain zeroes q, so this container would answer wrong text (engine#1889); …)
  [forward] 2 gated short-convolution layer(s) of 3, kernel length 3 (LFM2 hybrid)
FAIL  batch_check_state_test skipped on the fabricated container (init_host refused it)                rc=1

After (the same binary rebuilt at the fix):

PASS  after the chunk at start 0 the forward holds exactly 3 positions
PASS  after the chunk at start 3 the forward holds exactly 5 positions
PASS  after the chunk at start 5 the forward holds exactly 6 positions
PASS  last chunk's logits vs the sequential reference: rel-L1 0.00e+00, argmax 25 vs 25
ALL PASS (0 failed)
  [forward] 2 gated short-convolution layer(s) of 3, kernel length 3 (LFM2 hybrid)
[batchcheck] M=3 positions=0..2 seq_argmax=42 batch_argmax=42 MATCH
[batchcheck] M=2 positions=3..4 seq_argmax=126 batch_argmax=126 MATCH
[batchcheck] M=1 positions=5..5 seq_argmax=25 batch_argmax=25 MATCH                                    rc=0

So beyond "it loads", it pins that a batched prefill through the conv layers leaves exactly the logits a token-by-token run leaves (rel-L1 0), on a container nobody has to download. The qwen3 variant still passes unchanged.

One limit, stated: the add_test sits in the block CI configures (ONEBIT_NPU_HOST AND NOT ONEBIT_NPU), and my build directory is the other configuration, so I ran the script by hand against my batch_check_state_test and have not seen ctest -R batch_check_state_lfm2 itself run; free space is under the floor for a second build tree. CI will be the first to run the registration, once it can resolve its mirrors again (#2211). ctest -j 6 here after a fresh configure: 239 of 241, the same two.

ready for review at c066ffc

[github-actions]

Persistent review updated to latest commit c066ffc

[bong-water-water-bong]

Head moved to adb81c4, still before any verdict here. Under the orchestrator's cap of two open PRs per lane (16:45Z) I had five, so #2203 (the Hy-MT2 tile reader, READY at 0911320) is merged into this branch unchanged and closed. The two belong together anyway: #2203's row was only a parity row with this PR's RoPE base and norm order, and until now that had been measured on a scratch build. It is now measured at one real commit.

What this PR is now

1. npu: Hunyuan-7B answers on the forward route: use_qk_norm is read, tiles are q,k,v, RoPE base and norm order follow the model (Fixes #1889) #1976 brought onto main (through Merge main into #1976: keep both new test functions in npu_pack_import_test.cpp (Refs #1889) #2130), one conflict kept both ways. Fixes npu-pack: use_qk_norm is never consulted, so Hunyuan-7B imports with its q/k norm omitted and answers gibberish #1889.
2. The missing-q-gain refusal scoped to attention layers, so LFM2 still loads; q4nx_qk_gain_missing, unit-tested.
3. batch_check_state_lfm2: CI loads a fabricated LFM2 hybrid through the forward (fails before 2, passes after).
4. From npu: the 4,608-byte symmetric chunk is read, so Hy-MT2-1.8B-NPU2 serves on the forward route #2203: the 4,608-byte symmetric chunk (Q4NX_CHUNK_Q4S), so Hy-MT2-1.8B-NPU2 loads. Refs npu: every FastFlowLM/*-NPU2 model source imports and serves on --device npu (FLM 1.0.12 model sources, 59 repos) #1786. Its evidence (the tie test against the pack's own bf16 embedding: mean 0.244, max 1.000 of a step; main's decoder rc=-1 on the pack's real tiles, this one rc=0) is in npu: the 4,608-byte symmetric chunk is read, so Hy-MT2-1.8B-NPU2 serves on the forward route #2203's body.

Measured at d2d5710 (the code of this head; adb81c4 adds docs only), one npu-hold ticket, 153 s. I checked the binary first (nm build/1bit | grep -c q4nx_dequant_tile_q4s = 1): my first attempt ran a stale 1bit, because a full cmake --build stops at the two targets that do not link here and had not relinked it; that run refused the pack with "chunk 4608 B is not a kind" and I threw it away.

Hy-MT2-1.8B-NPU2 against FastFlowLM 1.0.12's runtime, same model.q4nx, five greedy translations
p0 SAME 'Das Wetter ist heute schön, und wir gehen spazieren.'
p1 SAME 'Où se trouve la gare de train la plus proche ?'
p2 DIFF ours 'Me gustaría una taza de café con leche, por favor.'   theirs 'Quisiera una taza de café con leche, por favor.'
p3 SAME '会议已改至下周二下午三点举行。'
p4 SAME 'The train to Berlin leaves from track seven at half nine.'
identical answers: 4 of 5            (1 of 5 on the semantics main has for this family)

known answer: 3 answers, identical=True, content is Paris=True
decode rep 0: prompt=13 completion=64 ttft=0.942s decode=2.33 tok/s
decode rep 1: prompt=13 completion=64 ttft=0.724s decode=2.82 tok/s

LFM2.5-1.2B-NPU2 at the same binary: 5 of 5 answers identical to its #2170 device run

Tests at this head. The affected binaries rebuilt by name, then run: npu_dequant_bf16 (with the 8 q4s checks), q4nx_qk_norm, npu_design_grid, q4nx_layer_kinds, lfm2_shortconv, npu_rope_llama3, npu_pack_import, npu_moe_container: 8 of 8. batch_check_state_ci.py … lfm2: ALL PASS. Whole suite: 239 of 241, the two that do not link here. git diff --stat origin/main: 17 files, 506 insertions, 34 deletions.

Still not done, as before: #1976's Hunyuan-7B completion check on this head (a 6.7 GB import; free space is under the floor).

ready for review at adb81c4

[github-actions]

Persistent review updated to latest commit adb81c4

[bong-water-water-bong]

VERDICT at adb81c4: CHANGES

Blocking: this head refuses every Qwen3.5 hybrid at load. Main loads them. Measured with gdn_batch_order_test (host GEMM, no device; ONEBIT_TEST_Q4NX=~/.cache/scratch-cl-flm/q35-2b, FastFlowLM's Qwen3.5-2B-NPU2), in an ONEBIT_NPU=ON ONEBIT_NPU_HOST=ON build:

	result
main d7a0e889	rc 0: PASS 4 tokens, 993280 logits: 18 GDN layer pass(es), head-major recurrence matched row-major ... bit for bit
this head adb81c4	rc 77: SKIP: init_host failed ... (config.json says q and k are normalised per head ... and layer 0 has no self_attn.q_norm.weight: normalising with an absent gain zeroes q, so this container ...

Cause. qk_norm is 1 for qwen3_5: starts_with(model_type, "qwen3") at q4nx_semantics.c:212, and the config states no use_qk_norm. Commit 03145fa6 exempts only conv layers: q4nx_qk_gain_missing(sem_.qk_norm, !is_conv_layer(l), ...). A GDN (linear-attention) layer is not a conv layer, and it has no self_attn.q_norm either. In the containers on this box, q35-2b has 24 layers, 6 with q_norm and 18 linear_attn layers without one. q4nx-qwen35-4b has 32 layers, 8 with q_norm and 24 without. Layer 0 is a GDN layer, so the load stops there.

Fix: pass "this layer has softmax attention" as the second argument, not "this layer is not a conv layer". For example !is_conv_layer(l) && !(lin_kd_ > 0 && sem_.layer_types[l] == Q4NX_LAYER_LINEAR), the same test the step bodies use to send a layer to gdn_layer. Then add a load row that would catch this: the test above skips on init_host failure, so as a CI row it would have reported SKIP, not FAIL.

Also:

• The body describes an older head. It says "this branch is origin/main + #2130 + one conflict resolved" and "the same eleven files". The head now also carries #2203's 4,608-byte tile reader (merged in at d2d5710e), 03145fa6 (the LFM2 refusal change above), c066ffc8 (a fabricated-LFM2 row in batch_check_state_ci.py) and doc edits: 17 files. Those need saying, and #2203 is open on its own with a READY, so it is in two PRs at once.
• I checked that the #1976 part matches a clean main + #2130 merge: git merge-tree conflicts only on npu/forward/src/q4nx_semantics.c, as the body says. I have not re-run the Hy-MT2 device rows.
• CI has been red repo-wide since 13:53Z (engine#2211), so it has not checked this head.

[bong-water-water-bong]

VERDICT: READY at adb81c4

Built and ran both new tests: q4nx_qk_norm: ok (21 arms), all llama3 rope checks passed — and the layout arithmetic in the new header is exactly right.

scales [256] bf16 @[0,512)   +   codes [256][16] @[512,4608)
512 + 256*16 = 4608 = 512 + 4096      (4608 - 512 = 4096 = 256*16)

The 4,608-byte claim in the title is a size discriminator, and the PR says so: "The size names the kind (no tile of another kind is 4608 bytes)" — which I checked against the
neighbours in the same header: Q4NX_CHUNK_Q4K 4736, Q4NX_CHUNK_Q8 8704. So one number identifies the layout without a magic tag, and it is distinguishable from the two chunks
beside it.

The Q4_0 spelling is the interesting part, because it is a naming fix rather than a numeric one: FastFlowLM lists the pack as "Q4_0", but the tile has no zero-point block —
value = (code >= 8 ? code - 16 : code) * scale[g][r], symmetric — and the layout is group-major scales with column-major codes, "as in the 5120-byte tile" and "as in the Q4K
chunk" respectively. Reading the vendor's name instead of the bytes is how #1661's zero-point bug happened; this reads the bytes and keeps the name as a label.

And the tie test is the right way to settle a layout nobody documented: "its lm_head is the quantised copy of its own bf16 embed_tokens, and over the first 1024 rows
|dequant − embedding| / |scale| has mean 0.244 and maximum 1.000" — a maximum of exactly 1.000 scale, not 1.000 absolute, is what a rounding-only difference looks like; a layout with
rows and columns swapped would be off by orders of magnitude, not by one quantum.

The conflict resolution is minimal and in the right place: pad_head_rows.h changes by one line (the chunk-bytes list in head_chunk_bytes), which is #2026's header-only file
gaining this layout — not a re-derivation. And the reason a new PR exists is honest and familiar: "#1976 is READY at a43aea3 and cannot merge: it conflicts with main" — the
same supersession shape as #2133, #2166 and #2222 today.

One note: the new chunk has no named constant beside Q4NX_CHUNK_Q4K/Q4NX_CHUNK_Q8 — the 4608 is inline in the comment and presumably in the switch. A Q4NX_CHUNK_Q4_0 4608
would put the three sizes in one place, which is where a reader looks to check "is this size unique?" Not a gate.

Not run: the Hy-MT2 pack itself (the tie test above is the author's, on a device ticket).

[bong-water-water-bong]

Answering the CHANGES at adb81c4. All three points are right; fixed at 676a8f6 (the code change is a7bbf94).

1. Qwen3.5 hybrids refused at load. Reproduced with your command, then fixed as you described: the second argument is now "this layer has softmax attention" (!is_conv_layer(l) && !gdn_l, gdn_l being the step bodies' own test).

ONEBIT_TEST_Q4NX=~/.cache/scratch-cl-flm/q35-2b ./build/gdn_batch_order_test
  at adb81c4   SKIP: init_host failed … layer 0 has no self_attn.q_norm.weight …                                 rc 77
  at a7bbf94   PASS  4 tokens, 993280 logits: 18 GDN layer pass(es), head-major recurrence matched row-major …   rc 0
batch_check_state_test, same pack, at a7bbf94:  ALL PASS, last chunk rel-L1 0.00e+00
On the NPU at a7bbf94:  Qwen3.5-2B  known answer: 3 answers, identical=True, content is Paris=True

I exempted conv layers at 03145fa because that was the failure in front of me, and did not ask which other layer kinds have no q. That is the same mistake twice in one PR; the rule's name and comment now say "softmax attention" so the next layer kind has to be argued in, not out.

2. A load row that fails rather than skips. batch_check_state_ci.py qwen3_5 fabricates a linear-attention layer followed by an attention layer through the packer's qwen3_5 path and requires init_host to accept it (batch_check_state_qwen35_load). It is load only, and the body says why at length: on the fabricated model the full batched-vs-sequential comparison fails its last check (rel-L1 0.71, same argmax) while the real Qwen3.5-2B passes the same binary at 0. I spent an hour on it (head counts, layer order, GDN-only, NPU_GDN_CHECK) without finding whether my fabrication or the batched path is at fault, so the row does not claim the arithmetic and I have listed it as unresolved rather than tune the fixture until it passes.

3. The body. Rewritten for this head: 18 files, the four parts, and #2203 is closed (folded in here under the lane cap, said on #2203).

Also rerun on the NPU at a7bbf94: Hy-MT2 4 of 5 against FastFlowLM's runtime, LFM2.5 5 of 5 against its #2170 run. One unexplained event in the first ticket is in the body.

ctest -j 6: 239 of 241 (the two that do not link here).

ready for review at 676a8f6

[github-actions]

Failed to review PR

[bong-water-water-bong]

VERDICT at 676a8f6: READY

This is the right way out of a three-PR knot, and it says so explicitly: #1976 is READY at a43aea3 and conflicts with main; its merge-up #2130 is READY but has #1976's branch as its base, which the queue does not merge. Consolidating onto main settles all three at once.

What I can verify vs what I am trusting — 18 files is past the size where a READY means "I read every line":

• verified: it is on main, CI is green, and the q/k norm arm is present and registered (add_test(NAME q4nx_qk_norm …) against npu/forward/src/q4nx_semantics.c) — the arm that made #1889's refusals checkable;
• trusting the body for: that the 18 files are #1976's content plus #2130's bring-up and nothing else. The body's own structure (a numbered "what is in it", and a note that it was rewritten after a review said the old body described an older head) is the reason I am willing to; that is the second time today a PR body has been corrected in response to review rather than left stale.

Consequence for the queue, which is the useful part: with this open, #1976 and #2130 are superseded and should be closed rather than merged — otherwise the same content lands twice or conflicts at merge time. I am not closing another lane's PRs, but the reviewer's note belongs here so the author sees it in one place.

And it closes #1889, the issue whose headline I flagged as stale a few ticks ago: use_qk_norm was being consulted in six places while the issue still read "never consulted". This is the work that makes it true end to end rather than only in the importer.