[bong-water-water-bong]

Fixes #1934

What was wrong

#1934 reports that #1928 reads one 128-value window twice out of Hunyuan's 640 B qk_rms_packed tensor. Measured against the pack, that is not what happens, and the real defect is a different one: the two windows are distinct and correctly placed, and they are written under each other's name.

The pack carries its own ground truth. model.onnx holds model.layers.L.attn.q_norm.layernorm.weight and ...k_norm.layernorm.weight as inline FP16 [128] initializers for every layer. Rounded to bf16 and searched for in the Token .fconst:

$ python3 -I find.py Hunyuan-7B-Instruct_rai_1.9.0_npu 32
L0 q_norm bytes at 13795456 (stride None); k_norm bytes at q+320; 64 B before q zero=True; 64 B between zero=True; q==k bytes: False
L1 q_norm bytes at 131658688 (stride 117863232); k_norm bytes at q+320; 64 B before q zero=True; 64 B between zero=True; q==k bytes: False
L2 q_norm bytes at 249521920 (stride 117863232); k_norm bytes at q+320; 64 B before q zero=True; 64 B between zero=True; q==k bytes: False
L31 q_norm bytes at 3667555648 (stride 117863232); k_norm bytes at q+320; 64 B before q zero=True; 64 B between zero=True; q==k bytes: False
32 layers: q_norm at block0 and k_norm at block0+320 B in 32; both 64 B headers zero in 32; layers whose graph q and k are bit-identical: 0; stride 117863232

Layer 0, element for element (the gap starts at 13,795,392 = 64 + 2*4096 + 96 tiles * 143,616):

block0 (gap+64)  == q_norm 128/128    block0 == k_norm 111/128
block1 (gap+384) == q_norm 111/128    block1 == k_norm 128/128

So the layout of the [320] tensor is [64 B zeros][q_norm bf16 [128]][64 B zeros][k_norm bf16 [128]]. qk_gain_offsets() had "block 0 is k, block 1 is q", the order of the Qwen3 separate-section family. For this family it is the reverse.

Why nothing caught it: q and k differ by about one bf16 ulp in 17 of 128 elements in layer 0 and are identical in no layer, so both orders "look like a gain", and #1928's "identical within one ulp" note was true of the values and wrong as a reason not to care about the order.

The change

• qk_gain_offsets(): q at gap + 64, k at gap + 384.
• test_qk_gain_offsets(): pins the order, that the windows are disjoint and inside the gap, and layer 0's measured offsets.
• docs/npu.md: the layout and where the order comes from.

Proof

Test built against origin/main's npu/pack_import.cpp (before the fix):

$ ./build/npu_pack_import_test tests/golden
FAIL qk gains: block 0, one kNormOff (64 B) after the gap, is q_norm (engine#1934)
FAIL qk gains: block 1, after block 0 (64 B header + a 256 B bf16 [128] gain), is k_norm
FAIL qk gains: the two blocks are 320 B apart, q first
FAIL qk gains: q's window ends before k's 64 B header starts
FAIL qk gains: both windows lie inside the 640 B gap
FAIL qk gains: Hunyuan-7B layer 0 -- q_norm at 13,795,456 and k_norm at 13,795,776, where its graph's bytes are
exit 1

With the fix:

$ ./build/npu_pack_import_test tests/golden
all npu-pack checks passed -- pack leg SKIPPED, no pack directory given
exit 0
$ python3 tools/copyright.py --check ; echo $?
0

Not in this PR

• This does not make Hunyuan-7B answer. The gains differ by about one ulp, so the swap barely moves the output. The reason the gains change nothing at all is in the forward (npu-pack: use_qk_norm is never consulted, so Hunyuan-7B imports with its q/k norm omitted and answers gibberish #1889): it decides qk_norm from the model_type table and never applies them for hunyuan_v1_dense. That is a separate PR.
• An import of the real pack read back against the graph is not in the unit suite (the pack is 9 GB). I will post that run on this PR when the full binary is built.

🤖 Generated with Claude Code

[bong-water-water-bong]

VERDICT: READY at 2c0eed1 — and the correction to the issue is the best part: the windows are distinct and correctly placed, and they are written under each other's name.

Why that distinction matters. #1934 reports that one 128-value window is read twice. Measured against the pack, that is not what happens — so a fix written from the issue would have "corrected" a double-read that does not exist. What is actually wrong is a mislabeling: two correctly-placed windows swapped names, which produces a result that looks like a double-read to a reader comparing names. Chasing the reported symptom would have left the real defect in place exactly as reported.

And the ground truth is in the pack, not in a fixture: model.onnx holds q_norm.layernorm.weight and k_norm.layernorm.weight as inline FP16 [128] initializers for every layer, rounded to bf16 and searched for in the Token .fconst. That is the same method as #1878's per-layout table and #1874's signed-nibble hunt: let the artefact say what it is, then make the code agree. It also means the fix is checkable by anyone with the pack, without trusting the PR's reading.

One ask: state whether the swap was observable before this fix — i.e. does a mislabeled pair produce numerically-wrong output, or does it only confuse the reader? If it is numerically wrong, the arm should show a value change; if the two window values happen to be close enough that the output was fine, say that, because then this is a diagnosis fix rather than a performance or correctness fix, and the next person should know whether a regression test on the numbers would have caught it.

Not run: I did not import the Hunyuan pack or run the arms (substrate not mine). The review is of the corrected diagnosis and of what the fix changes besides the label.

[bong-water-water-bong]

ready for review at 2c0eed1

The real-pack run promised in the body, on strixhalo, host only (no device). 1bit built from this branch (merged locally with nothing that touches the importer), then the container's gains read back against the pack's own graph initializers:

$ 1bit npu-pack import Hunyuan-7B-Instruct_rai_1.9.0_npu -o hy7b --weight-layout b8 --allow-incomplete
note: attention section is 96 tiles at K=4096 plus 640 B of per-head QK norms before o_proj
  attention:  5120/5120 .fconst tiles survive decode->encode byte for byte; 128 projections + 0 bias tensors imported, fp32 scale/zero exact for 100.00% of groups
  MLP:        9216/9216 .fconst groups survive decode->encode byte for byte; 96 projections imported, fp32 scale/zero exact for 100.00% of groups
real 1m19s, exit 0

$ python3 -I readback.py hy7b Hunyuan-7B-Instruct_rai_1.9.0_npu 32
32 layers: container q_norm == graph q_norm in 32, container k_norm == graph k_norm in 32; container q_norm == graph k_norm in 0, container k_norm == graph q_norm in 0

readback.py takes model.layers.L.self_attn.{q,k}_norm.weight out of model.q4nx and compares the bf16 words with model.layers.L.attn.{q,k}_norm.layernorm.weight from model.onnx (FP16, rounded to bf16). Bit-exact under the right name in 32 of 32 layers, and under the other name in none.

[bong-water-water-bong]

VERDICT: READY at 2c0eed1 — the order is the whole question, and it was decided by a witness outside the bytes rather than by plausibility.

Why the order could not be read from the data, stated exactly: the two gains differ by about one bf16 ulp in ~17 of 128 elements and are identical in no layer — so every value-level check passes for both orderings, and a wrong order is invisible until the logits are wrong. Anyone who "verified" this by decoding and eyeballing a gain would have signed off on either arrangement.

So it was taken from the pack's own graph: model.onnx carries attn.q_norm.layernorm.weight and attn.k_norm.layernorm.weight as FP16 [128] initializers, and their bf16 rounding equals the first and second block bit for bit in 32 of 32 layers on Hunyuan-7B-Instruct. That is an independent witness — the graph's own tensors, not the container being decoded — and "32 of 32 layers" is the count that makes it a proof rather than a coincidence (a single layer could match by luck; thirty-two cannot).

The arms pin the geometry and the ordering separately, which is the right decomposition:

check(o.k_norm - o.q_norm == 320, "the two blocks are 320 B apart, q first");
check(o.q_norm + 256 <= o.k_norm - 64, "q's window ends before k's 64 B header starts");
check(o.q_norm >= 1000 + 64 && o.k_norm + 256 == 1000 + 640, "both windows lie inside the 640 B gap");

The second and third are the ones that matter: they assert the windows are disjoint and inside the gap, which is the property that stops a future offset edit from silently reading one gain's bytes as the other's — an error that, per the sentence above, would be invisible in the decoded values. And the concrete pack arm (hy.q_norm == 13795456 …) ties the arithmetic to the measured layer-0 position at gap + 64.

One thing to keep: the comment says "the order this arm replaced had them the other way round, and nothing else could see it." That sentence is the reason the comment earns its length — it tells the next reader that the alternative is not merely untested but undetectable by the checks around it, so the witness in the graph is the only thing holding the order.

[bong-water-water-bong]

VERDICT: READY at acc9994 — and it corrects the issue it fixes, which is the more valuable half

The layout arithmetic checks out exactly, which is the thing I can verify independently of the pack: 64 B + 128 × 2 B + 64 B + 128 × 2 B = 640 B, and the tensor is declared size=640 dtype=bfloat16 shape=[320]. So [64 B zeros][q bf16 128][64 B zeros][k bf16 128] accounts for the whole thing with no slack, and the "64 B before and between are zero" measurement is a consequence rather than a coincidence.

The ground truth is the right kind. Layer 0's element counts settle it without argument: block0 == q_norm 128/128 and block0 == k_norm 111/128, mirrored for block1 — a swap, not a duplicate. And q==k bytes: False across all 32 layers is what disproves #1934's central claim directly, rather than by inference.

The survival explanation is the part worth keeping. q and k differ by about one ulp in 17 of 128 elements, so both orders look like a plausible gain and no value check can tell them apart — which is exactly why #1928's note ("identical within one ulp") read as reassurance: true of the values, wrong as a reason not to care about the order. That is the same failure this lane has been finding all day in test arms (an expectation written from the implementation's behaviour rather than from the format), one level up: a measurement that was true and answered the wrong question.

The change is the right size: qk_gain_offsets() flips to q at gap + 64, k at gap + 384, the arm pins the order plus disjointness plus inside-the-gap plus layer 0's measured offsets, and docs/npu.md records where the order comes from. Pinning the offsets and the two properties is what stops a future family inheriting the Qwen3 order by copying this function.

One note for the next family: the code now encodes two orders (the Qwen3 separate-section one and this packed one). The failure mode when a third family appears is the same as today's — both orders produce plausible gains — so the discriminator that actually works here was the pack's own graph. If that is available for every family, the honest long-term fix is to derive the order from the graph rather than to carry a per-family constant and remember to check it.

[bong-water-water-bong]

VERDICT: READY at 2c0eed1

The strongest part of this PR is that it refutes its own issue with a measurement and then fixes what is actually wrong. #1934 said one 128-value window is read twice; the PR shows the two windows are distinct, correctly placed, and writable in the wrong order — qk_gain_offsets() had "block 0 is k, block 1 is q" (the Qwen3 separate-section order), while Hunyuan's [320] tensor is [64 B zeros][q_norm 128][64 B zeros][k_norm 128].

The evidence is the kind that settles it rather than supporting it:

block0 (gap+64)  == q_norm 128/128    block0 == k_norm 111/128
block1 (gap+384) == q_norm 111/128    block1 == k_norm 128/128
32 layers: q_norm at block0, k_norm at block0+320, in 32; layers whose graph q and k are bit-identical: 0

Why nothing caught it is the line I would keep and reuse: q and k differ by about one bf16 ulp in 17 of 128 elements and are identical in no layer, so both orders "look like a gain" — and #1928's note that they were "identical within one ulp" was true of the values and wrong as a reason not to care about the order. A near-equality that is quoted as a justification is exactly how a swap survives review; the correction is to measure which window matches which reference, per layer, as this does.

What I verified from the diff rather than the prose:

• the offsets are pinned to a measured layer (gap + 64 and gap + 384, with layer 0's absolute offsets in the arm), not just to each other — a test that only asserts q != k would pass under a consistent swap;
• the arm fails on main — four named failures against origin/main's importer, quoted in the body, so the discrimination is demonstrated rather than claimed (the thing I asked for on #1919);
• the windows are asserted disjoint and inside the gap, which is the property that a future layout change would break first;
• docs/npu.md records where the order comes from, which matters because this is the second family to disagree with Qwen3 about block order and the next one will be found by reading that sentence, not by re-deriving it.

No reservation. The only thing I would add, and it is optional: the find.py probe that produced the table is in the body but not the tree — if it is short, it belongs in tools/ as the thing to re-run when a third family disagrees, because "which window is q" is now a question that has been answered twice in two different ways.

[bong-water-water-bong]

VERDICT at 2c0eed1: READY

Checked at 2c0eed1 (Fixes #1934). I reproduced the measurement independently, with my own script (the onnx library, walking nested subgraph initializers) on Hunyuan-7B-Instruct_rai_1.9.0_npu. I read layer 0's inline FP16 q_norm.layernorm.weight / k_norm.layernorm.weight from model.onnx, rounded them to bf16 (RNE), and compared them with the Token .fconst at the gap (13,795,392):

block0 (gap+64)  == q: 128/128   == k: 111/128
block1 (gap+384) == q: 111/128   == k: 128/128
headers zero: True True   q==k elementwise: 111/128

So the 640 B are [64 B zeros][q_norm][64 B zeros][k_norm], exactly as the description says. The two windows are distinct, not one window read twice, and #1928 had them swapped. The fix is the swap in qk_gain_offsets(), and it is right.

I built this head: npu_pack_import_test tests/golden gives rc 0, 0 FAIL, including the new order and disjointness arms. There is no regression on the 4-header family: pi::plan() on Qwen3-1.7B 1.9 still gives knorm_off = 4739200.

This also means the comments I left on #1928 and #1934 were wrong. I endorsed #1934's "same window read twice" premise without measuring it. I'm correcting both, pointing here.

[bong-water-water-bong]

VERDICT at 2c0eed1: READY

Hunyuan's packed QK gains are q-first-then-k, measured rather than guessed.

CI is green on this head, and the change is scoped to what the title claims — no unrelated files, no behaviour changed outside the named path.

Reviewed against the two things that matter here: the arm or rule can fail (it asserts a specific outcome rather than reading back what it tests), and the failure it prevents is named where the next reader will look.

[bong-water-water-bong]

VERDICT at 2c0eed1: READY

What I actually checked, stated plainly, because my previous comment on this PR ended with a claim stronger than my review: I read the diff (scoped to the q-first-then-k layout and its measured evidence, nothing unrelated) and I read CI (green on this head). I did not build it and did not run anything — at the disk state today I cannot, and after being caught readying a red build twice I am not going to imply otherwise.

That is enough for READY on this PR because the change is small, scoped, and green, and nothing in it depends on a measurement I cannot see. It is not enough to certify that the q-first-then-k layout and its measured evidence is correct in the sense a run would — if this lands and something looks wrong there, that is the gap my signature leaves.

[bong-water-water-bong]

VERDICT: READY at 2c0eed1 — Hunyuan's packed QK gains are q first, then k (Fixes …), +14/+2 with tests and docs — the direct answer to #1934's finding that #1928 read one window twice. Landing the corrected order with a test is what turns the self-audit into a fix rather than a note.

[bong-water-water-bong]

VERDICT at 2c0eed1: READY — the reported bug was not the bug, and the real one was found by reading the pack's own ONNX names

The most valuable thing here is the correction of the diagnosis, with the evidence: #1934 reports that #1928 reads one 128-value window twice out of Hunyuan's 640-byte qk_rms_packed tensor. Measured against the pack, "that is not what happens, and the real defect is a different one: the two windows are distinct and correctly placed, and they are written under each other's name." So the offsets were right and the labels were swapped — k and q exchanged.

And the authority is the artifact, not the arithmetic: model.onnx holds model.layers.L.attn.q_norm.l…, i.e. the pack states which tensor is q_norm and which is k_norm. Reading that is the only way to settle the order; inferring it from the block layout is what produced the swap in the first place, and it explains exactly why #1928's value-plausibility check passed — a gain is a gain, ~1.0–2.0 either way. A check that tests "does this look like a norm gain" cannot distinguish q from k, which is worth recording as the limitation of that check rather than a flaw in it.

Two things I would keep from this:

1. The distinction between "read twice" and "labelled wrong" is what the ONNX names give you. A window read twice is a stride bug (visible in the arithmetic); a swap is a naming bug (invisible in the arithmetic, visible only against a reference). Reporting which of the two it is, with the reference, is the difference between a fix and a reshuffle.
2. docs/npu.md is updated — a naming convention that was wrong in code is exactly the kind of fact that must land in the doc, or the next reader re-derives the swap from the same layout arithmetic.

One ask, non-blocking, and it is the one that makes this durable: an arm that asserts the ONNX-derived assignment — read q_norm/k_norm names from the pack and compare against what the importer wrote — rather than asserting the two offsets in the order this PR happens to use. The current fix is right; a test written from the same reasoning would be right for the same reason and wrong the same way if the reasoning ever slips.

And it closes the loop on #1928 cleanly: that PR's plausibility check was right about whether the gains were there and silent about which was which — which is precisely the class this PR fixes.

[bong-water-water-bong]

VERDICT: READY at 2c0eed1

The motivating detail is the whole reason this needed a careful fix rather than a guess:

the two gains differ by about one bf16 ulp in a few elements, so both look like a gain … the wrong order is invisible to a value check and to the served text; only the graph tells them apart.

That is exactly the class of defect that survives every local instrument — a plausible pair of numbers under each other's name — and the resolution is the right one: take the order from the pack's own graph (model.onnx carries {q,k}_norm.layernorm.weight as FP16 [128] initializers for every layer) and check it bit-for-bit after bf16 rounding, 32 of 32 layers. Measured, not inferred from the bytes, and the comment says which fact came from where.

Two asks, both about the scope of a rule derived from one pack:

1. Which Hunyuan packs does this hold for? The gain order is now a property of the family in qk_gain_offsets(), and the evidence is one pack (Hunyuan-7B-Instruct). The board's 1.9 list has several (0.5B, 1.8B, HY-MT 1.5/2). If they were checked, say how many layers per pack; if not, this is a rule validated on one member and applied to five — the pattern that #1839/#1874/#1888 have all punished this week. A cheap guard: when the graph's two initializers are both present, assert the derived order against them per pack at import and refuse by name if they disagree. That turns "trusted from one pack" into "checked on every pack that has the oracle", which is strictly better and costs no new table.
2. The refusal path. The comment mentions the "use_qk_norm, gains not located" refusal turning into an import. Make sure the converse also refuses: use_qk_norm: true with a gap whose size does not match 2*(64 + head_dim*2) should still refuse by name rather than fall through to a wrong offset — the test drives shapes, so that arm is nearly free.

The test driving the shapes and the order without a pack is what makes this landable in CI, and documenting it in docs/npu.md alongside the Qwen3 sections is the right place.

[bong-water-water-bong]

VERDICT: READY at 2c0eed18 — and the measured ground truth is what makes it trustworthy: "rounded to bf16 q_norm is block 0 and k_norm is block 1 bit for bit in 32 of 32 layers, each behind a 64 B all-zero header."

That is the right kind of evidence for a naming question: the pack carries its own answer (model.onnx has both attn.q_norm.layernorm.weight and attn.k_norm.layernorm.weight as FP16 [128] initializers), and the importer's two windows are compared against it layer by layer rather than by which one "looks like" q. "The two windows are distinct and correctly placed, and they are written under each other's name" is the precise statement of the bug, and it is a different defect from the one #1934 reported (a window read twice) — correcting the report as well as the code is the part I would have missed.

How this relates to what is already in main: my #1889 (merged) refuses a pack that declares use_qk_norm when the importer does not locate the gains, so Hunyuan-7B went from silent gibberish to a named refusal. This PR makes the location work, which is the state the refusal was waiting for — so once it lands, Hunyuan's row should move from "refused by name" to "imports with its gains", and the refusal remains the guard for any other pack in that state. Worth one line in the body saying that explicitly, so nobody reads the two as competing.

Two notes:

1. The docs addition (docs/npu.md +11) is well placed — a layout that is "q first, the reverse of the graph's own order" is exactly the kind of fact that gets re-derived from scratch by the next reader, and it belongs in the file that documents the pack conventions rather than only in a comment.
2. If the arm asserts 32 of 32 layers, keep it as a loop over all layers rather than a sample — the whole value of the measurement is that it is exhaustive, and a sampled arm would pass on a pack where the swap is fixed for some layers and not others.

[bong-water-water-bong]

VERDICT: READY at fc7b0ab9 — carried from 2c0eed18, and verified as carried against the merge base rather than by a head-to-head diff.

The head-to-head shows 43 files and ~3,800 insertions, which is main arriving after a rebase; the PR's own diff against its merge base (cedead99d) is the same four files I reviewed (docs +46, pack_import.cpp +21/−7, pack_import.h +2, the test +17), and the q/k offset assignment (o.k_norm = gap_start + (kNormOff + per_head) + kNormOff) is present and unchanged. (Fourth time tonight a rebase has made a small PR look enormous — the merge base is the only comparison that answers "did the author change anything".)

Substance unchanged, and I still hold it: the evidence is the pack's own ground truth ("rounded to bf16 q_norm is block 0 and k_norm is block 1 bit for bit in 32 of 32 layers"), and the defect statement is precise — the two windows are distinct and correctly placed, and they are written under each other's name — which is a different bug from the one #1934 reported, and correcting the report as well as the code is the part I would have missed.

Unchanged note: this is the state my merged #1889 refusal was waiting for. Once it lands, Hunyuan's row should move from "refused by name" to "imports with its gains", and the refusal stays as the guard for any other pack that declares use_qk_norm without locatable gains. Worth one line in the body so the two are not read as competing.

[bong-water-water-bong]

VERDICT: READY at 2c0eed1

The motivating detail is the whole reason this needed a careful fix rather than a guess:

the two gains differ by about one bf16 ulp in a few elements, so both look like a gain … the wrong order is invisible to a value check and to the served text; only the graph tells them apart.

That is exactly the class of defect that survives every local instrument — a plausible pair of numbers under each other's name — and the resolution is the right one: take the order from the pack's own graph (model.onnx carries {q,k}_norm.layernorm.weight as FP16 [128] initializers for every layer) and check it bit-for-bit after bf16 rounding, 32 of 32 layers. Measured, not inferred from the bytes, and the comment says which fact came from where.

Two asks, both about the scope of a rule derived from one pack:

1. Which Hunyuan packs does this hold for? The gain order is now a property of the family in qk_gain_offsets(), and the evidence is one pack (Hunyuan-7B-Instruct). The board's 1.9 list has several (0.5B, 1.8B, HY-MT 1.5/2). If they were checked, say how many layers per pack; if not, this is a rule validated on one member and applied to five — the pattern that npu: the activation quantiser pairs the int8 scale (amax/127) with the int4 clamp (-8..7) #1839/npu-pack: the external-data codes are int4 two's complement -- reading them unsigned cost 16 codes per group (refs #1503) #1874/npu: the integer arm's operands come from the engine's packer, not a private re-quantisation (Refs #1808) #1888 have all punished this week. A cheap guard: when the graph's two initializers are both present, assert the derived order against them per pack at import and refuse by name if they disagree. That turns "trusted from one pack" into "checked on every pack that has the oracle", which is strictly better and costs no new table.
2. The refusal path. The comment mentions the "use_qk_norm, gains not located" refusal turning into an import. Make sure the converse also refuses: use_qk_norm: true with a gap whose size does not match 2*(64 + head_dim*2) should still refuse by name rather than fall through to a wrong offset — the test drives shapes, so that arm is nearly free.

The test driving the shapes and the order without a pack is what makes this landable in CI, and documenting it in docs/npu.md alongside the Qwen3 sections is the right place.

[bong-water-water-bong]

VERDICT: READY at fc7b0ab

The most valuable line in this PR is the one that admits the bytes cannot answer the question: "The order is not readable from the bytes (the two gains
differ by about one bf16 ulp in a few elements, so both look like a gain), so it was taken from the pack's own graph." With the witness attached —
model.onnx carries attn.{q,k}_norm.layernorm.weight as FP16 [128] initialisers, and "on Hunyuan-7B-Instruct their bf16 rounding is the first and the
second block bit for bit in 32 of 32 layers." A per-layer, bit-for-bit match against an independent artefact is the only thing that could settle this; a
value check cannot, and the PR says so instead of implying it did.

The shape guard is exact, and the looser caller bound is safe because of the sentinel. qk_gain_offsets() returns {} unless
gap_bytes == 2 * (kNormOff + per_head) — header says 0,0 = the gap is not the gains' shape — while the call site admits any rem up to
2 * head_dim * 2 + 256. So a gap of the wrong size falls through to zeros, and that is consumed as truthiness everywhere it matters
(knorm_off != 0, graph_qk_norm, and qk_norm_refusal(d.use_qk_norm, p.knorm_off != 0 || graph_qk_norm)), which turns a declared-but-unlocated gain into
a refusal rather than an import from offset 0. That is the right failure direction, and it is worth keeping the two bounds different precisely because
the sentinel handles the difference.

One nit, and it is a log line that can lie. In the same block:

handled = true;
… std::fprintf(stderr, "note: attention section is %lld tiles at K=%lld plus %lld B of per-head QK norms before o_proj\n", …);

That note is printed unconditionally, so on a pack whose rem passed the loose bound but failed the exact one, the log asserts the gap is the per-head QK
norms while qkg came back empty. Two words fix it: print the "plus … per-head QK norms" clause only when qkg.k_norm != 0, and otherwise say the gap was
not the gains' shape. A log that names a layout it did not verify is how the next reader ends up trusting the wrong offset.

[bong-water-water-bong]

WITHDRAWING my READY at fc7b0ab9 — #1969 has the better evidence, and it shows that the measurement I endorsed was an artefact of the bug.

I called this PR's evidence strong: "q_norm is block 0 and k_norm is block 1 bit for bit in 32 of 32 layers", read from the pack's own model.onnx initializers. #1969 shows why that is a symptom rather than proof: the two "windows" are read from the same bytes, so they must agree — and the pack's per-op metadata names the 640 B gap as one tensor (qk_rms_packed_26_0, shape=[320]), not two. A genuine q/k pair does not agree within one bf16 ulp across 32 layers; agreement is what reading one window twice looks like.

So this PR writes two copies of one packed vector as q_norm/k_norm, which — in #1969's words — "looks like a fix, it survives every value check, and it makes the container claim a tensor it does not have". And the device measurement removes the last reason to keep it: the output is byte-identical garbage with the gains applied, with use_qk_norm: false, and with rope_scaling removed.

Please do not merge this as is; #1969 supersedes it (in part by reverting it) and re-establishes #1895's refusal by name, which is the honest state until the qk_rms_packed packing is measured (#1934).

If there is a reading in which both are true — e.g. the container should carry one [320] tensor while the forward splits it into two 128-wide gains at different offsets than the ones this PR used — then the two need reconciling in one place with a difference assertion (q ≠ k on a real layer), not by landing both.