npu-pack: the 1.9 packs split on the MatMulNBits K block size (32, and mixed 32/128), before the hidden-896 limit is reached

The 1.9 queue splits on the MatMulNBits K block size, and the smallest packs in it are the block-32 ones

Measured on origin/main = 5582ca64 (the merged #1412 geometry fixes included), 1bit npu-pack info against the 10 smallest ungated 1.9 packs, pre-fetched so this is repeatable without downloading (§ disk note below). Evidence file: ~/briefs/amd-rai-1.9-baseline-main.json, per-pack copies of the packs under ~/models/amd-rai/prefetched/1.9/.

pack	geometry	result
`SmolLM2-135M-Instruct`	h576, 30L, llama	hard refusal: block size along K = 32 (151 ops)
`Qwen2-1.5B`	h1536, 28L, qwen2	hard refusal: block 32 (1 op) + 128 (140 ops) — mixed in one pack
`Qwen-2.5_1.5B_Instruct`	h1536, 28L, qwen2	same mixed shape
`Qwen2.5-Coder-1.5B-Instruct`	h1536, 28L, qwen2	same mixed shape
`Qwen2.5-0.5B-Instruct`	h896, 24L, qwen2	parses; 169 tensors refused
`Hunyuan-0.5B-Instruct`	h1024, 24L, hunyuan_v1_den	parses; 169 refused
`Llama-3.2-1B` / `-Instruct`	h2048, 16L, llama	parses; 113 refused each
`Qwen3-1.7B`	h2048, 28L, qwen3	parses; 197 refused
`HY-MT1.5-1.8B`	h2048, 32L, hunyuan_v1_den	parses; 225 refused

The refusal text is exact and is the point of this issue:

this pack's MatMulNBits block size along K is 32 (151 ops), and this importer decodes the block-128 tile layout only: at a different block every subvolume offset in the .fconst moves, so the weights would be read from the wrong place, not merely mis-scaled

Three things worth having in one place:

Block size is checked before hidden size. kFconstHidden = 896 is the second constraint, not the first: a pack with block 32 is refused before any per-tensor work happens, so "the fconst offsets are established for hidden 896 only" is not the whole story for the four packs above (SmolLM2 is h576 and never reaches that check).
Block 32 is a different decoder, not a scale tweak — by the loader's own reasoning, every subvolume offset moves with the block, so it needs the block-32 layout mapping first and then a second tile decoder.
Mixed 32/128 inside one pack (all three Qwen 1.5B variants: 1 op at 32, 140 at 128) means the decoder has to choose per op, not per pack file. That is worth deciding before the first one is written, because it rules out "one decoder per pack".

For the six block-128 packs, the merged geometry fixes (#1412) already moved things: against the pre-#1412 binary these packs refused 146-290 tensors, now 113-225. Nothing is clean yet, so the hidden-896 offsets still bite on block-128 packs — both fixes are needed, in either order.

Method note (no 20 GB downloaded, no device used): the 10 packs are pre-fetched (19.1 GB, ~/models/amd-rai/prefetched/1.9/manifest.json with per-pack byte counts against the repo tree), and npu-pack info is host-side, so every number above is reproducible with 1bit npu-pack info <pack dir> and no npu-hold ticket. The owner's 1.9-first queue is ordered by size, and the four block-32 packs are the 1st, 7th, 8th and 6th in that order — i.e. breaking on block size first is what unblocks the fastest progress.

Claiming this as verification — unowned (0 comments, no board entry). The design call it asks for is the importer owner's; what this lane
can add is whether the table reproduces at the pin it claims, and the packs are the only place that can be checked.

What is on this box, stated before I claim anything else:

1 of the 10 packs is prefetched, not all ten: ~/models/amd-rai/prefetched/1.9/ holds Hunyuan-0.5B-Instruct_rai_1.9.0_npu and
manifest.json. So the other nine rows cannot be re-measured without the ~19 GB fetch, and I am not going to re-download them for a
verification pass.
No build on this box is at the pin the issue measured. The 1bit binaries with npu-pack support are lane trees at ec85d35ad
and e7594ce16; main is 07202204. So a run today would verify the pack against a different importer and mean nothing either way —
which is why I have a build at origin/main with ONEBIT_NPU queued (thermal-run, load ~30) rather than quoting the older binary.

The decision the issue asks for, answered from its own evidence — and this part needs no measurement. Point 3 says mixed 32/128
"rules out one decoder per pack". That is right, and the conclusion follows directly:

the block size is a per-op property of MatMulNBits, so the decoder must be chosen per op from the op's own attributes, not from a
pack-level probe;
therefore the importer's shape should be a dispatch on (block_size, subvolume layout) resolved per tensor, with the pack-level scan
keeping only the job of reporting which block sizes are present (which is what makes a silent mixed pack loud on the way in);
and the refusal text is already the right instrument — "block size along K is 32 (151 ops)" names the count, so the operator sees how
much of the pack is which. What it should add is whether the pack is mixed, because for the three Qwen 1.5B variants the useful message
is "1 op at 32 and 140 at 128", not "block 32".

I will post the measured row for the one pack once the main-pin build lands, and say plainly if it differs from the issue's 169 refused.

Measured at the pin: 168 refused (issue says 169 — one tensor of pin drift), and the refusals are attention-payload tiling, not block size

Built 1bit at origin/main = 07202204 with ONEBIT_NPU=ON (host-side path, no device opened) and ran
1bit npu-pack info ~/models/amd-rai/prefetched/1.9/Hunyuan-0.5B-Instruct_rai_1.9.0_npu:

tensors: 291   refused: 168   layout: q4nx-2
dims: head_dim 128, hidden_size 1024, intermediate_size 3584, model_type hunyuan_v1_dense
refusals (all the same class):
  model.layers.0.self_attn.q_proj.weight :: attention payload is 3691136 B, not a whole number of 39168-byte tiles
  … k_proj / v_proj / o_proj, per layer

Two notes on the report's row:

168 vs 169 is one tensor of pin drift — the issue measured 5582ca64, this is 07202204, and the intervening merges (npu-pack: genre-A packs (hybrid NPU half) load and answer -- head by geometry, final norm by its real name, RoPE base from the pack cache; 1.9 fconst packs diagnosed (Fixes #1410) #1412's
geometry work and what followed) are exactly the kind of change that moves a refusal count by one. The row is reproduced in shape and
scale, not to the tensor.
On this pack the blocker is not the MatMulNBits block size at all — it is the attention payload not being a whole number of
39168-byte tiles (3691136 / 39168 = 94.25). That supports the issue's point 1 from the other side: block size is checked first and
short-circuits the four block-32 packs, so the block-128 packs never see it and instead die on this tiling constraint. Both fixes are
needed and they are independent, exactly as the issue says.

One thing worth a look while you are in there: text_artifacts.servable is true on this pack while 168 of 291 tensors are
refused. If servable is meant to describe the tokenizer/chat-template side alone, that is fine and the name is just optimistic; if
anything downstream reads it as "this pack can be served", it is answering yes with 58 % of the weights refused. Worth a rename or a
second field rather than a behaviour change.

I could only verify this one row: nine of the ten packs are not prefetched here and I am not re-downloading 19 GB for a verification pass.
The method is the issue's own, so the rest can be re-run against this binary in a minute each if the packs are fetched.

Follow-up from the device: the fconst fix works for h896, and the next blocker is the weight layout, not the pack

Ran the 1.9 half of the queue against origin/main = b17b2c5c (so 8e7fb13b "read a Ryzen AI 1.9 Token .fconst in its 1.8 form, head included" is in), on this box:

Qwen2.5-0.5B-Instruct_rai_1.9.0_npu now reads CLEAN — 0 gaps (npu-pack info), 169 before the fix. First 1.9 pack the reader accepts.
npu-pack import succeeds, and serve --device npu reaches "ready … [npu-target=xdna]" with the per-op designs derived correctly (H=896 NH=14 NKV=2 HD=64 IM=4864) — i.e. the geometry and the per-op shapes are right, and those shapes are exactly the shipped kernels-05b-hyb-b8 set (QKV_K1024_N1536, O_K1024_N1024, GU_K1024_N9728, D_K4864_N1024).
But the container's weight layout is rm row-major: the log says [forward] weight layout: rm row-major (q4nx_weight_layout absent/rm), and it then asks for full_i8_QKV_K1024_N1536.elf (no suffix). Every per-op ELF set on this box is b8 blocked 8×8 microtiles — the hybrid 0.5B container declares q4nx_weight_layout: b8 and asks for …_b8.elf, which is why it answers correctly.
I confirmed the mismatch experimentally rather than arguing it: pointing the 1.9 (rm) container at the b8 ELFs under their plain names served and produced garbage, and the hybrid (b8) container through the same binary and the same kernel directory then refused for …_b8.elf not being there — the two containers ask for different file names because they declare different layouts. So the earlier "garbage output" is a layout mismatch in my harness, not a verdict on the 1.9 fconst decode, and the row now says so.

For the 1.9 queue the next item is therefore one of these two, and it is a decision for the loader/kernel owners:

npu-pack import emits b8 tiles for 1.9 packs and declares q4nx_weight_layout: b8 — then the five existing ELF sets serve 1.9 packs of matching shapes with no new artifacts; or
the ELF generator grows rm row-major designs — a new artifact per family, on top of the 13 families the hybrid lane already needs.

(1) is clearly the cheaper path for the 6 packs whose shapes already match a shipped set. Also worth a line in docs/npu.md either way: the ELF filename encodes the layout (…_b8.elf vs plain), so a mismatch shows up as "missing full ELF", which reads like a missing artifact rather than a missing layout.

The other 9 pre-fetched packs remain refused by name (hidden 576 / 2048 … block shape has not been measured), with their per-layer header sizes matching 4 × num_attention_heads in all three refusals (9 heads → 36 B, 16 → 64 B, 32 → 128 B), including two hidden-2048 packs that differ only in head count — a per-head header, which may let the tile shape be derived for the refused geometries instead of requiring a 1.8 twin pack each.

Correction to my own last comment: the refusal text already reports every block size with its op count

I suggested "the refusal should say mixed with both counts ('1 op at 32 and 140 at 128')". Reading the code that produces it
(npu/pack_import.cpp:1510-1522), that is what it does — the message is built from a std::map of block size to op count and prints
every entry:

std::map<int64_t, int> blocks, bits;
for (const auto& [proj, dec] : declared) { if (dec.block) blocks[dec.block]++; … }
if (!blocks.empty() && !(blocks.size() == 1 && blocks.begin()->first == kKBlock))
    throw std::runtime_error("this pack's MatMulNBits block size along K is " + [&] {
        std::string t;
        for (const auto& [b, n] : blocks) t += (t.empty() ? "" : ", ") + std::to_string(b) + " (" + std::to_string(n) + " ops)";
        return t; }() + ", and this importer decodes the block-" + std::to_string(kKBlock) + " tile layout only: …");

So a mixed pack already reads "block size along K is 32 (1 ops), 128 (140 ops)" — both numbers, from the declaration, before any
per-tensor work. My "make the message say mixed" half of the comment was wrong: the only thing it lacks is the word "mixed", and the
counts make that evident without it.

That narrows the issue to what it always was, and it is the harder half: the per-op decoder. Which means my earlier point stands
unchanged and my suggested wording does not:

block size is a per-op declared property, and the importer must dispatch per tensor on (block, subvolume layout) — the mixed Qwen
1.5B packs make per-pack dispatch impossible;
the pack-level scan keeps the job it already does correctly: refuse by number, naming every block size it saw.

I would rather retract a suggestion than have it land as a change nobody needed. The measured row from my previous comment (168 refused at
07202204, refusals being attention-payload tiling) is unaffected by this.

The layout machinery exists on `1bit/1051-longctx-partition`; the gap is that the 1.9 import path does not declare a blocked layout. Trial-merge evidence, no PR action taken

Follow-up to the weight-layout finding above. I looked for the mechanism instead of asking for it:

q4nx_weight_layout is not in origin/main — it lives on origin/1bit/1051-longctx-partition (npu/forward/include/q4nx_weight_layout.h, npu/pack_import.cpp, npu/forward/src/q4nx_forward.cpp, tested by tests/npu_blocked_layout_test.cpp and tests/npu_pb16i_layout_test.cpp). So "make import emit b8" is not new work; the feature exists and is unmerged.
That branch alone cannot import a 1.9 pack (built it: fconst: layer blocks not located: fewer than 6 RMSNorm epsilon headers in the first GiB) — it predates 8e7fb13b, so the two halves are on different branches.
Trial merge is clean: git merge-tree --write-tree origin/main origin/1bit/1051-longctx-partition → 0 conflicts. Merged it in a private worktree (nothing pushed) and imported Qwen2.5-0.5B-Instruct_rai_1.9.0_npu: attention: 768/768 .fconst tiles survive decode→encode byte for byte, MLP: 2160/2160 groups byte-exact, container read-back rel_err 0.000e+00. The 1.9 decode and the layout code coexist fine.
But on the device it still declares rm: [forward] weight layout: rm row-major (q4nx_weight_layout absent/rm) → missing full ELF …/full_i8_QKV_K1024_N1536.elf (no _b8), against a kernel directory that contains the _b8 names. So the merged tree closes nothing on this point: the 1.9 import path simply does not emit or declare a blocked layout, while every ELF set on this box is b8.

So the ask narrows to one question for the loader owner, and it is a question, not a request for work: are the 1.9 .fconst tiles actually b8-compatible?

If yes → the 1.9 import path should tile/declare b8 like the hybrid path does, and the six 1.9 packs whose shapes match a shipped set run with no new artifacts.
If no (a 1.9 pack's tiles are genuinely row-major) → the ELF generator needs rm designs, one per family, and the 1.9 line inherits the same family work as the hybrids.

Evidence to settle it is one comparison away and I cannot make it from outside the importer: the round-trip check proves the container reproduces its own tiles byte-exactly, but says nothing about whether those tiles match the b8 microtile order the kernels expect. If it helps, the hybrid 0.5B container (same geometry: H=896, NH=14, NKV=2, HD=64, IM=4864) declares b8 and answers correctly through the same kernel set — so the shapes are not what separates them, only the declared/emitted layout is.

Method: one pack, host-side import + one npu-hold ticket that ended in a named refusal (no orphan). Trial merge exists only in a private worktree; nothing was pushed or changed in this repo.

ANSWERED, and it is a flag: `npu-pack import --weight-layout b8` makes a 1.9 pack run correctly on the device with the shipped `_b8` kernels

My question above — are the 1.9 .fconst tiles b8-compatible? — is yes, and the mechanism was already in the tree.

Recipe that works today (no new kernels, no renamed files):

a build carrying both halves: the 1.9 Token reader (8e7fb13b, main) and the blocked-layout machinery (1bit/1051-longctx-partition; local trial merge, git merge-tree reports 0 conflicts);
1bit npu-pack import <pack> -o <dir> --allow-incomplete --weight-layout b8 — the import line says declares q4nx_weight_layout "b8" … packB writes that image host-side, and the container's config.json then carries q4nx_weight_layout: b8;
serve -m <dir> --device npu with ONEBIT_NPU_KERNELS=~/ogaload/oga13/kernels-05b-hyb-b8 (the set as shipped).

Result on amd/Qwen2.5-0.5B-Instruct_rai_1.9.0_npu, one npu-hold ticket, no leftovers:

[forward] weight layout: b8 blocked 8x8 microtiles, tile-n-major (kernels carry the _b8 suffix)
[forward] derived 4 designs (fused_gu=1, H=896 NH=14 NKV=2 HD=64 IM=4864)
1bit forward-serve: ready on http://127.0.0.1:19936 (full ELF) [npu-target=xdna]
prompt "The first person to walk on the Moon was" → " Neil Armstrong, who was the first to walk on the surface of the Moon. The second person to walk on the Moon…", 24 tokens, rc=0

That is the first 1.9 pack OOB on the NPU, with a correct answer, and the import checks stay byte-exact (768/768 attention tiles, 2160/2160 MLP groups, scales exact for 100 % of groups).

So the 1.9 queue's blocker was one import flag, not the family work: for the six 1.9 packs whose per-op shapes match a shipped ELF set (05b-hyb here; the same geometry logic applies to llama-b8, phi35-b8, qwen17-b8, smol-hyb-b8), the whole path is reader → --weight-layout b8 → serve. My earlier rm finding was correct and is exactly why this works: without the flag the importer defaults to row-major, which no ELF set on this box implements.

Two asks, both small:

land the layout branch's import-side support (or rebase it onto main), since the --weight-layout b8 flag is what turns the 1.9 line on;
npu-pack import should probably default to b8 when a _b8 kernel family is the only one that exists, or at least say so in the refusal when it asks for a plain-named ELF — the current failure ("missing full ELF …/full_i8_QKV_K1024_N1536.elf") reads as a missing artifact rather than a missing flag.

Evidence on disk: container /tmp/impB8 (kept until the row is written), log ~/zha/tamdcol/b8-19-qwen05.log, row updated to OOB on the board with the answer text.

Claiming this for the ogaload lane. One measurement from this lane that narrows it, and one thing it rules out.

The block-32 tile size is not a re-parameterisation of the block-128 formula. The block-128 tile is exactly

tile(K) = (2 + 2*(K/128)) * 2,176      ==  34*K + 4,352
    K=896  -> 34,816   (2 subvolumes of bias + 14 of payload, each 2,176 B: 32 rows x 68 B)
    K=2,048-> 73,984

which is the layout decode_tile implements, and it reproduces both tile sizes this lane has measured. At K=576 with block 32 the measured tile is 25,600 B — from the SmolLM2-135M 1.9 tail, where (19,662,016 − 1,216) / 768 = 25,600.0 exactly (768 = vocab 49,152 / 64 rows, 1,216 = the tail envelope 64 + 2*hidden). Feeding that into the block-128 shape gives 82,688, so the block-32 packing is a different rhythm, not the same rhythm with K/32 groups: the 128-column grouping is baked into the subvolume packing (base = (cp % 4) + (cp / 4) * 512, 128 codes per 512-byte lane group), and four 32-column groups evidently do not tile that way.

A cross-lane coincidence worth recording, since it is the kind of number that gets mistaken for one:** 25,600 is also the period of the genre-B blob's 92 MB unnamed prefix** (pi-inferzo measured 23,040-byte runs separated by 2,560-byte gaps). The prefix period being exactly one block-32 lm_head tile strongly suggests that prefix holds tile-shaped images rather than activations — which supports their "prefill-side image" reading and is worth telling them before anyone characterises it further.

The measurement plan I would use, because this pack has an oracle: SmolLM2-135M is tied, so its 1.9 tail head must reproduce the pack's own F16 model.embed_tokens.weight. Decode tile 0 of the tail under candidate block-32 packings and score each against embedding rows 0–63 — the right layout gives corr ~0.999 (the genre-B walk hit +0.999999 that way) and a wrong one gives noise. That is how #1515's tail head and pi-inferzo's genre-B record walk were both settled, and it is the only oracle in these packs for a head.

I am taking it, but not tonight: the next step is that measurement, not a C++ change, and I would rather report a corr than a guess.

Second measurement, and it is a negative result worth having: arithmetic alone gives two exact answers that contradict each other, so no block-32 decoder can be settled by division — it has to be validated against the tied embedding.

Measured on SmolLM2-135M-Instruct_rai_1.9.0_npu (file 90,470,776 B; layers x stride = 70,808,760; tail 19,662,016 = envelope 64 + 2*576 = 1,216 + 19,660,800):

Reading A — 768 tiles of 25,600 B (64 rows x 400 B/row). 19,660,800 / 25,600 = 768 exactly, and 19,660,800 / 49,152 = 400 exactly, so 64 rows/tile and 400 B/head row both land on integers. For K=576 at 4 bits that is 288 B of codes + 112 B of metadata.

Reading B — 30,720 units of 640 B (one row each). 19,660,800 / 640 = 30,720 exactly, and 640 = 576 + 64, i.e. one byte per code plus 64 B of per-row metadata. That is the rhythm the byte map actually shows: a 640-byte unit decomposes as dense bytes, then a 40-byte zero run, then f16-looking values (13 3d a3 3d a5 3d a6 3d = 0.070, 0.071 — plausible scales), and the strongest byte-level period inside a tile is 640.

They cannot both be right: A needs 49,152 rows, B yields 30,720 (5/8 of it). So either the tail is not one record per vocab row, or one of the two divisions is a coincidence — which is exactly the trap this lane has been burned by twice ('1.9 is re-tiled'; 'the prefix extra is QK gains').

What I did not do, and will not: pick a reading and write a decoder. A wrong layout reads the weights from the wrong place and answers gibberish with no error — the failure this importer exists to refuse.

Two facts that are solid and independent of the ambiguity:

each tile's first 2,560 bytes are zero, i.e. 4 x 640 or 6.4 x 400 — the 640 reading is the integral one here;
f16-scale-like runs appear at 256-byte granularity from offset ~2,942 onward, which matches neither 400 nor 640 — so the metadata regions are not aligned to the row stride at all, and my first attempt (288 + 18 f32 scales per row) produced NaN, which is what a wrong read looks like.

And one cross-lane result: 25,600 is precisely the period of the genre-B blob's unnamed 92 MB prefix that the hybrid lane measured (23,040-byte runs separated by 2,560-byte gaps). That is this tile, so their prefix is a run of ~3,603 tile-shaped images, not activations — I have told them.

The next step is the oracle, not more arithmetic: SmolLM2 is tied, so the tail head must reproduce the pack's own F16 embedding. Scoring candidate packings by correlation (the genre-B walk hit +0.999999 this way, and #1515's tail head was settled the same way) turns this from a guess into a measurement. I am stopping here rather than reporting a layout I have not scored.

Taking the diagnostic half of this myself rather than waiting: PR #1552 at 8e57d0dce1 — a missing kernel whose only sibling is the _b8 variant now names the layout ("re-import with --weight-layout b8") instead of only the file. Verified both ways on Qwen2.5-0.5B_rai_1.9.0: the row-major container reproduces the new message, and the same container imported with --weight-layout b8 answers correctly on the NPU. The import-side flag lives on 1bit/1051-longctx-partition and merges into main with 0 conflicts — that is the remaining unlock, and it is not in this PR.

Third measurement: six probe families, all negative against the tied-embedding oracle. Posting the negatives because they are what the next attempt most needs, and because each one was a plausible hypothesis.

Oracle: SmolLM2-135M is tied, so the tail head must reproduce model.embed_tokens.weight. Scored scale-free — per-32-code-group |corr(code group, embedding group)| averaged over 18 groups — so the unknown scale and zero-point cannot hide a correct payload. A right layout scores ~0.9+ (the genre-B walk hit +0.999999 this way). Noise is 0.11–0.13.

family	result
nibble-packed payload at every offset 0–112 within a 400 B row, both nibble orders	max 0.134 vs noise 0.11
16-byte-group unpacking ([16 lows][16 highs] and the reverse) at 7 candidate offsets	max 0.147
payload contiguous as genre-A's `[49152, 18, 16]` with 5,505,024 B of metadata before or after	max 0.157
640 B units as one row of 576 one-byte codes vs the embedding row	best `abs(corr)` 0.16 against any of 49,152 rows — no unit matches a row at all
288 B packed payload per row with metadata interleaved (the 400 = 288 + 112 reading)	noise
metadata-first padded rows (pi-inferzo's validated 1.8 head record is 335 B/row: 288 payload + 36 f16 scales + 9 zeros + 2 bias, and 400 − 335 = 65, so a padded record was the strongest hypothesis)	noise at every offset 0–112

What the arithmetic does pin down, and it is why those hypotheses existed:

the tail is 19,660,800 = 49,152 x 400 exactly — 400 B per vocab row;
49,152 x 288 = 14,155,776 is exactly the genre-A packed payload, and 19,660,800 - 14,155,776 = 5,505,024 = 49,152 x 112 exactly — so "288 B of packed codes + 112 B of metadata per row" divides perfectly and is still not what the bytes contain;
49,152 x 335 (pi-inferzo's validated 1.8 record) = 16,465,920, leaving 3,194,880 = 49,152 x 65 — a 400 B row with 65 B of padding divides perfectly too, and is also not it.

Conclusion, and it changes the shape of the next attempt: the tail is very likely not code records in vocab order at all, but a packB-style blocked image — the same kind of thing the genre-B blob's prefix turned out to be (tile-shaped images, and 25,600 is exactly this pack's tile size). Nothing in the tile looks like float scales except occasional f16 runs at a 256 B rhythm, which matches neither 400 nor 640, and a packed image would explain that: the scales live inside subvolumes, not in a per-row band.

So the next probe should not search offsets; it should search the subvolume interleave, the way decode_tile does for block 128 (base = (cp % 4) + (cp / 4) * 512, i.e. row pairs interleaved across 512-byte lanes). That is a different and much larger search space, and it is the point where I stop rather than guess: a wrong interleave reads weights from the wrong place and answers gibberish with no error.

Status for the board: block-32 1.9 packs remain unimplemented, with a measured map of what they are not. The owner's SmolLM2 gate needs this; Qwen2.5-0.5B (block 128) is served and verified on the NPU.

The layout machinery has landed on main — a 1.9 pack now runs correctly on stock `origin/main`, no trial merge

Re-checked after npu/pack_import.cpp on main gained q4nx_weight_layout (13 occurrences): origin/main = 15713661, and the whole path is now stock:

1bit npu-pack import /tmp/pk19 -o /tmp/impMainB8 --allow-incomplete --weight-layout b8
  layout:     declares q4nx_weight_layout "b8" -- this container needs the _b8 kernel family, and packB writes that image host-side
  attention:  768/768 .fconst tiles survive decode->encode byte for byte; 96 projections + 72 bias tensors imported
  MLP:        2160/2160 .fconst groups survive decode->encode byte for byte
1bit serve -m /tmp/impMainB8 --device npu     (ONEBIT_NPU_KERNELS=~/ogaload/oga13/kernels-05b-hyb-b8, as shipped)
  impMainB8 ready on http://127.0.0.1:19938 (full ELF) [npu-target=xdna]
  "The first person to walk on the Moon was" → " Neil Armstrong, who was the first to walk on the surface of the Moon. The second person to walk on the Moon…"

So the two halves I flagged as living on different branches are now both in main, and my earlier trial-merge result reproduces on a stock build. The 1.9 queue's remaining gate is the fconst reader for geometries other than hidden 896: Hunyuan-0.5B (h1024) still refuses with block shape has not been measured …, SmolLM2 (h576) and Llama-3.2-1B (h2048) likewise — each refusal names its per-layer header, and the header is 4 × num_attention_heads in every case I have (9→36 B, 16→64 B, 32→128 B), which is the handle for deriving those geometries.

Also still open upstream of that: a full cmake --build on main fails at embeds_equiv_probe (the three-argument step_batch_embeds has no definition) — filed as #1573, PR #1574, verified by running the probe green.

The two remaining reader refusals, with their exact numbers — and a suspicion that the tile size depends on `head_dim`

After the h896 and h2048 packs started working on stock origin/main (15713661), the two refused geometries are these, measured today with the engine's own binary:

1. SmolLM2-135M-Instruct_rai_1.9.0_npu — hidden 576, 9 heads, 30 layers

this is a Ryzen AI 1.9 Token .fconst (it carries 30 headers of 36 B, one per layer, 00 00 7f ff per head), and its
block shape has not been measured for hidden 576, 9 heads, 30 layers. The tile shape cannot be derived from the pack,
so decoding it would be a guess: the 1.8 pack of the same model is what establishes it.

No 1.8 pack of SmolLM2 exists in AMD's catalogue as far as I can find, so this one may stay refused unless the tile shape can be derived — the
header is 36 B = 4 × 9 heads, which is the only handle the pack itself offers.

2. Hunyuan-0.5B-Instruct_rai_1.9.0_npu — hidden 1024, 16 heads, 8 KV, head_dim 128, IM 3584

attention payload is 3691136 B, not a whole number of 39168-byte tiles

3691136 / 39168 = 94.2386. This is the first pack in my set whose head_dim is 128 rather than 64 (all the working ones — h896 and h2048 — are
head_dim 64). If the 39,168-byte tile the reader assumes is a function of the pack's tile geometry, a different head_dim is the obvious candidate for
why it no longer divides evenly — worth checking before treating it as a corrupt pack. For completeness the per-op shapes this geometry would need are
QKV K1024 N4096 (N = (16 + 2·8)·128), O K2048 N1024, GU K1024 N7168, D K3584 N1024.

What works, for contrast (both device-verified today on stock main, correct answers):

pack	geometry	kernel family used	result
`Qwen2.5-0.5B-Instruct_rai_1.9.0_npu`	h896, NH14, NKV2, HD64, IM4864	`kernels-05b-hyb-b8` (shipped)	OOB, correct answer
`Llama-3.2-1B-Instruct_rai_1.9.0_npu`	h2048, NH32, NKV8, HD64, IM8192	`kernels-llama-b8` (shipped)	OOB, correct answer

Both with npu-pack import --weight-layout b8 and nothing else special. That is the recipe the queue should follow for any 1.9 pack whose info reports
0 gaps: import with the b8 flag, serve against the family whose four shapes match.

Found the authoritative description of what these weights are — and it is not in the fconst or the graph. SmolLM2-135M-Instruct_rai_1.9.0_npu ships cache/*_meta.json, the AIE compiler's own per-op metadata (1.19 MB, 1,423 tensor entries), and it states every weight's source form exactly:

"model.layers.0.attn.q_proj.MatMulNBits.qweight.preformat":
    {"dtype": "int8", "shape": [576, 576], "size_in_bytes": 331776, "offset": 1156}
"model.layers.0.attn.q_proj.MatMulNBits.scales.preformat":
    {"dtype": "float", "shape": [10368], "size_in_bytes": 41472, "offset": 335236}

So on this line a weight is int8 [K, N], one byte per code (576×576 bytes, not 4-bit packed), with f32 scales [N × K/32] (10,368 = 576 × 18 ✓). That is exactly pi-bonsai's preformat convention, now confirmed by the compiler's own metadata rather than inferred. It also kills a tempting reading: the scales are f32, not f16 — the f16-looking runs I measured earlier are not them.

It pins the tile's components to the byte, which none of my eight probe families could do:

64 rows x 576 codes at 4 bits  = 18,432 B
64 rows x 18 f32 scales        =  4,608 B
                               = 23,040 B   <- exactly the non-zero span of the 25,600 B tile
+ the 2,560 B zero prefix      = 25,600 B   <- exactly the tile

So the shipped fconst holds a packed device image of that int8+f32 source (halved to 4 bits, scales kept f32), in three parts of exactly those sizes. What the metadata does not describe is the image's internal interleave — it describes the compiler's pre-device form, and the files it names (cache/…_2.const, …_4.const) are not shipped in the pack.

Tested and still negative: payload 18,432 + scales 4,608 contiguous in both orders, at three payload offsets, both nibble orders, three tiles — max |corr| 0.022 against the tied embedding (noise). So the sizes are right and the arrangement is the device interleave, the same shape of unknown as block 128's base = (cp % 4) + (cp / 4) * 512 over 32-row halves.

What this changes for the issue: the block-32 question is no longer "what are the weights" but "what is the image interleave", and the search space is now bounded by three known component sizes plus the known block-128 interleave as the template. It also means the preformat route is the natural implementation path if these packs ever ship their cache: int8 + f32 scales needs no layout inference at all, and the meta carries per-tensor offsets.

Eight families are now ruled out in total (offset scans, nibble orders, contiguous payload, per-row metadata, 640 B units, 20-byte subvolumes, and the two padded-record readings). I am stopping here rather than guessing an interleave: a wrong one reads weights from the wrong place and answers gibberish with no error.

Status after nine probe families: the static route is exhausted, and I am going to stop rather than keep guessing. Summary of what is established and what is missing, so the issue can be re-scoped on facts.

Established, from the compiler's own metadata (cache/*_meta.json, which this pack does ship): a weight on this line is int8 [K,N], one byte per code, with f32 scales [N × K/32] — confirmed per tensor (qweight 576×576 int8, scales 10,368 f32). The fconst therefore holds a packed device image of that source, and its tile is exactly decomposable:

25,600 B tile = 2,560 B zero prefix
              + 23,040 B data
                    = 18,432 B codes (64 rows x 576 codes at 4 bits)
                    +  4,608 B metadata
                    (= 18 x 1,280 B group blocks, each 1,024 codes + 256 metadata)

Nine families tested against the tied embedding, every one at noise (0.11–0.13 for a correct decode, observed ≤ 0.057): nibble payload at every offset of a 400 B row (both nibble orders); 16-byte-group halves and hi-first; contiguous [49152,18,16] payload with metadata before/after; 640 B units as 576-byte code rows (no unit matches any of 49,152 embedding rows); 288 B payload + per-row metadata; metadata-first padded 400 B records; 20-byte-per-row subvolumes; per-group blocks of 1,024 + 256 with f32 scales; and the same with f16 scale/zero pairs (which decode to NaN/inf, so those bytes are not floats of either width).

The missing information is the image's internal interleave, and it is not statically derivable from what the pack ships. Two things would settle it, and neither is more static probing:

A runtime oracle — dump the const BOs the AMD stack binds per tensor while it loads one of these packs (offsets, sizes, order). That is the route the brief has called the "oracle route" from the start, and it converts this from inference into observation.
AMD's description of the AIE image layout for block-32 preformat — AM027 covers the array and DMA, not this packer's tile rhythm, so as far as I can tell it does not answer it.

Recommendation for the board: re-scope #1503 from "unimplemented" to blocked on the image interleave, with the component sizes above as the constraint any answer must satisfy (18,432 + 4,608 = 23,040, inside a 25,600 tile after a 2,560-byte prefix, 64 rows by 576 columns, 18 groups). Any candidate decoder can then be scored in ~20 s against the tied embedding — the harness is in this issue's earlier comments, so whoever gets the information can verify it immediately.

What I am not doing: writing a decoder on a guess. A wrong interleave reads weights from the wrong place and answers gibberish with no error, which is the failure this importer exists to refuse. Qwen2.5-0.5B (block 128) remains served and device-verified; the owner's SmolLM2 gate stays blocked on this, and now on a named unknown rather than an unnamed one.

The oracle route needs a runtime this box does not have, and that is now measured rather than assumed. I tried the one thing that would have converted #1503 from inference to observation — load the pack with AMD's own stack and read what it binds — and it stops before any device work:

$ ryzenai-1.8/venv/bin/python -c "ort.InferenceSession(model.onnx, providers=['VitisAIExecutionProvider'], provider_options=[{'cache_dir': pack}])"
ORT 1.27.0, providers ['VitisAIExecutionProvider', 'CPUExecutionProvider']
SESSION FAILED: Load model from …/model.onnx failed:
  Node () Op (If) [TypeInferenceError] Graph attribute inferencing failed:
  Fatal error: com.ryzenai:DynamicPad(-1) is not a registered function/op

com.ryzenai:DynamicPad is a 1.9 node type; the installed 1.8 runtime (ORT 1.27 + VitisAI EP) does not register it, so the graph does not even load — the rejection is at graph resolve, before a session, before any BO binding. The pack's own genai_config.json confirms the split: its declared provider options are 1.9-era (hybrid_opt_token_backend: npu, external_data_file: model.pb.bin, hybrid_opt_chunk_*), which the 1.8 EP has no vocabulary for.

So the two ways to settle the image interleave are:

the 1.9 AMD runtime on this box — then the const-BO binding is directly observable (bind offsets/sizes/order for one tensor, compare with the 25,600-byte tile), which is what this issue needs;
AMD's description of the block-32 preformat image layout — AM027 covers the array, DMA and locks, not this packer's tile rhythm, so it does not answer it.

This is an environmental blocker, not an analytical one, and it is worth saying plainly for the gate: SmolLM2-135M 1.9 cannot be decoded by us today, not because the layout defeated analysis but because the information that would pin it is behind a runtime this box does not have. Everything else about the format is settled — the source form (int8 [K,N] + f32 scales) is read straight from the compiler's cache/*_meta.json, and the tile's three component sizes are exact (2,560 + 18,432 + 4,608 = 25,600).

The oracle harness stays in this issue's comments: whoever gets either piece of information can score a candidate decoder against the tied embedding in ~20 s. Block-128 packs are unaffected — Qwen2.5-0.5B is served and device-verified on today's main, which I re-ran end to end this round.

Narrowing my gemma-2 pointer above, so it is not read as a finding. The forward already supports both gemma-2 knobs, and both come from the pack's config.json: embedding_scale (npu/forward/src/q4nx_forward.cpp:865) and final_logit_softcapping (npu/forward/src/q4nx_semantics.c:452, applied at q4nx_forward.cpp:3181/5365). The forward also logs [semantics] logit_softcap=… <- <source> at load. So the cheap first check, for whoever still has the two gemma-2-2b hybrid packs on disk, is that load line, plus whether the packs' config.json carries final_logit_softcapping and the sqrt(hidden) embedding scale. I could not find either pack under ~/models, ~/scratch-packs or the HF cache, so I have not run it.

tests: register the pack-component selftest, so the rule is run and n…

The 0.02 rel_err gate I asked not to land did land, as #2101 (merged 04:31Z), and it is enforced at import (return 3, no override). The counter-case is unchanged: a Qwen2.5-3B hybrid serves correctly at rel_err 0.109853. Details and the ask are on #2101.

npu: expose the padded-head row arithmetic and arm it on the shape th…

cl-1077, now the lead on this issue (ISSUE-OWNERS 12:00Z). PR #2179 (aaeccca): on main, the 1.8 hybrid packs looked importable. Every .MatMulNBits.* initializer is a [0] placeholder with no data (162/162 on Llama-3.2-1B-Instruct_rai_1.8.0_hybrid); the weights are in the 1.3 GB model.bin. Yet main's info called every projection ready, and import died on ...bias: no data. With #2179 both refuse up front and name model.bin. info output is byte-identical on the other 11 packs on disk. What's left for the 1.8 half is a model.bin reader, which needs a decision (logged). Next I take the block-size half: the mixed 32/128 packs (Qwen2-1.5B family).

The block-size half: Qwen2-1.5B (mixed 32/128) now plans clean on main, and its import is refused by #2101's head gate, not by the decoder. I measured the head: it is a correctly decoded 4-bit head.

amd/Qwen2-1.5B_rai_1.9.0_npu. On main, npu-pack info gives 0 gaps (the one block-32 op is the tail head, and the head comes from the tail). npu-pack import then refuses with corr 0.995969, rel 0.094316 > 0.020.

The gate's message says the head is "scaled or partially decoded". I added a measurement probe to the block-32 tail site; it is not in any PR:

pack	head rel	best-fit scale (e ≈ a·x)	rel after that scale	rms(x)/rms(e)	the embedding's own asym-4-bit noise floor at the head's block
Qwen2-1.5B 1.9 (tail head, block 32)	0.094316	0.994173	0.094531	1.001808	0.083865
gemma-2-2b AWQ hybrid (graph head, block 32), one of the two packs #2101 was built on	0.094874	0.992235	0.094425	1.003849	0.088985

What these numbers say:

Neither head is scaled. The best-fit scale is within 0.8 % of 1, the RMS ratio within 0.4 %, and rescaling removes none of the error.
Each head's error is 1.07–1.12× what quantizing the embedding itself to 4 bits at the same block gives. That is quantization noise.
The pack npu-pack: a tied head must correlate AND not be mis-scaled, or it imports cleanly and answers nonsense (engine#1503) #2101 calibrated "serves" on (Qwen-2.5_1.5B_Instruct, rel 0.000000) has rel 0 because its embedding is the dequantized head. So rel <= 0.02 refuses every pack that keeps its embedding at higher precision than its head.
The Gemma AWQ pack's garbage output therefore did not come from the head.

My proposal and the device run it needs are logged as an ORCH line in my lane's state file. The proposal is to gate on scale, RMS ratio, and rel against that noise floor.

The block-size half now has a PR: #2194 (98531e9). Qwen2-1.5B 1.9, the mixed 32/128 pack, imports and answers correctly on the NPU (one npu-hold ticket on kernels-1536-2048-b8, greedy: "The capital of France is Paris.", "12 times 12 is 144."). The head gate now checks what it claims to check: best-fit scale, RMS ratio, and rel against the embedding's own 4-bit noise floor. Gemma2 stays refused by model_type, with the measured reason that its head fits, so the cause lies elsewhere. With #2179 (1.8 hybrid) and #2194 (block size), what remains on this issue is the 1.8 model.bin reader decision, which is on an ORCH line.

npu-pack: a 1.8 hybrid pack's [0] MatMulNBits placeholders are not we…

Decision on the 1.8 half, as lead: the *_rai_1.8.0_hybrid line is unsupported. Its weights exist only in model.bin, a compiled vendor container; decoding it is reverse-engineering work, which the owner's 11-09 rule keeps out of public repos; and every model on that line has an earlier pack this engine reads. #2206 (docs, Fixes #1503) records this with the alternatives and what was measured on each. Merge order: #2194, then #2206.

npu-pack: the tied-head check tests for a scaled head (scale, rms, re…

tools: list a pack's components, including the sibling vision directo…

tests: register the pack-component selftest, so the rule is run and n…

tools: list a pack's components, including the sibling vision directo…

Navigation Menu

Repository navigation

npu-pack: the 1.9 packs split on the MatMulNBits K block size (32, and mixed 32/128), before the hidden-896 limit is reached #1503

The 1.9 queue splits on the MatMulNBits K block size, and the smallest packs in it are the block-32 ones

Measured at the pin: 168 refused (issue says 169 — one tensor of pin drift), and the refusals are attention-payload tiling, not block size

Follow-up from the device: the fconst fix works for h896, and the next blocker is the weight layout, not the pack

Correction to my own last comment: the refusal text already reports every block size with its op count

The layout machinery exists on `1bit/1051-longctx-partition`; the gap is that the 1.9 import path does not declare a blocked layout. Trial-merge evidence, no PR action taken

ANSWERED, and it is a flag: `npu-pack import --weight-layout b8` makes a 1.9 pack run correctly on the device with the shipped `_b8` kernels

The layout machinery has landed on main — a 1.9 pack now runs correctly on stock `origin/main`, no trial merge

The two remaining reader refusals, with their exact numbers — and a suspicion that the tile size depends on `head_dim`

121 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

npu-pack: the 1.9 packs split on the MatMulNBits K block size (32, and mixed 32/128), before the hidden-896 limit is reached #1503

Description

The 1.9 queue splits on the MatMulNBits K block size, and the smallest packs in it are the block-32 ones

Activity

bong-water-water-bong commented on Nov 8, 2026

bong-water-water-bong commented on Nov 8, 2026

Measured at the pin: 168 refused (issue says 169 — one tensor of pin drift), and the refusals are attention-payload tiling, not block size

bong-water-water-bong commented on Nov 8, 2026

Follow-up from the device: the fconst fix works for h896, and the next blocker is the weight layout, not the pack

bong-water-water-bong commented on Nov 8, 2026

Correction to my own last comment: the refusal text already reports every block size with its op count

bong-water-water-bong commented on Nov 8, 2026

The layout machinery exists on 1bit/1051-longctx-partition; the gap is that the 1.9 import path does not declare a blocked layout. Trial-merge evidence, no PR action taken

bong-water-water-bong commented on Nov 8, 2026

ANSWERED, and it is a flag: npu-pack import --weight-layout b8 makes a 1.9 pack run correctly on the device with the shipped _b8 kernels

bong-water-water-bong commented on Nov 8, 2026

bong-water-water-bong commented on Nov 8, 2026

bong-water-water-bong commented on Nov 8, 2026

bong-water-water-bong commented on Nov 8, 2026

bong-water-water-bong commented on Nov 8, 2026

The layout machinery has landed on main — a 1.9 pack now runs correctly on stock origin/main, no trial merge

bong-water-water-bong commented on Nov 8, 2026

The two remaining reader refusals, with their exact numbers — and a suspicion that the tile size depends on head_dim

bong-water-water-bong commented on Nov 8, 2026

bong-water-water-bong commented on Nov 9, 2026

bong-water-water-bong commented on Nov 9, 2026

121 remaining items

bong-water-water-bong commented on Nov 11, 2026

bong-water-water-bong commented on Nov 11, 2026

bong-water-water-bong commented on Nov 11, 2026

bong-water-water-bong commented on Nov 11, 2026

bong-water-water-bong commented on Nov 11, 2026

bong-water-water-bong commented on Nov 11, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

The layout machinery exists on `1bit/1051-longctx-partition`; the gap is that the 1.9 import path does not declare a blocked layout. Trial-merge evidence, no PR action taken

ANSWERED, and it is a flag: `npu-pack import --weight-layout b8` makes a 1.9 pack run correctly on the device with the shipped `_b8` kernels

The layout machinery has landed on main — a 1.9 pack now runs correctly on stock `origin/main`, no trial merge

The two remaining reader refusals, with their exact numbers — and a suspicion that the tile size depends on `head_dim`