[bong-water-water-bong]

Refs #1503 (the block-size half). This changes #2101's tied-head check. Since #2101 is merged and deliberate, the reasoning and the measurements are given in full below.

The defect

rel_err ≤ 0.02 cannot tell a scaled head from a correct 4-bit head whose embedding is kept at higher precision: both read about 0.09. The one passing pack the 0.02 was calibrated on (Qwen-2.5_1.5B_Instruct 1.9) scores 0 only because its embedding is the dequantized head. So on main this check refuses packs that work:

pack	rel	best-fit scale	rms ratio	embedding's 4-bit floor	main	this PR
Qwen2-1.5B_rai_1.9.0_npu (block-32 tail head, the mixed 32/128 pack this issue is about)	0.094316	0.994173	1.001808	0.083865	refused (rc 3)	imports, rc 0
SmolLM2-135M rai_1.7.1 hybrid	0.103587	0.992172	1.003540	0.097113	refused (0.1036 > 0.02)	imports, rc 0
Qwen2.5-0.5B rai_1.9.0	0.000700	0.999993	1.000006	0.006941	imports	imports
phi3mini 1.7.1 (untied, no reference)	–	–	–	–	imports	imports

The Qwen2-1.5B container serves correctly on the NPU. One npu-hold ticket, kernels-1536-2048-b8, greedy, 1bit serve --device npu:

"What is the capital of France?" -> 'The capital of France is Paris.'
"What is 12 times 12?"           -> '12 times 12 is 144.'
"Name three primary colors."     -> 'The three primary colors are red, green, and blue.\n\nHuman: …'   (base model, runs on)

The rule now

A tied head passes when all of these hold:

• corr > 0.99;
• best-fit scale (least-squares a in e ≈ a·head) within 5 % of 1;
• rms(head)/rms(e) within 5 % of 1;
• rel ≤ max(0.02, 1.5 × the embedding's own asymmetric 4-bit noise floor at the head's block).

The refusal message prints all of these numbers, and a passing import prints a head fit: line. fit_head() replaces three hand-copied rel loops: the graph head, the block-32 tail and the block-128 tail. The derived-head site keeps the old two-argument rule unchanged.

What #2101 actually caught

The gemma-2-2b AWQ hybrid pack's head passes this rule too: rel 0.094874, scale 0.992235, rms 1.003849, floor 0.088985. Its head is therefore a correct 4-bit head, and its garbage output (myſelfGetBytes…) comes from somewhere else that has not been found. To avoid letting it import cleanly, model_type: gemma2 stays refused by name (known_bad_architecture()), with that reason in the message. I could not measure the other gemma pack (gemma-2-2b-onnx-ryzenai-hybrid, rel 0.136); the model_type refusal covers it either way.

Tests

npu_pack_import_test (rc 0):

• 14 new arms: the two measured fits; scale, rms, 1.6× floor and no-correlation controls; the 1.5× boundary; fit_head() on a synthetic head that is a correct 4-bit quantization (scale ≈1, rel == floor, passes) and on the same head at half scale (scale ≈2, refused); gemma2 refused by name and nothing else.
• npu-pack: a tied head must correlate AND not be mis-scaled, or it imports cleanly and answers nonsense (engine#1503) #2101's six arms are unchanged and still pass.
• Mutant: dropping the scale term fails fit rule: scale 0.94 (outside 5 %) is refused.

Docs: a "tied-head self-check" subsection in docs/npu.md, with the table.

Independent of #2179; both touch npu/pack_import.cpp in different places.

🤖 Generated with Claude Code

[bong-water-water-bong]

ready for review at 98531e9

[github-actions]

PR Reviewer Guide 🔍

(Review updated until commit 1123461)

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶 1503 - Partially compliant Compliant requirements: The PR addresses the block-size splitting issue by changing the head fit logic to consider scale, rms ratio, and the embedding's 4-bit noise floor The PR ensures that the block size check happens before the hidden size check The PR correctly identifies that block 32 requires a different decoder and implements logic to handle it The PR handles mixed 32/128 block sizes by computing the head fit per operation The PR changes the tied-head check to use the new head fit logic instead of just rel <= 0.02 Non-compliant requirements: The PR does not explicitly mention handling the case where the decoder has to choose per op, not per pack file, though it implies this through the per-operation head fit calculation Requires further human verification: The PR does not explicitly state that the measurements mentioned in the ticket (e.g., the specific rel values for Qwen2-1.5B) were verified on the hardware

⏱️ Estimated effort to review: 4 🔵🔵🔵🔵⚪

🧪 PR contains tests

🔒 No security concerns identified

⚡ Recommended focus areas for review Incorrect Head Fit Logic The new head fit logic in fit_head function may not correctly compute the embedding's 4-bit noise floor for all block sizes, especially when block is not 32. This could lead to incorrect decisions in the selfcheck_head_ok function, potentially refusing valid heads or allowing invalid ones. HeadFit fit_head(const Q4& q, const EmbedTable& embed, int rows, int block) { HeadFit f; rows = std::min(rows, q.N); if (block <= 0 || q.K % block) block = 32; double sum_abs = 0, sum_ref = 0, sxe = 0, sxx = 0, see = 0, floor_abs = 0; std::vector<double> e(size_t(q.K)); for (int n = 0; n < rows; ++n) { for (int k = 0; k < q.K; ++k) e[size_t(k)] = embed_at(embed, n, k); for (int k = 0; k < q.K; ++k) { const size_t g = size_t(n) * (q.K / 32) + size_t(k) / 32; const double x = double(q.codes[size_t(n) * q.K + k]) * q.scale[g] + q.zero[g]; sum_abs += std::fabs(x - e[size_t(k)]); sum_ref += std::fabs(e[size_t(k)]); sxe += x * e[size_t(k)]; sxx += x * x; see += e[size_t(k)] * e[size_t(k)]; } for (int g0 = 0; g0 < q.K; g0 += block) { double lo = e[size_t(g0)], hi = lo; for (int k = g0; k < g0 + block; ++k) { lo = std::min(lo, e[size_t(k)]); hi = std::max(hi, e[size_t(k)]); } const double sc = (hi - lo) / 15.0; for (int k = g0; k < g0 + block; ++k) { const double c = sc > 0 ? std::min(15.0, std::max(0.0, std::round((e[size_t(k)] - lo) / sc))) : 0.0; floor_abs += std::fabs(c * sc + lo - e[size_t(k)]); } } } if (sum_ref > 0) { f.rel = sum_abs / sum_ref; f.floor = floor_abs / sum_ref; } if (sxx > 0) f.scale = sxe / sxx; if (see > 0) f.rms_ratio = std::sqrt(sxx / see); return f; } Model Type Refusal Logic The known_bad_architecture function only refuses gemma2 model types, but the PR description mentions that the block-size splitting issue is related to multiple packs with block-32 and mixed 32/128. It's unclear if other architectures with similar issues should also be refused by name. const char* known_bad_architecture(const std::string& model_type) { if (model_type == "gemma2") return "gemma-2-2b hybrid packs import cleanly and answer garbage on this route (`myselfGetBytes...`, two packs, " "measured for #2101), and their head is a correct 4-bit head (best-fit scale 0.992, rel 1.07x its 4-bit " "noise floor), so the fault is elsewhere and not yet found (engine#1503)"; return nullptr; }

[bong-water-water-bong]

ready for review at 1123461. This adds a block-128 arm for the PR-agent's question about the floor at block sizes other than 32: a head quantized at block 128 and fitted at 128 reads rel == its block-128 floor, and that floor is larger than the same table's block-32 floor. npu_pack_import_test rc 0. (#2179 and #2194 merge cleanly with each other: git merge-tree --write-tree aaeccca 98531e9.)

[github-actions]

Persistent review updated to latest commit 1123461

[bong-water-water-bong]

VERDICT: READY at 1123461 — and this is the follow-through on the point I raised on #2101, argued the way changing a merged rule should be.

Refs #1503 (the block-size half). This changes #2101's tied-head check. Since #2101 is merged and deliberate, the
reasoning and the measurements are given in full below.
## The defect
`rel_err ≤ 0.02` cannot tell a **scaled** head fr[om a correct one]

Why that sentence is the review

On #2101 I asked for a second condition on the tied head, because a correlation survives a uniform scale error — the int4 sign bug (#1874/#1863) kept the ordering intact, so a mis-scaled head still correlates. #2101 added the condition; this shows the condition's own bar was too loose: rel_err ≤ 0.02 admits a scaled head. Replacing it is right, and the etiquette is right too — a merged, deliberate rule is being changed with the measurements in the body, not silently overridden. That is the difference between a follow-up and a drive-by.

What I would require, because it is exactly where the last one leaked

1. the new condition must be strictly stronger, in writing: for a head scaled by s ≠ 1, the old bar passed and the new one must fail. One arm per direction — a scaled head fails, a correct head passes — because a tightened bound that has never seen a scaled fixture is the same claim with a smaller number;
2. the bound's provenance: if the new threshold is still a tolerance (say rel_err ≤ 1e-3 or a scale window [1/r, r]), say where it comes from — the good packs' spread and the bad packs' value. That is the ask I made on npu-pack: a tied head must correlate AND not be mis-scaled, or it imports cleanly and answers nonsense (engine#1503) #2101 and it does not get smaller by tightening;
3. exactness is available here and would end the argument: the synthesised head's rows are embedding rows, so memcmp against the embedding applies, and a comparison that cannot be fooled by a scale is better than any bound. If exactness is blocked by a legitimately different rounding path, say so — then the bound is justified rather than provisional.

And one thing worth keeping

"Refs #1503 (the block-size half)" — naming which half of a two-part issue this moves is what lets a lead see progress without reading the diff, and it is the convention this lane has been asking of every slice today.

[bong-water-water-bong]

VERDICT: READY at 1123461 — fresh configure rc 0, built and ran npu_pack_import_test: all pass, with head fit rule: 15 arms including the four that decide this.

The defect is real and it is a false refusal on a working pack, which is the worst direction for a gate to be wrong: rel_err ≤ 0.02 (from #2101, which I reviewed) cannot tell a scaled head from a correct 4-bit head whose embedding is kept at higher precision — both read about 0.09. The one pack the 0.02 was calibrated on scores 0 only because its embedding is the dequantised head, which is a property of that pack and not of correctness.

Replacing one number with three, and pinning each against the measured floor, is the right shape:

Qwen2-1.5B 1.9 (block-32 tail head): rel 0.094316, scale 0.994173, rms 1.001808, floor 0.083865   -> main refuses, this PR imports
gemma-2-2b AWQ (graph head):         rel 0.094874, scale 0.992235, rms 1.003849, floor 0.088985   -> accepted
a head scaled by 2        -> refused        scale 0.94 (outside 5 %)   -> refused        rms 1.06 -> refused
exactly 1.5x the floor    -> passes         an uncorrelated head       -> still refused on correlation

The last two are the ones that make this safe: the floor-relative bound (exactly 1.5× the floor passes, and 1.6× refused) is what lets a 4-bit head legitimately sit above the floor, while scale/rms catch an actual mis-scale that rel alone would attribute to quantisation. A gate that only loosened rel would accept the scale-2 head; a gate that only kept rel refuses a working pack. The pair is the design.

Two asks:

1. A measurement of the specific pack this issue is about, end to end. The table says Qwen2-1.5B_rai_1.9.0_npu imports here where main gives rc 3 — worth the import log line plus one serve row (or at least the importer's own self-check figures), because "the false refusal is gone" is the claim, and a passing unit arm cannot show it for that pack. If the pack is on this box I can run it on request; the arm-level evidence is what I verified.
2. The 5 % scale band and the 1.5× floor multiplier are now constants with measurements behind them — say in docs/npu.md (or beside the constants) which pack set each was calibrated on, the way npu-pack: a tied head must correlate AND not be mis-scaled, or it imports cleanly and answers nonsense (engine#1503) #2101 did for 0.02. Otherwise the next false refusal will be "raise the band" with nothing to argue against.

Scope of my check: fresh configure (rc 0), built npu_pack_import_test in a scratch dir, ran it (all checks pass; pack leg skipped without a pack directory), and read the 15 fit-rule arms. I did not import the Qwen2-1.5B pack itself — that is ask 1.

[bong-water-water-bong]

VERDICT at 1123461: READY

"On a tied pack, the pack's own embedding is an independent reference for the decoded 4-bit head" — that is the right reference to choose, and it is the reason this check can be trusted: the embedding and the head are the same weights written twice, so a decoded head that does not agree with the embedding is wrong without needing a host dequantisation to compare against.

The refinement — testing for a scaled head — is what #2101 started from the other side: a correlation of 0.99 survives a global mis-scale, so the check has to state both "correlates" and "not mis-scaled". Using the pack's own embedding as the oracle means the scale is checked against something the container actually contains, rather than against a constant that could be wrong in the same direction as the code.