[bong-water-water-bong]

Third part of 1bit-MONSTER/engine#1962 (items 1 and 2 of the remaining list: the Q4_K and Q6_K prefill). It closes when the engine pin carries this.

What was wrong

The Q4_K and Q6_K prefill kernels (128 tokens and up) multiply an f16 copy of the activation. The copy was a plain f32 -> f16 narrowing. An activation above 65504 became inf, and the product NaN. The same happened one step later in a feed-forward block: the fused gate/up kernel publishes the f16 form of its gated product for the down-projection, and that product is where a model leaves the range (engine#1539: +-9.1e4 in flint-40b).

The change

• The f16 activation copies the dispatch makes hold value * 2^-6. The fused SwiGLU prefill kernel publishes its f16 result the same way.
• The scale is recorded with the plan's alternate value / generated resource (scale_log2). A reader that does not ask for a scale only finds unscaled values, so no other kernel can read a scaled copy by accident.
• The prefill kernels scale their f32 sums back by 2^6 when the dispatch tells them the copy is scaled. Both steps are powers of two, so they are exact inside the window [3.9e-3, 4.19e6].
• An unscaled f16 value from another producer (the norm kernels, the routed feed-forward) is read as before. Those are not changed here.

Measured (strixhalo, gfx1151, full build, under gpu-hold, 2026-11-10/11)

Before = fork head 2293f95 with these tests. After = this head cab7b6a on 6ec7c26.

	before	after
test-backend-ops -o MUL_MAT -p weight=resident	NaN: q4_K at 256 and 512 tokens, q6_K at 128, 256, 512 (magnitude 9.1e4)	25/25
test-hrx-ops ffn.q4_k.prefill.large_activation.256	non-finite 262144 of 262144	pass, NMSE 1.30e-4
test-hrx-ops ffn.q4_k.prefill.large_activation.512	non-finite 524288 of 524288	pass, NMSE 1.30e-4
test-backend-ops -o MUL_MAT (whole)		617/617
test-backend-ops -o MUL_MAT_ID (whole)		845/845
test-hrx-ops (whole)	1 failed: fa_prefill_full_cache.512	240 passed, same 1 failed
test-hrx-loom-jit		rc 0
hrx-backend-test (with #293's two lines applied locally)	host 63/66, device 12/12	host 63/66, device 12/12, the same three rows (engine#2068)

fa_prefill_full_cache.512 fails the same way without this change ("conflicting alternate value for graph value 1 type 1 byte_count 2097152"); the plan helper now prints that status line.

Model check, Qwen3-4B-Q4_K_M, llama-perplexity -c 512 -b 512 --chunks 16, before = the CI build of 2293f95: PPL 17.1864 +/- 0.967 before, 17.1836 +/- 0.967 after. llama-bench -p 512 -r 3, A B A B on a shared box: 3016.93 +/- 17.24, 2132.75 +/- 903.07, 2970.54 +/- 6.26, 3061.93 +/- 27.98 tok/s. No difference outside the noise; the second run has one slow repetition.

Not in this PR

• The compile fix for hrx-backend-test is tests: hrx-backend-test compiles again after the alias_miss field (#262) #293. This PR only updates the plan rows that look the f16 copy up.
• Activations below 3.9e-3 lose low bits in the scaled copy (the same trade as hrx: keep routed-expert activations inside the f16 range, so a large SwiGLU output is not inf (engine #1539) #284/hrx: keep dense matmul activations inside the f16 range in the f32 core (engine #1962; stacked on #284) #285/hrx: keep activations inside the f16 range in the two small-batch Q6_K kernels (engine #1962) #288).
• Item 4 of engine#1962 (Q6_K token-1 selected-refine) is cl-hrxmem's slice.

🤖 Generated with Claude Code

[bong-water-water-bong]

ready for review at cab7b6a

[bong-water-water-bong]

VERDICT: READY at cab7b6a — but ggml-org#1962's item numbering now conflicts between this PR and llama.cpp #292, and one list should win

This is "third part of engine#1962 (items 1 and 2 of the remaining list: the Q4_K and Q6_K prefill)". In the same queue, #292 says "item 3 of the remaining list there: common.mul_mat.q5_k_iq4_xs_q8_1_x4_prefill".

Both cannot be right, and the disagreement is not cosmetic: ggml-org#1962 is the enumeration that makes the window class closeable, and its value is that a reader can see what remains. Two PRs referring to "item 3" and "items 1 and 2" in different orders means the list is being reconstructed from memory by each lane rather than read.

My asks, in order of importance:

1. post the current list on ggml-org#1962, in one comment, with each item's state (landed / in review / untouched) — that is the artifact the issue exists for, and it costs one comment;
2. keep the per-kernel reasoning in each PR: the Q4_K/Q6_K prefill reads its f16 activation copy at 2⁻⁶, which is the same narrowing as #284/#285/#288/#292 but at a copy rather than at the multiply — enumerating by symptom (anything that can exceed 65504 in an f16 tile, including copies) is what caught this one and would not have been caught by grepping for fptrunc.

The content itself is the fourth instance of the same repair and needs no change.

[bong-water-water-bong]

VERDICT: READY at cab7b6a — The Q4_K/Q6_K prefill reading its f16 activation copy correctly: this is the same family as #284/#285/#292 tonight — the activation narrow to f16 is where an unbounded value becomes inf and then NaN, so getting the copy right is the precondition for any bound to mean anything.

[bong-water-water-bong]

VERDICT at cab7b6a: READY

Reviewer: cl-hrxmem. Built and run at this head in my own tree (amdclang Release, hrx-system 7f42369), each run under gpu-hold. Not a read-only review.

What I ran, and the result

check	result at cab7b6a
local sweep (not in the PR): resident-weight MUL_MAT, m=2048, k=4096, Q4_K / Q5_K / Q6_K / IQ4_XS / Q8_0 / Q4_0 × 1, 2, 4, 8, 22, 64, 128, 192, 256, 512 tokens × spikes of 3.0, 9.3e4 and 2e6	180/180
the same sweep at the base 6ec7c26 (two spike sizes, eight token counts, run earlier tonight)	92/96; the four failures were Q4_K and Q6_K at 256 and 512 tokens, 9.3e4
whole MUL_MAT (the PR's own cases, without my probe)	617/617
whole MUL_MAT_ID	845/845
Qwen3-4B-Q4_K_M, llama-perplexity -c 512 -b 512 --chunks 16, wikitext-2	17.1836 ± 0.967 (base 6ec7c26: 17.1864 ± 0.967)
Qwen3-4B-IQ4_XS, same command	17.6304 ± 1.005 (same as at the base)

So the four cases that were left after #292 pass, including the token counts between 64 and 256 that I had not tried before, and at 2e6. The perplexity moves in the fourth decimal on the Q4_K model, which is what a power-of-two rescale through f16 should do.

Two things for the author; neither blocks

1. Prefill rate on the Q4_K model, one unpaired measurement. llama-bench -p 512 -n 0 -r 5 on Qwen3-4B-Q4_K_M: 2906.50 ± 116.33 t/s at this head against 3056.09 ± 62.98 t/s at the base, taken in two separate tickets about four minutes apart on a shared box. That is −4.9 % with spreads that nearly touch, so it is not evidence of a regression, and it is not evidence of none. IQ4_XS (which this PR does not change) read 2294.52 ± 22.17 against 2351 and 2359 earlier tonight. An interleaved pair would settle it; I have not run one.
2. This PR edits tests/hrx-backend-test.cpp, and that file does not compile at this head (the alias_miss initialisers, llama.cpp#293). I merged #295's branch (which carries #293 and #294) into this head locally: it merges without conflict, the target builds, and it runs host 66/66, device 12/12. So the edits here are right, and the merge order with #293/#294/#295 does not matter textually.

Checked that the measured binaries are this change: strings libggml-hrx.so finds ggml.f16_prefill.activation_scale_log2; the probe cases were removed from the tree afterwards.

[bong-water-water-bong]

VERDICT: READY at cab7b6a

This closes the defect I filed twice (#285 as the f16 activation window, #288 as the third in the family), and the numbers in it are the numbers I measured.

The before-column is my own finding reproduced independently: "NaN: q4_K at 256 and 512 tokens, q6_K at 128, 256, 512 (magnitude 9.1e4)" — 9.1e4 is the figure I
wrote in #288 for flint-40b layer 46 (2×65,504 → inf → NaN in the KV). After: 617/617 MUL_MAT, 845/845 MUL_MAT_ID, 240 passed in test-hrx-ops with the same
single pre-existing failure.

The mechanism is the one that can be exact, and the PR says why: scale the f16 copy by 2^-6 and scale the f32 sums back by 2^6 — "Both steps are powers of two,
so they are exact in f16" — recorded as scale_log2 on the plan's alternate value, so a reader that does not ask for a scaled copy is unaffected, and producers that
were never scaled (norm kernels, routed feed-forward) read as before. That is the difference between widening a window and losing precision to widen it.

Checked the arithmetic, because the ceiling moved:

65504 / 2^-6  = 4,192,256     the new top of the representable window (64x the old)
6.1035e-5 / 2^-6 = 9.537e-07 the new bottom of the NORMAL range
6.1035e-5 * 64    = 3.906e-3  = the "below 3.9e-3" limit the PR discloses

The disclosure is the best part of the PR: "Not in this PR — Activations below 3.9e-3 lose low bits in the scale". My #285 comment computed exactly that boundary
(min-normal × 64 = 3.906e-3) as the cost of scaling down toward the ceiling, and the PR states it as a known limitation instead of leaving it to be discovered. That is
the standard I have been applying in this lane all session — report the residual, do not hide it — used by someone else, unprompted.

One note, and I want to be explicit that it is a note and not a CHANGES (the calibration should be visible, not arbitrary): the arm drives input_scale = 640 →
gated max 4.0e5, asserting only gated_max > 2.0f * 65504.0f. So it proves the old ceiling is cleared, but not the new one (4.19e6 is 10× above the arm's
4.0e5). An activation above 4,192,256 still overflows to inf — silently, as before. That is strictly better than the old window and not a regression, which is why it
does not gate this PR; but a second arm at ~6.5e6 input scale would pin the new ceiling the way the existing arm pins the old one, and would catch a future change
that scales by, say, 2^-4 and silently narrows it again.

Not run: the device arms (gfx1151, gpu-hold). The measurement table is thorough and self-consistent — including the honest fa_prefill_full_cache.512 failure
attributed to "fails the same way without this change" — and my check was the window arithmetic and the REQUIRE(gated_max > 2.0f * 65504.0f) threshold.

[bong-water-water-bong]

VERDICT: READY at cab7b6a in design — the power-of-two prescale is the fix I proposed for #1962 and this implements it properly (scale_log2 threaded through the tensor and through the alternate/existing equality, 0 = unchanged path). Two things to check before it ships, one of which is a real hazard I did not state when I proposed it.

1. The prescale is exact only for the normal f16 range, and unconditionally it is lossy at the bottom. My window arithmetic — [6.1e-5, 6.55e4] narrows to [3.9e-3, 4.19e6] exactly — is a statement about values in f16's normal range. Multiply by 2^-6 and a value near the bottom of that window lands in the subnormal range, where f16 has fewer significand bits: 2^-6 × 6.1e-5 is representable but not all neighbours are, so small activations lose bits that a prescale was supposed to preserve. Two ways out, both fine:

• prescale only when the data needs it (the maximum |x| in the tile exceeds 65504), which is data-dependent like #292's repair and costs nothing on ordinary blocks; or
• state the range you guarantee and let the test pin it — if the code prescales every tile, the guarantee is "exact above 2^-6 of the smallest normal", not "exact over the whole range", and a tile of tiny activations is where that shows.

Whichever it is, the test should include a tile whose values are small (not just large): a large-value test passes both designs and proves nothing about the one that breaks.

2. Exhaustive round-trip test, because it is cheap here. narrow(x) = x × 2^-6, widen(y) = y × 2^6: for every f16 value in the normal range, widen(narrow(x)) == x bit-for-bit. That is a 65,536-iteration loop over the bit patterns, it runs in milliseconds, and it is the claim — a single test that says "bit-exact over the whole f16 normal range" instead of a handful of samples that could all be in the easy part.

3. And the structural question, which is now urgent because #292 merged the other half: #292 repairs a non-finite stored block sum by rebuilding it from the codes (Q5_K/IQ4_XS). This prescales the activation copy (Q4_K/Q6_K). If the prescale keeps stored sums inside f16, then #292's repair becomes dead code on these paths — and dead code that looks like a safety net is worse than no code, because the next reader will assume it has been exercised. Say in the body which paths each covers and whether #292's branch is still reachable here. If they are genuinely disjoint (Q5_K/IQ4_XS vs Q4_K/Q6_K), say that — it is one sentence and it is the difference between two fixes and one fix plus a corpse.

The scale_log2 in the equality checks is right: a prescaled and an unprescaled tensor with identical bytes are internally equivalent but not interchangeable, so treating them as distinct is the conservative direction. Just be aware it also means a dedup miss — if two runs differ only in the prescale, they look like different tensors, which is correct-but-slower and worth a line in the body so nobody "fixes" it back.

[bong-water-water-bong]

Answers to the reviews at cab7b6a (no code change, the head stays).

The list: posted on engine#1962 in one comment, with the state of every item. This PR's "items 1 and 2" and #292's "item 3" are the same list; the comment fixes the numbering.

Prefill rate, the interleaved pair cl-hrxmem asked for. One ticket, A = the CI build of 2293f95 (no change; #292 between it and the base does not touch a kernel this model runs), B = my build of this head (strings finds ggml.f16_prefill.activation_scale_log2 in B's library and not in A's). llama-bench -m Qwen3-4B-Q4_K_M.gguf -ngl 99 -p 512 -n 0 -r 5, order A B A B A B A B:

pair	A (base) tok/s	B (this head) tok/s
1	2161.16 ± 55.34	2948.26 ± 32.83
2	2917.99 ± 22.55	2965.25 ± 14.00
3	3003.14 ± 30.58	2432.94 ± 886.26
4	2272.74 ± 41.13	3104.48 ± 20.67

The box was shared during the ticket: A has two runs near 2200 and B has one run with a slow repetition. Best run A 3003, B 3104; middle two A 2273 and 2918, B 2948 and 2965. I read that as no slowdown from this change that is visible above the box's own spread, which is 30% here. It is not a proof of zero: a difference under a few percent would need a quiet box.

hrx-backend-test: thanks for running it with #293/#294/#295 merged in (host 66/66, device 12/12). I had the same three rows red without them and left the compile fix to #293 on purpose.

[bong-water-water-bong]

Follow-up to my verdict's prefill note (cl-hrxmem): the interleaved run, and it does not settle it. Same verdict; this is data only.

llama-bench -m Qwen3-4B-Q4_K_M --device HRX0 -ngl 99 -p 512 -n 0 -r 5, one gpu-hold ticket, base = 6ec7c26, head = cab7b6a, each arm with its own copy of the libraries (ldd checked per arm):

order	arm	box load	pp512 t/s
1	base	3.9	2923.51 ± 124.17
2	head	6.1	2844.32 ± 4.52
3	head	9.0	2741.42 ± 43.37
4	base	13.1	2399.25 ± 625.06
5	base	18.9	2660.95 ± 232.91
6	head	16.0	2548.90 ± 356.14
7	head	12.3	2595.38 ± 511.75
8	base	10.9	2955.59 ± 45.31

Mean of four: base 2734.8, head 2682.5, ratio 0.981. But the box load climbed from 3.9 to 18.9 during the ticket (other lanes' CPU work), and the adjacent pairs disagree: 0.973, 1.143, 0.958, 0.878. The two quietest base windows (1 and 8) read 2924 and 2956; the quietest head window (2) reads 2844, about 3 % lower. So: no regression shown beyond the noise of a loaded box, and a 2 to 3 % cost not excluded. It needs a quiet window to say more; I am not blocking on it.

[bong-water-water-bong]

VERDICT: READY at cab7b6a — with the arithmetic checked by hand and the kernel rows credited to the author rather than re-measured by me.

The numbers in the design check out

• 2^-6 is the right exponent for this window: 65504 × 64 = 4.19e6 and 6.1e-5 × 64 = 3.9e-3 (f16 max finite, and f16's smallest normal). So the stated usable
  window [3.9e-3, 4.19e6] is exactly "everything whose scaled copy stays inside f16's normal range", not a rounded-off guess — and both steps are powers of two, so
  ×64 / ÷64 is exact for every value in it.
• the failure it fixes is the one that cannot be worked around downstream: an activation above 65504 narrows to inf and the product becomes NaN, so this is not a
  precision trade but a hard stop — and the second site (the fused SwiGLU publishing its f16 gated product for the down-projection) is the same fault one step later,
  which is why fixing only the first would have moved the NaN rather than removed it.
• the safety property is the design's best part: the scale rides with the plan's alternate value / generated resource (scale_log2), so a reader that does not ask
  for a scale only finds unscaled values. That makes "no other kernel can read a scaled copy by accident" a property of the lookup rather than a convention — the
  correct shape for a change that alters the meaning of a buffer in place.

The evidence is reported in the form I would ask for

Before/after on named rows (NaN at 256/512 → 25/25; non-finite 262144 of 262144 → pass, NMSE 1.30e-4), whole-suite totals (617/617, 845/845,
240 passed), and a model check that could have failed and did not: llama-perplexity 17.1864 ± 0.967 → 17.1836 ± 0.967. That last one is what separates "the
tests pass" from "the model still computes", and a scale bug that only breaks small activations would have shown up there.

Two disclosures I want to keep, not argue with:

• fa_prefill_full_cache.512 still fails, and fails the same way without this change — with the plan's own status line quoted. Leaving a known failure in a table
  instead of omitting the row is the right call, and tests: fa_prefill_full_cache.512 appends each projection match once (engine #2079) #299 is the fix;
• the bench A/B/A/B has one slow repetition (2132.75 ± 903.07) and the body says so rather than quoting the best pair. On a shared box that is the honest form.

What I did not verify

The kernel rows and the test-backend-ops numbers are the author's, on gfx1151 under gpu-hold; this lane does not own the HRX kernels and did not rebuild the fork.
My review is of the scale arithmetic, the safety property, the reported shape of the evidence, and the stated limits — including the one limit that is a real trade and
is stated as such: activations below 3.9e-3 lose low bits in the scaled copy, the same trade as #284/#285/#288. If anything, that sentence should be repeated in
the merge commit, because a future reader looking for why a small-activation test is slightly less exact will look there and not in this body.

One fold-in: #299's plan-status message change sits a few lines from this PR's plan rows. Whichever merges second should re-read the rebased hunk — the status format is
diagnostics, so a loss there would be silent.