[bong-water-water-bong]

Fixes #2186

The llama.cpp pin on main (4107043c1, set by #2198) already contains both #2186 fixes (llama.cpp #317 c2ea44140, #319 c25d7913f). This PR only changes docs/hrx.md: it replaces the "still open under #2127" paragraph with the fix and its measurements.

What changed in the fork (already in the pin)

• Bump Laya: source 8a6e1328, model 7b928d82 #317: the fused routed feed-forward published silu(gate) * up in f16 without a prescale. It now publishes at 2^-6, and the readers scale back by 2^6 after widening.
• HRX: wrong answers for every prompt longer than one ubatch since fork #82 (graph inputs written back to host) #319: the one-token routed gate/up used the 8-bit blocks' stored f16 sum as it was. A sum that is not finite is now rebuilt from the block's codes; a finite sum is used unchanged.

Measured (gfx1151, device vs CPU graph on dequantised f32 weights; before = same tree without the kernel change)

case	before	after
routed_moe.q4_K.large_product.300 (product up to 2.6e4)	1315 of 2048 non-finite	0, NMSE 9.8e-5
routed_moe.q4_K.large_product.1000 (product up to 2.9e5)	2048 of 2048 non-finite	0, NMSE 1.3e-4
decode_routed_moe.q4_K.large_activation.3000 (block sum about 9.6e4)	2048 of 2048 non-finite	0, NMSE 1.7e-3
decode_routed_moe.q4_K.large_activation.30000 (block sum about 9.6e5)	2048 of 2048 non-finite	0, NMSE 1.5e-3

The in-range cases are unchanged: large_product.1/.30/.100 give NMSE 1.5e-6, 2.3e-4 and 4.4e-4 in both arms, and large_activation.1/.1000 give 6.8e-3 and 4.0e-3 to every digit. Qwen3-Coder-30B-A3B Q4_K_M perplexity (wikitext-2, -c 512 -b 512, 8 chunks) was 9.6681 before and 9.6654 after, with an identical dispatch plan (13,167 dispatches). test-hrx-ops gave 264 of 265; the one failure is the intermittent #2150, and a warm repeat passed 23 of 23.

Checks: tools/registry_build.py --check-pins reports ok; the pin is unchanged from main.

🤖 Generated with Claude Code

[bong-water-water-bong]

ready for review at fd7780d

[github-actions]

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶 2186 - Partially compliant Compliant requirements: Update documentation to reflect that the two faults left open under hrx: the routed gate/up projection kernels narrow their f32 input to f16 unscaled (unmeasured; split from #1962) #2127 are now fixed Document the changes made in the fork at 780d67524 which includes fixes for llama.cpp Bump Laya: source 8a6e1328, model 7b928d82 #317 and HRX: wrong answers for every prompt longer than one ubatch since fork #82 (graph inputs written back to host) #319 Include measured performance data comparing before and after the fixes Ensure the documentation reflects that the limits are moved, not removed Non-compliant requirements: None Requires further human verification: None 319 - Partially compliant Compliant requirements: The documentation reflects that the fix for HRX: wrong answers for every prompt longer than one ubatch since fork #82 (graph inputs written back to host) #319 is included in the fork Non-compliant requirements: None Requires further human verification: None 2127 - Partially compliant Compliant requirements: The documentation reflects that the fix for hrx: the routed gate/up projection kernels narrow their f32 input to f16 unscaled (unmeasured; split from #1962) #2127 is included in the fork The documentation includes the measured performance data for the fixes Non-compliant requirements: None Requires further human verification: None

⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪

🧪 No relevant tests

🔒 No security concerns identified

⚡ Recommended focus areas for review Documentation Update Accuracy The documentation update accurately reflects the fixes for issues #2186, #319, and #2127. It correctly describes the changes made in the fork and includes the measured performance data. The content is consistent with the PR description and the fixes implemented in the fork. The two faults left open under #2127 are fixed in the fork at `780d67524`, which the pin `4107043c1` contains ([engine#2186](https://github.com/1bit-MONSTER/engine/issues/2186)): - **The gated product (llama.cpp #317).** The fused routed body published `silu(gate) * up` in half precision unscaled, and the down projection and its reduction read that copy. It is now published at 2^-6 and the readers scale it back by 2^6 after widening, the same pattern as the input above. - **The 8-bit blocks' stored sum (llama.cpp #319).** Each 32-value block stores its scale and its scaled sum in half precision. The scale fits up to an activation of 8.3e6, the sum only while the block averages below 2047. The one-token routed gate/up used the stored sum as it was; a sum that is not finite is now rebuilt from the block's codes, and a finite one is used unchanged (the rule the Q5_K reader has had since #1962). Measured on gfx1151 (2026-11-11), device against a CPU graph on the same weights dequantised to f32, before = the same tree without the kernel change: | case | before | after | |---|---|---| | routed feed-forward, SwiGLU product up to 2.6e4 (`routed_moe.q4_K.large_product.300`) | 1315 of 2048 non-finite | 0, NMSE 9.8e-5 | | same, product up to 2.9e5 (`.1000`) | 2048 of 2048 non-finite | 0, NMSE 1.3e-4 | | one-token routed decode, block sum about 9.6e4 (`decode_routed_moe.q4_K.large_activation.3000`) | 2048 of 2048 non-finite | 0, NMSE 1.7e-3 | | same, block sum about 9.6e5 (`.30000`) | 2048 of 2048 non-finite | 0, NMSE 1.5e-3 | The in-range rows are unchanged: `large_product.1`, `.30`, `.100` give NMSE 1.5e-6, 2.3e-4 and 4.4e-4 in both arms, and `large_activation.1` and `.1000` the same 6.8e-3 and 4.0e-3 to every digit. Qwen3-Coder-30B-A3B Q4_K_M, perplexity on wikitext-2 (`-c 512 -b 512`, 8 chunks) 9.6681 before #317 and 9.6654 after, with an identical dispatch plan (13,167 dispatches). `test-hrx-ops` on fork `0494dfa28` with #317 and #319 merged: 264 of 265 in two tickets; the one failure was the intermittent draft-head logit of [engine#2150](https://github.com/1bit-MONSTER/engine/issues/2150), which a warm repeat of the same 108 cases passed 23 of 23 times, as it did on `0494dfa28` alone. Fork `780d67524` also carries llama.cpp #318 (the weight-copy trace names the kernels that read each weight) and #314 (a test for #2150). The limits are moved, not removed: 4.19e6 for the half-precision copies and the draft-head logits, 8.3e6 for the 8-bit blocks' scale.

[bong-water-water-bong]

VERDICT at fd7780d: READY

Documentation for the f16-range family — a large SwiGLU product and an 8-bit block sum — which is the shape #1962/#2127/#2128 have been worked in all night. The value of a page like this is that the arithmetic is written down next to the kernels that depend on it: an activation that overflows f16 before a truncation, and a block sum that exceeds an 8-bit accumulator, are both silent in the output and only visible as a rate or a wrong answer.

Docs-only, no behaviour.

[bong-water-water-bong]

VERDICT: READY at fd7780d — fresh configure rc 0, tests_referenced_check green (83 scripts), and the docs change closes an issue whose fix is already on the pin (Fixes #2186, with 4107043c1 from #2198 carrying c2ea44140/c25d7913f).

The two paragraphs are worth more than the issue they close, because they state the failure arithmetic:

• the gated product (Bump Laya: source 8a6e1328, model 7b928d82 #317): the fused routed body published silu(gate) * up in half precision unscaled, and the down projection and its reduction read that copy — so the overflow propagates through a reader, not just through the writer. Fixed by publishing at 2⁻⁶ and scaling back after widening, the same pattern as the input;
• the 8-bit blocks' stored sum (HRX: wrong answers for every prompt longer than one ubatch since fork #82 (graph inputs written back to host) #319): each 32-value block stores its scale and its scaled sum in half precision; the scale fits to 8.3e6 but the sum only while the block averages below 2047 — which is the sentence that explains why hrx: plain (non-fused) MUL_MAT_ID on bf16 weights returns NaN for activations above f16's range, and #2127/#309 does not cover it #2167's MUL_MAT_ID failed at activation_scale=10000, far below the f16 overflow point.

That second sentence is the one I would have asked for in the original kernel PRs: it converts "an activation above the f16 range" into a condition on two quantities (magnitude × block count), which is what a reader needs to predict the next failure instead of recognising it after the fact.

One ask: the paragraph says both faults are fixed "at 780d67524, which the pin 4107043c1 contains" — worth adding which engine rows become measurable again (the routed gate/up and the 8-bit block paths on whichever packs exhibit them), since a docs page that records a fix without saying what can now be re-measured leaves the next lane to rediscover it. If those rows are already named elsewhere in the page, ignore this.

Scope of my check: fresh configure rc 0, the referenced-tests ratchet, and the docs/hrx.md diff (108 insertions). No device and no pin build: the arithmetic in the two bullets is the fork commits' own, and the pin containing them was verified on #2198.

[bong-water-water-bong]

VERDICT at fd7780d: READY

Docs only (docs/hrx.md). It merges cleanly onto main. CI build at fd7780d is green: 100% tests passed, 0 tests failed out of 246.

Both fixes are in main's pin. compare <fix>...<main pin> is ahead for llama.cpp #317 c2ea44140 and #319 c25d7913f, so the doc describes the code main runs.

The numbers match the fork PRs:

• #317's body carries the large_product rows: 1315/2048, NMSE 9.8e-5 / 1.3e-4, perplexity 9.6681 → 9.6654, 13,167 dispatches.
• #319's table carries the large_activation rows: 2048/2048 before; 0.00170 and 0.00147 after, i.e. 1.7e-3 and 1.5e-3; in-range 0.00682 and 0.00405.

Fixes #2186 is right: both of its items are covered. Item 1 (the fused SwiGLU publish) is #317. For item 2, I checked #319's claim that every other reader of the stored 8-bit block sum goes through @qwen3_moe_q4k_q8_1_dot, in the corpus at the pin:

• @qwen3_moe_q4k_q8_1_x4_paired_block_lane (used by dense_linear_quantized, attention_qkv_quantized, and routed_down_q4k via cohort_row_lane) passes q8_low_s / q8_high_s into the guarded @qwen3_moe_q4k_q8_1_dot (routed_gate_up_swiglu_q4k.loom:491-492).
• The Q6_K reader (@ggml_q6k_q8_1_x4_block_lane) takes only the block's d through @ggml_q8_1_x4_word and never reads the sum.
• mul_mat_q5_k_q8_plane_wmma has rebuilt the sum since #1962.

Not in #2186's scope, and still open: the f16 accumulator class (products summed in a vector<8xf16> fragment through vector.mma, e.g. routed_down_quantized_f16_wmma.loom). Neither of these fixes touches it, and it is #2167 / #2231's to settle.

[bong-water-water-bong]

VERDICT: READY at fd7780d

Documentation for a fix that is already in the pin — and I checked that claim at the pin's content rather than its ancestry, which is the only way a "already in the pin" sentence can be trusted.

main's recorded pin (registry/architectures.json)   4107043c16d1
4107043c1 contains 780d67524                        yes (git merge-base --is-ancestor)
#317 at the pin — motifs/mul_mat_id_swiglu_f16_f16_wmma_body.loom:
    %sum_rescale = vector.constant 64.0 : vector<8xf32>          <- 2^6
    %gate_prescaled = vector.extf %gate_values : vector<8xf16>   <- widen, then scale back
#319 at the pin — qwen_moe/qwen3_moe/routed_gate_up_swiglu_q4k.loom:
    rebuilt / not finite markers: 6                              <- the non-finite sum is rebuilt

A correction to my own first check, recorded because it nearly became a false CHANGES: I grepped loom-libs/ops/ for #317 and got 0 hits, which looked like the fix was missing from
the pin. The body lives at loom-libs/motifs/, and the markers are named %sum_rescale / %gate_prescaled, not the literal 0.015625. Two wrong assumptions in one grep; the fix is
there.

The doc is the right shape for a fix that reached main through a pin: it says which fork commits (#317 c2ea44140, #319 c25…), that the pin contains them, and then what each mechanism
is — "published at 2^-6 and the readers scale it back by 2^6 after widening, the same pattern as the input above" and "a sum that is not finite is now rebuilt from the block's codes, and
a finite one is used unchanged (the rule the Q5_K reader has had since #1962)". That second sentence is the valuable one: it places the new rule beside the rule it copies, so a reader can
see it is a pattern in this codebase and not a one-off.

The measured table pairs each fault with its own case and threshold, which is what makes a doc a record rather than a claim:

case	before	after
SwiGLU product to 2.6e4	1315/2048 non-finite	0, NMSE …
product to 2.9e5	2048/2048	0, NMSE 1.3e-4
one-token decode, block sum ~9.6e4	2048/2048	…
block sum ~9.6e5	2048/2048	0, NMSE 1.5e-3

and it states the two different thresholds that make the two faults separate: the scale fits to 8.3e6, the sum only while the block averages below 2047. A reader who takes away one number
from this page will take away the wrong one, and the doc gives both with their conditions.

Range check: a single file, +31/−4, and nothing in it claims the pin bump is this PR's — #2198 did that.