[bong-water-water-bong]

Refs 1bit-MONSTER/engine#2186 (the first of its two items; it reaches engine main with a later pin bump).

The fault, measured

The fused routed feed-forward keeps two planes in f16: the gate/up kernel publishes silu(gate) * up to one, and the routed down projection reads it and accumulates and publishes its sums in f16 to the other. So the product and every partial down sum had to stay under 65504.

New test-hrx-ops cases routed_moe.q4_K.large_product.<scale>: the routed graph of routed_moe.q4_k with every input multiplied by scale, compared with the CPU. At fork caa2d4dde, on gfx1151:

scale	largest silu(gate)*up (CPU)	before	after
1	0.18	NMSE 1.5e-6	NMSE 1.5e-6
30	260	2.3e-4	2.3e-4
100	2,886	4.4e-4	4.4e-4
300	25,976	1315 of 2048 outputs non-finite	0 non-finite, 9.8e-5
1000	288,623	2048 of 2048 non-finite	0 non-finite, 1.3e-4

At scale 300 the product is still inside f16; it is the down sums that overflow first. No shipped model is known to reach these magnitudes on this route (flint's 9.1e4 SwiGLU outputs take the single-precision routed kernels, fixed in #284/#285); this is the same class of fault on the route Qwen3-30B-A3B takes.

The change

• motifs/mul_mat_id_swiglu_f16_f16_wmma_body.loom: the product is published at 2^-6.
• The routed down projection is unchanged: it is linear, so its sums are at 2^-6 too.
• The three kernels that reduce the routed down plane (qwen3_moe_routed_down_weighted_reduce_f16_f32, ..._next_rmsnorm_f32, ..._next_rmsnorm_q8_1_x4) multiply by 2^6 after widening to f32.
• dispatch-routed-ffn.cpp: both planes are registered and looked up with scale_log2 6, the field hrx: the Q4_K/Q6_K prefill reads its f16 activation copy at 2^-6, so an activation above 65504 stays finite (engine #1962) #298 added to plan alternate values, so a reader that does not ask for the scale does not find them.

Both factors are powers of two, so ordinary values are rounded as before (the first three rows of the table are identical before and after). The window moves from 65504 to 4.19e6 for the product and the down sums.

About the test

Two things I had to get right before the numbers meant anything:

• The reference. With K-quant weights the CPU quantises the activation to 8 bits per block, which costs more precision than the device's f16. The CPU graph here holds the same weights dequantised to f32.
• The router logits. make_router_logits() repeats eight values over the 128 experts. With ties the CPU and the device each pick a valid but different top eight, which read as an 8 to 35 % "error" in both arms. The case uses distinct logits.

hrx-backend-test: the rows that look the two planes up now ask for scale 6 and assert that an unscaled lookup finds nothing.

Measured on the fix (gfx1151, 2026-11-11, each under its own gpu-hold ticket)

• test-hrx-ops: 261 passed, 0 failed, 261 total.
• hrx-backend-test host: 73 passed, 0 failed (71 passed, 2 failed before the two rows were updated, which is the scale doing its job).
• Qwen3-Coder-30B-A3B-Instruct Q4_K_M, which runs on this route (llm.routed_ffn.gate_up_swiglu_f16_wmma, llm.routed_ffn.down_q6k_f16_wmma_grouped, llm.routed_ffn.down_weighted_reduce in its dispatch log), before against fix:

  • perplexity, wikitext-2, -c 512 -b 512, 8 chunks: 9.6681 +/- 0.65886 against 9.6654 +/- 0.65844;

  • the plan is the same: 32 dispatch registrations before and after with identical names, and 13,167 dispatches over the same 36 graphs in both arms (GGML_HRX_LOG_DISPATCH=1 GGML_HRX_HOST_GAP=1, llama-bench -p 512 -n 16 -r 2). So recording the scale did not make any reader fall back to another path; the change is one multiply per value at each end.

  • llama-bench -p 512 -n 32 -r 3, ten alternated arms over three tickets. The box was loaded and the arms scatter by 500 tok/s within either side, so this shows no difference rather than proving none:

    	prefill tok/s	decode tok/s
    before	1,663 1,705 1,548 1,196 1,635	87.8 82.8 84.6 58.1 83.2
    fix	1,710 1,396 1,278 1,594 1,359	88.3 87.8 82.1 84.0 42.9

    A slow arm is slow in all three of its repetitions on either side, which is the process's state, not the kernel's. In the plan run above the dispatch replay time was 1,749 ms before and 1,384 ms after for the same 13,167 dispatches.

Not shown

• A rate measurement that could resolve a few per cent. The arms above cannot. What is shown is that the plan and the dispatch count are identical.
• The suite and the model rows were run on a build of caa2d4dde plus this change; the branch is rebased onto 0494dfa28 (two commits later, neither touches these files) and was not rebuilt there.
• Builds have no HIP add-on.
• The routed down sums can still overflow above 4.19e6, and the gate and up sums are accumulated in f16 at 2^-6 (hrx: the routed gate/up SwiGLU narrows its input at 2^-6, so an input above 65504 stays finite (engine #2127) #309), with the same ceiling.
• Item 2 of add top-k slider to web ui ggml-org/llama.cpp#2186 (other readers of the 8-bit activation blocks) is untouched.

🤖 Generated with Claude Code

[bong-water-water-bong]

ready for review at 58db3e2 (body updated: the dispatch plan is identical before and after on the routed model, 13,167 dispatches in both arms; ten alternated bench arms show no difference a loaded box can resolve)

[bong-water-water-bong]

VERDICT at 58db3e2: READY

The factor becomes a named constant instead of a literal, which is the right end state for a family that now has four members:

static constexpr int kRoutedFfnF16PlaneScaleLog2 = 6;

2⁻⁶ has appeared as 0.015625 in #285 (dense), #284 (routed experts), #309 (routed gate/up) and now here. Each was written as a literal, which is four places that can drift by one. Naming the log2 means the next kernel in the family picks up the same factor by reference, and a reviewer can see that it is the same factor rather than checking a decimal.

Passing it through find_alternate_value(...) rather than leaving the call's implicit default is the other half: the scale is an argument to the lookup, so the kernel that reads the plane and the kernel that wrote it cannot disagree about the exponent.

Refs ggml-org#2186 (first of its two items), reaching main with a later pin bump — the chain ggml-org#2141's checker guards.

[bong-water-water-bong]

VERDICT: READY at 58db3e2

The table is the right shape — it brackets the transition rather than sampling a range — and the explanation for why the product is not the thing that overflows is the part that makes it
believable.

scale	largest silu(gate)*up	before	after
1 / 30 / 100	0.18 / 260 / 2,886	NMSE 1.5e-6 / 2.3e-4 / 4.4e-4	identical
300	25,976	1315 of 2048 non-finite	0, 9.8e-5
1000	288,623	2048 of 2048 non-finite	0, 1.3e-4

At scale 300 the product is only 40 % of the f16 ceiling and the output is already 64 % non-finite, so the overflow is in the down sum, not the product — and the PR says that
outright ("it is the down sums that overflow first"). That matters because it is the kind of claim a reader could otherwise "fix" by widening the product alone: the three reducing
kernels (…_weighted_reduce_f16_f32, …_next_rmsnorm_f32, …) have to scale back too, and the change says they do.

The three identical rows are the exactness argument, visible rather than asserted: 2^-6 in and 2^6 out are powers of two, so values that fitted before round identically — which is
why 1/30/100 do not move at all. A prescale that changed the low rows would be a precision regression wearing a range fix.

What I like most is the honesty about reach: "No shipped model is known to reach these magnitudes" — so this is a latent fault being closed, not a model being fixed, and the PR
says so instead of implying a customer-visible bug.

One note, not a change: the five scales bracket the transition (100 passes / 300 fails / 1000 fails harder), which is the right selection, but the exact edge is unexplained. 25,976
per product × 2,048 terms would overflow; 2,886 × 2,048 would not, and 2,886 × 2,048 = 5.9e6 which is above 65,504 — so the number of terms that actually reach the sum is far below
2,048 (the router's top-k). A sentence giving the reduction width would let a reader predict the edge instead of tabulating it. Not a gate: the arm set finds the boundary empirically,
which is what an arm is for.

Checked in the source: the plan records the scale with the alternate values (scale_log2 6), the same field #298 added, and the dispatch registers both planes so the reader that
does not ask for a scaled copy still gets the stored layout. That is the pair behaving the same way in a second family, which is the argument for the mechanism being right rather than
coincidentally right here.

Not run: the device arms (gfx1151, GGML_HRX=ON). My check is the arithmetic in the table and the mechanism's placement.

[bong-water-water-bong]

VERDICT: READY at 58db3e2f3 — and I checked the one thing that could have made this quietly wrong.

The factor does not compound. Two f16 planes each published at 2^-6 is the obvious way to get a 2^-12 error: scale the SwiGLU product, then let the linear down projection carry that factor, then scale its output again. Reading the diff, it does not happen — each plane is unscaled at its own read site (%plane_unscale = vector.constant 64.0, applied to %stored and to %routed_stored respectively, both widening to f32), so each plane's factor is undone before it is used and the two never multiply. That is the property to keep: the invariant is "every reader of a scaled plane unscales it", and today each plane has exactly one reader.

The rest, checked:

1. The motivation is a measurement, not a worry: products up to 2.6e4 made the down sums overflow and 770 of 2048 outputs go inf. That is the right kind of evidence for a numeric change — a fault that was observed, with a count.
2. Scaling down is the correct direction for f16: relative precision is preserved (values move toward the well-populated part of the range), and the cost is an absolute floor 64x higher in original units, which is irrelevant at products of 1e4.
3. The plan records it (scale_log2 on both alternate values), so a reader has to ask for the scaling rather than discover it: that is what keeps a future investigation from measuring a 64x-shifted quantity against a stale expectation.
4. The comment states the mechanism and its direction — publish scaled, projection is linear and carries it, f32 reduce brings it back — which is exactly the chain a reviewer needs to verify, and it matches the MLIR I read.

One framing worth keeping in the record rather than only in this thread: this buys 64x of ceiling, it does not remove the ceiling. The overflow boundary moves from ~1e3 to ~6.5e4 per element (and the sum bound with it), so a future model with larger SwiGLU products will hit it again — the test's large_product.<scale> table is what will show it, and the next step at that point is a dynamic scale, not a larger constant. If that sentence is already in the body, ignore me; if not, it is one line that prevents the constant from being treated as a guarantee.

Also fine: the new test-hrx-ops cases exercise several scales rather than only the one that failed, which is what makes the ceiling discoverable instead of just the current failure fixed.