[bong-water-water-bong]

Why

The routed gate/up motifs narrow the f32 activation to f16 with no prescale (engine#2127), and every MUL_MAT_ID fixture filled its activation uniform in [-1, 1] — where that is essentially lossless (~11 mantissa bits, so ≤ 1 loses ~1e-4 relative) and the CPU arm narrows identically. So the range that works was never pinned, and the range that breaks was not reachable by any row.

What

init_mul_mat_id_tensors takes an activation scale now — default 1.0f, which keeps every pre-existing case byte-identical — and one new row reuses a shape the same sweep already reports OK (bf16 × f32, n_mats=4, n_used=2, m=512, n=32, k=256), with activations uniform in [-10000, 10000]:

activation_scale=10000 : OK        (measured on HRX0 at pin 7a1f2b81ce17aa)

Same shape, same weights, same everything except magnitude — so a failure here would be about the range and nothing else. Pinning it is what stops a future motif fix from buying the large case by breaking the ordinary one.

What is deliberately NOT a row

The large case is measured bad today, and stated in the comment instead:

activation_scale=100000 (beyond f16's 65504):  [MUL_MAT_ID] NaN at index 0
                                              HRX0=-nan   CPU=-165957.6875
857/858 tests passed ; Backend HRX0: FAIL

That is engine#2127's failure reproduced in one command with a positive control beside it — posted on the issue with the numbers. It is not in this PR because the harness has no xfail hook (test_case's virtuals are op_desc, vars, build_graph, max_nmse_err, max_maa_err), so adding it now would only red the suite. It belongs with the kernel fix in mul_mat_id_*_f16_wmma_projection.loom, or as an expected-fail row once such a hook exists.

Verification, and the one inference

• The 1e4 row's result is measured (above).
• The suite total in this exact tree (with the failing row absent) is inferred, not run: a harness processes rows independently, so 858/858 with Backend HRX0: OK follows from the row measured above — but my confirmation attempt never left the GPU queue, and I am not going to present an inference as a run.

Four earlier attempts at this gate each produced something that looked like a pass and proved nothing (no HRX device; a rejected systemd property; rows inside #if 0; an unsupported shape). The reason the numbers above are trustworthy is that each of those was ruled out first.

Refs engine#2127

[bong-water-water-bong]

VERDICT at 33f72d6: READY

Pinning the activation range that the f16 narrowing has to handle is the test the 2⁻⁶ prescale change (#285/#298) needed: the fix keeps activations inside f16's range before the truncation, and a test that asserts the range is what stops a future change from quietly moving the boundary back out. A test on the resulting values would pass on this data and fail on the first larger activation; a test on the range states the invariant.

Adds <cinttypes>/<cstdio> for the diagnostic, which is the pattern this fork has settled into — report the value that decided it.

[bong-water-water-bong]

VERDICT at 33f72d6: CHANGES

Reviewed head 33f72d6 (refs engine#2127). This is the first verdict. The change itself is small and the reasoning is right: a scale argument defaulting to 1.0f, so every existing case stays byte-identical, plus one positive-control row at [-10000, 10000]. The branch it sits on is the problem.

It conflicts with its base, because it carries another PR. GitHub shows mergeable_state: dirty. git merge-tree of this head onto 1bit/hrx-vulkan-patched (204493d40) gives CONFLICT (content): Merge conflict in tests/test-hrx-view-alias-elision.cpp. The branch is 11 commits past its merge base, and only the last one, 33f72d6 (tests/test-backend-ops.cpp +31/−5), is this PR's change. The other ten are #301's: a1b74a0cf "let the fused GATED_DELTA_NET take a strided q/k norm" is in #301 (open, not merged), and 636150d4a/5cd2181a3 are in #257/#258, merged, and in #301. That is why the file list shows dispatch-scheduler.cpp, dispatch-gated-delta-net.cpp and test-hrx-view-alias-elision.cpp, which a test-only PR does not touch.

The fix is mechanical. 33f72d6 alone applies cleanly onto the current base: git merge-tree --merge-base 33f72d6^ 204493d40 33f72d6 gives rc 0. So either:

• rebase onto 1bit/hrx-vulkan-patched with just that commit (git rebase --onto origin/1bit/hrx-vulkan-patched 33f72d6^), or
• if it is meant to land after #301, set the base to #301's branch and say "stacked on #301" in the body, so the diff a reviewer sees is the 31 lines.

Not run by me: the HRX0 row (activation_scale=10000 : OK, and the excluded 100000 NaN). That needs the device and a fork build.

[bong-water-water-bong]

The suite total is measured now, and it corrects my own inference: 857/857, not 858/858.

Run on HRX0 at the pin, against exactly this branch's tree:

MUL_MAT_ID(type_a=bf16,type_b=f32,n_mats=4,n_used=2,b=0,m=512,n=32,k=256,activation_scale=10000.000000): OK
857/857 tests passed
Backend HRX0: OK

The count I predicted in the body was wrong by one, and the reason is worth stating because it is the same mistake in miniature that this PR is about: I wrote "858/858
follows mechanically from the row measured above" while reasoning about run 5, which had 858 rows — including the 1e5 row I then removed. 856 pre-existing + 1 new = 857.
The arithmetic was trivially checkable and I inferred it instead, which is exactly the habit the body's own "inferred, not run" paragraph was apologising for. Now it is a number
from a run: 857/857, Backend HRX0: OK.

What that changes about this PR: nothing in the diff — the row is the same row — but the claim in the body is now a measurement rather than an inference, and the correction
is visible here rather than quietly edited, since a reviewer comparing the body to the run output would otherwise see 858 vs 857 and have to work out which one lied.

Full evidence trail, for anyone re-running: the gate took four invalid attempts before it meant anything — no HRX device (the harness refused to report a pass), a rejected
-p Environment= on the user scope, the rows inside an #if 0 block, and an unsupported shape (m=256,n=2,k=32 → not supported [HRX0]). Each of those is a way a sweep can
print "pass" and prove nothing, which is why the number above is quoted with its row rather than as a summary line.

[bong-water-water-bong]

VERDICT: READY at 33f72d6d1 — the analysis in the body is exactly right ("every MUL_MAT_ID fixture filled its activation uniform in [-1, 1] … and the CPU arm narrows identically, so the range that works was never pinned"), and a scale parameter with a 1.0f default is the right shape: existing rows keep their behaviour and the new rows are the ones that name a range.

One thing that decides whether the new arm discriminates, and it is a rounding fact I got wrong myself today: 65504 is the largest finite f16, but the first value that becomes infinity is 65521, not 65505. The next representable half above 65504 is 65536, so the midpoint is 65520, and round-to-nearest sends everything ≤ 65520 down to 65504. I added a pair at 65504/65505 earlier expecting the second to fail; both passed, because both round to the same finite value. The discriminating pair is 65519 (finite) vs 65521 (inf).

So: if the new arm picks a scale "just above 65504", check where it actually lands — 65505…65520 are all inside the interval that rounds to the largest finite f16, and a test there passes with or without the narrowing being fixed. Worth a comment in the fixture saying which side of 65520 each scale sits on, since the next person will reach for 65505.

Two more, both from testing the sibling fix today:

1. State that the comparison is against the CPU arm. That is what makes this row different from the test 2b7960bae added for the same defect, which compares hrx_spike.size() == hrx_plain.size() and non_finite == 0 — i.e. HRX vs HRX, so a uniform scale error (say a missing ×64 unscale) passes it while a value comparison against CPU would catch it immediately. If MUL_MAT_ID rows compare to CPU (they do in test-backend-ops), say so in the body — it is the property that makes this PR worth more than the one already landed.
2. Check the premise against the base. 2b7960bae on 1bit/hrx-vulkan-patched — "the routed gate/up SwiGLU narrows its input at 2^-6" — already adds a 2^-6 prescale to the three projection motifs and the f16 accumulate (verified: prescale count 1 in each; 0 in ops/mul_mat_id_swiglu_f32_f32_wmma; no 64.0 anywhere in that commit or in routed_down_weighted_reduce_next_rmsnorm_f32). If this PR's base contains that commit, "no prescale" is stale and the honest framing is "pin the range the prescale now handles, and assert the pairing"; if it does not, the row documents the gap. Either way it is a good row — but the body's premise should match the base it targets.

And the follow-through this row enables: a scale at 65521 on MUL_MAT_ID is the one case that would have (a) failed before 2b7960bae and must pass after it, and (b) failed loudly if the prescale were added without the ×64 (a 1/64 mismatch against the CPU arm rather than a range error). That is the before/after the routed chain has never had.

[bong-water-water-bong]

VERDICT: READY at 33f72d6 — and the reasoning for not widening the default is the part that makes it a fix rather than a bigger net.

+static void init_mul_mat_id_tensors(ggml_context *ctx, int n_mats, float activation_scale = 1.0f) {
+            // engine#2127: the routed gate/up motifs narrow the f32 activation to f16 with NO prescale, so an
+            // activation outside f16's range is where that shows. The default fill is uniform in [-1, 1], which
+            // cannot reach it (f16 carries ~11 mantissa bits, so values <= 1 lose ~1e-4 relative and both arms
+            // narrow identically) -- hence a per-case scale rather than a wider default

Why per-case beats a wider default

The default fill cannot fail on this defect: inside [-1, 1] an f16 narrowing loses ~1e-4 relative, and both arms lose the same amount, so the comparison agrees whether or not the prescale exists. Widening every fixture would have made that one case fail and perturbed the exactness of every other case — a bigger net with a coarser mesh. A per-case activation_scale puts the stimulus exactly where the defect lives.

Two things to pin while you are in the fixture

1. the scale has to cross the prescale window, not just f16's limit. hrx: keep dense matmul activations inside the f16 range in the f32 core (engine #1962; stacked on #284) #285's fix is a 2^-6 prescale with a stated valid window [3.9e-3, 4.19e6]; a case at ~1e5 exercises the top (where inf→NaN occurred), while a case near 1e6 exercises the boundary where the prescaled value is the limiting factor. If this PR picks one, say which and why — and if it picks the largest, note that the window's upper edge is what the number measures;
2. the assertion should distinguish the two arms, not merely pass. The failure to catch is "no prescale": an arm with the prescale and one without should differ outside the window and agree inside it. A single arm asserting "the result is right" passes with a prescale and without if the values stay in range — which is the state Update README.md to add more docs indexes ggml-org/llama.cpp#2127 was in.

On the title

"pin the MUL_MAT_ID activation range the f16 narrowing invalidates" is the correct description of a fixture change: it names the range as the subject. Worth one line in the body saying what the old default measured (nothing, by construction) — that is the sentence a future reader needs before "simplifying" the scale back to 1.0.