hrx: four Q4_K/Q6_K MUL_MAT registrations have no known-answer coverage - test_mul_mat's unflagged weight is refused, and test-backend-ops reports the refusal as an NMSE of k/3+1

What happens

Three HRX MUL_MAT registrations (four with q6_k_aligned_skinny) return results that are wrong by a large, regular factor when test-backend-ops exercises them with ordinary data — uniform [-1, 1] weights and activations, nothing special about magnitude. The normalised error is ≈ k/3 (k = the contraction length) at every shape tried, which is not rounding: it is a different computation.

Found while separating real overflows from non-overflows on #1962; it is independent of that issue and of llama.cpp#284/#285 (it reproduces on the unmodified CI bench build of main's pin).

Measurement

test-backend-ops test -b HRX0 -o MUL_MAT, plain test_mul_mat(type_a, f32, m, n, k) cases, one case per process, GGML_HRX_LOG_DISPATCH=1 to name the registration. strixhalo, 2026-11-10 18:32Z, CI bench build at a5dc597b8, one gpu-hold ticket, no model.

registration	weights, tokens	k = 512	2048	4096	8192
`common.mul_mat.q4_k_q8_1_x4_prefill`	q4_K, 512	171	684	1363	2730
`common.mul_mat.q6_k_token1_final_projection_q8`	q6_K, 1	166	690	1343	2762
`common.mul_mat.q6_k_i8_prepacked`	q6_K, 8	—	711	1368	2758
`common.mul_mat.q6_k_aligned_skinny`	q6_K, 8 (m ≥ 4096)	175	—	—	—

(NMSE against the CPU backend; the limit is 5e-4. Each cell is the same for m = 512, 2048, 4096, 8192 to within a few percent — 16 shapes per row for the first two. Where q6_K at 8 tokens is routed to kquant.mul_mat.decode_f32 instead, the same case passes.)

An error that grows linearly with k and does not depend on m is a coherent term, not noise — the shape of an offset or a packing step that was not applied.

What I do not know

Whether serving is affected. Models that use these paths produce sane text and perplexity on this box, so my reading is that the registrations expect a weight the loader has prepared (the names say "prepacked"; a reviewer of llama.cpp#285 noted these paths want weight-flagged buffers) and here they claim a bare test tensor they should decline. That is a reading, not a measurement.
Which of the two it is, then: (a) the matcher admits a tensor without the precondition it needs — a claim/dispatch inconsistency of the hrx: the CLIP graph runs on no HRX backend, so a vision model's projector falls back to the CPU at ~17 s/image #1467 kind, visible only in the test harness; or (b) a real wrong answer that serving happens not to reach at these shapes.

Why it matters either way

These registrations have no known-answer coverage: any test-backend-ops case that reaches them fails, so the suite (589/589 on MUL_MAT today) cannot contain one. They are prefill and decode paths for Q4_K/Q6_K — the most common weight formats — and the gate that would catch a regression in them does not exist. It also cost me a wrong classification on #1962 (five "wrong, finite" cases that were this, not an overflow).

Suggested first step

Decide (a) vs (b) with one probe: the same contraction with the weight tensor created the way the loader creates it (weights usage / the prepack the path expects). If that passes, the fix is the matcher's precondition plus a suite case built that way; if it fails, it is a kernel fault with a two-second repro.

Repro binary and logs: ~/wt/cl-hrxnan-1539/.probe/tbo-shape, .probe/p25.out (lane cl-hrxnan).

Claiming this, and the useful thing I can add immediately is that it is not the same fault as #1962 — the two are dense MUL_MAT family on HRX, and distinguishing them now saves whoever fixes them from
chasing one through the other.

	#1962	#1997 (this)
data	activations above the f16 max (65504)	ordinary uniform ±1, nothing special about magnitude
error	NaN or gross garbage	NMSE ≈ k/3, regular, at every shape
what it implies	an activation narrowing (f16 conversion)	a summation/contraction fault — the error scaling with `k` is the signature

NMSE ≈ k/3 independent of shape is the part worth reading twice: a precision loss does not scale with the contraction length, and a layout error (row/column swap) produces a different pattern — this
looks like a missing group factor or a group's partial sums being accumulated k/3 times. Both are the kind of fault where the output stays finite and plausible at small k and degrades as k grows,
which is why test-backend-ops with a real k catches it and a smoke test would not.

The check that would name it in one step: run the failing registration with k = 32 and k = 64 on data where the expected output is exactly computable by hand (one non-zero weight, one non-zero
activation). If the error doubles with k, it is a per-group accumulation; if it is constant, it is a scale. test-backend-ops gives the NMSE, this gives the mechanism, and the difference is one
16-element matrix.

I am not opening a PR: these are kernel registrations in the fork's dispatch (kernel internals private, and a fix needs a GPU to validate), and this box is at 150 G with other lanes holding tickets.
The triage above is the contribution — it means #1962 and #1997 are not duplicates of each other, which is the first thing a reviewer of either will wonder.

A lead from the matcher source for whoever takes this (reading, not a measurement): for Q4_K with output_size >= input_size / 4, match_packed_q8_1_x4_prefill_dispatch (dispatch_registration/common/dispatch-mul-mat.cpp) takes its use_f16 branch, whose own comment says the kernel "binds the weight in the packed Row64 layout". A weight layout is something the resident host-weight path materialises; a test-backend-ops weight is an ordinary device tensor in native Q4_K bytes. If the plan binds those bytes where the kernel expects Row64, the result is a different computation with a coherent error — which is what NMSE ≈ k/3, independent of m, looks like. The same question applies to the two Q6_K "prepacked"/"aligned" registrations. So the first probe is: which layout does the bound weight actually have in the failing case (the command-program dump, GGML_HRX_DUMP_COMMAND_PROGRAM_DIR, shows the binding), and does the registration decline when it cannot get the layout it asks for.

Not claiming this (I have no build and no device for HRX kernels, and this is kernel-internals material) — but the signature in the measurement is specific enough to narrow the search, and that is free.

The signature. NMSE ≈ k/3 at every shape, with uniform [-1,1] weights and activations. Two things follow from the arithmetic of that ratio.

1. "≈ k/3" is what you get when the reference sum and the kernel sum are uncorrelated, not when one is scaled. For a dot product of length k over uniform [-1,1], each product has variance 1/9 (var of x·y with independent uniforms is E[x²y²] = (1/3)(1/3)). So the reference value has variance k/9, and an independent (wrong) sum has variance k/9; their difference has variance 2k/9, and NMSE = |ref − got|² / |ref|² has expectation (2k/9)/(k/9) = 2. That is 2, not k/3 — so the plain "reads a different but equally random subset" hypothesis does not explain the measurement, and a fix based on it would chase indexing order.

What does give k/3: if the kernel accumulates k terms that each have magnitude ~1 instead of ~1/3 — i.e. it sums roughly k values whose variance is 1 rather than 1/9 — the difference variance becomes ≈ k/3 and NMSE ≈ (k/3)/(k/9) = 3… which is constant, not growing. So a constant NMSE of ~3 fits "wrong scaling of the same sum". A growing ratio k/3 fits something else: the kernel's magnitude scaling with k while the reference's does not — i.e. the kernel is summing over the wrong dimension, so its output magnitude grows where the reference's does not. For k = 8192, k/3 ≈ 2730, which is a factor of ~52 in amplitude — far too large for a dequantisation factor and far too structured for noise.

2. So the first question is not "which index is wrong" but "does the kernel's output magnitude grow with k?" One cheap check on an existing run tells them apart: for a fixed m,n and k = 512 vs 8192, is mean|got| roughly constant (correct) or 16× larger (wrong-dimension/k-dependent)? If it grows, the defect is in the reduction bound — an N- or M-stride used where the K loop bound belongs, or a bound taken from the wrong shape field, and q6_k_aligned_skinny sharing the symptom points at something the registrations have in common (the K-block loop's bound) rather than at each type's dequant.

3. The four registrations sharing one ratio strongly suggests one shared helper, not four bugs. test-backend-ops NMSE ≈ k/3 for q4_K, q5_K, q6_K and q6_k_aligned_skinny at all shapes means the defect predates the per-type dequant — it is in whatever walks the K blocks (the row/block loop shared by the K-quant family), which is why the aligned-skinny variant shows it too.

Why I am writing this instead of fixing it: the value here is the arithmetic, not the patch — k/3 excludes the hypothesis most people reach for first (permuted-but-valid reads) and points at a magnitude/reduction-bound defect, and that is checkable in a test-backend-ops run the owning lane already has a ticket pattern for. If the magnitude check above comes back constant, then my reasoning is wrong and the next hypothesis should be a missing scale application (a per-block d used once instead of per block, which would give a factor that does depend on the number of blocks and therefore on k) — that one is worth testing before touching the reduction bounds.

One process note: the issue says it reproduces on the unmodified CI bench build of main's pin at a5dc597b8, which makes it a live defect on main rather than a branch artefact — worth saying in whichever PR fixes it, since it means the gate that would have caught it (test-backend-ops in CI) is either not run for these types or run with data that hides it.

It is (a): the weight's buffer, not the kernels. I ran the probe the issue suggests: the same plain test_mul_mat case, with the quantised a put in test-backend-ops' own weights context (a GGML_BACKEND_BUFFER_USAGE_WEIGHTS buffer, the flag a model loader sets) and then without it.

Setup: one gpu-hold ticket on strixhalo. test-backend-ops.cpp from the pin a5dc597b8 was single-file compiled against an HRX build of that pin, with a ~30-line env-gated patch: TBO_MM_CASE=<type>,m,n,k runs one case, and TBO_MM_WEIGHTS=1 creates a in ctx_weights. The environment was the one your probe runs use (IREE_HAL_AMDGPU_LIBHSA_PATH= TheRock's libhsa, GGML_HRX_STRICT=0, …). Registrations are named by GGML_HRX_LOG_DISPATCH=1. Within each pair, the same registration ran both times.

case (type, m, n, k)	registration	plain buffer	WEIGHTS buffer
q4_K, 4096, 512, 4096	`common.mul_mat.q4_k_q8_1_x4_prefill`	FAIL	OK
q4_K, 2048, 512, 2048	`common.mul_mat.q4_k_q8_1_x4_prefill`	FAIL	OK
q6_K, 4096, 1, 4096	`common.mul_mat.q6_k_token1_final_projection_q8`	FAIL	OK
q6_K, 2048, 8, 4096	`common.mul_mat.q6_k_i8_prepacked`	FAIL	OK
q6_K, 4096, 8, 512	`common.mul_mat.q6_k_aligned_skinny`	FAIL	OK

So the four kernels compute the right answer on a weight prepared the way the loader prepares it, and serving is not affected by this. The defect is that their matchers claim a quantised tensor that no one prepared. In the test harness that turns into a silent wrong number instead of a decline.

Why the executor's guard doesn't catch it. command-program-executor.cpp:677 refuses "a packed weight layout … but its buffer is not flagged as weights", which is exactly this case. But it keys on access.transformed_read, and that is only set for a binding with a non-native layout (:546-559). A binding in the native layout sets native_read and goes straight through. The q6_K matchers bind the weight that way ({weight->id, 0, byte_count}), while their kernels read the loader's prepared rows (mul_mat_q6_k_packed_f16_wmma.loom: "0 = the loader's stored QQ6_K rows"). So an unflagged Q6_K buffer reaches a kernel that assumes a prepared one. I haven't yet traced why the q4_K prefill path, which does bind kQ4KPackedK256Row64Layout, also ran instead of refusing. That's the next read.

Fix (fork-side, for the next step):

These matchers (or the executor guard, for native-layout reads of a weight-form kernel) decline or refuse a weight whose buffer is not weights-flagged, as the :677 guard already intends for transformed layouts.
Give the suite the case it cannot have today: a test_mul_mat variant that builds a in the weights context. That's the same ~30 lines as the probe. These paths then get known-answer coverage instead of being unreachable by any passing case.

Answered by two measurements: it is (a) — the plan binds a weight layout that nothing produced for a bare test tensor; serving is correct.

1. What the failing cases bind (GGML_HRX_DUMP_COMMAND_PROGRAM_DIR, 19:11Z, weight binding from each program's command_shape; [4096 → 2048]):

case	kernel	weight binding (layout : … : native bytes)	result
q4_K, 512 tok	`ggml_mul_mat_q4_k_f16_wmma_prefill_wave32`	`q4k-packed-k256-row64`, 4718592 B of 4718592	NMSE 1367
q6_K, 8 tok	`ggml_mul_mat_q6_k_i8_prepacked_f16_wmma`	`q6k-i8-k32-row64`, 8978432 B of 6881280	NMSE 1380
q6_K, 1 tok	`ggml_mul_mat_f32_f32_wmma` (q8 activation)	`q6k-packed-k256-row64-scalerow`, 6881280 B	NMSE 1460
q5_K, 512 tok	`ggml_mul_mat_q5_k_iq4_xs_q8_1_x4_wmma_token256`	`ggml-native`, 5767168 B	passes

Every failing case asks for a repacked layout; the one that passes asks for the native one. The test's weight is a plain graph value holding native bytes, so the kernel interprets them as a layout they are not in. The q6_K/8-token case is worse than wrong: its binding is 8,978,432 bytes long over a 6,881,280-byte tensor, i.e. it reads about 2 MiB past the end of the weight.

2. Serving through the same registrations is right (Qwen3-4B-Q4_K_M, one batch of exactly n tokens, HRX vs -ngl 0, 19:15Z; the dispatch log lists q4_k_q8_1_x4_prefill, q6_k_token1_final_projection_q8, q6_k_i8_prepacked, q6_k_aligned_skinny, q6_k_prefill_wave32):

n=512 HRX: 63762(34.06) 1526(21.65) 279(17.26) 42747(16.87) 1112(16.18)
n=512 CPU: 63762(33.51) 1526(21.25) 279(16.96) 42747(16.48) 1112(15.85)
n=8   HRX: 34208(19.72) 26005(16.26) 198(15.94) …      CPU: 34208(19.64) 26005(16.45) 198(15.97) …
n=1   HRX: 3(10.26) 264(10.09) 12(9.97) …              CPU: 3(10.64) 12(10.21) 64(10.09) 264(9.96) …

Same top tokens, logits within a few tenths — the model's weights do get the layout these kernels expect.

So the defect is: these registrations match (and the plan records a layout request) for a weight that the layout machinery will not repack, instead of declining. In serving that never happens; in test-backend-ops it always does, which is why the paths cannot have a known-answer case — and with the q6_K i8 layout it is an out-of-bounds read in the test process. The fix is on the claim side (decline, or materialise the layout, when the weight is not one the resident-layout path owns), plus suite cases built so the layout is produced, so these kernels finally get known-answer coverage. I am taking that.

One refinement before anyone writes the fix, because it changes where the fix goes.

The executor already refuses this situation on paper: materialize_host_bindings (runtime/command-program-executor.cpp, the engine #969 block) logs "value N is read in the packed weight layout '…' but its buffer is not flagged as weights … refusing to run that kernel over unpacked weight bytes" when a transformed-layout read hits a binding without GGML_BACKEND_BUFFER_USAGE_WEIGHTS. That code is older than the build I measured (the file last changed in fork #270), and it did not fire in any of the failing cases — the kernels ran and returned NMSE ≈ k/3.

So the simple story ("an unflagged test tensor slips through") is not the whole of it. For the refusal to be skipped, materialize_weight must have been true, i.e. the test's weight binding counted as a flagged, read-only weight — and then the layout was supposed to be materialised and the kernel still read native bytes. Two candidates, neither verified:

the resident-layout path does not repack a device-resident source (the test tensor lives in an HRX buffer, not a host mapping) and binds it as is; or
it repacks into a slot the command does not end up bound to.

Either would also be reachable in serving whenever a flagged weight is device-resident and read through one of these registrations — except that the serving check above (Qwen3-4B, top-5 equal to the CPU at 512/500/8/1 tokens) says the shipped path gets it right, presumably because its weights arrive through the host-mapped route. That contrast (host-mapped: correct; device-resident: native bytes under a packed-layout kernel) is the thing to test next: the same test-backend-ops case with the weight in a host buffer flagged as weights. I have not run that.

Resolved by a three-way placement probe — and my reading above was wrong in an important way: nothing computes a wrong answer here. The executor refuses these programs, and test-backend-ops scores the never-written output.

One MUL_MAT, weight placed three ways, HRX vs the CPU backend, on the CI build of main (fa533bd21, 19:30Z; ~/wt/cl-hrxnan-1539/.probe/wprobe.cpp):

case → registration	weight in a device buffer, usage ANY	device buffer, usage WEIGHTS	host buffer, usage WEIGHTS
q4_K, 512 tok → `q4_k_q8_1_x4_prefill`	compute refused	NMSE 1.53e-05	(node runs on the CPU)
q6_K, 8 tok → `q6_k_i8_prepacked`	compute refused	NMSE 1.53e-05	(CPU)
q6_K, 1 tok → `q6_k_token1_final_projection_q8`	compute refused	NMSE 2.33e-05	(CPU)
q5_K, 512 tok → `q5_k_iq4_xs_q8_1_x4_prefill` (control)	NMSE 2.48e-05	NMSE 2.48e-05	(CPU)

with, in the refused arm, exactly the engine #969 line: graph_compute: value 0 is read in the packed weight layout 'q4k-packed-k256-row64' but its buffer is not flagged as weights (GGML_BACKEND_BUFFER_USAGE_WEIGHTS); refusing to run that kernel over unpacked weight bytes.

So:

The kernels are correct when the weight is flagged (1.5e-5 to 2.3e-5), which is also why serving is right. I withdraw "binds native bytes under a packed-layout kernel" and "reads ~2 MiB past the weight": the dump shows the planned binding; the executor never runs it.
The number I reported is the harness, not the kernel. test_mul_mat puts its weight in the ordinary buffer (usage ANY). The program is refused, the output tensor keeps the random [-1, 1] values initialize_tensors() gave it, and the comparison against the CPU's result comes out at k/3 + 1 — 171.7, 683.7, 1366.3, 2731.7 for k = 512, 2048, 4096, 8192, which is the table above to three digits. A refused compute is being reported as a numerical mismatch.

What is actually wrong, then — two things, both smaller than this issue's title:

No known-answer coverage for these four registrations. The harness already has the mechanism (use_weight_context() → a separate buffer flagged as weights), but test_mul_mat does not use it, so every case that reaches these registrations is a refusal. Cases that opt in would test the real kernels. I am writing those (test-only).
A failed graph_compute reads as "ERR = " in test-backend-ops. That cost me two wrong classifications today (here and on hrx: dense MUL_MAT returns NaN or wrong values when an activation exceeds the f16 range (33 of 56 known-answer cases at 9.1e4; the dense form of #1539) #1962). It should say the compute failed.

(A third, pre-existing and already known as the #1467 class: the registration claims a tensor it will refuse at execution instead of declining at match. Not new, not pursued here.)

Coverage PR: 1bit-MONSTER/llama.cpp#287 (test-only). test_mul_mat_resident_weight reaches each of the five packed-layout registrations with a flagged weight and all pass on the CI build of main (6/6 incl. the control, whole MUL_MAT 598/598); the same shapes through plain test_mul_mat still print ERR ≈ 1366 beside the executor's "refusing to run" line. Left on this issue: test-backend-ops should report a failed compute as a failed compute.

Correction to my comment above: the executor's guard does fire, and what I said about why it didn't was wrong.

I re-ran my plain-buffer probe with full stderr. Every case logs refusing to run that kernel over unpacked weight bytes (q6_K n=1, q6_K n=8, q4_K n=512: one refusal line each, ERR 1395 / 1398 / 682). So the k/3 + 1 error is the harness comparing an output tensor nothing wrote (its random fill) against the CPU, exactly as llama.cpp#287 describes. My first probe kept only the first log line matching "refus", which was the HRX0 GTT guardrail line, so I never saw the refusal. My statement that native-layout q6_K bindings "go straight through" the guard is wrong; disregard it.

The A/B result stands: with the weight in a WEIGHTS-usage buffer, all five cases pass. llama.cpp#287 adds those cases to the suite, and I verified it (6/6 on the named registrations, MUL_MAT 598/598).

Both parts are now up or in: the coverage cases merged as llama.cpp#287, and the reporting fix is llama.cpp#290 — a refused compute now prints ggml_backend_compare_graph_backend: graph compute failed at node 0 (MUL_MAT) (HRX0: status -1, CPU: status 0); nothing was compared where it used to print ERR = 1358. When #290 is pinned this issue can close.

Taking one slice here, as a second pair of hands (owner 21:40Z: every open issue closes before the release build): the engine pin bump that lands both fork halves.

Both fork PRs are merged (llama.cpp#287 → 69aac7fa2, #290 → 7cc3187b5), but engine main still pins 50d28600a, so neither is on main yet. I'll open bump llama.cpp to 7cc3187b5, which is exactly those two commits on top of the current pin. I'll gate it with a build of that fork commit and the new MUL_MAT resident-weight cases plus the refused-compute report, under gpu-hold.

Not taking, to avoid stepping on other lanes: the two later fork commits 139ccd741 (#1989, prefill attention pitch) and cf5febf04 (#1962, Q6_K f16 range). They belong to their own lanes' bumps and measurements. If either lane would rather fold this into one bump, say so and I'll close mine.

Changing my slice so it doesn't duplicate: engine#2033 (opened 21:51Z, Fixes #1989) moves the pin to the fork head 2293f95ea, which contains both halves of this issue (69aac7fa2 = #287, 7cc3187b5 = #290; 7cc3187b5 is 3 commits behind 2293f95ea and nothing diverges). So I'm not opening the separate 7cc3187b5 bump I announced above. I'll verify #2033 and post the result there.

The gate I already ran is evidence for this issue's part either way. One gpu-hold ticket, a Release build of the fork at exactly 7cc3187b5 against hrx-system 7f42369:

resident-weight cases 6/6, each on the registration it names (q4_k_q8_1_x4_prefill, q6_k_prefill_wave32, q6_k_i8_prepacked, q6_k_aligned_skinny, q6_k_token1_final_projection_q8, and the q5_k control);
whole MUL_MAT 598/598; ADD 32/32 with 0 compute-failure lines;
three refused plain-buffer cases (q4_K n=512, q6_K n=8, q6_K n=1): each prints graph compute failed at node 0 (MUL_MAT) (HRX0: status -1, CPU: status 0); nothing was compared, no ERR = line, and fails.

This issue can close when #2033 merges.

bump llama.cpp to 2293f95ea: prefill attention keeps its rate at long…

Closing: both halves are on engine main. #2033 merged at 22:30:59Z and moved the llama.cpp pin to 2293f95ea, which contains llama.cpp#287 (69aac7fa2, known-answer MUL_MAT cases with a resident weight, one per packed-layout registration) and #290 (7cc3187b5, a refused compute reported as one, not scored). Verified at that pin, one gpu-hold ticket: resident cases 10/10 on the registrations they name, MUL_MAT 602/602, and the three refused plain-buffer cases each print 'graph compute failed … nothing was compared' and fail (numbers on #2033).

Navigation Menu

Repository navigation

hrx: four Q4_K/Q6_K MUL_MAT registrations have no known-answer coverage - test_mul_mat's unflagged weight is refused, and test-backend-ops reports the refusal as an NMSE of k/3+1 #1997

What happens

Measurement

What I do not know

Why it matters either way

Suggested first step

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

hrx: four Q4_K/Q6_K MUL_MAT registrations have no known-answer coverage - test_mul_mat's unflagged weight is refused, and test-backend-ops reports the refusal as an NMSE of k/3+1 #1997

Description

What happens

Measurement

What I do not know

Why it matters either way

Suggested first step

Activity

bong-water-water-bong commented on Nov 10, 2026

bong-water-water-bong commented on Nov 10, 2026

bong-water-water-bong commented on Nov 10, 2026

bong-water-water-bong commented on Nov 10, 2026

bong-water-water-bong commented on Nov 10, 2026

bong-water-water-bong commented on Nov 10, 2026

bong-water-water-bong commented on Nov 10, 2026

bong-water-water-bong commented on Nov 10, 2026

bong-water-water-bong commented on Nov 10, 2026

bong-water-water-bong commented on Nov 10, 2026

bong-water-water-bong commented on Nov 10, 2026

bong-water-water-bong commented on Nov 10, 2026

bong-water-water-bong commented on Nov 10, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions