[bong-water-water-bong]

Refs 1bit-MONSTER/engine#2137. Second part, after #308 (Q4_K). Opt-in: the default path does not change. It also fixes a fault in an existing opt-in.

What was left after #308

Qwen3-4B Q4_K_M (216 q4_K, 37 q6_K; 2,375.9 MiB), GGML_HRX_Q4K_PACKED_TAIL=1, after a 1,700-token prompt and a 16-token reply: 2,744.0 MiB of device copies. The extra 368 MiB is 35 q6_K weights, each held twice: prefill reads q6_K in the stored layout at every batch size, decode reads it in the packed row layout.

The existing GGML_HRX_Q6K_PACKED_PREFILL=1 does not change that. Its dispatch selects a different kernel but binds the stored layout, and on 256-aligned batches that kernel reads the stored rows (measured: registration common.mul_mat.q6_k_prefill_wave32 binds ggml-native with the switch on).

The change

GGML_HRX_Q6K_PACKED_ROWS=1: prefill reads q6_K in the packed rows decode reads, at every batch size.

• motifs/dequant.loom: ggml_q6k_packed_row64_f32_vector4, four values from the packed layout (layout as written by materialize_q6_k_packed_k256_row64_scalerow). The tiled loaders of the generic matmul core and of the fused gate/up kernel take it as weight format 46, next to hrx: Q4_K batches off the 256 grid can read the packed layout, so a weight keeps one device copy (engine #2137, opt-in) #308's format 44.
• motifs/q6_k_f16.loom: ggml_q6k_packed_row64_f16_pair, the 16-wide pair for the aligned f16 prefill kernel. Same arithmetic as ggml_q6k_f16_pair, other addresses.
• motifs/mul_mat_quantized_f16_prefill.loom (the kernel's wide tile) and ops/mul_mat_q6_k_packed_f16_wmma.loom (its ordinary body, new weight_layout config, 2 = packed rows) call it.
• dispatch: the aligned q6_K prefill, the plain matmul, the post-op chains and the gate/up pair (now per weight, so a q4_K/q6_K pair works) bind the packed layout when the switch is on and the weight is in a weights buffer. The switch lives with hrx: Q4_K batches off the 256 grid can read the packed layout, so a weight keeps one device copy (engine #2137, opt-in) #308's in dispatch-q4k-packed-tail.h.

A fault in GGML_HRX_Q6K_PACKED_PREFILL=1, measured and fixed

Off its wide tile (tokens not a multiple of 256, or outputs not a multiple of 128) that switch's kernel reads 17,536-byte int8 tiles, while the dispatch binds the 13,440-byte stored rows. It read past the weight. Before this PR:

GGML_HRX_Q6K_PACKED_PREFILL=1 test-backend-ops test -b HRX0 -o MUL_MAT -p "type_a=q6_K.*weight=resident"
  MUL_MAT(type_a=q6_K,m=2048,n=128,k=4096,weight=resident):
buffer_get: HRX buffer download failed
Bus error (core dumped)                                           rc 135, 9 cases OK before it

The switch now selects that kernel only on the wide tile, where it reads the rows it is bound to. After: the same command, 28 OK, 0 FAIL, rc 0. The 128- and 384-token cases that found it are in this PR.

Measured at ccf14f9c9 (gfx1151, Qwen3-4B Q4_K_M, each block under a gpu-hold ticket)

"on" is GGML_HRX_Q4K_PACKED_TAIL=1 GGML_HRX_Q6K_PACKED_ROWS=1.

	off	on
device copies, -p 100	2,375.9 MiB	2,375.9 MiB
device copies, -p 1700	4,032.5 MiB	2,375.9 MiB
device copies, -p 1700 -n 16	4,400.6 MiB	2,375.9 MiB (one copy per weight)
llama-perplexity, wikitext-2, 12 chunks, -c 300 -b 300	19.2890	19.2890
same, -c 384 -b 384	18.1680	18.1680
same, -c 512 -b 512	16.9054	16.9054
test-backend-ops -o MUL_MAT on HRX0	641 OK, 0 FAIL	641 OK, 0 FAIL

llama-bench -p 300,512,1700 -n 16 -r 4, off on off on (tok/s; this block was run on the same sources one rebase earlier):

	off	on	off	on
prefill 300	1,135.8	1,238.6	1,117.6	1,279.5
prefill 512	3,172.1	3,126.1	3,145.1	3,292.5
prefill 1,700	2,211.7	2,319.6	1,901.1	2,307.0
decode	74.9	67.8	74.1	74.6

Prefill at 300 tokens is faster with the switches on, at 512 it is within the spread, at 1,700 it is faster. Decode binds the same packed copy in both arms; the 67.8 is one arm of four ([66 66 69 70]) and the other "on" arm reads 74.6, so I take it as the box, not the change, but it is one pair.

hrx-backend-test host: 69 passed, 0 failed. The matmul suite includes 14 new weight=resident q6_K cases (22 to 384 tokens, an output that is not a multiple of 128).

Not covered

• One model. No dense 27B row and no routed-expert row, so neither switch is proposed as a default here.
• The fused gate/up path with a packed q6_K weight has no op-level case (those cases use ordinary buffers); on this model the gate/up weights are q4_K.
• The 6-to-16-token q6_K kernel reads a third layout (int8 tiles). A model that prefills in such batches would still hold a second copy of those weights; not measured.
• As in hrx: Q4_K batches off the 256 grid can read the packed layout, so a weight keeps one device copy (engine #2137, opt-in) #308: the switches save memory only with hrx: keep the model weights in the loader's file mapping instead of a device buffer (engine #856) #304's file mapping. With GGML_HRX_MODEL_BUFFER_COPY=1 they add packed copies.

🤖 Generated with Claude Code

[bong-water-water-bong]

ready for review at ccf14f9 (opt-in; also restricts GGML_HRX_Q6K_PACKED_PREFILL to the tile where its kernel is correct; device and host rows in the body)

[bong-water-water-bong]

VERDICT at ccf14f9: READY

The Q6_K counterpart to #308, and it comes with a confession worth having: "It also fixes a fault in an existing opt-in." A follow-on that reports a defect in the flag it is extending is more valuable than one that only adds a case — the first opt-in path was shipped with a bug, and the second part found it because extending it meant reading it.

Opt-in again, with "the default path does not change", and filed against ggml-org#2137 — the residual ggml-org#856 deliberately left. That is now the second part of a residual being taken in order (#308 Q4_K, this Q6_K), which is what makes the "document the duplicate and track it" decision on ggml-org#856 defensible rather than a way of deferring.

The diff replacing the inline packed_tail conjunction with a helper continues the right direction: what was four && clauses on two formats is becoming a named predicate, which is where the shape decision belongs.

[bong-water-water-bong]

VERDICT: CHANGES at ccf14f9

The gate is right and the arms miss exactly the half that produced the bus error.

The new gate is token_count % 256 == 0 && match.output_size % 128 == 0 — two conditions, AND. The reported fault was:

MUL_MAT(type_a=q6_K,m=2048,n=128,k=4096,weight=resident):  buffer_get: HRX buffer download failed / Bus error, rc 135

m=2048 is the output size → 2048 % 128 == 0 ✔. n=128 is the batch → 128 % 256 != 0 ✘. So the failing case is "outputs aligned, tokens not" — the first condition was satisfied and the second
was the one that had to hold.

The arms you added are n ∈ {22, 100, 164, 300} for q6_K. Measured:

n	n % 256	n % 128	covers which half of the AND
22	22	22	neither condition satisfied
100	100	100	neither
164	164	36	neither
300	44	44	neither
128	128	0	outputs aligned, tokens not ← the reported case
384	128	0	same half, larger

Every arm sits in the "neither" cell. A dispatch that tested only token_count % 256 == 0 and dropped the output_size % 128 conjunct would pass all four — and that is the conjunct the
reported case exercised. The AND is only tested in one direction today.

Ask: add n = 128 (the exact reported geometry) and n = 384 for q6_K. Two lines, and they are the cheapest regression pin available for a fault that cost a Bus error and a core
dump. If m also has an off-128 variant worth covering, m = 2048, n = 128 is the reported one and m = 2100 would be its counterpart.

What I verified and like: the fix selects the from_packed kernel only when the tile is genuinely wide ("the from_packed kernel reads the stored rows on its wide tile (tokens a multiple of
256, outputs of 128) and 17,536-byte int8 tiles on every other tile, while this dispatch binds the stored layout"), which is the correct shape — the binding and the kernel's expectation
agree or the kernel is not chosen. Opt-in default unchanged (GGML_HRX_Q6K_PACKED_ROWS=1), and the appended arms for q4_K at n ∈ {6, 22, 164, 300} cover the same off-grid batch question
for the other type.

Measured, not run: the coverage arithmetic above. The device run (28 OK vs 9 OK then rc 135) is yours and needs GGML_HRX=ON.