[bong-water-water-bong]

Refs 1bit-MONSTER/engine#2137 (the remainder of ggml-org#856). Opt-in: the default path does not change.

The cause, measured

Since #304 the weights stay in the loader's file mapping and each layout HRX reads is a device copy of its own. Counted with one trace line per copy (added here, GGML_HRX_HOST_WEIGHT_TRACE=2) on Qwen3-4B Q4_K_M (216 q4_K, 37 q6_K; 2,375.9 MiB), fork tip 204493d40, one fresh llama-bench process per shape:

shape	device copies	weights held in two layouts
decode only, -p 512, -p 1024, -p 100	2,375.9 MiB each	none
-p 1700 (3 x 512 + 164)	4,032.5 MiB	214 q4_K (stored + packed)
-p 1700 -n 16	4,400.6 MiB	214 q4_K, 35 q6_K

Decode and the 256-aligned prefill kernels already share the packed row layout of a q4_K weight. The second copy comes from the batch that is not a multiple of 256 tokens (a prompt's remainder): it runs on the generic tiled kernels, which read the stored layout.

The change

GGML_HRX_Q4K_PACKED_TAIL=1 makes the tiled paths read q4_K in the packed row layout:

• kernels: a format-44 branch in the tiled loader of the generic matmul core (shared by the plain matmul and the post-op chains) and in the two loaders of the fused gate/up kernel. It calls the packed-row reader the low-token path already uses and narrows to f16. 26 lines of Loom in two files.
• dispatch: plain matmul, post-op chains, fused gate/up bind the packed layout when the switch is on, the batch has more than 5 tokens, the layout's own preconditions hold (input a multiple of 256, output of 64, no alias) and the weight is in a weights buffer. Not for the split-k variant.
• dispatch-q4k-packed-tail.h: the switch and the weight predicate, shared by the two dispatch files. It logs which path it took, once.
• GGML_HRX_HOST_WEIGHT_TRACE=2: one line per device copy made (id, offset, layout, type, bytes).

Why only weights buffers: the executor refuses a packed read of a tensor in an ordinary buffer. My first device run had that: three MUL_MAT(type_a=q4_K, n=22, ..., magnitude=...) cases failed with value 0 is read in the packed weight layout ... but its buffer is not flagged as weights. With the predicate those cases keep the stored layout.

Measured at c641c84d4 (gfx1151, each block one gpu-hold ticket)

Qwen3-4B Q4_K_M, switch off -> on:

	off	on
device copies, -p 100	2,375.9 MiB	2,375.9 MiB
device copies, -p 1700	4,032.5 MiB	2,375.9 MiB (216 of 216 q4_K packed only)
device copies, -p 1700 -n 16	4,400.6 MiB	2,744.0 MiB (the rest is the 35 q6_K, not touched here)
llama-perplexity, wikitext-2, 12 chunks, -c 300 -b 300	19.2890 +/- 1.66255	19.2890 +/- 1.66255
same, -c 512 -b 512 (control, aligned)	16.9054 +/- 1.08520	16.9054 +/- 1.08520
llama-bench -p 300 -n 0 -r 4, off on off on	851.5, 857.5 tok/s	963.1, 963.6 tok/s
llama-bench -p 1700 -n 0 -r 4, warm repetitions	1,738 1,748 1,738 / 1,730 1,732 1,745	1,778 1,786 1,783 / 1,777 1,764 1,780
same, first repetition	1,511 / 1,366	1,612 / 1,627

test-backend-ops test -b HRX0 -o MUL_MAT: 627 cases OK, 0 FAIL with the switch off, and the same with it on. That includes ten new weight=resident q4_K cases at 6, 22, 164 and 300 tokens (and the model's two feed-forward shapes at 164), which take the packed path with the switch on (the log line says read the PACKED layout). hrx-backend-test host: 69 passed, 0 failed.

What is not covered

• The fused gate/up kernel has no op-level case with a weight in a weights buffer, so its packed path is exercised only by the model rows above (perplexity equal to four decimals at an unaligned batch, 70 gate/up weights going from two copies to one).
• One model and one format mix. No dense 27B row, no MoE row.
• q6_K is unchanged: it still reads stored for prefill and packed for decode (368 MiB on this model).
• The switch only saves memory with hrx: keep the model weights in the loader's file mapping instead of a device buffer (engine #856) #304's mapping. With GGML_HRX_MODEL_BUFFER_COPY=1 the stored layout is the loader's buffer and costs nothing extra, so the switch adds packed copies there (measured on a pre-hrx: keep the model weights in the loader's file mapping instead of a device buffer (engine #856) #304 base: a 100-token prompt went from 4 packed copies to 148). That is why it is opt-in here and why a later default has to follow the mapping mode.

🤖 Generated with Claude Code

[bong-water-water-bong]

ready for review at c641c84 (opt-in, default path unchanged; device and host rows in the body)

[bong-water-water-bong]

VERDICT at c641c84: READY

The remainder of ggml-org#856, made opt-in — "the default path does not change" — for Q4_K batches that are off the 256 grid and can read past the packed layout: const bool packed_tail = use_wmma && match.gate_format == CommonMulMatWeightFormat::Q4K && …, with the dispatch in its own dispatch-q4k-packed-tail.h.

Two things make this the right shape for a follow-on to a release blocker: it cannot make the shipped path worse (opt-in, guarded, new header), and it is filed against ggml-org#2137, the tracked residual that ggml-org#856's fix deliberately left behind. A fix that names its own remainder is what let ggml-org#856 close on time; this is that remainder being taken seriously rather than forgotten.