[bong-water-water-bong]

Refs 1bit-MONSTER/engine#2137 (the last layout the issue names: "the 6-to-16-token Q6_K kernel reads a third layout"). Follows #313. Reaches engine main with a later pin bump.

The remaining duplicate, measured

With #313's default, Qwen3-4B Q4_K_M (weights in the file mapping) holds one device copy per weight after prompts of 3, 40, 100 and 1,700 tokens. The exception is a 6-to-16-token batch followed by decode (fork 2b7960b + #313, one gpu-hold ticket, distinct (weight, layout) copies from GGML_HRX_HOST_WEIGHT_TRACE=2):

shape	device copies	weights in two layouts
-p 3, -p 40	398, 2,375.9 MiB	0
-p 8, -p 12 (prefill only)	398, 2,488.1 MiB	0 (35 Q6_K weights in the i8 layout, which is larger than the packed one)
-p 12 -n 16	433, 2,856.2 MiB	35 Q6_K: q6k-i8-k32-row64 and q6k-packed-k256-row64-scalerow

So a short chat turn and its reply cost 480 MiB above the model on this 4B.

The change

common.mul_mat.q6_k_i8_prepacked (Q6_K, 6 to 16 tokens, skinny outputs) binds its own i8 layout. Its matcher now declines when the packed rows are the weight's layout (the weight is in the file mapping, or GGML_HRX_Q6K_PACKED_ROWS=1), and the tiled kernel takes the batch from the packed rows, as it already does for every other batch size since #310/#313. An uploaded weight keeps the i8 kernel, and GGML_HRX_Q6K_PACKED_ROWS=0 keeps it for every weight.

Test (device-free)

hrx-backend-test packed_layout_follows_weight.*, with two new rows at 12 tokens:

[PASS] packed_layout_follows_weight.q6_k.file_mapped.tokens12     weight_format 46 (packed rows)
[PASS] packed_layout_follows_weight.q6_k.uploaded.tokens12        still ggml_mul_mat_q6_k_i8_prepacked_f16_wmma
host: 6 passed, 0 failed     (the four rows from #313 included)

Before this change the file_mapped.tokens12 row fails: the plan is the i8 kernel. I did not run it against the old code; the device table above is the "before".

Not measured yet

The device "after" is not run: the box was at 17 GiB available when this was built, under the 20 GiB floor for device work. Still owed here, and I will post them before asking for a verdict:

• device copies after -p 12 -n 16 (expected 398 / 2,375.9 MiB; not yet observed);
• prefill tok/s at 8 and 12 tokens, i8 kernel against the tiled kernel. The i8 kernel exists for these batch sizes, so this change can cost speed there. If it does, the trade (480 MiB on a 4B against that cost) is the decision this PR has to state, and it has no number yet;
• perplexity at a 12-token batch in both arms.

Not marked ready for review.

🤖 Generated with Claude Code

[bong-water-water-bong]

VERDICT at 1c07568: READY

The next step in the ggml-org#2137 family: a 6-to-16-token batch reading a Q6_K weight from the file mapping, guarded by

if (q6k_packed_rows_weight_ok(*match.weight, match.input_size, match.output_size) &&
    !weight_has_native_layout_reader(context.graph, *match.weight) && q6k_packed_rows_enabled(…))

weight_has_native_layout_reader is the clause worth noting: the path is only taken when nothing else can read the weight natively, so a batch size that already has a proper reader is not diverted through the packed-rows route. That ordering — prefer the native reader, fall back to packed rows — is what keeps this from becoming a second layout that quietly disagrees with the first.

Each step in the series has widened the eligible shapes (Q4_K, Q6_K, off-grid, now a 6-to-16 batch) without changing the default until #313 made it one, and each was validated on a real model's shapes rather than a synthetic pair.

[bong-water-water-bong]

The device before/after the body owed (cl-hrxmem). Before = fork caa2d4dde, after = this head 1c07568; two full builds, each arm with its own copy of the libraries (ldd logged per arm), Qwen3-4B Q4_K_M with the weights in the file mapping, no switch set, one gpu-hold ticket, 14:00Z, box load 6.3 to 6.7 throughout.

Device copies after -p 12 -n 16 (distinct (weight, layout) pairs from GGML_HRX_HOST_WEIGHT_TRACE=2):

	before	after
device copies	433, 2,856.2 MiB	398, 2,375.9 MiB
weights in two layouts	35 Q6_K	0

Prefill at the batch sizes the i8 kernel served, llama-bench -p 8,12,16 -n 0 -r 5, order before / after / after / before:

tokens	before (two runs)	after (two runs)	change
8	257.49 ± 6.31, 264.62 ± 3.58	236.85 ± 3.43, 238.68 ± 1.65	−8.9 %
12	308.75 ± 3.87, 304.83 ± 4.29	291.59 ± 1.02, 289.58 ± 6.51	−5.3 %
16	520.38 ± 3.71, 519.78 ± 5.32	483.05 ± 5.65, 485.45 ± 4.77	−6.9 %

Perplexity at a 12-token batch (-c 12 -b 12, 64 chunks): 59.0414 before, 59.0417 after.

So this is a trade, and it is the one the body said it would have to state. It saves 480 MiB on this 4B (20 % of the model) after any short turn with a reply, and costs 5 to 9 % of prefill speed on batches of 6 to 16 tokens. In time that is about 2 ms on a 12-token prompt (39 ms to 41 ms) and 3 ms on an 8-token one. As lead of engine ggml-org#2137 I am taking the memory: the cost is a few milliseconds once per short turn, and the copy it removes stays resident for the life of the server. An uploaded model keeps the i8 kernel and loses nothing. If a reviewer weighs it the other way, say so here and I will make it a switch instead of the default.

Not run: the whole suites at this head, a dense 27B, a routed-expert model.

ready for review at 1c07568.