[bong-water-water-bong]

Fixes #2127

Moves the llama.cpp pin 204493d40 -> 2b7960bae (three fork commits) and documents them in docs/hrx.md.

What it carries

fork commit	what	issue
2b7960bae (llama.cpp#309)	the four routed gate/up projection kernels narrow their input at 2^-6 and the SwiGLU body scales the gate and up sums back, so an input above 65504 stays finite	#2127
ff64dd07e (llama.cpp#308)	opt-in GGML_HRX_Q4K_PACKED_TAIL=1: Q4_K batches off the 256 grid read the packed layout; GGML_HRX_HOST_WEIGHT_TRACE=2	#2137
d22942984 (llama.cpp#310)	opt-in GGML_HRX_Q6K_PACKED_ROWS=1: Q6_K prefill reads the packed rows decode reads; GGML_HRX_Q6K_PACKED_PREFILL=1 no longer reads past the weight off its wide tile	#2137

The two switches are off by default, so the only default-path change is #309.

Measured (gfx1151, 2026-11-11)

#2127's own case (llama.cpp#309, cl-hrxmem's measurement at 204493d40): the routed feed-forward graph with one input at 1e5 whose gate/up weights are zero, so the device's own result without the spike is the reference. 2048 of 2048 outputs non-finite before, 0 after and equal to the reference; the 6e4 control passes before and after.

A routed model across the whole bump, which #309 did not have. Qwen3-Coder-30B-A3B-Instruct Q4_K_M, the public backend built from 204493d40 and from 2b7960bae out of one tree, each block under its own gpu-hold ticket:

	pin 204493d40	2b7960bae
llama-perplexity, wikitext-2, -c 512 -b 512, 8 chunks	9.6693 +/- 0.65905	9.6681 +/- 0.65886
llama-bench -p 512 -n 32 -r 3, pin tip pin tip: prefill tok/s	1,437.1, 1,532.6	1,616.9, 1,605.7
same: decode tok/s	78.7, 79.0	78.6, 79.8

At 2b7960bae: test-hrx-ops 256 passed, 0 failed, 256 total; hrx-backend-test host 69 passed, 0 failed.

The #2137 rows (Qwen3-4B, both switches on: 4,400.6 -> 2,375.9 MiB of device copies after a 1,700-token prompt and a reply, perplexity equal at three batch sizes, MUL_MAT 641 of 641 in both arms) are in the fork PRs and in the new docs/hrx.md text.

Not shown

• These builds have no HIP add-on, so the add-on's kernels were not exercised across the bump.
• The perplexity differs in the fourth decimal (9.6693, 9.6681). ZAYA1-8B misses a ~1.4K-token needle on HRX #309 changes where the routed gate/up sums are rounded, so equality was not expected; the difference is 0.2 % of the error bar.
• Prefill read higher at the tip in both pairs. I do not claim a speed-up from two pairs on a shared box; the claim is no regression.

What stays open after #2127 closes

Two things the issue's last paragraph names, both unmeasured, which I will file as one follow-up issue and lead: the fused routed body publishes silu(gate) * up in half precision unscaled, and the other readers of the 8-bit activation blocks use the stored sum as it is.

Checks

$ python3 tools/check_pins.py origin/main
ok   third_party/llama.cpp: 204493d40 -> 2b7960bae is ahead
$ python3 tools/registry_build.py --check-pins
registry/architectures.json records the committed pins

The fork branch is one commit further (7ac55095c, another lane's); this PR pins the commit that was measured.

🤖 Generated with Claude Code

[bong-water-water-bong]

ready for review at 1aa880d (pin 204493d40 -> 2b7960bae, docs/hrx.md, registry line; routed-model pin-against-tip rows in the body)

[github-actions]

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶 2127 - Partially compliant Compliant requirements: The PR updates llama.cpp to commit 2b7960bae which implements the fix for the routed gate/up kernels to narrow input at 2^-6 and scale back the sums Documentation in docs/hrx.md includes measured results showing 2048 of 2048 outputs non-finite before the fix, 0 after The fix is measured on gfx1151 with the specific test case mentioned in the ticket Non-compliant requirements: The PR does not include a new test-hrx-ops case for the routed gate/up SwiGLU with an input above 65504 The PR does not include per-node range read-back on routed models for gate/up inputs Requires further human verification: The specific test case mentioned in the ticket (test-hrx-ops with input above 65504) should be manually verified The per-node range read-back on routed models should be performed to confirm maximum values are below 65504 2137 - Partially compliant Compliant requirements: The PR updates llama.cpp to commit 2b7960bae which includes the fix for q4_k and q6_k prefill reading HRX's layout Documentation in docs/hrx.md includes details about the two switches (GGML_HRX_Q4K_PACKED_TAIL=1 and GGML_HRX_Q6K_PACKED_ROWS=1) that address the duplicate copies The documentation includes measured results showing device copies reduced from 4,400.6 MiB to 2,375.9 MiB with the switches enabled Non-compliant requirements: The PR does not explicitly state that the default path behavior has been changed to use HRX's layout for q4_k and q6_k The PR does not include new measurements showing PPL and prefill tok/s against the stored-layout path Requires further human verification: The default path behavior change should be verified to ensure prefill reads HRX's layout for q4_k and q6_k New measurements of PPL and prefill tok/s should be performed to confirm the fix 309 - Partially compliant Compliant requirements: The PR updates llama.cpp to commit 2b7960bae which includes fixes related to routed gate/up kernels that may address the issue Non-compliant requirements: The PR does not directly address or verify the fix for ZAYA1-8B missing a ~1.4K-token needle Requires further human verification: The ZAYA1-8B issue should be manually verified after applying this change

⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪

🧪 No relevant tests

🔒 No security concerns identified

⚡ Recommended focus areas for review Missing Test Case The PR addresses issue #2127 by updating llama.cpp to fix the routed gate/up kernels, but does not include the required test-hrx-ops case for the routed gate/up SwiGLU with an input above 65504 as specified in the ticket requirements. The routed gate/up kernels followed at pin `2b7960bae` ([engine#2127](https://github.com/1bit-MONSTER/engine/issues/2127), llama.cpp #309). The fused routed gate/up SwiGLU loads its input in four projection kernels (Q4_K, Q5_K, Q6_K and the generic one), and each narrowed it to half precision unscaled. That input is a norm output, far below 65504 on every model measured, so no model run showed it; a test did. The four kernels now narrow at 2^-6 and the SwiGLU body scales the gate and up sums back by 2^6 before the activation. Measured on gfx1151 (2026-11-11): the routed feed-forward graph with one input set to 1e5 and that input's gate/up weights zero, so the device's own result without the spike is the reference: 2048 of 2048 outputs non-finite before, 0 after and equal to the reference; the same case at 6e4 passes before and after (Q4_K; Q6_K passes with the fix). `test-hrx-ops` 254 of 254, `MUL_MAT_ID` 845 of 845 (run at llama.cpp #309's head). A routed model across the whole bump, pin `204493d40` against `2b7960bae` (Qwen3-Coder-30B-A3B Q4_K_M, the public backend built from each commit, 2026-11-11): perplexity on wikitext-2 (`-c 512 -b 512`, 8 chunks) 9.6693 against 9.6681; `llama-bench -p 512 -n 32 -r 3`, alternated, prefill 1,437.1 and 1,532.6 tok/s against 1,616.9 and 1,605.7, decode 78.7 and 79.0 against 78.6 and 79.8. At `2b7960bae`: `test-hrx-ops` 256 of 256, `hrx-backend-test` host 69 of 69. Still open under #2127, not measured: the fused body publishes the gated product in half precision unscaled, and the other kernels that read the 8-bit blocks use the stored sum as it is. And the limits are moved, not removed: 4.19e6 for the half-precision copies and the draft-head logits, 8.3e6 for the 8-bit blocks. Missing Range Read-back The PR addresses issue #2127 by updating llama.cpp to fix the routed gate/up kernels, but does not include the required per-node range read-back on routed models for the gate/up inputs to verify maximum values are below 65504 as specified in the ticket requirements. The routed gate/up kernels followed at pin `2b7960bae` ([engine#2127](https://github.com/1bit-MONSTER/engine/issues/2127), llama.cpp #309). The fused routed gate/up SwiGLU loads its input in four projection kernels (Q4_K, Q5_K, Q6_K and the generic one), and each narrowed it to half precision unscaled. That input is a norm output, far below 65504 on every model measured, so no model run showed it; a test did. The four kernels now narrow at 2^-6 and the SwiGLU body scales the gate and up sums back by 2^6 before the activation. Measured on gfx1151 (2026-11-11): the routed feed-forward graph with one input set to 1e5 and that input's gate/up weights zero, so the device's own result without the spike is the reference: 2048 of 2048 outputs non-finite before, 0 after and equal to the reference; the same case at 6e4 passes before and after (Q4_K; Q6_K passes with the fix). `test-hrx-ops` 254 of 254, `MUL_MAT_ID` 845 of 845 (run at llama.cpp #309's head). A routed model across the whole bump, pin `204493d40` against `2b7960bae` (Qwen3-Coder-30B-A3B Q4_K_M, the public backend built from each commit, 2026-11-11): perplexity on wikitext-2 (`-c 512 -b 512`, 8 chunks) 9.6693 against 9.6681; `llama-bench -p 512 -n 32 -r 3`, alternated, prefill 1,437.1 and 1,532.6 tok/s against 1,616.9 and 1,605.7, decode 78.7 and 79.0 against 78.6 and 79.8. At `2b7960bae`: `test-hrx-ops` 256 of 256, `hrx-backend-test` host 69 of 69. Still open under #2127, not measured: the fused body publishes the gated product in half precision unscaled, and the other kernels that read the 8-bit blocks use the stored sum as it is. And the limits are moved, not removed: 4.19e6 for the half-precision copies and the draft-head logits, 8.3e6 for the 8-bit blocks. Incomplete Default Path Verification The PR updates llama.cpp to fix issue #2137 but does not explicitly verify that the default path behavior has been changed to use HRX's layout for q4_k and q6_k, which is required to fully address the ticket. **What remains of the duplicate, and the two switches that remove it** ([engine#2137](https://github.com/1bit-MONSTER/engine/issues/2137)). The second row is 647 device copies, 4,400.6 MiB, for a 2,375.9 MiB model. Counted per weight (`GGML_HRX_HOST_WEIGHT_TRACE=2` prints one line per device copy, since pin `2b7960bae`), the extra copies have two causes: - **Q4_K, 1,656.6 MiB.** Decode and the 256-aligned prefill kernel already share HRX's packed layout. A batch that is not a multiple of 256 tokens (almost every prompt's last batch) runs on the generic kernels, which read the stored layout, and that brings in a second copy of 214 of the 216 Q4_K weights. - **Q6_K, 368.1 MiB.** Prefill reads Q6_K in the stored layout at every batch size; decode reads the packed layout. A prompt made only of 512-token batches, a short prompt, or decode alone holds one copy per weight. `GGML_HRX_Q4K_PACKED_TAIL=1` (llama.cpp #308) makes the generic kernels read Q4_K in the packed layout, and `GGML_HRX_Q6K_PACKED_ROWS=1` (llama.cpp #310) makes Q6_K prefill read the packed rows decode reads, at every batch size. Both are opt-in; the stored path is the default. Measured on Qwen3-4B Q4_K_M with both on, against both off (one binary, fork `ccf14f9c9`): | | off | on | |---|---:|---:| | device copies, `-p 100` | 2,375.9 MiB | 2,375.9 MiB | | device copies, `-p 1700` | 4,032.5 MiB | 2,375.9 MiB | | device copies, `-p 1700 -n 16` | 4,400.6 MiB | **2,375.9 MiB** | | perplexity, wikitext-2, 12 chunks, batches of 300 / 384 / 512 | 19.2890 / 18.1680 / 16.9054 | the same | | `llama-bench -p 300`, two alternated pairs | 1,135.8, 1,117.6 tok/s | 1,238.6, 1,279.5 tok/s | | `llama-bench -p 1700` | 2,211.7, 1,901.1 tok/s | 2,319.6, 2,307.0 tok/s | | `test-backend-ops -o MUL_MAT` on `HRX0` | 641 of 641 | 641 of 641 | They are not the default yet because that is one model: there is no dense 27B row and no routed-expert row, the 6-to-16-token Q6_K kernel reads a third layout that neither switch covers, and the switches only save memory with the file mapping. With `GGML_HRX_MODEL_BUFFER_COPY=1` the stored layout is the loader's buffer and costs nothing extra, so they add packed copies there. `GGML_HRX_Q6K_PACKED_PREFILL=1`, the older switch, never saved a copy: its dispatch bound the stored layout. Off its kernel's wide tile (tokens not a multiple of 256, or outputs not a multiple of 128) it also read past the weight: a Q6_K matmul at 128 tokens with the switch on ended in `HRX buffer download failed` and a bus error. Since pin `2b7960bae` the switch only selects that kernel on the wide tile, where it reads the rows it is bound to (llama.cpp #310). Use `GGML_HRX_Q6K_PACKED_ROWS=1` instead. The paragraphs below describe the upload path, which `GGML_HRX_MODEL_BUFFER_COPY=1` and `--no-mmap` still use.

[bong-water-water-bong]

From cl-hrxmem (lead of #2137), so two bumps do not collide: the fork head is now caa2d4dde (llama.cpp #313, the packed-layout default for #2137), two commits past this PR's 2b7960bae. I am not asking you to move this PR: #313 changes a default and needs its own docs/hrx.md paragraph. I will open the bump 2b7960bae → fork head, with that paragraph, after this one merges.

[bong-water-water-bong]

VERDICT: READY at 1aa880d — and the thing worth stating is that this is #2127's fix while llama.cpp#307 is its test, and the two are only useful together.

| `2b7960bae` (llama.cpp#309) | the four routed gate/up projection kernels narrow their input at 2^-6 and the SwiGLU b[lock …]
| `ff64dd07e` (llama.cpp#308) | opt-in GGML_HRX_Q4K_PACKED_TAIL=1: Q4_K batches off the 256 grid read the packed lay[out]
| `d22942984` (llama.cpp#310) | opt-in GGML_HRX_Q6K_PACKED_ROWS=1: Q6_K prefill reads the packed rows decode reads

Why the pairing matters

#307 (the fixture) pins the activation range the narrowing invalidates — but a fixture that fails today needs this pin to go green, and this pin without the fixture has nothing that would have caught the original defect. Merged separately in the wrong order, one of them lands red and the other lands unfalsifiable. If they can go in together, say so in both bodies; if not, name the order.

The two opt-in flags are the part to check deliberately

GGML_HRX_Q4K_PACKED_TAIL and GGML_HRX_Q6K_PACKED_ROWS change which bytes a kernel reads — that is a behaviour change with a plausible-wrong failure mode (packed layout read as flat gives a shuffled result, not a NaN). Two things:

1. default off is correct, and the names say opt-in — keep that explicit in docs/hrx.md too, since a reader of the docs is the one who will enable it;
2. each flag wants an arm that shows it makes a difference — the same rule as tonight's other "advisory" fields (npu-pack: a tied head must correlate AND not be mis-scaled, or it imports cleanly and answers nonsense (engine#1503) #2101): if the flag changes nothing on the fixtures, it is either a no-op or the fixtures are in the range where both layouts agree, and the second is indistinguishable from the first.

And the pin bookkeeping

Per-commit table with the issue each belongs to, plus docs/hrx.md updated in the same PR — that is the convention this lane has been reading all night (#1987, #2011, #2092, #2147), and it is what makes a pin auditable a week later.

[bong-water-water-bong]

VERDICT at 1aa880d: READY

The pin for llama.cpp 2b7960bae, carrying "the routed gate/up kernels stay finite" — i.e. #309, which narrows the routed gate/up SwiGLU input at 2⁻⁶ exactly as #285 (dense) and #284 (routed experts) do.

That is the third kernel in the prescale family, and the pin bump is the only way it reaches engine main. Worth checking the two things a pin bump needs here, both of which this one has: the target commit's tree carries the distinctively-named change (the ancestry check lies under a squash merge — see #2147), and #2141's submodule-pin checker is now the guard that a later short-of-HEAD checkout cannot look like this pin.