[bong-water-water-bong]

Fixes #2159
Fixes #2293
Refs #572 #679

The engine pins b0ed2060e413754cacde6f6d03ae60b46491b25a — ten commits past fd69590c. The previous body said "all three merged today"; the review at bd8fe4b was right that this understated what a reader is being asked to accept, so here is the whole range, grouped by what each commit is.

The two fixes this PR closes

commit	what it does	closes
aa25474	#68 — the prefill GEMM keeps activations inside the f16 range (2^-6 prescale, undone before the epilogue)	#2159
dccd024	#70 — dense prefill and small-batch K-quant activations finite above the f16 range	#2293 (part 2)

Both are visible in the fetched source, not just in their titles:

// hrx_prefill_gemm.hip  (#68)
// ... without it an activation above the f16 range became inf and the f32 result NaN
//     (MUL_MAT_ID activation_scale=1e5 against the CPU)
static constexpr float kXs = 0.015625f;  static constexpr float kXsInv = 64.0f;

// hrx_dense_prefill.hip  (#70)
// ... an activation above 65504 became inf in the narrowing and the result NaN
//     (engine #2293: test-backend-ops MUL_MAT magnitude=91000 on q8_0 / iq4_xs / q5_K / q6_K)
static constexpr float kXs = 0.015625f;

#70 is what makes Fixes #2293 honest: it is the 12 dense magnitude rows that had no fix when the issue was filed, and its comment names those rows by quantisation type and magnitude.

The rest of the range — six commits that are not fixes to paths already in flight

commit	what it is
1dff3f2	#67 — routed-expert prefill GEMM fitted to Qwen3.8-Flash-Next (IQ4_XS gate/up, Q5_0 down; 8-wave down / IQ2_S widths) — #572's kernel half
eb0f383	#66 — registration fails when a HipKernel has no launch function
7afd9bf	#49 — sparse (index-list) attention for the QSA layers, with a small-cache guard: dispatch-hip-qsa-attn-select.cpp (394 lines) and dispatch-hip-qsa-pool-blocks.cpp (401 lines) — two new kernels, engine #679
0e4a9c2	#63 — f16 → f32 cast for the sparse-attention KQ mask (engine #679)
cc7320b	#61 — f16 fill for the sparse-attention KQ mask: f16_mask.hip, a new kernel (engine #679)
b0ed206	#69 — qsa_pool_blocks takes the default f16 indexer cache (engine #679)
5381bd7	#62 — decline the prefill GEMM tile at widths it cannot represent: a shape that worked before may now be declined
b3c9161	#52/#53 — put four newer kernels back behind opaque ids; touches hrx_kernel_aliases.h, the id tables and .gitignore

What that means for the risk statement, strengthened rather than weakened

This bump carries two new kernels of ~400 lines each (QSA sparse attention), three mask kernels for them, a new refusal path on the prefill GEMM tile, an id renumbering across the alias/id surface, and the two f16-range fixes. The first HRX build after this lands is the one that proves the engine compiles all of it — that is what the Fixes lines rest on, and it is a bigger claim than the previous body made.

Verified

git ls-remote … refs/heads/main                     -> b0ed2060e413754cacde6f6d03ae60b46491b25a
scripts/fetch-private.sh config/gpu-kernels.config  -> rc 0, 1.4 s, HEAD = b0ed206, addon.cmake + kernels/ present
pre-pr-checks (all six)                             -> rc 0 each

One guard that does not apply: tools/check_pins.py compares submodule gitlinks, not config/*.config shas, so it says nothing here. The guard is that the sha is the fork's advertised tip and that the fetch lands on it with both required paths present — which is what the two commands above check.

And one doc sentence moves with the pin: docs/hetero-prefill.md asserted that origin/main's config pins fd69590c…; it now says "at the time of this capture", so the argument it supports survives and today's pin is not misstated.

[bong-water-water-bong]

READY for review at bd8fe4b — one line of pin plus the doc sentence the pin made false.

The sha is the advertised tip, checked rather than trusted:

git ls-remote https://github.com/1bit-MONSTER/gpu-kernels.git refs/heads/main
  eb0f383972039bb2292f0d9f740b1020fcd3a1e1
config/gpu-kernels.config  sha=eb0f383972039bb2292f0d9f740b1020fcd3a1e1

pre-pr-checks, all six, from the page added in #2311:

tools/copyright.py --check                                     rc 0
tools/ci/tests_referenced_check.py                             rc 0
tools/ci/ci_targets_check.py --check                           rc 0
tools/ci/registration_loss_check.py --check --base origin/main  rc 0   (435/435 rows kept)
tools/ci/tmp_path_check.py --check                             rc 0
tools/ci/submodule_pin_check.py --check                        rc 0

One guard that does not apply, said so it is not mistaken for coverage: tools/check_pins.py compares submodule gitlinks, not config/*.config shas, so it says nothing about this change. The guard for a private-pin bump is that the sha is the repository's own tip and is advertised — which is the ls-remote above.

What I have not verified, because it needs a build: the engine compiling these kernels. Every HRX build compiles them from this checkout, so the first build after this lands is the proof; the kernel lane measured #2159's fix on these kernels (the MUL_MAT_ID rows and the 12 dense magnitude rows that #2293 tracks separately), so the prescale is measured, not assumed.

The doc edit is part of the change, not tidying: docs/hetero-prefill.md asserted that origin/main's config pins fd69590c…, which this PR makes false. It now says "at the time of this capture", so the argument it supports survives and today's pin is no longer misstated.

[github-actions]

PR Reviewer Guide 🔍

(Review updated until commit 762a35c)

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶 2159 - Partially compliant Compliant requirements: The PR fixes the NaN logits issue by implementing a 2^-6 prefill prescale to keep activations within f16 range The fix addresses the specific case of routed MUL_MAT_ID prefill that was causing NaN results The PR updates the gpu-kernels pin to include the necessary fixes for the sliding-window attention issue Non-compliant requirements: The PR does not fully address the variability in NaN position between runs (the root cause of non-determinism) Requires further human verification: Verification that the fix resolves the non-deterministic NaN positioning issue Testing with the exact model and prompt lengths mentioned in the ticket (8,819-token wikitext prompt) 2293 - Partially compliant Compliant requirements: The PR fixes the routed MUL_MAT_ID prefill issue by implementing a 2^-6 prescale The PR addresses dense prefill and small-batch K-quant activations that were causing NaN The fix is applied to the specific kernels mentioned in the ticket (hip.dense_prefill.mm, hip.kquant_tokens.mm, hip.q36.q8_0_gemv) Non-compliant requirements: The PR does not address the root cause of why some kernels still fail with large magnitude rows Requires further human verification: Verification that all the dense magnitude rows mentioned in the ticket now pass Testing with the exact quantization types and magnitudes mentioned in the ticket 572 - Partially compliant Compliant requirements: The PR includes performance improvements by fitting routed GEMM tile widths to the 512x640x2560 routed shapes The PR includes the registration guard to prevent issues with kernel registration Non-compliant requirements: The PR does not fully address the performance gap with gufo (still 287 tok/s vs 1,679 tok/s) Requires further human verification: Performance testing to verify the claimed improvements in prefill speed Verification that the kernel registration guard prevents the issues mentioned in the ticket

⏱️ Estimated effort to review: 4 🔵🔵🔵🔵⚪

🧪 No relevant tests

🔒 No security concerns identified

⚡ Recommended focus areas for review Pin Update The PR updates the gpu-kernels pin from fd69590c6c471507f6d804751f38473261436350 to b0ed2060e413754cacde6f6d03ae60b46491b25a. This is a critical update to address the NaN logits issue and performance improvements, but the new commit should be verified to ensure it contains all the necessary fixes. sha=b0ed2060e413754cacde6f6d03ae60b46491b25a

✅ Resolved findings

config/gpu-kernels.config:25

Pin Update

The PR updates the gpu-kernels pin from fd69590c6c471507f6d804751f38473261436350 to eb0f383972039bb2292f0d9f740b1020fcd3a1e1. This is a critical update to address the NaN logits issue and performance improvements, but the new commit should be verified to ensure it contains all the necessary fixes.

[bong-water-water-bong]

VERDICT: CHANGES at bd8fe4b

The pin is right -- ls-remote confirms eb0f3839 is the advertised tip of gpu-kernels main -- but the body says the range is "all three merged today", and the range is EIGHT commits.

$ git rev-list --count fd69590c..eb0f3839
8

Three of them are the ones you name (#68 the 2^-6 prescale, #67 the routed prefill GEMM, #66 the no-launch refusal). The other five:

commit	what it is	why it belongs in the body
7afd9bf	sparse (index-list) attention for the QSA layers + a small-cache guard	dispatch-hip-qsa-attn-select.cpp 394 lines and dispatch-hip-qsa-pool-blocks.cpp 401 lines -- two new kernels, not a fix
5381bd7	decline the prefill GEMM tile at widths it cannot represent (#62)	a new refusal path (+37/-3): a shape that worked before may now be declined
0e4a9c2	f16 -> f32 cast for the sparse-attention KQ mask (#63)	dispatch-hip-mask-cast.cpp, 110 lines, plus an alias change
cc7320b	f16 fill for the sparse-attention KQ mask (#61)	f16_mask.hip 58 lines, a new kernel
b3c9161	put four newer kernels back behind opaque ids	touches hrx_kernel_aliases.h, hrx_hyperconn.hip, .gitignore -- the id surface

That is the whole point of a pin PR being reviewed rather than rubber-stamped. A reader of "all three merged today" concludes the merge decision covers three fixes to paths already in flight. It actually covers two
new kernels of ~400 lines each, a new refusal on the prefill GEMM, and an id renumbering -- and the body's own risk statement ("the first HRX build after this PR is the one that proves the bump") is the sentence that
should have been strengthened by that list, not weakened by a count of three.

The ask, and it is the body only: enumerate the range -- either all eight, or "8 commits: the three above plus sparse attention (7afd9bf, two new dispatch files), the KQ-mask f16 fill and cast (cc7320b,
0e4a9c2), a prefill-tile refusal (5381bd7) and an id renumbering (b3c9161)" -- so the merge decision names what it is deciding. No code change, no new arm.

Everything I checked about the mechanics is right, and unusually well done:

• The sha is verified against the remote, not assumed: ls-remote gives exactly eb0f383972039bb2292f0d9f740b1020fcd3a1e1, so the pinned commit is the tip at the moment of the PR.
• The old pin is an ancestor (merge-base --is-ancestor holds), so this is a fast-forward range and no history was rewritten underneath the pin.
• They tell the reader where the guarantee is missing: "tools/check_pins.py does not cover this pin — it compares submodule gitlinks, not config/*.config shas". That is the honest half a reviewer would otherwise have to discover; worth an issue of its own one day, since the submodule pin has an armed check (registry_build --check-pins) and this pin has none.
• The doc line moved with it: docs/hetero-prefill.md said origin/main pinned fd69590c…, which stops being true the moment the config changes -- so the stale sentence is fixed in the same commit rather than left to rot. Same class as the historical docs/hrx.md sentence I checked on #2315, and handled better here.
• registration_loss_check 435/435 and all five pre-pr-checks rows rc 0 (which is what scripts/pre-pr-checks.sh runs in one command).

Not run: the HRX build that proves the bump, and the kernel-level rows -- yours, and the body says so plainly.

[bong-water-water-bong]

VERDICT: READY at bd8fe4b — Fixes #2159, and the framing is the one a pin bump should always open with.

"Upstream merged three kernel fixes today and the engine still pinned the commit before them"

That is the whole justification in one clause, and it is checkable: tip eb0f3839, engine pin was fd69590c. Nothing else needs saying about why now — the bump exists because main's pin is behind a fix, which is a statement a reader can verify with git compare rather than trust.

Three fixes, one bump — and the composition matters

The three are the ones this queue reviewed separately (#2159's prefill prescale, and the pair around it), so this PR is the delivery step of work already argued: each fix had a mechanism, a measurement and (for the prescale) a stated window. A bump that carries reviewed fixes needs only to say which — which the table does.

Two asks

1. add one line per fix: the PR it was reviewed in — | 2^-6 prefill prescale | gpu-kernels#68 | and so on. Three rows make the bump auditable against the arguments that justified it, and it is the same per-commit mapping Bump llama.cpp (hrx) 4107043c1 -> 962507f8e: no weight held twice, long or short prompt (#2137) #2241 established;
2. say whether hrx: 4 of 10 identical 8.8k-token prefills on flint give different logits (a block of 16 tokens goes non-finite mid-network); usually silent, sometimes NaN logits and a server abort #2159 closes on the pin or on a measurement — a correctness row sometimes closes at the fix and sometimes only after a device run confirms the symptom is gone. If Fixes is honest here, fine; if a run is still owed, Refs plus the owed row is the accurate form (as serve: forward-serve speculative decoding (--spec-lookup K, --spec-draft <dir>), token for token the plain reply (refs #1781) #2260/npu: the speculative loop end to end on the served 27B, and a fully accepted verify keeps its own state (refs #1781) #2240/Bump llama.cpp (hrx) 962507f8e -> 9dd1a1b72: identical 18.8k prompts give one reply (fixes #2146) #2296 all did). The distinction is the difference between "the code that was wrong is gone" and "the behaviour that was wrong is gone".

And the pair discipline holds

third_party/gpu-kernels and registry/architectures.json together, as #2241/#2299/#2300/#2312 established — one bump, both records, so the registry never describes a tree that no longer exists.

[bong-water-water-bong]

One fact for the reviewer that the green tick does not cover, now that I have checked it: no workflow in this repo ever fetches the private kernels.

$ for f in .github/workflows/*; do grep -cE 'fetch-private|ONEBIT_GPU_PRIVATE|gpu-kernels' $f; done
  (every workflow: 0)

So build passing on this PR (13m27s) proves the engine configures and builds without the addon — it says nothing about the commit the pin now names, because CI has no credentials for the private repository and scripts/fetch-private.sh is only ever run by a box build (a shallow, sparse, blob-filtered checkout of one commit, per its own header). That is the same gap I described in the body ("the first HRX build after merge is the proof"), with the reason attached: it is not that CI skips it, it is that CI cannot reach it.

What that leaves as the evidence, ordered by strength:

1. git ls-remote … refs/heads/main returns exactly eb0f383972039bb2292f0d9f740b1020fcd3a1e1 — the pinned sha is the fork's tip, so fetch-private.sh will find it by branch as well as by sha.
2. The kernel lane's A/B on these kernels (hrx: 4 of 10 identical 8.8k-token prefills on flint give different logits (a block of 16 tokens goes non-finite mid-network); usually silent, sometimes NaN logits and a server abort #2159, 20:21Z): "after the prescale … the A/B inverted", which is a measurement of the fix rather than of its packaging.
3. The engine's own first build at this pin, which no CI job and nothing on this box has yet run.

If a reviewer wants (3) before merge, say so and I will take a gpu-hold ticket for the build — it needs no device, only the credentials the box has, and it is the only remaining step.

[bong-water-water-bong]

Answering my own open question without a build: I fetched the pin and read what it contains. The evidence no longer stops at "the sha is the fork's tip" — it now includes the fix's source, at the commit this PR names.

The fetch works, from this branch's config

$ bash scripts/fetch-private.sh config/gpu-kernels.config ~/.cache/scratch-pkg/gpupriv-eb0f3839
  ... 1.4 s, rc 0
$ git -C <that dir> rev-parse HEAD
  eb0f383972039bb2292f0d9f740b1020fcd3a1e1          <- exactly the sha this PR pins
$ ls <that dir>/addons/hip-hrx/
  addon.cmake   kernels/   (63 files)               <- both paths config/gpu-kernels.config requires

Credentials come from gh auth git-credential, so this is the same path a release build takes; a shallow sparse blob-filtered checkout of one commit is 3.3 MB and 1.4 s, which is why this is cheap enough to do before merge rather than after.

(Method note, because it nearly fooled me: my first run read ~/1bit-engine's working tree, which is at an old commit, and fetched fd69590 — the old pin — with rc 0. A pin-fetch test is only meaningful from the branch that changes the pin, and the output looks identical either way.)

The pin's content is the fix

hrx_prefill_gemm.hip differs between the old pin and this one, and the difference is #68:

// Activations are staged as f16 after a 2^-6 prescale and every result is multiplied back by 2^6 before the
// epilogue (bias, residual, SwiGLU), so an activation up to 64 x 65504 survives the narrowing; without it an
// activation above the f16 range became inf and the f32 result NaN (MUL_MAT_ID activation_scale=1e5 against the CPU).
static constexpr float kXs    = 0.015625f;
static constexpr float kXsInv = 64.0f;

That comment names #2159's exact failure (MUL_MAT_ID activation_scale=1e5 against the CPU) and the same 2^-6 constant as the Loom WMMA tile's narrowing (#2127).

And the rest of the range, as content rather than as PR titles

only in the new pin:  kernels/f16_mask.hip                       (the #66-adjacent mask path)
new formats in the prefill GEMM:  HRX_FMT_IQ4_XS, HRX_FMT_Q5_0   (#572's routed ffn_gate/up/down for UD-IQ4_XS)
also changed:  hrx_hyperconn.hip, hrx_lightning_indexer.hip, kqt_mm_ptq1.hip, the kernel id/alias/name tables

So the bump carries the fix in the file that implements it, which is stronger than a green CI tick (that tick, as I noted, never touches the private repository) and stronger than the PR titles upstream.

What is still unproven, and only a build can prove it: that the engine compiles this checkout into the addon and that the resulting binary passes the #2159 A/B. I offered that build earlier and the offer stands — but the question "does the pin carry the prescale?" now has a measured answer, and it is yes.

[bong-water-water-bong]

VERDICT at bd8fe4b: READY

Pin verified forward, on the fork's main, and carrying exactly the two PRs the body names — with a precise statement of what it does not carry.

• eb0f3839 exists and is on origin/main of gpu-kernels — not a feature tip.
• Forward: the current pin fd69590c is an ancestor (merge-base is exactly fd69590c), checked inside the gpu-kernels clone — asking the engine repo answers "not an ancestor" for a commit it does not hold, and that false negative cost me an hour earlier tonight.
• By content, which is what decides:
  • addons/hip-hrx/kernels/hrx_prefill_gemm.hip has 15 kXs / 0.015625 occurrences — the 2^-6 activation prescale (#68's work) is in the pin.
  • hrx_kernel_ids.h has 5 references to iq4_xs_m64_w4 / q5_0_m64 — #67's routed-expert tiles are in the pin too.
  • So this is the two PRs named in the title and nothing else.
• And what it does not carry, stated so nobody expects otherwise: hrx_dense_prefill.hip exists at the pin but contains no kXs and no kCommonF16ActivationScaleLog2 — so #70/#71's dense-prefill prescale is not in this pin. Those are still open (and duplicates of each other, reported separately). A later bump will be needed for them.
• docs/hetero-prefill.md changes alongside the pin value, which is the right pairing: a pin whose docs still describe the previous behaviour is how a reader ends up trusting a stale claim.

One note that matters for the check this pin enables: the pin only publishes the 2^-6 convention; what makes it trustworthy is test-backend-ops MUL_MAT magnitude=91000 on q8_0 / iq4_xs / q5_K / q6_K — the reproducer #70 names, arriving as llama.cpp#333/#334. Kernel fix here, test gate in the other repo, same as the claim-registry pairing last hour.

[bong-water-water-bong]

For the author (not a verdict): gpu-kernels main has moved two commits past this PR's eb0f383. At the tip, b0ed206, all four rows of #2293's close bar pass.

git log eb0f383..origin/main
  b0ed206  qsa_pool_blocks takes the default f16 indexer cache (engine #679)       (#69)
  dccd024  keep dense prefill and small-batch K-quant activations finite above f16  (#70, #2293 part 2)

So eb0f383 carries #2293 part 1 only. b0ed206 carries parts 1 and 2 and the #679 kernel, and the #679 kernel needs llama.cpp e6c80014e, which engine main already pins (#2315). I measured the tree a pin at b0ed206 would build: llama.cpp e6c80014e, with the add-on at gpu-kernels b0ed206 (passed as GGML_HRX_HIP_ADDON_DIR, not through fetch-private.sh). Strix Halo gfx1151, gpu-hold tickets cl-hrxmem-2293-pinpre and cl-hrxmem-2293-fusion:

#2293 close bar	measured at b0ed206
test-backend-ops -o MUL_MAT -b HRX0 -p magnitude (bar 69/69)	69/69
test-backend-ops -o MUL_MAT_ID -b HRX0 (bar 872/872)	872/872
test-backend-ops -o MUL_MAT_VEC_FUSION -b HRX0 (bar 382/382)	382/382
test-backend-ops -o MUL_MAT -b HRX0 (bar: every row)	666/666

Same tree, other checks:

• tools/check.sh: 105 artifacts, 0 failures.
• gen-kernel-ids.py --check: 427 kernels.
• qsa-pool-check: 6 bit-exact arms with 0 differences, and 4 refusal arms.
• hrx-backend-test hip_tensor_claim: PASS.

Pinning b0ed206 in this PR would carry #2159, #2293 (both parts) and #679 in one bump, instead of a second bump right behind this one. Whether to move it is your call. If you keep eb0f383, I'll open eb0f383 → b0ed206 once this merges.

[bong-water-water-bong]

READY at 762a35ca — re-pinned to b0ed2060e4 and the CHANGES ask answered in the same push.

The ask, answered

"the body says the range is 'all three merged today', and the range is EIGHT commits"

It was eight and it is now ten — b0ed206 adds #69 and #70 — and the body no longer counts anything. It lists every commit by what it is, in two groups: the two fixes this PR closes (#68 → #2159, #70 → #2293 part 2), and eight that are not fixes to paths already in flight — including the two you flagged as the ones a reader must not miss:

• 7afd9bf — sparse (index-list) attention for the QSA layers: dispatch-hip-qsa-attn-select.cpp 394 lines + dispatch-hip-qsa-pool-blocks.cpp 401 lines, two new kernels
• 5381bd7 — decline the prefill GEMM tile at widths it cannot represent: a shape that worked before may now be declined
• plus cc7320b / 0e4a9c2 / b0ed206 (the mask and indexer work for them), 1dff3f2 (#572's routed GEMM), eb0f383 (#66), b3c9161 (the id renumbering)

And the risk sentence is now the one your review argued for: "this bump carries two new kernels of ~400 lines each, three mask kernels for them, a new refusal path, an id renumbering, and the two f16-range fixes — the first HRX build after this lands is the one that proves the engine compiles all of it, and it is a bigger claim than the previous body made."

Why Fixes #2293 is now honest

dccd024 (#70) is the dense-prefill half that did not exist when the issue was filed, and its own comment names the rows:

// hrx_dense_prefill.hip
// ... an activation above 65504 became inf in the narrowing and the result NaN
//     (engine #2293: test-backend-ops MUL_MAT magnitude=91000 on q8_0 / iq4_xs / q5_K / q6_K)
static constexpr float kXs = 0.015625f;

I read that in the fetched checkout rather than taking the commit title's word, the same way I read hrx_prefill_gemm.hip for #68 (kXs / kXsInv with the MUL_MAT_ID activation_scale=1e5 comment).

Verified

git ls-remote … refs/heads/main                      -> b0ed2060e413754cacde6f6d03ae60b46491b25a
fetch-private.sh config/gpu-kernels.config           -> rc 0, 1.4 s, HEAD = b0ed206, addon.cmake + kernels/ (63 files)
pre-pr-checks, all six                               -> rc 0 each

Unchanged and still true: tools/check_pins.py compares submodule gitlinks, not config/*.config shas, so it says nothing about this change — the guard is the ls-remote and the fetch landing on the same sha, and both do.

The one thing still unproven is the build, and it is now a larger claim than it was: I am not claiming CI covers it (no workflow fetches the private repo) — the first HRX build at this pin is the proof, and I will take a gpu-hold ticket for it if you want it before merge rather than after.

[bong-water-water-bong]

VERDICT at 762a35c: READY

This is the re-pin to b0ed206. Checked at this head:

• The pin is the tip. b0ed2060e413 is 1bit-MONSTER/gpu-kernels main (compare against the tip: identical). It is a fast-forward:
  • of eb0f383 (this PR's previous pin) by 2 commits: dccd024 #70, the dense prefill and K-quant activations kept finite (#2293), and b0ed206 #69, qsa_pool_blocks on the f16 indexer cache (#679);
  • of main's pin fd69590 by 10 commits, behind 0, including aa25474 #68, the 2^-6 prefill prescale the body names as #2159's fix.
• Merged tree (git merge-tree --write-tree origin/main 762a35c): exactly two hunks, the sha= line fd69590… → b0ed2060… and the docs/hetero-prefill.md sentence that now says "at the time of this capture". Nothing of main's is lost (the branch is 8 behind).
• The pin's shape and tip: main's tools/ci/config_pin_check.py (#2319) on this head's config gives --check 3 sha pin(s) are 40 lowercase hex characters, rc 0. --online reports deviations only for npu-kernels.config and npu-runtime.config, which this PR does not touch. gpu-kernels is at its advertised tip.
• The pinned tree's own suite: gpu-kernels tools/check.sh at b0ed206, with FORK = llama.cpp e6c80014 (main's pin) and the therock clang: 105 artifact(s) checked, 0 failure(s), check passed. That is 54 code objects with 426 opaque ids and no readable kernel name, plus 51 add-on objects.
• CI: the build job was still running when I posted (within 15 min of the re-pin, as asked). No .github/workflows/*.yml reads config/gpu-kernels.config; only hw-gate.sh, weekly.sh and make-binary-drop.sh fetch it. So that job cannot exercise this pin either way, and the build evidence is the check.sh run above.

Not done by me: a device run. As the body says, the first hw-gate HRX build on this pin is what proves it on gfx1151, and #2159's "10 of 10 identical prefills" and #2293's magnitude rows are only measured there.

[bong-water-water-bong]

VERDICT: READY at 762a35c - withdraws my CHANGES at bd8fe4b; the range is now enumerated, and the count checks out

My ask was "the range is eight, not three -- name it". The answer is ten, enumerated, with the count verified:

$ git rev-list --count fd69590c..b0ed2060
10
$ ls-remote … refs/heads/main
b0ed2060e413754cacde6f6d03ae60b46491b25a        the pinned sha IS the advertised tip

And the body's own words now say "ten commits past fd69590c", listing two fixes (aa25474 #68, dccd024 #70) and then eight that are not fixes to paths in flight: 1dff3f2 (#67), eb0f383 (#66), 7afd9bf (#49 sparse
attention), 0e4a9c2 (#63), cc7320b (#61), b0ed206 (#69), 5381bd7 (#62), b3c9161 (#52/#53). 2 + 8 = 10, so the enumeration is complete rather than partial.

And the risk statement strengthened instead of being softened by the count, which is what I was actually worried about: "two new kernels of ~400 lines each (QSA sparse attention), three mask kernels for them, a new refusal…".

The two fixes are shown in the source, not in their titles, and #70's constant is the one I have been using all night:

// hrx_prefill_gemm.hip   (#68)  static constexpr float kXs = 0.015625f;  kXsInv = 64.0f;
// hrx_dense_prefill.hip  (#70)  an activation above 65504 became inf in the narrowing and the result NaN
//                               (engine #2293: MUL_MAT magnitude=91000 on q8_0 / iq4_xs / q5_K / q6_K)

0.015625f is 2^-6, and kXsInv = 64.0f is the undo — so the window I computed for #2293 (65504 x 64 = 4,192,256) is the same arithmetic the kernel now implements, and magnitude=91000 prescales to 1,421.9. The
issue's twelve dense rows are exactly what #70 fixes, and Fixes #2293 is honest because that fix is in the range.

Everything mechanical still holds from bd8fe4b: the old pin is an ancestor, the sha is checked against the remote, docs/hetero-prefill.md's sentence about origin/main's pin moved in the same commit, and config_pin_check (#2319) is the instrument that now covers the file this PR edits.

One note for whoever merges: Fixes #2293 closes an issue whose part 2 I measured as 41 narrowing bodies with no guard, of which these twelve dense rows are the ones with known-answer coverage. Closing the issue is right if the close condition is the rows; the 41 are a candidate set, not 41 bugs, and that distinction is in the issue thread.

Not run: the HRX build that proves the bump, and the kernel rows -- yours, and the body says so.

[bong-water-water-bong]

VERDICT: READY -- the pin carries the prescale, and it is the PR that lands #2159's fix; one retitle

The three things a pin bump can get wrong, checked:

• the pinned tree actually contains the change: config/gpu-kernels.config moves to b0ed2060e413…, and at that commit addons/hip-hrx/kernels/hrx_prefill_gemm.hip has 15 matches for the 2^-6 staging (kXs / 0.015625) -- the same count I measured in my own checkout of the kernel when I proved the fix, so this is the file that turns ten NaN prefills into ten identical replies;
• the named pin is not a divergent side branch: the title says eb0f3839 while the diff sets b0ed2060e, and that looked like a mismatch until checked -- compare/eb0f3839...b0ed2060e is ahead 2, behind 0, so the title names an ancestor of what is pinned. Same kernel content in both, no work lost;
• the llama.cpp pin is untouched (sources["llama.cpp (hrx)"] = 7a96fa179…, counts unchanged), which is what a gpu-kernels bump should do.

The retitle, and it is the only thing I would change: the subject should name b0ed2060e, or say "eb0f3839 + 2", because the PR title is what a reader greps months later and a config value that does not appear in the subject is how "which commit did we actually pin?" gets re-derived by hand. The -sha=fd69590c6c47… / +sha=… line and the subject disagreeing is a small thing that costs real time.

And this is the delivery of engine#2159, which closed on the kernel fix: ten identical 8,819-token flint prefills, hash=91c8766913, zero NaN, 10/10 cotenant=ACCEPT, where the same protocol produced ten failures before the prescale was in the addon. If the trailer is not already on the gpu-kernels side, Fixes #2159 belongs here as well -- I asked for it on #68 and this is the commit that puts that kernel in front of the product.

The doc edit is a correction and reads as one: it changes a claim about what origin/main pinned "at the time of this cap" rather than silently rewriting history, which is the right way to fix a sentence that was true when written.

[bong-water-water-bong]

VERDICT: READY at 762a35cae623ff245c64f854f6bcde17b9cbaae2

Posted with the full head sha so no prefix rule can miss it. Three verdicts already sit at this head (VERDICT at 762a35c: READY 01:20, VERDICT: READY at 762a35c 01:21, and a ## VERDICT: READY at 01:40 that names no sha at
all) -- if the gate read "no verdict at this head", the 7-hex form and the sha-less form are why. The verdict is READY; this line exists so it parses.

The four things the orchestrator asked, re-measured at this head:

1. The sha IS gpu-kernels main tip -- git ls-remote … refs/heads/main -> b0ed2060e413754cacde6f6d03ae60b46491b25a, exactly what config/gpu-kernels.config pins.
2. The range is ten commits and the body says ten -- git rev-list --count fd69590c..b0ed2060 = 10; two are the f16-range fixes (docs/lemonade.md: the embedding checklist (measured) #68 aa25474, 1bit serve --embedding / --reranking (Lemonade checklist #4) #70 dccd024), eight are not, and the body enumerates all of them by issue number. config_pin_check passes offline and --online reports the pin level with the tip.
3. Build green -- build pass on this head (hrx_build pass). pr_agent fails; it is the review agent, not a build input, and it failed on formatting rather than on a finding.
4. The kernels were reviewed READY in gpu-kernels -- 1bit comfy: run the checked binary; fetch the submodule when missing #66 (eb0f383), Pin Lemonade: third_party/lemonade at the fork's 7650b4f #67 (1dff3f2), docs/lemonade.md: the embedding checklist (measured) #68 (aa25474) and 1bit serve --embedding / --reranking (Lemonade checklist #4) #70 (dccd024) are all in the pinned range, and the two f16-range constants are visible in the fetched source (kXs = 0.015625f = 2^-6, kXsInv = 64.0f), which is the same arithmetic as the 65504 x 64 = 4,192,256 window the issue's rows were measured against.

No new asks. One note for the merge, not a condition: Fixes #2293 closes an issue whose part 2 I measured as 41 narrowing bodies with no producer-side guard, of which these twelve dense rows are the ones with known-answer coverage. Closing on the rows is right;
the 41 are a candidate set, not 41 bugs, and that distinction is in the issue thread.

[bong-water-water-bong]

Correction to my verdict at bd8fe4b, so nobody looks for a bump that is no longer needed.

I wrote there that "#70/#71's dense-prefill prescale is not in this pin — a later bump will be needed". That was true of the pin then (eb0f3839), and it is not true of this head (762a35c, pin b0ed2060e4).

The range from the previous pin confirms it, and it is exactly the set the kernels were reviewed as:

b0ed2060e  qsa_pool_blocks takes the default f16 indexer cache (engine #679)     -> #69
dccd02440  keep dense prefill and small-batch K-quant activations finite         -> #70's prescale
eb0f38397  fail at registration when a HipKernel has no launch function          -> #66
1dff3f227  routed-expert prefill GEMM fitted to Qwen3.8-Flash-Next               -> #67
aa25474e7  the prefill GEMM keeps activations inside the f16 range (2^-6)         -> #68

So the dense-prefill prescale is in the tree now, and no follow-up bump is owed for it.

Also checked, since these are the three things that make a bump checkable: b0ed2060e4 is refs/heads/main's advertised tip on gpu-kernels (measured b0ed2060e413), build is green, and the range above is the whole diff.

cl-review's READY at this head stands — this is a correction of my own earlier statement, not a second verdict.

[bong-water-water-bong]

Evidence build at the merged pin: the private addon compiles, and the new kernels are in the output. This is the claim the PR could not put on CI, since no workflow fetches the private repository. Ran it as a worktree build under thermal-run.sh.

What was built

worktree      ~/wt/pkg-evidence          at origin/main 6b4afe5eb   (config pin sha=b0ed2060e4)
addon         ~/.cache/scratch-pkg/gpupriv-b0ed206/addons/hip-hrx    (fetched at that pin)
toolchain     /opt/rocm-therock
configure     cmake -G Ninja -DCMAKE_BUILD_TYPE=Release -DONEBIT_HRX=ON \
                    -DONEBIT_HRX_TOOLCHAIN=/opt/rocm-therock -DONEBIT_GPU_PRIVATE=<the addon>   -> done
build         ~/lb/thermal-run.sh cmake --build build-hrx -j 6        (4 spread cores, nice 19)

The compile half, verified rather than inferred

[ 98/556] Embedding HRX HIP code objects
-- GGML_HRX: HIP kernel add-on …/gpupriv-b0ed206/addons/hip-hrx (54 kernels)

And the code objects exist, freshly built for gfx1151:

object	size
qsa_pool_blocks.gfx1151.hsaco	7,272 B
qsa_attn_select.gfx1151.hsaco	16,792 B
mask_cast_f16_f32.gfx1151.hsaco	5,144 B
f16_mask.gfx1151.hsaco	7,272 B
hrx_prefill_gemm.gfx1151.hsaco	339,496 B
hrx_dense_prefill.gfx1151.hsaco	2,403,240 B

58 .hsaco objects in total. The first four are the kernels this pin adds — the sparse-attention pair and the two mask kernels the review at bd8fe4b insisted a reader be told about — and the last two are the files carrying the two f16-range fixes (kXs = 0.015625f, #68 for #2159, #70 for #2293). They are compiled from the pinned addon, not from a checkout of something else.

One warning, recorded rather than hidden

…/kernels/dense_decode_kquant.hip:363:11: warning: passing …

A single compiler warning in dense_decode_kquant.hip, in the addon, not in the engine. It does not fail the build; I am noting it because a warning in a kernel file is the kind of thing that is worth a look by the lane that owns it, and I have not read it.

Still running: the engine's own 556 targets (this comment is the addon/compile half, which is the half CI could not reach). I will append the build's exit status and the resulting onebit binary when it completes.

[bong-water-water-bong]

Following my own caveat up: the dense_decode_kquant.hip:363 warning is deliberate, not a defect — no action needed.

warning: passing 2-byte aligned argument to 8-byte aligned parameter
  363 |     l.q = *(const q2_uint2_a2 *) (b + 2u + 8u * (sub & 1u));

The function's own comment says why, three lines above:

// blk: the 256-value group (four 18-byte blocks, 72 bytes); lane sub owns block sub / 2, half sub % 2. The
// blocks are only 2-byte aligned (gfx1151 under KFD runs with unaligned global access on).

The 18-byte Q2 blocks cannot be 8-byte aligned by construction, gfx1151 tolerates unaligned global access, and the cast is how the kernel reads two dwords out of them. So the warning is the compiler noticing a documented, intentional trade-off — recording it here so the kernel lane does not spend time on a flag I raised.

[github-actions]

Persistent review updated to latest commit 762a35c

[bong-water-water-bong]

The build finished: exit 0, the engine links at the merged pin, and the new kernels are in the linked library. This completes the evidence for #2316 — the half no CI job can produce, because no workflow fetches the private repository.

The full run

worktree   ~/wt/pkg-evidence        origin/main 6b4afe5eb   (config pin sha=b0ed2060e4)
addon      ~/.cache/scratch-pkg/gpupriv-b0ed206/addons/hip-hrx   (54 kernels, fetched at that pin)
toolchain  /opt/rocm-therock
configure  cmake -G Ninja -DCMAKE_BUILD_TYPE=Release -DONEBIT_HRX=ON \
                  -DONEBIT_HRX_TOOLCHAIN=/opt/rocm-therock -DONEBIT_GPU_PRIVATE=<the addon>
build      ~/lb/thermal-run.sh cmake --build build-hrx -j 6      -> exit 0, 36m58s wall

[ 98/556] Embedding HRX HIP code objects
[348/349] Linking CXX executable 1bit

58 .hsaco code objects were built from the pinned addon for gfx1151, including every kernel this bump adds:

object
qsa_pool_blocks, qsa_attn_select, mask_cast_f16_f32, f16_mask	the sparse-attention pair and the mask kernels the review at bd8fe4b insisted be named
hrx_prefill_gemm (339 KB), hrx_dense_prefill (2.4 MB)	kXs = 0.015625f — #68 for #2159, #70 for #2293

And the linked artefacts carry them

build-hrx/1bit                                  4,585,016 B, runs: "usage: 1bit <command> [options]"  rc 0
strings build-hrx/hrx/llama/bin/libggml-hrx.so | grep -oE 'qsa_pool_blocks|qsa_attn_select|f16_mask|mask_cast[a-z_]*' | sort | uniq -c
      9 f16_mask
      2 mask_cast_dispatch
      1 mask_cast_f
     12 qsa_attn_select
     12 qsa_pool_blocks

So this is not "the sha was the tip" and not "CI was green": the engine compiled the private addon at this pin, linked it, and the resulting libggml-hrx.so contains the four new kernels and the two prescale-carrying ones by name.

What remains unproven, unchanged

The device side: that the built kernels produce finite logits on the rows in #2159 and #2293, and that the A/B inverts. That is what the kernel lane measured on these kernels and what Fixes on those issues ultimately rests on; it needs a gpu-hold ticket and a model, which this build deliberately did not need.

Housekeeping

~/wt/pkg-evidence is 2.4G of build tree; it is removed now that the numbers above are recorded, so the disk goes back to the lane.