From 19c2ea4f239fcaad002b2df569b4a91236ff9e03 Mon Sep 17 00:00:00 2001 From: bong-water-water-bong Date: Tue, 10 Nov 2026 14:10:59 -0400 Subject: [PATCH] bump llama.cpp to fa533bd21: a large expert activation no longer ends in all-NaN logits (fixes #1539) Carries fork #284 and #285 (the routed-expert and dense matmul kernels keep their activations inside the half-precision range) and #283. Adds the docs/hrx.md section for the fix; the first part of #1962. Co-Authored-By: Claude Fable 5.2 --- docs/hrx.md | 35 +++++++++++++++++++++++++++++++++++ registry/architectures.json | 2 +- third_party/llama.cpp | 2 +- 3 files changed, 37 insertions(+), 2 deletions(-) diff --git a/docs/hrx.md b/docs/hrx.md index f6c7c3313..f8ad5eea2 100644 --- a/docs/hrx.md +++ b/docs/hrx.md @@ -269,6 +269,41 @@ the fix, 0 of 15 after. `llama-bench -p 512 -n 128 -r 3`, two runs each: Sarashi at 26.49 and 25.92 tok/s, stories15M Q8_0 at 317.77 and 338.54 tok/s (the box was shared; the same binary gave 545.39 tok/s on a quieter run). +### Fixed: all-NaN logits after a large expert activation ([engine#1539](https://github.com/1bit-MONSTER/engine/issues/1539), [#1962](https://github.com/1bit-MONSTER/engine/issues/1962)) + +`flint-40b-a2.5b` on `HRX0` answered some requests and then failed a later one with +`HRX returned NaN logits: count=150144 of 150144`. The matmul kernels multiply in half precision, and a +half-precision value ends at 65504. An FFN down-projection reads the SwiGLU output, which nothing +normalises: flint's layer 46 reaches about 9.1e4 on an ordinary 22-token prompt. That value became +`inf`, the sum over the routed experts `inf - inf = NaN`, and the NaN was stored as those tokens' +keys and values. The request that overflowed still returned finite logits. The next, shorter request +on the same slot found the stale NaN row inside its masked attention window and returned NaN for the +whole vocabulary. So the failure depended on the prompt (which tokens drive an activation that large) +and on the order of requests, not on memory, load or timing, and the CPU backend, which multiplies in +single precision, was never affected. + +The routed-expert kernel (llama.cpp #284) and the dense kernel for single-precision activations +(llama.cpp #285) now scale the activations by 2^-6 before the half-precision stage and scale the +result back by 2^6. Both factors are powers of two, so values that fit before are rounded as before; +the usable range moves from [6.1e-5, 6.55e4] to [3.9e-3, 4.19e6]. + +Measured on gfx1151 (2026-11-10): + +- flint, the sequence that failed on 9 of 9 fresh loads (a 22-token request, then an 8-token one on + the same slot): 150144 of 150144 NaN before, 0 after; a read-back of all 3056 graph nodes finds no + non-finite value, and the first short request now returns the same logits as its repeat. +- `test-backend-ops -o MUL_MAT_ID`: the new case at flint's shape with activations of 9.1e4 returned + NaN against the CPU's 56214.23 before, and passes after; the whole op is 845 of 845. `-o MUL_MAT`: + the 47 new large-activation cases went from 23 to 47 passing; the whole op is 589 of 589. +- The range the shift gives up at the low end is not used by the models measured: the smallest + activation column maximum is 3.2e-2 on flint's expert down-projections (37278 columns) and 2.4e-2 on + the dense inputs of Qwen3-0.6B, Qwen3-4B, ZAYA1-8B and flint. Cases at those two values are in the + suite, so the shift has a test on each side. + +Still open under #1962: two prefill kernels that re-quantise the activations into 8-bit blocks +(Q5_K and Q6_K weights at 512 tokens) keep a per-block term in half precision and still return NaN +above 65504. And the limit is moved, not removed: an activation above 4.19e6 would overflow again. + ### MoE decode and the kernel cache (llama.cpp #32, #33) - **MoE decode.** With one to four tokens, a `MUL_MAT_ID` (a MoE expert matmul) runs as a GEMV over diff --git a/registry/architectures.json b/registry/architectures.json index b2fb3f4e9..705031c6d 100644 --- a/registry/architectures.json +++ b/registry/architectures.json @@ -1,7 +1,7 @@ { "about": "HF architecture -> GGUF architecture and the backends whose code accepts it. Generated by tools/registry_build.py from the pinned sources; do not edit.", "sources": { - "llama.cpp (hrx)": "a5dc597b81f10c1fd29356018e56da074380ce60" + "llama.cpp (hrx)": "fa533bd21053eeb7eab5354aef749c44cde86987" }, "counts": { "hrx": 298, diff --git a/third_party/llama.cpp b/third_party/llama.cpp index a5dc597b8..fa533bd21 160000 --- a/third_party/llama.cpp +++ b/third_party/llama.cpp @@ -1 +1 @@ -Subproject commit a5dc597b81f10c1fd29356018e56da074380ce60 +Subproject commit fa533bd21053eeb7eab5354aef749c44cde86987