Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 32 additions & 0 deletions docs/hrx.md
Original file line number Diff line number Diff line change
Expand Up @@ -389,6 +389,38 @@ layout (8.80 s and 8.85 s) and gives the same text. `test-hrx-fa-transposed-valu
backend (8 cases). Why 256-token launches are faster than one 512-token launch at long key counts is measured,
not explained.

### Fixed: a gated-delta-net layer behind a type cast failed the whole graph ([engine#1925](https://github.com/1bit-MONSTER/engine/issues/1925))

On Qwen3.8-Flash-Next, a long prompt failed at the first 512-token batch past 2,048 tokens of context when the
sparse attention path's mask chain ran on the device:

```
graph_compute: unsupported HRX node 160: L2_NORM output=f32[128,16,512,1]<-L2_NORM inputs=[f32[128,16,512,1]<-VIEW] consumers=[174:GATED_DELTA_NET]
```

The fused gated-delta-net dispatch did match that node. The fault was the order in which the graph was walked.
The scheduler walks a graph in dependency order and takes the matmuls first, which is what puts a layer's gate
and beta projections ahead of the `L2_NORM` its fused pattern starts from. A type cast is an in-place copy that
names its own output as an input. The walk counted that as a dependency, so the copy waited for itself, the walk
stopped there, and every node after it was taken in index order. In index order the `L2_NORM` comes before the
projections, and the pattern was refused because it would have read a projection that had not run. The sparse
path's mask chain contains such casts, so the failure needed both a long context and that chain on the device.

The walk no longer counts a node's own output as a dependency (llama.cpp #302). The fused matcher also keeps a
match when a node of its pattern that only renames storage was already passed.

Measured on gfx1151 (2026-11-11, Flash-Next UD-IQ3_XXS, `-c 19456 -np 1`, one 18,782-token request, a build of
the fork with only the two changed objects replaced): `unsupported HRX node` 1 before, 0 after; all 390 fused
gated-delta-net roots accepted; the whole prefill runs. `tests/test-hrx-gdn-passed-alias.cpp` plans the layer
behind a cast on the host and requires the same dispatches as without it. Qwen3.5-4B, which has the same layer
type and no sparse path, gives the same tokens on the old and the new library (128 of 128 in three alternated pairs), uses the same 33
dispatch registrations in the same order, and runs at the same rate (`llama-bench -p 512 -n 64 -r 5`, old, new,
old, new: decode 57.4, 58.3, 58.1, 58.2 tok/s; prefill 3,516, 3,042, 3,109, 3,213).

Not shown by this: that the answer at 18.8k is the same with the mask chain on the device as with it on the
CPU. Three of the four long requests in these runs, on the old library and the new, ended in all-NaN logits on
the first sampled token, which is [engine#1536](https://github.com/1bit-MONSTER/engine/issues/1536).

### MoE decode and the kernel cache (llama.cpp #32, #33)

- **MoE decode.** With one to four tokens, a `MUL_MAT_ID` (a MoE expert matmul) runs as a GEMV over
Expand Down
2 changes: 1 addition & 1 deletion registry/architectures.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"about": "HF architecture -> GGUF architecture and the backends whose code accepts it. Generated by tools/registry_build.py from the pinned sources; do not edit.",
"sources": {
"llama.cpp (hrx)": "7a1f2b81ce17aaa84a40b766d1b06e5fd4601196"
"llama.cpp (hrx)": "1703bfeaaaae16686e0ea5fd22b18b6fa2163fa1"
},
"counts": {
"hrx": 298,
Expand Down
Loading