Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 26 additions & 0 deletions docs/hrx.md
Original file line number Diff line number Diff line change
Expand Up @@ -304,6 +304,32 @@ Still open under #1962: two prefill kernels that re-quantise the activations int
(Q5_K and Q6_K weights at 512 tokens) keep a per-block term in half precision and still return NaN
above 65504. And the limit is moved, not removed: an activation above 4.19e6 would overflow again.

### Fixed: Qwen3-VL on `HRX0`: it decodes again, and its rotation runs on the device ([engine#1467](https://github.com/1bit-MONSTER/engine/issues/1467))

Two faults, both in the text model's graph rather than in the projector:

- **A refused view (llama.cpp #283).** A deepstack model adds a slice of the wide input embedding to each of
its first layers, taken as a view of a view. With the first view's node in another split, the HRX graph
never holds the wide tensor, the slice could not alias its input, and the graph was refused at
`unsupported HRX node 20: VIEW`. So Qwen3-VL-2B could not decode even text, and every image request failed
with it. Such a view is external to the graph and a layout op writes nothing, so the scheduler now skips it.
- **The rotation on the CPU (llama.cpp #286).** The model's rotation is the interleaved multi-section one
with sections that do not collapse to the plain rotation, so 56 `ROPE` nodes per text graph ran on the CPU
and cut the graph into 114 splits. The two generic rotation kernels gained a mode that reads the position
section each pair selects; the text graph is now 2 or 3 splits with `PAD` (and `GET_ROWS` on token
batches) left on the CPU.

Measured at pin `50d28600a` on gfx1151 (2026-11-10, power mode `performance`, Qwen3VL-2B-Instruct Q4_K_M with
`mmproj-Qwen3VL-2B-Instruct-F16`): `test-backend-ops -o ROPE -b HRX0` 166/166; 12 greedy text tokens
byte-identical to the CPU backend; `llama-bench -p 512 -n 64 -r 3` at 6,481 and 6,532 tok/s prefill, 133.7
and 135.5 tok/s decode (2,331 and 57.8 before the rotation moved). `llama-server` with the projector on
`HRX0`: the first request for the 448x448 test image takes 2.99 s (`prompt_ms` 2,782 for 227 prompt tokens)
against the 16.97 s this issue was filed on, and answers "A red rectangle and a green circle."

The projector's own graph still runs 103 of 626 nodes on the CPU for an image of that size (its 2-D rotation,
the attention score matmul and softmax, `IM2COL`, `UPSCALE`); with it offloaded, encoding the image is no
faster than with `--no-mmproj-offload` (0.5 to 1.1 s against 0.4 to 0.6 s).

### MoE decode and the kernel cache (llama.cpp #32, #33)

- **MoE decode.** With one to four tokens, a `MUL_MAT_ID` (a MoE expert matmul) runs as a GEMV over
Expand Down
2 changes: 1 addition & 1 deletion registry/architectures.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"about": "HF architecture -> GGUF architecture and the backends whose code accepts it. Generated by tools/registry_build.py from the pinned sources; do not edit.",
"sources": {
"llama.cpp (hrx)": "fa533bd21053eeb7eab5354aef749c44cde86987"
"llama.cpp (hrx)": "50d28600a5f7a38df390709b94c463df02785a54"
},
"counts": {
"hrx": 298,
Expand Down
Loading