From 79ed00ec685fdb96b145b8ccbfb26b09c999bdee Mon Sep 17 00:00:00 2001 From: bong-water-water-bong Date: Tue, 10 Nov 2026 15:46:32 -0400 Subject: [PATCH] bump llama.cpp to 50d28600a: Qwen3-VL's rotation runs on the device (fixes #1467) One fork commit past fa533bd21: 50d28600a (#286), the multi-section rotation in the two generic rotation kernels. With fa533bd21's #283 (already pinned) Qwen3-VL-2B decodes on HRX0 again and its text graph is 2-3 splits instead of 114. Measured at 50d28600a on gfx1151 (amdclang build, gpu-hold): test-backend-ops ROPE 166/166; text identical to the CPU for 12 greedy tokens; llama-bench pp512 6481 / 6532 tok/s, tg64 133.7 / 135.5 tok/s; llama-server, projector on HRX0, first image request 2.99 s (prompt_ms 2782) against the issue's 16.97 s. registry/architectures.json: the recorded source pin only. docs/hrx.md: the #1467 section. Co-Authored-By: Claude Fable 5.2 --- docs/hrx.md | 26 ++++++++++++++++++++++++++ registry/architectures.json | 2 +- third_party/llama.cpp | 2 +- 3 files changed, 28 insertions(+), 2 deletions(-) diff --git a/docs/hrx.md b/docs/hrx.md index f8ad5eea2..a1c96fb7c 100644 --- a/docs/hrx.md +++ b/docs/hrx.md @@ -304,6 +304,32 @@ Still open under #1962: two prefill kernels that re-quantise the activations int (Q5_K and Q6_K weights at 512 tokens) keep a per-block term in half precision and still return NaN above 65504. And the limit is moved, not removed: an activation above 4.19e6 would overflow again. +### Fixed: Qwen3-VL on `HRX0`: it decodes again, and its rotation runs on the device ([engine#1467](https://github.com/1bit-MONSTER/engine/issues/1467)) + +Two faults, both in the text model's graph rather than in the projector: + +- **A refused view (llama.cpp #283).** A deepstack model adds a slice of the wide input embedding to each of + its first layers, taken as a view of a view. With the first view's node in another split, the HRX graph + never holds the wide tensor, the slice could not alias its input, and the graph was refused at + `unsupported HRX node 20: VIEW`. So Qwen3-VL-2B could not decode even text, and every image request failed + with it. Such a view is external to the graph and a layout op writes nothing, so the scheduler now skips it. +- **The rotation on the CPU (llama.cpp #286).** The model's rotation is the interleaved multi-section one + with sections that do not collapse to the plain rotation, so 56 `ROPE` nodes per text graph ran on the CPU + and cut the graph into 114 splits. The two generic rotation kernels gained a mode that reads the position + section each pair selects; the text graph is now 2 or 3 splits with `PAD` (and `GET_ROWS` on token + batches) left on the CPU. + +Measured at pin `50d28600a` on gfx1151 (2026-11-10, power mode `performance`, Qwen3VL-2B-Instruct Q4_K_M with +`mmproj-Qwen3VL-2B-Instruct-F16`): `test-backend-ops -o ROPE -b HRX0` 166/166; 12 greedy text tokens +byte-identical to the CPU backend; `llama-bench -p 512 -n 64 -r 3` at 6,481 and 6,532 tok/s prefill, 133.7 +and 135.5 tok/s decode (2,331 and 57.8 before the rotation moved). `llama-server` with the projector on +`HRX0`: the first request for the 448x448 test image takes 2.99 s (`prompt_ms` 2,782 for 227 prompt tokens) +against the 16.97 s this issue was filed on, and answers "A red rectangle and a green circle." + +The projector's own graph still runs 103 of 626 nodes on the CPU for an image of that size (its 2-D rotation, +the attention score matmul and softmax, `IM2COL`, `UPSCALE`); with it offloaded, encoding the image is no +faster than with `--no-mmproj-offload` (0.5 to 1.1 s against 0.4 to 0.6 s). + ### MoE decode and the kernel cache (llama.cpp #32, #33) - **MoE decode.** With one to four tokens, a `MUL_MAT_ID` (a MoE expert matmul) runs as a GEMV over diff --git a/registry/architectures.json b/registry/architectures.json index 705031c6d..031dafa8d 100644 --- a/registry/architectures.json +++ b/registry/architectures.json @@ -1,7 +1,7 @@ { "about": "HF architecture -> GGUF architecture and the backends whose code accepts it. Generated by tools/registry_build.py from the pinned sources; do not edit.", "sources": { - "llama.cpp (hrx)": "fa533bd21053eeb7eab5354aef749c44cde86987" + "llama.cpp (hrx)": "50d28600a5f7a38df390709b94c463df02785a54" }, "counts": { "hrx": 298, diff --git a/third_party/llama.cpp b/third_party/llama.cpp index fa533bd21..50d28600a 160000 --- a/third_party/llama.cpp +++ b/third_party/llama.cpp @@ -1 +1 @@ -Subproject commit fa533bd21053eeb7eab5354aef749c44cde86987 +Subproject commit 50d28600a5f7a38df390709b94c463df02785a54