From 8d2a0307655dfd9d412ff99e808a51af06740e1a Mon Sep 17 00:00:00 2001 From: bong-water-water-bong Date: Wed, 11 Nov 2026 03:21:37 -0400 Subject: [PATCH 1/3] bump llama.cpp to 209091986: the model weights stay in the file mapping on HRX, so the loader no longer holds a second copy in GTT (fixes #856) Pin 1703bfeaa -> 209091986: llama.cpp #303 (engine #1962) and #304 (engine #856). #304 keeps the model in the loader's file mapping on HRX: an HRX0_Mapped host buffer with no device allocation, from which HRX makes its device copies. Qwen3-4B Q4_K_M peak GTT 5,665.3 -> 3,225.3 MiB on a short prompt; text identical; KLD 0; steady-state speed unchanged. What remains after a long prompt (prefill reads q4_k/q6_k in the stored layout, so HRX keeps a device copy in each layout: 6,311.8 -> 6,055.8 MiB) is documented in docs/hrx.md and tracked in #2137. tools/hrx_gtt_split.py counts an HRX0_Mapped buffer as host memory. Co-Authored-By: Claude Opus 5.5 --- docs/hrx.md | 35 +++++++++++++++++++++++++++++++---- registry/architectures.json | 2 +- third_party/llama.cpp | 2 +- tools/hrx_gtt_split.py | 9 +++++++-- 4 files changed, 40 insertions(+), 8 deletions(-) diff --git a/docs/hrx.md b/docs/hrx.md index d9feccefd..b91f37f20 100644 --- a/docs/hrx.md +++ b/docs/hrx.md @@ -864,7 +864,34 @@ length peak at 8.7 GiB instead of growing; greedy answers are identical, and lla is unchanged. A repeated prompt length right after a flush costs 1.75 s instead of 1.12 s, since its program is rebuilt. -### Server memory: HRX reads the weights instead of mapping them ([engine#856](https://github.com/1bit-MONSTER/engine/issues/856), [engine#1339](https://github.com/1bit-MONSTER/engine/issues/1339)) +### Server memory: the weights stay in the file mapping, and HRX holds the device copy ([engine#856](https://github.com/1bit-MONSTER/engine/issues/856), [engine#1339](https://github.com/1bit-MONSTER/engine/issues/1339)) + +**Since [llama.cpp #304](https://github.com/1bit-MONSTER/llama.cpp/pull/304) (engine#856).** With a model on HRX and the default +`-lm mmap`, the loader no longer uploads the model into an `HRX0` device buffer. HRX reports `buffer_from_host_ptr`, so the loader +wraps its file mapping as an `HRX0_Mapped` buffer. That buffer is host memory: no copy, no device allocation, and no import (the +userptr import described below is not used). HRX makes its own device copy of each weight from those pages. The model's file pages +stay in the page cache, which is file-backed and reclaimable. `GGML_HRX_MODEL_BUFFER_COPY=1`, or any +`GGML_HRX_HOST_WEIGHT_LIMIT_MB`, brings back the upload described in the next paragraph. Measured at llama.cpp `209091986` (#304 on `b6f65a273`) +and hrx-system `7f4236924aad`, with one binary for both arms (`tools/hrx_gtt_split.py run`, peak `drm-total-gtt`): + +| shape | upload (`GGML_HRX_MODEL_BUFFER_COPY=1`) | #304 | +|---|---:|---:| +| Qwen3-4B Q4_K_M, 5-token prompt, `-n 64` | 5,665.3 MiB | **3,225.3 MiB** | +| Qwen3-4B Q4_K_M, 1,282-word prompt, `-c 4096 -n 16` | 6,311.8 MiB | 6,055.8 MiB | +| Qwen3.8-27B UD-Q4_K_XL, 5-token prompt, `-n 16` | 19,075.3 MiB | 17,231.3 MiB | + +The generated text is identical in every pair, and `llama-perplexity --kl-divergence` against the upload's logits gives a mean KLD of 0.000000. +Every pair agrees on the top-1 token at every position (100.000 %). Speed is unchanged in steady state: `llama-bench -p 1024 -n 64 -r 3`, +alternated, gives pp1024 2,764.6 tok/s (upload) and 2,767.7 / 2,765.4 tok/s (#304), and tg64 71.5 against 72.1 / 72.3 tok/s. A fresh +process pays a one-time copy at its first prompt instead of at load time. + +**What remains of the duplicate** ([engine#2137](https://github.com/1bit-MONSTER/engine/issues/2137)): prefill reads q4_k and +q6_k weights in their stored layout, while decode reads HRX's layout, so after a long prompt HRX holds a device copy in each +layout. The second row shows it: 647 device copies, 4,400.6 MiB, for a 2,375.9 MiB model. `GGML_HRX_Q6K_PACKED_PREFILL=1` does +not change that row. A short-prompt server, or one that has not yet seen a long prompt, holds one copy per weight. + +The paragraphs below describe the upload path, which `GGML_HRX_MODEL_BUFFER_COPY=1` and `--no-mmap` still use. + With a model on HRX (`-ngl` > 0), the loader reads the weights into their device buffers with `read()` and drops each weight's page-cache pages as soon as it is read ([llama.cpp #234](https://github.com/1bit-MONSTER/llama.cpp/pull/234)). A mapped load @@ -875,8 +902,8 @@ of 14.3 s. Output and wikitext perplexity are unchanged. `--mlock` with mmap kee The earlier weight-mapping import ([llama.cpp #226](https://github.com/1bit-MONSTER/llama.cpp/pull/226), opt-in since [#231](https://github.com/1bit-MONSTER/llama.cpp/pull/231)) is gone, and `GGML_HRX_WEIGHT_HOST_PLACEMENT` does nothing. It registered the file mapping as a KFD userptr, which the kernel always write-faults, so it dirtied the file, changed -its mtime, and never finished on large models while writeback kept invalidating it (engine#1339). HRX's repacked copy of the -dense weights still sits next to the device buffer in GTT. +its mtime, and never finished on large models while writeback kept invalidating it (engine#1339). On that path, HRX's repacked copy of the +dense weights sits next to the device buffer in GTT. **What was tried to remove that duplicate, and why it does not work (2026-11-09).** The obvious route is to stop placing the native range on the device at all and let HRX repack on demand: `-ot '=CPU'` does *not* put a weight in a plain CPU @@ -1343,7 +1370,7 @@ in [hrx.md at 0baf286](https://github.com/1bit-MONSTER/engine/blob/0baf286/docs/ bit-identically. Cost: pp512 -4.7% on Qwen3-0.6B, the rest within noise. - **Weights read, page cache dropped** ([llama.cpp #234](https://github.com/1bit-MONSTER/llama.cpp/pull/234)): with a model on HRX the loader reads the weights instead of mapping the file and drops their page cache; replaces the weight-mapping - import of #226/#231 ([section above](#server-memory-hrx-reads-the-weights-instead-of-mapping-them-engine856-engine1339)). + import of #226/#231 ([section above](#server-memory-the-weights-stay-in-the-file-mapping-and-hrx-holds-the-device-copy-engine856-engine1339)). - **A cap on the graph program cache** ([llama.cpp #63](https://github.com/1bit-MONSTER/llama.cpp/pull/63)): `GGML_HRX_GRAPH_PROGRAM_CACHE`, default 64 ([section above](#server-memory-the-graph-program-cache-cap-llamacpp-63)). - **Hadamard-rotated Q4_0 files** ([llama.cpp #58](https://github.com/1bit-MONSTER/llama.cpp/pull/58)): diff --git a/registry/architectures.json b/registry/architectures.json index 1d43ec4ea..44da82f8f 100644 --- a/registry/architectures.json +++ b/registry/architectures.json @@ -1,7 +1,7 @@ { "about": "HF architecture -> GGUF architecture and the backends whose code accepts it. Generated by tools/registry_build.py from the pinned sources; do not edit.", "sources": { - "llama.cpp (hrx)": "1703bfeaaaae16686e0ea5fd22b18b6fa2163fa1" + "llama.cpp (hrx)": "20909198607426debddb34805846a788c6cb0414" }, "counts": { "hrx": 298, diff --git a/third_party/llama.cpp b/third_party/llama.cpp index 1703bfeaa..209091986 160000 --- a/third_party/llama.cpp +++ b/third_party/llama.cpp @@ -1 +1 @@ -Subproject commit 1703bfeaaaae16686e0ea5fd22b18b6fa2163fa1 +Subproject commit 20909198607426debddb34805846a788c6cb0414 diff --git a/tools/hrx_gtt_split.py b/tools/hrx_gtt_split.py index f45c6c7e8..78ba710c8 100755 --- a/tools/hrx_gtt_split.py +++ b/tools/hrx_gtt_split.py @@ -52,8 +52,9 @@ def is_gtt_buft(name): - """HRX0, HRX0_HOST, HRX1 ... are GTT on this APU; CPU / CPU_Mapped / CPU_REPACK are not.""" - return name.startswith("HRX") + """HRX0, HRX0_HOST, HRX1 ... are GTT on this APU; CPU / CPU_Mapped / CPU_REPACK are not, and neither is HRX0_Mapped: + since llama.cpp#304 that is the loader's file mapping wrapped as host memory, with no device allocation.""" + return name.startswith("HRX") and not name.endswith("_Mapped") def parse(text): @@ -243,6 +244,10 @@ def check(name, got, want): check("trace off: the report says why", NO_TRACE in report(s, 3925.0), True) check("not a -v log: the report says so", "warning: no `model buffer size` line" in report(parse("hello\n"), 1.0), True) + s = parse(FIXTURE_4B.replace("HRX0 model buffer", "HRX0_Mapped model buffer")) + check("HRX0_Mapped (llama.cpp#304) is host memory, not GTT", (s["model_gtt"], round(s["model_host"], 2)), (0.0, 2680.19)) + check("no loader copy in GTT: no duplicate line", "duplicate:" in report(s, 3225.3), False) + s = parse(FIXTURE_4B + "0.00.378.790 I load_tensors: HRX0 model buffer size = 100.00 MiB\n") check("a split buffer type is summed", s["model_gtt"], 2475.91) From e7f2b844cbe7c84e97ffa62ca85f23c2e38ea8cf Mon Sep 17 00:00:00 2001 From: bong-water-water-bong Date: Wed, 11 Nov 2026 04:20:42 -0400 Subject: [PATCH 2/3] docs/hrx.md: the device-copy count rises while GTT falls under #304, so the opt-out is not the smaller footprint (review on llama.cpp#304) Co-Authored-By: Claude Opus 5.5 --- docs/hrx.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/hrx.md b/docs/hrx.md index b91f37f20..f1ae9d088 100644 --- a/docs/hrx.md +++ b/docs/hrx.md @@ -881,6 +881,9 @@ and hrx-system `7f4236924aad`, with one binary for both arms (`tools/hrx_gtt_spl | Qwen3.8-27B UD-Q4_K_XL, 5-token prompt, `-n 16` | 19,075.3 MiB | 17,231.3 MiB | The generated text is identical in every pair, and `llama-perplexity --kl-divergence` against the upload's logits gives a mean KLD of 0.000000. +The number of device copies goes *up* while GTT goes down. With the upload, HRX copied only the 253 weights it reads in its own +layout, and every other read went to the loader's buffer. With #304 every weight HRX runs gets a device copy: 398 on the short +prompt. Those copies replace the 2,375.9 MiB loader buffer, so `GGML_HRX_MODEL_BUFFER_COPY=1` gives fewer device buffers and more GTT. Every pair agrees on the top-1 token at every position (100.000 %). Speed is unchanged in steady state: `llama-bench -p 1024 -n 64 -r 3`, alternated, gives pp1024 2,764.6 tok/s (upload) and 2,767.7 / 2,765.4 tok/s (#304), and tg64 71.5 against 72.1 / 72.3 tok/s. A fresh process pays a one-time copy at its first prompt instead of at load time. From f0937b21ad47431bd5e6e59cfabe8327bfc37aff Mon Sep 17 00:00:00 2001 From: bong-water-water-bong Date: Wed, 11 Nov 2026 04:41:22 -0400 Subject: [PATCH 3/3] bump llama.cpp to 204493d40: #304 as merged Co-Authored-By: Claude Opus 5.5 --- docs/hrx.md | 2 +- registry/architectures.json | 2 +- third_party/llama.cpp | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/hrx.md b/docs/hrx.md index f1ae9d088..b39fc8071 100644 --- a/docs/hrx.md +++ b/docs/hrx.md @@ -871,7 +871,7 @@ program is rebuilt. wraps its file mapping as an `HRX0_Mapped` buffer. That buffer is host memory: no copy, no device allocation, and no import (the userptr import described below is not used). HRX makes its own device copy of each weight from those pages. The model's file pages stay in the page cache, which is file-backed and reclaimable. `GGML_HRX_MODEL_BUFFER_COPY=1`, or any -`GGML_HRX_HOST_WEIGHT_LIMIT_MB`, brings back the upload described in the next paragraph. Measured at llama.cpp `209091986` (#304 on `b6f65a273`) +`GGML_HRX_HOST_WEIGHT_LIMIT_MB`, brings back the upload described in the next paragraph. Measured at llama.cpp `204493d40` (#304 as merged) and hrx-system `7f4236924aad`, with one binary for both arms (`tools/hrx_gtt_split.py run`, peak `drm-total-gtt`): | shape | upload (`GGML_HRX_MODEL_BUFFER_COPY=1`) | #304 | diff --git a/registry/architectures.json b/registry/architectures.json index 44da82f8f..c3d128c89 100644 --- a/registry/architectures.json +++ b/registry/architectures.json @@ -1,7 +1,7 @@ { "about": "HF architecture -> GGUF architecture and the backends whose code accepts it. Generated by tools/registry_build.py from the pinned sources; do not edit.", "sources": { - "llama.cpp (hrx)": "20909198607426debddb34805846a788c6cb0414" + "llama.cpp (hrx)": "204493d40cda9440095318ca59d0f140b3e08da6" }, "counts": { "hrx": 298, diff --git a/third_party/llama.cpp b/third_party/llama.cpp index 209091986..204493d40 160000 --- a/third_party/llama.cpp +++ b/third_party/llama.cpp @@ -1 +1 @@ -Subproject commit 20909198607426debddb34805846a788c6cb0414 +Subproject commit 204493d40cda9440095318ca59d0f140b3e08da6