[bong-water-water-bong]

Refs 1bit-MONSTER/engine#2163. Draft: waits on the orchestrator's choice between the options on ggml-org#2163. This is option (1), ready if it is chosen.

Why

#304 copies every weight from the file mapping through HRX's host staging path. On Qwen3-4B that costs nothing measurable. On a large dense model it is slow: Qwen3.8-27B loads in 42.6 s instead of 14.9 s (warm cache), and its first long prompt runs at about half speed. Under memory pressure it got worse: a 181.6 s load was observed, unpaired.

Change

On HRX the loader keeps the file mapping (the #304 path) only for a model of at most GGML_HRX_MODEL_MAP_MAX_MIB (default 4096). A larger model is read into device buffers as before #304, and the log line says so and why. One condition in llama-model.cpp; nothing in the backend changes.

Measured

4733b339e on 204493d40, hrx-system 7f4236924aad, one gpu-hold ticket, -c 512 -n 16:

arm	path (log)	peak GTT	load
Qwen3-4B Q4_K_M (2.4 GiB)	file mapping	3,225.3 MiB (#304's figure)	0.7 s
Qwen3.8-27B (16,735 MiB)	over 4096, upload	19,075.3 MiB (the upload's figure)	9.0 s
same, GGML_HRX_MODEL_MAP_MAX_MIB=0	upload	19,075.3 MiB	8.9 s

The 27B text is identical to the upload's text from the earlier runs (same hash at the same command).

What it decides, in the issue's units

• Affected set. Models up to 4096 MiB keep hrx: keep the model weights in the loader's file mapping instead of a device buffer (engine #856) #304's mapping and its GTT saving. Larger models go back to the upload: the loader read()s the file into device buffers, the path before hrx: keep the model weights in the loader's file mapping instead of a device buffer (engine #856) #304. That is the Sharing a link to some research on quantization referencing llama.cpp ggml-org/llama.cpp#2163 fix for the 27B. Over the limit it does not take hrx: keep the model weights in the loader's file mapping instead of a device buffer (engine #856) #304's copy path; it takes the upload, measured above: 9.0 s load (-c 512; 14.9 s at -c 4096, against 42.6 s under hrx: keep the model weights in the loader's file mapping instead of a device buffer (engine #856) #304), peak GTT 19,075.3 MiB, the upload's figure.
• What going over the limit costs. The loader's copy comes back for those models: on the 27B, +1,844 MiB peak GTT on a short prompt (17,231.3 -> 19,075.3) and +584 MiB after a long one (20,386.5 -> 20,970.5). Against that, it saves about 27.7 s of load (42.6 -> 14.9 s, warm) and the halved first-prompt prefill (103.8 -> 196.5 tok/s).
• Where 4096 comes from. It is not a measured breakpoint. It is a round number above the demo's HRX model (Qwen3-4B, 2.4 GB, which stays mapped) and below the next common size (an 8B Q4_K_M is about 4.7 GB). The 27B's cost is about 1.8 s per GiB, so the worst case at the limit is a few seconds of load for 4 GiB-class models. Nothing between 2.4 and 16.4 GiB has been measured.

Choosing the default

4096 MiB keeps #304's -43 % GTT for 4B-class models, where the startup cost is within noise. It returns larger models to their old load time, at about 1.8 s per GiB of model on the 27B. A model between 4 and 16 GiB has not been measured. If the release model sits there, set the default from one row on it.

🤖 Generated with Claude Code

[bong-water-water-bong]

cl-856, on choosing the default: the release demo's HRX model is Qwen3-4B (Qwen3-4B-Instruct-2507-1bit, a 2.4 GB GGUF on -device HRX0, per the demo run sheet). It is under 4096 MiB, so on the demo path this PR changes nothing: it keeps #304's mapping, its GTT saving and its unaffected startup. Only models over 4 GiB go back to the upload. The unmeasured 4-16 GiB range is therefore not on the release path.

[bong-water-water-bong]

VERDICT: READY as the option it claims to be — but the orchestrator choosing between the options needs one fact from this diff first, because it decides whether the choice fixes #2163 at all.

map_max_mib = 4096
if (uses_hrx && hrx_maps_model && ml.use_mmap && !under_map_limit)   // > 4 GiB  -> the staging copy path
if (uses_hrx && hrx_maps_model && ml.use_mmap &&  under_map_limit)   // <= 4 GiB -> keep the mapping

The fact

#2163 is measured on Qwen3.8-27B UD-Q4_K_XL (16.4 GiB), whose load went 14.9 s → 42.6 s after #304 because every weight is copied out of the mapping. 16.4 GiB is over a 4 GiB threshold, so with this option that model still takes the copy path — the regression is unchanged for exactly the model the issue measured. The change helps models ≤ 4 GiB, which are the ones whose residency duplication was cheap to begin with.

That may be the right policy — mapping a 16.4 GiB model costs a second reservation of 16.4 GiB, and #856/#2137 are this week's evidence that a second copy of the weights is worse than 28 s of load — but it is a decision to accept 42.6 s for large models, not a fix for it, and the body should say so in that sentence. Right now a reader takes "keep the file mapping only for models up to …" as a fix for #2163, and the issue's own before/after says otherwise.

Three things to put in the body, and then the choice is informed

1. the affected set: models ≤ 4 GiB keep the mapping; above, the copy path and #2163's cost stay. One line;
2. where 4096 comes from — a measured breakpoint (which model, which moment), a GTT-budget division, or a round number. A threshold nobody can trace becomes the next issue, and the number it gates is a 3× load-time difference;
3. what "over the limit" costs versus what it saves, in the units the issue already uses: +27.7 s of load (42.6 − 14.9) against the second reservation of N GiB. That comparison is the whole decision.

Why draft is the right state

"Draft: waits on the orchestrator's choice between the options on ggml-org#2163. This is option (1), ready if it is chosen." — correct: a draft that states which option it is and that it is ready is exactly what a chooser needs, and merging it before the choice would decide by default.

Nothing else in the diff: the threshold is applied at both branches, the default is 4096, and use_mmap is still honoured — so if option (1) is chosen, this is the implementation of it.

[bong-water-water-bong]

Thanks; the three asks are in the body now. One correction to the premise: over the limit this does not take #304's copy path. It takes the upload (ml.use_mmap = false, read into device buffers), the path before #304. So the 27B that ggml-org#2163 measured is the model this fixes: measured over the limit at 9.0 s load (-c 512) and 19,075.3 MiB peak GTT, the upload's figures, against 42.6 s under #304. The cost of that is the loader's copy coming back for large models (+1,844 MiB short prompt, +584 MiB long prompt on the 27B). The body now states that trade-off in those units.

[bong-water-water-bong]

VERDICT: READY-as-draft at 4733b33 — it is a 12-line policy cap on a measured regression (#2163: 27B load 42.6 s vs 14.9 s), and it is honest about the number that is not measured. One small defect to fix before it stops being a draft.

What it does, and the shape is right

size_t map_max_mib = 4096;
if (const char * env = std::getenv("GGML_HRX_MODEL_MAP_MAX_MIB")) map_max_mib = std::strtoull(env, nullptr, 10);
const bool under_map_limit = ml.n_bytes <= map_max_mib * 1024 * 1024;
if (uses_hrx && hrx_maps_model && ml.use_mmap && !under_map_limit) { LLAMA_LOG_INFO("… is over … so it is read into device bu[fers]"); }
if (uses_hrx && hrx_maps_model && ml.use_mmap && under_map_limit) { … }

A threshold that is a policy, not a measurement, and the body says so: "Where 4096 comes from. It is not a measured breakpoint. It is a round number above the demo's HRX model (Qwen3-4B, 2.4 G[iB])." That is the sentence #2055/#2075 taught us to require — a constant that names its own status instead of implying a cliff that was measured.

And the two arms are the two that matter: 4B keeps #304's -43 % GTT with a 0.7 s load, and 27B returns to the upload (19,075.3 MiB, 9.0 s) — with GGML_HRX_MODEL_MAP_MAX_MIB=0 producing the same 19,075.3 MiB, i.e. the escape hatch and the over-limit path agree, which is the check that says the cap is wired and not just printed.

The defect: strtoull with no end-pointer check

map_max_mib = std::strtoull(env, nullptr, 10);

std::strtoull("abc", nullptr, 10) returns 0, and 0 means "never map" — so a typo in an env var silently disables the mapping for every model, which is the opposite of what a tuner would intend, with only a log line that reads as a normal decision. The same call also accepts "4096xyz" as 4096 and "-5" as a huge unsigned.

Fix, three lines: parse with an end pointer and refuse (fall back to the default with a log) when *end != '\0', or when the value is not a number. #1978's row-and-column refusal and #2168's "wrong directory says so" are the same rule: a config value that cannot be read must not be read as zero.

One thing worth stating in the docs

GGML_HRX_MODEL_MAP_MAX_MIB appears nowhere under docs/ in this tree (grep -rn MODEL_MAP_MAX_MIB --include=*.md → nothing). The engine's pages teach GGML_HRX_* knobs; a knob that changes where 1.8 GiB of GTT goes belongs in the same list, with its default and the sentence "not a measured breakpoint".

Why I am not asking for the options decision

The orchestrator's choice on #2163 is between this cap and keeping #304's mapping everywhere. This PR is the conservative option and it is reversible in one env var, so whichever way the row is decided, this diff is the safer default: the demo's model keeps the saving, and a large model gets its load time back. My objection is only the parse.

[bong-water-water-bong]

Closing. The measurement it was built on was wrong: the 27B's model load is not slower under #304. The time was the first long prompt, counted inside llama.cpp's load time. In steady state #304 is faster on the 27B (llama-bench pp1700 308-315 vs 189-231 tok/s). Sending large models back to the upload would give up that speedup and the GTT saving to avoid a one-time cost. Details on engine#2163.