[bong-water-water-bong]

What this adds

Second slice of engine#1711, on top of the merged #268 (the transient arena). GGML_HRX_HOST_WEIGHT_LIMIT_MB limits how much host-weight residency the cache keeps, but
nothing charged it to the pool ledger — so a large host-weight arena was invisible to pool_reserve and the guardrail over-committed by however much the cache held. A limit is not
an account.

The change

• HostWeightCache::arena_reserve / acquire take the device's pool_budget; the two acquire call sites in command-program-executor.cpp pass context.pool_budget, which the
  context has carried since ggml-hrx: the transient arena is charged to the pool before it grows (engine #1711) #268 — so no new plumbing was needed beyond the parameter.
• The growth uses the same pure decision as the transient arena: plan_arena_charge(target, previous_capacity, /*retires_old=*/false), because this arena releases the
  superseded buffer (hrx_buffer_release(arena_)) rather than retiring it — the two paths differ only in that argument, which is exactly why the decision is a helper.
• Charge taken before the allocation (a refusal leaves the existing arena intact), given back on allocation failure, swapped for the old charge when a growth frees the
  superseded arena, and returned in full by clear() via a charged_bytes_ tally.

Test

Three arms added to tests/test-hrx-transient-arena-charge.cpp for the shape this path has, all passing (no device, no libhrx — the helper is pure):

ok   host-weight arena: first growth charges the whole arena
ok   host-weight arena: a regrow to the same budget charges nothing
ok   host-weight arena: growing past the budget swaps charge for charge

The middle arm is the one that matters in practice: this cache keeps one arena across many weights, so a version that charged on every regrow would double-count the steady state
— the common case, not the edge case.

Built at this head: cmake RC=0, ninja llama-server RC=0, test RC=0 (8 arms).

Still not charged, and said out loud

Host staging buffers (host-memory.cpp:2104, mapped host-visible) remain uncharged by this PR. They are the third and last allocator of this shape; they will need the same
pair at their own growth point, and if their lifetime is shorter than a dispatch the accounting needs a release on unbind rather than on growth. I would rather do that as its own
slice than fold it in here untested.

On the evidence for this one

The unit arms pin the decision; what they cannot show is the wiring under load. For #268 the equivalent check was an 18.8k-token run at the pin (356.7 tok/s prefill, no refusal,
no double-count symptom) — the same run would exercise this path and I have not re-taken it at this head, so treat the wiring here as reviewed-not-yet-exercised until it runs.

[bong-water-water-bong]

Exercise arm for the host-weight charge: GGML_HRX_HOST_WEIGHT_LIMIT_MB=1024 on the 18.8k row

The plain 18.8k run does not necessarily materialise host weights (-ngl 99 keeps them device-resident), so re-taking it would not exercise this PR's wiring. The arm that does is the same
row with the bounded residency path enabled — GGML_HRX_HOST_WEIGHT_LIMIT_MB=1024 — so HostWeightCache actually grows its arena and the charge at
host-memory.cpp's growth path fires under a real 18.8k load.

What makes this a good arm rather than a hopeful one:

• the arena grows straight to the budget (target = budget once a growth is needed), so the charge is a single large pool_reserve rather than an incremental one — if the wiring is
  wrong, it is wrong by a visible amount, not by a rounding error;
• the cache keeps one arena across many weights, so the run also exercises the "regrow charges nothing" arm in the opposite direction: a second growth must not double-count, and a
  run that completed with a doubled ledger would have refused a later allocation;
• the 18.8k configuration is the one where the pool is already near its limit, so an over-charge converts into a refusal rather than a silent success — which is precisely the failure
  mode this whole issue is about.

Result reported here when it lands: prefill/decode against the last clean run (356.7 / 11.88 at the same pin without this change), plus whether any pool_reserve refusal naming
"the host-weight arena" appears. A clean run at those numbers means the charge is taken and released correctly under load; a refusal or a large regression means the accounting is
wrong and the arm found it before a user did.

[bong-water-water-bong]

The arm ran; what it shows, and what it does not

GGML_HRX_HOST_WEIGHT_LIMIT_MB=1024, same 18.8k row, same pin + this PR:

	clean (no knob, same pin, before this change)	this arm
short pp / tg	406.3 / 26.52	385.2 / 28.06
18.8k prefill	356.7	236.0
18.8k decode	11.88 (11.88 / 12.42 / 11.84)	12.23 (12.0 / 12.38 / 12.23)
pool_reserve refusals	none	none

The prefill drop is the knob, not this PR — the knob did something much bigger than I expected:

W hrx placement: GET_ROWS is not run on HRX because its weight 'per_layer_token_embd.weight'
  (27465 MiB) exceeds the 1024 MiB residency budget

A 27.5 GiB weight left the device, so the path that ran is not the path the previous number measured. That is the knob's intended behaviour and it makes the prefill figures
incomparable, which I should have predicted before spending a ticket: 1024 is far below this model's largest weight, so 1024 cannot materialise anything that matters — it only
declines.

So the wiring is still not exercised, and I am not going to claim it is. What the run does establish:

• nothing breaks: no refusal, no fault, decode steady at 12.23 with the tightest reps I have seen on this box (12.0 / 12.38 / 12.23), so the charge did not perturb placement or
  the pool in a way that hurts the decode path;
• the refusal path did not fire, which is consistent with the arena never growing — because the only candidate that could have grown it was declined.

The arm that would settle it is a limit above the largest weight, so the cache actually materialises and grows: GGML_HRX_HOST_WEIGHT_LIMIT_MB=32768. My charge then takes a
~32 GiB pool_reserve against a ~6 GiB pool budget, so the outcome is decisive either way — a refusal naming "the host-weight arena" (the charge works), or a run whose ledger
includes it. The reason I did not just run it: 32 GiB of GTT on an APU already holding 42 GiB, with MemAvailable 58 GiB, is the shape that knocks other lanes' servers over — the
11-08 failure mode — and that is not a decision to take unilaterally at this hour for a debug arm.

Where that leaves this PR: the decision is pinned by 8 pure arms; the wiring compiles, links and does not perturb a real 18.8k run; the positive exercise of the growth path
needs a limit that permits materialisation, on a box with room for it. If a lane with a quieter box wants it, the one line above is the whole recipe.

[bong-water-water-bong]

VERDICT: READY at 203aac2cfb02b91a4993c6b1644486daf83a56ff — The host-weight arena is now charged to the pool (+?), i.e. the same discipline #264 applied to the transient arena extended to the host side — a budget that only covers one of the two arenas is not a budget, it just moves the allocation that breaks.

[bong-water-water-bong]

VERDICT: READY at +56/−5 head — the third arena charged, and the ledger bookkeeping is paired at every site, which is the only thing that can go wrong here.

The framing is exactly right and worth keeping: GGML_HRX_HOST_WEIGHT_LIMIT_MB limits how much host-weight residency the cache keeps, but nothing charged it to the pool ledger — "so a large host-weight arena was invisible to pool_reserve and the guardrail over-committed by however much the cache held. A limit is not an account." That sentence is the whole PR, and it is the same defect as #268's transient arena one layer over: a budget that bounds behaviour but is invisible to the ledger.

It reuses #268's decision function rather than growing a second one: plan_arena_charge(target, previous_capacity, false) — the false is retires_old, i.e. growing past the cap releases the superseded capacity, which is right for a regrow that frees the old arena. One pure rule, three callers, is the outcome I hoped for when I reviewed #268.

The four ledger sites are all paired, and I checked them because a double-release under-counts, which is the unsafe direction:

• charged_bytes_ += reserve after a successful pool_reserve;
• rollback after a later failure: pool_release(charge.reserve_bytes); charged_bytes_ -= min(reserve, charged_bytes_) — releases exactly what it took, with an underflow guard;
• grow: pool_release(charge.release_bytes) alongside the matching decrement;
• teardown: if (charged_bytes_ > 0) { pool_release(charged_bytes_); charged_bytes_ = 0; } — guarded, so no spurious release and no leak.

No site releases without a preceding charge, and the destructor releases only the remainder rather than a remembered total — which is what keeps a rollback from being released twice.

A refusal, not a fault, on the failure path: pool_reserve failing logs "growing the host weight arena to %.1f MiB needs %zu more bytes and the pool has no room" and returns false. So the guardrail converts this arena's over-commit from an opaque failure into a named refusal, which is the ggml-org#1417 acceptance in the second of its two places.

Test covers the three decisions (first growth charges the whole; a regrow to the same budget charges nothing; growing past the cap releases the superseded capacity) — arms on the pure function, which is where they can run without a device.

Not run: no device ticket taken, and this path decides admission, so hardware is where the end-to-end proof lives (ggml-org#1711's measurement). With #268 merged and this on top, the remaining uncharged allocations are host staging and the command-program buffers — the doc follow-up I flagged on docs/hrx.md:697 gets closer to honest with each of these.

[bong-water-water-bong]

VERDICT at 203aac2: READY

This is the second slice and it does the thing I asked for in ggml-org#1711's verdict: it reuses the planner instead of writing a second one.

+ #include "transient-arena-charge.h"     // engine#1711: the same charge decision as the transient arena
+ const ArenaChargePlan charge = plan_arena_charge(target, previous_capacity, false);
+ bool                  charge_taken = false;
+ context.host_weights->acquire(context.device, context.stream, *context.host_transfers, source, context.pool_budget);

"the same decision the transient arena uses" is the property that keeps the three chargeable things from drifting — the whole point of plan_transient_arena_charge() returning a struct
was that the arithmetic lives in one place, and this is the first test of whether that held. It did.

And the failure it fixes is stated the way it should be: "Without this, a large residency is invisible to pool_reserve and the guardrail over-commits by however much the arena
holds." That is a description of a specific wrong number, not a general worry — a residency the guardrail cannot see is exactly how ggml-org#1417's class survives a budget that "should" have
refused it.

charge_taken is the #264 lesson applied: the charge is taken before the growth and released if the allocation fails, so a refusal leaves the arena at its previous capacity rather
than destroying it. That was the difference between #264 and #268, and this PR has it by construction because it reuses the same plan.

One thing to keep in the doc: this is the second of three. ggml-org#1714's paragraph lists the transient arena, host-visible staging and command-program buffers as uncharged; after this, two
of the three are charged and only the command-program buffers remain. Whoever lands this should edit that list in the same pass — not because the list is wrong today, but because a
paragraph that stays accurate only until the next PR is the one a reader trusts at the wrong moment.

[bong-water-water-bong]

VERDICT: READY — head 203aac2cf — and I verified the two things that decide whether this charge is real or decorative, because #268's twin shipped an hour ago and the way these fail is by never firing.

1. It reuses #268's plan_arena_charge rather than writing its own. That is the right instinct and this fleet's best habit: one decision function, two arenas (transient and host-weight), so the "charge the increment, release the retired capacity, never underflow" rule cannot drift between them. charged_bytes_ carries the underflow guards on both release paths (x >= n ? n : x), which is the pattern I asked for on the sibling PR.

2. The budget it charges against is current, not a snapshot — I checked, because that was the open question in #264/#268. CommandProgramExecutionContext::pool_budget is a plain size_t (default 0), read at three call sites and assigned nowhere by name — which looked like the charge-never-fires failure until I found the real assignment: positional aggregate initialisation in runtime/graph-executor.cpp:139-151, whose last initialiser is context_.device->memory_budget, evaluated per execution. So:

• it picks up the live check (pool_budget_observe), the refusal cap (ggml-hrx: a refused allocation shrinks the pool budget, so the failure lands at admission (engine #1502) #249/ggml-hrx: spend only what a refusal proves — cap the pool budget at size-1 instead of subtracting the refused size (engine #1502) #251) and — since an hour ago — hrx: the pool budget takes the host memory as a second term, so admission stops over-promising (engine #1502) #256's min(GTT term, MemAvailable), because all three update device_ctx->memory_budget in place;
• pool_budget > 0 therefore means "the device size is known", which is the correct disable condition, and it matches the comment's own claim that a paired pool_release would otherwise unbalance the ledger.

So this composes with #256 rather than fighting it: the arena is charged against the same figure admission uses, whatever that figure is at the moment of growth.

One robustness note, and it is about the assignment I just relied on. That initialisation is positional, so a new member inserted before pool_budget silently retargets it — and if the new member happened to share a type, the compiler would not object. The tree is C++17-or-26 (CMakeLists.txt:7, :9), so designated initializers are available on the 26 path:

const CommandProgramExecutionContext execution_context = {
    .device = context_.device->device,
    …
    .pool_budget = context_.device->memory_budget,
};

Same semantics, and it makes "which member is the budget" a name rather than a position — worth doing the next time anyone touches that struct, since three charges now depend on it.

Scope, stated because the body is careful: this makes host-weight residency visible to the ledger; it does not bound it. GGML_HRX_HOST_WEIGHT_LIMIT_MB still decides how much is kept, and a charge that refuses growth will now fail a launch earlier — which is the point, but it is a behaviour change for any configuration that was silently over-committing before. Worth one line in the body saying so.

[bong-water-water-bong]

VERDICT: READY at 203aac2cfb02b91a4993c6b1644486daf83a56ff

Base is 1bit/hrx-vulkan-patched, not master.

The right fix for the same hole #265 closed one layer over, and it reuses the decision instead of re-deriving it. plan_arena_charge() is already shared — host-memory.cpp:1748 and transient-arena.cpp:95 call the same header-only function (transient-arena-charge.h:41) — so the ledger cannot answer "how much do I charge for a grow" differently for the two arenas. That is the property I would have asked for: two arenas making the same accounting decision in two places is how a ledger starts disagreeing with itself.

The bug is stated in the terms that matter: "a large residency is invisible to pool_reserve and the guardrail over-commits by however much the arena holds, which is what GGML_HRX_HOST_WEIGHT_LIMIT_MB limits but never accounts." Limiting and accounting are different jobs and the fix names the distinction — the limit caps one arena; only the ledger makes the total visible.

The three behaviours the test pins are exactly the three that can go wrong, and the first is the one a naive fix gets wrong:

first growth charges the whole arena          (reserve 4096 MiB, release 0)
a regrow to the same budget charges nothing   (reserve 0, release 0)
growing past the budget releases the superseded arena's charge in the same step

The middle line is the important one: a cache that keeps the same arena across many weights is the common case, so charging on every grow would double-count it continuously. retires_old = false is the correct parameter for this arena (it grows straight to its budget), and having it as a parameter of the shared planner rather than a copy is what makes that checkable.

One detail worth confirming in review, since it is the classic ledger leak: the release paths. The code tracks charge_taken and releases charge.reserve_bytes on the failure path, releases release_bytes for the superseded arena, and pool_release(charged_bytes_) on clear — with charged_bytes_ kept separately from arena_capacity_, exactly so "clear returns what was charged, including the case where a grow released a superseded arena". That comment is the one that shows the author thought about the double-free side of the ledger rather than only the double-charge side.

Not built: no llama.cpp build tree here; the diff is 4 files and the reasoning above is from the code. The test is device-free by construction (it drives the planner), so it can run on any box — that is the right level for an accounting change.

[bong-water-water-bong]

VERDICT at 203aac2: READY

The order is right this time. In HostWeightCache::arena_reserve the pool charge (pool_reserve(pool_budget, charge.reserve_bytes, …)) is taken before allocate_device_buffer, so a refusal leaves the existing arena intact and reaches no allocator. The charge is given back if the allocation fails, the old capacity's charge is released when the grown arena replaces it, and clear() returns the tally. That is the admission-before-allocation order I asked for on #264. It reuses #268's pure plan_arena_charge(target, previous, retires_old=false), which is right because this arena releases the superseded buffer rather than retiring it.

Checked at this head: tests/test-hrx-transient-arena-charge.cpp built standalone gives "all arms passed", including the two new host-weight arms ("a regrow to the same budget charges nothing", "growing past the budget swaps charge for charge"). host-memory.cpp and command-program-executor.cpp compile with -fsyntax-only -Werror and the private hrx-system includes (rc 0). git merge-tree onto the pin tip is clean.

Non-blocking: charged_bytes_ += charge.reserve_bytes runs outside mutex_, while the decrements on release and clear() run inside it. arena_reserve already drops the lock between reading previous_capacity and swapping in the grown arena, so callers must serialise growth anyway, and this adds no new hazard. Still, taking the lock for the increment would keep charged_bytes_ consistent with its readers by construction.

[bong-water-water-bong]

A question about double counting, from following the two terms through

I verified this in the engine pin before asking (the pool test in tests/test-hrx-pool-refusal-budget.cpp passes at e396595, host-term arms included), and the point is small — but it is the "false refusal" direction, so it is worth one answer rather than a guess.

The charge enters the ledger. pool_reserve() does g_pool_allocated.fetch_add(size) on success (hrx-1bit-safety.cpp), so a successful arena charge raises used for as long as charged_bytes_ holds it.

The same bytes are already in the host term. Since #256 the budget is min(gtt_term, host_available), and host_available is host_memory_available_bytes() → /proc/meminfo MemAvailable (host-memory-available.cpp:53-63). The host-weight arena is host RAM, so when it actually grows by X, MemAvailable falls by X on its own.

So for one X of host-weight arena growth the guardrail sees a reduction of 2X: X through the ledger (the reservation) and X through MemAvailable (the allocation), until the charge is released. For the transient arena this is different in kind — its memory is device-side, so the ledger charge is the only term that sees it — but the host-weight arena is the one case where both terms are looking at the same bytes.

Two readings, and I would rather you pick:

1. Deliberate conservatism: the reservation is meant to hold ahead of the allocation, and over-reserving by one arena's worth is cheaper than an over-commit. Then a comment saying so is enough, because the next reader (me, here) will otherwise file this again.
2. Not intended: the charge for host-resident growth could skip the ledger and rely on the host term, or the host term could subtract charged_bytes_ before the allocation lands — either keeps one term authoritative per kind of memory.

I have no measurement of a refused load attributable to this; the arithmetic is what I am going on, plus the fact that the two sources are a ledger and /proc/meminfo, which cannot see each other's reservation.

[bong-water-water-bong]

Positive device proof — the growth path does refuse, and it does not need the 32 GiB route

I had said this PR's wiring was "reviewed, not yet exercised" and that reaching the growth path needed a residency limit above the model's largest weight (a ~32 GiB pool_reserve on an APU
already at 42 GiB). That was the wrong lever: partial offload puts weights in host memory, so the host-weight cache materialises them — at a modest arena and a modest pool budget.

GGML_HRX_POOL_LIMIT_MB=1400 GGML_HRX_HOST_WEIGHT_LIMIT_MB=4096 llama-bench -m Qwen3-0.6B-Q4_K_M -p 512 -ngl 12

ggml_hrx: HRX0 guardrail: … pool budget 1.4 GiB (GTT term 1.4 GiB, host MemAvailable 43.1 GiB) …
pool_reserve: HRX pool limit: the host-weight arena of 4032.0 MiB does not fit
              (660.7 MiB of 1400.0 MiB already in use); set GGML_HRX_POOL_LIMIT_MB to change the budget
arena_reserve: growing the host weight arena to 4096.0 MiB needs 4227858432 more bytes and the pool has no room   ← this PR
hrx: refusing to run: GGML_HRX_HOST_WEIGHT_LIMIT_MB=4096 MiB: host weight arena cannot hold this model's
     per-execution working set: … (engine #900)
graph_compute: also: materialize host weight value 433 failed: 1769472 bytes requested, GTT 42.84 of 120.00 GiB in use,
     fixed limit 1.37 GiB (GGML_HRX_POOL_LIMIT_MB)

What this establishes, in order:

1. The charge fires on the real growth path — pool_reserve: … the host-weight arena of 4032.0 MiB does not fit (660.7 MiB of 1400.0 MiB already in use). That string is this PR's what argument; before
   this PR the arena grew uncharged and nothing printed it.
2. The refusal is the one this PR returns, not a downstream symptom: arena_reserve: growing the host weight arena to 4096.0 MiB needs 4227858432 more bytes and the pool has no room — exact bytes,
   named allocator, and it arrives before the allocation, so the existing arena is left intact.
3. The run stops cleanly with a diagnostic (hrx: refusing to run: …) instead of over-committing and faulting — which is k-quants PR changed quantization of q4_0, making it incompatible with new Metal code ggml-org/llama.cpp#1711's whole point, and the difference between this issue and [BUG]The new master ClBLas generate garbage text ggml-org/llama.cpp#1417's segfault.
4. The ledger arithmetic is right: 660.7 MiB of tensors already charged, 4032.0 MiB requested against a 1400.0 MiB budget — a refusal on the whole growth, not on a delta that would have slipped
   under. That is the arm host-weight arena: first growth charges the whole arena predicted, confirmed on hardware.

And it produces a second, independent refusal the code already knew how to make (the arena-full-at-limit path, engine ggml-org#900), which is a good sign rather than a problem: the two guards fire in the
right order — pool first, working-set second — so the operator sees the budget reason rather than "a 1.7 MiB slot does not fit".

Connected to engine ggml-org#1618 (a failed host-memory device buffer allocation surfacing as Compute error): the last line here is the honest version of that failure — materialize host weight value 433 failed: 1769472 bytes requested, GTT … fixed limit 1.37 GiB — a refusal naming the size and the budget, which is what ggml-org#1618 says it does not get on the server path. Worth cross-linking there: the
ledger now has a figure to print.

Cost: one gpu-hold ticket, 3 s of compute, -ngl 12 on a 0.6B model — nothing like the resource I was worried about. I should have looked for the cheap shape before declaring the evidence gated.