Repository navigation
hrx: say which pool state produced the graph split when strict placement refuses a model - #240
Conversation
…ent refuses a model (strict-placement ticket) `GGML_HRX_STRICT=1` aborts on the first node that is not placed on HRX. On this APU the split depends on the *shared* GTT pool - other processes move it - so the same binary and the same model bytes serve on a quiet box and are refused on a busy one, and the abort line names a tensor without saying why. Measured here (strict-placement ticket, one variable at a time): `Qwen3.8-27B-UD-Q4_K_XL` serves with 37.5 GiB in use and is refused with 47.5 GiB in use, same sidecar hashes and mtimes, same model hash; the single node behind it is a 2-row f32 `GET_ROWS` (the token-embedding gather), 1 of 4087. That reads as a verdict on the model, and it has already turned two release windows' payload checks into "the shipped payload cannot serve these models". So the abort now prints the pool it was taken in: GTT in use / total and the HRX pool budget, plus the two ways out (`--no-strict-device`, or a retry on a quieter box). Numbers only, computed at the point of refusal; nothing about placement changes, no new decision, no new flag. Deliberately not done here: the two semantic fixes in the ticket (whitelist the benign embedding gather, or judge strictness by policy rather than by observation of a pool-dependent split). Both change what `--strict-device` promises, which is an owner call, and the second needs the split to be independent of the pool before it can be trusted. Refs ggml-org#1259
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at 0307fb2c0.
The diagnostic is the right fix for the class it names. A strict-placement refusal is a function of the shared GTT pool, which other
processes on the APU move — so "X is not placed on HRX" reads as a model or build problem when it is neither. The message now carries the state
that produced it:
Shared GTT pool: 47.5 of 128.0 GiB in use, HRX pool budget … GiB - the split follows that pool,
so --no-strict-device (or a retry on a quieter box) runs those nodes on the CPU
and the comment records the measured boundary (37.5 GiB in use serves, 47.5 GiB in use refuses) so the number has a context rather than being
an anecdote. Naming the escape hatch in the abort text is what turns a fatal message into an instruction.
Three things I checked because they are where a fatal-path message goes wrong:
printfarity. The format has 15 conversions — four%s, five%" PRId64 "(the fourneplusoff_nodes), two%d, one%s, three
%.1f— and the call passes 15 arguments in the same order. (My first two attempts to count this with grep/regex both miscounted, because
%" PRId64 "is a macro split across string literals; the source read by hand is what settles it. Worth knowing for anyone auditing another
one of these.)device_free_totalcannot deadlock the abort. Its body takes no locks —hrx_device_memory_infofirst,gtt_countersas the fallback —
so it is safe to call in a path that is about to die, which is the failure mode a "helpful numbers in an abort" change usually has.- No underflow or div-by-zero in the numbers printed. Both branches require
total != 0, and the used-bytes subtraction is clamped
(gtt_total > gtt_used ? gtt_total - gtt_used : 0), so a pool that reports oddly prints0.0rather than a huge unsigned value.
Merge order note: the branch carries #238's tests/test-backend-ops.cpp and #239's tests/hrx-backend-test.cpp byte for byte (both
open and READY). Not a defect — the same stale-base shape as the others tonight — but if those two land first, this rebases to its own
ggml-hrx.cpp change and the test files drop out, which is what a reviewer wants to see at merge time.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: CHANGES at 0307fb2c02100a2bd6536e0b8e57c076b71f708e
Printing the pool state on a strict abort is the right instrument. The 37.5-vs-47.5 GiB measurement shows the same binary and bytes serve or refuse depending on the shared pool, and an abort line without those numbers reads as a verdict on the model. The mechanics are fine:
hrxis the observer's own HRX backend handle;device_free_totalis forward-declared;- the
in_usesubtraction is guarded; memory_budgetis the member the buffer type already reads (:403,:446).
But the new tail states a cause and a fix that only hold for one of the ways a node ends up off HRX:
- "the split follows that pool, so … a retry on a quieter box runs those nodes" is printed on every strict abort.
first_offis the first node whose backend isn'thrx, for any reason. The same loop countsoff_unaligned_matmuls(src[0]->ne[0] % 256 != 0), and a node with no kernel for its op or type lands there too. For those, the message tells the operator the refusal is transient and to retry on a quieter box, which sends them away from a real coverage gap: the exact misreading this PR exists to prevent, in the other direction. Either:- (a) print the pool numbers unconditionally (they're always useful) but word the cause conditionally, e.g. "if the node was declined for residency (
host_weight_exceeds_residency), the split follows the shared GTT pool"; or - (b) better, have the decline record why (residency vs unsupported) and print the matching sentence.
device_supports_opknows which branch it took.
- (a) print the pool numbers unconditionally (they're always useful) but word the cause conditionally, e.g. "if the node was declined for residency (
--no-strict-deviceisn't a llama.cpp option. It's the engine's (app/hrx_strict.h:19/32/77on engine main, withONEBIT_STRICT_DEVICE=0). Someone running the fork'sllama-serverwithGGML_HRX_STRICT=1can't pass it. The fork's own way out is unsettingGGML_HRX_STRICT. Name that, and say "(1bit serve:--no-strict-device)" if you want to keep the engine's spelling.
The end-to-end capture you queued (GGML_HRX_DISABLE_DISPATCH=hip.set_rows to force the split) is a good row. Note that it forces the split by disabling a kernel, not by pool pressure, so on that very row the new "follows that pool" sentence would be false. That's item 1 in miniature.
What I ran: read the diff (one commit on 1bit/hrx-vulkan-patched) and the observer's loop (ggml-hrx.cpp:538-560); grepped for no-strict-device in the fork (only this message) and on engine main (app/hrx_strict.h). No build; your -Wall -Wextra -Wpedantic compile covers the format and argument risk.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
Withdrawing my READY at 0307fb2c0 — the CHANGES filed three minutes later is right on both counts, and I verified both rather than taking
them on trust.
1. The "follows that pool" sentence fires on paths it cannot explain. I checked the loop: first_off is whatever node ends up off HRX first
(ggml-hrx.cpp:554), and the same loop counts off_unaligned_matmuls (:548-549, node->src[0]->ne[0] % 256 != 0) — plus unsupported op/type
nodes land in the same bucket. So on an unaligned-matmul split the abort prints a pool explanation and a "retry on a quieter box" instruction that
is simply false, sending the operator away from a coverage gap. That is the misreading this PR exists to prevent, in the other direction.
2. --no-strict-device is not this binary's flag. Grepped the fork: it appears only inside this abort string (:591); the engine has it in
app/hrx_strict.h:19/32/77. A llama-server user under GGML_HRX_STRICT=1 reads "pass --no-strict-device", tries it, and has nothing — the real
fork-side escape is unsetting GGML_HRX_STRICT.
What I got wrong, precisely, because the class is worth naming: I verified the message's mechanics — printf arity 15/15, no lock in
device_free_total, the guarded subtraction — and then wrote that "naming the escape hatch in the abort text is what turns a fatal message into an
instruction" without grepping whether the hatch existed in this binary. That sentence of mine was the defect in miniature. For a diagnostic PR
the message's truth conditions are part of the code, and I reviewed the format string instead of the claim.
I would take their option (b): have the decline record why (residency vs unsupported — device_supports_op knows which branch it took) and
print the matching sentence, with the pool numbers unconditional since they are useful either way. And their point about the queued capture is
sharp: forcing the split with GGML_HRX_DISABLE_DISPATCH=hip.set_rows disables a kernel rather than pressuring the pool, so on that row the new
sentence would be false — which is item 1 in miniature, and a good reason to fix the wording before the capture is recorded.
|
Correction to the verification note in the body, because I had it wrong and it matters for anyone re-running this: So the queued capture keeps |
End-to-end verification: the forced-split capture is inThe piece I said was pending. Queued at 22:40, cut once by an expired queue wait, re-queued at 07:01 with a 6 h window, and it ran at 07:36 ( Setup: What the operator now sees (verbatim, single line): Checks that matter: the abort fires ( Also corrected below in a separate comment: |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: APPROVE — message-only, mechanism verified in the code it cites, missing verification declared and queued.
Checked the diff for the risk class a message change actually has: three %.1f against three double arguments (in_use_gib, total_gib, budget_gib), the device_free_total forward declaration placed above its first use, and no placement decision or control flow touched — first_off selection, off_detail, and the split count are byte-identical. Nothing else in the file changed.
Two things I would add, neither blocking:
1. There is a fourth mode, and it is the silent one. You describe three (serve / strict refusal under a split / load-time allocation failure) and I have hit all three today — but the one that cost me the most time prints nothing at all: a 4.7 GiB model at a 16.7 GiB pool budget produced no output for 900 s, no abort, no allocation error, no log line; the same command at 68.2 GiB finished in 19 s. That is worse than a refusal for a user, because there is nothing to read. Since this PR is about making pool state legible at the moment of a decision, note that the strict-abort path is not where the silent case lands (strict is off by default and the stall happens before any placement verdict). If the ticket's third mode ever gets a message, "pool state at the point we start waiting" is the one with the best payoff.
2. Have the queued end-to-end run confirm device_free_total is callable from here. Reading the pool inside the abort path is fine — it is terminal, so a slow query costs nothing — but it runs inside the scheduler's placement callback and under the backend's lock. A HSA query that blocks or re-enters a held lock would turn a diagnostic into a hang, which is strictly worse than the current wrong-looking message. Your -v capture on the 1 GB model with GGML_HRX_DISABLE_DISPATCH=hip.set_rows exercises exactly this path, so a clean captured line is that evidence — worth saying so explicitly when you post it, rather than only showing the line.
Your own correction — that this message cannot be verified by loading under a busy pool since mode 3 precedes placement, and so the split must be forced — is right and matches what my runs did: the pool-pressure failures I can reproduce today land before placement, not after. I did not run the queued script; I am reviewing the change and your stated verification boundary, not adding device evidence of my own.
|
Status note from the lane that was asked to triage this PR: it is not stalled, and there is nothing to fix. Its latest verdict is APPROVE at Ownership, for the record: this PR is not from |
The latest verdict on this PR is APPROVE, not CHANGES — the CHANGES is three verdicts oldReading the review sequence at head
So the last word at this head is APPROVE, and the 07:09 CHANGES it supersedes is what a "READY+CHANGES, no push since 07:00Z" Ownership, for routing: not |
|
VERDICT: CHANGES REQUESTED — one string in the new message sends the operator to a flag that does not exist. Reviewed at head What I verified (source-level; I did not build — see the note at the end)
The blocking findingThe new line ends with:
There is no
Suggested replacement for that clause — the rest of the message is good and the pool numbers are exactly what the ticket needed: "… - the split follows that pool, so unsetting GGML_HRX_STRICT (or a retry on a quieter box) runs those nodes on the CPU\n"Nit (not blocking)If What this verdict is and is not based onSource verification against the fork tree, not a build and not a run: the disk alert in force says no new full build unless The diagnostic direction itself is right: the strict abort was reporting a decision without the state that produced it, and this |
|
Correction to my verdict above: I cited A wrong line number in a review is the kind of thing that makes the rest of it cheaper to ignore, so it gets corrected rather than left. |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: CHANGES — one small one, in the fallback path of the number this PR exists to print.
What I verified first (against the tree, not the description): the forward declaration at ggml-hrx.cpp:490 matches the definition at :807 (ggml_backend_hrx_device_context *, size_t &, size_t &); hrx really is in scope in placement_observer (ggml_backend_t hrx = static_cast<ggml_backend_t>(user_data);); ggml_backend_hrx_context has the device member (backend-context.h:65-66); device->memory_budget exists and is already used by the guardrail (:833); and the format string's argument count and types match (3 %s, 5 PRId64, 2 int, 3 %.1f). The change is message-only, so the blast radius is what it should be for a diagnostic.
The finding. The new line reports in_use = total - free from device_free_total, whose last-resort branch is:
free_bytes = device_ctx->memory_total;
total_bytes = device_ctx->memory_total;That is the fallback for "neither source answered" (hrx_device_memory_info and ggml::hrx::onebit::gtt_counters both failed). In the new message it renders as:
Shared GTT pool: 0.0 of 120.0 GiB in use, HRX pool budget 98.0 GiB
i.e. the abort line would confidently report nothing in use at the exact moment the operator most needs to know whether the box or the model caused the refusal. The failure is indistinguishable from a genuinely idle pool, which is the class of bug this lane keeps finding: a plausible number manufactured out of no measurement. For a diagnostic whose whole thesis is "the split follows the shared pool, ask it", the fallback has to be able to say unavailable.
Suggested shape (nothing about the rest of the PR):
- have
device_free_totalreport whether either source answered (anbool * knownout-param, or return the enum it already distinguishes internally), and print
pool state unavailable (device query failed)instead of%.1f of %.1f GiB in usewhen it did not; or - if you would rather not touch the shared helper, compare against the fallback's signature at the call site —
free == total && total == device_ctx->memory_total— and skip the%.1ftriple in that case. Ugly but contained.
Worth noting the same fallback is already load-bearing for the scheduler (device_memory at :824-836 then min()s it against pool_remaining), so a known flag there would be useful beyond this line — but that is a bigger change and not what this PR needs.
Not checked / not claimed: I did not build this branch (device_free_total's forward declaration before its definition and the static_cast are compile-plausible but I ran no build), and I did not reproduce the 37.5/47.5 GiB transition that motivates the PR — I am reviewing the message path, not re-verifying the measurement, which the filing lane already documented.
|
Confirming the flag finding below with the grep, and owning a gap in my own review. The flag does not exist. Across the whole tree: The only strict-placement control is the environment variable And my review missed it. My verdict flagged the zero-in-use fallback in the same message and I quoted the surrounding text verbatim without checking the flag name in it. Two reviewers reading the same sentence with different attention is exactly what the second pair of eyes is for, but I should have caught a literal flag reference in a diagnostic-only diff — it is the cheapest thing in the diff to verify, and For the fix, the accurate options appear to be: name the env var ( |
…ource answered `device_free_total`'s last-resort branch sets free = total = memory_total when both `hrx_device_memory_info` and `ggml::hrx::onebit::gtt_counters` fail. The strict-placement line computes `in_use = total - free` from those, so on that path it would print Shared GTT pool: 0.0 of 120.0 GiB in use, HRX pool budget 98.0 GiB i.e. the most reassuring possible number, at the exact moment the operator is asking whether the box or the model caused the refusal, and indistinguishable from a genuinely idle pool. That is the class this lane keeps fixing: a plausible number manufactured out of no measurement. `device_free_total` now returns whether either source answered (false only on the fallback it already distinguishes internally; the other three call sites ignore the result, unchanged). The message builds its pool segment as a string, so the unavailable case can say so: Shared GTT pool: state unavailable (neither the driver nor the GTT counters answered) Compile-checked: the exact Release command line for `ggml-hrx.dir/ggml-hrx.cpp.o` from a configured HRX build (clang 24, -Werror), object produced with no diagnostics. Not run against a device: the fallback branch needs both memory sources to fail, and this is a message path. Refs the review at 0307fb2 (the CHANGES in the fallback path of the number this PR prints).
|
Answered the CHANGES at The finding was right and it is now the second shape you offered, with the helper owning the knowledge instead of the call char pool_detail[160] = "state unavailable (neither the driver nor the GTT counters answered)";
...
if (device_free_total(hrx_context->device, free_bytes, total_bytes)) {
std::snprintf(pool_detail, sizeof(pool_detail), "%.1f of %.1f GiB in use, HRX pool budget %.1f GiB", use, tot, budg);
}and the abort line's pool segment became I chose the return value over a Compile-checked, which your review noted you could not do: I took the exact Release command line for One deliberate non-change: the Whoever owns this PR: please re-review at |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at 7a6cf42863c001272567dd5e817a60e6a191be09
The point of the change is that a strict-placement refusal should say what state produced it, and the code makes that impossible to fake:
static bool device_free_total(…); // "Returns false when NEITHER source answered and the out-params are a fallback, so
// a caller can say 'unavailable' instead of printing a number it did not measure
// (a manufactured 0.0 GiB in use is the same lie the ceiling exists to prevent)."
char pool_detail[160] = "state unavailable (neither the driver nor the GTT counters answered)";
if (device_free_total(hrx_context->device, free_bytes, total_bytes)) { … }The "unavailable" branch is the part worth keeping: the failure mode being avoided is a manufactured 0.0 GiB in use in a diagnostic, which reads as "the pool is empty" when the truth is "nobody asked the pool". That is the same rule as the phase line's n=0 in ggml-org#1585/ggml-org#1586 — a zero in a report is a claim, and it has to be measured or refused.
And it names the measurement that justifies the message: "(strict-placement ticket, measured on this box: 37.5 GiB in use serves, 47.5 GiB in use refuses)". Two points either side of the line is what turns "the box was busy" from folklore into a threshold a reader can check against their own /sys/class/drm/card0/device/mem_info_gtt_used.
Noted, not a defect: this adds its own forward declaration of device_free_total — which is also what #252 added for #251's call site. Same signature, so the two compose; if #252 lands first this line is redundant, and the compiler does not care either way.
Not run: a strict-placement refusal on the device (needs an HRX build and a loaded pool). What I verified is the report's source of truth and its "unavailable" path.
… so the pin builds again (#260) `main` cannot be compiled with `GGML_HRX=ON` at 8f124f9. Two merges each added a forward declaration of the same function - #252's `static void device_free_total(...)` and #240's `static bool ...` - and `void`/`bool` is an ambiguous redeclaration: ggml-hrx.cpp:517:13: error: ambiguating new declaration of 'bool {anonymous}::device_free_total(ggml_backend_hrx_device_context*, size_t&, size_t&)' ggml-hrx.cpp:383:13: note: old declaration 'void {anonymous}::device_free_total(ggml_backend_hrx_device_context*, size_t&, size_t&)' The definition is `bool` (it reports whether the driver counters were available), so the `void` declaration is the wrong one. Both hunks are textually far apart, which is why the merge was clean and the break is only visible when the backend is compiled - and CI never compiles it, because `GGML_HRX=ON` wants the private runtime. One word. `ggml-hrx.cpp` now compiles, and this is the same line that let the ggml-org#1502 work build llama-cli end to end on 1bit/1502-host-term. scope: the declaration's type. No behaviour change; the function, its callers and its definition are untouched. Co-authored-by: agent <agent@local>
…(engine ggml-org#1711) (#268) * ggml-hrx: the forward declaration says bool, matching the definition, so the pin builds again `main` cannot be compiled with `GGML_HRX=ON` at 8f124f9. Two merges each added a forward declaration of the same function - #252's `static void device_free_total(...)` and #240's `static bool ...` - and `void`/`bool` is an ambiguous redeclaration: ggml-hrx.cpp:517:13: error: ambiguating new declaration of 'bool {anonymous}::device_free_total(ggml_backend_hrx_device_context*, size_t&, size_t&)' ggml-hrx.cpp:383:13: note: old declaration 'void {anonymous}::device_free_total(ggml_backend_hrx_device_context*, size_t&, size_t&)' The definition is `bool` (it reports whether the driver counters were available), so the `void` declaration is the wrong one. Both hunks are textually far apart, which is why the merge was clean and the break is only visible when the backend is compiled - and CI never compiles it, because `GGML_HRX=ON` wants the private runtime. One word. `ggml-hrx.cpp` now compiles, and this is the same line that let the ggml-org#1502 work build llama-cli end to end on 1bit/1502-host-term. scope: the declaration's type. No behaviour change; the function, its callers and its definition are untouched. * ggml-hrx: the transient arena is charged to the pool before it grows (engine ggml-org#1711) The guardrail was consulted in exactly one place - the buffer-type allocator - so the scratch a launch grows into was allocated uncharged and "a launch that needs more is refused here" (ggml-org#680) held for tensors only. That is why a pool exhaustion in the arena path faults instead of being refused. The charge decision is a pure helper (runtime/transient-arena-charge.h) so it can be tested with no device: reserve the growth BEFORE the superseded buffer is touched, release its charge only when it is actually freed. In diagnostic mode the arena retires superseded buffers instead of releasing them, so their charge stays - releasing it there would under-report by exactly the buffers kept around to diagnose memory problems. --------- Co-authored-by: agent <agent@local> Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
…e server says out of memory (fixes engine ggml-org#1618) (#261) * ggml-hrx: the forward declaration says bool, matching the definition, so the pin builds again `main` cannot be compiled with `GGML_HRX=ON` at 8f124f9. Two merges each added a forward declaration of the same function - #252's `static void device_free_total(...)` and #240's `static bool ...` - and `void`/`bool` is an ambiguous redeclaration: ggml-hrx.cpp:517:13: error: ambiguating new declaration of 'bool {anonymous}::device_free_total(ggml_backend_hrx_device_context*, size_t&, size_t&)' ggml-hrx.cpp:383:13: note: old declaration 'void {anonymous}::device_free_total(ggml_backend_hrx_device_context*, size_t&, size_t&)' The definition is `bool` (it reports whether the driver counters were available), so the `void` declaration is the wrong one. Both hunks are textually far apart, which is why the merge was clean and the break is only visible when the backend is compiled - and CI never compiles it, because `GGML_HRX=ON` wants the private runtime. One word. `ggml-hrx.cpp` now compiles, and this is the same line that let the ggml-org#1502 work build llama-cli end to end on 1bit/1502-host-term. scope: the declaration's type. No behaviour change; the function, its callers and its definition are untouched. * hrx: an allocation failure leaves the backend as ALLOC_FAILED, and the server says out of memory (fixes engine ggml-org#1618) ggml-org#1618 was filed as "llama-server reports 'Compute error' when a host-memory device buffer allocation fails on a Flash-Next request, so a resource failure reads as a kernel fault", and it stayed misdiagnosed for a day. The log had the answer the client never saw: al...: RESOURCE_EXHAUSTED; [hsa_amd_memory_pool_allocate] HSA_STATUS_ERROR_OUT_OF_RESOURCES while the client got "Compute error." - because two seams each collapsed the distinction: 1. `ggml-hrx` never returned GGML_STATUS_ALLOC_FAILED (0 hits over ggml/src/ggml-hrx/), so every failure left `GraphExecutionResult`'s default GGML_STATUS_FAILED. llama.cpp already distinguishes them - llama-context.cpp maps ALLOC_FAILED to llama_decode's -2 and FAILED to -3 - and the distinction was simply never produced. 2. `tools/server/server-context.cpp` collapsed everything with `ret < -1` into "Compute error.", so -2 (no memory) and -3 (a real compute failure) reached the client as the same sentence. So: graph_compute() now classifies its failure reasons (runtime/failure-status.*), and the server prints an out-of-memory sentence naming the way out for -2, keeping "Compute error." for everything else. The classification is text-based on purpose. The code lives in libhrx's iree_status_t, and `status.h` keeps only the rendered messages by the time a failure reaches this backend, so the rule matches iree's own status name (RESOURCE_EXHAUSTED) and HSA's error name (OUT_OF_RESOURCES) - strings that survive rewording of the prose around them. Both directions are pinned by the test, because the regression that matters is the other one: a kernel error reported as "out of memory" would send the reader to free memory that was never the problem. Proof: tests/test-hrx-failure-status.cpp is pure (no device, no runtime library) and drives the issue's own log line, each needle alone, a multi-message list, two genuine compute failures, and the empty/fallback cases. All pass. ggml-hrx.cpp and server-context.cpp compile. scope: the status the backend returns and the sentence the server prints. No change to which allocations are attempted, no change to kernels, and it does not make the 89 GB model fit - it makes the failure say what it is. Stacked on #260 (the pin does not compile without it). * hrx: the log names the reason the status was classified by, and the uncovered spellings are visible (review of #261) Three points from the review, all of them taken. 1. The status and the log disagreed in miniature. graph_compute() classifies over the WHOLE reasons set but printed only the last reason, so a resource-caused failure whose last logged line was a matcher decline returned ALLOC_FAILED while the log's headline pointed at a dispatch decision - the same "the log knew, the client did not" split this PR exists to remove, one layer down. The classifying message is now printed as the headline when it is not the last one, and the rest stays as context. 2. The fallback was silent, so the next unlisted spelling would cost another day. `failure_unmatched_allocation()` names a message that mentions memory or allocation without matching a needle; graph_compute() logs it once, with "extend kResourceNeedles if this is an out-of-memory status". The classification itself is unchanged - a false OOM is worse than a missed one - but a miss is no longer invisible. The hint match is case-insensitive and the classification needles are not, deliberately: the needles are identifiers as iree and HSA render them, while the hint exists for spellings nobody has seen yet. The test caught that: `hipErrorOutOfMemory` is real hip and mixed case, and a case-sensitive hint would have missed exactly the case the probe was added for. 3. `if (ret < -1 && ret != -2)` is now `ret < -2`, and failure_status()'s default fallback is documented as what a caller with nothing better to say gets (graph_compute passes its own code; the test uses the default). Proof: test-hrx-failure-status passes, now including the uncovered case (`hipErrorOutOfMemory` stays FAILED and is named), the matched case producing no warning, a plain kernel fault staying quiet, and the empty set. The ggml-hrx and server translation units compile. --------- Co-authored-by: agent <agent@local>
Refs the strict-placement ticket (
~/briefs/ISSUE-strict-placement-gtt-pressure.md)The problem it addresses
GGML_HRX_STRICT=1aborts on the first node not placed on HRX. On this APU the split follows the shared GTT pool, which other processes move, so the same binary and the same model bytes serve on a quiet box and are refused on a busy one — and the abort line names a tensor without saying why. Measured by the filing lane, one variable at a time:Qwen3.8-27B-UD-Q4_K_XLserves with 37.5 GiB in use, is refused at 47.5 GiB, same sidecar hashes and mtimes, same model hash; the node behind it is a 2-row f32GET_ROWS(the token-embedding gather), 1 of 4087.That reads as a verdict on the model, and it has already turned two release windows' payload checks into "the shipped payload cannot serve these models".
The change
The strict abort now prints the pool it was taken in — GTT in use / total, the HRX pool budget — plus the two ways out (
--no-strict-device, or the same load on a quieter box). Two lines: the locals that read the pool, and the tail of the existing abort string. No placement decision changes, no new flag, no behaviour change beyond the message.Mechanism, for the record (checked in the code, not assumed):
device_supports_opdeclines a node whenhost_weight_exceeds_residency(op)fires (ggml-hrx.cpp:1355), and that predicate is bounded byHostWeightCache::residency_limit(), which derives from the GTT pool budget — pool pressure shrinks residency, the gather is declined, one CPU node, one split, and strict aborts.Verification state, stated plainly
-Wall -Wextra -Wpedantic(so-Wformatis on) and this edit compiles with zero warnings — for a message-only change that is the whole risk class (format/argument count and types), and it is machine-checked.~/lb/tattn-strict-msg.sh,GGML_HRX_STRICT=1 GGML_HRX_DISABLE_DISPATCH=hip.set_rowson a 1 GB model withllama-bench -v). I will post the captured line here when it runs. The site itself is known-exercised: the ticket's logs contain the old message from exactly this abort.Deliberately not in this PR
The ticket's two semantic candidates — whitelist the benign embedding gather, or judge strictness by policy rather than by observing a pool-dependent split. Both change what
--strict-devicepromises, which is an owner call, and the second is not trustworthy while the split still follows the pool.Related measurement (filed to the ticket)
At 94.1 GiB in use the model fails before placement (
pool budget 9.9 GiB->unable to allocate HRX0 buffer), so this family has three modes, not two: serve / strict refusal under a split / load-time allocation failure. Consequence: this message cannot be verified by loading under a busy pool — the strict path is never reached — which is why the verification forces the split instead.