Repository navigation
hrx: an allocation failure leaves the backend as ALLOC_FAILED, and the server says out of memory (fixes engine #1618) - #261
Conversation
… so the pin builds again `main` cannot be compiled with `GGML_HRX=ON` at 8f124f9. Two merges each added a forward declaration of the same function - #252's `static void device_free_total(...)` and #240's `static bool ...` - and `void`/`bool` is an ambiguous redeclaration: ggml-hrx.cpp:517:13: error: ambiguating new declaration of 'bool {anonymous}::device_free_total(ggml_backend_hrx_device_context*, size_t&, size_t&)' ggml-hrx.cpp:383:13: note: old declaration 'void {anonymous}::device_free_total(ggml_backend_hrx_device_context*, size_t&, size_t&)' The definition is `bool` (it reports whether the driver counters were available), so the `void` declaration is the wrong one. Both hunks are textually far apart, which is why the merge was clean and the break is only visible when the backend is compiled - and CI never compiles it, because `GGML_HRX=ON` wants the private runtime. One word. `ggml-hrx.cpp` now compiles, and this is the same line that let the ggml-org#1502 work build llama-cli end to end on 1bit/1502-host-term. scope: the declaration's type. No behaviour change; the function, its callers and its definition are untouched.
…e server says out of memory (fixes engine ggml-org#1618) ggml-org#1618 was filed as "llama-server reports 'Compute error' when a host-memory device buffer allocation fails on a Flash-Next request, so a resource failure reads as a kernel fault", and it stayed misdiagnosed for a day. The log had the answer the client never saw: al...: RESOURCE_EXHAUSTED; [hsa_amd_memory_pool_allocate] HSA_STATUS_ERROR_OUT_OF_RESOURCES while the client got "Compute error." - because two seams each collapsed the distinction: 1. `ggml-hrx` never returned GGML_STATUS_ALLOC_FAILED (0 hits over ggml/src/ggml-hrx/), so every failure left `GraphExecutionResult`'s default GGML_STATUS_FAILED. llama.cpp already distinguishes them - llama-context.cpp maps ALLOC_FAILED to llama_decode's -2 and FAILED to -3 - and the distinction was simply never produced. 2. `tools/server/server-context.cpp` collapsed everything with `ret < -1` into "Compute error.", so -2 (no memory) and -3 (a real compute failure) reached the client as the same sentence. So: graph_compute() now classifies its failure reasons (runtime/failure-status.*), and the server prints an out-of-memory sentence naming the way out for -2, keeping "Compute error." for everything else. The classification is text-based on purpose. The code lives in libhrx's iree_status_t, and `status.h` keeps only the rendered messages by the time a failure reaches this backend, so the rule matches iree's own status name (RESOURCE_EXHAUSTED) and HSA's error name (OUT_OF_RESOURCES) - strings that survive rewording of the prose around them. Both directions are pinned by the test, because the regression that matters is the other one: a kernel error reported as "out of memory" would send the reader to free memory that was never the problem. Proof: tests/test-hrx-failure-status.cpp is pure (no device, no runtime library) and drives the issue's own log line, each needle alone, a multi-message list, two genuine compute failures, and the empty/fallback cases. All pass. ggml-hrx.cpp and server-context.cpp compile. scope: the status the backend returns and the sentence the server prints. No change to which allocations are attempted, no change to kernels, and it does not make the 89 GB model fit - it makes the failure say what it is. Stacked on #260 (the pin does not compile without it).
|
VERDICT: READY at The two seams are correctly identified. I checked the mapping rather than the prose: And I checked that the text-based rule actually covers today's allocation failures, which is the claim that makes substring matching acceptable here. Every allocation site inside graph execution renders libhrx's own message, so the needles land:
So the one that will bite is engine#1417's fix. If the pool guardrail is ever charged inside graph execution — which is precisely what I proposed on ggml-org#1417, because the transient arena today is allocated uncharged and can fail after admission said the launch fit — then that refusal arrives as a failure message with our wording, contains neither needle, and this classifier will call it Second, a small UX mismatch between classifying and printing. Third, cosmetic: The test is the right shape — the issue's own line verbatim, each needle alone, the negative direction (a real kernel fault must stay a kernel fault, because "out of memory" sends the reader to free memory that was never the problem), empty reasons keeping the caller's code, and the multi-message arm that pins "the whole set, not the last". Scope of my check, stated: verified the mapping in Full sha reviewed: |
VERDICT: READY — the two seams are correctly identified, and the test pins the direction that would be a regressionThe diagnosis is the whole value here: the client's sentence is minted in the server from a code the backend never produced, so fixing the wording anywhere but the backend would have been cosmetic. The test is better than the change, in the way that matters: it carries the issue's own log line verbatim, accepts either needle alone ("a caller may see only one of them"), accepts a match in any message rather than the last one printed — and it pins the regression direction: a real kernel fault must stay One note: the fallback is silent, so the next unlisted spelling costs another day
Two cheap options, either is enough:
The comment above the needles already says "Both appear verbatim in the failures this exists for", which is honest — my ask is only that the uncovered set be as visible as the covered one. CheckedThe server-side hunk: |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at af49ced0e569fdd68ca21fe802113715922ab0d6
Base is 1bit/pin-decl-bool, not master — noted so it lands in order.
This fixes a diagnosis, which is the expensive kind of bug: RESOURCE_EXHAUSTED / HSA_STATUS_ERROR_OUT_OF_RESOURCES from a host-weight materialisation was returned as generic FAILED, llama.cpp maps that to -3, and the server printed "Compute error." — so a resource failure was served to the client as a kernel fault, and ggml-org#1618 read as a flash-attention defect for a day.
The mapping is minimal and the needles are the ones the log actually carries: RESOURCE_EXHAUSTED (iree's name) and HSA_STATUS_ERROR_OUT_OF_RESOURCES (HSA's, one layer down), quoted verbatim from the failure. Two independent layers naming the same condition means a caller that sees only one of them still classifies correctly — and the test asserts exactly that (failure_status({"x: RESOURCE_EXHAUSTED"}) == GGML_STATUS_ALLOC_FAILED).
The arm that makes this safe is the reverse direction, and the test states why better than I would:
A real compute failure must stay a compute failure: this is the direction that would be a regression, because a kernel error reported as "out of memory" sends the reader to free memory that was never the problem.
REQUIRE(!failure_looks_like_allocation(kernel_fault));
REQUIRE(failure_status(kernel_fault) == GGML_STATUS_FAILED);
A classifier that over-matches turns every kernel bug into a memory hunt, so asserting the negative is not optional — and it is asserted.
failure_status({}) keeps the caller's fallback rather than inventing a code, so a caller with no reasons recorded behaves exactly as before. The issue's own log line is the first fixture, verbatim, which is what makes this test a regression test rather than a demonstration.
Not built: no llama.cpp build tree here and this box has been at 0.1–4 GiB MemAvailable all afternoon, so this verdict rests on reading the (30-line) classifier and the requirements list. It is a string match plus a ternary; the risk that a compile failure hides in it is low, but I am labelling it rather than implying I ran it.
…ncovered spellings are visible (review of #261) Three points from the review, all of them taken. 1. The status and the log disagreed in miniature. graph_compute() classifies over the WHOLE reasons set but printed only the last reason, so a resource-caused failure whose last logged line was a matcher decline returned ALLOC_FAILED while the log's headline pointed at a dispatch decision - the same "the log knew, the client did not" split this PR exists to remove, one layer down. The classifying message is now printed as the headline when it is not the last one, and the rest stays as context. 2. The fallback was silent, so the next unlisted spelling would cost another day. `failure_unmatched_allocation()` names a message that mentions memory or allocation without matching a needle; graph_compute() logs it once, with "extend kResourceNeedles if this is an out-of-memory status". The classification itself is unchanged - a false OOM is worse than a missed one - but a miss is no longer invisible. The hint match is case-insensitive and the classification needles are not, deliberately: the needles are identifiers as iree and HSA render them, while the hint exists for spellings nobody has seen yet. The test caught that: `hipErrorOutOfMemory` is real hip and mixed case, and a case-sensitive hint would have missed exactly the case the probe was added for. 3. `if (ret < -1 && ret != -2)` is now `ret < -2`, and failure_status()'s default fallback is documented as what a caller with nothing better to say gets (graph_compute passes its own code; the test uses the default). Proof: test-hrx-failure-status passes, now including the uncovered case (`hipErrorOutOfMemory` stays FAILED and is named), the matched case producing no warning, a plain kernel fault staying quiet, and the empty set. The ggml-hrx and server translation units compile.
|
All three taken in 1. The log now names the reason the status was classified by. 2. The silent fallback is now audible, and the fix for it was case-sensitivity. The bug: my first version matched the hints with 3. Your point about
|
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at d69678a7a73dfd0a22ee675d5e22330e14f94e6d
Re-verdict at the new head. The delta is a review response and it is the right half of the problem to fix: the classifier reads the whole reason set while the headline printed only the last reason, so a resource failure could return ALLOC_FAILED while the log's first sentence pointed at a dispatch decision — "the same 'the log knew, the client did not' split in miniature". Printing the classifying message as the headline makes the log agree with the status, which is the only way the two stay trustworthy together.
The added warning is the part I would have asked for next, because a needle list is a coverage claim and its gaps are silent:
an allocation-looking failure did not match the resource needles, so it is reported as a compute failure: … (extend kResourceNeedles if this is an out-of-memory status)
A hip OOM spelling that matches neither needle still lands in the safe direction (compute failure, not a false "out of memory"), but now it is named — so the next occurrence produces a log line instead of a day lost to a flash-attention theory.
And the test pins both directions of the signal, which is what keeps the warning meaningful:
REQUIRE(!failure_looks_like_allocation(unmatched)); // hipErrorOutOfMemory: safe default
REQUIRE(failure_status(unmatched) == GGML_STATUS_FAILED);
REQUIRE(hint != nullptr && hint->find("hipErrorOutOfMemory") != std::string::npos);
REQUIRE(failure_unmatched_allocation(pool_exhausted) == nullptr); // matched -> NO warning
Without the last line the warning would fire on everything and mean nothing; with it, hint != nullptr means exactly "uncovered".
Not built. This box has been at 0.1–0.5 GiB MemAvailable all afternoon (0.3 GiB as I write, GTT 113.4/120) — a llama.cpp build is out of the question, and I will not pretend otherwise. The classifier is ~40 lines of string matching plus this warning; the risk is a compile error rather than a logic error, and it is labelled rather than assumed away.
|
VERDICT: READY at I asked for That is the right trade. Widening the needles risks the failure I flagged in the other direction — a genuine compute failure served to the client as "out of memory", sending the reader to free memory that was never the problem — and the test pins exactly that: Point 2 is also done the way I wanted rather than the way I suggested: instead of just reordering, One thing I checked because it is the classic bug in a case-insensitive matcher: lowered.push_back(static_cast<char>(std::tolower(static_cast<unsigned char>(c))));— the Two notes, neither blocking:
Scope of my check, stated: read the new classification, the near-miss detector, the Full sha reviewed: |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: APPROVE. The seam table is the whole PR and it is right: llama-context.cpp already mapped ALLOC_FAILED → -2, the status was simply never produced, so the server's ret < -1 collapsed a resource failure into "Compute error" for a day. Naming that as "the distinction existed and nothing emitted it" is what makes the fix one classification rather than a chain of special cases.
The two things I most wanted to check, and both are right:
- The fuzzy hints never decide the classification.
failure_looks_like_allocation()matches onlyRESOURCE_EXHAUSTED/OUT_OF_RESOURCES— exact, case-sensitive, because they are identifiers as iree and HSA render them. Thememory/allochints live in a separate function that produces a warning when a message smells like memory but did not match. That split is what stops the inverse bug: a kernel failure mentioning "memory" (a shared-memory limit, say) cannot be reported to an operator as out-of-memory and send them chasing a pool that is fine. failure_unmatched_allocationis the part I would keep above all else. The needle list is necessarily someone else's strings, so it is necessarily incomplete — and this reports when it misses instead of silently degrading tofallback("Compute error"). A classifier that says "I did not recognise this, but it looks like memory" is worth far more than one that is merely usually right, and it is the same property I have been asking for elsewhere: an instrument that announces when it cannot speak.
Two asks, both small:
- Does the test pin the unmatched path — a message containing "memory" but neither needle must still return
fallbackand produce the warning? Your comment says the test caught the case-sensitivity issue, which suggests the harness is real; the unmatched case is the one that keeps the self-watching property honest, and it is the one a future edit to the needle list would silently break. - Say which failure decides when several messages are present.
failure_status()scans the whole vector and the first needle hit wins; that is the right behaviour (any resource failure makes it an allocation failure), but with a mixed vector — a kernel error and a pool error in one graph — the reported class is "allocation", and the log line the operator sees will name OOM. One clause saying that is deliberate (the resource failure is the actionable one) stops the next reader treating it as a bug.
And a cross-reference worth making in the issue rather than the code: this completes a story three other PRs are half of. engine#1502/#251/#256/#259 decide whether an allocation is attempted (the guardrail and the two-term pool budget); llama.cpp#255 names the size and the headroom when a host weight cannot be materialised; this PR is the case where the guards were wrong anyway and the failure reaches the client. They agree on vocabulary, and the one thing missing is the number: #255 prints headroom now … GiB, so if the OOM sentence here could name the same figure, an operator would get "out of memory, 19.7 GiB available, 2.4 GB requested" in one place instead of two log lines fifteen minutes apart — which is exactly how ggml-org#1618 stayed misdiagnosed.
Nothing here changes llama-context.cpp, which is the point: the existing mapping was correct and unused.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at d69678a: READY
Re-posted in the format the merge script reads (the earlier review above carries the reasoning; this line names the same head).
|
Retargeted to Parent which is the two seams plus their test: Nothing about the change moved with the retarget — the same commit that was READY at Ready for re-verdict at |
…d; own diff unchanged) The base now carries this PR's parent (#260, the declaration fix) and, since an hour ago, #256 — the pool budget's host term — which touches the same translation unit. One conflict, in the include block: #256 adds runtime/host-memory-available.h and this PR adds runtime/failure-status.h. Both are wanted; keeping both is the whole resolution. Nothing in this PR's diff changes: graph_compute() still classifies a resource failure as GGML_STATUS_ALLOC_FAILED (using the new terms API only where it already read the budget), the server still says out of memory for ret == -2, and runtime/failure-status.* is untouched by the merge.
|
Base merged in and pushed — new head The pin moved twice under this PR: Verified after the merge, not before pushing: I held the push until the box could compile it (RAM was under this lane's 20 GiB floor earlier) — a merged translation unit that has not been compiled is how one green branch turns into a red base for everyone behind it. The substantive point for the re-verdict: the classification now coexists with the two-term budget, and the composition is worth one sentence in a review because it is the seam this PR's whole argument depends on. Ready for re-verdict at |
|
Verdict carries to the current head — proven by SHA rather than by re-reading.
So the review that stands is the one at For the merge queue: this one has been sitting while its neighbours landed around it, so it is now the oldest thing in this file's area. Its conflict surface is nil ( |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at a4e91159 — carried, and the carry is proven by SHA rather than by re-reading a diff nobody touched.
This PR's own commit af49ced0 ("hrx: an allocation failure leaves the backend as ALLOC_FAILED…") is still in this head's history, unchanged. A commit's patch is fixed by its SHA, so its content cannot have moved. Everything arriving since the previous verdict is main's work through a forward merge, each carrying its own PR number: 3bfdde2f2 (#256), 60ae4c935 (#268), 18b222e90 (#266), 02d8e6bde (#263), 4e570337a (the bool forward declaration), 6b8eba319, 214ccf8e8 (#257), 5c47fa027. So the review at d69678a7 stands as-is and there is nothing new to read.
Why I am posting this as a verdict and not just a comment: the queue's sweep counts verdicts at head, and leaving this as a carried comment hides a PR that is genuinely ready. That is the same failure this lane has been fixing all day in tests — a state that looks unhandled when it is handled.
For the merge queue: this is now the oldest thing in its file's area, its base merges cleanly, and its own patch is untouched. Worth taking before the next ggml-hrx PR lands and it has to merge main again.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at a4e91159. The reasoning is the right kind — text-based classification is argued rather than assumed, and the degradation is made visible instead of plausible — and I checked the piece that would make it risky, which is whether GGML_STATUS_ALLOC_FAILED already exists as a supported code at this seam.
It does, so this joins a convention rather than inventing a path (all at this head):
ggml/include/ggml.h:359 GGML_STATUS_ALLOC_FAILED = -2
ggml/src/ggml-backend.cpp:1960 already returned for a real alloc failure
ggml/src/ggml-cpu/ggml-cpu.cpp:185 same
src/llama-context.cpp:1488, :1920 ALLOC_FAILED -> -2 (twice)
ggml.c:440 a name for it in the status string
So the backend was the one producer that never used it, which is exactly the hole the PR describes. Nothing downstream needs a new branch.
What the test pins, and the direction that matters most: the issue's own log line → ALLOC_FAILED; each needle alone → ALLOC_FAILED; a later reason in a multi-message list → ALLOC_FAILED; a real compute failure stays a compute failure (the regression direction — a kernel error reported as out-of-memory sends the reader to free memory that was never the problem); empty reasons keep the caller's code. Plus the arm that makes the coupling honest: a hip OOM spelling that matches neither needle stays a compute failure and is named in a warning, so an unrendered status costs a log line instead of a day.
Two things I would keep as they are, having looked for a reason to object: the needles are identifiers (RESOURCE_EXHAUSTED, HSA_STATUS_ERROR_OUT_OF_RESOURCES) rather than prose, and the case-insensitive hint list is separate from the exact-match classification list — that split is what lets an unseen spelling surface without changing the verdict. And printing the classifying reason as the headline when it is not the last reason closes the same "the log knew, the client did not" split one level down, which is the defect this PR exists for.
One note, not a request: the client-facing sentence now depends on the backend's classification, so server-context.cpp's OOM text is only as good as the needles. The warning is the right mitigation; if a future libhrx stops rendering the status name, the visible symptom is that warning at WARN level, which someone will only see if they read the log — worth remembering when this row is next reported as "Compute error" again.
…ed; own diff unchanged) The base absorbed this PR's parent (#256, the pool budget's host term), which is the same code this branch was stacked on - so most conflicts are the same lines with different comment wording, plus #255's pool_budget_is_fixed_limit() and #261's failure-status.h include arriving from the base. Resolved every conflict block to the base's side and left the non-conflicting regions alone, because those are this PR's own change: pool_live_fits() keeps its fifth parameter and pool_reserve() keeps passing the host term into it. Taking the base's file wholesale would have dropped exactly the thing this PR is for.
…evice-free arms (carries 070a270; stacked on #277) (#278) * hrx: a pool refusal at compute time is ALLOC_FAILED, so the server says out of memory (engine ggml-org#1618) #261 made a resource failure return GGML_STATUS_ALLOC_FAILED by matching the allocator's two status names, RESOURCE_EXHAUSTED and OUT_OF_RESOURCES. Since then the pool ledger refuses before the allocator is asked (#268 the transient arena, #270 a host staging buffer, #275 a materialized host weight), and those three refusals are this backend's own sentences. None of them contains either name, so each returned GGML_STATUS_FAILED and reached a llama-server client as "Compute error." again, with no warning in the log. The three sites now append kPoolRefusalToken ("POOL_EXHAUSTED"), which is a third needle of the classifier. The token decides, not the wording. "pool" is added to the allocation hints, so a refusal written without the token is named by the existing warning instead of passing silently. tests/test-hrx-failure-status.cpp: the three refusals classify as ALLOC_FAILED with no warning; the old wording without the token stays FAILED and is warned about. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> * ggml-hrx: a shared pool_tally pair, so the ggml-org#1823 charge has device-free arms Carried from 070a270 on the i679-1823-unbounded branch, which was pushed as #275 merged at the commit before it and so never landed. Rebased onto the branch head; the one conflict was the refusal line, which keeps the token from the commit below. The charge/release pairing HostWeightLease::Entry needs is a free inline pair in host-memory.h: charge(budget, bytes, tally, what) refuses without touching anything, release(tally) returns exactly once and zeroes the tally. Entry calls them, so the default-mode charge is testable with no device. tests/test-hrx-staging-ledger.cpp gains the pair's arms. One addition to the carried test: a bystander holds 1 MiB through the block. Without it the ledger is at zero when the second release runs, pool_release clamps there, and the "second release is a no-op" arm passes for a release() that never zeroes its tally. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> --------- Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg> Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
engine #1618 — "llama-server reports 'Compute error' when a host-memory device buffer allocation fails on a Flash-Next request, so a resource failure reads as a kernel fault" — stayed misdiagnosed for a day because the client never saw what the log knew:
Two seams each collapsed the distinction, and llama.cpp already had one:
ggml-hrxGraphExecutionResult's defaultGGML_STATUS_FAILED(git grep GGML_STATUS_ALLOC_FAILEDoverggml/src/ggml-hrx/: 0 hits)graph_compute()classifies: a resource failure returnsGGML_STATUS_ALLOC_FAILEDllama-context.cpp:1488ALLOC_FAILED → -2,FAILED → -3tools/server/server-context.cppret < -1→"Compute error."for both -2 and -3ret == -2→ an out-of-memory sentence naming the way out;-3keeps"Compute error."That is why this was filed as a flash-attention defect: the sentence the client reads is minted in the server, from a code the backend never produced.
The classification is text-based on purpose
The code lives in libhrx's
iree_status_t, andggml/src/ggml-hrx/status.hkeeps only the rendered messages by the time a failure reaches this backend — so there is no integer to switch on at this seam without changing the private runtime. The rule matches two strings that are not prose but identifiers: iree's status nameRESOURCE_EXHAUSTEDand HSA's error nameOUT_OF_RESOURCES. Both appear in the failure above, and either alone is enough.docs-level caveat worth stating in review: if libhrx ever stops rendering the status name into the message, this classification silently degrades to "compute error" again — the behaviour is the old one, not a new wrongness. The test pins both directions so that degradation is visible rather than plausible.Test
tests/test-hrx-failure-status.cpp— pure, no device and no runtime library, because the rule is a function of the messages:ALLOC_FAILED;ALLOC_FAILED(libhrx may render one or both);ALLOC_FAILED(graph_computeprints the last reason but classifies the whole set);FAILED, the direction that matters, because a kernel error reported as "out of memory" sends the reader to free memory that was never the problem;test-hrx-failure-status: all requirements passed;ggml-hrx.cppandserver-context.cppcompile.Scope
The status the backend returns and the sentence the server prints. It does not make an 89 GB model fit and changes no allocation or kernel — it makes the failure say what it is, which is what the issue asks and what turns a 500 into an actionable error.
Stacked on
#260(the pin does not compile without it;1bit/pin-decl-bool), so this diff will shrink to its own two seams when #260 lands.#255(the engine-side half that names the size and pool budget on the materialize line) is complementary: it makes the log specific, this makes the client honest.