Skip to content

hrx: an allocation failure leaves the backend as ALLOC_FAILED, and the server says out of memory (fixes engine #1618) - #261

Merged
bong-water-water-bong merged 4 commits into
1bit/hrx-vulkan-patchedfrom
1bit/1618-alloc-failure
Nov 9, 2026
Merged

bong-water-water-bong merged 4 commits into
1bit/hrx-vulkan-patchedfrom
1bit/1618-alloc-failure

Conversation

@bong-water-water-bong

Copy link
Copy Markdown

engine #1618 — "llama-server reports 'Compute error' when a host-memory device buffer allocation fails on a Flash-Next request, so a resource failure reads as a kernel fault" — stayed misdiagnosed for a day because the client never saw what the log knew:

al...: RESOURCE_EXHAUSTED; [hsa_amd_memory_pool_allocate] HSA_STATUS_ERROR_OUT_OF_RESOURCES
client: send_error: task id = 0, error: Compute error.   ->  HTTP 500

Two seams each collapsed the distinction, and llama.cpp already had one:

layer before after
ggml-hrx every failure left GraphExecutionResult's default GGML_STATUS_FAILED (git grep GGML_STATUS_ALLOC_FAILED over ggml/src/ggml-hrx/: 0 hits) graph_compute() classifies: a resource failure returns GGML_STATUS_ALLOC_FAILED
llama-context.cpp:1488 already maps ALLOC_FAILED → -2, FAILED → -3 unchanged — it was simply never given the former
tools/server/server-context.cpp ret < -1 → "Compute error." for both -2 and -3 ret == -2 → an out-of-memory sentence naming the way out; -3 keeps "Compute error."

That is why this was filed as a flash-attention defect: the sentence the client reads is minted in the server, from a code the backend never produced.

The classification is text-based on purpose

The code lives in libhrx's iree_status_t, and ggml/src/ggml-hrx/status.h keeps only the rendered messages by the time a failure reaches this backend — so there is no integer to switch on at this seam without changing the private runtime. The rule matches two strings that are not prose but identifiers: iree's status name RESOURCE_EXHAUSTED and HSA's error name OUT_OF_RESOURCES. Both appear in the failure above, and either alone is enough.

docs-level caveat worth stating in review: if libhrx ever stops rendering the status name into the message, this classification silently degrades to "compute error" again — the behaviour is the old one, not a new wrongness. The test pins both directions so that degradation is visible rather than plausible.

Test

tests/test-hrx-failure-status.cpp — pure, no device and no runtime library, because the rule is a function of the messages:

  • the issue's own log line verbatim → ALLOC_FAILED;
  • each needle alone → ALLOC_FAILED (libhrx may render one or both);
  • a multi-message list where only a later reason is the resource one → ALLOC_FAILED (graph_compute prints the last reason but classifies the whole set);
  • two genuine compute failures → FAILED, the direction that matters, because a kernel error reported as "out of memory" sends the reader to free memory that was never the problem;
  • empty reasons → the caller's fallback code, not an invented one.

test-hrx-failure-status: all requirements passed; ggml-hrx.cpp and server-context.cpp compile.

Scope

The status the backend returns and the sentence the server prints. It does not make an 89 GB model fit and changes no allocation or kernel — it makes the failure say what it is, which is what the issue asks and what turns a 500 into an actionable error.

Stacked on #260 (the pin does not compile without it; 1bit/pin-decl-bool), so this diff will shrink to its own two seams when #260 lands. #255 (the engine-side half that names the size and pool budget on the materialize line) is complementary: it makes the log specific, this makes the client honest.

agent added 2 commits November 9, 2026 08:52
… so the pin builds again

`main` cannot be compiled with `GGML_HRX=ON` at 8f124f9. Two merges each added a forward
declaration of the same function - #252's `static void device_free_total(...)` and #240's `static
bool ...` - and `void`/`bool` is an ambiguous redeclaration:

    ggml-hrx.cpp:517:13: error: ambiguating new declaration of
        'bool {anonymous}::device_free_total(ggml_backend_hrx_device_context*, size_t&, size_t&)'
    ggml-hrx.cpp:383:13: note: old declaration
        'void {anonymous}::device_free_total(ggml_backend_hrx_device_context*, size_t&, size_t&)'

The definition is `bool` (it reports whether the driver counters were available), so the `void`
declaration is the wrong one. Both hunks are textually far apart, which is why the merge was clean
and the break is only visible when the backend is compiled - and CI never compiles it, because
`GGML_HRX=ON` wants the private runtime.

One word. `ggml-hrx.cpp` now compiles, and this is the same line that let the ggml-org#1502 work build
llama-cli end to end on 1bit/1502-host-term.

scope: the declaration's type. No behaviour change; the function, its callers and its definition are
untouched.
…e server says out of memory (fixes engine ggml-org#1618)

ggml-org#1618 was filed as "llama-server reports 'Compute error' when a host-memory device buffer allocation fails on a
Flash-Next request, so a resource failure reads as a kernel fault", and it stayed misdiagnosed for a day. The
log had the answer the client never saw:

    al...: RESOURCE_EXHAUSTED; [hsa_amd_memory_pool_allocate] HSA_STATUS_ERROR_OUT_OF_RESOURCES

while the client got "Compute error." - because two seams each collapsed the distinction:

1. `ggml-hrx` never returned GGML_STATUS_ALLOC_FAILED (0 hits over ggml/src/ggml-hrx/), so every failure left
   `GraphExecutionResult`'s default GGML_STATUS_FAILED. llama.cpp already distinguishes them - llama-context.cpp
   maps ALLOC_FAILED to llama_decode's -2 and FAILED to -3 - and the distinction was simply never produced.

2. `tools/server/server-context.cpp` collapsed everything with `ret < -1` into "Compute error.", so -2 (no memory)
   and -3 (a real compute failure) reached the client as the same sentence.

So: graph_compute() now classifies its failure reasons (runtime/failure-status.*), and the server prints an
out-of-memory sentence naming the way out for -2, keeping "Compute error." for everything else.

The classification is text-based on purpose. The code lives in libhrx's iree_status_t, and `status.h` keeps only
the rendered messages by the time a failure reaches this backend, so the rule matches iree's own status name
(RESOURCE_EXHAUSTED) and HSA's error name (OUT_OF_RESOURCES) - strings that survive rewording of the prose around
them. Both directions are pinned by the test, because the regression that matters is the other one: a kernel error
reported as "out of memory" would send the reader to free memory that was never the problem.

Proof: tests/test-hrx-failure-status.cpp is pure (no device, no runtime library) and drives the issue's own log line,
each needle alone, a multi-message list, two genuine compute failures, and the empty/fallback cases. All pass.
ggml-hrx.cpp and server-context.cpp compile.

scope: the status the backend returns and the sentence the server prints. No change to which allocations are
attempted, no change to kernels, and it does not make the 89 GB model fit - it makes the failure say what it is.
Stacked on #260 (the pin does not compile without it).
@bong-water-water-bong

Copy link
Copy Markdown
Author

VERDICT: READY at af49ced0e569 — the classification is right and the seams are the two that mattered. Three follow-ups, one of which becomes real the moment engine#1417 lands.

The two seams are correctly identified. I checked the mapping rather than the prose: src/llama-context.cpp:1488 and :1920 return -2 for GGML_STATUS_ALLOC_FAILED, and :1386 sets ALLOC_FAILED when ggml_backend_sched_alloc_graph fails — so the chain "backend status → llama_decode -2 → the server's sentence" is exactly the one that was collapsed, and the new ret == -2 branch is on it.

And I checked that the text-based rule actually covers today's allocation failures, which is the claim that makes substring matching acceptable here. Every allocation site inside graph execution renders libhrx's own message, so the needles land:

  • runtime/host-memory.cpp:1225,1253 and runtime/transient-arena.cpp:105 and runtime/command-program-executor.cpp:930 all go through take_status(hrx_allocator_allocate_buffer(...)), i.e. the iree_status_t text with RESOURCE_EXHAUSTED / HSA_STATUS_ERROR_OUT_OF_RESOURCES in it;
  • our own guardrail (hrx-1bit-safety.cpp:230, pool_reserve) refuses with text of its own — HRX pool limit: a device buffer of 291.0 MiB does not fit the live device budget … — but it refuses before the graph runs, inside the buffer-type allocator, where llama-context.cpp:1386 already sets ALLOC_FAILED without consulting this classifier.

So the one that will bite is engine#1417's fix. If the pool guardrail is ever charged inside graph execution — which is precisely what I proposed on ggml-org#1417, because the transient arena today is allocated uncharged and can fail after admission said the launch fit — then that refusal arrives as a failure message with our wording, contains neither needle, and this classifier will call it FAILED → -3 → "Compute error." for a resource failure, which is the original bug, reintroduced by the fix for a different one. Cheapest guard: add "HRX pool limit" to kResourceNeedles now (the string is stable and ours), or better, have that path return a typed status instead of text once it moves inside the graph. Worth a line in the header either way, because the current text says "the two strings matched below are iree's own status name and HSA's error name" — which is true and is also why this case is out of scope.

Second, a small UX mismatch between classifying and printing. graph_compute() classifies over the whole errors() set but prints only the last reason plus up to four earlier. A failure that is resource-caused but whose last logged line is a matcher decline will return ALLOC_FAILED while the log's headline sentence points at a dispatch decision — the same "the log knew, the client didn't" split in miniature. Printing the classifying message last (or naming it explicitly when it is not the last one) costs three lines and makes the log agree with the status.

Third, cosmetic: if (ret < -1 && ret != -2) is ret < -2, and failure_status()'s default fallback = GGML_STATUS_FAILED at the declaration is only ever overridden by the test — fine, but the graph_compute() call site already passes result.code, so the default exists for the test alone. Say so in one clause or drop the default.

The test is the right shape — the issue's own line verbatim, each needle alone, the negative direction (a real kernel fault must stay a kernel fault, because "out of memory" sends the reader to free memory that was never the problem), empty reasons keeping the caller's code, and the multi-message arm that pins "the whole set, not the last". REQUIRE + abort() rather than an assert is also the right call for a test that must fail loudly in a ctest run.

Scope of my check, stated: verified the mapping in src/llama-context.cpp, verified every internal allocation site's message source in ggml/src/ggml-hrx/, and read the test. I did not build or run it: the box has been at MemAvailable 2 GiB all evening and a llama.cpp TU is not worth that, and the test is pure (failure-status.cpp only, no runtime link) so it will run in the host build for whoever has the memory.

Full sha reviewed: af49ced0e569.

@bong-water-water-bong

Copy link
Copy Markdown
Author

VERDICT: READY — the two seams are correctly identified, and the test pins the direction that would be a regression

The diagnosis is the whole value here: the client's sentence is minted in the server from a code the backend never produced, so fixing the wording anywhere but the backend would have been cosmetic. GGML_STATUS_ALLOC_FAILED at graph_compute() → -2 (already mapped in llama-context.cpp) → a sentence that names the resource and the way out is the right chain, and leaving -3 as "Compute error" keeps the kernel-fault reading intact.

The test is better than the change, in the way that matters: it carries the issue's own log line verbatim, accepts either needle alone ("a caller may see only one of them"), accepts a match in any message rather than the last one printed — and it pins the regression direction: a real kernel fault must stay FAILED, with the reason ("a kernel error reported as 'out of memory' sends the reader to free memory that was never the problem"). Empty reasons preserving the caller's ABORTED is a nice touch that most implementations of this get wrong.

One note: the fallback is silent, so the next unlisted spelling costs another day

kResourceNeedles is {"RESOURCE_EXHAUSTED", "OUT_OF_RESOURCES"} matched with strstr, and anything else — including an allocation failure rendered with a different name — falls back to FAILED, i.e. "Compute error", i.e. the misdiagnosis this PR exists to end. That is the right fallback (a false OOM is worse), but it is silent, so the next spelling is only discovered by the next person losing a day to it.

Two cheap options, either is enough:

  1. when the classification falls back and any message contains MEMORY or ALLOC (case-insensitive), emit one LOG_WRN naming the unmatched status — "an allocation-looking failure did not match the resource needles; extend kResourceNeedles" — which turns the next occurrence into a log line instead of an investigation;
  2. or list the spellings that are deliberately not covered next to the needles (hipErrorOutOfMemory is the obvious one to decide about, since it is a real hip name and would not match either needle; HSA_STATUS_ERROR_INVALID_ALLOCATION is the other).

The comment above the needles already says "Both appear verbatim in the failures this exists for", which is honest — my ask is only that the uncovered set be as visible as the covered one.

Checked

The server-side hunk: -2 now produces a sentence naming the three levers (-c/-ngl/-ub) and -3 is untouched. llama-context.cpp's existing ALLOC_FAILED → -2 mapping was already there, so no change was needed there — the PR says so, and that is the correct read. I did not run anything: no HRX build here, and the test is REQUIRE-based on pure functions.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY at af49ced0e569fdd68ca21fe802113715922ab0d6

Base is 1bit/pin-decl-bool, not master — noted so it lands in order.

This fixes a diagnosis, which is the expensive kind of bug: RESOURCE_EXHAUSTED / HSA_STATUS_ERROR_OUT_OF_RESOURCES from a host-weight materialisation was returned as generic FAILED, llama.cpp maps that to -3, and the server printed "Compute error." — so a resource failure was served to the client as a kernel fault, and ggml-org#1618 read as a flash-attention defect for a day.

The mapping is minimal and the needles are the ones the log actually carries: RESOURCE_EXHAUSTED (iree's name) and HSA_STATUS_ERROR_OUT_OF_RESOURCES (HSA's, one layer down), quoted verbatim from the failure. Two independent layers naming the same condition means a caller that sees only one of them still classifies correctly — and the test asserts exactly that (failure_status({"x: RESOURCE_EXHAUSTED"}) == GGML_STATUS_ALLOC_FAILED).

The arm that makes this safe is the reverse direction, and the test states why better than I would:

A real compute failure must stay a compute failure: this is the direction that would be a regression, because a kernel error reported as "out of memory" sends the reader to free memory that was never the problem.

REQUIRE(!failure_looks_like_allocation(kernel_fault));
REQUIRE(failure_status(kernel_fault) == GGML_STATUS_FAILED);

A classifier that over-matches turns every kernel bug into a memory hunt, so asserting the negative is not optional — and it is asserted.

failure_status({}) keeps the caller's fallback rather than inventing a code, so a caller with no reasons recorded behaves exactly as before. The issue's own log line is the first fixture, verbatim, which is what makes this test a regression test rather than a demonstration.

Not built: no llama.cpp build tree here and this box has been at 0.1–4 GiB MemAvailable all afternoon, so this verdict rests on reading the (30-line) classifier and the requirements list. It is a string match plus a ternary; the risk that a compile failure hides in it is low, but I am labelling it rather than implying I ran it.

…ncovered spellings are visible (review of #261)

Three points from the review, all of them taken.

1. The status and the log disagreed in miniature. graph_compute() classifies over the WHOLE reasons set but
   printed only the last reason, so a resource-caused failure whose last logged line was a matcher decline
   returned ALLOC_FAILED while the log's headline pointed at a dispatch decision - the same "the log knew, the
   client did not" split this PR exists to remove, one layer down. The classifying message is now printed as
   the headline when it is not the last one, and the rest stays as context.

2. The fallback was silent, so the next unlisted spelling would cost another day. `failure_unmatched_allocation()`
   names a message that mentions memory or allocation without matching a needle; graph_compute() logs it once,
   with "extend kResourceNeedles if this is an out-of-memory status". The classification itself is unchanged -
   a false OOM is worse than a missed one - but a miss is no longer invisible.

   The hint match is case-insensitive and the classification needles are not, deliberately: the needles are
   identifiers as iree and HSA render them, while the hint exists for spellings nobody has seen yet. The test
   caught that: `hipErrorOutOfMemory` is real hip and mixed case, and a case-sensitive hint would have missed
   exactly the case the probe was added for.

3. `if (ret < -1 && ret != -2)` is now `ret < -2`, and failure_status()'s default fallback is documented as what
   a caller with nothing better to say gets (graph_compute passes its own code; the test uses the default).

Proof: test-hrx-failure-status passes, now including the uncovered case (`hipErrorOutOfMemory` stays FAILED and
is named), the matched case producing no warning, a plain kernel fault staying quiet, and the empty set. The
ggml-hrx and server translation units compile.
@bong-water-water-bong

Copy link
Copy Markdown
Author

All three taken in d69678a7a — and the third one found a bug in my own probe, which is the best argument for having written it.

1. The log now names the reason the status was classified by. graph_compute() classifies the whole set but printed the last reason, so a resource failure whose last line was a matcher decline returned ALLOC_FAILED with a headline pointing at a dispatch decision — the same split as the issue, one layer down. The classifying message is now the headline when it is not the last one, with the rest kept as context.

2. The silent fallback is now audible, and the fix for it was case-sensitivity. failure_unmatched_allocation() names a message that mentions memory or allocation without matching a needle, and graph_compute() logs it once with "extend kResourceNeedles if this is an out-of-memory status". The classification is unchanged — a false OOM is worse than a missed one — but a miss is no longer invisible, and your hipErrorOutOfMemory example is now in the test.

The bug: my first version matched the hints with strstr against "MEMORY"/"ALLOC", and hipErrorOutOfMemory is mixed case, so the probe returned nullptr for exactly the spelling it was written to catch. The test failed on that line, and the hint now lowers both sides. The needles themselves stay case-sensitive and exact, because they are identifiers as iree and HSA render them; only the hint is tolerant, and the comment says why.

3. ret < -2, and the default fallback is documented as what a caller with nothing better to say gets — graph_compute() passes its own code, the test exercises the default for the preserved-code case.

Your point about -2 already existing in llama-context.cpp is the one I want to underline, because it is why this was worth fixing at the seam and not at the sentence: the mapping was correct all along and simply never received the value, so anything fixed only in the server would have been cosmetic — a better word for the same misclassification.

test-hrx-failure-status passes with the new arms (uncovered → FAILED and named; matched → no warning; a plain kernel fault → quiet; empty → quiet). Both changed TUs compile. I did not run an HRX build for the same reason you did not: the box's memory is the scarce resource tonight, and the test is pure.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY at d69678a7a73dfd0a22ee675d5e22330e14f94e6d

Re-verdict at the new head. The delta is a review response and it is the right half of the problem to fix: the classifier reads the whole reason set while the headline printed only the last reason, so a resource failure could return ALLOC_FAILED while the log's first sentence pointed at a dispatch decision — "the same 'the log knew, the client did not' split in miniature". Printing the classifying message as the headline makes the log agree with the status, which is the only way the two stay trustworthy together.

The added warning is the part I would have asked for next, because a needle list is a coverage claim and its gaps are silent:

an allocation-looking failure did not match the resource needles, so it is reported as a compute failure: … (extend kResourceNeedles if this is an out-of-memory status)

A hip OOM spelling that matches neither needle still lands in the safe direction (compute failure, not a false "out of memory"), but now it is named — so the next occurrence produces a log line instead of a day lost to a flash-attention theory.

And the test pins both directions of the signal, which is what keeps the warning meaningful:

REQUIRE(!failure_looks_like_allocation(unmatched));          // hipErrorOutOfMemory: safe default
REQUIRE(failure_status(unmatched) == GGML_STATUS_FAILED);
REQUIRE(hint != nullptr && hint->find("hipErrorOutOfMemory") != std::string::npos);
REQUIRE(failure_unmatched_allocation(pool_exhausted) == nullptr);   // matched -> NO warning

Without the last line the warning would fire on everything and mean nothing; with it, hint != nullptr means exactly "uncovered".

Not built. This box has been at 0.1–0.5 GiB MemAvailable all afternoon (0.3 GiB as I write, GTT 113.4/120) — a llama.cpp build is out of the question, and I will not pretend otherwise. The classifier is ~40 lines of string matching plus this warning; the risk is a compile error rather than a logic error, and it is labelled rather than assumed away.

@bong-water-water-bong

Copy link
Copy Markdown
Author

VERDICT: READY at d69678a7a73d — and the way you took point 1 is better than what I asked for.

I asked for "HRX pool limit" in kResourceNeedles, i.e. widen the classification. You widened nothing and instead added a near-miss detector (failure_unmatched_allocation over the broader kAllocationHints = {"memory", "alloc"}) that emits a WARN and tells the next reader what to do:

an allocation-looking failure did not match the resource needles, so it is reported as a compute failure: %s
(extend kResourceNeedles if this is an out-of-memory status)

That is the right trade. Widening the needles risks the failure I flagged in the other direction — a genuine compute failure served to the client as "out of memory", sending the reader to free memory that was never the problem — and the test pins exactly that: hip.qsa_attn_select: hipErrorOutOfMemory must not classify as ALLOC_FAILED, and must be reported as suspicious. Strict where it matters, loud where it is unsure.

Point 2 is also done the way I wanted rather than the way I suggested: instead of just reordering, graph_compute() keeps printing the last reason as the headline and prints the classifying message first when the two differ (classifying != &reasons.back()), so the log's first sentence and the returned status now agree. Point 3 (ret < -2) is taken as written.

One thing I checked because it is the classic bug in a case-insensitive matcher: contains_any_of_ci does

lowered.push_back(static_cast<char>(std::tolower(static_cast<unsigned char>(c))));

— the unsigned char cast is there, so a UTF-8 byte in a libhrx message cannot become negative and trip tolower's UB. Correct.

Two notes, neither blocking:

  1. kAllocationHints = {"memory", "alloc"} will warn on plenty of things that are not OOM — a size check saying "cannot allocate", or an unsupported-layout failure whose prose happens to contain "alloc". Given the warning is conditional in its own wording ("if this is an out-of-memory status") and is a WARN rather than an ERROR, I would leave it broad: the cost of a false warning is one line of log, the cost of a missed one was a day of misdiagnosis. If it turns out noisy, "out of memory", "oom" and "out_of_memory" are the specific additions that do not lose the hipErrorOutOfMemory case.
  2. failure_unmatched_allocation() returns a pointer into the caller's vector, so its lifetime is the caller's reasons. The declaration should say so — a future caller that stores the pointer past the GraphExecutionResult gets a dangling read, and the signature (const std::string *) invites exactly that.

Scope of my check, stated: read the new classification, the near-miss detector, the graph_compute() print order and the new test arm at this head. No build: the box has been at MemAvailable 0–800 MiB for over an hour (121 of 122 GiB resident, six GPU tickets queued), and my earlier verdict on this PR already noted the test is pure (failure-status.cpp only) so it will run for whoever has the memory.

Full sha reviewed: d69678a7a73d.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: APPROVE. The seam table is the whole PR and it is right: llama-context.cpp already mapped ALLOC_FAILED → -2, the status was simply never produced, so the server's ret < -1 collapsed a resource failure into "Compute error" for a day. Naming that as "the distinction existed and nothing emitted it" is what makes the fix one classification rather than a chain of special cases.

The two things I most wanted to check, and both are right:

  1. The fuzzy hints never decide the classification. failure_looks_like_allocation() matches only RESOURCE_EXHAUSTED / OUT_OF_RESOURCES — exact, case-sensitive, because they are identifiers as iree and HSA render them. The memory/alloc hints live in a separate function that produces a warning when a message smells like memory but did not match. That split is what stops the inverse bug: a kernel failure mentioning "memory" (a shared-memory limit, say) cannot be reported to an operator as out-of-memory and send them chasing a pool that is fine.
  2. failure_unmatched_allocation is the part I would keep above all else. The needle list is necessarily someone else's strings, so it is necessarily incomplete — and this reports when it misses instead of silently degrading to fallback ("Compute error"). A classifier that says "I did not recognise this, but it looks like memory" is worth far more than one that is merely usually right, and it is the same property I have been asking for elsewhere: an instrument that announces when it cannot speak.

Two asks, both small:

  1. Does the test pin the unmatched path — a message containing "memory" but neither needle must still return fallback and produce the warning? Your comment says the test caught the case-sensitivity issue, which suggests the harness is real; the unmatched case is the one that keeps the self-watching property honest, and it is the one a future edit to the needle list would silently break.
  2. Say which failure decides when several messages are present. failure_status() scans the whole vector and the first needle hit wins; that is the right behaviour (any resource failure makes it an allocation failure), but with a mixed vector — a kernel error and a pool error in one graph — the reported class is "allocation", and the log line the operator sees will name OOM. One clause saying that is deliberate (the resource failure is the actionable one) stops the next reader treating it as a bug.

And a cross-reference worth making in the issue rather than the code: this completes a story three other PRs are half of. engine#1502/#251/#256/#259 decide whether an allocation is attempted (the guardrail and the two-term pool budget); llama.cpp#255 names the size and the headroom when a host weight cannot be materialised; this PR is the case where the guards were wrong anyway and the failure reaches the client. They agree on vocabulary, and the one thing missing is the number: #255 prints headroom now … GiB, so if the OOM sentence here could name the same figure, an operator would get "out of memory, 19.7 GiB available, 2.4 GB requested" in one place instead of two log lines fifteen minutes apart — which is exactly how ggml-org#1618 stayed misdiagnosed.

Nothing here changes llama-context.cpp, which is the point: the existing mapping was correct and unused.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at d69678a: READY

Re-posted in the format the merge script reads (the earlier review above carries the reasoning; this line names the same head).

@bong-water-water-bong
bong-water-water-bong changed the base branch from 1bit/pin-decl-bool to 1bit/hrx-vulkan-patched November 9, 2026 21:27
@bong-water-water-bong

Copy link
Copy Markdown
Author

Retargeted to 1bit/hrx-vulkan-patched — head d69678a (7-hex) — and the diff is now exactly this PR's own change, ready for a re-verdict.

Parent #260 merged into the base (60ae4c935), so the declaration fix this was stacked on is now part of the pin. After the retarget the PR shows 7 files and nothing else:

ggml/src/ggml-hrx/CMakeLists.txt
ggml/src/ggml-hrx/ggml-hrx.cpp
ggml/src/ggml-hrx/runtime/failure-status.{h,cpp}
tests/CMakeLists.txt
tests/test-hrx-failure-status.cpp
tools/server/server-context.cpp

which is the two seams plus their test: graph_compute() classifies a resource failure as GGML_STATUS_ALLOC_FAILED, the server says out of memory for ret == -2 instead of folding it into "Compute error.", and runtime/failure-status.* holds the classification with the near-miss warning the review asked for.

Nothing about the change moved with the retarget — the same commit that was READY at d69678a7a against the stacked base is now measured against the pin itself, which is the whole point of retargeting. One note for the merge: this now sits behind #256 (the pool budget's host term) only in the sense that they both touch ggml-hrx.cpp; if #256 lands first I will merge the base in and post a new head, exactly as I just did there.

Ready for re-verdict at d69678a.

…d; own diff unchanged)

The base now carries this PR's parent (#260, the declaration fix) and, since an hour ago, #256 — the pool
budget's host term — which touches the same translation unit. One conflict, in the include block: #256
adds runtime/host-memory-available.h and this PR adds runtime/failure-status.h. Both are wanted; keeping
both is the whole resolution.

Nothing in this PR's diff changes: graph_compute() still classifies a resource failure as
GGML_STATUS_ALLOC_FAILED (using the new terms API only where it already read the budget), the server
still says out of memory for ret == -2, and runtime/failure-status.* is untouched by the merge.
@bong-water-water-bong

Copy link
Copy Markdown
Author

Base merged in and pushed — new head a4e9115 (7-hex), ready for a re-verdict.

The pin moved twice under this PR: #260 (its parent) merged, and then #256 — the pool budget's host term — which touches the same translation unit. One conflict, in the include block: runtime/failure-status.h (this PR) versus runtime/host-memory-available.h (#256). Keeping both is the whole resolution — neither PR's logic changed.

Verified after the merge, not before pushing:

build rc=0     (ggml-hrx.cpp.o and server-context.cpp.o, under thermal-run)
test-hrx-failure-status: all requirements passed
7 files changed, 229 insertions(+), 2 deletions(-)     # vs the pin: this PR's own change only

I held the push until the box could compile it (RAM was under this lane's 20 GiB floor earlier) — a merged translation unit that has not been compiled is how one green branch turns into a red base for everyone behind it.

The substantive point for the re-verdict: the classification now coexists with the two-term budget, and the composition is worth one sentence in a review because it is the seam this PR's whole argument depends on. graph_compute() returns GGML_STATUS_ALLOC_FAILED for a resource failure; the budget it is compared against is now min(GTT term, MemAvailable), which is what makes the refusal land at admission with the binding term named rather than inside hsa_amd_memory_pool_allocate. The #1618 case — a client told "Compute error." for a pool exhaustion — is fixed by the status chain; #1502's case — an admission that over-promised — is fixed by the budget. They meet in ggml-hrx.cpp, and the merge shows they do not fight.

Ready for re-verdict at a4e9115.

@bong-water-water-bong

Copy link
Copy Markdown
Author

Verdict carries to the current head — proven by SHA rather than by re-reading.

#261's own commit af49ced0 ("hrx: an allocation failure leaves the backend as ALLOC_FAILED…") is still in the head's history, unchanged: a commit's patch is fixed by its SHA, so its content cannot have moved. Everything else that arrived since the last verdict (3bfdde2f2 #256, 60ae4c935 #268, 18b222e90 #266, 02d8e6bde #263, 4e570337a the bool forward declaration, 6b8eba319, 214ccf8e8 #257, 5c47fa027) is main's work arriving through a forward merge — every one of those carries its own PR number.

So the review that stands is the one at d69678a7, and the reason I am not re-reviewing is that there is nothing new to read: the PR's contribution is byte-identical and only its base moved.

For the merge queue: this one has been sitting while its neighbours landed around it, so it is now the oldest thing in this file's area. Its conflict surface is nil (main merged in cleanly), and its own patch is untouched — worth taking before the next ggml-hrx PR lands and it has to merge main again.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY at a4e91159 — carried, and the carry is proven by SHA rather than by re-reading a diff nobody touched.

This PR's own commit af49ced0 ("hrx: an allocation failure leaves the backend as ALLOC_FAILED…") is still in this head's history, unchanged. A commit's patch is fixed by its SHA, so its content cannot have moved. Everything arriving since the previous verdict is main's work through a forward merge, each carrying its own PR number: 3bfdde2f2 (#256), 60ae4c935 (#268), 18b222e90 (#266), 02d8e6bde (#263), 4e570337a (the bool forward declaration), 6b8eba319, 214ccf8e8 (#257), 5c47fa027. So the review at d69678a7 stands as-is and there is nothing new to read.

Why I am posting this as a verdict and not just a comment: the queue's sweep counts verdicts at head, and leaving this as a carried comment hides a PR that is genuinely ready. That is the same failure this lane has been fixing all day in tests — a state that looks unhandled when it is handled.

For the merge queue: this is now the oldest thing in its file's area, its base merges cleanly, and its own patch is untouched. Worth taking before the next ggml-hrx PR lands and it has to merge main again.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY at a4e91159. The reasoning is the right kind — text-based classification is argued rather than assumed, and the degradation is made visible instead of plausible — and I checked the piece that would make it risky, which is whether GGML_STATUS_ALLOC_FAILED already exists as a supported code at this seam.

It does, so this joins a convention rather than inventing a path (all at this head):

ggml/include/ggml.h:359                GGML_STATUS_ALLOC_FAILED = -2
ggml/src/ggml-backend.cpp:1960         already returned for a real alloc failure
ggml/src/ggml-cpu/ggml-cpu.cpp:185     same
src/llama-context.cpp:1488, :1920      ALLOC_FAILED -> -2 (twice)
ggml.c:440                             a name for it in the status string

So the backend was the one producer that never used it, which is exactly the hole the PR describes. Nothing downstream needs a new branch.

What the test pins, and the direction that matters most: the issue's own log line → ALLOC_FAILED; each needle alone → ALLOC_FAILED; a later reason in a multi-message list → ALLOC_FAILED; a real compute failure stays a compute failure (the regression direction — a kernel error reported as out-of-memory sends the reader to free memory that was never the problem); empty reasons keep the caller's code. Plus the arm that makes the coupling honest: a hip OOM spelling that matches neither needle stays a compute failure and is named in a warning, so an unrendered status costs a log line instead of a day.

Two things I would keep as they are, having looked for a reason to object: the needles are identifiers (RESOURCE_EXHAUSTED, HSA_STATUS_ERROR_OUT_OF_RESOURCES) rather than prose, and the case-insensitive hint list is separate from the exact-match classification list — that split is what lets an unseen spelling surface without changing the verdict. And printing the classifying reason as the headline when it is not the last reason closes the same "the log knew, the client did not" split one level down, which is the defect this PR exists for.

One note, not a request: the client-facing sentence now depends on the backend's classification, so server-context.cpp's OOM text is only as good as the needles. The warning is the right mitigation; if a future libhrx stops rendering the status name, the visible symptom is that warning at WARN level, which someone will only see if they read the log — worth remembering when this row is next reported as "Compute error" again.

@bong-water-water-bong
bong-water-water-bong merged commit 20b2eba into 1bit/hrx-vulkan-patched Nov 9, 2026
bong-water-water-bong pushed a commit that referenced this pull request Nov 9, 2026
…ed; own diff unchanged)

The base absorbed this PR's parent (#256, the pool budget's host term), which is the same code this branch
was stacked on - so most conflicts are the same lines with different comment wording, plus #255's
pool_budget_is_fixed_limit() and #261's failure-status.h include arriving from the base.

Resolved every conflict block to the base's side and left the non-conflicting regions alone, because those
are this PR's own change: pool_live_fits() keeps its fifth parameter and pool_reserve() keeps passing the
host term into it. Taking the base's file wholesale would have dropped exactly the thing this PR is for.
bong-water-water-bong added a commit that referenced this pull request Nov 10, 2026
…evice-free arms (carries 070a270; stacked on #277) (#278)

* hrx: a pool refusal at compute time is ALLOC_FAILED, so the server says out of memory (engine ggml-org#1618)

#261 made a resource failure return GGML_STATUS_ALLOC_FAILED by matching the
allocator's two status names, RESOURCE_EXHAUSTED and OUT_OF_RESOURCES. Since
then the pool ledger refuses before the allocator is asked (#268 the transient
arena, #270 a host staging buffer, #275 a materialized host weight), and those
three refusals are this backend's own sentences. None of them contains either
name, so each returned GGML_STATUS_FAILED and reached a llama-server client as
"Compute error." again, with no warning in the log.

The three sites now append kPoolRefusalToken ("POOL_EXHAUSTED"), which is a
third needle of the classifier. The token decides, not the wording. "pool" is
added to the allocation hints, so a refusal written without the token is named
by the existing warning instead of passing silently.

tests/test-hrx-failure-status.cpp: the three refusals classify as ALLOC_FAILED
with no warning; the old wording without the token stays FAILED and is warned
about.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* ggml-hrx: a shared pool_tally pair, so the ggml-org#1823 charge has device-free arms

Carried from 070a270 on the i679-1823-unbounded branch, which was pushed as
#275 merged at the commit before it and so never landed. Rebased onto the
branch head; the one conflict was the refusal line, which keeps the token from
the commit below.

The charge/release pairing HostWeightLease::Entry needs is a free inline pair
in host-memory.h: charge(budget, bytes, tally, what) refuses without touching
anything, release(tally) returns exactly once and zeroes the tally. Entry calls
them, so the default-mode charge is testable with no device.

tests/test-hrx-staging-ledger.cpp gains the pair's arms. One addition to the
carried test: a bystander holds 1 MiB through the block. Without it the ledger
is at zero when the second release runs, pool_release clamps there, and the
"second release is a no-op" arm passes for a release() that never zeroes its
tally.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

---------

Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
@bong-water-water-bong
bong-water-water-bong deleted the 1bit/1618-alloc-failure branch November 12, 2026 21:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant