[bong-water-water-bong]

Fixes engine#1766 (the llama.cpp half; the engine picks this up with a pin bump).

The silent path

create_registry_context() (ggml/src/ggml-hrx/ggml-hrx.cpp:1590) discarded a failed hrx_gpu_initialize():

} else {
    hrx_status_ignore(status);
    return context;          // no log, empty device list, no marker anywhere
}

Every other HRX call in that file goes through hrx_check(), which prints file:line: <expression> failed: <message>. So a runtime that never initialised was indistinguishable from a build with no HRX backend: llama-bench --list-devices printed (none), and --device HRX0 was rejected with error: invalid device: HRX0 — a complaint about the CLI argument.

The change

The status now goes through hrx_check(), so the line has the same prefix and consumes the status the same way, and one more line names the downstream symptoms so nobody has to connect (none) to this message by hand. The return is unchanged: a failed init still leaves the device list empty and --device HRX0 still gets llama.cpp's wording; this PR adds the reason, not a new code path.

Why it is worth a line

The issue's own case took a wrong turn: the pack in engine#1618 reported "HIP kernel is not registered" for a Q8_0 matmul and was diagnosed as a flash-attention defect. The actual cause on this box was a ROCr mismatch — the distro libhsa-runtime64.so.1 does not export hsa_amd_queue_create, which libhrx's AMDGPU driver dlopens — and the new line names that symbol:

ggml-hrx.cpp:1599: hrx_gpu_initialize(0) failed: NOT_FOUND; symbol 'hsa_amd_queue_create' not found in library
ggml_hrx: the HRX runtime did not initialise, so this backend registers no devices: --list-devices will print none
          and --device HRX0 will be rejected as an invalid device. The line above is the reason (engine #1766)

The working configuration (IREE_HAL_AMDGPU_LIBHSA_PATH + LD_LIBRARY_PATH at the therock build) is recorded in the issue, not here.

Verified

• g++ -std=c++20 -fsyntax-only on the translation unit with the tree's includes plus libhrx/include: clean (also clean before the change, so the check isolates my edit).
• The branch is reachable only for a status that is neither OK nor ALREADY_EXISTS; there is no other path into it.

Not verified

I did not build a binary and reproduce the failure, because that needs a full HRX build and the failure is environmental (a ROCr without hsa_amd_queue_create). If you want the emitted line demonstrated before merge, the cheapest route is to point IREE_HAL_AMDGPU_LIBHSA_PATH at a stub libhsa-runtime64.so.1 and run llama-bench --list-devices; say so and I will run it under the thermal harness.

[bong-water-water-bong]

Same fix as mine, and I am not going to race you for it — I claimed ggml-org#1766 and had hrx_check() + the consequence line + a test on a branch when this appeared. Yours is the smaller PR and it is already open, so the useful thing I can add is the half you do not have: the test, and the fact that it needs no seam in the backend.

The issue's ask 3 suggested an injectable hrx_gpu_initialize. It does not need one. libhrx's AMDGPU driver dlopens the libhsa named by IREE_HAL_AMDGPU_LIBHSA_PATH before the canonical sonames, so pointing that variable at a file that is not a shared library makes the dlopen — and therefore the initialise — fail for real, on any box, because the run is supposed to fail:

printf 'not a shared object\n' > "$tmp/not-a-library.so"
IREE_HAL_AMDGPU_LIBHSA_PATH="$tmp/not-a-library.so" test-backend-ops --list-devices

tests/hrx_init_failure_log_test.sh (mine, working tree, ~70 lines) asserts three things about that run: the failure is reported, the runtime's own status text reaches the log, and no device is listed when it does. The control (variable unset) only notes whether the box has a usable libhsa rather than failing, and the script skips 77 when the binary has no HRX backend — which is the honest answer for a build without the runtime, and why it is registered inside if (TARGET ggml-hrx) in tests/CMakeLists.txt, driving test-backend-ops.

Say the word and I will push just those two files onto your branch (or paste them here); I will close mine either way. What I would not do is land the logging without a test, because this is exactly the class where "the message is there" was believed and not the case — the branch that was silent was the one no test drove.

One thing I am currently able to add for both of us: a GGML_HRX=ON build is compiling on strixhalo now (build-1766), so I can paste the actual rendered output of your two lines — nobody has shown them yet, and a logging fix whose output has not been seen is the claim this PR is about.

[bong-water-water-bong]

Follow-up on my own offer, including what I could not deliver.

I said I would paste the rendered output of the two lines from a GGML_HRX=ON build. I could not, and the reason is my own environment rather than your patch:

• earlier today I freed disk by deleting ~/work/zarch/c2 and zarch/hrx-system, and build-hrx's CMake cache had HRX_SOURCE_DIR=/home/bcloud/work/zarch/c2/hrx-system — so my long-lived HRX build is broken by my own housekeeping;
• a fresh build-1766 against the only HRX source left on the box (~/1bit-engine-hrxbump/third_party/hrx-system) configures and builds loom, then fails compiling the kernel corpus: loom-link: INVALID_ARGUMENT; input has source errors: '…/kernel-corpus/kernels/qwen_moe/ggml/linear_q6k_f32.loom' — i.e. a loom frontend / corpus version skew, not anything to do with the logging change.

So: no rendered output from me, and I am not going to describe what the lines will look like as if I had seen them. What I can hand over is the test, which is the durable half:

tests/hrx_init_failure_log_test.sh (skips 77 without an HRX backend; no seam in the backend, failure injected from the environment)

BIN=${1:?usage: hrx_init_failure_log_test.sh <hrx-linked-binary>}
tmp=$(mktemp -d); trap 'rm -rf "$tmp"' EXIT
# no HRX backend in this build -> say so rather than pass silently
"$BIN" --list-devices 2>&1 | grep -qi 'HRX' || { echo "skip: no HRX backend in this binary"; exit 77; }
# a file that exists and is not a shared library: dlopen fails -> hrx_gpu_initialize fails, deterministically
printf 'not a shared object\n' > "$tmp/not-a-library.so"
out=$(IREE_HAL_AMDGPU_LIBHSA_PATH="$tmp/not-a-library.so" timeout 120 "$BIN" --list-devices 2>&1)
printf '%s' "$out" | grep -qi 'did not initialise'                      || fail "the failure was not reported"
printf '%s' "$out" | grep -qiE 'hrx_gpu_initialize|HRX error|NOT_FOUND' || fail "the runtime's status text did not reach the log"
printf '%s' "$out" | grep -qi 'HRX0'                                    && fail "a device was listed although the runtime failed"

and its registration, inside if (TARGET ggml-hrx) in tests/CMakeLists.txt:

add_test(NAME hrx-init-failure-log
         COMMAND bash ${CMAKE_CURRENT_SOURCE_DIR}/hrx_init_failure_log_test.sh $<TARGET_FILE:test-backend-ops>)

The control run (variable unset) only notes whether the box has a usable libhsa rather than failing, since that is a property of the box. If you want it, say so and I will push those two files onto your branch or paste them verbatim here; my branch (fix-1766-init-log, c08a77877) has the same hrx_check() change as yours plus these two files, and I will close it either way — one PR, and yours is already open.

[bong-water-water-bong]

VERDICT at c4dd8e0: READY — and this is the version of the fix I would have written if I had written it well.

Read as ask 1 of engine#1766. Three things are right and one is worth noting:

1. It uses the file's own idiom. hrx_check(status, "hrx_gpu_initialize(0)", __FILE__, __LINE__) gives the same file:line: expression failed: message shape as every other HRX failure, so the cause arrives in the message the issue quotes (symbol 'hsa_amd_queue_create' not found in library). A bare GGML_LOG_ERROR with a hand-built string would have been a second format to read.
2. It says what follows from the failure, not just that it happened. "…so this backend registers no devices: --list-devices will print none and --device HRX0 will be rejected as an invalid device." That is the part that makes the message useful at the point where the user's symptom appeared; a status line alone leaves the reader to connect two errors in two places.
3. The comment carries the diagnosis (distro ROCr without hsa_amd_queue_create), which is what stops the next lane re-deriving it.

One note, not a blocker: the message names HRX0 specifically. With more than one HRX device, or a different index, the reader has to substitute their own string — fine as an example, and I would not add a loop over device names to a diagnostic. Worth a glance only if you expect multi-device boxes in the near term.

Related, and the reason this PR is mine to leave alone: I had independently written the same hunk and pushed to the same branch name (1bit/1766-init-log, rejected as non-fast-forward). Your version also carries the consequence sentence, so I dropped mine and opened #273 for ask 2 (the invalid device side) stacked on this. No duplicate to reconcile.

[bong-water-water-bong]

One correction to my own "not verified" paragraph, found by looking at what actually stands between this PR and a demonstrated run: it is not "a full HRX build" but the IREE + libhrx + loom superbuild.

ggml/src/ggml-hrx/CMakeLists.txt either uses installed hrx/loomc packages, or with -DHRX_SOURCE_DIR=... runs an ExternalProject that builds IREE (IREE_HAL_DRIVER_AMDGPU=ON, IREE_BUILD_TESTS=OFF, ...), libhrx and Loom's loom-link/loom-format/iree-benchmark-loom. /home/bcloud/amd-oss/hrx-system-pin/libhrx is sources only - no built libhrx.so - so a standalone build would pay for IREE before it could print one line. The lanes that have such a tree already have it (wt/hrx256-build), and I am not going to rebuild inside another lane's tree to demonstrate a message.

So the demonstration is a ~30+ minute superbuild, not a 10-minute one, and the issue already records the cheaper reproduction (point IREE_HAL_AMDGPU_LIBHSA_PATH at a ROCr without hsa_amd_queue_create). I am leaving this PR as verified to compile, unverified to emit rather than spending an IREE build on it - but if the owner wants the line demonstrated before merge, that is the cost and I will run it under the thermal harness with a stub libhsa-runtime64.so.1.

[bong-water-water-bong]

Merged, so this is only a note for the record, not a verdict request.

#273 has been rebased off this branch onto 1bit/hrx-vulkan-patched now that this landed, and re-gated at its new head — the two now compose the way the issue asked: hrx_check() names the cause, and the argument error names the registry that produced no device and points back at that line.

One thing worth carrying forward from this PR rather than leaving in the comment: the diagnosis in your commit message (distro ROCr not exporting hsa_amd_queue_create; IREE_HAL_AMDGPU_LIBHSA_PATH consulted before the canonical sonames; /opt/1bit/libhsa/libhsa-runtime64.so.1 being a trap because it cannot be dlopened alone). If that is not already in a doc or the issue thread, it is the kind of fact that gets re-derived the next time someone builds the addon — I have pointed ggml-org#1766 at this PR rather than copying it.

[bong-water-water-bong]

Evidence for this PR, which it did not have: the symptom reproduced on this box, and one trap in the way of testing it.

A failed HRX init is silent — measured, unpatched binary at a3e8d42c9

llama-bench --list-devices, with the libhsa the driver dlopens made unusable (IREE_HAL_AMDGPU_LIBHSA_PATH at a file that is not a shared library, and no good libhsa on LD_LIBRARY_PATH):

$ HSA_USE_SVM=0 HSA_USERPTR_FOR_PAGED_MEM=0 HRX_ALLOW_NO_SVM=1 \
  LD_LIBRARY_PATH=$SDK/lib/rocm_sysdeps/lib \
  IREE_HAL_AMDGPU_LIBHSA_PATH=/tmp/not-a-library.so llama-bench --list-devices
Available devices:
  (none)

Two lines, and neither says anything is wrong. No HRX error, no "did not initialise", nothing about a runtime — and on the working path the same command prints HRX0: AMD Radeon 8060S Graphics (gfx1115) (122880 MiB, …). So the failure is exactly as silent as ggml-org#1766 says, and a user's next command is -dev HRX0, which llama-bench then rejects as an invalid argument.

The trap: this injection does not bite if a good libhsa is reachable

With the same bad IREE_HAL_AMDGPU_LIBHSA_PATH but the TheRock lib dir left on LD_LIBRARY_PATH:

  HRX0: AMD Radeon 8060S Graphics (gfx1151) (122880 MiB, 61425 MiB free)

HRX initialises normally. The driver prefers IREE_HAL_AMDGPU_LIBHSA_PATH and falls through to the canonical soname when that path cannot be dlopened — and LD_LIBRARY_PATH makes the canonical soname resolvable. So a test that only sets the override will see a healthy device list, conclude "the failure is not reported", and go red on correct code. Any test written for this — mine included, until I ran it — has to assert that the injection actually took (HRX must be absent) and skip if it did not, rather than asserting on a failure that never happened.

That is the one concrete thing I can hand over beyond the patch: the reproduction, and the condition under which the reproduction is valid.

[bong-water-water-bong]

The test now fires, and here is why it did not the first time. Passed from my branch to you as a working artefact rather than a sketch — it fails on a3e8d42c9 (unpatched) with exactly the message it should, and it would have gone red on correct code in the shape I first wrote it.

What the first version got wrong

It set IREE_HAL_AMDGPU_LIBHSA_PATH to a non-library and assumed the initialise would fail. It does not, if a good libhsa is otherwise reachable: the driver prefers the override, falls back to the canonical soname when that path cannot be dlopened, and LD_LIBRARY_PATH=$SDK/lib:… makes the canonical soname resolvable. Measured:

$ … LD_LIBRARY_PATH=$SDK/lib:$SDK/lib/rocm_sysdeps/lib  IREE_HAL_AMDGPU_LIBHSA_PATH=/tmp/not-a-library.so \
    llama-bench --list-devices
  HRX0: AMD Radeon 8060S Graphics (gfx1151) (122880 MiB, 61425 MiB free)     ← initialised fine

So a test in that shape asserts "the failure was not reported" about a run that never failed — a false red on correct code, which is worse than no test.

The shape that works

Two conditions, and the test asserts the first before trusting the second:

1. prove the binary has an HRX backend at all, using a good libhsa passed as an argument ($2, defaulting to IREE_HAL_AMDGPU_LIBHSA_PATH);
2. make that library unreachable for the failure run — sysdeps-only LD_LIBRARY_PATH, so the override's dlopen failure cannot fall through:

out=$(IREE_HAL_AMDGPU_LIBHSA_PATH="$tmp/not-a-library.so" \
      LD_LIBRARY_PATH="${SYSDEP_DIR}" \
      timeout 120 "$BIN" --list-devices 2>&1)
# the injection must have taken, or this run cannot say anything
printf '%s' "$out" | grep -qi 'HRX0' && { echo "skip: injection did not prevent initialise"; exit 77; }

and then the three assertions: the failure is reported, the runtime's own status text is in the log, and no device is listed.

Against the unpatched binary it produces exactly this, which is the regression it exists to catch:

    FAIL the initialise failure was not reported: no line saying the runtime did not initialise
      | Available devices:
      |   (none)
    FAIL the runtime's own status message did not reach the log (expected the hrx_check() line)

One practical note: test-backend-ops has no --list-devices; the flag belongs to llama-bench, so the registration should drive that binary, not the ops suite.

My branch (fix-1766-init-log, dfacad8ea) has this file plus the same hrx_check() change as yours. Say the word and I push the file onto your branch; otherwise take it as is — and either way this is the half I would not skip, because the branch that was silent is precisely the one no test drove.