Skip to content

hrx: a refused graph says when the scheduler's dependency walk stalled (Refs engine #1925) - #305

Merged
bong-water-water-bong merged 1 commit into
1bit/hrx-vulkan-patchedfrom
1bit/walk-stall-report
Nov 11, 2026
Merged

bong-water-water-bong merged 1 commit into
1bit/hrx-vulkan-patchedfrom
1bit/walk-stall-report

Conversation

@bong-water-water-bong

Copy link
Copy Markdown

Refs 1bit-MONSTER/engine#1925 (closed by #302 and engine#2134). Diagnostic only: no plan changes.

Why

When the scheduler's dependency walk cannot reach a node, it appends the remaining nodes in index order and matches them in that order. A fused pattern's root can then come before a node the pattern reads, and the refusal names the matcher:

unsupported HRX node 160: L2_NORM ... consumers=[174:GATED_DELTA_NET]

engine#1925 was that. Four lanes instrumented the matcher over a day, and every guard was satisfied; the cause was the walk (an in-place CPY that waited for itself, fixed in #302). Nothing in the output said the graph had been walked out of order.

What

  • GraphTraversalOrder records how many nodes the walk did not reach (stalled_count()) and the first of them (first_stalled_node()).
  • DispatchScheduler::schedule_graph adds one line to the diagnostics it already prints with a refusal:
dependency walk stalled: 2 of 2 nodes were not reached and are matched in index order, from node 0 (SCALE)
fusion order: common.scale_f32 at node 0 (SCALE) refused: dispatch 0 loom_libs:ggml_scale_f32 binding 0 input reads value 1 produced by node 1 (SCALE) that is not scheduled yet
unsupported HRX node 0: SCALE output=0:f32[64,1,1,1]<-SCALE inputs=[1:f32[64,1,1,1]<-SCALE] consumers=[1:SCALE]

A graph that plans prints nothing, stalled or not. 23 lines of source.

Proof (host, no device)

tests/test-hrx-gdn-passed-alias.cpp gains check_stall_report (output above is from it):

At 0e502ebdf on fork tip b6f65a273, amdclang Release build: test-hrx-gdn-passed-alias rc 0 (the seven earlier cases unchanged), hrx-backend-test host: 69 passed, 0 failed.

Not run: anything on the device. The change adds no dispatch and alters no order.

🤖 Generated with Claude Code

…d (Refs engine ggml-org#1925)

When the walk cannot reach a node it appends the rest in index order and matches them in that order. A pattern
root can then come before a node its pattern reads, and the refusal names the matcher, not the walk. engine ggml-org#1925
was that for days: "unsupported HRX node 160: L2_NORM" with every matcher guard satisfied.

GraphTraversalOrder now records how many nodes it did not reach and the first of them, and the scheduler adds one
line to the diagnostics it prints with a refusal:

  dependency walk stalled: 2 of 2 nodes were not reached and are matched in index order, from node 0 (SCALE)

Nothing is printed for a graph that plans. The count is zero for every graph llama.cpp builds since #302; the test
asserts that for a ggml_cast and builds a two-node cycle by hand for the other side.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@bong-water-water-bong

Copy link
Copy Markdown
Author

ready for review at 0e502eb (diagnostic only, host proof in the body)

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 0e502eb: READY

A refusal that says the dependency walk stalled, with the counts, is the diagnostic ggml-org#1925 needed and did not have:

if (traversal.stalled_count() != 0) {
    const size_t first = traversal.first_stalled_node();
    pending_diagnostics.log("dependency walk stalled: %zu of %zu nodes were not reached and are matched in …

That is exactly the shape I praised in #302's write-up — 183 accepted vs 1 declined told a reader where a matcher was wrong in one line. Here the same idea is applied to the traversal: not "the graph was refused" but "N of M nodes were not reached, and they are being matched anyway", with the first stalled node named.

Diagnostic only, no plan changes, and it lands after #302 — which is the right order: the fix first, then the instrument that would have shortened the search.

@bong-water-water-bong

Copy link
Copy Markdown
Author

VERDICT: READY 0e502eb

This is the right fix for the reporting half of ggml-org#1925, and I checked the three ways a diagnostic can be worse than none:

1. It cannot print garbage. Both counters are initialized in-class (stalled_count_ = 0, first_stalled_node_ = 0), and first_stalled_node() is only read under
stalled_count() != 0, so a graph that never stalls cannot produce "from node ". That is the failure mode I look for first in a diagnostic-only PR, because
the numbers are the part nobody re-checks once the message looks plausible.

2. It is only said when it matters. pending_diagnostics.log(...) reaches the user through the refusal path, so a healthy graph prints nothing new. The comment says
that explicitly and the code matches it — the alternative (logging on every stall) would have been noise on a path that, per the header's own note, is "Zero for every
graph llama.cpp builds today"
.

3. The test pins the numbers, not just the sentence. It builds the stall by hand (two SCALE nodes reading each other — real graphs do not produce one, so a
hand-made graph is the only honest way to cover it) and asserts nodes().size() == 2, stalled_count() == 2, first_stalled_node() == 0, then the message fragments
"dependency walk stalled: 2 of 2 nodes were not reached" and "from node 0 (SCALE)". Grepping for a substring alone would have passed with a wrong count; this does not.
The second arm — a ggml_cast leaves nothing unreached — is the regression guard for #302 itself, which is the more valuable half.

One thing worth considering later, not a blocker: stalled_count() counts all unreached nodes, so a graph that ever contains legitimately dead nodes (built but not
consumed by the output) would report a non-zero count on an unrelated refusal, and a reader could chase the order instead of the real cause. The current wording is
factual ("were not reached and are matched in index order"), so it does not mislead — but if that situation ever arises, the sentence may want to admit "which can be a
cause or a consequence" the way the source comment already does.

Context: my measurements on engine#1346 say the batched-NaN signature reports the slot's own logits, so the remaining long-context faults are in producing kernels and
graph scheduling — exactly the space this diagnostic makes legible, instead of the day four lanes spent instrumenting a matcher that was already correct.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant