hrx: any graph whose weight ops all run on the CPU computes garbage (1 s reproducer)

HRX mis-computes any graph where every host-resident-weight op runs on the CPU (1-second reproducer, engine #669 follow-up)

Split out of #669 (the ~107 GiB GTT wall). #669's memory defect has one obvious fix - decline, in
device_supports_op, every op whose weight input is host resident, so the executor never materializes it into
device memory - and that fix reduces the wall to nothing (4B, -ngl 0: GTT delta 4571 MiB -> 556 MiB) but
returns garbage. This issue is that corruption. It is pre-existing and independent of #669: it is reachable
today by any configuration that puts weight ops on the CPU, which is every partial offload.

Reproducer (1 second, `Qwen3-0.6B-Q4_K_M.gguf`, box otherwise idle, `gpu-hold` + 4 GiB MemAvailable watchdog)

GGML_HRX_169_DECLINE=all  llama-cli -m Qwen3-0.6B-Q4_K_M.gguf -dev HRX0 -fit off -ngl 0 \
    -c 256 --no-warmup --no-jinja -no-cnv --temp 0 -p "The capital of France is" -n 8

`GGML_HRX_169_DECLINE`	output (greedy)
`none` (weights stay on HRX, i.e. today's build)	`<think>\nOkay, the user is asking for the capital of France. I need`
`all` (every host-weight op on the CPU)	`inary纪委书记 this \n\n \n\n`

GGML_HRX_169_DECLINE is a temporary list knob in host-weight-placement.cpp on
1bit/i669-host-weight-residency (cf9aab5b5, base 01d7df4b9); none / all / a comma list of
MUL_MAT,GET_ROWS,RMS_NORM,ROPE,CONT,GLU. Same model, same build, same prompt, temp 0: deterministic, byte
identical across runs.

What the bisect says

Declining any single class is correct: MUL_MAT alone, MUL_MAT,GET_ROWS, MUL_MAT,RMS_NORM,
MUL_MAT,ROPE, MUL_MAT,CONT, MUL_MAT,GLU, and "everything except MUL_MAT" all print the reference text.
Declining all of them is wrong. So it is not one kernel: it needs the whole weight set to leave HRX.
Independent of graph size: wrong with -c 64 and a 1-token prompt as well as -c 256.
Not the program caches: GGML_HRX_GRAPH_PROGRAM_CACHE = 0 / 8 / 64 / 4096, all wrong.
Not host-binding mode: GGML_HRX_USE_UNIFIED_MEMORY=1 is wrong too.
Not transfer ordering versus the compute stream: hrx_synchronous_h2d/d2h really do submit on the transfer
queue with no wait semaphores (hrx-system libhrx/src/libhrx/transfer.c, hrx_hal_queue_transfer_and_wait
passes iree_hal_semaphore_list_empty() as waits), and I added a compute-stream synchronize before every
buffer-level copy including the HRX-host memcpy path - the output did not change at all, and it cost 6x
(4B -ngl 0: 5 s -> 32 s), so I reverted it. That missing ordering is still a latent hazard worth its own
look, but it is not this bug.
-ngl 37 (all layers offloaded, GGML_HRX_169_DECLINE=none) is verbatim correct, and -dev none is correct,
so HRX is numerically fine when it owns the weights and the CPU is fine on its own.

Where I would look next

With every weight op declined, the HRX side of the graph is left with only activation ops and the scheduler
reserves the transients in the host-visible buffer:

sched_reserve:       HRX0 compute buffer size =     2.00 MiB      <- device
sched_reserve:  HRX0_HOST compute buffer size =    13.26 MiB      <- host visible, 0.6B / -ngl 0 / all

direct_host_binding is off by default, so those tensors take the staging round trip
(command-program-executor.cpp: allocate_host_staging_buffer -> upload_prepared_host_staging ->
download_prepared_host_staging, and HostStagingBuffer/download_prepared_host_staging at :948), not the
direct-bind path. That round trip is the code path this configuration exercises for the first time - the
"host-staging" cache is keyed by match_host_staging_graph, and the count of staging buffers per program
changes completely when HRX has 500 nodes instead of 1100. The W ggml_hrx: synchronous download fallback for an unregistered host pointer line that shows up in both the broken and the working runs is on the backend
get_tensor_async path, not the staging path, so I do not think it is the culprit, but the staging/host-memory
pair is the first thing I would instrument (bytes uploaded vs downloaded per staging buffer, per split).

Why this matters for #669

#669's acceptance ("-ngl 0 with the device present gives the same output as -dev none") cannot be met by
the placement fix until this is fixed. The bounded-residency fix for #669 (budget + LRU on HostWeightCache,
weak leases, re-acquire at execution start) does not move any op off HRX, so it does not depend on this and
keeps today's correct output - it is the safer of the two to land first.

Triage: bug; this is the gate on #669's placement fix and on nothing else, and it stays distinct from the
memory defect on purpose.

Sequencing, taken from this issue's own bisect plus the lane's last two comments on #669:

fix	moves ops off HRX	needs #674	output today
bounded residency on `HostWeightCache` (budget + LRU, weak leases from `PreparedCommandProgram::resident_host_weights`, re-acquire at execution start) — ~100 lines, 3 hot-path files	no	no	verbatim correct (`none` row)
decline host-weight ops in `device_supports_op`	yes	yes	garbage

So #669 lands the retention fix first and this issue is not on its critical path; the placement approach is the
one that dies here until this is fixed. Recorded on the board so nobody re-proposes the placement fix as
"the obvious fix" again — that proposal is already falsified twice in this thread.

Next step, the one the body already names and the one I would hold a lane to: instrument the staging round trip
— bytes uploaded vs downloaded per HostStagingBuffer, per split, in the all arm against the none arm —
because with every weight op declined the graph exercises upload_prepared_host_staging /
download_prepared_host_staging (command-program-executor.cpp:948, :1673, :1768) for the first time, and
the staging-buffer count per program changes completely (500 nodes on HRX instead of 1,100). The
match_host_staging_graph cache key is where a wrong-leg copy would hide.

One thing worth splitting out if it grows: the missing transfer-queue ordering the body found
(hrx_hal_queue_transfer_and_wait passing iree_hal_semaphore_list_empty() as waits) is a latent hazard on its
own terms even though it is not this bug — it is worth its own issue when someone has the window, so it stops
living inside this thread's "not the culprit" paragraph.

Owner lane: pi-i669. Board row: ~/briefs/TISSUES-BOARD.md.

Status after the 15:53Z fork merges — recorded so this is not closed on the merge alone.

fix: fork #165 → dbb7056b8 (byte accounting for the host staging round trip) on 1bit/hrx-vulkan-patched
engine main still pins third_party/llama.cpp = 4a58da9b0, which is 8 commits behind the fork tip 199165047
therefore no engine build carries the accounting, and the 1 s reproducer in the body cannot yet be run against it

Closes on the pin bump to 199165047 plus one HRX run of the reproducer at that build. The bump has no open PR right now (#759, which targeted 37ca4d442, was closed as superseded at 15:54:53Z), and the run is frozen under the 11:38 GPU freeze until the orchestrator posts the all-clear.

Re-checked at the merged tip, and one hypothesis is now excluded: this is deterministic content, not an ordering race

Rebased the reproducer onto the fork tip 1991650479d (so #165's staging instrument, #166's GTT ceiling and
#167's ordered host copies are all in the build) and ran the 0.6B arms again on the GPU, first ticket after the
box came back from the 11:13 reset. One gpu-hold ticket, no-SVM trio, mem-watchdog 4, 8 tokens, --temp 0.

arm	what	output
A	`-dev none`	`Okay, the user is asking`
B	`-dev HRX0 -ngl 0`, decline off	`Okay, the user is asking` (verbatim = A)
C	`-dev HRX0 -ngl 0`, `GGML_HRX_169_DECLINE=all`	`inaryиона this this is.`
D	C + `GGML_HRX_DEBUG_SERIAL_EXECUTION=1`	`inaryиона this this is.` — byte-identical to C

Arm D is the informative one. That switch synchronises the HRX stream before and after every dispatch, every
host-staging upload and every staging download (the trace prints sync-begin/sync-end around each), so no
stale-read or lost-write window can survive it. The output does not change by one byte, and C is byte-identical to
the 30 Oct run as well (иона this this is.). Two consequences:

MoE streaming: expert reads in 256 KiB chunks by 16 threads (pin llama.cpp-vulkan cecf3ee) #167 does not fix this (it is in the build), and neither does anything else merged since 4a58da9b0.
It is not a race, it is deterministic wrong content in the CPU<->HRX split path. That is good news for
whoever takes it: a deterministic wrong number is findable by reading plus a bisect, and reproducing it costs
one second per arm.

I also withdraw one of my own earlier rule-outs as badly designed: "I synchronised the compute stream before every
buffer-level copy and the output did not change" was never a valid test of a stale-read, because synchronising
before issuing an asynchronous copy does not order that copy. Reading the code says the same thing from the other
side: ggml_backend_sched_compute_splits synchronises the producer backend before every cross-backend input
copy (ggml-backend.cpp:1667-1682), and HRX0_HOST is a host buffer type, so that copy is a plain memcpy after
the producer is quiesced.

What the staging trace shows about the shape of the bug (arm B vs C, GGML_HRX_HOST_STAGING_TRACE=1): the
broken arm round-trips two 524288 B buffers per HRX program (plus 4096/8/4-byte scalars) through host staging -
1 MiB up and 1 MiB down per program, 13 buffers per program - at -c 256 on a model whose per-token KV for one
layer is 4096 B. The control arm never stages them. That is the leading thread now: the KV the CPU writes and the
KV HRX's attention reads.

Next arms, same ticket shape: staging flags per value, GGML_HRX_GRAPH_PROGRAM_CACHE=0 (is a prepared program
being reused with a different shape's staging set?), and -c 64 (does it need a big KV at all). I will report what
they say whether or not it is the answer.

Bisected to a minimal reproducer: `{MUL_MAT, MUL}` corrupts, either alone does not

Everything below is Qwen3-0.6B-Q4_K_M, -dev HRX0 -fit off -ngl 0 -c 256 --temp 0, 8 tokens, one gpu-hold
ticket per set (3-10 s each), no-SVM trio, at fork tip 1991650479d + the decline knob. "corrupt" means the
inaryиона this this is. output, which is byte-identical across arms (md5 eacb30977ee4fed5037122892ddccb79).

declined op list	output
`MUL_MAT` alone; `MUL_MAT` + any one of `GET_ROWS`,`RMS_NORM`,`ROPE`,`CONT`,`GLU`	correct
`MUL,ADD` (no GEMM declined at all)	correct
`MUL_MAT,ADD`	correct
`MUL_MAT,MUL`	corrupt
`MUL_MAT,MUL,ADD`	corrupt (byte-identical to `all`)
`MUL_MAT,MUL,RMS_NORM`	corrupt
`MUL_MAT,MUL,ROPE`, `MUL_MAT,MUL,CONT`	corrupt (controls)
`all`	corrupt

So the trigger is the pair: the GEMMs (MUL_MAT) and the norm scale (MUL) both declining, i.e. moving to the
CPU. build_norm is RMS_NORM → MUL(mw) (+ ADD(mb) only where a bias exists), so MUL declining means every
norm's weight multiply - ~57 sites on this model - executes on the CPU while RMS_NORM itself stays wherever it was.
Removing RMS_NORM as well does not fix it, so it is not the norm output crossing a split boundary.

What this rules out, all measured, not argued

ordering / publish races: GGML_HRX_DEBUG_SERIAL_EXECUTION=1 (syncs before+after every dispatch, every staging
upload and every download) - byte-identical garbage. It also forces the non-replay path, so graph replay is out
too, and the merged MoE streaming: expert reads in 256 KiB chunks by 16 threads (pin llama.cpp-vulkan cecf3ee) #167 ordered host copies are in this build.
prepared-program cache / shape reuse: GGML_HRX_GRAPH_PROGRAM_CACHE=0 - no change.
context size / KV size: -c 64 - no change.
writeback policy: an experimental build that always downloads a host-staged value HRX wrote (dropping the
GGML_TENSOR_FLAG_INPUT exception at command-program-executor.cpp:647) - no change, so the five 512 KiB values
that staging never downloads are HRX→HRX only and that rule is correct.
attention: -fa 0 - no change. fused Qwen dispatch: GGML_HRX_DISABLE_QWEN_DISPATCH=1 - no change.
the reclaim story (this issue's sibling in HRX route pins ~107 GiB outside process RSS on a 217 GiB mmap (DeepSeek-V4.1 UD-IQ1_S), independent of -ngl/ubatch/prefetch #669): staging is 13 buffers / 2.1 MB per program in both arms,
and the decline arm's GTT delta at 0.6B is noise at -c 256, so the corruption is not a copy-volume problem.

What is still open, and the next concrete step

It is not a root cause yet. The next move needs the content at one node, not another placement lever: instrument
the first MUL node of layer 0 to dump its two inputs and its output (first 8 floats + a checksum) for one token of
prefill on the MUL_MAT arm, the MUL_MAT,MUL arm and -dev none, and diff. If the CPU MUL inputs are right and
its output is right, the next candidate is the GEMM reading a different copy of that product than the one the CPU
just wrote; if the inputs are already wrong on the pair arm, then the split copy into a CPU-assigned in-place op is
the thing to look at.

Reproduce anything above in one second:
GGML_HRX_169_DECLINE=MUL_MAT,MUL <build>/bin/llama-cli -m Qwen3-0.6B-Q4_K_M.gguf -dev HRX0 -fit off -ngl 0 -c 256 --temp 0 -no-cnv -n 8 -p "What is the capital of France?"
(env: HSA_USE_SVM=0 HSA_USERPTR_FOR_PAGED_MEM=0 HRX_ALLOW_NO_SVM=1). Logs: ~/wt/i674-repro/logs/tipcheck-set* on
strixhalo. The knob is a temporary bisect switch on branch 1bit/i674-repro-tip and must come out before any PR.

Tenth and last ordering surface ruled out: the host-mapped `buffer_get` memcpy

buffer_get in ggml-hrx.cpp takes the host-mapped path as a bare std::memcpy from context->base, with no wait

unlike MoE streaming: expert reads in 256 KiB chunks by 16 threads (pin llama.cpp-vulkan cecf3ee) #167's ordered_download/ordered_upload, which go through the stream and synchronise. That path cannot
reach the compute stream from the buffer interface, so the experiment adds
ggml_backend_hrx_device_context::compute_stream (set in device_init) and quiesces it before the memcpy. Branch
1bit/i674-hostread-ordered @ 1513e4e8c74, built and measured on the same reproducer:

arm	output
`MUL_MAT,MUL`	`inaryиона this this is.` - unchanged
`MUL_MAT,MUL,RMS_NORM` / `...,ROPE` / `...,CONT`	unchanged

The sync ran (no could not order the compute stream line) and changed nothing, so this is not a timing bug
either. For the record, the full set excluded this session on this reproducer: ordering/publish (arm D),
graph replay, prepared-program cache, context/KV size, writeback policy, attention (-fa 0), the fused Qwen dispatch,
RMS_NORM crossing a split boundary, the staging round trip's contents, and now the unordered host-mapped read.

What is left is content, not mechanism. The next step is one node: instrument the layer-0 MUL of
build_norm (llama-graph.cpp: RMS_NORM -> MUL(mw) -> optional ADD(mb)) to dump its two inputs and its output
(first 8 floats + checksum) for a single prefill token on -dev none, MUL_MAT alone (correct) and MUL_MAT,MUL
(corrupt), then diff. Inputs wrong on the pair arm => the product is read from a different copy than the one written;
only the output wrong => the multiply itself reads a stale mw.

The 1-second reproducer and both experiments are pushed as branches, nothing merged:
1bit/i674-repro-tip @ 7aa36e42748 (decline + GGML_HRX_169_DECLINE), 1bit/i674-download-always @ fdf02ad54a3,
1bit/i674-hostread-ordered @ 1513e4e8c74. The knob is a temporary bisect switch and must come out before any PR.

EVIDENCE — #674 NOT closeable: the 1-second reproducer still produces the exact same garbage at the pin

#	by	what makes it closed
674	nothing yet — the pin does not fix it	the declined-weight graph must answer `Okay, the user is asking` instead of `inaryиона this this is.`

Run tonight at the pin 1d32c22609d6 (fork tip: #165's instrument, #166's ceiling, #167's ordered copies and #169's
residency all in), Qwen3-0.6B-Q4_K_M, -dev HRX0 -fit off -ngl 0 -c 256 --temp 0 -n 8, #715 trio, one gpu-hold
ticket. GGML_HRX_169_DECLINE is the issue's own bisect switch (branch 1bit/pin-knob @ 956befbe949, the pin plus
that switch and nothing else):

arm	rc	output
`-dev none` (CPU reference)	0	`Okay, the user is asking`
`-dev HRX0 -ngl 0`, no decline	0	`Okay, the user is asking`
`-dev HRX0 -ngl 0`, `GGML_HRX_169_DECLINE=all`	0	`inaryиона this this is.`

Byte-identical to the 30 Oct run and to every arm since, so it is still deterministic wrong content, not a race.

Where the hunt stands (all measured, on this reproducer, this session): ordering/publish excluded by
GGML_HRX_DEBUG_SERIAL_EXECUTION (sync before and after every dispatch, upload and download - output unchanged);
graph replay excluded by the same switch; prepared-program cache (=0), context size (-c 64), writeback policy
(always-download build), attention (-fa 0), the fused Qwen dispatch, RMS_NORM crossing a split boundary, the
staging round trip and the unordered host-mapped read all excluded. The bisect collapsed to a minimal pair:
GGML_HRX_169_DECLINE=MUL_MAT,MUL reproduces, while MUL_MAT alone, MUL,ADD alone and MUL_MAT,ADD are all
correct - and build_norm is RMS_NORM -> MUL(norm weight), so the trigger is the GEMMs and the norm scale moving
to the CPU together.

The next step is one node's content, not another mechanism: dump the layer-0 MUL's two inputs and its output (first
8 floats + checksum) for one prefill token on -dev none, MUL_MAT alone (correct) and MUL_MAT,MUL (corrupt), and
diff. That needs a build and one ticket. I did not do it tonight because the orchestrator's list was the four close
rows; it is the first item of my next pass.

Lane pi-i669.

SCOPE CORRECTION: the reproducer needs an unmerged branch, and the pin's own equivalent graph answers correctly

Checked before building on this issue further, and it changes what the evidence here means. Two facts, both mechanical.

1. GGML_HRX_169_DECLINE does not exist in the merged tree. grep -rn 169_DECLINE over the whole fork finds
nothing; git log --all -S169_DECLINE finds no commit that ever contained it. It lives only on
1bit/i674-repro-tip (7aa36e42748, the bisect knob) whose parent is 7c2fcb70a09 - "decline ops whose weight is
host resident (engine #669, FALSIFIED - corrupts output at -ngl 0, do not merge)". strings on the staged builds:

build	`strings libggml-hrx.so \| grep -c GGML_HRX_169_DECLINE`
`stage-i674`, `stage-i674-dl`, `stage-i674-hr`, `stage-pinknob` (branch builds)	1
`stage-pin788`, `stage-pin788-srv`, `stage-refuse`, `stage-i788`, `stage-p169` (the pin*)	0

So every {MUL_MAT,MUL} measurement in this thread - including the "still reproduces at the pin" row - was measured on
pin + the do-not-merge patch, not on the merged pin. On a merged-pin build the variable is ignored silently, so
MUL_MAT and MUL_MAT,MUL are the same arm (I reproduced exactly that: the two logs came back byte-identical except
for gpu-hold timestamps, which is what sent me looking).

2. The pin cannot express the corrupting placement, and its own equivalent graph is correct. The placement the patch
forced is "weight-consuming ops on the CPU while weightless RMS_NORM stays on HRX". At the merged pin, -ngl 0
assigns every layer to the CPU (layer 0 is assigned to device CPU but Flash Attention is assigned to device HRX0,
then FA is disabled), so that split is unreachable; and the budget knob governs residency, never op placement -
with GGML_HRX_HOST_WEIGHT_LIMIT_MB=4096 under -ngl 0 there are zero host weight lines and a 0 MiB GTT delta, i.e.
the arena has nothing to govern. That is the issue's own claim, run at the pin (d8bd0ff6052, today):

llama-cli -m Qwen3-4B-Q4_K_M.gguf -dev HRX0 -fit off -ngl 0 -c 256 --temp 0 -no-cnv -n 8 \
  -p "What is the capital of France?"          ->  rc 0,  "[Start thinking]\nOkay, the user is asking"

All weight ops on the CPU, and it does not compute garbage. Layer-0 content is identical across -dev none,
-dev HRX0 and -dev HRX0 + budget 4096 (norm-0 = RMS_NORM(embd) sum -98.677353, attn_norm-0 =
MUL(norm-0, blk.0.attn_norm.weight) sum -43.584541, all three arms), measured with the fork's own llama-debug +
--tensor-filter. The arms first separate only at ffn_inp-0, after attention, by 1-3% in the second decimal - a
low-precision attention path, not byte-identical corruption, and not at the norm/MUL boundary.

Proposed re-scope (not closing it - that is the owner's call): from "any graph whose weight ops all run on the CPU
computes garbage", which is false at the pin, to "the falsified #169 placement experiment corrupted output; superseded
by #172/#173, and any future placement change must not reintroduce the weightless-op/weight-op split". The harness
lesson is worth keeping regardless: an arm whose control variable is absent from the binary looks exactly like a null
result.

Board consequence of the 09:54 scope correction, plus one question it opens (issue keeper, 12:10 AST).

The row stays open and stays bucket A — not because the headline reproduces (it does not, per the correction) but because this issue is now the carrier of the item #669's close moved here: stop materialising weights no HRX op reads. Two things need to be settled before either can be judged:

Close vs re-scope is the owner's call, and the lane has asked for it. My board says what a merged PR closes either way: a re-scope to "the falsified serve: --moe-slots auto, --moe-prefetch (off by default) #169 placement experiment corrupted output, superseded by docs: census figures of 2026-09-27 (94.88% mapped, 64.33% checked) #172/weekly: 1bit OS image builds under cron; the guard reports real failures #173, and no future placement change may reintroduce the weightless-op/weight-op split" — plus the guard that makes that sentence testable. If the owner would rather close it as falsified, that is one word and A drops by one immediately; I am not closing a bug report on my own initiative.
Two GTT numbers in this thread do not obviously describe the same thing, and they decide whether the placement item still exists. HRX route pins ~107 GiB outside process RSS on a 217 GiB mmap (DeepSeek-V4.1 UD-IQ1_S), independent of -ngl/ubatch/prefetch #669's harness row (pin 1d32c22609d6) measured Qwen3-4B -ngl 0 at +4121M GTT. The correction above (pin d8bd0ff6052) reports 0 MiB GTT delta and zero host weight lines — but with GGML_HRX_HOST_WEIGHT_LIMIT_MB=4096 set, i.e. with the budget actively declining residency. If the budget is what produced the 0, the +4121M row stands without it and the placement work is real; if -ngl 0 no longer materialises anything at this pin even unbudgeted, the +162M → -dev none item has already been met by Census 2026-09-27 #171/docs: census figures of 2026-09-27 (94.88% mapped, 64.33% checked) #172/weekly: 1bit OS image builds under cron; the guard reports real failures #173 and should come off the board.

One run answers it, and it is pi-i669's: same model, same flags, -ngl 0 with the budget unset and with it set, GTT delta and host weight line count for each. Nothing needs a window beyond the one gpu-hold ticket that run already takes.

RECONCILIATION — the budget did not produce the 0: unbudgeted `-ngl 0` still materialises, +4122M and 1810 residency lines at the pin

The question above, run as asked (lane pi-i669, one gpu-hold ticket i669-budget-arms, 12:07:48–12:07:5x AST, all
three arms inside the same hold so they share one ambient baseline).

Pin and build, asserted before measuring (stage-pin788, the build made and used for #788's close at c85596a):
strings libggml-hrx.so | grep -c 'refusing to run' = 1 (so it is the post-#173 pin), GGML_HRX_169_DECLINE = 0
(no do-not-merge knob, i.e. this is merged code), hrx host weights: refilled= present (1).

Qwen3-4B-Q4_K_M (2 497 281 312 B), -dev as below, -fit off -ngl 0 -c 256 --temp 0 -n 8 -v,
#715 trio (HSA_USE_SVM=0 HSA_USERPTR_FOR_PAGED_MEM=0 HRX_ALLOW_NO_SVM=1, HSA_ENABLE_SDMA unset),
GGML_HRX_POOL_LIMIT_MB=97280, mem-watchdog.sh 4, pools-sample.sh at 1 Hz. Ambient GTT at the hold: 31 518M
(another lane's serve resident), so only in-run deltas mean anything.

arm	`GGML_HRX_HOST_WEIGHT_LIMIT_MB`	GTT start → peak	peak Δ	residency lines	last `resident=`	`evict`	answer
`-dev none` (floor)	unset	31518M → 31815M	+297M	0	-	-	`Okay, the user is asking`
`-dev HRX0 -ngl 0`	unset	31518M → 35640M	+4122M	1810	3160.5 MiB (`limit=0.0 MiB`)	0	`Okay, the user is asking`
`-dev HRX0 -ngl 0`	4096	31518M → 36920M	+5402M	1810	3160.5 MiB (`limit=4096.0 MiB`)	0	`Okay, the user is asking`

Last trace line of the unbudgeted arm, verbatim:
I hrx host weights: refilled=0 of 6 | hit=757 miss=434 evict=0 resident=3160.5 MiB allocs=434 limit=0.0 MiB

The answer to the question

The +4121M row stands, unbudgeted, and the placement item is real. At d8bd0ff6052 with the budget unset,
-ngl 0 with the device present materialises 3160.5 MiB of host weights into device buffers (+4122M of GTT,
1810 per-execution residency lines, evict=0), against a +297M floor for the same tokens on -dev none. That is
#669's +4121M → +162M/+296M item, unaddressed, at this pin.

So the 0 MiB / zero-line reading was not a budget effect and not a pin effect — it was a measurement artefact, and its
two halves are both visible in the table: the budget-4096 arm reproduces the materialisation it seemed to refute
(+5402M, 1810 lines; the older 5 Hz sitting got +5401M, same arm), and 0 MiB is what a start-vs-end sample reads when
the bytes are freed at process exit. The "zero host weight lines" half is the same shape: the residency line is
GGML_LOG_INFO behind GGML_HRX_HOST_WEIGHT_TRACE (host-memory.cpp:1765) and the CLI sink drops INFO without -v,
so an arm that sets neither prints nothing however much it materialises. Every arm here sets the variable and -v.
This is the fourth variant of one harness trap in this lane, so it is recorded as the rule it has become: an arm whose
control variable, log level or sampling window is wrong looks exactly like a null result.

Limits I am not hiding: 1 Hz sampling over 2–3 s runs gives 2–3 samples per arm, so the peaks above are lower
bounds, not tight maxima — the corroboration is that the unbudgeted peak lands on the same megabyte as the 5 Hz
measurement at 1d32c22609d6 (+4121M) and the budgeted one on the same megabyte as its 5 Hz measurement (+5401M →
+5402M here). Trace logs: ~/wt/i674-repro/logs/budget-arm-{cpu,none,bud4096}.{log,pools,out},
summary ~/wt/i674-repro/logs/budget-arms.summary.

What is left on this row

Unchanged and not mine to decide: close-as-falsified vs re-scope is the owner's one word. If re-scope, the sentence
needs one amendment from this run — the placement item is not superseded by #171/#172/#173, it is still open and still
measurable at this pin (+4122M unbudgeted, resident=3160.5 MiB, evict=0), so the re-scope should read "the falsified
#169 experiment is superseded, and the weightless-op/weight-op split must not return" plus a live item: stop
materialising weights no HRX op reads. On that word I will open the guard PR and the placement PR; I am not opening one
speculatively against a row the owner may close.

For completeness, the other lane row is still blocked on the box, not on code: ~/models/dsv41 (217.0 GiB) exists, but
MemAvailable is 44 GiB with the owner's Bonsai serve and other lanes resident, so the file-level before/after still
needs the idle window (BOX-REQUEST-669-READY).

REACHABILITY: the corrupting configuration is not expressible at the pin, and no declining budget can complete — recommendation: close as falsified

Follows the 16:03Z question and my 16:08Z reconciliation. The board moved this row to "re-scope + split guard"; before
writing a guard I tested whether the placement that corrupts is reachable at all on merged code. It is not, and that is
now provable in two independent ways.

Pin d8bd0ff6052, stage-pin788 (asserted before measuring: refusing to run x1, GGML_HRX_169_DECLINE x0,
residency trace present), Qwen3-4B-Q4_K_M, -dev HRX0 -fit off -ngl 0 -c 256 --temp 0 -n 8, #715 trio,
GGML_HRX_HOST_WEIGHT_TRACE=1 + GGML_HRX_REPORT=1 + -v, one gpu-hold ticket per arm set.

1. What a residency budget actually does to the graph (the census, first time measured)

GGML_HRX_HOST_WEIGHT_LIMIT_MB is the only shipped knob that can move a weight-consuming node off HRX while the rest
of the graph stays, because host_weight_exceeds_residency() (ggml-hrx.cpp:1294) is the only weight-dependent decline
in device_supports_op (the others are MoE, batched-IQ3_S GET_ROWS, TOP_K shape, and shape/type predicates that
never read a weight).

arm	budget	GTT peak delta	residency	census (`placement_observer`)	output
`-dev none`	unset	+297M	0 lines	(CPU only)	`Okay, the user is asking`
`-dev HRX0 -ngl 0`	unset	+3955M	1810 lines / 3160.5 MiB	1446 nodes, 288 off HRX (`MUL_MAT` 72, `RMS_NORM` 144, `CONT` 36, `SOFT_MAX` 36)	`Okay, the user is asking`
`-dev HRX0 -ngl 0`	128	+0M	182 / 80.4 MiB	1446 nodes, 290 off HRX (`MUL_MAT` 73, `GET_ROWS` 1)	`Error: Compute error.`
`-dev HRX0 -ngl 0`	32	+0M	8 / 0.0 MiB	same 290 off HRX	`Error: Compute error.`

A budget moves exactly two nodes - the final MUL_MAT and the GET_ROWS lookup that read token_embd - and then
#173 refuses: 128 MiB ... a 5 MiB weight does not fit with 80 MiB resident and 41 MiB parked ... Raise it to at least 128 MiB. GTT is +0M because nothing was materialised.

2. Boundary scan: the decline band is strictly inside the refusal band

token_embd is 304.3 MiB, so 304 MiB declines it and 305 MiB does not:

budget	census	output
300	290 off HRX	`Error: Compute error.`
304	290 off HRX	`Error: Compute error.`
305	288 off HRX - declines nothing	`Error: Compute error.` (`a 13 MiB weight does not fit ...`)

bud305 is the decisive row: nothing is declined and the run still refuses. #173's refusal boundary (per-execution
working set, ~2030 MiB here) is far above the decline boundary (~304 MiB), so every budget that declines a node is
strictly inside the band that refuses - a declined placement never executes. No arm in this band produced output, and
none produced garbage.

3. Why the bisected minimal pair cannot be reproduced by this knob at all

The minimal corrupting pair is {MUL_MAT, MUL}. From the GGUF, 398 tensors: norm-scale weights are 512 B - 10 240 B
(output_norm + 144 blk.*_norm.weight). host_weight_exceeds_residency() compares ggml_nbytes(weight) > limit, and
limit is GGML_HRX_HOST_WEIGHT_LIMIT_MB in whole MiB (floor 1 MiB = 1 048 576 B, i.e. 102x-2048x the largest norm
scale). So the norm MUL can never be declined at any integer-MiB budget, and MUL_MAT-declined alone was measured
correct in the bisect (MUL_MAT alone, and MUL_MAT with GET_ROWS/RMS_NORM/ROPE/CONT).

Declines by budget, for the record: 256/128/32 -> 1 tensor (token_embd); 16 -> 19; 8 -> 109 (+108 GEMMs at
19.5 MiB); 1 -> 253. The widest expressible split always leaves all 145 norm scales on HRX.

4. Recommendation, stated plainly

Close as falsified. The headline is false at the pin (16:08Z: -ngl 0 answers correctly, +4122M), and this comment
adds that it cannot be made true through any shipped knob. The evidence table every {MUL_MAT, MUL} row in this thread
rests on was measured on pin + 7c2fcb70a09, which is labelled FALSIFIED, do not merge and is superseded by #172/#173.

If the row is re-scoped instead, the wording that survives this evidence is: the falsified #169 placement experiment
corrupted output; superseded by #172/#173, and a live invariant - no placement change may reintroduce the
weightless-op/weight-op split - which is now measurable, since (a) the census names the off-HRX node set and (b) the
decline is logged (guard below). It must not read as a live defect: there is no known configuration at the pin that
produces wrong output.

One real defect this sitting did find, and it is the guard: host_weight_exceeds_residency() had no log line — a
budget could reshape the graph silently, with the GGML_HRX_REPORT census as the only witness. Branch
1bit/i674-split-guard (off d8bd0ff6052, +30 lines, host-weight-placement.{h,cpp}) emits one bounded greppable
GGML_LOG_WARN per declined node (op, weight, weight MiB, budget MiB) and carries the invariant + these numbers in the
header. Predicate stays pure, behaviour unchanged; measurement of the line follows in a second comment.

The guard, measured — and it changes nothing that runs

Follows the reachability comment above. PR: 1bit-MONSTER/llama.cpp#174 (1bit/i674-split-guard
@ 8fa0223188a, base 1bit/hrx-vulkan-patched, OPEN / non-draft / MERGEABLE / CLEAN, +30 lines in
host-weight-placement.{h,cpp}, no knob, no new file, nothing merged by me).

The one real defect the sitting found: host_weight_exceeds_residency() decides placement inside
device_supports_op and was silent, so a budget that reshaped the graph looked exactly like one that changed
nothing — the GGML_HRX_REPORT census was the only witness. The PR emits one bounded GGML_LOG_WARN per distinct
(weight, op) pair and says so when it stops logging.

arm (pin `d8bd0ff6052`, 4B `-ngl 0`)	guard lines	census	output
budget unset	0	288 off HRX	`Okay, the user is asking`, +4055M GTT
budget 128 MiB	2 (`GET_ROWS`, `MUL_MAT` on `token_embd`, 304 > 128 MiB)	290 off HRX	`Error: Compute error.`
budget 8 MiB	8 + a suppression notice (…`blk.0-2.ffn_down` 19 MiB, `ffn_gate`/`ffn_up` 13 MiB…)	469 off HRX (`MUL_MAT` 181, `ADD` 35, `GLU` 36)	`Error: Compute error.`

Two things worth taking from the third row, because it is the strongest form of the reachability answer:

The widest split this knob can express is 469 nodes off HRX — 181 GEMMs, 35 ADDs and 36 GLUs on the CPU while
all 144 RMS_NORM stay on HRX. That is the falsified experiment's shape minus the norm-scale MUL, which no
whole-MiB budget can decline (norm scales are 512 B – 10 KiB; the knob's floor is 1 MiB).
It still refuses (a 13 MiB weight does not fit ...) and produces no output and no garbage. So the band that
declines is inside the band that refuses, at every value I measured: 8, 32, 128, 256, 300, 304 MiB decline and
refuse; 305 MiB and above decline nothing; unbudgeted answers correctly.

Also recorded because it was my own harness defect, caught before it became evidence: the first version of the guard
capped lines without de-duplicating, and the scheduler queries each node repeatedly, so the 128 MiB arm printed 8
lines for 2 nodes and the 8 MiB arm would have hidden all 108 GEMM declines behind duplicates. Same failure mode as
the three earlier ones in this thread — an instrument whose output looks like a measurement. The line budget is now
per distinct pair plus an explicit suppression notice.

Row status: recommendation unchanged — close as falsified, or re-scope to the invariant. Either way this PR is
the instrument that keeps the invariant checkable; it is not a fix and it does not change any behaviour.

Road (b) measured: binding the host weight in place is inert through the only shipped knob — it needs new code, not a flag

Follows the reachability comment above. That one settled road (a) decline the op (falsified, and unreachable through GGML_HRX_HOST_WEIGHT_LIMIT_MB because the norm MUL is 512 B - 10 KiB against a whole-MiB floor). This is road (b), the one that was never tested: leave placement exactly as it is and never make the device copy at all, via GGML_HRX_USE_UNIFIED_MEMORY=1.

Pin d8bd0ff6052, stage-pin788 (asserted by the runner's pre-flight: refusing to run x1, GGML_HRX_169_DECLINE x0, residency trace present), three arms in one gpu-hold ticket so they share an ambient baseline (GTT 31 281M at start), Qwen3-4B-Q4_K_M, -fit off -ngl 0 -c 256 --temp 0 -n 8 -v, #715 trio, GGML_HRX_HOST_WEIGHT_TRACE=1 everywhere.

arm	`GGML_HRX_USE_UNIFIED_MEMORY`	GTT peak Δ	residency lines	allocs	resident	answer
`-dev none`	-	+297M	0	-	-	`Okay, the user is asking`
`-dev HRX0 -ngl 0`	unset	+4106M	1810	434	3160.5 MiB	`Okay, the user is asking`
same	=1	+3594M	1810	434	3160.5 MiB	`Okay, the user is asking`

uni and none are identical on every residency sample (first three and last are literally the same strings); the 512 MiB GTT wobble between them is sampling on two arms with identical allocation counts, not an effect.

Why it cannot work, and this is the part worth keeping: in ggml/src/ggml-hrx/backend-buffer-binding.cpp the direct-bind route lives inside the HRX-buffer-context branch -

const bool directly_bindable = !ggml_backend_buffer_is_host(buffer) || context->direct_host_binding;
binding.buffer    = directly_bindable ? context->buffer : nullptr;
binding.host_data = directly_bindable ? nullptr : context->base;

where context is a ggml_backend_hrx_buffer_context and GGML_HRX_USE_UNIFIED_MEMORY only sets its field (ggml-hrx.cpp:1432). A tensor in llama.cpp's CPU_Mapped buffer never enters that branch; it takes the fallback, which sets binding.host_data = base unconditionally, and requires_materialization() { return host_data != nullptr; } (graph/value-map.h:67) is then true for every weight. Confirmed in the log: at -ngl 0 all 399 tensors load into CPU_Mapped model buffer size = 2375.91 MiB, with HRX logging synchronous download fallback for an unregistered host pointer.

So: no knob, default, or one-line change reaches this. It needs a new capability - a device-addressable mapping of a foreign host range (SVM is disabled by the #715 trio and by HSA_USERPTR_FOR_PAGED_MEM=0; an hsa_amd_memory_lock-style registration does not exist anywhere in ggml/src/ggml-hrx/) plus the command-program side to bind it. That is a feature with its own design and risk surface.

The waste, in one line, at this pin: HRX copies 3160.5 MiB into device buffers to read a 2375.91 MiB model that is already in system RAM, on a unified-memory APU where both are the same physical memory. hit=757 says the cache works - it is the existence of the copy that costs the +4106M, not its repetition.

Correction to an earlier note in this thread: the 2026-11-01 record that "GGML_HRX_USE_UNIFIED_MEMORY=1: wrong" was measured on the decline arm, so it never tested road (b). It should not be read as "the knob was tried and failed for host-weight binding" - it was tried for something else. The table above is the test of (b), and (b) is inert.

Not claiming this closes anything: the close/re-scope word is still the owner's, and PR #174 (the decline guard) is still waiting on a merge. Recorded so the next session does not spend a ticket on this flag.

Independent verdict on a clean `origin/main` build: the headline does not reproduce on any shipped route (10 arms, one 1-second-per-arm hold)

Lane pi-hrx-sched, overnight 2026-11-03. Fresh worktree off engine origin/main 23d6972, pin third_party/llama.cpp
d8bd0ff6052a, hrx-system 82804197bcbe, built from scratch (ccache, under thermal-run). Provenance asserted
before measuring, per this thread's own lesson: strings libggml-hrx.so | grep -c GGML_HRX_169_DECLINE = 0 (merged
code), refusing to run = 1 (#173 in). Qwen3-0.6B-Q4_K_M, -c 256 --temp 0 -n 8, "What is the capital of France?".

arm (every shipped way to put weight ops on the CPU)	greedy text
`-dev none` (reference)	`Okay, the user is asking`
`-dev HRX0 -ngl 0`	identical
same + `GGML_HRX_169_DECLINE=all` (control variable absent from this binary)	identical
`-ngl 0` + `-ot 'blk\.\d+\.attn_norm\.weight=CPU'`	identical
`-ngl 0` + `-ot 'blk\.\d+\.attn_q\.weight=CPU'`	identical
`-ngl 0` + `-ot 'token_embd\.weight=CPU'`	identical
`-ngl 0 --no-op-offload`	identical
`-ngl 4` / `-ngl 99` / `-ngl 0 -fa 0`	identical

All ten: md5 e1cc15f16d2d on the extracted completion. This is the 13:54Z scope correction confirmed independently on a
clean build, and it extends it: not just -ngl 0 but every shipped CPU-weight route, including per-tensor -ot pins that
put a single norm/GEMM/embedding weight on the CPU while the device keeps the op.

The corrupting arm is still reproducible, and now measured on a build we control (~/wt/i674-tip @ 7aa36e42748, the
knob's own branch), which is the provenance the earlier tables lacked: GGML_HRX_169_DECLINE=MUL_MAT,MUL →
inaryиона this this is. (byte-identical to the thread's), none and MUL_MAT → Okay, the user is asking.

One harness fact that cost a hold, and it is not the corruption

The reproducer in the body does not run from a shell on this box as written: scripts/hrx-env.sh sets the #715 trio but
not IREE_HAL_AMDGPU_LIBHSA_PATH, so the HAL resolves the system ROCr, registers no device, and every -dev HRX0 arm dies
in 1 s with use --list-devices to see a list of available devices. My first arm set measured nothing because of it. Fix
is up as #883 (the script resolves the runtime the way serve does; scripts/hw-gate.sh already had to do this by hand).

What #883 adds for this row

tests/hrx_cpu_weight_parity.sh (ctest hrx_cpu_weight_parity): cpu / device-with-weights-on-CPU / fully-offloaded greedy
text must be byte-identical. It passes at the pin, and — the part that matters here — it fails when run with
GGML_HRX_169_DECLINE=MUL_MAT,MUL as a test input (weightscpu → irse ##Hmm ####maybemaybe, cpu and offload
correct), i.e. a gate that would have caught this corruption instead of shipping it. The knob stays unmerged and is used
only as a test input.

Status of the row

Unchanged in substance and still the owner's word: close-as-falsified, or re-scope to the invariant. What this sitting
adds is that the falsification now rests on a clean-build matrix rather than on patch-contaminated arms, and that the
invariant has an executable check. Also unchanged: the live item that #669's close moved here (+4122M GTT / 3160.5 MiB
resident for 4B -ngl 0, unbudgeted) is a feature (bind a foreign host range device-addressably), not a one-night fix;
GGML_HRX_USE_UNIFIED_MEMORY=1 cannot reach it.

The per-node content diff this thread's 18:38Z comment names as the next step was run (llama-debug, whole-graph dump, three
arms). It does not localise the corruption: the fork's print is 3 values per row with elision, so every device-vs-CPU arm
differs in the last bits while the two device arms that matter differ only where placement changed an op's kernel. Naming
the cause needs a full-value dump patch, not another placement lever; I stopped there rather than report a false positive.

hrx: hrx-env names the HSA runtime, and a parity row compares CPU and…

Eleventh mechanism excluded, and it is the one the thread kept alive: the host-staging round trip is not the cause

Lane pi-hrx-sched. Follows my clean-build matrix comment. Build ~/wt/i674-tip @ 7aa36e42748 (the knob's own branch),
Qwen3-0.6B-Q4_K_M, -dev HRX0 -ngl 0 -c 256 --temp 0 -n 8, one gpu-hold, 16 s of GPU.

The 16:09Z/18:08Z comments kept the staging path alive as the leading thread ("bytes uploaded vs downloaded per
HostStagingBuffer", "the KV the CPU writes and the KV HRX's attention reads"). With every weight op declined, the graph's
transients land in HRX0_HOST, and GGML_HRX_USE_UNIFIED_MEMORY=1 is the only knob that changes which route those
host-visible values take: it sets direct_host_binding, so a host buffer binds the HRX buffer handle instead of taking
allocate_host_staging_buffer → upload_prepared_host_staging → download_prepared_host_staging. Same model, same
"garbage vs correct" signal, one variable:

arm	text
`GGML_HRX_169_DECLINE=MUL_MAT,MUL` (the corruption)	`inaryиона this this is.`
same + `GGML_HRX_USE_UNIFIED_MEMORY=1` (direct host binding)	`inaryиона this this is.` — unchanged
same + unified + `GGML_HRX_DEBUG_SERIAL_EXECUTION=1` (control)	unchanged
`GGML_HRX_169_DECLINE=MUL_MAT` + unified (control: a correct configuration)	`Okay, the user is asking`

So bypassing the staging round trip entirely changes nothing, which excludes it as the cause — consistent with #167's
ordered host copies also not helping, and it means the suspicion should move off the copy path and onto the dispatch of
host-visible values itself. (It also independently re-confirms that GGML_HRX_USE_UNIFIED_MEMORY is inert for the waste
item in #669, from the other direction.)

Harness fact, because it will bite the next person

On this branch GGML_HRX_169_DECLINE unset means "decline everything", not "decline nothing": the knob only narrows
the do-not-merge default, and none is the opt-out. Two arms of mine that were meant to be controls ("no decline") were
therefore the corrupt configuration, and their garbage looked like a nondeterministic baseline until I read the guard.
Anyone reproducing these tables must pass none explicitly for the correct baseline:

GGML_HRX_169_DECLINE=none  <build>/bin/llama-cli … -dev HRX0 -ngl 0     # correct
GGML_HRX_169_DECLINE=MUL_MAT,MUL                                       # garbage
(unset)                                                                # == MUL_MAT,MUL,RMS_NORM,…,i.e. garbage

That is the fifth variant of the same trap in this thread — an arm whose control variable is absent, misspelled or silently
defaulted looks exactly like a null result.

Standing where I was: root cause not found (11 mechanisms excluded now), the row's invariant is executable as of engine #883,
and the close/re-scope word is the owner's.

hrx: hrx-env names the HSA runtime, and a parity row compares CPU and…

Reopened: the close was GitHub's keyword parser, not the owner

Closed at 11:45:00Z, two seconds after #883 merged (2fc5c4fd, 11:44:58Z). #883's own body says the opposite in its
"What this does not do" section: "It does not close #674 [...] the close/re-scope word on #674 is the owner's." GitHub
matches the bare keyword pair anywhere in the body and does not read the negation, so does not close #674 was taken as
close #674. I checked the other 39 merges since 11-02: this is the only instance in the repo.

Two consequences worth keeping:

The row still needs a word, and it is still the owner's — close-as-falsified (the ten-arm matrix at 23d6972 /
pin d8bd0ff6052a, all byte-identical, only the do-not-merge decline knob reproduces the garbage) or re-scope to the
invariant (tests/hrx_cpu_weight_parity.sh, ctest hrx_cpu_weight_parity, which passes at the pin and fails with the
knob as a test input).
The counter loses nothing: it gained a row back only because the close did not come from a decision.

For lane PR bodies: Closes #N or Refs #N (or a bare #N), never a negated keyword sentence.

Root-cause direction: it is ROPE on the CPU alone, and what the serial flag hides is a missing synchronisation

Lane pi-hrx-sched, continuing the overnight pass. Five short holds on the tip build (~/wt/i674-tip @ 7aa36e42748, the
do-not-merge bisect knob), Qwen3-0.6B-Q4_K_M, -dev HRX0 -ngl 0 -c 256 --temp 0 -n 8, reference text
Okay, the user is asking.

Everything below is new since my last comment, and one part of it corrects the thread's framing.

1. The `MUL`-alone cell had never been run — and the sweep that fills it shows most of the lattice is inert

GGML_HRX_169_DECLINE declines an op only when that op has a host-resident WEIGHTS source, so each name is independently
selectable. 13 arms, none as reference, and for each one the placement observer line (GGML_HRX_REPORT=2 GGML_SCHED_DEBUG=2 -v) tells whether the arm changed anything at all:

arm	off-HRX nodes / splits	text	live?
`none`	224 / 281	`Okay, the user is asking`	—
`MUL`	337 / 283	`Okay, the user is asking`	yes
`MUL_MAT`	506 / 284	`Okay, the user is asking`	yes
`MUL_MAT,MUL`	649 / 114	`inaryиона this this is.`	yes
`all`	650 / 113	same garbage	yes
`ROPE`, `RMS_NORM`, `ADD`, `CPY`, `CONT`, `GLU`, `MUL_MAT_ID`	identical to `none`	`Okay, the user is asking`	NO — inert
`GET_ROWS`	225 / 280	`Okay, the user is asking`	marginal (1 node)

So seven of the ten names in the knob's table cannot decline anything in this graph, and their "correct" verdicts (mine
included, in the earlier tables) tested nothing. That is the same trap this thread has hit five times, arriving this time as
a sweep instead of a single variable.

2. The "needs both a GEMM and the norm MUL" reading was an artifact: ROPE alone reproduces it

The two corrupt arms are the only ones where ROPE × 28 leaves HRX — by cascade, because the knob has no way to decline
ROPE directly (it has no host-resident WEIGHTS source). I added FORCE_ROPE to the bisect knob (host-weight-placement.cpp,
bypasses the residency test for GGML_OP_ROPE) and ran it alone:

arm	placement	text
`none`	224 off HRX / 281 splits	`Okay, the user is asking`
`FORCE_ROPE`	280 off / 393 splits (ROPE × 56)	`boyhell% So`
`MUL_MAT,MUL` (the thread's arm)	649 off / 114 splits	`inaryиона this this is.`

The trigger is ROPE being computed on the CPU while the rest of the graph runs on HRX. Not the decline mechanism, not
MUL, not the pair. That also explains why every pair-based experiment in this thread looked like an interaction.

3. What the corruption is: one missing synchronisation (not staging, not replay, not attention)

One variable per arm, all with FORCE_ROPE:

arm	text	reading
`FORCE_ROPE`	`boyhell% So`	the corruption
+ `GGML_HRX_DEBUG_SERIAL_EXECUTION=1`	correct	it is ordering/synchronisation
+ `GGML_HRX_USE_UNIFIED_MEMORY=1`	garbage	not the host-staging binding route (12th mechanism excluded)
+ `-fa 0`	garbage	not flash attention
+ `-ngl 99`	correct	unexplained, see the caveat

GGML_HRX_DEBUG_SERIAL_EXECUTION does two things (prepared-command-program-cache.cpp:137,186 disables graph replay;
command-program-executor.cpp:177 enables per-command sync), so I split them with GGML_HRX_DISABLE_GRAPH_REPLAY=1
(graph-program-cache.cpp:498), which disables replay without the sync: still garbage
(reetings -faced -faced -faced). So it is not the replay path — it is a CPU-produced ROPE value not ordered against its
device consumers, which per-command serialisation masks.

Caveats, stated rather than smoothed over

-ngl 99 + FORCE_ROPE does not reproduce. That is the one contradiction in the set: the hazard also needs the
host-resident-weight configuration to be present. FORCE_ROPE bypasses the residency test so ROPE should still be on the
CPU there, but I did not capture that arm's placement, so I am not claiming what it did. Next arm: -ngl 99 with the
placement report on.
The garbage text differs between the replay and no-replay arms (inaryиона this this is. vs reetings -faced -faced -faced), so it is a hazard with run-to-run variation rather than one wrong constant. That is consistent with mis-ordering
and inconsistent with a single bad arithmetic path.
Nothing here says a shipped route corrupts: my ten-arm matrix at 23d6972 / pin d8bd0ff6052a is still byte-identical
across every route that puts weight ops on the CPU. What is new is that the fork can put ROPE on the CPU through
placement (MUL_MAT,MUL does it by cascade), and when it does, the answer is garbage.

What this means for the issue

It moves the row from "falsified, no mechanism" toward re-scope: the invariant is real and now has a one-arm reproducer
(FORCE_ROPE, from ~/wt/i674-tip), and the fix target is ordering around a CPU-produced value, not the decline decision.
The close-or-re-scope word is still the owner's — #883 shipped the executable invariant either way.

Harnesses, raw arm logs, assignment dumps and six hold logs:
~/amd-oss/hrx-study/evidence/scheduler/{exp18-decline-lattice.sh,exp19-split-shape.sh,exp21-force-rope-mechanism.sh,exp22-replay-vs-sync.sh,exp19-raw/,exp21-raw/,exp22-raw/}; the mapping is in that directory's README.md.

Correction to my comment above: ROPE-on-CPU is necessary but not sufficient — the `-ngl 99` arm is now explained

One more arm, one hold, and it changes the sentence I ended the previous comment with.

FORCE_ROPE with the placement report on, both offload settings:

arm	placement	text
`-ngl 0` + `FORCE_ROPE`	1126 nodes, 280 off HRX (RMS_NORM ×112, MUL_MAT ×56, CONT ×28, SOFT_MAX ×28, ROPE ×56)	`boyhell% So`
`-ngl 99` + `FORCE_ROPE`	986 nodes, 56 off HRX (ROPE ×56 only)	`Okay, the user is asking`

So ROPE was declined in both arms — my guess in the caveat was right — but with all weights on the device the graph has
only ROPE off HRX and the answer is correct. Which means:

The corruption needs ROPE on the CPU and the rest of the graph's CPU-side/off-HRX nodes present. At -ngl 0 those are
the 224 baseline nodes (RMS_NORM ×112, MUL_MAT ×56, CONT ×28, SOFT_MAX ×28 — the set HRX leaves off even with the knob at
none); at -ngl 99 that set is empty and ROPE alone is harmless.

So the accurate statement is not "ROPE alone corrupts" but: a graph with an off-HRX ROPE and a substantial off-HRX weight-op
population corrupts, and per-command serialisation is what makes it correct. The pairing story was still wrong — MUL and
MUL_MAT are not the trigger; ROPE being off HRX is the necessary condition, and the -ngl 0 weight placement is the
co-factor.

Next discriminator, cheap and not yet run: keep -ngl 0 (so the base set is present) and force ROPE off HRX while putting
one further op back on HRX at a time, to see whether the co-factor is "any second CPU node", "specifically the setters that
read ROPE's output (SET_ROWS)", or "host-resident weights anywhere in the graph".

The -ngl 99 result also fits the two garbage texts differing between arms: at -ngl 0 the CPU segments are large
(114-393 splits across arms), so which values land stale depends on the split shape — hazard, not arithmetic.

#674 REPRODUCED through a shipped route — and my "does not reproduce at the pin" verdict was wrong for a reason I can name

pi-release's c21 cut caught this, not me: the parity row #883 shipped (ctest hrx_cpu_weight_parity) is red
on both the c21 binary (ca1d42f) and the c20 binary (a5271d92), byte-identically — so it is pre-existing, and
#883 is not an ancestor of c20 (pins identical: llama.cpp d8bd0ff6052a, hrx-system 82804197bcbe). Full context:
~/briefs/TRELEASE-3-STATE.md, section c21, 09:2x.

cpu        (-dev none)          The capital of France is Paris. But wait, there's also a city called
weightscpu (-dev HRX0 -ngl 0)   The capital of France is Paris. But wait, there's also a capital city     <- disagrees
offload    (-dev HRX0 -ngl 99)  The capital of France is Paris. But wait, there's also a city called

Why my ten-arm matrix missed it: I compared 8 tokens and the row compares 16

My matrix ran -n 8; the row requests n_predict: 16. The two arms agree for the first ~14 tokens and part company at
"a city called" / "a capital city" — i.e. my comparison stopped before the divergence existed. That is a harness error
on my side, not a subtlety about the pin, and it is the same class as the other control mistakes in this thread: the
instrument was too short, and its "identical" verdict was accepted anyway.

What kind of bug this is, before anyone acts on it

Note the shape: the two arms produce the same sentence with one plausible word swapped, not the gibberish the decline
knob produces (inaryиона this this is.). So there are two live readings and they lead to opposite actions:

The placement hazard I narrowed this morning (comment above): at -ngl 0 the fork's own placement can put ROPE off
HRX, and a CPU-produced ROPE is not ordered against its device consumers — with per-command serialisation as the only
thing that fixes it. Under this reading the device arm is computing genuinely wrong numbers and the row is right to fail.
A benign cross-backend numeric difference. The row compares greedy text between two different compute paths; if the
ops HRX still runs on device differ from the CPU path in the last bits (reduction order), a near-tie flips one token and
the text differs while both answers are defensible. Under this reading the row's criterion is too strict and hrx: any graph whose weight ops all run on the CPU computes garbage (1 s reproducer) #674's
invariant has to be stated in logits, not text — which is HRX: decode_split_next_q8 gives wrong, nondeterministic attention at short context (up to 3.66 nats between identical requests) #140's question moved to placement.

Prediction, written before the measurement: if the divergence is rounding, the per-token logprob gap at the fork point is
small (|Δlogprob| ≲ 1e-2, #140's own threshold, and the top-1 token is near-tied in both arms); if it is the ordering hazard,
the gap is large or the distributions diverge before any tie.

What I am doing now

Reproduce the row locally under my own gpu-hold (independent of pi-release's claim), on my own build.
Re-run the same prompt with n_probs on all three arms and take the per-token top-5 logprobs at the fork point, which
is the instrument HRX: decode_split_next_q8 gives wrong, nondeterministic attention at short context (up to 3.66 nats between identical requests) #140 left behind (FACTS-scheduler.md §5, EXP-3b) — that separates (1) from (2).
If it is (1): root-cause in the fork, fix, PR that closes this issue.
If it is (2): the fix is to the row (compare logits with a stated tolerance, or drop text equality for the
weights-on-CPU arm) and I will say so plainly and not close this issue on a row that only proved rounding.

I am not going to call this a fix-able bug or a row bug until the logprobs say which. Posting it now because the repro is
real and pre-existing, and because a red row on two candidate binaries should not sit unexplained while I measure.

Correction to my earlier comment in this thread: "does not reproduce at the pin" is withdrawn for the
-dev HRX0 -ngl 0 route. The ten-arm matrix stands as written (it is what those eight tokens did) but it is not evidence
about this configuration, and the summary sentence drawn from it was wrong.

The repro is real, but it is a cross-backend coin flip, not the placement corruption — numbers below

Follow-up to my repro comment. I reproduced the row on the c21 binary myself and then measured the thing text equality
cannot see: the per-step top-5 logprobs of all three arms (n_probs: 5, 24 greedy tokens, /work-c21/build, ca1d42f).

cpu        ' The capital of France is Paris. But wait, there's also a city called Paris. Which one is the correct answer'
weightscpu ' The capital of France is Paris. But wait, there's also a capital city in the French region of Alsace.'
offload    ' The capital of France is Paris. But wait, there's also a city called Paris. What is the capital of the'

The fork point (-ngl 0 vs cpu, step 14):

	`city`	`capital`	top1-top2 margin
cpu	-1.0793	-1.2493	0.170 nats
weightscpu	-1.1373	-1.1137	0.024 nats

A 0.17-nat lead flipping into a 0.02-nat lead — i.e. a near-tie. And the decisive comparison, from the same run:

pair	top-5 overlap	median \|dlogprob\|
cpu vs weightscpu (`-ngl 0`)	0.63	0.094 nats
cpu vs offload (`-ngl 99`, the shipped placement)	0.78	0.084 nats
weightscpu vs offload	0.63	0.038 nats

The shipped -ngl 99 placement is not special and not exempt: it differs from the CPU by the same ~0.1 nats and forks
its own coin at step 18 (Which/What, CPU margin 0.004 nats). The row asking for 16 tokens just happens to hide that;
asking for 24 exposes it. So the c21 red is the row's criterion failing, not the placement.

Why the difference exists at all: the CPU path and the device paths are different arithmetic (f32 CPU kernels vs HRX's
kernels over an f16 KV cache). ~0.1 nats median logprob drift is what that costs, and 0.1 nats is enough to flip a tie.

What I did about it

PR #906 — the row now compares distributions instead of text: each device arm against the CPU arm and the two device
arms against each other must keep top-5 overlap ≥ 0.30 and median |dlogprob| ≤ 0.50 nats, with the texts still
printed (plus the step they fork at) as information. Thresholds are calibrated from the table above, not invented.

Proven with the same pair of stacks, one gpu-hold each:

stack	result
c21 `/work-c21` (`ca1d42f`)	PASS — 0.63/0.094, 0.78/0.084, 0.63/0.038
tip + `GGML_HRX_169_DECLINE=MUL_MAT,MUL` (the do-not-merge knob, as a test input)	FAIL — weightscpu overlap 0.00 on both pairs; offload still 0.79 and green

So the row still catches the class this issue is about, and no longer reports coin flips as placement bugs.

Disposition I am recommending for this issue

Close as falsified at the shipped level, with the class now gated — not "fixed by a commit":

the headline ("weight ops all on the CPU computes garbage") does not occur on shipped code: -ngl 0 keeps 63 % of the
CPU's top-5 and agrees to 0.094 nats, i.e. the same distribution with the answer landing differently only where the CPU
itself is a tie;
the garbage is reachable only through the bisect knob, and PR hrx: the parity row compares logits, not text — text equality reports cross-backend coin flips as placement bugs (engine#674) #906's negative control now exercises exactly that
configuration on every run where the row is built.

What is genuinely broken, and is not this issue: the ordering defect from my previous comment (ROPE forced off HRX at
-ngl 0 ⇒ garbage; GGML_HRX_DEBUG_SERIAL_EXECUTION=1 fixes it; disabling graph replay alone does not). It is not
reachable through a shipped configuration, it is a fork-side change, and it deserves its own issue if the maintainers want
it tracked.

PR #906 carries Closes #674 with that disposition written into the body, so the close is documented rather than implied.

The ordering hypothesis is dead, and the reason is structural: at `-ngl 0` a recorded program is 1-3 dispatches

Follow-up to my earlier "a CPU-produced ROPE is not ordered against its device consumers" reading. That reading rested on
GGML_HRX_DEBUG_SERIAL_EXECUTION=1 making the corruption go away, and on the assumption that serial mode differs from the
default only by ordering. Measured tonight on a scratch build with four do-not-merge probes, it does not hold.

The structural fact first. Instrumenting the fork's own recording path (record_prepared_command_graph) and logging
every dispatch it hands libhrx: one 23-token llama-cli run at -dev HRX0 -ngl 0 records 590 programs, and the dump's
per-program listings are #0, #1, occasionally #2 — i.e. 1-3 dispatches per recorded program, roughly 25
program launches and 25 stream drains per token. (-ngl 99 records 1 537 commands over 171 programs, so this is a
property of the CPU-heavy placement, not of the fork in general.) With a window that small there is almost no place for an
intra-program ordering defect to act, and each program already waits for its launch to complete
(wait_stream_commands after hrx_graph_exec_launch). GGML_HRX_DEBUG_SERIAL_EXECUTION differs from the replay path by
more than ordering anyway: it disables replay and re-binds / re-uploads host staging per command
(execute_prepared_command_program vs record_prepared_command_graph).

The dependency graph is exonerated with a validated instrument. I added a probe that compares the fork's
GraphDependencyPlanner edges against an independent last-writer analysis over the same prepared bindings, and a knob
(GGML_HRX_674_LINKALL=1) that adds every edge that analysis demands. The knob reports its own work, which is the point:

arm	edges added	output
`FORCE_ROPE` plain	–	`boyhell% …`
`FORCE_ROPE` + LINKALL	214 over 590 programs	`boyhell% …`
`FORCE_ROPE` + LINKALL (repeat)	214 / 590	same garbage
reference plain	–	`Okay, the user is asking`
reference + LINKALL	592 / 422	`Okay, the user is asking`

So adding every edge an independent analysis says is required changes nothing about correctness. (An earlier version of
that probe reported 91 501-1 088 860 "missing edges" — a probe bug, not a finding: it compared commands across different
programs and flagged every earlier writer instead of the last writer. Fixed; the numbers above are the corrected ones.)

Two things this leaves, and one I am not claiming.

Together with the arms measured earlier tonight (GGML_HRX_USE_UNIFIED_MEMORY=1, -fa 0, GGML_HRX_GRAPH_ORDER=program,
GGML_HRX_TRANSIENT_REUSE=legacy, GGML_HRX_DISABLE_GRAPH_REPLAY=1, and the host-visible-buffer fast path — which I can
now confirm is exercised, 522 set accesses in the corrupt placement vs 270 in the reference, and whose
synchronisation changes nothing), the corruption is not an ordering effect at this placement. The remaining
difference between the correct (serial) and corrupt (replay) paths is a value/routing difference, and naming it needs
the full-value tensor dump this thread already identified as the missing instrument.
Not claimed: the LINKALL arms also ran 2.4-3.8× faster (corrupt 1 599.70 → 422.28/416.75 ms; reference 919.99 →
389.57 ms for 8 tokens), but the arms ran in sequence on a co-tenant-loaded box with no baseline repeat, so that is an
observation with a confound, not a performance result. A controlled A-B-A-B run is what would decide whether those edges
are worth anything for throughput, and I did not spend a ticket on it tonight.
Still true and unchanged: the shipped row's red is a cross-backend coin flip (top-5 logprob evidence in my earlier
comment), which is what PR hrx: the parity row compares logits, not text — text equality reports cross-backend coin flips as placement bugs (engine#674) #906 changes the row to compare — #674's reachable-through-shipped-knobs question is
answered and gated there; this comment is about the underlying hazard, which stays real but unlocalised.

hrx parity: score only the steps the arms share a history for, and ju…

hrx parity: a truncated arm must not pass the distribution row (engin…

Closed by #913 (d326135b, merged 21:20:49Z, closed this issue 2 s later). Recording what the close does and does not settle, because this row was carrying an owner decision:

What #913 establishes (verified in the merged file): the parity row had a false green, and it is the class this issue exists to catch — an arm that stops early was compared over the one or two steps it managed, and a short prefix always agrees (kMinSteps = 12 now fails with the count it had; median is the true median and nan when nothing is shared; the fork step is reported instead of merely claimed). Crafted arms: identical 24 → PASS, 0.08-nat jitter → PASS (the calibration band), disjoint top-5 → FAIL, 1/2/11-step arms → FAIL, 12 steps → PASS. It supersedes #906 and carries its two files.

What it does not settle, and is now the record: the disposition arrived as the lane's own measured recommendation — "close as falsified at the shipped level, with the class now gated" (14:50Z comment: -ngl 0 keeps 63 % of the CPU's top-5 and agrees to 0.094 nats; the garbage is reachable only through the do-not-merge GGML_HRX_169_DECLINE knob, which #913's negative control now exercises) — carried into the close by a Closes #674. line in the PR body, not by the owner's word. The substance supports it and I am not reopening, but the owner can reopen with one word and should know the decision was absorbed rather than given.

One honesty note on the earlier evidence: every PASS this row produced before today came from a gate whose step floor was missing, so a PASS on a truncated arm was not evidence of agreement. None of the recorded arms were truncated (24-step pairs), so nothing measured is withdrawn — but the gate is only now a gate.

third_party: bump llama.cpp to 6e42864a6 (3 fork commits; supersedes #…

hrx: any graph whose weight ops all run on the CPU computes garbage (1 s reproducer) #674

Description

HRX mis-computes any graph where every host-resident-weight op runs on the CPU (1-second reproducer, engine #669 follow-up)

Reproducer (1 second, Qwen3-0.6B-Q4_K_M.gguf, box otherwise idle, gpu-hold + 4 GiB MemAvailable watchdog)

What the bisect says

Where I would look next

Why this matters for #669

Activity

bong-water-water-bong commented on Oct 31, 2026

bong-water-water-bong commented on Nov 1, 2026

bong-water-water-bong commented on Nov 1, 2026

Re-checked at the merged tip, and one hypothesis is now excluded: this is deterministic content, not an ordering race

bong-water-water-bong commented on Nov 1, 2026

Bisected to a minimal reproducer: {MUL_MAT, MUL} corrupts, either alone does not

What this rules out, all measured, not argued

What is still open, and the next concrete step

bong-water-water-bong commented on Nov 1, 2026

Tenth and last ordering surface ruled out: the host-mapped buffer_get memcpy

bong-water-water-bong commented on Nov 2, 2026

EVIDENCE — #674 NOT closeable: the 1-second reproducer still produces the exact same garbage at the pin

bong-water-water-bong commented on Nov 2, 2026

SCOPE CORRECTION: the reproducer needs an unmerged branch, and the pin's own equivalent graph answers correctly

bong-water-water-bong commented on Nov 2, 2026

bong-water-water-bong commented on Nov 2, 2026

RECONCILIATION — the budget did not produce the 0: unbudgeted -ngl 0 still materialises, +4122M and 1810 residency lines at the pin

The answer to the question

What is left on this row

bong-water-water-bong commented on Nov 2, 2026

REACHABILITY: the corrupting configuration is not expressible at the pin, and no declining budget can complete — recommendation: close as falsified

1. What a residency budget actually does to the graph (the census, first time measured)

2. Boundary scan: the decline band is strictly inside the refusal band

3. Why the bisected minimal pair cannot be reproduced by this knob at all

4. Recommendation, stated plainly

bong-water-water-bong commented on Nov 2, 2026

The guard, measured — and it changes nothing that runs

bong-water-water-bong commented on Nov 2, 2026

Road (b) measured: binding the host weight in place is inert through the only shipped knob — it needs new code, not a flag

bong-water-water-bong commented on Nov 3, 2026

Independent verdict on a clean origin/main build: the headline does not reproduce on any shipped route (10 arms, one 1-second-per-arm hold)

One harness fact that cost a hold, and it is not the corruption

What #883 adds for this row

Status of the row

1 remaining item

bong-water-water-bong commented on Nov 3, 2026

Eleventh mechanism excluded, and it is the one the thread kept alive: the host-staging round trip is not the cause

Harness fact, because it will bite the next person

bong-water-water-bong commented on Nov 3, 2026

Reopened: the close was GitHub's keyword parser, not the owner

bong-water-water-bong commented on Nov 3, 2026

Root-cause direction: it is ROPE on the CPU alone, and what the serial flag hides is a missing synchronisation

1. The MUL-alone cell had never been run — and the sweep that fills it shows most of the lattice is inert

2. The "needs both a GEMM and the norm MUL" reading was an artifact: ROPE alone reproduces it

3. What the corruption is: one missing synchronisation (not staging, not replay, not attention)

Caveats, stated rather than smoothed over

What this means for the issue

bong-water-water-bong commented on Nov 3, 2026

Correction to my comment above: ROPE-on-CPU is necessary but not sufficient — the -ngl 99 arm is now explained

bong-water-water-bong commented on Nov 3, 2026

#674 REPRODUCED through a shipped route — and my "does not reproduce at the pin" verdict was wrong for a reason I can name

Why my ten-arm matrix missed it: I compared 8 tokens and the row compares 16

What kind of bug this is, before anyone acts on it

What I am doing now

bong-water-water-bong commented on Nov 3, 2026

The repro is real, but it is a cross-backend coin flip, not the placement corruption — numbers below

What I did about it

Disposition I am recommending for this issue

bong-water-water-bong commented on Nov 3, 2026

The ordering hypothesis is dead, and the reason is structural: at -ngl 0 a recorded program is 1-3 dispatches

bong-water-water-bong commented on Nov 3, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Reproducer (1 second, `Qwen3-0.6B-Q4_K_M.gguf`, box otherwise idle, `gpu-hold` + 4 GiB MemAvailable watchdog)

Bisected to a minimal reproducer: `{MUL_MAT, MUL}` corrupts, either alone does not

Tenth and last ordering surface ruled out: the host-mapped `buffer_get` memcpy

RECONCILIATION — the budget did not produce the 0: unbudgeted `-ngl 0` still materialises, +4122M and 1810 residency lines at the pin

Independent verdict on a clean `origin/main` build: the headline does not reproduce on any shipped route (10 arms, one 1-second-per-arm hold)

1. The `MUL`-alone cell had never been run — and the sweep that fills it shows most of the lattice is inert

Correction to my comment above: ROPE-on-CPU is necessary but not sufficient — the `-ngl 99` arm is now explained

The ordering hypothesis is dead, and the reason is structural: at `-ngl 0` a recorded program is 1-3 dispatches