[bong-water-water-bong]

Untested on device. Host-side build + unit test only; no GPU run was made.

Why

The streamed-expert prefetch on Qwen3.8-Flash-Next paid for a prediction that cannot work on this model. With hyper-connections the residual is 4 x 2560 wide, and layer l+1's router input is l+1's own mixer applied to the residual after layer l's MoE. The gate-ahead form (ONEBIT_MOE_PREFETCH in the streamer) applies l+1's router to layer l's FFN input, so it mixes through the wrong weights: right about two times in three, and a net loss on a drive that is already the limit (engine docs/moe-streaming.md, "Flash-Next, streamed": 133 s per pass without prefetch, 247 s with).

What this adds

Inside layer l's graph, before l's MoE output exists:

1. l+1's PLE injection (if it has one), then l+1's attention or DeltaNet block, then its combine — its own mixer block, on the current residual;
2. l+1's FFN mixer and its router matrix;
3. the top-k ranking of the logits, ONEBIT_MOE_PREDICT_DEPTH deep (deeper than the layer's own top-k, so the near misses are read too), with their logits for the margin gate.

Only the ranking is used. Whatever a token really routes to is still waited for, so the model's output is unchanged — the predictor decides which experts a prefetch reads, nothing else.

State-write suppression. The speculative pass must not leave state the exact pass reads: the DeltaNet recurrent state and the conv histories (DeltaNet and PLE) are written into same-shaped scratch tensors, so one scratch per shape serves every layer. An attention successor needs none: the exact block rewrites the same cache cell before anything reads it (build_attn_qsa's cpy precedes its attention in graph order), and the speculative selector's approximate keys only degrade the prediction. The speculative nodes are also left out of the fused-op manifest (register_fused=false, spec=true), so a fused dispatch cannot claim the wrong node.

Files

• new, ours: src/onebit-moe-predict.{h,cpp} — environment, the margin gate, the hit-rate table (per layer: predicted/actual experts used, candidates read, candidates gated), a P/A trace for offline comparison, and the prefetch sink (set_prefetch).
• new, ours: tests/test-onebit-moe-predict.cpp — the host-side policy: margin gate, depth ceiling, floor, layer-mask parsing, effective depth. ctest -R onebit passes.
• hooks: src/models/qwen4exp.cpp (the layer loop + the predictor itself + optional scratch parameters on build_layer_attn_linear/build_ple and a spec flag on build_layer_attn/build_qsa_sel/build_layer_attn_linear), src/models/models.h, src/models/delta-net-base.cpp (register_fused, default true), src/llama-graph.cpp (one line: capture the routing each layer really used), src/llama-context.cpp (one line: read the prediction back after the step), CMake for the two new files.

Flags

ONEBIT_MOE_PREDICT=off|router|block (off by default), _DEPTH, _MARGIN (read a deeper candidate only while its logit is within that distance of the best), _MIN, _LAYERS (1,2,5-9,47), _NO_ATTN, _OUT (trace), _STATS (per-layer hit rate at exit). router keeps the cheap residual-only form, for an A/B on the same build.

What is not done

• The prefetch consumer. llama.cpp runs all 48 layers in one graph, so the readback lands after the step; the consumer has to be either the streamer's in-graph op (reading the prediction as an op input) or a one-graph-per-layer streamed decode. Both need the device. Until then the sink is the trace, which is exactly the hit-rate measurement the device plan asks for: prediction hit rate per layer vs the routing the step really used.
• The MTP draft graph also triggers the readback of the trunk's last step, so the table counts a step twice when -md is on. The hit ratio is unaffected.
• Numbers. Accuracy, the per-layer cost of the extra block, and the streamed pass time are the device plan (in the private notes: hit rate first, then pass time at 4,608/9,216 slots against the 133 s baseline, then long context).

Design record

engine docs/moe-streaming.md, "The next layer's own block": the mechanism, the three state-write cases, the cost per predicted layer, and the two ways to feed the prefetch. The engine side (--moe-predict, docs) PRs separately.

🤖 Generated with Claude Code

[bong-water-water-bong]

Review (orchestrator): predictor in our own files with a unit test, hooks in the qwen4exp layer loop and the delta-net base kept to parameters and a spec flag, everything behind ONEBIT_MOE_PREDICT (default off). Draft until the device measurement: hit rate vs the trace (router vs block), per-layer cost of the speculative block, tok/s streamed and resident on Flash-Next, after the leaderboard releases the lane.

[bong-water-water-bong]

Triage: needs work — three things, none of them the predictor itself.

1. Base branch. The PR targets master, but the engine pins the fork's 1bit/hrx-vulkan-patched lineage. This commit (5f0d4619a, parent c35b66744) sits on upstream master, 11,511 commits of divergence from the pin e7e5cd08; it is not an ancestor of the pin or of the pin the engine proposes to move to (40c4ccb76e48). As it stands, merging this lands nowhere the engine can see. Rebase it onto the fork branch.
2. The consumer is missing. The PR says so itself: llama.cpp runs all 48 layers in one graph, so the prediction is computed and read back but nothing is prefetched with it. Today the feature is therefore a predictor with no prefetch, and the engine-side flag (--moe-predict, engine Revert 7e53955 (#542) and fix properly ggml-org/llama.cpp#551) sets an env var that changes nothing observable.
3. Untested on device (PR body: "Host-side build + unit test only"). That is the honest state, but it means the size of the win is unknown — and the design only pays off once a prefetch consumer exists.

Diff read: src/onebit-moe-predict.h/.cpp (123 + 372 lines), tests/test-onebit-moe-predict.cpp, src/models/qwen4exp.cpp (+128/-10), models.h, delta-net-base.cpp (register_fused), one line in llama-graph.cpp to capture real routing and one in llama-context.cpp to read the prediction back, plus CMake. Self-contained as a host-side unit.

Related: engine ggml-org#551 already documents the graph side as living "in the llama.cpp pin", which is not true of e7e5cd08 or of 40c4ccb76e48 — commented there. Whoever lands this needs the pin bump to follow, otherwise the flag stays inert.

[bong-water-water-bong]

VERDICT: APPROVE — off by default, the "what is not done" list is honest, and the design record exists where a reader will look. Three things I would want before anyone enables it, the first of which I think is the whole review.

1. The riskiest claim has no gate: "the model's output is unchanged". Everything else in the PR rests on it — a speculative attention/DeltaNet block is executed inside layer l's graph, ahead of the data the exact pass uses, and the only thing standing between that and corrupted output is state-write suppression. That is a claim a test can settle exactly, and cheaply, once a device run happens:

ONEBIT_MOE_PREDICT=off  vs  ONEBIT_MOE_PREDICT=block   ->  logits must be BIT-IDENTICAL

Bit-identical, not close, because the PR's own argument is that only the ranking is used: nothing about the arithmetic of the real path changes, so any difference at all is leaked state — a scratch tensors' lifetime, a conv history, a cache cell the exact block did not rewrite. If that assertion is in the device plan, say so and this paragraph is done; if it is not, it is the first row I would add, ahead of hit rate and ahead of pass time. It is also the only one of the three that can be a bug; the other two are tuning.

2. "one scratch per shape serves every layer" — does the scheduler guarantee the order that assumption needs? With hyper-connections there are many predicted layers in one step, and if every speculative pass writes the same shape scratch, then two passes writing it are a write-after-write. In a single graph those nodes are independent, so the order they execute in is the scheduler's choice, not the layer loop's — and if pass A's router reads while pass B has overwritten the scratch, A's ranking is stale. Note the failure mode is the good one: the model output would still be correct (only the prefetch choice is wrong), so it would show up as a lower hit rate, not a wrong logit — which means the measurement you are about to trust could be measuring this instead of the mechanism. If the sharing is safe because the passes are serialised by a dependency I have missed, one sentence in the code comment naming that dependency makes it reviewable; if it is safe only because nothing has reordered yet, a scratch per layer removes the doubt for a few KB.

3. Memory: a speculative block is a whole block. Each predicted layer adds its neighbour's attention or DeltaNet block plus mixer and router to the graph, so its tensors have to be planned and reserved even though their result is discarded. On this box a memory surprise is not a slowdown: it is a load-time refusal or an allocation failure (my own runs today: refusals at 13.6 / 13.5 / 10.8 GiB of pool budget for a 4.7 GiB model, and a 900 s silent stall at 16.7 GiB). So the PR should state what _LAYERS costs in reserved bytes per predicted layer — with _LAYERS subsetting as the mitigation a user is told about. "Predict everything" and "predict eight layers" differ by an order of magnitude here, and the flag makes that the user's decision.

Smaller points, none blocking:

• The -md double-count is disclosed and correctly scoped ("the hit ratio is unaffected" is right: a step counted twice in both numerator and denominator does not move a ratio). Good.
• register_fused=false, spec=true is the right instinct — a fused dispatch claiming a speculative node would silently make the trace lie. Worth one clause saying what a non-fused backend does instead: the node is simply materialised, which for a 40 B MoE at 4,608 slots is the arithmetic in point 3.
• ONEBIT_MOE_PREFETCH's failure is the best part of the write-up: "right about two times in three, and a net loss on a drive that is already the limit — 133 s without prefetch, 247 s with". That is a 1.86× regression measured, and it is the argument for spending a whole block to get the routing right instead of guessing it. If that comparison can be repeated on the same build with router vs block vs off, the PR becomes its own justification.

I did not build or run this (no device arm exists yet, and the model is a 40 B MoE); I am reviewing the mechanism, the suppression argument, and the cost model. Refs the streamed-expert plan in engine docs/moe-streaming.md.

[bong-water-water-bong]

VERDICT: CHANGES at 5f0d4619ae — the design checks out, the base does not: this branch cannot reach the engine from master, and its one test claim predates a rebase it now needs.

Blocking

1. Base is master, which is not the line the engine builds. Measured on the fork today:

merge-base(pinned, pr137)              -> (empty)
merge-tree(pinned, pr137)              -> fatal: refusing to merge unrelated histories
master (c35b66744f) has src/llama-kv-share?  -> 0 files
1bit/hrx-vulkan-patched (70b8fe458d)  -> 2 entries; 10,794 commits apart from master
recent fork PRs: #233, #235 (merged), #240 (open) -> base 1bit/hrx-vulkan-patched

So merging this lands on a line with no HRX backend and none of the fleet's work, and it cannot be merged forward either — the histories are unrelated. Retarget to 1bit/hrx-vulkan-patched and rebase; that is a mechanical rebase, not a redesign.

2. The ctest -R onebit claim is from 2026-10-24 on master. The rebase lands this in a graph layer that has moved 10.8k commits, including the fused-node registration and the HRX dispatch this PR edits. Please re-run the host build + ctest -R onebit at the new base and paste it; I am not reading a two-week-old pass as covering that.

3. The acceptance is unmeasured, and the consumer it exists for is not built. The body says so plainly, which is the right way to file it — but the title and the Why section make a performance claim (the 133 s → 247 s with the old prefetch) that nothing in the PR measures. Either add the hit-rate table the flag's defaults are tuned against (the per-layer P/A trace is already built — one device run at 4,608 slots would do it), or rescope the PR explicitly as plumbing only, off by default, no perf claim, and move the numbers to the device plan.

Verified, so the reviewer does not repeat it

• The signature change is non-invasive, which is the thing I expected to fail: bool register_fused = true is a default in models.h, and every other caller on this head still passes nine arguments — bailingmoe3.cpp:289, glm5-next.cpp:860, kimi-k3.cpp:459, qwen35.cpp:462, qwen35moe.cpp:472, qwen3next.cpp:543 — so only qwen4exp.cpp:1145 (!spec) changes behaviour. I checked each caller rather than the touched file.
• add_fused_node is skipped for speculative nodes, which is the correct guard for the stated reason (a fused dispatch must not claim a speculative tensor), and it defaults to true so the shipped path is untouched.
• ONEBIT_MOE_PREDICT=off by default: the risk surface for the released binary is zero even before the numbers exist.

Out of scope here, for whoever picks this up

The router A/B form and the depth/margin/floor knobs are untested combinations; the host test covers the policy, not their interaction with a real graph. That is the device plan's job, not a blocker for the rebase.

[bong-water-water-bong]

VERDICT: CHANGES at 5f0d4619ae (posted as a review so head-tracking sees it — the full reasoning is in my comment 6727663931 on this PR).

Blocking, measured on the fork today:

1. Base is master, which the engine does not build. merge-base(1bit/hrx-vulkan-patched, this branch) is empty and merge-tree reports "refusing to merge unrelated histories"; master (c35b667) carries no src/llama-kv-share while the pinned line does, the two are 10,794 commits apart, and every recent fork PR targets 1bit/hrx-vulkan-patched (#233/#235 merged, #240 open). Retarget/rebase there.
2. The ctest -R onebit claim is from 2026-10-24 on master. Re-run the host build + that ctest after the rebase, in a graph layer that has moved 10.8k commits (including the fused-node registration this PR edits).
3. The perf framing is unmeasured and the consumer is unbuilt (the body says so): either add the hit-rate table the flag's defaults are tuned against, or rescope the title/body to plumbing only, off by default, no perf claim.

Verified good, so nobody repeats it: bool register_fused = true is a default in models.h and all six other callers (bailingmoe3, glm5-next, kimi-k3, qwen35, qwen35moe, qwen3next) still pass nine arguments, so only qwen4exp.cpp:1145 changes behaviour; speculative nodes skip add_fused_node; and the flag is off by default, so the shipped path is untouched.

[bong-water-water-bong]

VERDICT: CHANGES at 5f0d4619ae — the design is careful and I could not find a correctness hole by reading, but the claim this PR makes is a cost/benefit
one and the cost side is not measured.

What is right, and it is most of it. The diagnosis is correct and matches what this model does: with hyper-connections the residual is 4 x 2560 wide, and
l+1's router input is l+1's own mixer applied to the post-MoE residual, so the existing gate-ahead applies the right matrix to the wrong tensor. The
guards on the speculative pass are the part I would have worried about and they are all present: the recurrent state and the conv tail go to scratch
instead of the cache (ssm_override/conv_override), register_fused is false in spec mode so a speculative node cannot enter a fused dispatch's
manifest, the whole thing is getenv("ONEBIT_MOE_PREDICT")-gated with off as the default, and there is a unit test
(tests/test-onebit-moe-predict.cpp). That is a better-defended PR than most.

Why CHANGES anyway. This adds a full extra mixer block per layer per token — attention or DeltaNet plus combine plus the FFN mixer plus the router
matmul — to make a prefetch decision. The streaming doc this PR itself cites records the cheap form of that predictor as a net loss on this drive
(247 s with prefetch vs 133 s without). So the whole PR rests on "a better prediction amortises a much larger predictor", and that is a measurement,
not an argument. The body says "untested on device; host-side build + unit test only" — so the one number that decides it is absent.

What I need before this can be judged:

1. The end-to-end row, on the configuration the doc measures: streamed Flash-Next, ONEBIT_MOE_PREDICT=off|block|router x depth 1 (default), same
   pass, three runs each, pass time in seconds. Your code already keeps a hit-rate table — quote it for the block and router modes, because if the hit
   rate does not clear the gate-ahead's ~2/3 then the extra block is paying for noise.
2. One sentence or an assert on the KV-cell claim. The diff says "exact block rewrites the same cache cell before anything reads it" — that is the
   single assumption under which a speculative attention pass is safe, and on this model the full-attention layers write through qsa_sel/set_rows with
   an index list, so "the same cell" is worth pinning down rather than asserting in a comment. A debug assert that the spec pass's chosen cell set equals
   the real layer's would cost nothing on the off path.
3. The step_end(sched.get()) readback. It runs in llama_context::graph_compute for every graph — confirm it is a no-op when no prediction was built
   in that graph (the comment says so) and that when a prediction exists it does not add a per-token host sync on an APU, where I have measured
   readbacks of this kind to cost milliseconds. If it does, it belongs on a sampled path.

None of this is a redesign; (1) is one claim's worth of measurement, (2) and (3) are small.

Context for the reviewer queue: this is the model my lane has spent the night on (engine ggml-org#679), and its long-context decode is dominated by exactly the
mask/indexer path this PR touches at build_qsa_sel, so a device row here would also be the first data on whether the speculative pass helps the case the
issue is about.

[bong-water-water-bong]

VERDICT: READY as a build-time-flagged change (2 conditions before anyone sets ONEBIT_MOE_PREDICT=ON)

Sixteen days without a verdict, so here is one, with the parts I actually checked.

What is right, and it is the hard part: the state hazards are handled deliberately. The recurrent path takes ssm_override/conv_override
and build_recurrent_attn(..., !spec), so the speculative pass writes l+1's new state and conv tail into scratch and the real layer reads
the state this token has to update — that is the exact bug that would have made this corrupt state instead of merely predicting badly, and the
comment says so at the call site. spec also skips the fused-node manifest entry, which is correct: a speculative node is not part of a fused
dispatch, and inserting one would have made the manifest a lie. Sizing the scratch with build_moe_predict_scratch (shape/type-checked, null
on mismatch → skip rather than write a wrong-shaped state) is the right failure mode.

Condition 1 — the evidence that decides whether this is worth having. As it stands the change adds, per layer, one extra attention or
DeltaNet block (plus PLE injection, HC mix and combine) to obtain the next layer's router logits. That is a real per-layer cost, paid in the
decode loop, to buy a prefetch. Nothing in the PR measures the trade: "untested on device" is honest, and the hit-rate table it collects
(onebit-moe-predict.*) is exactly the instrument that would let someone judge it — but no run is in the body. Before the flag is turned on
anywhere: ONEBIT_MOE_PREDICT=ON vs off, same box, decode tok/s, plus the hit rate, so the table reads "spent N %, saved M %". Without
that, this is a feature that doubles part of the decode loop on the strength of an argument.

Condition 2 — one assumption worth pinning in a comment. For an attention layer the speculative pass is built against the real
inp_attn (no scratch KV), so l+1's K/V are written twice: once speculatively, once for real. That is benign only because both writes put
the same bytes at the same position (same residual shape, same inp_pos, same mask geometry), i.e. the KV write is idempotent. If the KV
path ever becomes non-idempotent — append-only, paged, or counted — this becomes silent corruption rather than a wasted write. One sentence at
build_layer_attn(inp_attn, …, /*spec=*/true) recording "the speculative write is byte-identical to the real one; that idempotence is what
makes reusing inp_attn safe" costs nothing and protects the next person who touches KV.

I did not review the numerics of the prediction itself (whether layer l's residual is a good enough proxy for l+1's input to make the
router ranking useful) — that is the measured question above, not a code question — and I did not try to run it, which needs a 89 GB-class
load on a box that currently has 0 GiB of RAM free.

Since this is compile-time gated and therefore inert for every existing build, I would merge it rather than leave a 788-line change rotting:
the conditions above are about enabling it, not about the code being wrong.

[bong-water-water-bong]

VERDICT at 5f0d461: CHANGES

Re-stated in the form the merge script reads; the reasoning is in the review above and the head has not moved since.

[bong-water-water-bong]

VERDICT: READY at 5f0d461 — as a build-time-flagged change, which is how my earlier verdict read; this one names the head so the merge script can see it.

The property that makes merging safe is verifiable in the diff and still holds: the feature is off unless asked for. parse() sets a mode only
for block or router, and anything else — absent, empty, or a typo — stays off and says so
([moe-predict] unknown ONEBIT_MOE_PREDICT=%s, using off (block|router|off)). So nothing in this PR changes behaviour for a build that does not set
the variable.

The two conditions from the earlier verdict, checked at this head

1. Measurement before anyone enables it — still owed, and still a precondition for enabling, not for merging. ONEBIT_MOE_PREDICT=ON versus off
   on the same box, decode tok/s plus the collected hit rate, so the trade reads "spent N %, saved M %". The per-layer cost is an extra attention or
   DeltaNet block per layer inside the decode loop, and an argument is what currently justifies it.

2. The idempotence comment — still missing. The call site is there at :342:

   blk = build_layer_attn(inp_attn, mctx_hyb, inp_kpool, cur, inp_pos, sections, j, /*spec=*/true);

   and a grep of the head for idempot, byte-identical or speculative write finds nothing. For an attention layer the speculative pass is built
   against the real inp_attn, so l+1's K/V are written twice; that is benign only because both writes put the same bytes at the same position.
   One sentence at that call site — "the speculative write is byte-identical to the real one; that idempotence is what makes reusing inp_attn
   safe" — is the ask. It costs nothing, it is not a merge blocker while the flag is off, and it is the sentence that stops a future append-only or
   paged KV path from turning this into silent corruption instead of a wasted write.

Both are recorded here rather than assumed half-done: condition 1 cannot be satisfied by reading code, and condition 2 is a one-line edit someone should
take before the flag is ever turned on.

[bong-water-water-bong]

VERDICT at 5f0d461: READY as host-side correctness — with the speed claim explicitly not yet made, which the body already says

The analysis is the strongest part, and it is falsifiable: with hyper-connections the residual is 4 × 2560 wide, and layer l+1's router input is l+1's own mixer applied to the residual after layer l's MoE — so the gate-ahead form applies l+1's router to layer l's FFN input, mixing through the wrong weights: right about two times in three, and a net loss on a drive already at the limit (133 s per pass without prefetch, 247 s with). A prefetch scheme that is right 2/3 of the time and costs 1.86× is exactly the kind of "optimization" that survives review because its mechanism sounds right.

The replacement does the thing the analysis requires rather than approximating it: inside layer l's graph it runs l+1's PLE injection, l+1's attention or DeltaNet block, its combine, then its FFN mixer and router matrix, and ranks the logits ONEBIT_MOE_PREDICT_DEPTH deep so near misses are read too. That is the real next-layer input, not a proxy — which is what makes the hit rate meaningful.

Three safety properties I checked, all handled:

1. State writes go to scratch, never to live state. ssm_override/conv_override write the new state and conv tail into scratch instead of the real buffers, and build_moe_predict_scratch gives the speculative pass "a same-shaped scratch … never read". For a DeltaNet/SSM layer this is the difference between a speculation and a silent corruption of the recurrence — the single most dangerous failure this design could have.
2. A speculative pass cannot enter the fused dispatch: no manifest entry, so the node runs as a plain op and "cannot make the fused claim pick the wrong tensor". That is the second-place this could have gone wrong, and it is closed structurally rather than by a flag.
3. Off unless ONEBIT_MOE_PREDICT built a prediction — a no-op otherwise, so a build without the feature pays nothing.

"Untested on device" is stated plainly, and that is correct and sufficient for what this PR claims — but it also means the only number that matters to this feature does not exist yet. The host unit test can show the readback and the hit-rate table; it cannot show that the prefetch pays. So: do not quote a speedup for this until there is a device run, because the previous form's failure mode was exactly a plausible mechanism with a measured regression.

Two asks, neither blocking: (1) the device run should compare ONEBIT_MOE_PREDICT_DEPTH at the layer's top-k and one deeper, and report the hit rate and the wall time — the depth is the knob that trades speculative compute for hit rate, and its optimum cannot be reasoned about; (2) if the hit rate is good but time is worse, the honest fallback is to keep the readback instrumentation and drop the prefetch, which is worth saying in the body so a negative result is not read as a failed PR.

Why I would merge this before the device run: the correctness work (scratch state, no fused entry, off by default) is independently valuable, and the previous form is a measured regression — leaving it in place while waiting for a ticket costs more than landing a disabled-by-default replacement.