[bong-water-water-bong]

NPU_BATCH_CHECK runs the tokens sequentially, then reset()s and lets the batched pass run. The reset threw away the state the batched pass continues from, so for every chunk at start_pos > 0 the two sides ran from different states — the replay from an empty cache at position start, the batch from the caller's prefix — and the oracle reported a divergence that was not in the kernels. #1900's chunking (width 1 on a sealed pack) made it fire; it is latent for any prompt wider than the M=128 design.

The replay now snapshots the prefix at start_pos and restores it, so both sides start from the same state and the comparison stays exact for a chunked pass instead of being disabled for one. start_pos == 0 has no prefix (snapshot_state(0, …) is defined to refuse), and a snapshot that did not take is not a licence to compare against the replay's own state: both fall back to reset().

Measured before/after on identical inputs, device-free

~/scratch-packs/imp1519b (host GEMM, so the width is 1 and every id is its own chunk — i.e. every start_pos > 0 case), --ids "1 2 3 4 5", NPU_BATCH_CHECK=1, same pack and same ids, two builds:

chunk	before (reset())	after (snapshot/restore)
start=0	MATCH	MATCH
start=1	MISMATCH (0x411bf754 vs 0x409d210c)	MATCH
start=2	MATCH	MATCH
start=3	MATCH	MATCH
start=4	MATCH	MATCH

Only start=1 diverged: start=2 and 3 "matched" because the argmax happened to coincide at id 0 — which is why the oracle keeps the whole row and compares it bitwise, and why one green chunk says nothing about the next.

Fixes #1905. Refs #1900, #1904, #1781.

[bong-water-water-bong]

VERDICT at 8ebfc05: CHANGES

Checked at 8ebfc05 (Fixes #1905, q4nx_forward.cpp +9 −1). onebit_npu_forward_syntax builds. The snapshot is taken before the replay and restored after it, so the success path is right.

Blocking: a failed restore silently falls back to reset().

if (!(had_prefix && restore_state(oracle_prefix, start_pos))) reset();

When start_pos > 0 and either the snapshot or the restore fails, this empties the state the real batched pass then runs from. The oracle's comparison is false again (the #1905 MISMATCH), and the run's own output comes from an empty cache. The comment says "the row comparison below still applies", but it then compares a correct sequential replay against a batch computed from nothing. With start_pos == 0, reset() is right. With a real prefix that could not be kept, the only honest outcomes are refusing (as #1909 does, by name) or skipping the comparison and leaving the state alone. restore_state is documented to refuse a truncated or mismatched snapshot (:354), so this branch is reachable, not hypothetical.

Also: #1909 fixes the same #1905 and conflicts with this in q4nx_forward.cpp. It handles this exact case by refusing. I'd land #1909 and close this.

[bong-water-water-bong]

VERDICT: APPROVE at 8ebfc05 — this was a false MISMATCH in the instrument, which is the worst place for a bug of this shape.

What was wrong

NPU_BATCH_CHECK replayed the tokens sequentially and then reset() — which emptied the cache the replay had just filled and dropped the caller's prefix. So for any chunk at start_pos > 0, the sequential side ran from "one token at position start_pos from scratch" while the batched side continued the previous chunk: two different states compared against each other, and a MISMATCH reported that was not in the kernels.

That is the failure mode this whole oracle family exists to exclude — an instrument that reports failure for its own reason. Every chunked run (prefill included) would have been a guaranteed MISMATCH, and a real one would have been lost in the noise.

The fix

StateSnapshot oracle_prefix;
const bool had_prefix = start_pos > 0 && snapshot_state(start_pos, oracle_prefix);
if (!(had_prefix && restore_state(oracle_prefix, start_pos))) reset();

Snapshot → replay → restore, with reset() kept for the case that has nothing to keep. Both signatures match the header (snapshot_state(int tokens, StateSnapshot&) const, restore_state(const StateSnapshot&, int tokens)), and start_pos == 0 behaves exactly as before.

The fallback comment is the part I like: "a snapshot that did not take is not a reason to compare against the replay's own state: both fall back to reset()" — it refuses to compare two states that happen to agree because neither was restored.

The gap: nothing pins it

OPTS.files is q4nx_forward.cpp alone, so there is no arm. The discriminating case is cheap and exact: a NPU_BATCH_CHECK run at start_pos > 0 must report MATCH — it fails on the old code and passes on this one, which is the definition of an arm that carries the fix rather than the intention. Without it, a later refactor that reintroduces the reset() is green.

Until then the only evidence is reasoning plus this PR's own account of the symptom, and I have spent tonight reviewing four other PRs whose subject was a check that could not fail. Worth a follow-up even if this lands as is.

For my own work: this is the same oracle as engine#1813/#1781 gate 1, so the fix is load-bearing for the k=7 row identity I am waiting to measure — a false MISMATCH there would have been read as a kernel divergence.

[bong-water-water-bong]

Closing as superseded by #1909 (473fdf4), which fixes the same #1905 and does the two things this PR got wrong.

The review was right and the blocking point is the one that matters: my fallback was

if (!(had_prefix && restore_state(oracle_prefix, start_pos))) reset();

When start_pos > 0 and the snapshot or the restore fails, that empties the state the real batched pass then runs from — so the run's own output comes from an empty cache and the comparison goes false again. restore_state is documented to refuse a truncated or mismatched snapshot, so the branch is reachable, not hypothetical. #1909 refuses by name instead, which is the only honest outcome when the prefix you compared against cannot be put back. It also names the positions compared (positions=%d..%d), which my version did not.

What this PR had that is worth keeping is a measurement, and I have posted it on #1909: on imp1519b (host GEMM, width 1, so every id is its own chunk) only start=1 diverged before the fix — start=2 and 3 "matched" because the argmax happened to coincide at id 0. That is the case for #1909's better log line: a green chunk proves nothing, and the old output did not show which positions it covered.

No duplicate fix from me; #1909 is the one to land.

[bong-water-water-bong]

VERDICT: READY at 8ebfc05 — a false-positive oracle, which is the expensive direction: it accuses the kernels of a defect the harness invented.

// engine#1905: keep the state the batched pass continues from. This used to `reset()`, which emptied the cac[he the]
// replay had just filled AND dropped the caller's prefix -- so for every chunk at start_pos > 0 the two side[s started]
// from different states ("one token at position start from scratch" vs "position start continuing the previo[us]
// chunk") and the oracle reported a MISMATCH that was not in the kernels. Snapshot the prefix, replay, resto[re]

Three things are right here:

• the mechanism is named precisely: reset() did two wrong things at once — it emptied the cache the replay had just filled, and it dropped the caller's prefix. Either alone produces a mismatch; together they make the two sides differ by construction for every chunk with start_pos > 0;
• the disjunction is spelled out in the two readings it compared ('one token at position start from scratch' vs 'position start continuing the previous chunk') — which is exactly the sentence that would have found this in an hour instead of a day;
• the fix is the invariants, not the symptom: snapshot the prefix, replay, restore. Nothing about the comparison was weakened, so the oracle keeps its teeth.

The general lesson worth keeping

A harness that mutates the state it is comparing reports failures that belong to the harness. This is the twin of the certifying fixture (#1863) and of the vacuous arm (#1805): one asserts correctness it cannot see, this one asserts a defect that does not exist — and both send the reader to the wrong file. The difference is that a false positive also costs credibility for every real mismatch the oracle reports afterwards.

Nothing to change.