From 93f15c6361ca94efc8cd8a13a900360e37b181e2 Mon Sep 17 00:00:00 2001 From: bong-water-water-bong Date: Wed, 11 Nov 2026 10:57:10 -0400 Subject: [PATCH 1/4] window11 --compare accepts the decode head on the host as the one declared row difference (Refs #1785, #2191) A run with NPU_HEAD_HOST prints 'dec HEAD M=1' where a device-head run prints 'dec HEAD M=16'. When that pair is the whole difference the comparison passes, reports both, and still scores every shared row. Co-Authored-By: Claude Fable 5.2 --- docs/npu-plan.md | 7 ++++++- scripts/npu-window11.sh | 45 +++++++++++++++++++++++++++++++++++++++-- 2 files changed, 49 insertions(+), 3 deletions(-) diff --git a/docs/npu-plan.md b/docs/npu-plan.md index f69262d33..f022ad64b 100644 --- a/docs/npu-plan.md +++ b/docs/npu-plan.md @@ -152,7 +152,12 @@ path untouched. `scripts/npu-window11.sh` is the gate the plan change was validated with: three fixed greedy prompts plus the prof table, compared between a run with the plan and one without. `--compare` refuses to run without -`--tol PCT` or `--control C1.log C2.log`; `--self-test` checks that policy. +`--tol PCT` or `--control C1.log C2.log`; `--self-test` checks that policy. The prof rows of the two +runs must be the same set of (phase, role, M), with one declared exception: a run with the decode lm head on +the host (`NPU_HEAD_HOST`, [npu.md](npu.md)) prints `dec HEAD M=1` where a device-head run prints +`dec HEAD M=16`. When that pair is the whole difference the comparison passes, prints both head rows and +which side each is on, and scores every shared row as usual; any other missing or added row still fails +(engine#1785). Tests: `tests/npu_plan_test.cpp` (ctest `npu_plan`) pins the loader and the refusals; `tests/npu_plan_generate_test.py` (ctest `npu_plan_generate`) runs the generator and `--check` on a diff --git a/scripts/npu-window11.sh b/scripts/npu-window11.sh index 36000691c..72e26d95f 100755 --- a/scripts/npu-window11.sh +++ b/scripts/npu-window11.sh @@ -31,6 +31,8 @@ # so two runs can be compared without trusting a human to eyeball two terminals. The gate a change has to # pass is: 3/3 transcripts byte-identical, the prof table's rows (phase, role, M) identical, and no row's # ms/launch outside the tolerance. +# One row difference is a declared mode and passes: `dec HEAD M=1` (the lm head on the host, NPU_HEAD_HOST) +# against `dec HEAD M=16` (on the device), when that pair is the whole difference (engine#1785). # # The tolerance is NOT a constant: engine#834 measured the same binary against itself on this box and the rows # moved up to 7.7 %, so a 2 % assertion cannot be evaluated here at all. `--compare` therefore refuses to run @@ -229,8 +231,26 @@ if ha or hb: f"{sum(1 for t in ka if t.startswith('[qkv-dump]'))} qkv dumps)") rows_a, rows_b = set(pa), set(pb) -if rows_a != rows_b: - print(f"FAIL prof rows differ: only in A {sorted(rows_a - rows_b)}, only in B {sorted(rows_b - rows_a)}") +# The one row difference that is a declared mode and not a regression (engine#1785, #2191): the decode lm +# head computed on the host (NPU_HEAD_HOST) prints `dec HEAD M=1`, one host pass per token, where the device +# head prints `dec HEAD M=16`, one row per launch. The two rows time different things, so they are reported +# side by side and not scored against each other; the logits are the same numbers either way, which the text +# rows above already assert. It is accepted only when that pair is the WHOLE difference: any other row +# missing or added still fails, and so does a log with both head rows or with neither. +HEAD_DEV, HEAD_HOST = ("dec", "HEAD", "16"), ("dec", "HEAD", "1") +only_a, only_b = rows_a - rows_b, rows_b - rows_a +head_moved = None +if (only_a, only_b) == ({HEAD_DEV}, {HEAD_HOST}): + head_moved = ("A", HEAD_DEV, pa, "B", HEAD_HOST, pb) +elif (only_a, only_b) == ({HEAD_HOST}, {HEAD_DEV}): + head_moved = ("B", HEAD_DEV, pb, "A", HEAD_HOST, pa) +if head_moved: + dev_side, dk, dp, host_side, hk, hp = head_moved + print(f"prof rows: {len(rows_a & rows_b)} identical (phase, role, M), plus the decode head: on the device in " + f"{dev_side} (HEAD M=16, {statistics.median(dp[dk]):.3f} ms per launch) and on the host in {host_side} " + f"(HEAD M=1, {statistics.median(hp[hk]):.3f} ms per token); reported, not scored against each other") +elif rows_a != rows_b: + print(f"FAIL prof rows differ: only in A {sorted(only_a)}, only in B {sorted(only_b)}") fail = 1 else: print(f"prof rows: {len(rows_a)} identical (phase, role, M)") @@ -341,6 +361,27 @@ with open(pb, "w") as f: rc, out = run(pa, pb, "--tol", "10") check("statistic in force: median-of-ratios" in out and "median-of-ratios" in out, "two rounds per row: the paired statistic is named and printed", out) + +# The decode head on the host (engine#1785): `dec HEAD M=1` against `dec HEAD M=16` is the one accepted row +# difference, in either order, and only when it is the whole difference. +hd, hh = ("dec", "HEAD", "16"), ("dec", "HEAD", "1") +dev = write("dev.log", {**base, hd: 3.100}) +host = write("host.log", {**base, hh: 9.000}) +rc, out = run(dev, host, "--tol", "10") +check(rc == 0 and "on the device in A" in out and "on the host in B" in out and "not scored" in out, + "device head against host head passes and says which side is which", out) +rc, out = run(host, dev, "--tol", "10") +check(rc == 0 and "on the device in B" in out and "on the host in A" in out, "the same in the other order", out) +rc, out = run(dev, write("host_slow.log", {keys[0]: 2.600, keys[1]: 5.000, hh: 9.000}), "--tol", "10") +check(rc == 1 and "outside tolerance" in out, "a host-head log is still scored on every shared row", out) +rc, out = run(dev, write("host_extra.log", {**base, hh: 9.000, ("dec", "Z", "16"): 1.0}), "--tol", "10") +check(rc == 1 and "prof rows differ" in out, "a host-head log with another row added still fails", out) +rc, out = run(dev, write("nohead.log", base), "--tol", "10") +check(rc == 1 and "prof rows differ" in out, "a log with no head row at all still fails", out) +rc, out = run(dev, write("both.log", {**base, hd: 3.100, hh: 9.000}), "--tol", "10") +check(rc == 1 and "prof rows differ" in out, "a log with both head rows still fails", out) +rc, out = run(dev, write("host8.log", {**base, ("dec", "HEAD", "8"): 9.000}), "--tol", "10") +check(rc == 1 and "prof rows differ" in out, "a head row at any other M is not the host head", out) print() print("self-test: " + ("PASS" if not failures else "FAIL")) sys.exit(1 if failures else 0) From 27b3d9212d62d0d4b2091f2f04973c3bdde2035d Mon Sep 17 00:00:00 2001 From: bong-water-water-bong Date: Wed, 11 Nov 2026 09:57:35 -0400 Subject: [PATCH 2/4] npu: a hybrid model's full-attention layers no longer re-dequantise q/k/v/o on every decode token (Refs #1785) load_layer() filled wq_/wk_/wv_/wo_ each token although their only reader, q4nx_assemble_B, is skipped once the layer's packed QKV and O images are resident. Skipped when both are; NPU_ATTN_RELOAD=1 keeps the old behaviour. Qwen3.5-2B: load_layer 12-13 ms -> 0.04 ms per token, logits byte-identical. Co-Authored-By: Claude Fable 5.2 --- docs/npu.md | 13 +++++++++++++ npu/forward/src/q4nx_forward.cpp | 12 +++++++++++- 2 files changed, 24 insertions(+), 1 deletion(-) diff --git a/docs/npu.md b/docs/npu.md index baedf4d8e..8256de551 100644 --- a/docs/npu.md +++ b/docs/npu.md @@ -2058,6 +2058,19 @@ On FastFlowLM's Qwen3.5-2B-NPU2 (IM 6144, H 2048, a 151 MB tail that the earlier run in one ticket measured decode at 362–367 ms per token with the cache on (both arms) against 485–520 ms with it off in the quieter of the two off arms and 478–707 ms in the other, same text in all twelve runs (engine#1785). +The full-attention layers of such a model had the same reload, for their own four buffers: `load_layer()` +dequantised `q`, `k`, `v` and `o` into `wq_`/`wk_`/`wv_`/`wo_` on every decode token, and the only reader of +those buffers (`q4nx_assemble_B`) is skipped once the layer's packed QKV and O images are on the device. The +reload is now skipped when **both** images are resident (the batched path guards its O assemble with the QKV +key, so a half-cached layer still loads everything) and the run is not `NPU_INFER_HOST_GEMM`. +`NPU_ATTN_RELOAD=1` keeps the old per-token reload. On Qwen3.5-2B-NPU2 (6 full-attention layers of 24), an +A B B A run in one ticket with `NPU_PHASE_PROF=1`, 46 decode tokens per arm, medians: `load_layer` 13.0 and +11.6 ms per token with the reload against 0.04 and 0.03 ms without; the layer loop's host share 23.9 and 22.2 +ms against 10.6 and 11.1 ms; device wait unchanged (240.7 to 247.3 ms in all four). All 144 logits dumps of +the other three arms are byte-identical to the first arm's, so this removes work and changes no value. The +whole token moved from 347 and 345 ms to 332 and 341 ms, which on that box (load 20 to 27) is inside the +spread between two arms of one mode; the host share is the number to read (engine#1785). + ### The prefix cache's snapshot: two kinds of plane `snapshot_state`/`restore_state` (`npu/forward/include/q4nx_forward.h`) hand a prefill's state to the next diff --git a/npu/forward/src/q4nx_forward.cpp b/npu/forward/src/q4nx_forward.cpp index 9dca11690..e74b47369 100644 --- a/npu/forward/src/q4nx_forward.cpp +++ b/npu/forward/src/q4nx_forward.cpp @@ -2041,12 +2041,22 @@ bool Q4nxNpuForward::load_layer(int l) { // copies of the big projections are never needed: skipping them here is the // difference between a first token that dequantises the whole 17.9 GB model and // one that opens 13.4 GB of images. The norms and every small member still load. + // The four attention f32 buffers (wq_/wk_/wv_/wo_) are read only by q4nx_assemble_B(), which step() and + // step_batch() skip once this layer's packed QKV (key 0) and O (key 1) images are on the device -- the + // same argument as ffn_tail_cached() for the MLP and proj_cached for a GDN layer. A dense model never + // gets here on a cached layer (step()'s `dense_weights_cached`), but a hybrid one does on every token: + // its full-attention layers re-dequantised q, k, v and o each decode step, 17.5 ms of a 285 ms token on + // Qwen3.5-2B (6 of 24 layers), for buffers nothing read (engine#1785). BOTH keys must be cached: + // step_batch() guards its O assemble with the QKV key, so a half-cached layer still loads everything. + // NPU_ATTN_RELOAD=1 keeps the old per-token reload, for the A/B. + static const bool attn_reload = getenv("NPU_ATTN_RELOAD") != nullptr; + const bool attn_cached = !attn_reload && !host_gemm_ && xd_ && xd_->weight_cached(l, 0) && xd_->weight_cached(l, 1); if (is_conv_layer(l)) { // LFM2 conv layer: two projections where an attention layer has four. Same f32-until-packed rule. LhScope _t_p(LH_LOAD_PROJ); if (!deq(&lw->shortconv_in_proj_weight, wqkvlin_.data(), 3 * H, H)) return false; if (!deq(&lw->shortconv_out_proj_weight, woutlin_.data(), H, q4nx_pad_k(H))) return false; - } else if (!xd_->pb16i()) { + } else if (!xd_->pb16i() && !attn_cached) { if (!deq(&lw->q_proj_weight, wq_.data(), NH * hd * (attn_gate_ ? 2 : 1), kpad_)) return false; // q|gate rows if (!kv_shared) { From cd491e26afc74f8c420843f48a69c9dbf3446645 Mon Sep 17 00:00:00 2001 From: bong-water-water-bong Date: Wed, 11 Nov 2026 13:00:46 -0400 Subject: [PATCH 3/4] docs: the P4 gate's logits line states the prompts that were run (was #2197, Refs #803) Co-Authored-By: Claude Fable 5.2 --- docs/npu-p4-target-split.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/npu-p4-target-split.md b/docs/npu-p4-target-split.md index 960bf0327..0fffee654 100644 --- a/docs/npu-p4-target-split.md +++ b/docs/npu-p4-target-split.md @@ -236,7 +236,7 @@ What is still not wired, and stated as such: `gdn_step`/`attn_step` dispatch nee ## 4. Gate (unchanged from #803) 1. `window11`: 3 greedy prompts, text identical to the current binary. -2. Logits KLD **0** on 8 prompts (the same prompts the hw-gate uses), same model dir and kernel set. +2. Logits KLD **0** on 8 prompts, same model dir and kernel set in both arms. (This line used to ask for the hw-gate's prompts; the row below was run on eight fixed completions and says so.) **Measured (2026-11-11, on the NPU, under `npu-hold` tickets).** Baseline: the binary at `540e1fd` (before the decode-family move), run twice as the control. Candidate: `main` at `d0fa51c` (with #2166). From 16c40c0db1d4d68369dbea482657d36ea4202d5b Mon Sep 17 00:00:00 2001 From: bong-water-water-bong Date: Wed, 11 Nov 2026 12:21:56 -0400 Subject: [PATCH 4/4] docs/npu-plan: the Not-yet list says what P5, P6 and P7 landed and what was not built (Refs #799) The P6 entry said the tau=0.6 acceptance rows were missing; they are on the leaderboard. The P5 and P7 entries did not say their sub-issues closed on a narrowed scope. Co-Authored-By: Claude Fable 5.2 --- docs/npu-plan.md | 17 ++++++++++++++--- 1 file changed, 14 insertions(+), 3 deletions(-) diff --git a/docs/npu-plan.md b/docs/npu-plan.md index f022ad64b..be2cce852 100644 --- a/docs/npu-plan.md +++ b/docs/npu-plan.md @@ -194,6 +194,9 @@ layers[0] {"index": 0, "role": "gdn", "qkv": {"K": 5120, "N": 10240, "fused" ## Not yet +The tracker (#799) closes with P4 (#803). What follows is the part of the plan that was **not built**, so that closing +it does not read as "all seven phases shipped": each entry says what landed, what did not, and why. + - **P4** (#803): **A1 landed** in #808 (`0a2cbb9`) — the `NpuTarget` interface (`npu/target/target.h`), the XDNA target behind it, and `open()` consuming the plan instead of duplicating the derivation. The step-by-step record is [npu-p4-target-split.md](npu-p4-target-split.md) §3, so this list does not snapshot it, but the two ends that touch @@ -209,9 +212,17 @@ layers[0] {"index": 0, "role": "gdn", "qkv": {"K": 5120, "N": 10240, "fused" ([npu-p4-target-split.md](npu-p4-target-split.md) §4). `XdnaTarget::load_block` remains a named no-op, so weight residency and decoding stay as they were — the cache fills lazily inside `gemv`, and turning that into an explicit block step is an optimisation whose evidence is a tokens/s row, not a structural move. -- **P5** (#804): Hexagon target (t2 gemv on HVX/HMX, QEMU v73 oracle) has no engine rows yet. -- **P6** (#805): the shared drafter protocol is on main; tau=0.6 and Hexagon acceptance rows are still missing. -- **P7** (#806): the `kv` contract above is in the plan and its arithmetic is gated in CI (`ctest -R npu_kv_int8`, +- **P5** (#804, closed 2026-11-07 on the scope the owner set: a toolchain check): the t2 GEMV is bit-exact in QEMU + v73 on the container's own layer-0 tensors and has a cycle model, and `tests/npu_t2_target_gate.py` (#1272) holds + any t2 target to the pinned accumulation order. **Not built:** a Hexagon `NpuTarget` in the engine (`npu/target/` + holds `xdna` only, and `--npu-target hexagon` is refused by name), silicon timing of our kernel (no board on the + fleet), and the decoder layer composed on the DSP. +- **P6** (#805, closed 2026-11-07): the shared drafter protocol is on main and measured on XDNA at both gates, 8 + prompts x 64 tokens, text identical 8/8 — tau 0.6 (the default): hit rate 78.77 %, depth-1 acceptance 78.34 %, 1.83 + tokens per pass; tau 0: 46.69 %, 75.14 %, 2.87 ([npu-leaderboard.md](npu-leaderboard.md), the `27b-mtp-*` rows). + Leaf fill was rejected with a test (`tests/mtp_draft_test.cpp` [16]) rather than built. **Not measured:** acceptance + on a second target, because there is no second target (P5). +- **P7** (#806, closed 2026-11-07 with the refusal in place, the writer not built): the `kv` contract above is in the plan and its arithmetic is gated in CI (`ctest -R npu_kv_int8`, which needs no device: see below), and the **target-side refusal** is enforced as well (`npu_target_caps`, so an `int8` plan cannot be run as f16 by any target, including P5's Hexagon). Still open, in the order it can be taken: the cache **writer/reader** in the