Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 21 additions & 3 deletions docs/npu-p4-target-split.md
Original file line number Diff line number Diff line change
Expand Up @@ -126,9 +126,11 @@ proved by the byte-exact gate rather than assumed. It is listed in §5.
the artifact, refuses by name, and owns `init_elf`; the forward no longer creates an `I8Ctx` for the
prefill set. The device is passed at construction (`XdnaTarget(xrt::device*)`), which keeps
`NpuTarget::open(plan, kernels_dir, family, err)` exactly as documented here. **Done** (#1380).
- **still in the forward, and therefore still owed**: the M=16 decode family and the lm-head context.
Moving them needs only the setup state that `configure(Setup)` already carries (dirs, family, plan,
layout flags), which is why that struct exists.
- **the lm-head context and the M=16 decode family are the target's** too: `XdnaTarget::open_head()`
(#1458, #1478) and `XdnaTarget::open_decode_family()` (#2166, A2b). The forward still names the decode
designs (`open_decode_family_local()`, from the plan's `decode.dirs`) and the target opens and configures
the M=16 contexts. What remains of context setup in the forward is the optional wide and narrow sealed
families (`load_family` in the same function), which it still configures field by field.
- `load_block(first_layer, n_layers, err)` — **not started, and its premise is now measured, which changes
what it is for.** It was described here as taking over "`load_layer()`'s packing half (`wcache_` fill +
pb16i image load)"; at this tree that half is **already the target's**: `wcache_` lives in
Expand Down Expand Up @@ -235,6 +237,22 @@ What is still not wired, and stated as such: `gdn_step`/`attn_step` dispatch nee

1. `window11`: 3 greedy prompts, text identical to the current binary.
2. Logits KLD **0** on 8 prompts (the same prompts the hw-gate uses), same model dir and kernel set.

**Measured (2026-11-11, on the NPU, under `npu-hold` tickets).** Baseline: the binary at `540e1fd`
(before the decode-family move), run twice as the control. Candidate: `main` at `d0fa51c` (with #2166).
Both built against the same runtime. Model `ZR1-1.5B-npu-q4km-v2` with its 8 kernels (4 roles at M=16),
8 fixed prompts through `/v1/completions` (so both arms tokenise the same text), 8 greedy tokens each,
`--dump-logits` on, compared with `tools/logits_kld.py`:

| comparison | positions | mean KLD | max KLD | dumps byte-identical | texts identical |
|---|---|---|---|---|---|
| baseline run 1 vs run 2 (control) | 64 | 0.000000 | 0.000000 | 64 of 64 | 8 of 8 |
| baseline vs `main` | 64 | 0.000000 | 0.000000 | 64 of 64 | 8 of 8 |

The dumps are the same bytes, so the KLD is 0 exactly, not to a tolerance. Two limits: the prompts are
eight fixed completions, not the hw-gate's list, and the model is the 1.5B one, because the 27B container
did not fit above the box's memory floor that day; the 27B row of this gate is item 1's (window11, three
texts identical across the same two binaries, on #803).
3. `NPU_PROFILE=1` profile table: every row within 2 % of the pre-split binary, same row count.
4. `ctest` green; `--npu-target xdna` is the default and an unknown target name is refused by name.
The flag is `forward-serve`'s (the per-op route's own command) and `serve` carries it through to that child,
Expand Down
18 changes: 8 additions & 10 deletions docs/npu-plan.md
Original file line number Diff line number Diff line change
Expand Up @@ -196,16 +196,14 @@ layers[0] {"index": 0, "role": "gdn", "qkv": {"K": 5120, "N": 10240, "fused"
and two of the three context sets are now **the target's**: the M=128 prefill ELF set in
`XdnaTarget::open_prefill()` (#1380) and the lm-head context in `XdnaTarget::open_head()` (#1458, #1478) — the
forward calls `open_head()` (`q4nx_forward.cpp:1357`) and then asks the target for the context
(`head_ctx()`), rather than building it. **The M=16 decode family is still the forward's**, and that is the
correction to an earlier version of this paragraph: the target allocates the memory
(`XdnaTarget::make_ctx16()`), but the forward still decides which decode contexts exist and configures each one
field by field — `MD`, `KD`, `ND`, `blocked_b`, `pb16i`, `t2s`, the plane count
(`q4nx_forward.cpp:1306-1330`) — and `docs/npu-p4-target-split.md` §3 says the same in as many words ("still
in the forward, and therefore still owed: the M=16 decode family"). What is left on A2 is therefore two things,
not one: **A2c** moves that decode-family setup behind the target (it needs nothing the setup state does not
already carry), and `XdnaTarget::load_block` remains a named no-op, so weight residency and decoding stay as
they were — the cache fills lazily inside `gemv`, and turning that into an explicit block step is an optimisation
whose evidence is a tokens/s row, not a structural move.
(`head_ctx()`), rather than building it. **The M=16 decode family is the target's as well** (#2166, A2b:
`XdnaTarget::open_decode_family()`; the forward names the designs), so all three context sets are; the optional wide and narrow sealed families are
still set up by the forward. The P4
gate's two output rows are measured: window11 text identity on `bonsai2-27b-NPU` (on #803) and logits KLD 0 on
eight prompts, with every logits dump byte-identical before and after the move
([npu-p4-target-split.md](npu-p4-target-split.md) §4). `XdnaTarget::load_block` remains a named no-op, so weight
residency and decoding stay as they were — the cache fills lazily inside `gemv`, and turning that into an
explicit block step is an optimisation whose evidence is a tokens/s row, not a structural move.
- **P5** (#804): Hexagon target (t2 gemv on HVX/HMX, QEMU v73 oracle) has no engine rows yet.
- **P6** (#805): the shared drafter protocol is on main; tau=0.6 and Hexagon acceptance rows are still missing.
- **P7** (#806): the `kv` contract above is in the plan and its arithmetic is gated in CI (`ctest -R npu_kv_int8`,
Expand Down
Loading