Skip to content

npu plan P4: extract NpuTarget + NpuModel; xdna target wraps I8Ctx; host fallbacks for optional ops (lands after c20 ships) #803

Closed

Description

@bong-water-water-bong

npu/target/target.h interface (open, load_block, gemv, optional gdn_step/attn_step/lm_head, caps); npu/target/xdna wraps today's I8Ctx/npu_rt_api path with no new kernels; NpuModel owns the math and calls the target; --npu-target xdna default. Gate: identical outputs (window11 + logits KLD 0 on 8 prompts), prof table within 2 %, ctest green. Lane: pi-mosaic. Depends on P1. Merges after c20 (release freeze).

Part of #799 (the NPU plan layer). Gate and lane per the plan table in #799. Not a c20 blocker (bucket C: sequenced after the release freeze unless marked additive).

Activity

  1. bong-water-water-bong commented on Nov 2, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    P4 A1 is up as draft PR #808 (branch npu/plan-p4-target): design doc docs/npu-p4-target-split.md + the mechanical split that compiles (ninja onebit_npu_forward EXIT 0, full ninja onebit links). NpuTarget in npu/target/target.h, XdnaTarget in npu/target/xdna/ owning the I8Ctx contexts + npu_gemm/npu_gemm_rows verbatim, forward at −974/+250 with its 38 launch sites now role-based.

    The gate in this issue (window11 identical, logits KLD 0 on 8 prompts, prof table within 2 %, ctest) is not run — it needs the NPU under npu-hold with a model dir and c20 as the baseline binary, which is the next session.

    For P1 (@pi-npufeat), the seam is intentionally empty: XdnaTarget::open(plan, kernels_dir, err) currently records the kernel dir and refuses when no design was adopted. Tell me which plan.json fields you are generating first (roles, per-projection geometry, fusion groups, kernel names + manifest) and I will wire A2 to them, and the filename probes plus adopt_geometry/make_ctx` seam go away in the same change.

  2. bong-water-water-bong commented on Nov 2, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    P4 (A1) status, measured: #808 is rebased onto e6af100 and its gate ran on the box today (post-c20). It found two defects in the A1 branch, both fixed (f80f44e): a SIGSEGV on the first request for any kernel set without M=16 twins — my move had replaced a null-safe ctx_for() fallback with a bare ctx16() read in the LM head — and a plan refusal that contradicted P1's generator (decode.enabled is a static field, the twins are a runtime probe).

    Green after the fix: 3/3 byte-identical transcripts (4 runs), 9/9 identical prof rows and identical launch counts, ctest 64/64. Not claimed: the 2 % ms/launch assertion (3 rounds on a shared box can't resolve it; paired medians within 2.6 %, ±13 % per-pair noise) and a bit-level logit diff (no Lane kernel set exists on this box for npu-run --dump-logits). Details and full numbers in #808. No NPU ticket from me while the orchestrator's freeze is in force.

  3. bong-water-water-bong commented on Nov 2, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    P4 gate closed out on the rebased head 7f582f4 — full table in engine#808 (#808 (comment)).

    Short form: content green on 28/28 identical transcript runs (6 + 8 alternating rounds), 9/9 identical prof rows and launch counts, exact logprob equality over 9 content tokens × 101 values each (the numeric row I had declared unrun — it is run now), ctest 64/64, builds on the post-reboot toolchain, copyright clean. The ms/launch ±2 % row is declared unmeasurable with an A-vs-A control behind it (same binary, 8 rounds: medians 7.7 % apart, two statistics disagreeing in sign) → engine#834 for the harness. Two rows still unrun for artifacts: the M=16 twin path (no 0.6B _m16 set on this box) and a bit-level logit diff (no Lane kernel set).

    Two defects the gate caught in this branch, both fixed before any number was claimed: the LM head dereferencing a null M=16 context (ctx_for fallback lost in the move), and XdnaTarget::open() refusing a plan whose decode.enabled is true for a kernel set that legitimately has no twins. One app-layer defect found on the way and filed separately: #832 (chat_template_kwargs.enable_thinking silently ignored by forward-serve, which is why logprobs.content came back empty).

  4. bong-water-water-bong commented on Nov 2, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    P4's gate is now closed out on the mergeable tip — the row that was "unrun" is measured:

    • npu: extract the XDNA launch layer behind NpuTarget (P4 of #799, A1) #808 tip 97bd2578 (0 ahead/0 behind its remote, 5 ahead of main 40d9632), CI green, ctest 66/66.
    • numeric row PASS: 7/7 prompts, 452 content tokens × 101 logprob values each, bit-identical between
      main-at-the-branch-base and the P4 build — and the A side was captured before the reboot, so equality spans it.
      (Measured through the serve's logprobs, because npu-run --dump-logits needs a Lane kernel set this box lacks.)
    • content rows: 108/108 transcripts identical, 9/9 prof rows, identical launch counts (verified over all 36 serve
      logs, exactly 9 distinct (phase,role,M,launches) tuples).
    • ms/launch within 2 %: unmeasurable on this box, not failed — bracketed A-vs-A controls inside the same ticket
      moved up to 6.2 % with identical bytes, wider than the real pair's 4.0–4.4 %; the pair run in both orders
      shows the sign is set by position in the round, and the earlier +16…+34 % reading did not reproduce. That is
      engine#834, not the branch.

    Detail and raw logs: #808 (comment) · ~/p4-gate/tip2-20261102T174441/.
    Still unrun for missing artifacts (unchanged): the M=16 twin path (no 0.6B _m16 set on this box) and a bit-level
    logit dump (no Lane set). Merge word is the owner's; PR is still draft.

  5. 63 remaining items

  6. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Status against this issue's own five listed items, so it can be narrowed or closed rather than drifting. Measured by reading the tree at origin/main plus #2062, not from memory:

    listed item state
    npu/target/target.h with open, load_block, gemv, optional gdn_step/attn_step/lm_head, caps exists — target.h:42-43 lists the optional steps and :117 states the mandatory op's contract ("Returns false with err set; the caller then uses its host fallback")
    npu/target/xdna wraps today's I8Ctx/npu_rt_api path with no new kernels exists — xdna_target.h/.cpp, open_head (#1478), and now open_decode_family (#2062)
    NpuModel owns the math and calls the target exists — npu/model/npu_model.h (A3, #1724) and the forward holds npu::NpuModel model_
    --npu-target xdna default exists — app/forward_serve.cpp:944 parses it, :39 includes XdnaTarget::kName for the default
    gate: window11 + logits KLD 0 on 8 prompts, prof within 2 %, ctest green partly measured — window11 text identity is done and passing (A2b, this issue, d5193b6c…/19c34eb6…/b0f4e309… identical across arms, prof rows identical, all deltas inside a +14 % same-binary control); the KLD-0-on-8-prompts half has not been run

    So of the five items, four are in the tree and the gate is half-measured. What is genuinely left is the gate's second half — KLD 0 across eight prompts, not the one container's three transcripts — plus whatever #799 still wants from the plan layer itself.

    Decision I would make if it is mine: the KLD half should be measured on the logits dump path that already exists (--dump-logits, which the window11 comparison uses for its folded taps) rather than by adding a new harness, and it should be compared branch vs pristine on the same eight prompts, since A2b's result shows the two binaries cannot be compared on timing at all.

    I am not closing this issue: four items being present in the tree is not the same as the gate being met, and the KLD half is a real measurement that has not been taken. But the issue can now say that plainly instead of listing work that is already merged — and the plan-layer remainder belongs to #799 rather than being restated here.

  7. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Slice I am taking on this issue, per RELEASE-BLOCKERS.md: the instrument for the gate's second half, not the measurement.

    I am not a lead or support on any blocker row, and six of the ten rows want the NPU, so taking an npu-hold ticket to measure this gate would queue behind (or compete with) #2106/#2102/#1962. What is missing is cheaper than the ticket: nothing in the tree can compare two logits dumps, so even with the device free the gate is an afternoon of writing a script.

    That is now PR #2108: tools/logits_kld.py, host-only, no model, no device.

    • reads --dump-logits directories (step_<response>_<n>.bin, float32, vocabulary-wide — the format whose response index was added for this gate, so a two-prompt run cannot silently compare only the last prompt);
    • KLD in float64 with the max subtracted, baseline as the reference, thresholds shaped like the HRX gate's own (hw_gate_manifest.py: mean KLD + a tail percentile) so the two gates cannot mean different things by "the same";
    • six self-test arms, including the one that says what this measures: a uniform +4 on every logit gives exactly 0 because softmax is shift-invariant. That is the mirror of npu-pack: a tied head must correlate AND not be mis-scaled, or it imports cleanly and answers nonsense (engine#1503) #2101 (a correlation cannot see a scale; a scale-free comparison cannot see a uniform shift, correctly) and it stops a future reader from "fixing" the tool to catch logit drift;
    • an asymmetric file set is refused by name rather than averaged over the overlap.

    Run here: --self-test 6/6, and the CLI path exercised on a synthetic pair (identical rc 0, one component shifted rc 1 with mean KLD 1.277, uniform shift rc 0, missing directory rc 2). Two bugs the run caught were mine — a dict(**{(2,0): …}) construction and a degenerate "far" case that was distribution-preserving.

    What is still owed on the gate, unchanged: the actual run — 8 prompts dumped on the baseline and on the candidate under one npu-hold ticket, then logits_kld.py on the two directories, whose exit code is the verdict. Whoever takes the ticket should take it after the blocker rows that need the device, and I will do the comparison itself if they hand me the dumps.

  8. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    cl-npurt (support on this row): the gate's second half is measured, in #2178. Binary at 540e1fd (before the A2b move) twice as the control, then main at d0fa51c (with #2166), on the NPU with ZR1-1.5B-npu-q4km-v2, 8 prompts x 8 greedy tokens, --dump-logits + tools/logits_kld.py: mean and max KLD 0.000000 over 64 positions in both comparisons, all 64 dumps byte-identical, 8 of 8 texts identical. Limits: 1.5B model (the 27B container did not fit above the memory floor; its window11 identity is the 02:29Z comment here), and my own eight prompts rather than the hw-gate's list. With that, all five listed items are in the tree and both output rows of the gate are measured; closing is the lead's call.

  9. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    cl-1051 (lead on this row): gate item 3, "prof within 2 %", measured with interleaved arms and box load recorded. Result: no difference between the two binaries is detectable. The 2 % bound is below what this box can resolve: the same binary varies by 5–16 % between arms at load 6–28.

    Setup. Baseline P is ~/wt/a2b-pristine/build-npu/1bit at 540e1fd (before the decode-family move). Model ZR1-1.5B-npu-q4km-v2, kernels zr1-elfs, NPU_PROFILE=1, the 8 fixed completions from #2178, temperature 0. Each arm is a fresh forward-serve inside one npu-hold ticket. The metric is the sum over all [prof] rows of launches × ms/launch, i.e. total device-launch time.

    session arms (load at arm start) total vs first P arm
    cl-npurt, 08:27–08:35 local (#2178 logs, max_tokens 8) P, C0 (6df0386), P, C (d0fa51c) P +0.0, C0 +0.6 %, P +4.9 %, C +8.7 %
    mine, run 2, 09:00–09:09 local (max_tokens 8) P1 C1 P2 C2 P3 C3 (10.7 → 8.1) P +0.0 / +5.4 / +15.5 %; C −2.0 / +8.7 / −1.5 %
    mine, run 1, 08:41 local (max_tokens 32, cut by the ticket) P1 (28.4), C1 (14.6, d0fa51c build), P2 partial (6.5) P1 is 29 % slower than both C1 and P2: load, not the binary
    • Pairs in run 2: C against the P just before it is −2.0, +3.1 and −14.8 %. The median C arm is 6.5 % faster than the median P arm.
    • Launches: launch counts and every [prof] dispatch line are identical in all 13 arms (the structural half of the prof table). Texts are identical in all arms. No request errors.
    • Run 2's C binary is not clean main. It is ~/wt/cl-npurt-build-1788 rebuilt at 08:47 local from main 7035ac1 plus uncommitted edits to q4nx_forward.cpp in cl-npurt's tree. The d0fa51c evidence is cl-npurt's session and my run 1. I could not freeze a clean main build: the disk is under the 175G floor for a new build.

    Reading. The candidate is never outside the same-binary spread, and in the paired arms it is as often faster as slower. A 2 % bound cannot be checked on this box under shared load. Meeting it needs a quiet box, as with the rate corrections in #2140.

    Gate state: all five listed items are in the tree. Outputs are identical: window11 on the 27B (02:29Z comment) and KLD 0 with 64/64 identical dumps on the 1.5B (#2178). The prof structure is identical and the timing is within noise (this comment). CI is green on main. I am asking the orchestrator to accept this measured ceiling so the issue can close. If a quiet-box window is wanted instead, the run is ~/.cache/scratch-cl-1051/p803/ab.sh, about 10 min under one npu-hold ticket.

  10. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    cl-1051: the caveat in my 13:11Z comment is now removed. That comment's run-2 candidate was a dirty tree. This run's candidate is a clean origin/main build at d7a0e88 (~/.cache/scratch-cl-1051/b803/1bit, sha256 d0d509a0…), configured with the same flags as the pristine binary: ONEBIT_NPU_PRIVATE=nk-dx, JINJA off, own runtime, the same runtime directory, gcc-16, Release. The two arms now differ only by source. Same harness as before: one npu-hold ticket, P C P C P C, load 4.4–9.8 (quieter than either earlier session).

    arm P (540e1fd) C (main d7a0e88)
    1 +0.0 % +7.3 %
    2 +12.2 % +11.5 %
    3 +20.8 % +7.4 %
    • The P binary alone spans 21 % across its three arms. C spans 4 %, and its median is 4.3 % below P's median. Against the P arm just before it, C is +7.3, −0.6 and −11.1 %.
    • Dispatch lines and launch counts are identical in all 6 arms, texts are identical, and there were no request errors.

    So on a clean, flag-matched main the result stands: no slowdown is visible, and a 2 % bound is below this box's same-binary spread even at load 4–10. I'm leaving the ORCH ask as it was: accept that ceiling to close, or give a quiet-box window. cl-npurt, if #2178's doc should carry gate item 3, these numbers are yours to cite. Logs are in ~/.cache/scratch-cl-1051/p803/ (top level is this run, then run1/ and run2/).

  11. added 2 commits that reference this issue on Nov 11, 2026
  12. bong-water-water-bong commented on Nov 12, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    CLOSE CHECKLIST for #803 (cl-1051, lead), read from origin/main at 11a0ca2

    Each item is from this issue's own body. "Done" means the code is in the tree at 11a0ca2 and the row cites the PR that put it there.

    # item from the issue body state where
    1 npu/target/target.h interface: open, load_block, gemv, optional gdn_step/attn_step/lm_head, caps done #808 (A1) put the interface in. The optional ops return false and the model takes its host fallback, a seam that #1632 implemented. At 11a0ca2, target.h:110/115/118 holds open/load_block/gemv, :143-145 the three optional steps and :147-148 supports()/caps().
    2 npu/target/xdna wraps today's I8Ctx / npu_rt_api path, no new kernels done #808 moved the launch layer, #1380 the M=128 ELF set, #1458/#1478 the lm-head context, and #2062/#2166 the M=16 decode family. xdna_target.h:60 XdnaTarget now owns every I8Ctx. No kernel was added.
    3 NpuModel owns the math and calls the target done as A3 landed it #1695 and #1724 moved the target and every state member into npu/model/npu_model.h: geometry c_, semantics sem_, norm scratch, KV, MLA and recurrent state. The arithmetic statements stay in Q4nxNpuForward's method bodies and reach that state through reference members. That is deliberate: no expression moved, which is what made the window11/KLD identity provable (docs/npu-p4-target-split.md "A3 as landed"). There is no npu_model.cpp. Moving the method bodies would be a new slice with no behaviour change, and per the owner's "no new slices" I'm not opening one.
    4 --npu-target xdna is the default done #808 added the flag and #1664 documents it. With no flag, forward-serve uses XdnaTarget::kName = "xdna". An unknown name is refused by name before the device opens: --npu-target hexagon: this build has no such target (compiled: xdna).
    5a gate: window11 text identical done A2b gate (comment above, 11-11 02:29Z): three window11 runs on bonsai2-27b-NPU, texts identical across arms. Each A3 slice also passed 3/3 identical texts and 13/13 identical rows, and they pass against the c25 reference.
    5b gate: logits KLD 0 on 8 prompts done #2178 (cl-npurt): 540e1fd ×2 as control vs main d0fa51c, 8 prompts. 64/64 dumps byte-identical, so mean and max KLD are exactly 0.
    5c gate: prof table within 2 % open, needs the device Measured twice, both times on a loaded box. My 11-11 14:33Z run on a clean, flag-matched main build: dispatch lines and launch counts identical in all 6 arms and no slowdown visible (C median 4.3 % below P). But the old binary alone spans 21 % across its own three arms, so a 2 % bound can't be resolved without a quiet box.
    5d gate: ctest green done (measured today, host-side) origin/main 11a0ca2, ONEBIT_NPU=ON with the private kernel set, gcc-16, Release, run under thermal-run with ctest -j 6: 100 % passed, 0 failed out of 269, 15 of them skipped (model or tool not present). Four tests were excluded because they touch the device or its live lock while NPU OFF is in force: npu_dmabuf_import (it submits a kernel), npu_device_lock, device_holders and device_route. Build note: since #2037, main needs a runtime source with the ext::bo(dev, fd, size, host) constructor. A runtime directory older than that fails to link npu_rt_shim_test and npu_dmabuf_import_test, which is npu_rt_shim_test doing its job. With that runtime source the build links clean.

    Open after this: only 5c, and it is a device run. Under NPU OFF I'm not queuing it. The exact run owed is in the ORCH line in my state file: a quiet-box npu-hold window of about 10 min for the interleaved P/C NPU_PROFILE=1 arms on bonsai2-27b-NPU, with the old binary 540e1fd vs main. Alternatively, the owner can accept the measured ceiling (identical dispatch, no visible slowdown, box spread above 2 %), and then this issue can close.

    #799: this issue is its P4. What remains of the plan layer is #799's own rows. That includes XdnaTarget::load_block(), still a documented no-op, which docs/npu-p4-target-split.md §3 found is a first-token-latency step and not needed to accept this issue.

  13. bong-water-water-bong commented on Nov 12, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Gate 5c (prof table within 2 %) measured on the NPU. Every #803 checklist item is now met.

    This follows the close checklist above (issuecomment-6825952062), where 5c was the one open row.

    The run. One npu-hold ticket (cl1051-803-prof5, 16 min limit, exited 0 after 667 s).

    • Arms interleaved four times, P C P C P C P C, with NPU_PROFILE=1.
    • Model ZR1-1.5B-npu-q4km-v2 with its kernel set, 8 fixed greedy completions per arm.
    • P is the pre-split binary at 540e1fd (sha256 33bb7e8d…).
    • C is origin/main at 11a0ca2 (sha256 802515ef…), built against the same runtime source as P.
    • Load at arm start ranged 4.7–16.6.

    Route-matched. Since the 11-11 run, two later merges changed main's default route: #2227 puts the decode lm head on the host, and #1963 runs 17–128-row prefill chunks as 16-row launches. That is not this issue's change. In a run with defaults (run3, same ticket shape), C dispatched differently (458 rows vs 451, 8208 launches vs 8808) and came out 16 % faster. That number can't answer the gate. So the C arms here run with NPU_HEAD_HOST=0 NPU_PREFILL_SPLIT=0, the two documented switches back to the old route, and the comparison isolates P4.

    Identity. In all 8 arms the texts are identical, the dispatch lines are identical, every arm has 451 profile rows with 8808 launches, and there were no request errors. No amdxdna kernel line was logged during or after the ticket.

    Per-row result, median ms/launch over the four arms of each binary:

    row launches P C C/P P own spread C own spread
    dec D 16 1788 8.111 8.025 −1.1 % 11.5 % 29.3 %
    dec GU 16 1788 17.393 17.066 −1.9 % 14.5 % 33.5 %
    dec HEAD 16 648 17.008 16.496 −3.0 % 13.5 % 33.7 %
    dec O 16 1788 1.918 1.952 +1.8 % 10.4 % 25.1 %
    dec QKV 16 1788 2.030 2.068 +1.9 % 10.6 % 19.9 %
    pre D 128 252 10.490 10.535 +0.4 % 19.3 % 49.3 %
    pre GU 128 252 21.644 21.560 −0.4 % 17.4 % 46.0 %
    pre O 128 252 2.784 2.830 +1.6 % 17.8 % 49.8 %
    pre QKV 128 252 3.082 3.142 +2.0 % 23.2 % 49.2 %
    • 8 of 9 rows are within ±2 %. The ninth, dec HEAD, is 3.0 % faster on main. No row is slower than P by more than 2.0 %.
    • Totals: median C/P −1.80 % over all columns and −1.85 % on device wait. The pairs were +0.9, +19.4, −5.7 and +2.3 %. The +19.4 % pair is arm C2, which started as load jumped from 4.7 to 12.5, and its rows are uniformly slow, as on a loaded box.
    • Caveat: each binary's own spread (10–50 %) is larger than 2 %, so this is a statement about medians over interleaved arms, not about any single arm.

    Checklist status: items 1–4, 5a (window11), 5b (KLD 0), 5c (this row) and 5d (ctest 269/269 at 11a0ca2) are all met. I'm leaving the close to the owner. What remains of the plan layer belongs to #799.

    Logs are in ~/.cache/scratch-cl-1051/p803/run4/ (route-matched) and run3/ (defaults).

  14. bong-water-water-bong commented on Nov 12, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Closing. I checked cl-1051's close checklist (14:12Z) and gate 5c (16:04Z) against origin/main at 9a589d7, which is later than the checklist's 11a0ca2.

    • Item 1: npu/target/target.h:110/115/118 holds open, load_block and gemv, and :143 onward has the optional steps that return false.
    • Item 2: XdnaTarget (xdna_target.h:60) owns the I8Ctx contexts, and no kernel was added.
    • Item 3: the state is in npu/model/npu_model.h, as the checklist describes ("done as A3 landed it").
    • Item 4: forward_serve.cpp:1206 defaults npu_target to XdnaTarget::kName ("xdna"), and :1265 refuses an unknown name.
    • Items 5a, 5b and 5d are as cited.
    • 5c: I re-read the run4 logs. In all 8 interleaved arms the 8 completions have the same text sha256s, and run.log has no amdxdna line. The route-matched medians are within ±2 % on 8 of 9 rows, and the ninth (dec HEAD, −3.0 %) is faster on main.

    The plan-layer work that remains belongs to #799, and #799 closes with this issue. XdnaTarget::load_block() is still a documented no-op. That is a first-token-latency step (docs/npu-p4-target-split.md §3) and not part of this acceptance.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    backlogReal, but does not hold the release buildenhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions