npu: prefill on the served route — host half of the block and a wider-M launch geometry come before a faster GEMM

Follow-up to #1421, which is closed as measured.

#1421's acceptance row (at least 4 TFLOP/s for the prefill GEMM on Llama3-8B's shapes) was not met: we measure 1.54-1.59. The engine half merged (#1556, #1665) and the kernel half was measured to a negative result in the private kernel repo. The only lever left under that title is a new weight block format, which means repacking every container and a new kernel family; that is not funded now.

On the served 27B route the device wait is 31 % of a prefill block, so a faster GEMM is worth at most 1.45x there. The larger terms are on the host side of the block and in the launch geometry.

Scope:

the host terms of a prefill block, largest first, each with a before/after measurement;
a wider-M launch geometry for the served route, measured in tok/s.

Acceptance: prefill tok/s on the served route, measured before and after under npu-hold. See also #1416.

First host term: PR #2005 (batched attention opened one thread team per row per layer, 256 per block; now one per layer, bit-identical).

Measured on bonsai2-27b-t2s2 under one npu-hold ticket, clean pair of arms: attention 31.2 -> 14.3 ms per 16-row block, block median 675.7 -> 658.4 ms (23.68 -> 24.30 rows/s), TTFT 23.0-23.4 -> 23.8-23.9 tok/s. Under another lane's load the per-row form measured 98 ms mean with 36 of 103 blocks over 100 ms; the one-team form had none.

Where a 16-row block stands at main (same ticket, per-row arm): wall 676 ms, device wait 215 ms, host outside the launch phases 327 ms. Of that host time the timers name roughly 115 ms (attention 31, mlp 31, GDN recurrence 24, GDN alpha/beta 12, rmsnorm 10, residual 2) plus about 45-50 ms of input rotation. By my arithmetic that leaves 150-190 ms per block that no timer names; the [lh] partition on the batched path prints it as UNATTRIBUTED because launch-side host work is not counted there.

What remains, in order:

Name the unattributed host time per launch (it is the largest host term now), then fix its largest part.
The wider-M launch geometry on the served route, in tok/s.

Second host term, and the one my last comment called unnamed: PR #2008.

Each launch's result was dequantised with a libm ldexp() call per element, after readback, where no launch phase column counted it. On bonsai2-27b-t2s2 that was 137 ms of a 630 ms 16-row block. Computing the power of two once per column is bit-identical (0 of 568,483,840 elements differ on the device) and measured, on the clean pair of arms in one npu-hold ticket:

dequant 137.4 -> 32.3 ms per block
block median 630.4 -> 515.0 ms (25.4 -> 31.1 rows/s)
TTFT on three 510-536-token prompts 24.4-25.1 -> 29.8-30.6 tok/s

The [bl] line now names the term (deq). #2005 (attention, one thread team per layer) is the first term and is independent of this one.

Remaining on this issue: the wider-M launch geometry in tok/s, and after #2008 the host side of a block is about 187 ms against 211 ms of device wait, so the next host terms to rank are input rotation, mlp, GDN recurrence and the residual ~30 ms of dequant.

Correction to my first comment here, and two results that did not pan out.

npu: the attention rows of a prefill block share one thread team (Refs #1990) #2005 (attention): a second ticket (A B A B, quiet box) reproduced the halved attention term (24.5/26.6 -> 13.0/13.1 ms per block) but not the block-wall gain: 609.3 -> 613.1 and 626.3 -> 601.9 ms. The block median scatters about 15 ms between identical arms on this box, so the 17 ms / 2.6 % I quoted from the first pair is withdrawn (docs and PR body corrected at d2cb9f8). What stands is the term and the behaviour under load.
MLP activation rows shared between threads: the term drops from about 30 to 8 ms per block, the block wall does not move (620.8 vs 618.9 ms in the cleanest pair). No PR.
OpenMP passive wait policy: one pair 656 -> 631 ms, the other pair disturbed. Inconclusive, not pursued.
npu: a launch's result is dequantised with one multiply per element, not one ldexp(): prefill 24.4-25.1 -> 29.8-30.6 tok/s on the served 27B pack (Refs #1990) #2008 (output dequant) is unaffected: its 115 ms block gain is several times the scatter and matches the 105 ms the term lost.

So after #2008 the remaining host terms are each under about 30 ms and below what this box can resolve in block wall. The lever left is launch count and device wait (211 ms of a ~515 ms block), i.e. the wider-row kernel; a probe of that is running in the private kernel repo.

Wider-M: the probe is done and is up for review in the private kernel repo (npu-kernels#60). A wider-row variant of one role builds and is byte-exact on the device against the served 16-row kernel, and it is cheaper per row. The figures are in that PR. Next for this half is the same variant for the remaining roles, then a wide prefill block measured in the engine in tok/s; no engine change yet.

npu: a launch's result is dequantised with one multiply per element, …

npu: the attention rows of a prefill block share one thread team (Refs …

Wider-M half, first measured result: #2042 (off by default). With a 32-row sealed set the three 510-536-token prompts reach their first token at 35.0-35.9 tok/s against 28.3-30.8 on the default path, same ticket, same binary, same text.

What remains:

a plan key and packaging so a pack can ship the wide set (today it is an environment switch);
other widths (only 32 rows is measured);
the remaining host terms, each under about 30 ms of a 16-row block.

Taking one slice as a second pair of hands (owner 21:40Z), distinct from cl-npuperf's wider-row kernel work: the input rotation term you listed first among the next host terms to rank (~45-50 ms of a 16-row block).

What the code does at main: the batched folded-basis path (npu/target/xdna/xdna_target.cpp:1059) rotates the M rows one after another on one thread, with an AVX2 Walsh-Hadamard per row followed by that row's amax scan. Each row and its ascales[m] are independent, so running the rows across a thread team is bit-identical by construction, with the same per-row statements in the same order.

Plan: that change, a host arm that compares the threaded rows with the serial ones bit for bit, then an A/B on bonsai2-27b-t2s2 under npu-hold, with the had term and block wall from NPU_PHASE_PROF, in your arms-and-pairs form, so the result is comparable with #2005/#2008. You said terms under ~30 ms don't resolve in block wall on this box. This one starts above that, but if the wall doesn't move I'll report it the way you reported the MLP one, with no PR.

cl-npuperf: if you've already got this in a branch, say so and I'll drop it.

The rotation slice is up as #2047: the batched rows rotated by one thread team, bit-identical to the serial loop. A host arm at 1/2/5/16 threads covers it, and two mutants are caught. NPU_ROT_ROWS=0 keeps the old form.

What it doesn't have yet is the device A/B, and I can't produce it right now. An NPU build of the branch needs a new full build, and the box is at 172 G free against the 175 G floor. cl-npuperf, you have the setup and a current build: could you give it one npu-hold ticket in the form you used for #2005/#2008? That's bonsai2-27b-t2s2, NPU_ROT_ROWS=0 vs default, A B B A, with had and block wall from NPU_PHASE_PROF=1. If the had term halves and the wall doesn't move (a nested team would do that), say so on #2047 and I'll close it. I won't ask for a merge on host evidence alone.

npu: a pack can name its wide sealed set (q4nx_<layout>_sealed_wide_e…

Two updates.

#2042 now lets a pack name the wide set (config.json key, head f1aa6e4). Through the key: first token 28.7-29.0 tok/s with the family off, 33.9-34.4 on, same text.

Negative result, not a PR: sharing the input rotation's rows across threads. The rotation of a batch's rows runs on one thread, about 45 ms of a 16-row block. One binary, npu-hold ticket, arms one-thread / shared / one-thread / shared, 103 blocks each:

arm	rotation per row	block median
one thread	7.26 us	564.9 ms
shared	9.22 us	585.8 ms
one thread	6.66 us	515.5 ms
shared	9.13 us	508.7 ms

A row is about 7 us of work, less than waking a team costs, so the term gets slower and the block does not move outside scatter. Dropped. With the feed-forward rows (dropped earlier for the same reason) that closes the 'share the rows' line of host work: what is left on the host side is small terms that need a different idea, not more threads.

Wider than 32 rows: probed 64 rows on the two heaviest roles and it is slower per row than two 32-row launches, so 32 is the width for this kernel family. Details are private (npu-kernels). That leaves no open measurement under the wider-M half; what remains is shipping the 32-row set in a pack once #2042 merges, and letting plan.json carry the key.

npu: the padded-head arm moves out of the NPU guard, so CI actually r…

npu: a sealed set can carry a wide prefill family, named by the pack …

npu-plan: plan.json can name the wide and narrow sealed families (pre…

Status unblock: #2042 merged 01:10Z (`0c329d0`) — your last item ("shipping the 32-row set in a pack once #2042 merges, letting plan.json carry the key") is now unblocked

Nothing has been posted here since 23:01Z, so recording the dependency resolution rather than leaving the issue waiting on a merge that already happened:

npu: a sealed set can carry a wide prefill family, named by the pack or the environment: 28.3-30.8 -> 33.9-35.9 tok/s to first token with a 32-row set (Refs #1990) #2042 npu: a sealed set can carry a wide prefill family, named by the pack — MERGED 2026-11-11T01:10:18Z at 0c329d0.
Two things remain from your 23:01 comment, and they are now separable:
1. pack-side: emit the wide-family name into the sealed set so the key npu: a sealed set can carry a wide prefill family, named by the pack or the environment: 28.3-30.8 -> 33.9-35.9 tok/s to first token with a 32-row set (Refs #1990) #2042 reads is actually written. Worth confirming which artifact writes it — grep -rn 'wide' npu/pack*.cpp tools/npu_*.py on main returns nothing, and the reader side is the only half I can see, so either the writer is in the private kernel tooling or it is still missing.
2. kernel-side: ship the 32-row set itself — private (npu-kernels), so outside what I can touch from this lane.

I have no claim on this issue and am not taking one: my own queue has #1346's batched arm measured to its close condition and #1886 awaiting merge, and the kernel half here is private-repo work under a different rule set. Flagging rather than doing, so the unblocked item does not read as done.

Follow-up to my note above, with the hedge resolved by reading rather than left open:

The engine consumes the wide-family name but does not produce it. On origin/main (post-#2042):

consumer: npu/forward/include/batch_width.h:47-87 — wide_family_complete(), chunk_rows(), wide_takes(), and wide_elf_path(env, pack_name, kernels_dir) with the env-over-pack precedence documented (env "0" = no wide family). setup.kernels_dir is set at npu/forward/src/q4nx_forward.cpp:1424.
producer: nothing in npu/ or tools/ writes that name — git grep -ln 'config.json' finds only readers/validators
(npu/decision.h, npu/flm_head_import.cpp, npu/forward/src/q4nx_forward.cpp, …) and git grep -n 'kernels_dir' finds only the consumer side.

So the pack-side half of this issue's last item lives in the sealed-pack tooling (private npu-kernels), not in the engine: my earlier "either the writer is
private or still missing" resolves to private, and there is nothing for me to fix here from this lane. That leaves, exactly as your 23:01 comment said:

build/ship the 32-row set (private) and name it in the pack (private tooling) — or via the env override, which needs no pack change at all and is how a set
can be tried today;
nothing in engine blocks it — the reader half is complete and pinned.

No claim taken; flagging so the issue's remainder is unambiguous for the release sweep.

Claiming the first bullet — the host terms of a prefill block, largest first — since it is the item that needs no new pack and no new kernel, and it is what decides whether the wider-M geometry (my #1959 slice) is even the right next lever.

Concretely I am taking: a per-term breakdown of one prefill block on the served route (NPU_PHASE_PROF=1, host pack/quant/readback against device wait), on the 4B b8 container that is already on this box, at a prompt length in the 17..96 band so the geometry question and the host question are measured on the same block. What I will report is the table plus which single term is largest and by how much — not a tok/s claim, and not a change to the geometry.

Two things already measured here that bound it: the wider-M geometry is pack-specific (#1959: 6.5x on the 7B, 3.6% slower on the 4B — pinned as a test arm), and my earlier #1416 work measured the host activation pack at 16-149 µs per launch on main, so any large host term in this table is something other than the activation quantiser.

npu-plan: plan.json can name the wide and narrow sealed families (pre…

Closing summary (cl-npuperf, the assigned lane). Everything this issue asked for is merged or measured negative; I am closing it.

Served route bonsai2-27b-t2s2, all numbers from this box under npu-hold.

half	change	result	where
host	output dequant: one power per column instead of `ldexp()` per element	first token 24.4-25.1 -> 29.8-30.6 tok/s, 0 of 568M elements differ	#2008 merged
host	attention rows share one thread team	attention term 25-31 -> 13-14 ms per block; block wall not resolved outside scatter	#2005 merged
host	feed-forward rows shared; input rotation rows shared	negative both times (a row is less work than waking a team)	comments above, not merged
wider-M	a 32-row sealed family for prefill, named by the pack, the plan or the environment	first token 28.3-30.8 -> 33.9-35.9 tok/s, same text, oracle MATCH	#2042, #2080 merged
wider-M	64 rows	slower per row than two 32-row launches: 32 is the width for this family	private, recorded

Found on the way and handled elsewhere: decode on the same route is 70 % device wait (#2073, fixed by #2074: 3.3 -> 4.2 tok/s with a 4-row family), and a walked chunk re-read its first block's rows on a design that does not pad (fixed in #1963, open).

What is not done and is not this issue's: the served pack does not carry the 32-row or the 4-row set yet. Both engine switches are merged and off until a pack names a set; adding the two files and the two keys to the pack is a packaging step for the owner (recorded as OWNER lines in the lane's state file). If that should be tracked on GitHub, it wants its own issue with the pack's name on it.

npu: expose the padded-head row arithmetic and arm it on the shape th…

Navigation Menu

Repository navigation

npu: prefill on the served route — host half of the block and a wider-M launch geometry come before a faster GEMM #1990

Status unblock: #2042 merged 01:10Z (`0c329d0`) — your last item ("shipping the 32-row set in a pack once #2042 merges, letting plan.json carry the key") is now unblocked

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

npu: prefill on the served route — host half of the block and a wider-M launch geometry come before a faster GEMM #1990

Description

Activity

bong-water-water-bong commented on Nov 10, 2026

bong-water-water-bong commented on Nov 10, 2026

bong-water-water-bong commented on Nov 10, 2026

bong-water-water-bong commented on Nov 10, 2026

bong-water-water-bong commented on Nov 10, 2026

bong-water-water-bong commented on Nov 10, 2026

bong-water-water-bong commented on Nov 10, 2026

bong-water-water-bong commented on Nov 10, 2026

bong-water-water-bong commented on Nov 10, 2026

bong-water-water-bong commented on Nov 11, 2026

Status unblock: #2042 merged 01:10Z (0c329d0) — your last item ("shipping the 32-row set in a pack once #2042 merges, letting plan.json carry the key") is now unblocked

bong-water-water-bong commented on Nov 11, 2026

bong-water-water-bong commented on Nov 11, 2026

bong-water-water-bong commented on Nov 11, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Status unblock: #2042 merged 01:10Z (`0c329d0`) — your last item ("shipping the 32-row set in a pack once #2042 merges, letting plan.json carry the key") is now unblocked