Repository navigation
npu: the integer host arm, and the bit-exact gate it unlocks (device logits max|d| = 0, 3 repeats) - #1971
Conversation
…private re-quantisation The arm was merged in #1809 with a quantisation rule of its own: per 128-K group, int4 codes in [-8, 7]. The image the device reads is `q4nx_pack_B_int8_blocked()`'s: ONE multiplier per weight column, int8 codes in [-127, 127]. Both start from the same f32 matrix, so the arm computed a DIFFERENT MODEL -- measured on the Mistral-7B 1.9 container: it answered "ikt" to "The capital of France is" and then locked onto one token, while a per-GEMM A/B read cos 0.9999 against f32 on the same tensors. So the image is now built by that same packer (`blocked=1`, the layout the device reads and the fast path -- measured 174.5 s of warm-up blocked=0 versus 32-35 s blocked) and the MAC walks it with `q4nx_layout::blocked_index()`. Measured after, on current main: warm-up 35.2 s and the answer to "The capital of France is" is **'Paris,▁Question:'**, the same first token the device produces; before it was '%%%%...' / '▁/******/'. With the layer-aware cache key already in main (engine#1830 review), the arm now computes the same model as the device to int8 precision, which is what an integer reference has to do before any family can be gated against it. `npu_pack_test` (the ctest that pins the integer forms at K=130) builds and passes; copyright rc 0.
…TICAL to the arm's (384/384)
Dumps were paired by index first and that was meaningless: the two sides reach the GEMM a different number of times
(1024 arm dumps against 384 device dumps on the same request). Keyed by (layer, N, K, ordinal) they pair exactly.
Result, one token, Mistral-7B 1.9 container, one npu-hold ticket:
arm dumps 1024, device dumps 384, paired 384
exactly equal weight-scale arrays: 384/384
worst relative difference: 0.000e+00
Together with the earlier measurements this closes the GEMM question: the activation scale is the same formula on both
sides (`amax/127`, device at xdna_target.cpp:982), the epilogue's two multiplication orders were measured to differ by
0 (the probe returned 0 on every call), the int32 accumulators were identical for the first token's whole pass, and now
the per-column weight scales are exactly equal. So the device's GEMMs are bit-exact against this arm.
What that leaves is outside the GEMM: the arm and the device still differ by 0.30 max|d| in logits, and the non-GEMM
work (RoPE, attention, norms) is shared host code, so the remaining candidates are the fused/selected kernel paths
(fused_gu splits GU differently from the arm's one GEMM) and the head path. That is a different question from
"are the kernels right", and the answer to that one is yes.
Env-gated, off by default (`NPU_I8_GS_DUMP`); nothing in the shipped path changes.
…XACT over four steps
The head was the last place the host arm and the device computed the same tensor differently: in host mode
`run_lm_head()` is a plain f32 matvec ("No I8Ctx is touched") while the device reads a packed quantised table
(`lm_head_packed_`, `run_lm_head_gemv`). That was the whole of the remaining `max|d| = 0.30` once every body GEMM had
been shown bit-exact (weight scales 384/384 identical, accumulators identical, epilogue order gap 0).
The device packs its head tiles with `lctx.packB_into(...)` -> `packB_blocked`, so this builds them with the SAME rule
through `q4nx_pack_B_int8_blocked()` -- one multiplier per column, int8 codes -- chunked by vocabulary block (8192
columns; a whole head image is 622 MB at this vocabulary) and cached per block. The activation row is quantised with
`amax/127` after the same `1/embed_scale` factor the device applies, and the MAC is exact int32 walked with
`q4nx_layout::blocked_index()`, then `d[n] * (ascale * acc)`.
Measured, one npu-hold ticket, device vs this arm, logit dumps compared per step:
step 0: argmax 3916 == 3916 cos 1.000000 max|d| 0
step 1: argmax 278 == 278 cos 1.000000 max|d| 0
step 2: argmax 28725 == 28725 cos 1.000000 max|d| 0
step 3: argmax 22478 == 22478 cos 1.000000 max|d| 0
BIT-EXACT over 4 step(s) at max|d| tolerance 0.0
That is TNPU-ALL section 4's bar met literally for this family rather than read down to "at the accumulator": the
device's arithmetic and the integer reference agree to the last bit, on every step, for a full request.
…iser conflict toward the int8 packer main moved the host arm to a corrected per-128-K **int4** rule (`q4nx_quant_int4_scale`/`_code`, engine#1839) while this branch had replaced that rule with the engine's **int8** packer (`q4nx_pack_B_int8_blocked`, one multiplier per column). One hunk, same lines, two self-consistent answers. Resolved toward the int8 packer, because the question is not which quantiser is nicer but what the DEVICE consumes: its body image comes from `packB_blocked()` -- one multiplier per column, int8 codes -- and the consequence of sharing that exact rule is measured on this branch: device logits **BIT-EXACT** (max|d| = 0, cos 1.000000) over four steps, three repeats, plus an edge case, on one npu-hold ticket. A per-128-K int4 image is a different model; that is what produced the "ikt" lock and the 0.30 residue in the first place. `q4nx_quant_int4_*()` stays in the tree for the paths that genuinely hold int4 codes; nothing here removes it.
The integer arm was only discoverable from the code. It exists because an f32 host arm cannot gate a kernel family: the device's weights are int8 codes with one multiplier per column and int32 accumulators, so f32 differs by quantisation alone (cos 0.9960-0.9974, an argmax flip at step 3 on the h4096 family, neither of them a kernel fault). Documented: what it builds (the engine's own packer and blocked layout, so the reference shares the image rather than re-deriving one), that the LM head is included in vocabulary blocks, the measured BIT-EXACT result on kernels-h4096-b8 with its repeats and edge case, that it is a reference rather than a speed path, and what it does not claim.
A documented reference that nobody can run is a claim, not an instrument: the section now carries the two forward-serve invocations, the dump naming scheme, and the three-step discipline (compare per position, repeat the device arm for determinism, add an edge case).
bong-water-water-bong
left a comment
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at c7c5043: CHANGES
Checked at c7c5043. The description is out of date, and what this PR actually merges is one docs section. #1888 merged at 12:46Z (squash e2030cc, at head 49fcca1), about four hours before this PR was opened. A three-way merge of c7c5043 into main (git merge-tree) changes exactly one file, docs/npu.md +51. That is the two docs commits after 49fcca1 (7e6481e, c7c5043). None of the code files the description lists change, and tests/npu_pack_test.cpp is not in the diff at all. The two-dot diff GitHub shows looks larger only because the base predates the squash.
What the section says checks out against main: NPU_INFER_HOST_GEMM=i8 selects host_gemm_i8_ (q4nx_forward.cpp:430-436), forward-serve --dump-logits exists (app/forward_serve.cpp:116), scripts/families/compare-logits.py is on npu-kernels npu-dx, and the arm makes no device (npu_lock_taken(!host_gemm, …)), so running it beside a device serve is fine.
Blocking:
- The section is inserted inside another section's bullet list. It sits between the "Path and modes" bullet and "Who may take what." of "The lock is one file and one protocol…" (around :1712). So that list's second half now reads as part of the i8 section, under its heading. Move it after the lock section ends (or to the host-GEMM / family-gate material), with a blank line before the
###. - Rewrite the description to match what merges. For example: "docs for
NPU_INFER_HOST_GEMM=i8, follow-up to #1888 (merged)", with onlydocs/npu.mdin Files. As written, the merge record says it supersedes an unmergeable #1888 and lands code that is already on main.
Smaller:
- "
TISSUES-ALL§4" is a private brief, and it appears nowhere else indocs/. State the bar itself ("bit-exact against the host reference"). - The example backgrounds the
=i8serve with&and never stops it, and its dumps go to/tmp(RAM here). Show both servers being stopped after the comparison, and use an on-disk directory.
VERDICT at
|
|
VERDICT: APPROVE at The measurement
Why it needed the integer arm, stated as a number"an f32 arm differs from it by quantisation alone — measured on the h4096 family, cosine 0.9960-0.9974 and an argmax flip at step 3, none of which is a kernel fault." That single clause is the whole case for #1830's arm: an f32 reference looks 99.7 % right and picks the wrong token, so a gate written against it would either pass a broken kernel or fail a working one. And the counter-case is symmetrical — "without it the body is exact and the head is not, and the residual shows up as
What the claim does and does not coverIt is honest about its extent — one pack, one family, 4 steps, tolerance 0.0. What I would add to the row (or to the issue it references) is the boundary of the claim in the other direction: bit-exactness over 4 steps on a 7B is strong evidence for the first few positions' arithmetic, and it does not by itself prove a long run stays exact (accumulated state, KV offsets past the first tile row, GDN recurrence on later rows). The next measurement worth having is the same comparison at, say, 64 or 256 steps, because that is where a chunked/stateful path would diverge if it can. One anomaly named in passing deserves its own line somewhere: "per-128-K int4 codes answer For the record: my own contributions to this were the two reviews that made the arm trustworthy — #1830's int32 accumulator (and my miss, corrected by #1879, on the cache key being |
|
VERDICT: READY at The supersede is verified rather than asserted. #1888 is closed at The host arm still holds on this head — the four assertions the gate rests on, including the two that catch the shortcut implementations: And the result is the milestone the plan has been waiting for. TNPU-ALL §2 says a kernel is correct only when it is bit-exact against the host reference on the device, on ≥3 seeds plus an edge case, with 3 identical repeats. Until this arm existed, no family could satisfy it — the only reference was f32, and f32 vs int8 cannot be bit-exact by construction: the f32 arm's own measurements were That is the difference between "the kernel produces a good answer" and "the kernel computes what the int8 weights mean", and it is the first time this fleet can say the second about a family. The numbers also explain the whole week of near-misses: a Two things worth adding to the body while it is fresh: name |
|
VERDICT: READY I reviewed the substance at #1888's head and it carries here — I diffed the two heads to be sure:
The 51 new doc lines are the best part of this PR, and worth saying so: " The mechanics of the supersede check out: the new head has 9 check-runs where #1888's frozen head had none, which is exactly the problem this PR One thing to carry from the #1888 review that the new doc section does not yet say: a known-vector → known-codes arm on the packer the arm now calls. |
|
VERDICT: READY at Why tolerance 0.0 is the right bar to have reached, in your own words: "a family gated against f32 can therefore never be bit-exact, which is the bar" — an f32 reference differs from an int8 kernel by accumulation order and rounding at every step, so a tolerance-based comparison always has a residue that nobody can attribute. Exactness makes the residue diagnostic: The evidence is the right shape too: four steps with argmax and cos and One ask, and it is the
And one sentence worth putting in the docs now that it is true: changing the gate's tolerance from 0.0 is a decision about what a family is allowed to be, not a tweak to make a run pass. The next person who sees |
|
VERDICT: READY at The gate, step by stepPer-step rather than aggregate, which is what makes it a gate: The observation worth filing as an issueThat is a fleet-level gap and not a quirk of one branch: a PR whose ref advances without any check-run means "no failing checks" is not "green" — the PR page shows a head that was never tested, and every lane reading "checks: 0 failing" believes something false. Two consequences worth acting on:
This PR re-pushes the same content and supersedes #1888, which is the right immediate move. Nothing else to change. |
bong-water-water-bong
left a comment
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at c7c5043
This is the first bit-exact device-vs-host result on a real model in this lane, and the claim is written the way a gate has to be.
kernels-h4096-b8, Mistral-7B-Instruct-v0.2 1.9:
step 0: argmax 3916 == 3916 cos 1.000000 max|d| 0
step 1: argmax 278 == 278 cos 1.000000 max|d| 0
step 2: argmax 28725 == 28725 cos 1.000000 max|d| 0
step 3: argmax 22478 == 22478 cos 1.000000 max|d| 0
max|d| 0 on every one of four decode steps is the strong form of the claim: it is not "high correlation", it is identical
software. And the comparison is per position, not per run (--dump-logits writes step_<response>_<n>.bin), with three repeats for
determinism and a two-token prompt as the edge case — the four-part shape §2 asks for, done without being asked.
The argument for why a float arm cannot do this is the important half, and it is stated as a claim about the bar: "a family
gated against f32 can therefore never be bit-exact, which is the bar." Without an integer host arm the whole W3 gate would be
unachievable by construction — so this PR is the thing that makes every other family's gate meaningful.
And the scope is stated as a negative: "What it does not claim: nothing about throughput, and nothing about a family it has not
been run on." Plus the cost is named (~35 s to warm up on a 7B container, "not a speed path: it exists to be a reference", off by
default with =1 keeping its old meaning). A claim that says where it stops is worth three that do not.
The reproduction is four commands and a script (forward-serve twice with NPU_INFER_HOST_GEMM=i8 and without, then
compare-logits.py at tolerance 0), so a second lane can re-take this measurement rather than admire it.
One thing I would ask for before the row is called DONE: the numbers above are one model on one kernel set. The remaining
question is whether max|d| 0 survives a second family (kernels-1536x2048-b8 exists and is device-proven) — one more row in the
same table would turn "this family is bit-exact" into "the arm is a gate".
Not built: no device, and the harness needs the NPU runtime; this verdict rests on the numbers as stated, their shape, and the
scope paragraph — not on a run.
bong-water-water-bong
left a comment
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at c7c5043
The reason for superseding is a finding in its own right and belongs in the fleet's record: #1888's head froze at 49fcca1 while its branch ref advanced to c7c5043 — two pushes landed, verified with git ls-remote, neither produced a check-runs entry or an Actions run, so the PR's displayed head was not the code being reviewed. That is a failure mode none of the verdict scripts catch: a push that produces no CI at all looks exactly like a push whose checks have not started, and a verdict recorded against the displayed head then certifies code that is not in the branch.
It is also the third variant of tonight's registration/visibility class: #1595 (a test that never runs), #1858 (a test file with no registration), and now a commit for which no CI is ever scheduled. The common question is "what evidence do we have that this ran?", and here the answer was "none, and nothing said so".
The substance is the #1888 change re-expressed on a branch that can be seen: the integer arm's operands come from the engine's packer (q4nx_pack_B_int8_blocked()), and the body says it unlocks the bit-exact gate with device logs — which is the end state the #1809 → #1830 → #1888 chain has been building toward, and the thing that makes maxabs = 0 a claim about the kernel rather than about a reference.
No asks. Consolidation: close #1888 — its change is here, on a head CI can see, and leaving it open would invite a second merge of the same operands fix.
|
Same instrument, second family, now also green: It took a prerequisite this branch could not provide: Qwen3-8B is 36 layers, so the pack was refused until its |
Supersedes #1888, whose head froze at
49fcca1while its branch ref advanced toc7c5043: two pushes landed (verified withgit ls-remote), neither produced acheck-runsentry or an Actions run, and the PR'shead.shastayed at49fcca1through a draft toggle and a body patch. So #1888 cannot be reviewed or merged at the right head — this PR is the same branch under a new name, atc7c5043, so CI and review have something to look at. Nothing was rebased or rewritten; #1888's discussion stays as the record of how the work got here.What this is
An integer reference arm for the NPU forward (
NPU_INFER_HOST_GEMM=i8), and the gate it unlocks: onkernels-h4096-b8with Mistral-7B-Instruct-v0.2 1.9, the device's logits and this arm's agree to the last bit.…and, in a second ticket, three repeats of the device arm (
BIT-EXACTeach), device-vs-device determinism run 1 == run 2 == run 3 (BIT-EXACT), and an edge case (promptHi, all four stepsBIT-EXACT). That isTNPU-ALL§4's bar taken literally rather than read down to "at the accumulator".Why an f32 arm could not do this
The device's weights are int8 codes with one multiplier per column and its accumulators are int32. An f32 host arm therefore differs from it by quantisation alone — measured on this family: cos 0.9960–0.9974 with an argmax flip at step 3, none of which is a kernel fault. The arm's whole value is that it shares the device's arithmetic:
q4nx_pack_B_int8_blocked()(one multiplier per column, int8 codes, blocked order), walked withq4nx_layout::blocked_index();amax/127after the same1/embed_scalefactor the device applies;d[n] * (ascale * acc)— the device's own epilogue, atxdna_target.cpp:1157;max|d| ≈ 0.3residue that survived every body-side fix.The cache is keyed by
(layer, wkey, K, N), because a role key alone is shared by every layer — that mistake made layers 1–31 reuse layer 0's weights and blew the residual up 3300×.What it does not claim
Nothing about throughput (the arm warms up ~35 s on a 7B container and is a reference, not a fast path), and nothing about the other four-h4096 families (
h4096-im12288,h4096-mha,h4096-chatglm): the same gate needs a pack with their geometry, which is a download, not a code change, and this box has ~140 G free against a 150 G floor.Files
npu/forward/src/q4nx_forward.cpp— the integer arm's body path, the integer head (chunked by vocabulary block), and the cache key;npu/forward/include/q4nx_forward.h— the arm's state and the head images;npu/forward/src/q4nx_pack.c,npu/forward/include/q4nx_pack.h—q4nx_host_gemm_i8[,_grouped]on the engine's packed image;tests/npu_pack_test.cpp— the arm's ctest (host, no device);docs/npu.md— what the flag is for, the two-arm command sequence, and the measured table.The conflict with
main's per-128-K int4 rule (#1839) was resolved toward the int8 packer, and the reason is in the merge commit: the question is what the device consumes, and the device consumespackB_blocked.q4nx_quant_int4_*()stays in the tree for the paths that genuinely hold int4 codes.