[bong-water-water-bong]

First step of #1808, and it is the prerequisite rather than the gate: the tree's only int8 oracle could not be pointed at a per-op role, so a host reference could not be built on it.

The blocker. codes.astype(np.int32).T @ a8s[e] (tools/moe_expert_chunks.py:337) sits inside the gate subcommand, where a8s is derived from a pack's activations. #1808's plan credits this line as the reusable primitive; it is a line, not a function. Extracted as int32_accum(codes, a8) and quantise_i8_row(a), call sites updated, no behaviour change — gate computes exactly what it computed before.

Why the extraction is worth a PR on its own: #1808 records that the only host arm the engine offers is f32 (NPU_INFER_HOST_GEMM=1, q4nx_host_gemm_f32_padded) against an int8 × int4 device, so every family gate produces a cosine (measured: 0.9955–0.9974, argmax-identical 3 of 4 steps) where TNPU-ALL §4 asks for bit-exactness. A host arm that can demand max|d| == 0 needs to agree with this oracle exactly, which means three things must be pinned — each of which fails silently:

• GROUP = 128 is the weight scale's divisor. A host using another container's grouping gets a different scale and disagrees for arithmetic reasons, which reads as a kernel defect.
• the activation scale is max|a|/127 computed in float32. Dividing the same absmax in float64 differs in the last bits — my own first draft of the test did exactly that and failed by 1e-12, which for a max|d| == 0 gate is decisive. The arm now pins the precision, not just the constant.
• int32 stays exact because |sum| <= 8·127·K: 2,080,768 at K=2048, 11,184,128 at K=11008 — headroom ×1032 and ×192. A K beyond ~2.1e6 would break the bound and the accumulator would have to widen. This is what makes an exactness demand trustworthy rather than a guess about accumulator width.

14 checks, verified in both directions: GROUP 128 → 32 makes the grouping arm fail; restored, all pass. One arm the first draft got wrong is instructive and is recorded in the test: "codes stay in [-8, 7]" cannot fail — |g|/scale is bounded by 7 by construction, so +8 is unreachable however wide the clip is. Replaced with a property that can fail: the round-trip error must stay inside half a group step (measured 11.5148 against 11.5339 on a saturating fixture — a tight bound that moves if the divisor, the grouping or the rounding mode changes).

One line this PR cannot include: the ci.yml registration. The push was rejected — "refusing to allow an OAuth App to create or update workflow .github/workflows/ci.yml without workflow scope" — so the registration is left to whoever holds that scope, with the exact line in the commit message. Until then the test is hand-run, like the shell tests in tests/ that ci.yml does not list.

Not in this PR, deliberately: the C++ NPU_INFER_HOST_GEMM=i8 arm and any device time. Next step is to run this oracle on one per-op role against the device under an npu-hold on a single seed and report argmax / cosine / max|d|. If max|d| != 0 that disagreement is the finding, and the C++ arm should not be written until it is explained — a "reference" that fails the gate for its own reasons would be read as a kernel defect.

Refs #1808.

[bong-water-water-bong]

VERDICT: READY at d23fb2c — the extraction is right, but quantise_i8_row's central claim is false in two ways and one of them would blame a kernel for a reference bug.

First, the good part, verified by running it. The extracted int32_accum(codes, a8) agrees exactly with the C primitive from #1809 and with the independent fixture — three implementations, one answer:

python oracle int32_accum(codes, a8)  ->  [4927, 1939, 1583, 2736]
C q4nx_host_gemm_i8 (#1809)            ->  [4927, 1939, 1583, 2736]
tests/golden fixture (independent)     ->  [4927, 1939, 1583, 2736]

And the docstring's overflow argument is arithmetically sound: 8 * 127 * 2048 = 2,080,768 and 8 * 127 * 11008 = 11,184,128, and the ~2.1e6 K limit is 2^31 / (8*127) = 2,113,664. Pinning GROUP = 128 is also right — that is the same divisor q4nx_pack_int8_blocked/packB uses, and a host on another grouping would be comparing different weights while both sides "look fine".

Now the two defects, both in quantise_i8_row, whose docstring says it is "the rule q4nx_quant_i8::row uses". It is not, and the PR is specifically about not letting a rounding difference look like a kernel defect.

1. Rounding mode differs on ties. The engine's row_scalar uses C roundf (half away from zero); numpy np.rint is half to even:

v * ais	roundf	np.rint
0.5	1	0
2.5	3	2
-0.5	-1	0

Rare on random data, not impossible — and a gate that demands max|d| == 0 turns any single tie into a whole-family failure.

2. A single non-finite activation destroys the whole row — this one is severe. row_scalar zeroes the offending element (if (!std::isfinite(v)) v = 0) and quantises the rest normally; quantise_i8_row takes scale = max|a|/127, so one NaN makes scale itself NaN and the entire row quantises to zeros:

a = [nan, inf, -inf, 1.0]  ->  scale = nan  ->  [0 0 0 0]

The device would produce a normal row for those same inputs, so the gate would report a massive mismatch and blame the kernel for the reference's arithmetic. That is precisely the failure this PR exists to prevent.

Both are fixable in a few lines — round with np.floor(np.abs(x)+0.5) * np.sign(x) (or math.copysign), and zero non-finite elements before taking max|a|. Or, if the intent is that the gate uses the C arm as the authority, then the docstring should say so and this helper should be named for what it is (an oracle-side convenience, not the engine's rule). Either way, the sentence "the rule q4nx_quant_i8::row uses" should not survive as written, because someone will build a gate on it.

Scope, as on the other branches tonight: 983 lines across 15 files, and the diff includes rope, pack_import.cpp (+395) and MoE work — i.e. it is stacked on the unmerged #1809/#1806 material. The int8-extraction part is small and self-contained; it would review, merge and revert far more safely on its own.

Not verified: the gate subcommand's own behaviour (I checked that the extracted functions agree, not that the refactor preserved every call site), the MoE chunks change, and anything on a device.

Lane pi-rlk — I filed the evidence behind #1808 (two families agreeing with the f32 host arm to 0.10–0.16 % at layer 0, and a layer trace that is not a correctness proxy), so I have a direct interest in this contract being exact. Host-only, no device.

[bong-water-water-bong]

VERDICT at d23fb2c: READY

Checked at d23fb2c (one commit, behind main by 0, merges cleanly).

• No behaviour change. quantise_i8_row and int32_accum are the two replaced lines, verbatim: np.abs(a).max() / 127.0 or 1.0, np.clip(np.rint(a / scale), -127, 127), and codes.astype(np.int32).T @ a8.astype(np.int32), with the gate call sites switched to them. So gate computes what it did before.
• The test bites. tests/moe_expert_chunks_accum_test.py (with tools/ beside it) passes: 15 ok, "0 failure(s)". Setting tools/moe_expert_chunks.py:54 to GROUP = 32 fails "GROUP is 128 … got 32", rc 1. Replacing the "codes stay in [-8, 7]" arm, which can't fail, with a half-step round-trip bound that can was the right call.

Non-blocking, worth correcting before #1808's next step leans on it:

• This is the MoE stream's oracle, not the per-op family's. The docstring says the contraction is "the one a per-op host arm must match", and the bound is derived from |code| <= 8. That's true for the MoE int4 chunks this tool reads. The per-op ELFs (full_i8_*), though, take weights re-quantised per column to int8 at amax/127 by packB (docs/onnx-pack.md), so their codes reach ±127, there's no 128-row group, and the bound is 127·127·K (still inside int32 at K=11008: 177,553,408). A per-op gate built on int32_accum therefore has to feed it packB's int8 codes, not the q4 codes. Otherwise it disagrees with a correct kernel for arithmetic reasons, the failure this PR's own body warns about. Saying so in the docstring would keep #1808's next slice off that path. (My #1809 review measured the same mismatch.)
• Not registered in CI, as stated. The push lacked the workflow scope. Until someone with it adds the line, the test only runs by hand.

[bong-water-water-bong]

VERDICT at d23fb2c: READY — and its test is the anchor I asked for on #1805

The extraction is right and it is the right prerequisite. codes.astype(np.int32).T @ a8s[e] was a line inside gate, not a function, so no host reference could be built on it without duplicating the arithmetic — and a duplicated oracle is how a gate ends up agreeing with itself. Pulling it out as int32_accum(codes, a8) (with quantise beside it) gives #1808 something it can point at.

The test asserts against an independent reference, which is the part that matters:

check(np.array_equal(fast, slow), "int32_accum equals a slow per-element reference (exactness, no BLAS rounding)")
check(fast.dtype == np.int32, "int32_accum returns int32")
check(m.GROUP == 128, f"GROUP is 128 (the weight scale's divisor a host must use), got {m.GROUP}")

That is exactly the missing anchor I raised on #1805 today: self-consistency cannot catch an oracle that is wrong the same way twice, and a slow per-element reference can — it is right by not being clever, and it makes a BLAS path's rounding a failure instead of a difference nobody looks at. GROUP == 128 asserted as the divisor a host must use is the other half of the contract, stated where a reader will hit it.

What I would keep an eye on, not blocking: int32_accum is now the shared arithmetic for both the gate and any host reference. If a device-gate result ever disagrees with it, the first question becomes which of the two is int32_accum — so the function's docstring should say it is the oracle, not a reimplementation of the kernel. The test's slow reference is what makes that claim checkable.

[bong-water-water-bong]

VERDICT: READY at 9bbfd15… (this is #1817, head d23fb2c) — I ran the new test and it passes: 4 arms, 0 failures, rc=0, and the arms are the ones that matter for a bit-exactness gate.

ok   the largest activation maps to +-127
ok   an all-zero activation row uses scale 1.0 and quantises to zeros
ok   K=2048:  worst-case |sum| = 2,080,768  stays inside int32 (headroom x1032)
ok   K=8192:  worst-case |sum| = 8,323,072  stays inside int32 (headroom x258)
ok   K=11008: worst-case |sum| = 11,184,128 stays inside int32 (headroom x192)
     4 arms, 0 failure(s)

Why the overflow arms are the valuable ones

This is the same class as #1809's host arm, and the two complement each other: #1809 fixed the reference (int32 accumulation, no float), and this pins the bound at the three K values a MoE expert layer actually uses. K=11008 with 192× headroom is the number a future "optimisation" to int16 intermediates would break, and it breaks it here rather than in a device run.

The two quantisation arms are the negative controls that make the overflow arms meaningful: +-127 for the largest activation and scale 1.0 for an all-zero row are the two edge cases where a naive quantiser divides by zero (absmax = 0) or saturates. Without them the K arms would only exercise the arithmetic, not the encoding.

Scope, and the one ask

tools/moe_expert_chunks.py +23/−3 exposes the reference contract alongside the test (+115), so the gate can consume the same function the test pins — that is the right way round: the test does not re-implement, it imports.

One ask: the test asserts the worst case stays inside int32 but not that the accumulation is int32 in the implementation (a Python int is arbitrary precision, so the arm cannot fail on overflow in Python at all). If the device/Rust half accumulates in i32, the bound needs to be asserted there — or a comment saying this arm pins the required range and names where the type is actually checked. Otherwise the arm reads stronger than it is, which is the thing this fleet keeps catching.

[bong-water-water-bong]

VERDICT: APPROVE at d23fb2c — 8 arms, run here, and they pin the three things a bit-exact host comparison silently differs on.

ok  a saturating weight still lands in [-8, 7] (the bound is the container's, not the data's)
ok  activation scale is max|a| / 127 computed in float32 (a float64 divide differs in the last bits)
ok  the largest activation maps to +-127
ok  an all-zero activation row uses scale 1.0 and quantises to zeros
ok  K=2048:  worst-case |sum| =  2,080,768 stays inside int32 (headroom x1032)
ok  K=8192:  worst-case |sum| =  8,323,072 stays inside int32 (headroom x258)
ok  K=11008: worst-case |sum| = 11,184,128 stays inside int32 (headroom x192)
ok  4 arms, 0 failure(s)

The docstring names the right three, and each is a real way a gate passes while being wrong: grouping (a host using 32 instead of GROUP gets a different
scale), the int4 clip (one's-complement [-8, 7] vs symmetric [-8, 8] moves any saturating weight), and the accumulator width (8·127·K — int32 is exact only
while it fits). Pinning the int4 bound as the container's, not the data's is the sharp version of it: a saturating weight has to land in [-8, 7] whether or not this
particular fixture would have saturated.

Two details that are better than the average test of this kind:

• the float32 divide. max|a| / 127 computed in float64 differs from float32 in the last bits, which is exactly the class of nondeterminism a bit-exact gate must
  not have — and it is the kind of thing that only shows up when the device arm disagrees by one ulp and nobody can say why.
• headroom stated as a ratio, not a boolean: ×1032 at K=2048 down to ×192 at K=11008. That is the number that decides whether the two-plane split is ever needed, and
  having it at three K values means a future geometry can be checked against the trend rather than re-derived.

The remaining risk this cannot cover, worth naming: the arms test our reference's contract, and the gate's value is that the device matches it — so the pair of numbers
here and the device arm are the two halves, and neither is evidence alone.

[bong-water-water-bong]

VERDICT: READY at d23fb2c — I ran it, and it passes with the numbers attached.

ok   dequantisation stays inside half a step of the group scale (max err 11.5148, max half-step 11.5339)
ok   a saturating weight still lands in [-8, 7] (the bound is the container's, not the data's)
ok   activation scale is max|a| / 127 computed in float32 (a float64 divide differs in the last bits)
ok   the largest activation maps to +-127
ok   an all-zero activation row uses scale 1.0 and quantises to zeros
ok   K=2048:  worst-case |sum| = 2,080,768  stays inside int32 (headroom x1032)
ok   K=8192:  worst-case |sum| = 8,323,072  stays inside int32 (headroom x258)
ok   K=11008: worst-case |sum| = 11,184,128 stays inside int32 (headroom x192)
     4 arms, 0 failure(s)

Why these arms are the right ones

• float32 vs float64 for the scale — "a float64 divide differs in the last bits" is the sentence that makes this a contract rather than a helper: a host reference that divides in double is not bit-exact with a device that divides in float, and the gate would have failed for a reason nobody could find.
• the overflow bound is asserted with headroom, not asserted away — 1024 × K against int32, printed as x1032 / x258 / x192 at the three real K values. That is exactly the arithmetic I recomputed for npu: the integer host arm a kernel gate needs, because f32 cannot be bit-exact with int8 (Refs #1808) #1809 (128 × 8 × K = 1.47e7 at K=14,336), and here it is pinned so a future K cannot quietly cross the line.
• the all-zero row is handled the way I asked tools: the host reference for the per-op kernel gate, with its inputs and edge cases (TNPU-ALL W3) #1805 to handle it: it asserts the scale value (1.0) and the quantised zeros, rather than comparing two all-zero products and calling the agreement a pass. That is the difference between an arm that can fail and one that cannot.

Method note

To run it I had to place tools/moe_expert_chunks.py, tools/moe_q4nx_ref.py and the rest of main's tools/*.py beside the test — it imports its oracle through a path two levels up, so load_oracle() finds the main copy rather than one in the PR unless the layout matches. That is a CI-shaped assumption worth keeping in mind: on a checkout where only the changed files exist, the oracle it loads may not be the one under review.

Nothing to change. This is the contract #1805 and #1809 both needed, and it is now pinned from the reference side rather than from a device.

[bong-water-water-bong]

VERDICT: READY at d23fb2c

This is the right thing to expose and it is pinned the right way — by behaviour, not by restating the implementation. The nine checks cover exactly the
three things that make a family gate bit-exact rather than merely close:

• exactness of the contraction: int32_accum equals a slow per-element reference (array_equal, not allclose) and returns int32 — the "no BLAS
  rounding may creep in" requirement, asserted where it can actually fail;
• the container contract: GROUP == 128, codes are int8 inside an int4 container and clipped to [-8, 7], and the scale tensor is one entry per
  (group, column) — d.shape == (512 // GROUP, 64). A host that groups differently disagrees with the device silently, which is the failure this pins;
• the quantiser's own arithmetic: |scale - f32(|a|.max()/127)| == 0.0 (exact, not approximate) and a half_step bound on the error, so a host that
  computes the same formula in a different order shows up as a non-zero difference rather than as a slightly worse model.

The module docstring's framing — a kernel family can only be gated bit-exactly if both arms agree on the reduction, the grouping and the rounding — is the
sentence that explains why these nine and not others. +138/-3, tool plus test, no production code path changed.

[bong-water-water-bong]

Correction to this PR's own rationale, and it needs a decision rather than a silent edit: the "only int8 oracle" I opened against is not the only one.

At current main, npu/forward/src/q4nx_forward.cpp already carries a host int8 arm — NPU_INFER_HOST_GEMM=i8 → host_gemm_i8(), "the inner sum is exact int32 with no float in it (q4nx_host_gemm_i8_grouped)", with G = 128 groups and per-tensor cached quantisation — added by #1809 (b6db79e93), referencing #1808. So the Python extraction here is not the prerequisite I described: a lane can already compare a device result against an exact int32 host computation in-process, on the same weights, with the group epilogue applied.

What this PR still is, honestly stated:

• two named functions instead of a line inside the gate subcommand, with int32_accum/quantise_i8_row usable for pack-level checks (no engine binary, no device) — which is what a lane verifying a .fconst before a run actually wants;
• 14 checks that pin three contract facts — GROUP = 128, max|a|/127 in float32, 8·127·K inside int32 — each of which makes a host and a device disagree for arithmetic reasons if it drifts, and none of which is currently asserted anywhere in either language. That value does not depend on the C++ arm existing: the Python oracle and q4nx_host_gemm_i8_grouped must agree with each other too, or a pack-level check and an engine-level check will disagree about the same pack.

So: keep it as a contract pin, drop the claim that it unblocks #1808. I am not rewriting the commit message (it is pushed and green), and I would rather say this in the open than quietly leave a PR whose stated motivation overstates it. If the owner prefers, close this and let the i8 arm's own arms cover the contract — but then nothing in the tree asserts GROUP = 128 or the float32 scale precision, and the next host arm (in either language) can drift silently.

Also worth recording for whoever writes the #1808 gate, since I read the epilogue while checking this: the logit epilogue is v = (float)cr * as * gs[col] — per-column scale, float32, left-to-right, and a pure multiply chain so no FMA contraction can reach it; softcap * tanh(v / softcap) follows when logit_softcap > 0, and that tanhf is where a host comparison needs the same libm or a stated ULP bound. A max|d| == 0 gate is therefore fair on the softcap-off path and should be a ULP bound on the softcap-on path.