[bong-water-water-bong]

Refs #1990 (second host term; this is the one #2005's "What remains" called unnamed).

What it was

Every launch's result is turned into floats as float32(S * 2^e_ref) * ascale. 2^e_ref depends only on the column, but it was computed with a libm ldexp() call per element. That code runs after readback, so none of the launch phase columns (pack, ql, wait, rb) counted it, and the batched [lh] table printed it under UNATTRIBUTED.

A 16-row prefill block on the served 27B pack is 416 launches of up to 16 x 17,408 elements. Measured, this was the block's largest host term.

What changes

q4nx_pb16i::dequant_rows() computes the power once per column and multiplies. I8Ctx::dequant_pb16i() calls it, so the layer launches and the head both use it. A second zero-fill of the output buffer before the dequant is dropped (the buffer is already sized, and every element is written).

The result is the same bits, not an approximation: a double times a normal power of two is exact while the product is a normal double, which holds for every exponent from -1022 to 960. A column outside that range sends the whole launch through the per-element form, as before.

The [bl] line gains a deq field (LH_DEQ_OUT), so the term is named from now on. NPU_DEQ_LDEXP=1 keeps the old form for an A/B; NPU_DEQ_LDEXP=2 runs both and counts differing elements.

Measured

bonsai2-27b-t2s2, one npu-hold ticket (466 s), one binary, arms A B B A (A = NPU_DEQ_LDEXP=1), 103 16-row blocks per arm, NPU_PHASE_PROF=1. amdxdna error count 2561 -> 2561.

arm	dequant per block (mean)	block wall (median)	host outside launch phases (median)	TTFT tok/s (3 prompts)
A1 ldexp	137.4 ms	630.4 ms	294.5 ms	25.11, 24.51, 24.40
B1 per-column	32.3 ms	515.0 ms	186.9 ms	29.77, 30.46, 30.60
B2 per-column	51.9 ms	620.9 ms	217.8 ms	30.04, 19.84, 16.65
A2 ldexp	360.2 ms	1363.9 ms	776.3 ms	11.83, 11.82, 11.13

The claim is the first pair: block median 630.4 -> 515.0 ms (25.4 -> 31.1 rows/s), TTFT 24.4-25.1 -> 29.8-30.6 tok/s.

The box slowed partway through B2 and stayed slow through A2 (device wait rose too, 212 -> 330 ms), so I do not quote the second pair as a speedup. The same thing happened in the second half of my two earlier tickets today; I have not established whether that is other lanes' load or something about a long ticket. Order was old-form first, so a warm-up or thermal drift would count against this change, not for it.

Bit identity on the device, same ticket (NPU_DEQ_LDEXP=2):

[deqcheck] calls=8000 elements=568483840 differing=0

Greedy 8-token text sha256 is d819d7c8ae824fb6 in all five arms.

Host microbench of the two forms alone (M=16, one plane / two planes):

N= 5120: ldexp 0.145 / 0.160 ms   per-column 0.021 / 0.043 ms   bit-identical
N=10240: ldexp 0.295 / 0.321 ms   per-column 0.043 / 0.086 ms   bit-identical
N=17408: ldexp 0.503 / 0.538 ms   per-column 0.073 / 0.142 ms   bit-identical

One thing to know when reading the table: the measured binary also carried #2005's attention change, in every arm. This branch is that change removed and rebased onto main; I built it (1bit links) and ran its unit test, but did not repeat the device run on this exact tree.

Tests

tests/npu_pb16i_layout_test.cpp (already run by CI as npu_pb16i_layout) gains an arm: dequant_rows() against scale_i64() per element, compared as bit patterns, for one and two planes, container-like exponents, both edges of the table range, exponents below and above it, and sums that include 0, +-1 and the int32 extremes.

Mutants: upper bound raised past the exact range -> FAIL; ascale folded into the double multiply -> FAIL; wrong column's power -> FAIL. Lowering the lower bound does not fail, and cannot: anything that small rounds to a float zero in both forms. The bound stays at -1022 because that is where the exactness argument holds.

🤖 Generated with Claude Code

[bong-water-water-bong]

ready for review at 93d5843

[bong-water-water-bong]

VERDICT at 93d5843: READY

Checked at 93d5843. A three-way merge onto main is clean.

The "same bits" claim holds, by reading and by test. scale_i64(s, e) is (float)std::ldexp((double)s, e), and the table form is (float)((double)s * ldexp(1.0, e)). Both take the same (double)s first (so a two-plane sum past 2^53 rounds identically), scale once by an exact power of two, and round to float once before * asc. So they agree whenever the product is a normal double. With |s| < 2^63, that holds for e in [-1022, 960], which is what the guard tests. The two-plane recombination is sum_of()'s (hi << 21) + lo, verbatim.

• tests/npu_pb16i_layout_test.cpp (NPU-private target, compiled directly with g++-16): "ok (8 geometries)". The new arm covers both planes, the int32 extremes, values past 2^53, and exponents at and beyond both table edges.
• Mutants:
  • kExpHi = 1100 fails "exponents above the table range … (3 of 128 differ)", because 0 * inf gives NaN where ldexp(0, e) gives 0;
  • kExpLo = -1100 still passes. That's correct, not a gap: below 2^-1022 the double product and ldexp both land far under float's range and round to the same float. The low bound is conservative.
• npu/target/xdna/xdna_target.cpp and npu/forward/src/q4nx_forward.cpp compile at this head (single-file, with an existing NPU build's commands pointed at this tree).

The measurement reads the same way as #2005's, and is written up just as honestly: only A1→B1 is claimed. The box slowed through B2/A2, and with the old form run first, drift would count against the change. The device check ([deqcheck] … differing=0 over 568 M elements) plus the identical greedy sha256 across arms is the right evidence for a "same bits" change.

[bong-water-water-bong]

VERDICT: READY at 2008's head 93d5843 — a perf fix whose cost was invisible because no phase column counted it, which is the interesting part.

+**That single rounding costs one multiply per element, not one `ldexp()` (#1990).** A launch's result is
+`float32(S * 2^e_ref) * ascale` for every element, and `2^e_ref` depends only on the column. It was computed
+with a libm `ldexp()` call per element, after readback, where no launch phase column counted it: a 16-row
+block's largest host term. `q4nx_pb16i::dequant_rows()` computes the power once per column and multiplies.

Why this was hard to see, and why the fix is safe

• it happened after readback, so it was in none of the launch's phase columns — the profile said "device wait" and "host pack" and neither included it. That is how a per-element libm call survives: it is not in the accounting, so no report is wrong enough to look wrong;
• 2^e_ref is per column, so hoisting it is not an approximation: the multiply is (S * 2^e) * ascale in the same order the code already used, and the power is a column constant. A power-of-two multiply is exact, so this is a pure de-duplication of work — no numerical change, which is exactly what a bit-exactness gate requires;
• the name of the caller is in the comment (q4nx_pb16i::dequant_rows()), so the reader can find the new shape without diffing.

The ask, which is about the instrument

The phase columns should gain an entry for post-readback host work — otherwise the next per-element call hides in the same gap. A perf fix that leaves the measurement blind to its own class will be followed by the same bug somewhere else.