From 0226bcc8ec48e075b691505d6615b041190a0461 Mon Sep 17 00:00:00 2001 From: bong-water-water-bong Date: Mon, 9 Nov 2026 08:54:25 -0400 Subject: [PATCH] docs/npu: mark the 18.64 tok/s prefill row retracted, and put the TTFT figure in its place MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `docs/npu.md`'s AM027 ceiling table carried `prefill, M=128, 18.64 tok/s` as a measurement, and the paragraph under it read "our prefill is at 6.1 %" from that number. 18.64 is retracted: it was quoted against request-wall figures in later comparisons where the ticket's load window was not sustained. The page is the one place a reader goes to judge where this kernel sits against the per-tile ceiling, so the number it divides into 2*params has to be the one that survives. Replace it with the TTFT-based figure and say what it is: 16.81 tok/s, `prompt_tokens / TTFT`, 2048-token prompt, distinct prompts, no shared prefix — 0.91 TFLOPS, 5.5 % of the 32-tile bf16 ceiling (was 1.01 TFLOPS / 6.1 %). The conditions go in the notice because 16.81 is also an upper-band figure: it was measured in the first half of the ticket. The ceiling arithmetic is untouched — 512 INT8 and 256 bf16 multipliers per tile at 1 GHz are properties of the architecture (AM027 Appendix A Table 14, Chapter 1 Table 1), and only the measured column moves. The retracted row stays in the table, struck through and marked, rather than deleted: a reader who saw the old number in another page needs to find it here and learn it was withdrawn, and a row that silently disappears cannot do that. --- docs/npu.md | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/docs/npu.md b/docs/npu.md index 3d55f0af7..99ad5f273 100644 --- a/docs/npu.md +++ b/docs/npu.md @@ -519,14 +519,21 @@ actually applied comes from this box's driver and from the measurement itself. | measurement | GEMM datatype | tiles | ceiling at **that** datatype | measured | % of that ceiling | |---|---|---:|---:|---:|---:| | gate, M=16, K=5120 N=17408 (2.852 GFLOP in 0.589 ms) | int8 | 8 | 8.19 TOPS | 4.84 TOPS | **59 %** | -| prefill, M=128, 18.64 tok/s | bf16 | 32 | 16.4 TOPS | 1.01 TFLOPS | **6.1 %** | +| prefill, M=128, ~~18.64 tok/s~~ — **RETRACTED, see below** | bf16 | 32 | 16.4 TOPS | — | — | +| prefill, M=128, 16.81 tok/s (TTFT, 2048-token prompt) | bf16 | 32 | 16.4 TOPS | 0.91 TFLOPS | **5.5 %** | | published W4A16 reference, 8.06 TFLOP/s | int8 | 8 | 8.19 TOPS | 8.06 TFLOPS | **98 %** — i.e. an int8-MAC-bound result | +**Retraction notice — 18.64 tok/s is retracted.** It was quoted against *request-wall* figures in later comparisons where the ticket's load window was not +sustained; the replacement is the TTFT-based 16.81 above (`prompt_tokens / TTFT`, distinct prompts, no shared prefix, measured in the first half of the +ticket — so also an upper-band figure). Any page still carrying 18.64 must mark it retracted rather than repeat it. The ceiling arithmetic is unaffected: +the tile count and the per-tile rate are properties of the architecture, and what changes is 2·params ÷ tok/s — 0.91 TFLOPS, i.e. 5.5 % of the 32-tile bf16 +ceiling. + The ceiling column is *per datatype* — 1.024 TOPS/tile for int8 and 0.512 for bf16 — so the rows are not comparable to each other by tile count; each % is against its own datatype's ceiling. Two readings that follow, and that a whole-array denominator would hide: our **decode** GEMM is at **59 %** of the tiles it occupies, so the remaining -headroom there is the stream (40.2 GB/s measured) and not the MACs; our **prefill** is at **6.1 %**, which is the launch geometry above, not arithmetic. +headroom there is the stream (40.2 GB/s measured) and not the MACs; our **prefill** is at **5.5 %**, which is the launch geometry above, not arithmetic. And the reference result being ~98 % of an 8-tile int8 ceiling is why 4-bit designs on this architecture are int8 MACs with an on-core expansion: Table 14 lists **no INT4 multiplier** row, and the array's 4-bit mode — *"Vector unit supporting MAC operations for multiple precisions (for example, 512x 8-bit × 8-bit and 512x 4-bit × 4-bit)"*, Chapter 4's Vector Multiplication Unit — runs at the **same 512-per-tile rate as int8**. So a narrower