Skip to content

docs/npu: mark the 18.64 tok/s prefill row retracted, and put the TTFT figure in its place #1665

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
bong-water-water-bong merged 1 commit into from
Nov 9, 2026
Merged

docs/npu: mark the 18.64 tok/s prefill row retracted, and put the TTFT figure in its place #1665

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 9 additions & 2 deletions docs/npu.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -519,14 +519,21 @@ actually applied comes from this box's driver and from the measurement itself.
| measurement | GEMM datatype | tiles | ceiling at **that** datatype | measured | % of that ceiling |
|---|---|---:|---:|---:|---:|
| gate, M=16, K=5120 N=17408 (2.852 GFLOP in 0.589 ms) | int8 | 8 | 8.19 TOPS | 4.84 TOPS | **59 %** |
| prefill, M=128, 18.64 tok/s | bf16 | 32 | 16.4 TOPS | 1.01 TFLOPS | **6.1 %** |
| prefill, M=128, ~~18.64 tok/s~~ — **RETRACTED, see below** | bf16 | 32 | 16.4 TOPS | — | — |
| prefill, M=128, 16.81 tok/s (TTFT, 2048-token prompt) | bf16 | 32 | 16.4 TOPS | 0.91 TFLOPS | **5.5 %** |
| published W4A16 reference, 8.06 TFLOP/s | int8 | 8 | 8.19 TOPS | 8.06 TFLOPS | **98 %** — i.e. an int8-MAC-bound result |

**Retraction notice — 18.64 tok/s is retracted.** It was quoted against *request-wall* figures in later comparisons where the ticket's load window was not
sustained; the replacement is the TTFT-based 16.81 above (`prompt_tokens / TTFT`, distinct prompts, no shared prefix, measured in the first half of the
ticket — so also an upper-band figure). Any page still carrying 18.64 must mark it retracted rather than repeat it. The ceiling arithmetic is unaffected:
the tile count and the per-tile rate are properties of the architecture, and what changes is 2·params ÷ tok/s — 0.91 TFLOPS, i.e. 5.5 % of the 32-tile bf16
ceiling.

The ceiling column is *per datatype* — 1.024 TOPS/tile for int8 and 0.512 for bf16 — so the rows are not comparable to each other by tile count; each % is
against its own datatype's ceiling.

Two readings that follow, and that a whole-array denominator would hide: our **decode** GEMM is at **59 %** of the tiles it occupies, so the remaining
headroom there is the stream (40.2 GB/s measured) and not the MACs; our **prefill** is at **6.1 %**, which is the launch geometry above, not arithmetic.
headroom there is the stream (40.2 GB/s measured) and not the MACs; our **prefill** is at **5.5 %**, which is the launch geometry above, not arithmetic.
And the reference result being ~98 % of an 8-tile int8 ceiling is why 4-bit designs on this architecture are int8 MACs with an on-core expansion:
Table 14 lists **no INT4 multiplier** row, and the array's 4-bit mode — *"Vector unit supporting MAC operations for multiple precisions (for example,
512x 8-bit × 8-bit and 512x 4-bit × 4-bit)"*, Chapter 4's Vector Multiplication Unit — runs at the **same 512-per-tile rate as int8**. So a narrower
Expand Down
Loading