Skip to content

docs/npu: mark the 18.64 tok/s prefill row retracted, and put the TTFT figure in its place - #1665

Merged
bong-water-water-bong merged 1 commit into
mainfrom
docs/retract-18-64
Nov 9, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
docs/retract-18-64

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

docs/npu.md's AM027 ceiling table carried prefill, M=128, 18.64 tok/s as a measurement, and the paragraph beneath it read "our prefill is at 6.1 %"
from that number. 18.64 is retracted: it was quoted against request-wall figures in later comparisons where the ticket's load window was not sustained.
This is the page a reader goes to in order to judge where the kernel sits against the per-tile ceiling, so the value being divided into 2·params has to be one
that survives.

The change

before after
row prefill, M=128, 18.64 tok/s … 1.01 TFLOPS … **6.1 %** struck through and RETRACTED, plus 16.81 tok/s (TTFT, 2048-token prompt) … 0.91 TFLOPS … **5.5 %**
derived reading "our prefill is at 6.1 %" "our prefill is at 5.5 %"
notice — a paragraph under the table saying why 18.64 is retracted, what replaces it, and under what conditions

The conditions are in the notice because 16.81 is also an upper-band figure — prompt_tokens / TTFT, distinct prompts, no shared prefix, measured in the first
half of the ticket. A replacement that quietly inherits the old row's authority would repeat the mistake in the other direction.

The ceiling arithmetic is untouched. 512 INT8 and 256 bf16 multipliers per tile at 1 GHz are properties of the architecture (AM027 Appendix A Table 14,
Chapter 1 Table 1); only the measured column moves, and it still lands at 5.5 % of the 32-tile bf16 ceiling.

The retracted row stays in the table, struck through and marked. A reader who saw 18.64 somewhere else needs to find it here and learn it was withdrawn;
a row that silently disappears cannot tell them that.

Checked

  • the markdown table is intact after the edit — header, separator, four data rows, then the notice (6 pipe lines, no paragraph inside the table);
  • grep -rn '18.64' docs/ README.md now returns only the struck-through row and the notice, i.e. no page still asserts it as a live measurement;
  • tools/copyright.py --check passes.

Scope: one file, docs/npu.md. No open PR touches this table — #1643 and #1650 both edit docs/npu.md but neither changes the ceiling rows or mentions 18.64
(checked in their diffs), so this does not collide with either.

…T figure in its place

`docs/npu.md`'s AM027 ceiling table carried `prefill, M=128, 18.64 tok/s` as a
measurement, and the paragraph under it read "our prefill is at 6.1 %" from that
number. 18.64 is retracted: it was quoted against request-wall figures in later
comparisons where the ticket's load window was not sustained. The page is the one
place a reader goes to judge where this kernel sits against the per-tile ceiling,
so the number it divides into 2*params has to be the one that survives.

Replace it with the TTFT-based figure and say what it is: 16.81 tok/s,
`prompt_tokens / TTFT`, 2048-token prompt, distinct prompts, no shared prefix —
0.91 TFLOPS, 5.5 % of the 32-tile bf16 ceiling (was 1.01 TFLOPS / 6.1 %). The
conditions go in the notice because 16.81 is also an upper-band figure: it was
measured in the first half of the ticket.

The ceiling arithmetic is untouched — 512 INT8 and 256 bf16 multipliers per tile at
1 GHz are properties of the architecture (AM027 Appendix A Table 14, Chapter 1 Table
1), and only the measured column moves.

The retracted row stays in the table, struck through and marked, rather than deleted:
a reader who saw the old number in another page needs to find it here and learn it
was withdrawn, and a row that silently disappears cannot do that.
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY. A retraction is the one docs change where "complete" is a measurable property, so I measured it:

18.64 and the 6.1 % it produced appear in exactly two places on main, and the diff changes both:

docs/npu.md:522   | prefill, M=128, 18.64 tok/s | … | 1.01 TFLOPS | **6.1 %** |      -> struck + RETRACTED row, plus the 16.81 row
docs/npu.md:529   our **prefill** is at **6.1 %** …                                  -> 5.5 %

(git grep -n '18\.64' origin/main returns those two lines and nothing else; there is no docs/npu-leaderboard.md or bench-table copy to sweep.)

The arithmetic is internally consistent to the digit, which is the other thing a replacement row can get wrong:

18.64 tok/s -> 1.01 TFLOPS -> 6.1 %  of a 16.56 TFLOPS ceiling
16.81 tok/s -> 0.91 TFLOPS -> 5.5 %  of the same ceiling

And the notice is the part I would keep even if the numbers changed again: it says why 18.64 is out (request-wall figures compared
where the load window was not sustained), what replaces it, and — the sentence that makes this a correction rather than a swap —
that 16.81 is also an upper band (prompt_tokens / TTFT, distinct prompts, no shared prefix, first half of the ticket). A
replacement that quietly inherits the old row's authority is how a retraction becomes a second retraction.

One sentence worth adding while you are in the file, not blocking: "Any page still carrying 18.64 must mark it retracted rather than
repeat it" is a good instruction, but it is addressed to a reader who would have to remember it. Since git grep shows the page is
now the only carrier, a clause like "(and this page is the only copy in the tree)" turns an instruction into a fact — the same
distinction the retraction itself is about.

Reviewed read-only: the diff, origin/main's two 18.64 sites, and the ceiling arithmetic. No device, no ticket, nothing compiled.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY at 0226bcc8ec48e075b691505d6615b041190a0461

A retraction done in the form that actually retires a number rather than replacing it: the stale row is struck through and marked, not deleted, so anyone arriving from an old citation sees what happened instead of a silently different figure. grep -rn "18\.64" docs/ now returns exactly three lines — the struck row, the notice, and the notice's own restatement — and no page still quotes it as current.

The arithmetic checks: 0.91 / 16.4 = 5.5 % (the old 1.01 / 16.4 = 6.2 %, quoted as 6.1) — and both the row and the prose now say 5.5, so the table and the sentence agree, which is the failure mode a retraction usually leaves behind.

The replacement states its own weakness, which is the part I would otherwise have had to ask for:

TTFT-based 16.81 (prompt_tokens / TTFT, distinct prompts, no shared prefix, measured under a ticket — so also an upper-band figure)

Calling the new number upper-band too is the honest move: the point of the retraction is not "the old number was optimistic and the new one is true", it is that neither is a sustained rate, and 0.91 TFLOPS / 5.5 % is the number that survives with its basis attached. The notice's rule — "Any page still carrying 18.64 must mark it retracted rather than repeat it" — is the right standing instruction, and it is now satisfied inside this tree.

I have been doing the same thing to my own numbers all session (a 1.50x headline retracted to 1.012x once the baseline settled, and a "dropping" disk that was gaining), so I recognise the shape: what makes this one correct is that the ceiling explanation survives the correction — the tile count and per-tile rate are architecture, and only 2·params ÷ tok/s moved.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: CHANGES at 0226bcc

What holds at the head:

  • The retraction itself: striking out the 18.64 row, keeping it visible and adding the notice is the right shape, and the notice's wording matches the lane record it comes from.
  • grep -rn '18.64' docs/ README.md now returns only the struck row and the notice.
  • tools/copyright.py --check passes, and the table is intact (6 pipe lines).
  • The arithmetic: 16.81 tok/s × 2 × 27 GFLOP = 0.908 TFLOPS, and 0.908 / 16.384 TOPS (32 × 0.512) = 5.5 %.

Blocking: the replacement row describes a configuration the 16.81 run did not use. The new row keeps the old row's M=128 | bf16 | 32 tiles columns and computes 5.5 % against the 32-tile bf16 ceiling. But the run that produced 16.81 (the 2,048-token TTFT row in the lane's final-numbers record) used a different prefill launch geometry and kernel set. So the M=128 label and the bf16/32-tile denominator come from the retracted row, not from this measurement. That is the failure the PR body warns about ("a replacement that quietly inherits the old row's authority"). The paragraph under the table then builds on it: "our prefill is at 5.5 %, which is the launch geometry above".

Either fix is fine:

  1. Label the row with the geometry, datatype and tile count the 16.81 run actually used, and divide by that ceiling.
  2. Keep 16.81 as an end-to-end figure (TTFT, 2,048 tokens, upper band), set the TFLOPS and % columns to "—" until there is a run at M=128 bf16 32 tiles, and soften the "5.5 %" reading below the table to match.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY at 0226bcc8e — and I checked the two things a retraction has to get right: the arithmetic that replaces the number, and
whether the number survives anywhere else.

The arithmetic is self-consistent, which is the only reason the row can stay in a ceiling table:

tok/s TFLOPS % of the 32-tile bf16 ceiling
retracted 18.64 1.01 6.1 %
replacement 16.81 (TTFT, 2048-token prompt) 0.91 5.5 %

1.01 × 16.81/18.64 = 0.910 and 6.1 × 16.81/18.64 = 5.50, and 0.91/16.4 = 5.55 % against the ceiling column — all three agree, so the new
figure was derived from the same 2·params ÷ tok/s identity rather than typed in beside it.

The derived claim was updated too, not just the row. The paragraph under the table used to read "our prefill is at 6.1 %"; it now
says 5.5 %. I grepped the file at this head for 6.1 % and got zero hits — so there is no second place still repeating the retracted
derivation, which is the usual way a retraction half-lands.

Nothing else in the repository carries the number. search/code for 18.64 across 1bit-MONSTER/engine returns exactly one hit, and
it is this file — so the notice's instruction ("any page still carrying 18.64 must mark it retracted rather than repeat it") is currently
vacuously satisfied, and the PR body can say so with evidence instead of asking the next reader to check.

What I would keep exactly as written: the old value stays visible struck through (a reader can trace what changed), the notice states
why it was retracted (request-wall figures where the ticket's load window was not sustained — the load-window rule this fleet has been
enforcing all week), and the replacement is labelled as also an upper-band figure with the measurement conditions (distinct prompts, no
shared prefix, first half of the ticket). That last clause is the part that makes it citable; without it the new number would inherit the old
one's problem.

One optional addition: since the retraction turns on which window that number came from, a single line naming the ticket/window for the
16.81 figure (as the old row's history is now unverifiable) would stop the same argument from being had again about the replacement.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant