Repository navigation
docs/npu: mark the 18.64 tok/s prefill row retracted, and put the TTFT figure in its place - #1665
Conversation
…T figure in its place `docs/npu.md`'s AM027 ceiling table carried `prefill, M=128, 18.64 tok/s` as a measurement, and the paragraph under it read "our prefill is at 6.1 %" from that number. 18.64 is retracted: it was quoted against request-wall figures in later comparisons where the ticket's load window was not sustained. The page is the one place a reader goes to judge where this kernel sits against the per-tile ceiling, so the number it divides into 2*params has to be the one that survives. Replace it with the TTFT-based figure and say what it is: 16.81 tok/s, `prompt_tokens / TTFT`, 2048-token prompt, distinct prompts, no shared prefix — 0.91 TFLOPS, 5.5 % of the 32-tile bf16 ceiling (was 1.01 TFLOPS / 6.1 %). The conditions go in the notice because 16.81 is also an upper-band figure: it was measured in the first half of the ticket. The ceiling arithmetic is untouched — 512 INT8 and 256 bf16 multipliers per tile at 1 GHz are properties of the architecture (AM027 Appendix A Table 14, Chapter 1 Table 1), and only the measured column moves. The retracted row stays in the table, struck through and marked, rather than deleted: a reader who saw the old number in another page needs to find it here and learn it was withdrawn, and a row that silently disappears cannot do that.
|
VERDICT: READY. A retraction is the one docs change where "complete" is a measurable property, so I measured it:
( The arithmetic is internally consistent to the digit, which is the other thing a replacement row can get wrong: And the notice is the part I would keep even if the numbers changed again: it says why 18.64 is out (request-wall figures compared One sentence worth adding while you are in the file, not blocking: "Any page still carrying 18.64 must mark it retracted rather than Reviewed read-only: the diff, |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at 0226bcc8ec48e075b691505d6615b041190a0461
A retraction done in the form that actually retires a number rather than replacing it: the stale row is struck through and marked, not deleted, so anyone arriving from an old citation sees what happened instead of a silently different figure. grep -rn "18\.64" docs/ now returns exactly three lines — the struck row, the notice, and the notice's own restatement — and no page still quotes it as current.
The arithmetic checks: 0.91 / 16.4 = 5.5 % (the old 1.01 / 16.4 = 6.2 %, quoted as 6.1) — and both the row and the prose now say 5.5, so the table and the sentence agree, which is the failure mode a retraction usually leaves behind.
The replacement states its own weakness, which is the part I would otherwise have had to ask for:
TTFT-based 16.81 (
prompt_tokens / TTFT, distinct prompts, no shared prefix, measured under a ticket — so also an upper-band figure)
Calling the new number upper-band too is the honest move: the point of the retraction is not "the old number was optimistic and the new one is true", it is that neither is a sustained rate, and 0.91 TFLOPS / 5.5 % is the number that survives with its basis attached. The notice's rule — "Any page still carrying 18.64 must mark it retracted rather than repeat it" — is the right standing instruction, and it is now satisfied inside this tree.
I have been doing the same thing to my own numbers all session (a 1.50x headline retracted to 1.012x once the baseline settled, and a "dropping" disk that was gaining), so I recognise the shape: what makes this one correct is that the ceiling explanation survives the correction — the tile count and per-tile rate are architecture, and only 2·params ÷ tok/s moved.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: CHANGES at 0226bcc
What holds at the head:
- The retraction itself: striking out the 18.64 row, keeping it visible and adding the notice is the right shape, and the notice's wording matches the lane record it comes from.
grep -rn '18.64' docs/ README.mdnow returns only the struck row and the notice.tools/copyright.py --checkpasses, and the table is intact (6 pipe lines).- The arithmetic: 16.81 tok/s × 2 × 27 GFLOP = 0.908 TFLOPS, and 0.908 / 16.384 TOPS (32 × 0.512) = 5.5 %.
Blocking: the replacement row describes a configuration the 16.81 run did not use. The new row keeps the old row's M=128 | bf16 | 32 tiles columns and computes 5.5 % against the 32-tile bf16 ceiling. But the run that produced 16.81 (the 2,048-token TTFT row in the lane's final-numbers record) used a different prefill launch geometry and kernel set. So the M=128 label and the bf16/32-tile denominator come from the retracted row, not from this measurement. That is the failure the PR body warns about ("a replacement that quietly inherits the old row's authority"). The paragraph under the table then builds on it: "our prefill is at 5.5 %, which is the launch geometry above".
Either fix is fine:
- Label the row with the geometry, datatype and tile count the 16.81 run actually used, and divide by that ceiling.
- Keep 16.81 as an end-to-end figure (TTFT, 2,048 tokens, upper band), set the TFLOPS and % columns to "—" until there is a run at M=128 bf16 32 tiles, and soften the "5.5 %" reading below the table to match.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at 0226bcc8e — and I checked the two things a retraction has to get right: the arithmetic that replaces the number, and
whether the number survives anywhere else.
The arithmetic is self-consistent, which is the only reason the row can stay in a ceiling table:
| tok/s | TFLOPS | % of the 32-tile bf16 ceiling | |
|---|---|---|---|
| retracted | 18.64 | 1.01 | 6.1 % |
| replacement | 16.81 (TTFT, 2048-token prompt) | 0.91 | 5.5 % |
1.01 × 16.81/18.64 = 0.910 and 6.1 × 16.81/18.64 = 5.50, and 0.91/16.4 = 5.55 % against the ceiling column — all three agree, so the new
figure was derived from the same 2·params ÷ tok/s identity rather than typed in beside it.
The derived claim was updated too, not just the row. The paragraph under the table used to read "our prefill is at 6.1 %"; it now
says 5.5 %. I grepped the file at this head for 6.1 % and got zero hits — so there is no second place still repeating the retracted
derivation, which is the usual way a retraction half-lands.
Nothing else in the repository carries the number. search/code for 18.64 across 1bit-MONSTER/engine returns exactly one hit, and
it is this file — so the notice's instruction ("any page still carrying 18.64 must mark it retracted rather than repeat it") is currently
vacuously satisfied, and the PR body can say so with evidence instead of asking the next reader to check.
What I would keep exactly as written: the old value stays visible struck through (a reader can trace what changed), the notice states
why it was retracted (request-wall figures where the ticket's load window was not sustained — the load-window rule this fleet has been
enforcing all week), and the replacement is labelled as also an upper-band figure with the measurement conditions (distinct prompts, no
shared prefix, first half of the ticket). That last clause is the part that makes it citable; without it the new number would inherit the old
one's problem.
One optional addition: since the retraction turns on which window that number came from, a single line naming the ticket/window for the
16.81 figure (as the old row's history is now unverifiable) would stop the same argument from being had again about the replacement.
docs/npu.md's AM027 ceiling table carriedprefill, M=128, 18.64 tok/sas a measurement, and the paragraph beneath it read "our prefill is at 6.1 %"from that number. 18.64 is retracted: it was quoted against request-wall figures in later comparisons where the ticket's load window was not sustained.
This is the page a reader goes to in order to judge where the kernel sits against the per-tile ceiling, so the value being divided into 2·params has to be one
that survives.
The change
prefill, M=128, 18.64 tok/s … 1.01 TFLOPS … **6.1 %**16.81 tok/s (TTFT, 2048-token prompt) … 0.91 TFLOPS … **5.5 %**The conditions are in the notice because 16.81 is also an upper-band figure —
prompt_tokens / TTFT, distinct prompts, no shared prefix, measured in the firsthalf of the ticket. A replacement that quietly inherits the old row's authority would repeat the mistake in the other direction.
The ceiling arithmetic is untouched. 512 INT8 and 256 bf16 multipliers per tile at 1 GHz are properties of the architecture (AM027 Appendix A Table 14,
Chapter 1 Table 1); only the measured column moves, and it still lands at 5.5 % of the 32-tile bf16 ceiling.
The retracted row stays in the table, struck through and marked. A reader who saw 18.64 somewhere else needs to find it here and learn it was withdrawn;
a row that silently disappears cannot tell them that.
Checked
6pipe lines, no paragraph inside the table);grep -rn '18.64' docs/ README.mdnow returns only the struck-through row and the notice, i.e. no page still asserts it as a live measurement;tools/copyright.py --checkpasses.Scope: one file,
docs/npu.md. No open PR touches this table — #1643 and #1650 both editdocs/npu.mdbut neither changes the ceiling rows or mentions 18.64(checked in their diffs), so this does not collide with either.