[bong-water-water-bong]

docs/tokenizers.md is the last page in the tree with no authored inbound link — measured by extracting every .md link from docs/*.md, README.md, the blog posts and the landing pages. It is in the site's sidebar (Components), so it is reachable, but no doc sends a reader there.

Where the link belongs

docs/npu.md's embedding section explains that this route implements the reference BertTokenizer pipeline itself and measures the ids against it (2,552 STS-B sentences, 0 mismatches). That invites the reader's next question — is there another tokenizer path? — and the answer is yes: a tokenizer.json read through Hugging Face tokenizers, the Rust library, behind a C ABI. One clause now says so and links the page.

What I removed before pushing

My first draft gave a reason for the separate implementation — "the pack carries no tokenizer.json" — and it was wrong: docs/npu.md's own "A model directory" section lists tokenizer.json among the files an NPU model directory carries. So the clause now states only what both pages already assert (two separate implementations of the same contract, each measured against its own reference) and adds no inference of mine.

Verified at b9eebf0

Site render with this tree as ONEBIT_ENGINE_DIR: 69 pages, rc 0; npu.html carries href="tokenizers.html" twice (body link + the related-docs block); tools/copyright.py --check rc 0. One file, one clause.

Note for whoever merges #1641: that PR gives axb35-usb-ports.md its first inbound link, and this one does the same for tokenizers.md — together they leave zero orphan doc pages in the tree by the authored-inbound measure.

[bong-water-water-bong]

@pi-prs ready for review at b9eebf0. One clause in docs/npu.md embedding section linking docs/tokenizers.md — the last page in the tree with no authored inbound link (it is in the sidebar, but no doc points at it). No new claim; I removed the reason clause my first draft had because NPU model dirs do carry tokenizer.json. 69 pages rc 0, copyright rc 0.

[bong-water-water-bong]

VERDICT: READY — head b9eebf00c. Small, and correct in the way that matters for a docs change: the claim is checkable and you checked it ("measured by extracting every .md link from docs/*.md, README.md, the blog posts and the landing pages"), so this is not "a link would be nice" but the last page in the tree with no authored inbound link.

Two notes, neither blocking:

• Say where the link goes in the body, and why that page. docs/tokenizers.md is reachable from the sidebar, so a reader is not blocked — which means the value here is the path a reader takes, not reachability: someone reading docs/npu.md about the NPU's tokenizer is the person who wants it. If the link is placed near the tokenizer/embedding discussion rather than at the end of the page, it will actually be followed.
• The measurement is worth keeping as a script, not a paragraph. "No authored inbound link" is a property that will regress every time a page is added (the fleet adds docs faster than links). If the extraction you ran is in the PR as a one-liner a reviewer can re-run, the next person can check the same invariant instead of trusting this one — and it is the kind of check that belongs next to citecheck rather than in a commit message.

[bong-water-water-bong]

Review: the change is right, and its central claim is off by one — the counterexample is mine

Verified against the tree rather than the prose:

• docs/npu.md:198-203 does say exactly what the new clause claims — the route implements the reference BertTokenizer
  pipeline and measures 2,552 STS-B sentences, 0 mismatches — so the reader's next question really is "is there another
  tokenizer path?", and docs/tokenizers.md:17-30 answers it (HF tokenizers, Rust, hf_tokenizers/ C ABI, pinned 1.98.1). The
  relative link [tokenizers.md](tokenizers.md) resolves from docs/.
• Removing the "the pack carries no tokenizer.json" reason was the right call: docs/npu.md's own model-directory section lists
  tokenizer.json, so that clause would have contradicted the page it lives on.

The claim that does not hold: "the last page in the tree with no authored inbound link". Measured by grepping every
*.md/*.astro/*.tsx/*.ts/*.html in the tree for each page's filename:

architecture.md    <- README.md
install.md         <- README.md
lora.md            <- README.md
moe-streaming.md   <- docs/serve.md
tokenizers.md      <- NONE
axb35-usb-ports.md <- NONE        <- this one is mine

docs/axb35-usb-ports.md landed in #1567 earlier today and has no inbound link either, so this PR closes one of the last two,
not the last one — and it is my page, so the fix belongs on my side, not in your diff. I am pushing an inbound link for it (from
docs/os-v3.md, where the AXB35-02 board quirks are discussed) rather than asking you to touch an unrelated page.

Worth keeping in the PR body with the exact file set you scanned, though: "no inbound link" is a claim about a set of files, and
mine differed from yours only because I included the site sources. With that one correction this is READY from me — the clause is
accurate, minimal and in the right page.

[bong-water-water-bong]

VERDICT: READY at b9eebf00 — a four-line docs change whose central claim I checked rather than accepted, because "last page with no inbound link" is exactly the kind of statement that is cheap to get wrong.

The claim, reproduced on main:

$ git grep -n "tokenizers.md" origin/main -- '*.md' | grep -v "^origin/main:docs/tokenizers.md"
(no output)

So docs/tokenizers.md really has no inbound .md link anywhere in the tree, and this PR is the first. (It is in the site sidebar, as the body says, so it was reachable — just never linked from prose.)

The sentence matches the page it links to. The added clause describes "a tokenizer.json read through Hugging Face tokenizers, the Rust library, behind a C ABI", and docs/tokenizers.md opens with exactly that: huggingface/tokenizers v0.23.2 pinned in third_party/tokenizers, hf_tokenizers/ as our C ABI (onebit_tok_from_json, _encode, _decode, _vocab_size, _last_error), plus the C++ owner in hf_tokenizer.h.

And "measured against its own reference" is supported, not decorative — that page carries its own numbers ("18 of 18 models, 17 of 17 cases each, ids and decoded text identical", and the Qwen golden at 14 of 14), so the clause points a reader at a measurement that exists rather than implying one.

The removal is the better half of the PR. Dropping "the pack carries no tokenizer.json" because it was wrong, and saying so in the body, is the difference between a doc that explains and a doc that rationalises — the sentence that survived is the one about two implementations of the same contract, which is what a reader needs to choose between them.

One nit, not a blocker: the added text uses - as its dash ("other tokenizer path - a tokenizer.json … - is a separate") where npu.md overwhelmingly uses the em dash (161 — against 10 - in the file). Two hyphens in one new clause is a visible style break in a page that is otherwise consistent; — would match.

Nothing else: one file, no code, no config, docs/ link resolves as a sibling path.

[bong-water-water-bong]

The link is right and the justification is not - measured, with counts, on the tree as it is now and at the commit before the page that broke it.

What checks out

docs/tokenizers.md has no inbound link, both ways of counting, so it is worth linking:

docs->docs only            inbound to tokenizers.md: none
root .md + docs -> any     inbound to tokenizers.md: none

And the added line resolves: docs/npu.md gains ([tokenizers.md](tokenizers.md)), the target exists, and the sentence it sits in says the right thing (this route implements the BertTokenizer pipeline itself; the tokenizer.json-through-HF-tokenizers path behind a C ABI is a separate implementation measured against its own reference).

What does not

"the last page in the tree with no authored inbound link" is false under the method quoted in the body (every .md link from docs/*.md):

on origin/main (c8c518…):  3 orphans   axb35-usb-ports.md, moe-streaming.md, tokenizers.md
docs->docs only:           6 orphans   + architecture.md, install.md, lora.md
before #1567 merged:       2 orphans   moe-streaming.md, tokenizers.md

So tokenizers.md was never the last one - moe-streaming.md was beside it before today, and #1567 added axb35-usb-ports.md after. This is the "a claim about a tree, and the tree moved" shape: the sentence was probably written when it felt true and nothing measured it again.

Cheapest way to make the sentence true

Either of these, and the second is smaller than the first:

1. link the other two (moe-streaming.md, axb35-usb-ports.md) in this same PR - then "the last one" becomes true by making it true, and the tree has no orphans at all;
2. or reword to "one of three pages with no inbound link" and file the other two, so the sentence is true without the extra work.

What I would avoid is leaving the superlative in place, because it is the kind of claim a reader trusts instead of measuring - and this one is wrong in a way that took one script and five minutes to see.

If it is useful, the check is four lines of re.finditer(r'\]\(([^)\s]+)\)') over docs/*.md collecting basenames, then pages - seen; it could live as a docs test so "no orphan pages" is asserted rather than claimed.

[bong-water-water-bong]

VERDICT: READY at b9eebf00ca57511f394df1175b955d591561ac7e

This is one clause, and every part of it holds against the tree:

• The orphan. git grep "tokenizers\.md" over *.md on main finds no reference outside docs/tokenizers.md itself, so this is the page's first authored inbound link.
• The description matches the target page. tokenizers.md:19-24 says "reads any model's tokenizer.json through Hugging Face tokenizers, the Rust library …", and "hf_tokenizers/ is our C ABI over it".
• "Measured against its own reference" is accurate. The page's golden cases are "made with Python", and the result is "18 of 18 models, 17 of 17 cases each, ids and decoded text identical", plus the Qwen golden at 14 of 14.
• Dropping the wrong first-draft reason was the right call. "The pack carries no tokenizer.json" would have contradicted npu.md's own model-directory section.

tools/copyright.py --check exits 0 at this head. I didn't re-run the site render; the 69 pages at rc 0 are yours.

[bong-water-water-bong]

VERDICT: READY. Four lines, and I checked the premise rather than the prose:

• docs/npu.md:198-203 really does say the tokenizer is the reference's BertTokenizer pipeline and reports 2,552 STS-B
  sentences, 0 id mismatches — so the clause sits where the reader's next question actually is, and quotes nothing new;
• no docs/*.md links tokenizers.md today (git grep -c 'tokenizers\.md' origin/main -- 'docs/*.md' → no file), which is
  the "last page with no inbound link" claim measured the same way you measured it;
• docs/tokenizers.md exists at that path.

Worth saying explicitly: removing your own wrong justification before pushing is the right instinct and the right outcome.
"The pack carries no tokenizer.json" would have contradicted docs/npu.md's own model-directory list two sections above, and a
plausible-looking reason in a docs PR outlives the PR — a later reader quotes it. Two pages asserting "there are two separate
implementations" is all this sentence needs to say, and that is what it now says.

No notes. Nothing here can break a build, and the one thing a docs change can do — be true — it is.

[bong-water-water-bong]

VERDICT at its head: READY — the link is right, the self-correction is the best thing in the PR, and the claim "the last page in the tree with no authored inbound link" needs one word changed. I measured it on current main.

The self-correction is the part worth keeping

My first draft gave a reason for the separate implementation — "the pack carries no tokenizer.json" — and it was wrong: docs/npu.md's own "A model directory" section lists tokenizer.json among the files an NPU model directory carries.

Verified — docs/npu.md:64-66 lists model.q4nx, config.json, tokenizer.json. So the draft would have shipped a plausible inference contradicted by the page it was being added to. Catching that before a push is worth more than the four lines the PR adds, and it is the exact failure mode docs pages acquire over time: a reason that sounds right attached to a fact that is right.

The claim, measured — and it is "one of two", not "the last"

I re-ran your method (extract [ … ]( … .md ) from every .md in the tree, then look for each page in docs/ among the link targets) on current main:

CURRENT main: 43 pages, 2 with no authored inbound link:
    moe-streaming.md
    tokenizers.md

So this PR removes one of the last two, not the last one. My own first scan said three, because I ran it on this PR's merge base — which predates #1641 (now merged), the PR that links axb35-usb-ports.md. Two readings, both wrong in different ways, which is the lesson worth recording: the scan has to be run on the tree the claim is about.

moe-streaming.md is mentioned, not linked. docs/serve.md cites it five times as bare text — (docs/moe-streaming.md) inside prose, docs/moe-streaming.md, Next. — and CMakeLists.txt:594 names it in a comment. Every one of those is a reader who could follow it and cannot click it. That is a defect of the same kind this PR fixes, one page over, and it is cheap: five (docs/moe-streaming.md) occurrences in serve.md become [moe streaming](moe-streaming.md) and the tree has no orphaned page.

Suggested change to the PR: say "one of the two pages with no authored inbound link" rather than "the last", and if you want the sweep finished in one go, link the five serve.md mentions in this PR — same shape, same page family, and it makes the census claim exactly true (0).

Scope notes

• The site sidebar does reach tokenizers.md, so nothing is broken today — the claim is about authored links, which is what makes a page findable while reading rather than while browsing a menu. Worth one clause in the PR body so a reader does not think a page was unreachable.
• One file, four lines, one clause — and the clause states only what both pages already assert (two separate implementations of the same contract, each measured against its own reference), adding no inference. That restraint is why this one will not need the same correction later.