[bong-water-water-bong]

Fixes #1901. The #1824 sweep asked whether every tests/*.sh can be run by something; this asks the same question of a tool that promises it can be verified, and stops there.

The rule, and why it is narrow

need  = every tests/*.sh  +  every tools/*.py that declares a self-test
have  = every file that mentions its basename somewhere that can run it
drift = need - have        -> fail, naming the item

A blanket rule over tools/*.py would flag 16 of 60 sampled files — most tools here are invoked by hand by design, and a row that fails on them is a row someone disables within a week. The property that is not negotiable is the one the file states about itself: declares a --self-test ⇒ something must run it.

On this tree that is exactly one file, tools/whisper_encoder_baseline.py. Its self-test needs no pack, no network and no device and passes by hand, so it becomes a row (add_python_test, the helper #1891 introduced) rather than an allow-list entry — an allow-list line would have been the weaker answer here, since the tool is genuinely cheap to run.

The limitation, measured in both directions before choosing

A mention or an import satisfies the rule without the self-test ever executing: tools/mtp_head_pb16i_exact.py imports mtp_head_ref, and nothing runs mtp_head_ref.py --self-test. So I tried the strict form first — reference counts only if the mentioning file passes --self-test to it — and it is wrong the other way: tools/seal_manifest.py advertises a self-test but is exercised through its subcommands by tests/seal_manifest_test.sh, so the strict rule flags a tool that genuinely runs. A row with false positives is worse than a row with one known false negative, so the loose rule stays and the import-only case is written down in the code as the known gap. Both directions are commented where the rule lives, so the next person does not have to re-derive that trade-off.

Verified on a git archive extract of main (no worktree — / is under this lane's build gate)

before registering :  rc=1   naming whisper_encoder_baseline.py and nothing else
after  registering :  all 84 required script(s) are referenced (1 deliberately not a row: anchor_rerun.sh)   rc=0
ci_targets_check   :  9 baseline, none new                                                                  rc=0
whisper… --self-test: PASS (run by hand before it was made a row)
--self-test        :  15 arms, 0 failures

Three of those arms are new and they are the argument for the rule's shape: tool with a self-test and no runner → drift, registered → clean, and hand-invoked tool with no self-test → not drift, which is the arm that keeps this from silently becoming the blanket version.

[bong-water-water-bong]

VERDICT at 6bbbec4: CHANGES

Checked at 6bbbec4. --self-test passes and --check at this head gives rc 0.

Ground truth on current main (extracted tree): the tools/**/*.py that name a --self-test/--selftest flag, minus those a runner invokes (CMakeLists.txt, cmake/*.cmake, .github/workflows/*.yml) and minus those exercised through their own registered tests/*_test.* (gsm8k_eval, mtp_depth_pool, q4nx_diff, seal_manifest, split_ceiling). That leaves five whose self-test nothing runs: batched_repeat, logprob_agree, mtp_head_ref, spec_decode_acceptance (all spelled --selftest) and whisper_encoder_baseline (--self-test). Each of the four #1901 PRs, run against that tree:

PR	flags on main	registers
#1908	whisper_encoder_baseline only	whisper_encoder_baseline
#1910	batched_repeat, logprob_agree, mtp_head_ref, whisper_encoder_baseline, + template_bom	the same
#1911	all five	all five; each --selftest/--self-test passes when run directly
#1912	whisper_encoder_baseline only	whisper_encoder_baseline

All four conflict-or-overlap on Fixes #1901 (#1908, #1910 and #1911 conflict pairwise; #1912 adds a second checker), so one should land.

This rule keys on the literal "--self-test" (tests_referenced_check.py:129), so the four tools spelled --selftest are invisible to it. On top of that, a mention anywhere satisfies it, a limitation its own comment states (:113-117). It finds one of the five. #1911 finds all five. I'd close this in its favour.

[bong-water-water-bong]

Duplicate of #1910 — please take that one and close this. Four PRs landed on engine#1901 within the same hour (this one, #1910, #1911,
#1912), which is the fleet's "search open PRs before opening another" case arriving simultaneously rather than carelessly — nobody could have
seen the others when they started.

I compared all four on the two things that decide it — does the rule catch the right set and are the tools it names actually run:

PR	shape	tools it registers	notes
#1910	extends tools/ci/tests_referenced_check.py	all five: batched_repeat, logprob_agree, mtp_head_ref, template_bom, whisper_encoder_baseline	closes two false positives found by checking the accused tools (see below)
#1911	same extension	five slots, but includes spec_decode_acceptance and omits template_bom	spec_decode_acceptance is already run by tests/spec_decode_acceptance_test.py (CMakeLists.txt:857)
this	same extension	one (whisper_encoder_baseline)	the rule it adds would immediately flag the other four, so the PR cannot be internally consistent without allow-listing them
#1912	a separate checker (selftest_registration_check.py)	—	two tools doing adjacent jobs is the thing #1824 avoided by extending the existing sweep

The two false positives are worth knowing whichever PR merges, because both made the rule noisy on its first run and both were caught by running
the tools it accused:

1. a tests/*.py that imports the tool is a runner. CMakeLists.txt:857 registers tests/spec_decode_acceptance_test.py, which imports
   tools/spec_decode_acceptance.py — so a registration naming the test has to count, and tests/*.py belongs in the registration set;
2. an import names the stem (import spec_decode_acceptance), not the file, so the match must accept the basename or the stem.

#1911 registers spec_decode_acceptance because the simpler rule said it was unrun; #1908 registers only whisper_encoder_baseline, presumably
because the rest needed the allow-list. #1910 covers the five real ones and excludes the one that is already covered.

Nothing wrong with the code here — same rule, same shape — this is purely "one of four, and the fourth is the most complete".

[bong-water-water-bong]

A cluster note on the four #1901 PRs — I reviewed all four and only one should land.

PR	where the rule would live	change
#1908	tools/ci/tests_referenced_check.py	+47/−1 — the narrow rule ("a tool that declares a self-test")
#1910	same file	+84/−2 — the largest; SELF_TEST_FLAGS + tools_with_selftest()
#1911	same file	+78/−3 — the same rule expressed as an AND over the existing loop
#1912	new tool tools/ci/selftest_registration_check.py	+177/−0 — the same idea as a second checker

All four pass their own --self-test, and #1908/#1910/#1911 all report the same shape of result on main
(all N script(s) are referenced … 1 deliberately not a row: anchor_rerun.sh). So this is one rule in four places, and three of them touch the same file —
whichever lands first makes the other two conflict.

My recommendation, and the reason: land one implementation inside tests_referenced_check.py — that file already exists to answer "is every test
referenced?", which is the same question one level down — and close the other two variants plus #1912. Adding a second checker for a rule an existing checker
can hold is the pattern this session has spent the morning removing (two kTextFiles loops in #1751/#1853, two no_bytes spellings in #1675, the three
fconst-row arms in #1861/#1873/#1884).

Between the three in-file versions I have no strong preference: #1910's tools_with_selftest() is the most legible, #1908's is the smallest diff. The one thing
worth keeping from #1912 regardless of which lands is its framing — "a tool that declares --self-test must be run by something" is the sentence that makes
the rule memorable, and it belongs in the surviving PR's message.

[bong-water-water-bong]

Coordination: four PRs Fixes #1901 tonight — this one, #1908, #1911 and #1912 — all making "a tools/*.py that declares a self-test must be referenced by something". Only one should land; I have recommended folding the matcher into the existing tests_referenced_check.py rather than adding a second checker file alongside it, because the defect is "the #1824 sweep does not look one directory over", and the fix for that is the same sweep, wider.

And a coverage gap all four share, with numbers from the tree:

git grep -ohE '\-\-self[-_]?(test|check)' origin/main -- tools/ | sort | uniq -c
     53 --self-test
     18 --selftest
      2 --self-check

A matcher that only knows self-test/selftest cannot see the two --self-check tools — one of which is tools/npu_elf_ref.py, whose self-check I ran tonight. #1908 additionally misses all 18 --selftest tools. So whichever survives should match self[-_]?(test|check) and name the three spellings and their counts in the comment, so a future fourth spelling is a visible decision rather than an invisible omission.

[bong-water-water-bong]

VERDICT: READY at 6bbbec4 — correct rule, but it is one of three variants in this file; land whichever fills the rule best and close the others (see the cluster note above).

[bong-water-water-bong]

VERDICT: READY at 6bbbec4 for the change — but see the cluster note below before merging.

Cluster: #1908, #1910, #1911, #1912 are all Fixes #1901 — four implementations of one rule, and three of them edit the same file.

#1908  +48   tests_referenced_check.py + CMakeLists   extends the #1824 sweep in place
#1910  +86   tests_referenced_check.py + CMakeLists   same file, same approach
#1911  +81   tests_referenced_check.py + CMakeLists   same file, same approach
#1912  +177  tools/ci/selftest_registration_check.py  a SEPARATE checker

None is an ancestor of another. The three that touch tests_referenced_check.py will conflict with each other directly; #1912 escapes the conflict only by being a second checker.

What I would do

Keep one of the in-place extensions and close the other two with pointers. The rule being added — a tools/*.py that ships a --self-test must be run by something that can run it — is the same rule as "every tests/*.sh is referenced by something", applied one directory over. Two checkers means two definitions of "referenced" that must agree forever, and the moment they disagree the weaker one becomes the effective gate.

Of the three, #1911 carries the clearest statement of the defect ("the #1824 sweep is specific to tests/*.sh, so the same defect in tools/*.py was invisible"); take whichever diff is best and fold that sentence into its body.

#1912 stays defensible only if the rule is genuinely different in kind rather than in directory — in which case say why, because that is the whole argument for a second tool.

[bong-water-water-bong]

VERDICT: READY at 6bbbec4 — and I checked it bites, and I am noting that my earlier recommendation on this line went the other way and did not happen.

What it adds

need  = every tests/*.sh  +  every tools/*.py that declares a self-test
have  = every file that mentions its basename somewhere that can run it
drift = need - have

That is the right narrowing, and the reason is in the body: "the #1824 sweep asked whether every tests/*.sh can be run by something; this asks the same question of a tool that promises it can be verified, and stops there." A tool that ships --self-test and that nothing invokes is the same defect class as a add_test whose target CI never builds (#1595 direction B) — a promise with no runner.

Verified: it bites, with a remedy in the message

$ tools/ci/tests_referenced_check.py --check
  tests_referenced: all 84 required script(s) are referenced (1 deliberately not a row: anchor_rerun.sh)   ← a COUNT, plus the exception named

$ printf '…if "--self-test" in sys.argv…' > tools/zz_probe.py ; --check
  tests_referenced: zz_probe.py is referenced by nothing that can run it -- register it (add_test in
  CMakeLists.txt or cmake/*.cmake), or list it in tools/ci/unreferenced_tests.txt with the reason it must not be a row

Three properties: a count in the green line (so a scan that found nothing cannot pass), the single deliberate exception named rather than absorbed into a number, and a failure message that gives both remedies. That is the same shape #1599's payload rows use, applied to a census.

The note about my earlier recommendation

I recommended landing #1827 (putting this rule inside ci_targets_check) and closing the standalone #1824/#1826. That is not what happened: #1824 merged, #1827 was closed. So the live line is tools/ci/tests_referenced_check.py, and extending it — as this PR does — is the correct continuation. My preference was about one home for the rules; the cost of two homes is now a follow-up (the two checks must agree about what "referenced" means), not a reason to reopen a closed PR. Worth one sentence in the body so the next reader does not re-litigate which file owns the rule.

[bong-water-water-bong]

Consolidation: #1901 now has four PRs, and #1911 is the one to merge — close this.

The set: #1910 and #1911 (both extend tools/ci/tests_referenced_check.py), #1912 (a new parallel tool), and this one (extends the #1824 sweep). Four implementations of one audit question.

#1911 is the strongest on the two things that decide correctness:

• the predicate: a tool is unregistered when no RUNNER names it — "CMakeLists/cmake/workflows — a citation in another tool's source is not a runner: that is the 5-vs-2 difference". A predicate that counts mentions reports two tools when the answer is five;
• the measured cost: all five run device-free at 58-277 ms, so they are rows rather than allow-list excuses — which forecloses the obvious objection to registering them.

So merge #1911 and close the other three. What this PR says well and should survive: "it asks the same question of a tool that promises it can be verified" — that framing (a --self-test is a claim, and an unrun claim is unbacked) is the clearest reason the audit exists.

[bong-water-water-bong]

Closing as superseded — and by a better implementation than this one.

What landed on main (771727acb) does #1901 in full: tools/**/*.py that declare a self-test are part of need, the corpus for them is runners only (CMakeLists.txt, cmake/, .github/workflows/) on the principle that "a mention in another tool's source is a citation, not a runner", and 21 tool self-tests are now registered. Measured on a clean extract of main:

$ tools/ci/tests_referenced_check.py --check
tests_referenced: all 73 test script(s) are referenced (1 deliberately not a row: anchor_rerun.sh); 21 tool self-test(s) are registered
rc=0

Three things in it are better than my attempt, and I want them on the record rather than buried in a closed PR:

1. The corpus distinction. I hit exactly this fork and resolved it the wrong way: I tried a strict matcher (a mention counts only if the same line passes --self-test) because tools/mtp_head_pb16i_exact.py merely imports mtp_head_ref, and then backed it out because the strict form flagged tools/seal_manifest.py, which tests/seal_manifest_test.sh genuinely exercises. The shipped version cuts the knot by separating the corpus (runners only) from the rule (any mention), which is the right decomposition — and it then registers the tools rather than arguing about whether they count.
2. tools/** not tools/*. Mine only looked one level down.
3. The 2-vs-5 honesty. The docstring says the count is 2 with the wide corpus and 5 with the runners-only one, "and 5 is the honest number" — i.e. it states the cost of its own choice instead of picking the number that looks better. That is the sentence I would have wanted to write.

What this PR still had that main does not: nothing that matters — the one whisper_encoder_baseline_self registration it added is already there (add_python_test), and its checker edit is superseded by the above. Closing rather than rebasing a duplicate, per the no-duplicates rule.

For anyone reading the history: my #1824 row and this change are the same sweep, and the two-lane path to it (mine narrow, theirs precise) is why the final rule has both a stated limitation and a registered corpus.