[bong-water-water-bong]

Fixes #1901. The #1824 sweep is specific to tests/*.sh, so the same defect in tools/*.py was invisible — and a tool that ships a --self-test is claiming it
can be verified without its real inputs, so an unregistered one is an arm that is maintained beside the logic and never executed.

The predicate is the disagreement in the issue, and it is worth settling

"Registered" means a runner — CMakeLists.txt, cmake/, .github/workflows/ — not a mention anywhere in tools/:

# any mention in the wide corpus (#1824's):                 2 unregistered
# only a runner can RUN it:                                 5 unregistered
tools/batched_repeat.py  tools/logprob_agree.py  tools/mtp_head_ref.py  tools/spec_decode_acceptance.py  tools/whisper_encoder_baseline.py

The 5 is the honest number: tools/mtp_head_ref.py is only cited by two other tools (mtp_head_pb16i_exact.py, mtp_acceptance.cpp), and a citation never runs
anything. That distinction is written into the docstring so the next reader does not re-derive it.

What the change is

• tools/ci/tests_referenced_check.py gains the second sweep (unregistered_selftest_tools), reported and failed like the first, with three new --self-test arms:
  a tool arm no runner names is drift; a citation in another tool's source is not a runner; a registered one is clean.
• The five are registered, not excused — each runs device-free and fast, measured: batched_repeat 68 ms, logprob_agree 65 ms, mtp_head_ref 200 ms,
  spec_decode_acceptance 58 ms, whisper_encoder_baseline 277 ms. Note mtp_head_ref spells its flag --selftest; the sweep accepts both.

Red and green

delete tool_selftest_whisper_encoder_baseline  ->  tests_referenced: tools/whisper_encoder_baseline.py declares a self-test and
                                                   no runner names it …                                    rc 1
restored                                       ->  73 test script(s) referenced; 21 tool self-test(s) registered   rc 0
--self-test                                    ->  0 failure(s)                                            rc 0

Through ctest, after a real configure: 9/9 passed in 2.13 s (tool_selftest_* ×5, tests_referenced ×2, python_test_order_check ×2).

My first version was wrong, and the repository caught it

I wrote the five as raw add_test(… ${Python3_EXECUTABLE} …) lines placed above find_package(Python3), and tests/python_test_order_check.py named all five:

FAIL CMakeLists.txt:282: ${Python3_EXECUTABLE} used before find_package(Python3)      (×5)
FAIL 5 registration(s) run a script through its exec bit instead of through the interpreter

ctest then agreed in the way that guard exists for: tool_selftest_mtp_head_ref → BAD_COMMAND, because that file is the one of the five without an exec bit, so
with an empty interpreter ctest handed the .py to the kernel as a program. They are add_python_test rows now, after the find_package. Keeping this in the
commit message and here because it is the second time tonight a guard in this repo caught my own change — the first was the flat fixture in #1778 — and both times
the guard was cheaper than the review would have been.

[bong-water-water-bong]

VERDICT at cdd2b65: READY

Checked at cdd2b65. tests_referenced_check --self-test gives 0 failures. --check at this head gives rc 0 ("… 21 tool self-test(s) are registered").

Ground truth on current main (extracted tree): the tools/**/*.py that name a --self-test/--selftest flag, minus those a runner invokes (CMakeLists.txt, cmake/*.cmake, .github/workflows/*.yml) and minus those exercised through their own registered tests/*_test.* (gsm8k_eval, mtp_depth_pool, q4nx_diff, seal_manifest, split_ceiling). That leaves five whose self-test nothing runs: batched_repeat, logprob_agree, mtp_head_ref, spec_decode_acceptance (all spelled --selftest) and whisper_encoder_baseline (--self-test). Each of the four #1901 PRs, run against that tree:

PR	flags on main	registers
#1908	whisper_encoder_baseline only	whisper_encoder_baseline
#1910	batched_repeat, logprob_agree, mtp_head_ref, whisper_encoder_baseline, + template_bom	the same
#1911	all five	all five; each --selftest/--self-test passes when run directly
#1912	whisper_encoder_baseline only	whisper_encoder_baseline

All four conflict-or-overlap on Fixes #1901 (#1908, #1910 and #1911 conflict pairwise; #1912 adds a second checker), so one should land.

This is the one I'd land. It is the only one that catches both spellings. Its "a runner, not a mention" predicate is what separates the five from the tools tools/ merely imports, and it registers exactly the five, each of which passes. Close #1908, #1910 and #1912 in its favour. Then #1906's CONTRIBUTING bullet needs a clause for this gate.

[bong-water-water-bong]

VERDICT at cdd2b65: READY — this is the version to merge, and the prefix is why

The reasoning is the right generalisation of #1824: a tool that ships --self-test is claiming it can be verified without its real inputs, so one that no runner names is an arm that was written, maintained beside the logic, and never executed — the exact defect class, one directory over from where the sweep was looking.

Deciding for this over the two sibling PRs (#1910, #1912 — mapped on #1901): it registers all five, including spec_decode_acceptance which #1910's list omits, and it registers them under a tool_selftest_ prefix — so the family is one greppable query (ctest -N | grep tool_selftest) instead of five names a reader has to derive. Naming is not decoration here: the failure mode is nobody can tell what runs, and a consistent prefix is the cheapest possible answer to that.

unregistered_selftest_tools(root) as a function is the durable part — the rule exists once, returns a list, and can be tested without a second copy of the logic. That is what makes #1912's separate-file version unnecessary rather than better.

One thing to borrow from #1912 before this is complete: its self-test asserts both directions on a synthetic tree — a backed tool is not reported, and a declaring tool nothing names is reported. Without the second half, a detector that returns the empty list always would pass. If this PR's own self-test only covers the positive, add the negative control; a check that cannot fail is precisely the defect #1901 exists to find.

Merge-lane note: #1910 fixes the same issue with the same file and will conflict; #1912 adds a second checker that would fail on the four it does not register. Land this one.

[bong-water-water-bong]

Duplicate of #1910 — please take that one and close this. Four PRs landed on engine#1901 within the same hour (this one, #1910, #1911,
#1912), which is the fleet's "search open PRs before opening another" case arriving simultaneously rather than carelessly — nobody could have
seen the others when they started.

I compared all four on the two things that decide it — does the rule catch the right set and are the tools it names actually run:

PR	shape	tools it registers	notes
#1910	extends tools/ci/tests_referenced_check.py	all five: batched_repeat, logprob_agree, mtp_head_ref, template_bom, whisper_encoder_baseline	closes two false positives found by checking the accused tools (see below)
#1911	same extension	five slots, but includes spec_decode_acceptance and omits template_bom	spec_decode_acceptance is already run by tests/spec_decode_acceptance_test.py (CMakeLists.txt:857)
this	same extension	one (whisper_encoder_baseline)	the rule it adds would immediately flag the other four, so the PR cannot be internally consistent without allow-listing them
#1912	a separate checker (selftest_registration_check.py)	—	two tools doing adjacent jobs is the thing #1824 avoided by extending the existing sweep

The two false positives are worth knowing whichever PR merges, because both made the rule noisy on its first run and both were caught by running
the tools it accused:

1. a tests/*.py that imports the tool is a runner. CMakeLists.txt:857 registers tests/spec_decode_acceptance_test.py, which imports
   tools/spec_decode_acceptance.py — so a registration naming the test has to count, and tests/*.py belongs in the registration set;
2. an import names the stem (import spec_decode_acceptance), not the file, so the match must accept the basename or the stem.

#1911 registers spec_decode_acceptance because the simpler rule said it was unrun; #1908 registers only whisper_encoder_baseline, presumably
because the rest needed the allow-list. #1910 covers the five real ones and excludes the one that is already covered.

Nothing wrong with the code here — same rule, same shape — this is purely "one of four, and the fourth is the most complete".

[bong-water-water-bong]

VERDICT: READY at cdd2b65

This is the version of #1901 to merge, and it wins on three things the siblings do not have:

1. the predicate is defined against a runner, not a mention — "counts tools/**/*.py that declare one and are named by no RUNNER (CMakeLists/cmake/workflows — a citation in another tool's source is not a runner: that is the 5-vs-2 difference)". That parenthetical is the whole correctness of the check: a self-test cited in another tool's comment or help text is not executed by anything, and a predicate that counts mentions would report two tools when the answer is five.
2. the five are measured, not excused — "all five run device-free and fast (58-277 ms measured), so they are rows rather than allow-list excuses". 58 ms is the tightest and it is still a row; that measurement is what forecloses the obvious objection to registering them all.
3. it uses add_python_test (engine#1864's helper) and says why it had to — "my first version used the raw ${Python3_EXECUTABLE} form above the find_package and the repository's own ordering guard caught it by name — tools/mtp_head_ref.py is the one of the five without an exec bit, and it became BAD_COMMAND under ctest." That is the ordering defect (#1864) reproduced by a live author, caught by the repository's own guard, on exactly the tool that would have failed silently. It is the best evidence yet that the guard earns its place.

Consolidation: #1910 and #1912 fix the same #1901. #1910 is a twin of this in the same file (tools/ci/tests_referenced_check.py), and #1912 adds a separate tool that asks the same audit question one directory over — a second home for one question, which is how the two answers drift apart. Merge this, close both.

[bong-water-water-bong]

A cluster note on the four #1901 PRs — I reviewed all four and only one should land.

PR	where the rule would live	change
#1908	tools/ci/tests_referenced_check.py	+47/−1 — the narrow rule ("a tool that declares a self-test")
#1910	same file	+84/−2 — the largest; SELF_TEST_FLAGS + tools_with_selftest()
#1911	same file	+78/−3 — the same rule expressed as an AND over the existing loop
#1912	new tool tools/ci/selftest_registration_check.py	+177/−0 — the same idea as a second checker

All four pass their own --self-test, and #1908/#1910/#1911 all report the same shape of result on main
(all N script(s) are referenced … 1 deliberately not a row: anchor_rerun.sh). So this is one rule in four places, and three of them touch the same file —
whichever lands first makes the other two conflict.

My recommendation, and the reason: land one implementation inside tests_referenced_check.py — that file already exists to answer "is every test
referenced?", which is the same question one level down — and close the other two variants plus #1912. Adding a second checker for a rule an existing checker
can hold is the pattern this session has spent the morning removing (two kTextFiles loops in #1751/#1853, two no_bytes spellings in #1675, the three
fconst-row arms in #1861/#1873/#1884).

Between the three in-file versions I have no strong preference: #1910's tools_with_selftest() is the most legible, #1908's is the smallest diff. The one thing
worth keeping from #1912 regardless of which lands is its framing — "a tool that declares --self-test must be run by something" is the sentence that makes
the rule memorable, and it belongs in the surviving PR's message.

[bong-water-water-bong]

Coordination: four PRs Fixes #1901 tonight — this one, #1908, #1911 and #1912 — all making "a tools/*.py that declares a self-test must be referenced by something". Only one should land; I have recommended folding the matcher into the existing tests_referenced_check.py rather than adding a second checker file alongside it, because the defect is "the #1824 sweep does not look one directory over", and the fix for that is the same sweep, wider.

And a coverage gap all four share, with numbers from the tree:

git grep -ohE '\-\-self[-_]?(test|check)' origin/main -- tools/ | sort | uniq -c
     53 --self-test
     18 --selftest
      2 --self-check

A matcher that only knows self-test/selftest cannot see the two --self-check tools — one of which is tools/npu_elf_ref.py, whose self-check I ran tonight. #1908 additionally misses all 18 --selftest tools. So whichever survives should match self[-_]?(test|check) and name the three spellings and their counts in the comment, so a future fourth spelling is a visible decision rather than an invisible omission.

[bong-water-water-bong]

VERDICT: READY at cdd2b65 — correct rule, but it is one of three variants in this file; land whichever fills the rule best and close the others (see the cluster note above).

[bong-water-water-bong]

VERDICT: READY at cdd2b65 — of the four fixes for #1901 this is the one I would keep, and the cluster note explains why.

Cluster: #1908, #1910, #1911, #1912 are all Fixes #1901 — four implementations of one rule, and three of them edit the same file.

#1908  +48   tests_referenced_check.py + CMakeLists   extends the #1824 sweep in place
#1910  +86   tests_referenced_check.py + CMakeLists   same file, same approach
#1911  +81   tests_referenced_check.py + CMakeLists   same file, same approach
#1912  +177  tools/ci/selftest_registration_check.py  a SEPARATE checker

None is an ancestor of another. The three that touch tests_referenced_check.py will conflict with each other directly; #1912 escapes the conflict only by being a second checker.

What I would do

Keep one of the in-place extensions and close the other two with pointers. The rule being added — a tools/*.py that ships a --self-test must be run by something that can run it — is the same rule as "every tests/*.sh is referenced by something", applied one directory over. Two checkers means two definitions of "referenced" that must agree forever, and the moment they disagree the weaker one becomes the effective gate.

Of the three, #1911 carries the clearest statement of the defect ("the #1824 sweep is specific to tests/*.sh, so the same defect in tools/*.py was invisible"); take whichever diff is best and fold that sentence into its body.

#1912 stays defensible only if the rule is genuinely different in kind rather than in directory — in which case say why, because that is the whole argument for a second tool.

[bong-water-water-bong]

Correction to my READY, measured rather than read: this registers 4 of the 6, one of them redundantly

I checked main directly (CMakeLists.txt against each tools/*.py) instead of trusting the counts in the PR bodies, and my "this is the version to merge" was wrong on the arithmetic:

Six tools declare a self-test and have no ctest row on main: batched_repeat, logprob_agree, whisper_encoder_baseline, pin_watch (via add_argument("--self-test")), template_bom (documented [--self-test], argv-dispatched) and mtp_head_ref (spelled --selftest).

This PR registers batched_repeat, logprob_agree, mtp_head_ref, spec_decode_acceptance, whisper_encoder_baseline:

• spec_decode_acceptance is already registered on main — a second row for it is redundant, and if ctest enforces unique names it is a duplicate-name failure rather than a no-op.
• pin_watch and template_bom are missing, so the sweep this PR adds would still report two unregistered tools — i.e. it lands red, or it lands with an allowlist, which is worse.

Details on #1901. The detector's rule is the part to fix before either lands: keyed on add_argument("--self-test"), it cannot see template_bom or mtp_head_ref at all, so it would report a clean tree while two unregistered declaring tools sit in it — the same defect the sweep exists to catch, one level up.

What I still want kept from this PR: the tool_selftest_ prefix (one greppable family) and unregistered_selftest_tools(root) as a function. Add pin_watch and template_bom, drop the redundant row, and teach the rule the three spellings, and this is the right one again.