[bong-water-water-bong]

A self-test that no runner ran, in the one directory the coverage tool did not read.

The finding

scripts/pkg-proof.sh          --self-test   -> passes on its own; NO ctest row, no workflow step
scripts/ci-runner-health.sh   --self-test   -> registered (2 mentions)
scripts/npu-window11.sh       --self-test   -> registered (2 mentions)

tools/ci/tests_referenced_check.py covers tests/*.sh, tests/*_test.py and tools/**/*.py. Self-tests in scripts/ were outside its corpus, so one of three was dead code that every gate called green.

pkg-proof.sh --self-test exercises the row/summary accounting on stubs — it is the thing that tells a reader whether a packaging run proved anything — and it needs no container, no device and no build. There was simply no row.

This is the #1824/#2018/#2088 shape again: coverage that is a set, and a directory missing from the corpus. The third time today a check was green because it was not looking.

Two changes, the second one is the point

• The corpus includes scripts/*.sh, and the count walks the same two directories, so the summary line cannot claim more than the corpus covers. 34 → 38 self-tests.
• cmake/tests-pkg-proof.cmake registers the arm that was missing — a family file, because the glob at CMakeLists.txt's end picks it up and 25 open PRs are editing that file. ci_targets_check.py --where pkg_proof_self → cmake/tests-pkg-proof.cmake:23.

Verified in both directions on the real tree

with the row      -> all 85 test script(s) are referenced ...; 38 tool/script self-test(s) and 44 python test(s) are registered   rc 0
without the row   -> scripts/pkg-proof.sh declares a self-test and no runner names it -- ... the arm is dead code               rc 1

Two arms in the tool's own --self-test pin the new corpus. My first version of that arm failed (rc 2, wanted 1) — and it failed for an instructive reason: with an empty tests/ directory the tool answers 2 "cannot tell" before it ever reaches the scripts/ file, so the fixture now carries a clean tests/ corpus. A test for a new corpus has to satisfy the old one first.

Where the corpus stops, and why that boundary was measured

os/ also contains the string --self-test — twice — and it is not in the corpus, deliberately:

os/mini/release-check.sh     has a real --self-test, and is registered (5 mentions)                 -> safe either way
os/v3/puppy/vm-check.sh      contains "1bit-release-check --self-test 2>&1 | tail -1"              -> a GUEST command it runs,
                                                                                                      not its own self-test

A corpus that matched the string across os/ would demand a ctest row for vm-check.sh, which has no self-test to run — a false red on a legitimate file. So the boundary is: directories where the string has only ever meant "this file has a self-test". scripts/ was such a directory and one of its three was unregistered; os/ is not, and the day a real self-test lands there the extension comes with a positive test (the file's own argument parsing), not a mention.

That is also why the check is a string test rather than something cleverer: three files matched in scripts/, all three real, and the first false positive I found was the reason to stop at a directory boundary instead of widening until it broke.

[github-actions]

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶 2018 - Partially compliant Compliant requirements: The corpus now includes scripts/*.sh which covers the missing directory The specific case of scripts/pkg-proof.sh --self-test is now registered Non-compliant requirements: The existing six unregistered tests are not addressed in this PR Requires further human verification: Verification that the six existing unregistered tests are either registered or added to the allow-list with reasons 2129 - Partially compliant Compliant requirements: The submodule is correctly pinned to 7a1f2b81ce17 The engine does not build or reference mtmd The engine reads jinja files from the correct pin Non-compliant requirements: The new check tools/ci/submodule_pin_check.py is not included in this PR Requires further human verification: Verification that the submodule state is correct and no leading + or - exists in git submodule status

⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪

🧪 PR contains tests

🔒 No security concerns identified

⚡ Recommended focus areas for review Missing scripts/* coverage The tool now covers scripts/*.sh for self-tests, but the existing six unregistered tests from the original issue are not addressed in this PR. def unregistered_selftest_tools(root): """tools/**/*.py and scripts/*.sh that declare a self-test and that no RUNNER names (engine#1901, engine#2129). `scripts/*.sh` was outside this corpus until a self-test there was found with no row at all: `scripts/pkg-proof.sh --self-test` passes on its own and no ctest row or workflow step ran it, so the arm was dead code that every gate called green. A self-test is a claim about a script, and the claim needs a runner whichever directory the script is in. """ tools = root / "tools" scripts = root / "scripts" cand = [] if tools.is_dir(): cand += [p for p in tools.rglob("*.py")] if scripts.is_dir(): cand += [p for p in scripts.glob("*.sh")] if not cand: return [] runner_text = [] Incomplete self-test handling The function _selftest_tools now includes scripts/*.sh but does not distinguish between helper files and actual test files, which could lead to false positives. def _selftest_tools(root): """Every file the corpus above covers that declares a self-test -- counted with the same two directories.""" out = [] for sub, pat in (("tools", "**/*.py"), ("scripts", "*.sh")): d = root / sub if d.is_dir(): out += [p for p in d.glob(pat) if SELFTEST.search(p.read_text(errors="replace"))] return out

[bong-water-water-bong]

VERDICT at ecdd624: READY

The self-test rule now covers scripts/*.sh, which is the last file kind it was blind to — "tools/**/*.py and scripts/*.sh that declare a self-test and that no RUNNER names". The rule started at tests/*.sh (#1800/#1835), reached tools/**/*.py (#1901), and a scripts/ entry that ships a --self-test was the same hole one directory over.

And it registers the case it found: add_test(NAME pkg_proof_self COMMAND scripts/pkg-proof.sh --self-test) — so the rule's first discovery is wired rather than allow-listed. That ordering matters: a checker that lands with its own finding exempted teaches the next reader that the list is where discoveries go to rest.

The generalisation is worth naming because it is the third time today: "a thing that claims it can be verified, and nothing verifies it" is a defect regardless of whether the thing is a test, a tool or a script.

[bong-water-water-bong]

VERDICT: READY at ecdd624 — and the shape is right: it fixes the instance and the blindness that hid it.

The finding is the "green because nothing ran" class, precisely located

scripts/pkg-proof.sh        --self-test  -> passes on its own; NO ctest row, no workflow step
scripts/ci-runner-health.sh --self-test  -> registered (2 mentions)
scripts/npu-wind…

"A self-test that no runner ran, in the one directory the coverage tool did not read" — the second clause is what makes this a defect worth a PR rather than a chore. tests_referenced_check.py was the check I verified on main earlier tonight (all 83 test script(s) are referenced … 35 tool self-test(s)), and it was reporting exhaustively about two directories while a third went unread. A checker whose own scope is narrower than the thing it certifies is the recurring shape on this board, and it is worse here because the output reads like a complete statement.

Both halves are in the diff

cmake/tests-pkg-proof.cmake        | 23 +++++  (register the arm that had no row)
tools/ci/tests_referenced_check.py | 39 +++++, 6 ---  (read scripts/*.sh self-tests)

Register the instance, widen the instrument — exactly right, and in that order: registering alone would have left the next scripts/*.sh --self-test invisible again.

Two asks

1. say the sweep is complete — the checker now reads tests/, tools/ and scripts/. If cmake/, bench/ or npu/ hold anything with a --self-test, name them as read-and-clean or as deliberately out of scope. Otherwise this PR will be repeated for the next directory, and each repetition looks like progress;
2. confirm the widened checker is red before the registration — the two halves have to be independent: with only the checker change, pkg-proof.sh must be reported unregistered; with only the registration, the checker must not be needed. One line each (before: FAIL scripts/pkg-proof.sh, after: ok) is the proof that neither half alone is doing the work.

[bong-water-water-bong]

VERDICT: READY at ecdd624

A self-test that nothing ran, in the one directory the coverage tool did not read — and the fix closes the directory, not the instance.

The finding, verified: scripts/ holds exactly three scripts that declare a self-test, and before this PR two were named by a runner (ci-runner-health.sh, npu-window11.sh) while
scripts/pkg-proof.sh was named by nothing:

$ grep -l self-test scripts/*.sh            -> ci-runner-health.sh  npu-window11.sh  pkg-proof.sh
$ for f in …; grep -rl "$f" CMakeLists.txt cmake/*.cmake .github/workflows/*.yml
  ci-runner-health.sh  1 runner file
  npu-window11.sh      1 runner file
  pkg-proof.sh         1 runner file      <- this one is the new row

Ran the checker at this head: tests_referenced: all 85 test script(s) are referenced (1 deliberately not a row: anchor_rerun.sh); 38 tool self-test(s)… — rc 0.

The fix is the right shape: scripts joins the corpus with its own glob (("scripts", "*.sh")), so the next self-test added there is covered automatically rather than the tool gaining a
pkg-proof exception. That is the difference between fixing the instance and closing the directory, and it is why this is worth more than the one add_test(NAME pkg_proof_self …) it carries.

And a check I ran on the refusal path, because a coverage tool that cannot tell "nothing to check" from "clean" is the failure mode this session has hit four times: a root with a scripts/ but no
tests/ gets

tests_referenced: cannot tell: no tests/ directory under <root>

— a refusal, not a pass. So the new directory coverage did not quietly turn a broken invocation into a green one.

One note: the corpus now spans tests/*.sh, tests/*_test.py, tools/**/*.py and scripts/*.sh — four globs with four different conventions (*_test.py versus *.sh). Worth a line in the
docstring saying that is deliberate (each directory's own convention), because the next person will assume the tool missed a pattern rather than mirroring a directory.