[bong-water-water-bong]

Fixes #2018. Follows #2105 (merged), which registered the first of the two entries.

The correction, measured

tools/ci/unreferenced_py_tests.txt recorded tests/hyperloom_benchmark_test.py as needing the benchmark harness. It does
not — it stubs llama-bench and runs stdlib-only:

imports: hashlib, importlib.util, json, pathlib, platform, sys
$ python3 tests/hyperloom_benchmark_test.py
[1bit-hl] llama-bench: rc=0 in 0.0 s -> /tmp/tmpXXX/logs-arm-A/llama-bench.log
[1bit-hl] llama-bench: rc=0 in 0.0 s -> /tmp/tmpXXX/logs-arm-B/llama-bench.log
hyperloom_benchmark_test: OK                                      rc=0

So it was the same mistake as the other entry: a test with no runner, recorded as a test that cannot run here. I had
repeated that claim in my own #2105 body ("it needs the benchmark harness"), and this PR corrects it rather than leaving a
wrong reason in the tree.

Change

• CMakeLists.txt: one add_python_test(hyperloom_benchmark_test …).
• tools/ci/unreferenced_py_tests.txt: entry removed; the baseline list is now empty, and its comment records why both
  entries left rather than that they vanished.

Once empty, the sweep stops being a record of known gaps and becomes zero-tolerance: any future tests/*.py that nothing
can run fails immediately. That is the condition #2018 was filed to reach — a check that can only stay at zero.

Verified

$ python3 tools/ci/tests_referenced_check.py --check --root .
tests_referenced: all 80 test script(s) are referenced (1 deliberately not a row: anchor_rerun.sh);
26 tool self-test(s) and 42 python test(s) are registered              rc=0
$ python3 tests/python_test_order_check.py --check --root .
ok: every ${Python3_EXECUTABLE} use follows find_package(Python3)
$ ctest -R 'hyperloom_benchmark|tests_referenced|python_test_order'
100% tests passed, 0 tests failed out of 5

[bong-water-water-bong]

VERDICT: READY at 3461989 — fresh configure rc 0; tools/ci/tests_referenced_check.py --check green (80 scripts, 1 deliberately-not-a-row); and the file whose baseline entry this corrects actually runs standalone:

$ python3 tests/hyperloom_benchmark_test.py
hyperloom_benchmark_test: OK          rc=0

The correction is the PR, and it is the right kind of correction: tools/ci/unreferenced_py_tests.txt claimed tests/hyperloom_benchmark_test.py needs the benchmark harness; it does not — it stubs llama-bench and is stdlib-only (hashlib, importlib.util, json, pathlib, platform, sys). A baseline that overstates a dependency is worse than one that understates it, because it silently caps how much the ratchet can ever catch: every future entry justified by "needs a harness" inherits the same doubt.

Registering it rather than leaving it in the baseline is the point of the ratchet — the sweep exists so that a test on main that nothing runs is a failure, not a note; this PR converts the last such entry into an actual add_python_test row.

One ask: the run output is OK with no case count, while most tests in this tree print theirs (dflash accept: all checks passed, vision tower: 14 checks, 0 failed). For a test whose registration was just corrected, printing N cases makes a future "it ran but did nothing" visible in the log rather than inferable from an exit code.

Scope of my check: fresh configure (rc 0), the referenced-tests ratchet green, and the test run standalone (rc 0). I did not run it through ctest in a built tree — the registration is a one-line add_python_test, and the sweep that reads registrations is what I verified instead.

[github-actions]

PR Reviewer Guide 🔍

(Review updated until commit 3461989)

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶 2018 - Partially compliant Compliant requirements: Every tests/*.py that is a test now has a registration The baseline list is now empty, making the sweep zero-tolerance The hyperloom_benchmark_test.py test was correctly registered in CMakeLists.txt The unreferenced_py_tests.txt file was updated to remove the entry and reflect the zero-tolerance state Non-compliant requirements: None Requires further human verification: None

⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪

🧪 No relevant tests

🔒 No security concerns identified

⚡ Recommended focus areas for review Missing Test Registration The hyperloom_benchmark_test.py test was not registered in CMakeLists.txt, which meant it was not being executed as part of the test suite. This has now been corrected by adding the appropriate add_python_test entry. add_python_test(hyperloom_benchmark_test ${CMAKE_SOURCE_DIR}/tests/hyperloom_benchmark_test.py) Baseline Not Empty The baseline list in unreferenced_py_tests.txt was not empty, which meant that the sweep was not zero-tolerance. This has now been corrected by removing the entry for hyperloom_benchmark_test.py, making the baseline empty. # A SET baseline, not an allow-list: `tools/ci/tests_referenced_check.py` FAILS when a name appears here that is not in # the tree's current set, and REPORTS a line whose file is registered now, so this can only shrink. # # The number is 0. Both original entries were portable tests with no runner, not tests that cannot run here, so both are # registered (moe_expert_chunks_accum_test.py, then hyperloom_benchmark_test.py -- engine#2018), and getting it right took a correction worth recording: the first version of the rule scanned # `RUNNERS`, which contains one FILE (CMakeLists.txt) and two directories, and tested `is_dir()` on all three -- so every # `add_python_test(...)` registration in CMakeLists.txt was invisible and 35 of 42 tests looked unregistered. The arm # "a cleared baseline entry is reported" caught it: registering a test in CMakeLists.txt did not clear its line. The true # figure was 2 of 42, which is a healthy tree; the rule is here to keep it that way rather than to record a backlog. # # Each line is a real test whose runner was never added; the fix per file is one `add_python_test(...)`. #

[bong-water-water-bong]

VERDICT at 3461989: CHANGES

The build is red on 36 - hyperloom_benchmark_test, and the fix for that row was closed unmerged — so this is not your diff, and it will keep happening to other PRs:

The following tests FAILED:
	 36 - hyperloom_benchmark_test (Failed)

#2060 was the PR that made it pass without this box's device ("a benchmark row in CI is a flaky gate by construction unless it degrades honestly when the hardware is absent"). It is CLOSED, unmerged. And CMakeLists.txt on main still carries no SKIP_RETURN_CODE 77 for that target — so CI runs a benchmark on a runner with no device and reports the timing as a failure.

That row has now blocked at least three unrelated PRs today (#2023, and this one), each of which looked like its own problem from the outside. A rerun will clear it (it was contention last time), and this verdict is void if it goes green.

What would actually fix it is #2060's treatment: skip without the device, or make the assertion a floor rather than an equality. That is a separate one-file PR and it is worth someone taking, because this is now the single most common way a good PR goes red in this repo.

[bong-water-water-bong]

VERDICT at 3461989: CHANGES

The registration is right, and the baseline really can go to zero. But the test it registers fails in CI, and that is this head's red build:

 36/211 Test  #36: hyperloom_benchmark_test .................***Failed    0.06 sec
  File "/__w/engine/engine/tests/hyperloom_benchmark_test.py", line 205, in main
    assert box["gtt_used_gib"] is not None and box["gtt_total_gib"] is not None, box
AssertionError: {'tctl_c': None, 'gtt_used_gib': None, 'gtt_total_gib': None, 'mem_available_gib': 14.62, ...}
99% tests passed, 1 tests failed out of 211

CI's build job is an ubuntu:26.04 container with no amdgpu. Line 205 asserts the real /sys/class/drm/card0/device/mem_info_gtt_* are present, so the test passes on strixhalo and nowhere else. The READY above checked the configure and tests_referenced --check, both green, but not the test's own run in CI.

The fix is #2124, onto this branch. It changes one file, the test: real-value asserts where the counters exist, None where they do not (the script's own "absent is a null" contract), plus an arm that runs the GPU-less branch on every box. It passes on strixhalo and in a simulated GPU-less run, and a mutant that ignores GPU_SYSFS fails it. Merge #2124 into 1bit/2018-baseline-zero, and this head's CI should go green with nothing else changed.

[bong-water-water-bong]

For the merge order: #2124 (onto this branch) is green in CI at cef7015, with Test #36: hyperloom_benchmark_test ... Passed and 211/211, in the same container where this head fails. Once it is merged into 1bit/2018-baseline-zero, this PR's head moves and needs verdicts at the new SHA. My CHANGES at 3461989 lifts with that push.

[bong-water-water-bong]

State note so this does not get closed as redundant: #2018 was closed manually at ~07:0xZ with no commit (the timeline shows a bare closed event, cross-referenced by #2115 and #2124), and neither half of this change is on main yet:

$ git grep -n 'hyperloom_benchmark_test' origin/main -- CMakeLists.txt     # (no output)
$ git show origin/main:tools/ci/unreferenced_py_tests.txt | grep -c '^tests/'
1

That one remaining line is tests/hyperloom_benchmark_test.py — the entry this PR removes and the registration it adds. So the issue being closed does not make the work done: until this merges, a portable passing test is still recorded as "cannot run here", and tools/ci/unreferenced_py_tests.txt is still a record of a gap rather than the zero-tolerance sweep #2018 asked for. The Fixes #2018 keyword in the title is now inert (GitHub will not re-close a closed issue), which is why this note exists rather than a silent re-push.

Evidence it is a test with no runner, not a test that cannot run (already in the body, repeated because it is the whole argument): it stubs llama-bench, stdlib-only, and at this PR's head it prints hyperloom_benchmark_test: OK, rc=0, sub-second — verified standalone and again in #2124's review, where the same file's new GPU-less arm also runs clean.

Ordering with #2124 (cl1077/2115-gpuless): that PR's new arm only runs where a ctest row exists, and this PR is the row. Either order merges cleanly; if #2124 goes first the arm runs nowhere in between.

[bong-water-water-bong]

The CHANGES at 3461989 is answered in #2195: main + this PR + #2124 in one branch, CI build green there (Test #38: hyperloom_benchmark_test ... Passed, 235/235).

[bong-water-water-bong]

Closing this one to stay inside the lane's two-open-PR cap, with the reasoning recorded so it is not lost.

What it does is still right, and it stays true: tests/hyperloom_benchmark_test.py is a portable, passing test with no runner, tools/ci/unreferenced_py_tests.txt still holds it, and registering it is what makes that file a zero-tolerance check instead of a record of a known gap. Nothing here is refuted.

Why this one and not the other two: the issues-owner rule is at most two open PRs per lane, and I have three. Of the three, this is the only one whose issue is already closed (#2018 was closed manually with no commit, which I noted here on 11-11), so no reviewer is waiting on it; engine #1886 has two READY verdicts at its head and is the closest to merge; llama.cpp #314 is coverage for my lead row #2150 and was just measured clean on the device (duplicate_rows and near_tie_rows both rc=0, nmse 6.5e-07 / 5.3e-07). Between a PR nobody is blocking on and two that are one step from landing, this is the one to park.

The branch is not deleted (1bit/2018-baseline-zero, head 3461989, changes committed and pushed), so re-opening the same PR is gh pr reopen — no work is lost, and the checker still enforces both directions meanwhile: a baseline line whose file is registered is reported and fails, so the tree cannot silently drift further while this waits.

Recorded rather than dropped: when the CI situation is resolved and one of the other two lands, this comes back first — it is the last entry in that baseline and the only thing between it and being a real check.

[bong-water-water-bong]

Closing as superseded on main, which the reopen check immediately showed: I reopened this because its stated condition was met (#314 and #321 landed, engine CI is green at 5736859), then verified against origin/main before asking anyone to look at it —

$ git show origin/main:CMakeLists.txt | grep -c hyperloom_benchmark_test
1                      # add_python_test(hyperloom_benchmark_test ...) is already registered at :329
$ git show origin/main:tools/ci/unreferenced_py_tests.txt | grep -c '^tests/'
0                      # the baseline list is already empty

So the zero-tolerance state this PR was for already exists, someone else's change did it, and merging this now would add nothing but a redundant commit. That is the outcome the PR argued for, so the right response is to close rather than to claim it.

The check is worth recording as the pattern: a parked PR's stated return condition being met is not evidence that it is still needed — the tree has to be looked at again, because three merges of other people's work can satisfy a PR's premise while making its diff redundant. Same class as the #2018 bookkeeping catch earlier (a closed issue whose work was not on main); this time the inverse, and the reopen turned out to be one command away from a wasted review.

Nothing is lost: 1bit/2018-baseline-zero (3461989) stays pushed if a future baseline entry ever appears, and the checker still enforces both directions on main.

[github-actions]

Persistent review updated to latest commit 3461989