ci(cli-tri): count, name and ratchet the tests that skip themselves

[gHashTag]

Closes #2377. #2370 stays open — this does not fix it.

What I verified first

Re-checked #2370's claims on origin/master c142c2f03802c13407649d521e64380b0c68c819 before changing anything.

git grep -niE "elan|lake|lean-toolchain|lean-action" origin/master -- .github/workflows/
returns nothing. Every lean substring hit across the workflow files is a false
positive — boolean, clean, elaborated. cli-tri.yml's only install step is:

      - name: install yosys
        run: sudo apt-get update -qq && sudo apt-get install -y -qq yosys

The three skips reproduce verbatim, and the issue's list is exact:

test fpga::tests::test_measured_to_lean_standalone_builds_in_temp_lake_package ... skip: lake not on PATH
test fpga::tests::test_smoke_gate_json_synthetic_validate_lean_standalone ... SKIP: lake not on PATH
test fpga::tests::test_smoke_gate_validate_lean_standalone_matches_snapshot ... SKIP: lake not on PATH

The part that decided the design

None of the three uses #[ignore]. Each println!s a reason and returns, so libtest
counts them as passed:

test result: ok. 173 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

ignored is 0. And cargo test captures the stdout of a passing test, so the reason
is not in the log at all — grep -icE 'skip' over the full default log returns 2, and
both hits are test names (..._sections_skipped ... ok,
skipped_is_not_a_verdict_...), not reasons.

So there is nothing to parse unless you ask for it. --nocapture makes the reason exist;
--test-threads=1 makes it attributable, because libtest lands the reason on the
test NAME ... line it has already opened and parallel workers interleave those.

Shape: ratchet, not reporting-only

Reporting-only would detect nothing — it prints a number into a green run and a fourth
test could start skipping tomorrow with the notice quietly reading 4. The ratchet keeps
that reporting and fails on growth, which is the actual regression.

The baseline is emitted by the same script in --emit-baseline mode from a real test log.
It is not a hand-typed 3: that would restate what #2370 already established and would
not survive a rename.

Today's three skips are not turned into failures. That is the owner's policy call, and
a skip is not a failure.

Three bars

TRUE — reports the real set, exit 0:

::notice title=skip ratchet::3 test(s) skipped at runtime and were counted as passed (baseline 3)
No new runtime skips. Baseline holds.

ALIVE — it runs in the real pipeline. End-to-end with the exact committed command
sequence, set -o pipefail correctly propagated a test failure through tee (exit 101)
while the ratchet still read the log and reported 3.

BITING — a fourth deliberately-skipping test was planted in fpga::tests (skipping on
a nonexistent binary) and the suite re-run. cargo test said:

test result: FAILED. 172 passed; 2 failed; 0 ignored; 0 measured; 0 filtered out

The mutant counted as passed and ignored stayed 0 — that is the defect itself.
The ratchet caught it:

::notice title=skip ratchet::4 test(s) skipped at runtime and were counted as passed (baseline 3)
**FAILED — the skip set grew.** New runtime skip(s):
- `fpga::tests::test_mutant_fourth_silent_skip` — `skip: definitely-not-a-real-binary-xyz not on path`
::error title=skip ratchet::new runtime skip: fpga::tests::test_mutant_fourth_silent_skip (skip: definitely-not-a-real-binary-xyz not on path)

Exit 1, reporting 4 rather than 3. The mutant was then removed;
cli/tri/src/fpga.rs is byte-identical to master (git diff is empty).

Degenerate inputs were checked too: a log with no test result: line (build died) and a
missing log both report "skip set NOT evaluated" and exit 0, rather than reporting an
empty set as a clean bill of health.

What this is not

• No Lean toolchain installed. elan plus a toolchain is multi-gigabyte and
  unaffordable on this machine (~1.1 GB free all session). The real fix for No workflow installs lake/elan, so every Lean smoke-gate test skips on every runner #2370 is that
  install, and it is still undone — hence No workflow installs lake/elan, so every Lean smoke-gate test skips on every runner #2370 stays open.
• This is not coverage. It cannot make a skipped test run. It makes the skipping
  legible and stops the set growing unnoticed.
• It only recognises the in-repo convention — a leading SKIP: / skip: on a line of
  test stdout. A test that returns early printing something else, or printing nothing, is
  as invisible to this as it is today. Real hole, stated deliberately.
• The baseline was generated on macOS with the demo bitstream present and yosys 0.63. If
  the ubuntu runner's set differs, the ratchet reddens on its first run rather than
  passing quietly — the intended direction.
• The verifying job is cli-tri's build, which is not a required check, so
  auto-merge will not wait for it. It is watched on master after merge.

[github-actions]

📓 NotebookLM Notebook linked to this PR

• Notebook ID: b83263109fb055dc
• URL: https://notebooklm.google.com/notebook/b83263109fb055dc

This notebook contains session context, decisions, and artifacts for this work.

[github-actions]

PR Dashboard

Generated at: 2026-08-22 01:01:52 UTC

• test(sim): commit the differential harness behind the DMA write-address fix (Refs #2348) #2379 test(sim): commit the differential harness behind the DMA write-address fix (Refs #2348) | Created: 2026-08-22 | Updated: 2026-08-22 | 15 failing / 19 total
• ci(cli-tri): count, name and ratchet the tests that skip themselves #2378 ci(cli-tri): count, name and ratchet the tests that skip themselves | Created: 2026-08-22 | Updated: 2026-08-22 | 15 failing / 22 total
• corpus: freeze the damage, patch one class at a time, and say where one rule stops #2161 corpus: freeze the damage, patch one class at a time, and say where one rule stops | Created: 2026-08-14 | Updated: 2026-08-21 | 0 failing / 9 total
• Unbreak FPGA CI and correct two unbacked hardware claims #2081 Unbreak FPGA CI and correct two unbacked hardware claims | Created: 2026-08-11 | Updated: 2026-08-20 | 5 failing / 27 total

Summary

Status	Count
Total Open PRs	4
PRs with Failing Checks	1
PRs with All Checks Green	3
READY	1
FAILING	1
PENDING	0

Seal Status

• ⚠️ STALE -- sha256(compiler.rs)=65f033d04125 != manifest seal=87e5cbd3ad94.
  The committed NMSE numbers were certified against an older compiler.rs.
  Run scripts/reseal-check.sh locally for the two-step reseal command (advisory; not a merge gate).