Fix/a closed row names a test that touches behaviour - #293
Merged
evkir merged 4 commits intoSep 22, 2026
Merged
Conversation
Risk 20 stood closed for fifteen days on a test that read a config field while an unscoped run spent fifty-one seconds touching a protected range. The existing guard was green and right to be: it asks whether the named test exists, not what the test asserts. Two probes settled that "measures behaviour" is not a syntactic property. A test calling validate_exploit_scope and checking its verdict is indistinguishable from one calling from_env and reading .strict_scope, so a rule failing the second accuses rows 1 and 6 wrongly. This reports instead of judging: structural, boundary, entrypoint or value, with helpers in the same module followed. Entrypoint is decided by the argument, not the receiver. CliRunner and asyncio are the receivers of every command run here; the product rides in as an argument. Measured over the register today: structural 2, boundary 2, entrypoint 6, value 6 across sixteen references. The cases are synthetic. A test reading today's register measures this morning's history and turns red when an unrelated test is renamed, which teaches the reader to edit the expectation instead of the code. The script is loaded by path, as the badge gates are, and named by a step in the typecheck job: scripts/ resolves on sys.path only because of how the package is installed here.
The page now carries a Held by column, written by the script rather than by whoever edits the row, and a guard that fails when the column and the tree disagree. Measured today: structural 2, boundary 2, entrypoint 6, value 6 over sixteen references. Risk 20 reads boundary/value, which is what it would have said out loud while it stood closed on a config read. No level is forbidden. Two probes established that the distinction cannot be made from syntax, so a value row is not a defect: it is a row whose guard could be stronger, now readable as such without opening the test. Both readers of this page matched the new column into the evidence cell and went on working. That is the wrong reason for a check to be green, so each pattern was widened to know the shape it reads. A column of claims decays the way the last table did: typing-scope.md carried a module at 17 errors while the tree said 12, and nobody was wrong when it was written. Hence the comparison rather than a note. Known gap, recorded rather than fixed: a row the pattern cannot parse leaves both sides of that comparison at once and is not reported missing. Counting the rows is a second guard, not this one.
The gap the last commit recorded is closed. A row the pattern cannot parse is absent from the page side and the tree side together, so the comparison could not see it: rows 20 and 24 carry a slash and would have vanished silently under a narrower pattern. The pipe lines are now split into heads, rules and body, and a body line the pattern misses is reported. Counting rows against a fixed number would have been a claim about today instead. Row 4 named test_the_manifest_carries_the_profile, which asserts two keys are present in the manifest. The risk is that two runs seeing different surfaces published the same provenance, and the test beside it -- test_two_profiles_do_not_share_a_hash -- is the one that fails when they do. Both have existed since 2026-09-08; the row named the weaker of the two. The level does not change, which is the point: entrypoint says how the test runs, not whether it asks the right question. That limit is deliberate and now has a name. Restoring the old reference under mutation leaves the suite green, because no measurement here reads what a risk means. The column narrows where to look; it does not judge.
The release notes carry the column under Added. Risk 22 -- one developer, measuring by hand -- said its mitigation was that every closed row names a test rather than a memory. That is still true and now says more: the row names what kind of test, measured rather than asserted. Neither makes a second pair of eyes out of one, and the row says so; both shorten how long a row can be wrong without anyone noticing.
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
evkir
deleted the
fix/a-closed-row-names-a-test-that-touches-behaviour
branch
September 22, 2026 19:52
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
How it was measured
Checklist
ruff format --check cyberai/ tests/andruff check cyberai/ tests/passpytest -W ignore::DeprecationWarning -m "not slow and not smoke"passes