gft self-check: fourteen more plants raise the assert column 2/34 -> 16/34 (Refs #2161) - #2702
Merged
Conversation
…16/34 (Refs #2161) The column was the number of planted faults, not a verdict on 32 assertions (§123). The model was tested by predicting before measuring: 2 plants -> 2/34, +7 -> 9/34, +7 more -> 16/34. Three for three. Cost was never the obstacle and nobody had checked. Each new plant fires in the arithmetic block at the top of `__main__`, before any training: 0.06s apiece. The control went 11.8s -> 12.5s for eight times the coverage. The old assumption that more plants meant more whole-program runs was generalised from a sample of two, both of which are full runs. Within a family the fault has to be surgical -- adjacent assertions test adjacent cases, so the obvious fault breaks them all and only the first is seen. `elif t == hf and (s & 1)` -> `... and False` kills the ODD tie only, leaving the even one true so it passes and the odd one speaks. One resisted and is left alone: disabling `_magsub`'s `if rem > half` also breaks the renormalisation-carry case, which is checked earlier, so the plant fires that instead. Separating them would mean encoding the exact remainders, and then the control is a second copy of the subject. Also: `spawned()` now asserts the plant edited the SUBJECT. All subject code sits above the control and every assertion under test below it, so the first changed byte must precede `def self_check(`. T124 caught a plant that edited the control's own source; T211 one that edited the assertion checking the result. Both went green. Neither was caught by reading; both would have been caught by this line.
Contributor
|
📓 NotebookLM Notebook linked to this PR
This notebook contains session context, decisions, and artifacts for this work. |
Contributor
PR DashboardGenerated at: 2026-08-24 10:55:07 UTC
Summary
Seal Status
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
§123 established that
--assert's 2 of 34 was the number of planted faults, not a verdict on 32 assertions. This tick tested that model the only way a model can be tested: by predicting the number before measuring it.Three for three.
Cost was never the obstacle, and nobody had checked
Each new plant fires in the arithmetic block at the top of
__main__, long before any training runs: 0.06s apiece. The whole control went 11.8s → 12.5s for eight times the coverage.Three iterations deferred this column partly on an unexamined assumption that more plants meant more whole-program runs. The two existing plants are full runs, so the assumption generalised from a sample of two. A cost you have not measured is a reason you have not checked.
Within a family, the fault has to be surgical
Adjacent assertions test adjacent cases of the same code, so the obvious fault breaks them all and only the first is ever seen. To surface the second member, the fault must falsify it while leaving the first true:
The even tie still correctly declines to round up, passes, and lets the odd one speak. Seven of eight designed this way hit their target exactly.
The eighth is the lesson arriving inside the lesson
Disabling
_magsub'sif rem > halfto surface "strictly above half" also breaks the renormalisation-carry case — checked earlier — so the plant fires that one instead.Left alone on purpose. A plant narrow enough to separate them would have to encode the exact remainders, and then the control becomes a second copy of the thing it checks, failing for its own reasons and agreeing with the subject about all of them.
A guard that mechanises two earlier lessons
T124 caught a plant whose needle's first occurrence was the control's own source. T211 caught one whose needle's only occurrence was the assertion checking the result. Both went green; both had names that were lies.
Every piece of subject code sits above the control; every assertion under test sits below it. One comparison decides it:
Positive control — a plant that edits an assertion's text is now refused:
Neither earlier case was caught by reading; both would have been caught by this line.
A smaller one worth keeping
Listing the file's assertions with a throwaway
line.strip().startswith("assert ")scored prose inside the control's docstring as an assertion — the exact bug fixed inassert_sitesone tick earlier. The tool had learned; my one-off script had not.The tell was the classification coming out nonsensical: zero training assertions in a file that is mostly training assertions.
18 survivors remain — the emitter and held-out-accuracy assertions, which do cost a full run each, plus one precondition (
gen_deep'slen(sizes) >= 2) whose only caller sits below the guard.§124.
Refs #2161