Skip to content

tri: restore cost and diffbin, and the same 634 specs now report 13 field losses where the old tool reported zero - #2159

Merged
gHashTag merged 1 commit into
masterfrom
w699-restore-loop-tools
Aug 20, 2026
Merged

tri: restore cost and diffbin, and the same 634 specs now report 13 field losses where the old tool reported zero#2159
gHashTag merged 1 commit into
masterfrom
w699-restore-loop-tools

Conversation

@gHashTag

Copy link
Copy Markdown
Owner

Closes #2158

What this establishes

Two measurement tools were unrecoverable, and the recovery attempt is documented rather than asserted. scripts/tri_loop/cost.py and diffbin.py were written, used to produce the "634 specs, 0 regressions" figure quoted in #2151, and never committed to any branch; the working copy was later re-cloned. All six routes named in the order were checked:

route result
dangling git objects one git stash WIP commit, holding only triage.py
reflog records the clone at b9287250 -- that is the loss, not the content
shell history no .bash_history or .zsh_history exists
CI artifacts fpga-synthesis, fpga-verilog, wp18-conformance-report only
PR and issue comments no comment carries the source of either script
session snapshot prose describing the scripts, not the scripts

So both are reimplemented from a written contract. That is the stronger outcome, because recalling what the old one roughly did would have reproduced its defect.

The old aggregate was wrong, and the same corpus now says so. Re-run over the identical 634 specs with the identical two binaries (/tmp/t27c.base and /tmp/t27c.fixed, both of which survived):

    616  unchanged
     13  field-loss
      1  strict-improvement
      4  malformed-input-tradeoff
      0  unknown

Previously: "0 regressions". specs/tri/agent/handoff.t27 goes from 35 parsed fields to 12. Verified by hand on specs/tri/crypto/sha256.t27, where base yields state and count and the candidate yields only state, having absorbed count : u64 into its type text.

The finding that decides #2151, and it is a favourable one. All 17 files whose field sets moved are inside the damaged set from #2154 -- 13 of 13 field-loss and 4 of 4 malformed-input-tradeoff. Not one well-formed spec changed behaviour, which is what 0 unknown is carrying. The recovery change is therefore safe on clean input and its entire cost is confined to files that are themselves corrupt. That reorders the queue: repair #2154 first and the field-loss question largely dissolves, because the inputs producing it stop existing.

The contract, and why each clause is there

tri diffbin -- five categories matched in order: unchanged, field-loss, strict-improvement, malformed-input-tradeoff, unknown. field-loss is tested before strict-improvement, so a change that removes a phantom and also drops a declared field is a loss, not an improvement. Exit status is driven by field-loss and unknown counts, never by a summary verdict.

The discrimination the old tool lacked: a removed field is a phantom only if its type text in the base was EMPTY. That is the signature of an identifier lifted out of a type-argument list and promoted to a field. A removed field with a non-empty base type was declared by the author, and its removal is a loss. Counting fields cannot tell the two apart, which is how the old aggregate missed them.

Second correction: only an ExprIdentifier whose parent is a StructDecl counts as a field. t27c parse prints that kind for every identifier in the tree including function bodies, so matching on kind alone would count expression operands as struct fields.

tri cost -- per stratum: n, median, p95, min-max ms/KB, coefficient of variation. Alpha only at n >= 8, printed beside its r2 and the KB range it was fitted over. No cross-family alpha is printed at all. Not printed-with-a-caveat: not printed. A single exponent across these strata is a metric of corpus composition rather than of the parser (#2133), and a number gets quoted while its caveat does not travel with it. Measured on specs/tri, alpha was withheld for 12 of 13 strata for insufficient n, and the one printed (utils, 0.14) carries r2 0.50 over a 2.2-3.0 KB range -- a slope through a cloud, which the output says.

Build profile is labelled, so a debug millisecond cannot be quoted as a release one.

tri damage -- classifies the corrupt annotations of #2154 by shape and repairs nothing: 125 lines, 65 files, 15 distinct shapes, one emitted fixture per shape. Repair can then be a reviewable diff of a stated size per class instead of one sweep over 125 lines.

Its first draft reported 429 damaged lines. 230 of those were target : < 5000ns -- a legitimate less-than bound, not an unbalanced bracket; 17 were match arms, 10 were multi-line array literals, 6 were function signatures the field regex should never have matched. So the headline measured the regex, not the corpus. The fix was deleting the two bad signals, not tuning a threshold until the number looked right. Two survive: [[], which the language has no syntax for, and an odd ". r#" is excluded after reading the three lines it flagged.

This also revises the previous loop's figure upward from 115 lines / 63 files, because the older pattern missed the [?[] and [std.StringHashMap(...)" shapes.

scripts/ci/loop-tools-tracked.sh -- fails when a required tool is missing, when it exists but git does not track it, when anything under scripts/tri_loop/ is untracked, or when the generic dispatch line is gone. Verified to fail in exactly the pre-loss state (FAIL, four untracked files) and to pass after the commit. A check only verified in its passing state is decoration.

Its own check 4 was wrong in its first draft: it grepped scripts/tri for each command name, and tri triage passed only because the word appears in a comment. A check that detects its own documentation is worth nothing, so it now verifies the dispatch line and the resolved target file.

Dispatcher defect fixed. scripts/tri looked for a built t27c and exited before dispatching loop helpers. tri triage and tri damage read the tracker and the spec text and never touch the compiler, yet a machine with no build could not run them. Loop dispatch now precedes the binary lookup, and the new workflow job runs with no compiler present to keep it that way.

What this does NOT establish

Reproduce

./scripts/tri damage specs
./scripts/tri diffbin /tmp/t27c.base /tmp/t27c.fixed specs --jsonl /tmp/d.jsonl
./scripts/tri cost /tmp/t27c.fixed specs/tri --repeat 1 --min-kb 2
bash scripts/ci/loop-tools-tracked.sh

@github-actions

Copy link
Copy Markdown
Contributor

PR Dashboard

Generated at: 2026-08-14 19:26:54 UTC

Summary

Status Count
Total Open PRs 22
PRs with Failing Checks 8
PRs with All Checks Green 14
READY 6
FAILING 8
PENDING 0

Seal Status

  • ⚠️ STALE -- sha256(compiler.rs)=cd2822f290eb != manifest seal=87e5cbd3ad94.
    The committed NMSE numbers were certified against an older compiler.rs.
    Run scripts/reseal-check.sh locally for the two-step reseal command (advisory; not a merge gate).

@github-actions

Copy link
Copy Markdown
Contributor

📓 NotebookLM Notebook linked to this PR

This notebook contains session context, decisions, and artifacts for this work.

This was referenced Aug 18, 2026
gHashTag added a commit that referenced this pull request Aug 19, 2026
`tri diffbin` reported five categories, and the first one carried two different
statements at once: "both binaries parsed and the field sets are identical", and
"neither binary parsed the file". Both incremented the same counter.

Same binaries, same 150-file slice, two versions of the same tool [measured]:

  five categories:  150 unchanged, 0 field-loss, 0 unknown      -- total agreement
  six categories:    59 unchanged, 91 not-evaluated (both-error) -- coverage 39.3%

Nothing about the compilers changed between those two lines. The first line is the
shape that had been quoted in pull requests. Over the 634-spec library the split
was 330 measured against 286 unmeasured: a 52 % base printed as 100 %.

What changes:
  * six mutually exclusive categories, with the partition ASSERTED at runtime
    rather than claimed in a docstring
  * every not-evaluated row carries a reason code -- both-error, base-timeout,
    candidate-timeout, environment-failure, excluded-source-loss, or a named
    other. "No verdict" for six reasons is six facts, and only some of them are
    about the compiler. An uncoded not-evaluated is a hard error, exit 2.
  * no PASS while any unknown remains, whatever the other counts say
  * coverage printed on every run, and required in the same sentence as any "no
    regressions" claim. A caveat in a neighbouring paragraph does not travel with
    the number; readers quote numbers, not paragraphs.
  * --exclude-status takes a corpus-status JSON and holds out
    unrecoverable-source-loss files under their own code. It REFUSES to run if the
    list selects nothing, because a silently empty exclusion produces a report
    claiming files were held out when none were.

Full corpus, single completed run, uniform 12 s threshold, 1089 files [measured]:
524 unchanged/ok, 343 unchanged/fail, 0 regressions, 1 strict-improvement, 221
not-evaluated (195 both-timeout, 26 candidate-timeout). Coverage 868/1089 = 79.7 %.

The 26 ok->timeout files are the threshold, not a slowdown. Timed directly, three
runs each way: median candidate/base ratio 1.010, min 0.985, max 1.026, on files
taking 10.8-11.7 s against a 12 s wall [measured]. A 1-3 % jitter moves them
across it.

Also: R15 and R16 added to docs/loop/LOOP-RULES.md and resealed; corpus-parse,
corpus-status, diffmodes and loop-rules registered in loop-tools-tracked.sh, which
fails on an untracked tool -- the exact state that already destroyed two of these
scripts and every number they had produced.

Overlap disclosed: cost.py, diffbin.py, damage*.py, triage.py and
loop-tools-tracked.sh also appear in the branches behind #2159 and #2161. They are
included here because a measurement fix must not sit in a deferred branch while
its output is being quoted. Whichever lands first, the other needs a rebase; the
diffbin.py here supersedes both.

Not claimed: that coverage improved. It did not. Coverage rises only by repairing
files or excluding them with a status -- never by recategorising them.
@gHashTag
gHashTag force-pushed the w699-restore-loop-tools branch from 6cff8f5 to 5da9ab1 Compare August 19, 2026 21:07
@github-actions

Copy link
Copy Markdown
Contributor

📓 NotebookLM Notebook linked to this PR

This notebook contains session context, decisions, and artifacts for this work.

@github-actions

Copy link
Copy Markdown
Contributor

PR Dashboard

Generated at: 2026-08-19 21:11:41 UTC

Summary

Status Count
Total Open PRs 25
PRs with Failing Checks 10
PRs with All Checks Green 15
READY 7
FAILING 10
PENDING 0

Seal Status

  • ⚠️ STALE -- sha256(compiler.rs)=375b2f88cc2f != manifest seal=87e5cbd3ad94.
    The committed NMSE numbers were certified against an older compiler.rs.
    Run scripts/reseal-check.sh locally for the two-step reseal command (advisory; not a merge gate).

gHashTag added a commit that referenced this pull request Aug 19, 2026
`tri diffbin` reported five categories, and the first one carried two different
statements at once: "both binaries parsed and the field sets are identical", and
"neither binary parsed the file". Both incremented the same counter.

Same binaries, same 150-file slice, two versions of the same tool [measured]:

  five categories:  150 unchanged, 0 field-loss, 0 unknown      -- total agreement
  six categories:    59 unchanged, 91 not-evaluated (both-error) -- coverage 39.3%

Nothing about the compilers changed between those two lines. The first line is the
shape that had been quoted in pull requests. Over the 634-spec library the split
was 330 measured against 286 unmeasured: a 52 % base printed as 100 %.

What changes:
  * six mutually exclusive categories, with the partition ASSERTED at runtime
    rather than claimed in a docstring
  * every not-evaluated row carries a reason code -- both-error, base-timeout,
    candidate-timeout, environment-failure, excluded-source-loss, or a named
    other. "No verdict" for six reasons is six facts, and only some of them are
    about the compiler. An uncoded not-evaluated is a hard error, exit 2.
  * no PASS while any unknown remains, whatever the other counts say
  * coverage printed on every run, and required in the same sentence as any "no
    regressions" claim. A caveat in a neighbouring paragraph does not travel with
    the number; readers quote numbers, not paragraphs.
  * --exclude-status takes a corpus-status JSON and holds out
    unrecoverable-source-loss files under their own code. It REFUSES to run if the
    list selects nothing, because a silently empty exclusion produces a report
    claiming files were held out when none were.

Full corpus, single completed run, uniform 12 s threshold, 1089 files [measured]:
524 unchanged/ok, 343 unchanged/fail, 0 regressions, 1 strict-improvement, 221
not-evaluated (195 both-timeout, 26 candidate-timeout). Coverage 868/1089 = 79.7 %.

The 26 ok->timeout files are the threshold, not a slowdown. Timed directly, three
runs each way: median candidate/base ratio 1.010, min 0.985, max 1.026, on files
taking 10.8-11.7 s against a 12 s wall [measured]. A 1-3 % jitter moves them
across it.

Also: R15 and R16 added to docs/loop/LOOP-RULES.md and resealed; corpus-parse,
corpus-status, diffmodes and loop-rules registered in loop-tools-tracked.sh, which
fails on an untracked tool -- the exact state that already destroyed two of these
scripts and every number they had produced.

Overlap disclosed: cost.py, diffbin.py, damage*.py, triage.py and
loop-tools-tracked.sh also appear in the branches behind #2159 and #2161. They are
included here because a measurement fix must not sit in a deferred branch while
its output is being quoted. Whichever lands first, the other needs a rebase; the
diffbin.py here supersedes both.

Not claimed: that coverage improved. It did not. Coverage rises only by repairing
files or excluding them with a status -- never by recategorising them.
@gHashTag
gHashTag force-pushed the w699-restore-loop-tools branch from 5da9ab1 to c426e79 Compare August 19, 2026 21:13
@github-actions

Copy link
Copy Markdown
Contributor

📓 NotebookLM Notebook linked to this PR

This notebook contains session context, decisions, and artifacts for this work.

gHashTag added a commit that referenced this pull request Aug 19, 2026
`tri diffbin` reported five categories, and the first one carried two different
statements at once: "both binaries parsed and the field sets are identical", and
"neither binary parsed the file". Both incremented the same counter.

Same binaries, same 150-file slice, two versions of the same tool [measured]:

  five categories:  150 unchanged, 0 field-loss, 0 unknown      -- total agreement
  six categories:    59 unchanged, 91 not-evaluated (both-error) -- coverage 39.3%

Nothing about the compilers changed between those two lines. The first line is the
shape that had been quoted in pull requests. Over the 634-spec library the split
was 330 measured against 286 unmeasured: a 52 % base printed as 100 %.

What changes:
  * six mutually exclusive categories, with the partition ASSERTED at runtime
    rather than claimed in a docstring
  * every not-evaluated row carries a reason code -- both-error, base-timeout,
    candidate-timeout, environment-failure, excluded-source-loss, or a named
    other. "No verdict" for six reasons is six facts, and only some of them are
    about the compiler. An uncoded not-evaluated is a hard error, exit 2.
  * no PASS while any unknown remains, whatever the other counts say
  * coverage printed on every run, and required in the same sentence as any "no
    regressions" claim. A caveat in a neighbouring paragraph does not travel with
    the number; readers quote numbers, not paragraphs.
  * --exclude-status takes a corpus-status JSON and holds out
    unrecoverable-source-loss files under their own code. It REFUSES to run if the
    list selects nothing, because a silently empty exclusion produces a report
    claiming files were held out when none were.

Full corpus, single completed run, uniform 12 s threshold, 1089 files [measured]:
524 unchanged/ok, 343 unchanged/fail, 0 regressions, 1 strict-improvement, 221
not-evaluated (195 both-timeout, 26 candidate-timeout). Coverage 868/1089 = 79.7 %.

The 26 ok->timeout files are the threshold, not a slowdown. Timed directly, three
runs each way: median candidate/base ratio 1.010, min 0.985, max 1.026, on files
taking 10.8-11.7 s against a 12 s wall [measured]. A 1-3 % jitter moves them
across it.

Also: R15 and R16 added to docs/loop/LOOP-RULES.md and resealed; corpus-parse,
corpus-status, diffmodes and loop-rules registered in loop-tools-tracked.sh, which
fails on an untracked tool -- the exact state that already destroyed two of these
scripts and every number they had produced.

Overlap disclosed: cost.py, diffbin.py, damage*.py, triage.py and
loop-tools-tracked.sh also appear in the branches behind #2159 and #2161. They are
included here because a measurement fix must not sit in a deferred branch while
its output is being quoted. Whichever lands first, the other needs a rebase; the
diffbin.py here supersedes both.

Not claimed: that coverage improved. It did not. Coverage rises only by repairing
files or excluding them with a status -- never by recategorising them.
@github-actions

Copy link
Copy Markdown
Contributor

PR Dashboard

Generated at: 2026-08-19 21:17:11 UTC

Summary

Status Count
Total Open PRs 24
PRs with Failing Checks 8
PRs with All Checks Green 16
READY 8
FAILING 8
PENDING 0

Seal Status

  • ⚠️ STALE -- sha256(compiler.rs)=375b2f88cc2f != manifest seal=87e5cbd3ad94.
    The committed NMSE numbers were certified against an older compiler.rs.
    Run scripts/reseal-check.sh locally for the two-step reseal command (advisory; not a merge gate).

@github-actions

Copy link
Copy Markdown
Contributor

PR Dashboard

Generated at: 2026-08-20 00:40:35 UTC

Summary

Status Count
Total Open PRs 21
PRs with Failing Checks 6
PRs with All Checks Green 15
READY 7
FAILING 6
PENDING 0

Seal Status

  • ⚠️ STALE -- sha256(compiler.rs)=cbbfac87dff3 != manifest seal=87e5cbd3ad94.
    The committed NMSE numbers were certified against an older compiler.rs.
    Run scripts/reseal-check.sh locally for the two-step reseal command (advisory; not a merge gate).

@github-actions

Copy link
Copy Markdown
Contributor

📓 NotebookLM Notebook linked to this PR

This notebook contains session context, decisions, and artifacts for this work.

Closes #2158)

Two measurement tools were lost and every number they had produced became
unreproducible with them. cost.py and diffbin.py were written, quoted in #2151,
and never committed; the working copy was later re-cloned. Six recovery routes
came back empty -- dangling objects held only a git stash WIP with triage.py, the
reflog records the clone rather than the content, shell history is absent, CI
artifacts hold only FPGA outputs, no PR or issue comment carries the source, and
the session snapshot preserved prose about the scripts instead of the scripts.
So these are reimplementations from a written contract. Recalling what the old
ones roughly did would have reproduced the old one's defect.

That defect was the specification for the new one. It reported "0 regressions"
over 634 specs while files were losing declared struct fields, because a per-file
judgement had relabelled the loss as an acceptable trade and the aggregate then
printed the judgement as if it were a measurement.

  No differential result may be called "0 regressions" unless the metric
  actually checks the claimed class of loss.

diffbin now assigns five ordered categories -- unchanged, field-loss,
strict-improvement, malformed-input-tradeoff, unknown -- with field-loss tested
before strict-improvement, so removing a phantom while dropping a declared field
is a loss and not an improvement. Phantom and declared are told apart by a stated
rule: a removed field is a phantom only if its base type text was empty. Only an
ExprIdentifier whose parent is a StructDecl counts, so identifiers in function
bodies stay out of the totals.

Re-measured on the same 634 specs and the same two binaries: 616 unchanged, 13
field-loss, 1 strict-improvement, 4 malformed-input-tradeoff, 0 unknown.
handoff.t27 goes from 35 parsed fields to 12. All 17 files that moved are inside
the damaged set and no well-formed spec changed at all, which is what 0 unknown
is carrying.

cost reports per stratum with n, median, p95, min-max ms/KB and coefficient of
variation, alpha only at n >= 8 with its r2 and KB range, and no cross-family
alpha at all: that number is a metric of corpus composition rather than of the
parser (#2133), and a printed number gets quoted while its caveat does not
travel with it.

damage classifies the corrupt annotations by shape rather than repairing them
(#2154): 125 lines, 65 files, 15 shapes, one fixture each. The first draft
reported 429, of which 230 were the legitimate bound `target : < 5000ns`, so the
fix was deleting two bad signals rather than tuning a threshold.

loop-tools-tracked.sh fails when a loop tool is missing, untracked, or unrouted,
and was verified to fail in exactly the pre-loss state. The dispatcher no longer
looks for a built compiler before running helpers that never use one.
@gHashTag
gHashTag force-pushed the w699-restore-loop-tools branch from 0d5998f to 8631f8e Compare August 20, 2026 00:42
@github-actions

Copy link
Copy Markdown
Contributor

PR Dashboard

Generated at: 2026-08-20 00:42:44 UTC

Summary

Status Count
Total Open PRs 21
PRs with Failing Checks 6
PRs with All Checks Green 15
READY 7
FAILING 6
PENDING 0

Seal Status

  • ⚠️ STALE -- sha256(compiler.rs)=cbbfac87dff3 != manifest seal=87e5cbd3ad94.
    The committed NMSE numbers were certified against an older compiler.rs.
    Run scripts/reseal-check.sh locally for the two-step reseal command (advisory; not a merge gate).

@github-actions

Copy link
Copy Markdown
Contributor

📓 NotebookLM Notebook linked to this PR

This notebook contains session context, decisions, and artifacts for this work.

@gHashTag
gHashTag merged commit 99e9098 into master Aug 20, 2026
23 of 27 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

The two tools every parser claim rests on are lost, and the one that survived reported zero regressions while fields were being dropped

1 participant