Skip to content

The ref leg: conformance verdict becomes legs == REFERENCE where the evaluator judges (TOR-8); 30 selftest scenarios incl. the co-drift class - #39

Merged
O6lvl4 merged 1 commit into
mainfrom
ref-leg
Aug 21, 2026

Conversation

@O6lvl4

@O6lvl4 O6lvl4 commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

Second slice of #10: the runner adjudicates against the reference (TOR-8).

  • scripts/conformance.py --legs ref (in the default leg set): for every program in spec/wasm_cross + spec/programs, where the reference evaluator produces observables, native and wasm must each equal the reference's — agreement between the two targets is no longer sufficient there. A reference abstain (ledgered class) is counted and skipped, never a pass or fail; a malformed protocol reply or evaluator fault is red. // @ref-allow: <reason> tracks a disagreement under adjudication and goes STALE when healed — grain_functions.almd carries the first one (finding F1, named-argument evaluation order).
  • scripts/selftest-conformance.py: 21 → 30 scenarios with a scripted stub als-ref — including the class this whole instrument exists for: co-drift (targets agree with each other, both differ from the reference) is red. proofs/runner-coverage.toml floor 85.1% → 86.3% (--write in this PR, per the gate's ratchet).
  • conformance.yml builds ref/ (pinned 1.95.0, cached) and runs the leg by default; the statement records the reference's version beside the binary and wasmtime.
  • QUALIFICATION limitation 2 narrows rather than lifts: on the evaluated subset the runner judges truth; on the abstained subset it still judges agreement, bounded by the shrink-only proofs/ref-abstain.toml.

Smoke: --legs ref --limit 40 → 11 judged / 11 pass / 29 abstained. Full-corpus ref runs happen in the conformance workflow (weekly / dispatch), as with every other leg.

🤖 Generated with Claude Code

… evaluator judges (TOR-8) — abstains counted never verdicts, @ref-allow tracks adjudicated findings with stale detection, grain_functions carries F1's directive; nine selftest scenarios including the co-drift class, coverage floor 85.1 to 86.3; conformance workflow builds ref/ and runs the leg by default
@O6lvl4
O6lvl4 merged commit 32803c1 into main Aug 21, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant