feat(ci): conformance vectors execute against generated RTL - #2403
Conversation
The checked-in conformance corpus had never been run: the job compiled everything, executed nothing, and echoed a stored verdict. A new runner generates hierarchical-call testbenches from the vector JSON and fails loudly on mismatch; mac's 8 mapped cases pass, and a planted fault (Trit_pos flipped in a copy) produces 4 FAILs and exit 1. The job now gates on the executed-vector registry and prints everything outside it as open debt -- the formal-v1 doctrine: a thin real gate over a broad vacuous one. Enabled by #2275 (mac elaborates with zero errors). Refs #2241.
PR DashboardGenerated at: 2026-08-22 11:05:23 UTC
Summary
Seal Status
|
|
📓 NotebookLM Notebook linked to this PR
This notebook contains session context, decisions, and artifacts for this work. |
|
📓 NotebookLM Notebook linked to this PR
This notebook contains session context, decisions, and artifacts for this work. |
PR DashboardGenerated at: 2026-08-22 11:09:29 UTC
Summary
Seal Status
|
|
Holding this one rather than merging, on a single line — the rest of the change is good and
if python3 tools/run_conformance_vvp.py "$m" "$v" build/fpga/conformance | tee "build/fpga/conformance/${m}.log"; then
echo "| $m | EXECUTED, all cases passed |" >> $GITHUB_STEP_SUMMARY
else
...
fails=$((fails+1))
fiWithout Measured, not argued: The step has no This is the defect #2242 removed from this same file, and the corrected pattern is # Audit 2026-08-19: without pipefail the if tested tee's exit (always 0),
set -o pipefailWhat this means for the negative control. The planted-fault run in the PR body One line fixes it — Everything else here reads well, and the doctrine in the comment — "a thin gate that is |
Refs #2241. First execution of the vector corpus: 8/8 mac cases pass, planted-fault negative control fails correctly. Job gates on the executed registry; the remainder is visible debt. Workflow file changed → full fpga-build validates in-PR; conformance can go green with real content for the first time.
🤖 Generated with Claude Code