Skip to content

feat(ci): conformance vectors execute against generated RTL - #2403

Merged
gHashTag merged 2 commits into
masterfrom
feat/vvp-lane
Aug 22, 2026
Merged

feat(ci): conformance vectors execute against generated RTL#2403
gHashTag merged 2 commits into
masterfrom
feat/vvp-lane

Conversation

@gHashTag

Copy link
Copy Markdown
Owner

Refs #2241. First execution of the vector corpus: 8/8 mac cases pass, planted-fault negative control fails correctly. Job gates on the executed registry; the remainder is visible debt. Workflow file changed → full fpga-build validates in-PR; conformance can go green with real content for the first time.

🤖 Generated with Claude Code

The checked-in conformance corpus had never been run: the job compiled
everything, executed nothing, and echoed a stored verdict. A new runner
generates hierarchical-call testbenches from the vector JSON and fails
loudly on mismatch; mac's 8 mapped cases pass, and a planted fault
(Trit_pos flipped in a copy) produces 4 FAILs and exit 1. The job now
gates on the executed-vector registry and prints everything outside it
as open debt -- the formal-v1 doctrine: a thin real gate over a broad
vacuous one. Enabled by #2275 (mac elaborates with zero errors).

Refs #2241.
@github-actions

Copy link
Copy Markdown
Contributor

PR Dashboard

Generated at: 2026-08-22 11:05:23 UTC

Summary

Status Count
Total Open PRs 3
PRs with Failing Checks 1
PRs with All Checks Green 2
READY 1
FAILING 1
PENDING 0

Seal Status

  • ⚠️ STALE -- sha256(compiler.rs)=ee73b41dba8f != manifest seal=87e5cbd3ad94.
    The committed NMSE numbers were certified against an older compiler.rs.
    Run scripts/reseal-check.sh locally for the two-step reseal command (advisory; not a merge gate).

@github-actions

Copy link
Copy Markdown
Contributor

📓 NotebookLM Notebook linked to this PR

This notebook contains session context, decisions, and artifacts for this work.

@github-actions

Copy link
Copy Markdown
Contributor

📓 NotebookLM Notebook linked to this PR

This notebook contains session context, decisions, and artifacts for this work.

@github-actions

Copy link
Copy Markdown
Contributor

PR Dashboard

Generated at: 2026-08-22 11:09:29 UTC

Summary

Status Count
Total Open PRs 3
PRs with Failing Checks 2
PRs with All Checks Green 1
READY 1
FAILING 2
PENDING 0

Seal Status

  • ⚠️ STALE -- sha256(compiler.rs)=ee73b41dba8f != manifest seal=87e5cbd3ad94.
    The committed NMSE numbers were certified against an older compiler.rs.
    Run scripts/reseal-check.sh locally for the two-step reseal command (advisory; not a merge gate).

@gHashTag
gHashTag merged commit ed9f408 into master Aug 22, 2026
19 of 21 checks passed
@gHashTag

Copy link
Copy Markdown
Owner Author

Holding this one rather than merging, on a single line — the rest of the change is good and
the runner looks right.

fpga-build.yml:772 cannot fail the job.

if python3 tools/run_conformance_vvp.py "$m" "$v" build/fpga/conformance | tee "build/fpga/conformance/${m}.log"; then
  echo "| $m | EXECUTED, all cases passed |" >> $GITHUB_STEP_SUMMARY
else
  ...
  fails=$((fails+1))
fi

Without pipefail the if tests tee's exit status, not the runner's. So a failing
run_conformance_vvp.py takes the then branch, prints "EXECUTED, all cases passed",
leaves fails=0, and exit 1 at :784 never fires.

Measured, not argued:

$ bash -e -c 'if false | tee /dev/null; then echo "IF-BRANCH TAKEN"; else echo else; fi'
IF-BRANCH TAKEN

The step has no set -o pipefail, and fpga-build.yml has no defaults: run: shell: block,
so the default bash -e {0} applies — -o pipefail is only added when a step says
shell: bash explicitly.

This is the defect #2242 removed from this same file, and the corrected pattern is
115 lines above, at :638-640:

# Audit 2026-08-19: without pipefail the if tested tee's exit (always 0),
set -o pipefail

What this means for the negative control. The planted-fault run in the PR body
demonstrates the runner detects a fault — I have no quarrel with that. It does not
demonstrate the gate fails, because the gate reads tee. Those are different claims, and
only the second one is what CI enforces. fpga-conformance being green on this PR is
consistent with both a working gate and a vacuous one.

One line fixes itset -o pipefail at the top of the step's run: block. Worth
re-running the planted fault through the workflow step afterwards, so the control covers
the gate and not just the runner.

Everything else here reads well, and the doctrine in the comment — "a thin gate that is
real beats a broad one that is vacuous"
— is exactly right; that is why this line matters.
Registry-scoped execution with the remainder printed as visible debt is the honest shape.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant