Skip to content

ci(fpga): pipefail the conformance step so a failing runner fails the job - #2417

Merged
gHashTag merged 1 commit into
masterfrom
fix/2415-conformance-pipefail
Aug 22, 2026
Merged

ci(fpga): pipefail the conformance step so a failing runner fails the job#2417
gHashTag merged 1 commit into
masterfrom
fix/2415-conformance-pipefail

Conversation

@gHashTag

Copy link
Copy Markdown
Owner

Closes #2415

What

fpga-build.yml's conformance step, landed by #2403, cannot fail. Line 772:

if python3 tools/run_conformance_vvp.py "$m" "$v" build/fpga/conformance | tee "build/fpga/conformance/${m}.log"; then
  echo "| $m | EXECUTED, all cases passed |" >> $GITHUB_STEP_SUMMARY
else
  echo "| $m | FAILED (see log) |" >> $GITHUB_STEP_SUMMARY
  fails=$((fails+1))
fi

Without pipefail the if tests tee's exit status. A failing runner takes the then
branch, fails stays 0, and the exit 1 at :784 never fires.

The step has no set -o pipefail, and the workflow has no defaults: run: shell: block, so
the default bash -e {0} applies — -o pipefail is added only for an explicit
shell: bash.

Measured, with a stub runner that fails the way a real mismatch would

$ cat runner.sh
#!/bin/sh
echo "case 3: MISMATCH expected 0x2a got 0x00"
exit 1

As landed on master:

case 3: MISMATCH expected 0x2a got 0x00
| mac | EXECUTED, all cases passed |
fails=0
STEP EXIT 0 — job green

With set -o pipefail, same runner:

case 3: MISMATCH expected 0x2a got 0x00
| mac | FAILED |
fails=1
::error::conformance execution failed
step exit 1

The gate reports "all cases passed" on the line directly below its own log saying
MISMATCH.

This is the defect #2242 removed, in the same file

#2242 fixed exactly this in fpga-formal, and the corrected pattern sits 115 lines
above
the new one, at :638-640:

# Audit 2026-08-19: without pipefail the if tested tee's exit (always 0),
set -o pipefail

On the negative control

#2403's body reports a planted-fault control failing correctly, and I have no quarrel with
it: it demonstrates the runner detects a fault. It does not demonstrate the gate
fails, because the gate reads tee. Those are different claims and only the second is what
CI enforces. fpga-conformance going green is consistent with both a working gate and a
vacuous one — which is precisely why the control has to run through the workflow step, not
beside it.

Raised on the PR before merge (#2403 review comment) and merged unchanged; filing so it is
tracked rather than lost.

Not in dispute

The rest of #2403 is good and this issue does not touch it: registry-scoped execution with
the remainder printed as visible debt is the honest shape, and its own comment — "a thin
gate that is real beats a broad one that is vacuous"
— is exactly the right doctrine.

Related: #2241, #2242, #2403, #2376.

@github-actions

Copy link
Copy Markdown
Contributor

📓 NotebookLM Notebook linked to this PR

This notebook contains session context, decisions, and artifacts for this work.

@github-actions

Copy link
Copy Markdown
Contributor

PR Dashboard

Generated at: 2026-08-22 12:10:24 UTC

Summary

Status Count
Total Open PRs 4
PRs with Failing Checks 1
PRs with All Checks Green 3
READY 2
FAILING 1
PENDING 0

Seal Status

  • ⚠️ STALE -- sha256(compiler.rs)=3ea6c4644e23 != manifest seal=87e5cbd3ad94.
    The committed NMSE numbers were certified against an older compiler.rs.
    Run scripts/reseal-check.sh locally for the two-step reseal command (advisory; not a merge gate).

@gHashTag
gHashTag enabled auto-merge (squash) August 22, 2026 12:10
… job

if python3 run_conformance_vvp.py ... | tee ...; then tests tee's exit
status without pipefail, so a MISMATCH printed one line above was
reported as "EXECUTED, all cases passed" and the job stayed green.

Same defect #2242 removed from fpga-formal, 115 lines below the corrected
pattern in this same file.

Closes #2415
@gHashTag
gHashTag force-pushed the fix/2415-conformance-pipefail branch from effe2f9 to 394e9a2 Compare August 22, 2026 12:15
@github-actions

Copy link
Copy Markdown
Contributor

PR Dashboard

Generated at: 2026-08-22 12:15:52 UTC

Summary

Status Count
Total Open PRs 4
PRs with Failing Checks 1
PRs with All Checks Green 3
READY 2
FAILING 1
PENDING 0

Seal Status

  • ⚠️ STALE -- sha256(compiler.rs)=3ea6c4644e23 != manifest seal=87e5cbd3ad94.
    The committed NMSE numbers were certified against an older compiler.rs.
    Run scripts/reseal-check.sh locally for the two-step reseal command (advisory; not a merge gate).

@github-actions

Copy link
Copy Markdown
Contributor

📓 NotebookLM Notebook linked to this PR

This notebook contains session context, decisions, and artifacts for this work.

@gHashTag
gHashTag merged commit 506357c into master Aug 22, 2026
24 of 26 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

The conformance step reports 'all cases passed' when the runner fails: no pipefail on the if

1 participant