Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
145 changes: 122 additions & 23 deletions .github/actions/reprocase-scan/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ inputs:
description: 'Entry URL to scan (e.g. http://localhost:3000/ — start your site in an earlier step).'
required: true
reprocase-ref:
description: 'ReproCase version to install (tag, branch or commit). Defaults to the ref this action was pinned to.'
description: 'ReproCase version to install: a tag (v0.5.1), a branch (main), or a full commit SHA. Defaults to the ref this action was pinned to.'
required: false
default: ''
config:
Expand All @@ -24,11 +24,11 @@ inputs:
required: false
default: ''
fail-on:
description: 'Exit 1 when "verified" or "suspected" findings exist, or "never" (report only).'
description: 'Exit 1 when "verified" or "suspected" findings exist, or "never" (report only). Does NOT affect the baseline gate — see fail-on-new.'
required: false
default: 'verified'
fail-on-new:
description: 'When compare-with is set: exit 1 if any NEW findings exist. Default true.'
description: 'When compare-with is set: "true" (default) fails the action if new findings exist vs baseline; "false" reports the diff but does not gate on new findings.'
required: false
default: 'true'
max-pages:
Expand All @@ -53,8 +53,11 @@ outputs:
description: 'clean (exit 0) | findings (exit 1) | error (exit 2)'
value: ${{ steps.scan.outputs.verdict }}
exit-code:
description: 'Raw scanner exit code (0/1/2).'
value: ${{ steps.scan.outputs.exit-code }}
description: 'Final action exit code (0/1/2).'
value: ${{ steps.decision.outputs.exit-code }}
scan-exit-code:
description: 'Raw scanner exit code before gating.'
value: ${{ steps.scan.outputs.scan-exit-code }}
new-findings:
description: 'Number of new findings vs the baseline (empty when no baseline was given).'
value: ${{ steps.gate.outputs.new-findings }}
Expand All @@ -67,24 +70,45 @@ runs:
node-version: ${{ inputs.node-version }}

- name: Install ReproCase
id: install
shell: bash
env:
REPROCASE_REF: ${{ inputs.reprocase-ref || github.action_ref }}
run: |
set -euo pipefail
REF="${REPROCASE_REF}"
case "$REF" in
''|refs/*) REF="v0.5.0" ;;
esac
# Strip leading "refs/" so the composite action's own ref value works.
case "$REF" in refs/*) REF="${REF#refs/}";; esac
# Default: a known-good release tag (never silently fall back to
# another ref when the user explicitly provided one that doesn't exist).
if [ -z "$REF" ]; then REF="v0.5.1"; fi
SRC="${RUNNER_TEMP}/reprocase-src"
rm -rf "$SRC"
echo "Installing ReproCase from github.com/geyutu6755/ReproCase ($REF)"
git clone --depth 1 --branch "$REF" https://github.com/geyutu6755/ReproCase.git "$SRC"
# Try tag/branch first via --branch; fall back to SHA via raw checkout.
if git clone --depth 1 --branch "$REF" https://github.com/geyutu6755/ReproCase.git "$SRC" 2>/dev/null; then
echo "Cloned tag/branch: $REF"
else
echo "--branch clone failed; trying SHA checkout"
git clone https://github.com/geyutu6755/ReproCase.git "$SRC"
cd "$SRC"
if git cat-file -e "$REF^{commit}" 2>/dev/null; then
git checkout "$REF"
echo "Checked out SHA: $(git rev-parse HEAD)"
else
echo "::error::ReproCase ref '$REF' not found as a tag, branch or commit SHA in geyutu6755/ReproCase. Available tags: https://github.com/geyutu6755/ReproCase/tags"
exit 1
fi
fi
cd "$SRC"
ACTUAL=$(git rev-parse HEAD)
echo "reprocase-commit=$ACTUAL" >> "$GITHUB_OUTPUT"
echo "Checked out: $ACTUAL"
# Global installs omit devDependencies, and the TypeScript build needs
# them — so build here (npm ci includes dev deps) and install the
# built package globally.
npm install --no-fund --no-audit
# them — so build here (npm ci uses the lockfile for reproducibility)
# and install the built package globally. The prepare script runs
# after dist/ already exists, so it's a no-op at that point.
npm ci --no-fund --no-audit
npm run build
npm install -g --no-fund --no-audit .
reprocase --version
Expand Down Expand Up @@ -115,7 +139,7 @@ runs:
reprocase "${ARGS[@]}"
CODE=$?
set -e
echo "exit-code=$CODE" >> "$GITHUB_OUTPUT"
echo "scan-exit-code=$CODE" >> "$GITHUB_OUTPUT"
case "$CODE" in
0)
echo "verdict=clean" >> "$GITHUB_OUTPUT"
Expand All @@ -124,27 +148,41 @@ runs:
1)
echo "verdict=findings" >> "$GITHUB_OUTPUT"
echo "ReproCase: scan completed — findings exist at or above the threshold."
# Do NOT exit 1 here: the gate step needs to run to compare against
# the baseline. The final decision step handles the exit code.
;;
2)
echo "verdict=error" >> "$GITHUB_OUTPUT"
echo "::error::ReproCase could not run properly (exit 2). This is an environment or configuration problem, NOT a verdict about your website. See the log above; 'reprocase doctor' diagnoses most causes."
exit 2
;;
*)
echo "verdict=error" >> "$GITHUB_OUTPUT"
echo "::error::ReproCase exited unexpectedly (code $CODE)."
exit 1
;;
esac
exit "$CODE"

- name: Compare with baseline (new-fault gate)
id: gate
if: ${{ inputs.compare-with != '' }}
if: ${{ inputs.compare-with != '' && steps.scan.outputs.verdict != 'error' }}
shell: bash
env:
BASELINE: ${{ inputs.compare-with }}
FAIL_ON_NEW: ${{ inputs.fail-on-new }}
FAIL_ON_NEW_RAW: ${{ inputs.fail-on-new }}
run: |
set -uo pipefail
# Normalise fail-on-new: case-insensitive, trim whitespace.
FAIL_ON_NEW=$(echo "$FAIL_ON_NEW_RAW" | tr -d '[:space:]' | tr '[:upper:]' '[:lower:]')
case "$FAIL_ON_NEW" in
true|'') GATE_FLAG="--fail-on new" ;;
false) GATE_FLAG="" ;;
*)
echo "::error::Invalid fail-on-new value '$FAIL_ON_NEW_RAW'. Use 'true' or 'false'."
exit 1
;;
esac
echo "fail-on-new-normalised=$FAIL_ON_NEW" >> "$GITHUB_OUTPUT"
if [ ! -f "$BASELINE" ]; then
echo "::notice::Baseline '$BASELINE' not found — first run on this branch? Skipping the new-findings gate; this scan becomes the new baseline candidate."
echo "new-findings=" >> "$GITHUB_OUTPUT"
Expand All @@ -154,19 +192,78 @@ runs:
echo "::error::Scan produced no result.json — the scan may have failed before writing results. Cannot compare."
exit 1
fi
GATE_ARGS=(compare "$BASELINE" reprocase-report/result.json --out reprocase-report/compare --fail-on new)
GATE_ARGS=(compare "$BASELINE" reprocase-report/result.json --out reprocase-report/compare)
[ -n "$GATE_FLAG" ] && GATE_ARGS+=("$GATE_FLAG")
set +e
reprocase "${GATE_ARGS[@]}"
CODE=$?
set -e
NEW=$(node -e "const r=require('./reprocase-report/compare/compare.json');console.log(r.summary.newCount)" 2>/dev/null || echo 0)
echo "new-findings=$NEW" >> "$GITHUB_OUTPUT"
case "$CODE" in
0) echo "ReproCase compare: no new findings." ;;
1) echo "::error::ReproCase compare: $NEW new finding(s) vs baseline (coverage shrank = unconfirmed, see report)." ; exit 1 ;;
2) echo "::error::ReproCase compare failed to run (bad baseline file?). Fix the baseline or remove compare-with." ; exit 1 ;;
0)
echo "ReproCase compare: no new findings (gate passed)."
;;
1)
if [ "$FAIL_ON_NEW" = "true" ]; then
echo "::error::ReproCase compare: $NEW new finding(s) vs baseline. Gate: FAIL (fail-on-new=true)."
exit 1
else
echo "::warning::ReproCase compare: $NEW new finding(s) vs baseline. Gate: PASS (fail-on-new=false — report only)."
fi
;;
2)
echo "::error::ReproCase compare failed to run (bad baseline file?). Fix the baseline or remove compare-with."
exit 1
;;
esac

- name: Final CI decision
id: decision
if: ${{ always() }}
shell: bash
env:
SCAN_VERDICT: ${{ steps.scan.outputs.verdict }}
SCAN_EXIT: ${{ steps.scan.outputs.scan-exit-code }}
GATE_CONCLUSION: ${{ steps.gate.conclusion }}
INSTALL_CONCLUSION: ${{ steps.install.conclusion }}
FAIL_ON_NEW_RAW: ${{ inputs.fail-on-new }}
HAS_COMPARE: ${{ inputs.compare-with != '' }}
run: |
set -uo pipefail
# Propagate the gate's new-findings output.
NEW="${{ steps.gate.outputs.new-findings }}"
echo "new-findings=$NEW" >> "$GITHUB_OUTPUT"

FAIL_ON_NEW=$(echo "$FAIL_ON_NEW_RAW" | tr -d '[:space:]' | tr '[:upper:]' '[:lower:]')

# Priority 1: installation or scan environment error (exit 2).
if [ "$INSTALL_CONCLUSION" = "failure" ] || [ "$SCAN_VERDICT" = "error" ]; then
echo "exit-code=2" >> "$GITHUB_OUTPUT"
echo "Final verdict: error (environment/configuration problem)."
exit 2
fi

# Priority 2: baseline gate failed (new findings with fail-on-new=true,
# or compare infrastructure error).
if [ "$HAS_COMPARE" = "true" ] && [ "$GATE_CONCLUSION" = "failure" ]; then
echo "exit-code=1" >> "$GITHUB_OUTPUT"
echo "Final verdict: gate failed (new findings vs baseline)."
exit 1
fi

# Priority 3: scan findings at or above the fail-on threshold.
if [ "$SCAN_VERDICT" = "findings" ]; then
echo "exit-code=1" >> "$GITHUB_OUTPUT"
echo "Final verdict: findings exist (scan exit $SCAN_EXIT)."
exit 1
fi

# Priority 4: all checks passed.
echo "exit-code=0" >> "$GITHUB_OUTPUT"
echo "Final verdict: clean."
exit 0

- name: Publish privacy-stripped report
if: ${{ always() }}
shell: bash
Expand All @@ -176,11 +273,13 @@ runs:
# evidence, never uploaded. Screenshots are of YOUR site: keep them,
# they belong to you. result.json / report.html / issues.md are
# redacted at source (token/session query params).
rm -rf reprocase-report/artifacts/traces
ls reprocase-report
if [ -d reprocase-report ]; then
rm -rf reprocase-report/artifacts/traces
ls reprocase-report
fi

- uses: actions/upload-artifact@v4
if: ${{ always() }}
if: ${{ always() && hashFiles('reprocase-report/**') != '' }}
with:
name: ${{ inputs.report-name }}
path: reprocase-report/
Expand Down
27 changes: 27 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,32 @@ All notable changes to ReproCase are documented here.
Format based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/);
versioning follows [Semantic Versioning](https://semver.org/).

## [0.5.1] — 2026-11-06

GitHub Actions composite-action reliability release. Three real defects found by the
external-repo CI runs and code audit.

### Fixed

- fail-on-new input was declared but its env var never read (P1): the gate always used
the new-findings gate regardless of the user setting. Now: true gates on new findings;
false reports the diff without gating; invalid values produce an explicit error.
- Scan step exited 1, causing GitHub Actions to skip the gate step (P1): the scan step
now completes on exit 0 or 1 (recording the verdict) and only exits 2 for environment
errors. A new Final CI decision step evaluates scan verdict + gate conclusion in priority
order (error > gate > findings > clean) and produces the final exit code.
- reprocase-ref SHA support was claimed but not implemented (P2): git clone --branch
only works for tags/branches. Now: --branch first, then fallback to full clone + git
checkout for SHAs, with an explicit error for unresolvable refs. The checked-out commit
is verified and reported.
- npm install to npm ci for lockfile-reproducible installs; hashFiles guard prevents
empty artifact upload.

### Verification

160/160 tests; npm audit 0; external repo validation (clean green, incident red, fix green,
honest-error green); four-OS CI matrix.

## [0.5.0] — 2026-11-05

Developer-workflow integration release: run ReproCase in your own GitHub Actions pipelines,
Expand Down Expand Up @@ -476,6 +502,7 @@ First verified release.
- No accessibility, performance, or visual-regression coverage beyond horizontal overflow.
- Single-browser (Chromium) at this stage; Firefox/WebKit untested.

[0.5.1]: https://github.com/geyutu6755/ReproCase/releases/tag/v0.5.1
[0.5.0]: https://github.com/geyutu6755/ReproCase/releases/tag/v0.5.0
[0.4.0]: https://github.com/geyutu6755/ReproCase/releases/tag/v0.4.0
[0.3.3]: https://github.com/geyutu6755/ReproCase/releases/tag/v0.3.3
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -152,7 +152,7 @@ fault through the real Playwright runner.

## Status

v0.5.0 — developer-workflow integration release (composite GitHub Action, baseline new-fault gating, honest CI exit semantics, cluster-aware issue markdown). See the [changelog](CHANGELOG.md), the [Actions guide](docs/github-actions.md) and the [compatibility matrix](docs/compatibility.md).
v0.5.1 — GitHub Actions reliability release (fail-on-new semantics, exit-code ordering, SHA support, npm ci). See the [changelog](CHANGELOG.md).
`npm i -g github:…#tag` now works and the quick start drops the manual build), a reconciled
Node baseline (**22+**, since Node 18/20 are End-of-Life; CI-tested on 22 & 24), the deprecated
Vitest `poolOptions` config fixed, and refreshed real screenshots. Detection itself is unchanged
Expand Down
2 changes: 1 addition & 1 deletion README.zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -139,7 +139,7 @@ npm run test:all # 单元 + 集成(真实浏览器)+ CLI 端到端

## 状态

v0.5.0 —— 开发者工作流集成版本(复合 GitHub Action、基线新增故障门禁、诚实的 CI 退出语义、聚类感知 Issue 文档)。详见 [变更日志](CHANGELOG.md) 与 [Actions 指南](docs/github-actions.md)。
v0.5.1 —— GitHub Actions 可靠性修复版本(fail-on-new 语义、退出码顺序、SHA 支持、npm ci)。详见 [变更日志](CHANGELOG.md)。
[变更日志](CHANGELOG.md)、[测试基线](docs/test-baselines.md) 与[兼容性矩阵](docs/compatibility.md)。

## 许可证
Expand Down
12 changes: 9 additions & 3 deletions docs/github-actions.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ Pin the action to a released tag (`@v0.5.0`) or a commit SHA for supply-chain sa
| `config` | — | Path to a `reprocase.config.json` in your repo (overrides URL budgets). |
| `compare-with` | — | Path to a baseline `result.json` — enables the new-findings gate. |
| `fail-on` | `verified` | `verified` / `suspected` / `never` — which findings fail the step. |
| `fail-on-new` | `true` | With `compare-with`: fail when new findings exist vs the baseline. |
| `fail-on-new` | `true` | With `compare-with`: `true` fails on new findings vs baseline; `false` reports the diff without gating. Scan findings still gate via `fail-on`. |
| `max-pages` | scanner default (25) | Crawl budget. |
| `detection-window-ms` | scanner default (1500) | Async observation window (higher = catches later hydration faults). |
| `report-name` | `reprocase-report` | Artifact name. |
Expand All @@ -74,8 +74,8 @@ Pin the action to a released tag (`@v0.5.0`) or a commit SHA for supply-chain sa

| Code | Meaning | CI effect |
| --- | --- | --- |
| `0` | Scan completed, nothing at/above the threshold | step passes |
| `1` | Scan completed — findings at/above the threshold (a **verdict about your website**) | step fails, report artifact is still published |
| `0` | Clean scan + no new findings vs baseline | step passes |
| `1` | Findings at/above threshold (a **verdict about your site**) OR new findings vs baseline (a **quality gate**) | step fails, report artifact is still published |
| `2` | ReproCase could not run properly (bad config, unreachable URL, missing browser) | step fails with a `::error::` annotation stating this is an **environment problem, not a website verdict** |

An environment error never masquerades as "your site is fine" — the action fails loudly and
Expand Down Expand Up @@ -103,6 +103,12 @@ Behaviour:
cannot prove.
- **Update the baseline deliberately**: after accepting a scan, copy its
`result.json` into `.reprocase/baseline/` in a reviewed commit.
- **fail-on-new=false**: the compare report is generated and the new/resolved/persistent
counts are visible in the output and the report, but the action does not gate on new
findings. Scan findings still gate via `fail-on`.
- **fail-on never + compare-with**: the scan never gates on findings, but the baseline
gate can still fail on new findings. Useful for report-only runs that still want
regression detection.

## Pull requests

Expand Down
4 changes: 2 additions & 2 deletions package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "reprocase",
"version": "0.5.0",
"version": "0.5.1",
"description": "Automated website fault detection & reproduction. Scans a site in a real browser, verifies each fault independently, and emits reproducible Playwright regression tests plus human- and CI-friendly reports.",
"type": "module",
"license": "MIT",
Expand Down
Loading
Loading