Repository navigation
test: guard ci.yml gating jobs against continue-on-error - #1000
Merged
Merged
Conversation
.github/workflows/ci.yml is the only pull_request-triggered workflow, so its validate, lint and e2e jobs are the entire pre-merge gate. Adding continue-on-error: true to any of them leaves every step running and reporting while the job reports success, and no test notices. The same line on the 'Run unit tests with coverage' step discards the unit suite and all four coverage thresholds on its own. The suite guarded this in one direction only: workflow-scripts.test.mjs asserts that e2e-coverage *is* non-gating. Nothing asserted the converse. coverage-gate-thresholds.test.mjs pins the threshold values and that ci.yml runs the gate command; both survive the exit code being ignored. Add tests/ci-gating-jobs.test.mjs, which scans every workflow and pins that the set of jobs declaring continue-on-error is exactly one allowlisted entry (ci.yml#e2e-coverage, with its reason), that no step of a gating job declares it, that ci.yml still defines validate, lint and e2e, and that an allowlist entry cannot outlive the job it exempts. The scanners take the parsed workflows as an argument so each violation path is exercised against a fixture rather than only against a green tree. No workflow file is modified. Signed-off-by: quality <quality@hive.kubestellar.io> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
Author
|
Important Held for human review by the hive's ACMM level gate. This PR was opened by the "quality" agent while Hive policy required a human checkpoint for that agent. Non-outreach agents are held at ACMM L3–L5; the Hive will automatically remove the |
This was referenced Oct 3, 2026
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Test Improvement
Adds
tests/ci-gating-jobs.test.mjs. No workflow file is modified — thechange is one new file under
tests/..github/workflows/ci.ymlis the onlypull_request-triggered workflow, so itsvalidate,lintande2ejobs are the entire pre-merge gate. Addingcontinue-on-error: trueto any of them leaves every step running and every stepreporting, and makes the job report success. The same line on the
Run unit tests with coveragestep discards the unit suite and all four coveragethresholds on its own. Nothing in the suite noticed.
The existing guards are blind to it by construction:
tests/workflow-scripts.test.mjspins the relationship in one direction only —that
e2e-coverageis non-gating (assert.equal(coverage['continue-on-error'], true)).Nothing asserted the converse.
tests/coverage-gate-thresholds.test.mjspins the threshold values and thatci.yml runs the gate command. Both survive the exit code being ignored.
What the new file pins
only allowlisted jobs opt out of failingcontinue-on-errorci.yml's gating jobs are gatingvalidate,lintore2eis disarmed (names the job in the message)no step inside a gating job suppresses its own failureevery workflow parses and ci.yml still defines its gating jobspull_requestthe non-gating allowlist does not outlive its jobsThe allowlist holds exactly one entry —
ci.yml#e2e-coverage— with its reasonwritten next to it. This follows the
FLOORSconvention incoverage-gate-thresholds.test.mjs: a legitimate future non-gating job stayspossible, but it must be argued for in the allowlist rather than slipped in as one
line of YAML.
continue-on-erroraccepts an expression, so only a literalfalsecounts as pinned off;
${{ ... }}is treated as opting out.Verification
The scanners take the parsed workflows as an argument rather than closing over the
repository's own, so every violation path is exercised against a fixture — a guard
whose failure path never runs is a guard nobody has checked.
Each failure mode was also confirmed end-to-end by mutating a scratch copy of
ci.ymland restoring it (the committedci.ymlis byte-identical tomain):continue-on-error: trueon thevalidatejobonly allowlisted jobs opt out of failing, ✖ci.yml's gating jobs are gatingcontinue-on-error: trueonRun unit tests with coverageno step inside a gating job suppresses its own failure—ci.yml#validate → Run unit tests with coveragee2e-coveragestops opting outonly allowlisted jobs opt out of failing, ✖the non-gating allowlist does not outlive its jobslintjob deletedevery workflow parses and ci.yml still defines its gating jobsci.ymlFull gate,
npm run test:unit:coverage:check(TZ=UTC), run locally on this branch:Exit 0, zero failing tests; overall coverage is unchanged against the
99.37 / 95.02baseline on
main.npx prettier --checkandnpx cspellboth pass on the new file.Scope
Disjoint from every open hold-gated PR: this touches no file in #989, #991, #994,
#996 or #998. In particular it does not touch
tests/tools/coverage-report.mjs,tests/tools/e2e-coverage-report.mjsorpackage.json, so it does not overlap thereporter work in #991/#992/#996.
Related Issue
Closes #999
Filed by quality agent (hold-gated mode). Human review required.
— hive: agent=quality backend=copilot model=claude-opus-5 copilot=1.0.88