Skip to content

Two parked rules return with the run's own evidence: tests run twice, an artifact that duplicates another upload - #35

Merged
askalf merged 18 commits into
mainfrom
rules/parked-two-with-evidence
Nov 8, 2026
Merged

askalf merged 18 commits into
mainfrom
rules/parked-two-with-evidence

Conversation

@askalf

@askalf askalf commented Nov 8, 2026

Copy link
Copy Markdown
Owner

Closes #34. Closes #32 (the last two of its four rules).

Why

The two rules were taken out of #33 after twelve review rounds: both recommend deleting a step, and a workflow file alone cannot establish that the second copy is the first (another directory, environment, runtime, configuration, or a destination that fails after the artifact went up). #34 kept every case the review raised and posed the design question: observations, or read more than the file.

This reads more than the file. The file picks the candidates, with every #34 case as a negative test, and the newest sampled successful run's own record confirms them. Without that record the rule is silent; the file alone never fires either rule.

tests run twice

  • File: the same one-command test step twice (the thirteenth-round reading, restored: command, condition, directory, step env and shell; within a job adjacent; across jobs the same runs-on, env, container, services, defaults, condition and every step before, now including each preparation step's working-directory and shell; no matrix; a pipe anywhere disqualifies the step).
  • Run: the two steps' sections of the job log (##[group]Run <first line> opens each step; across jobs one section from each job's log) list the same tests, by name where the runner prints them (TAP as node --test does, go test -v, pytest -v, a jest or vitest listing), else by the count of the summary line. A log that lists different tests, shows none, could not be read, opens a different number of sections for the header than the file has such steps (a step's own output can print one), or belongs to a job that does not map one-to-one to the file's job, confirms nothing.
  • Confidence measured. Estimate as before: the shorter step, from its own job's sampled runs; a shared display name prices at the pooled minimum.

an artifact that duplicates another upload

  • File: the thirteenth-round reading, restored, plus the artifact's name (name: literal, else artifact).
  • Run: the newest successful run's artifact of that name exists, unexpired, once, and its zip's central directory lists exactly one file, the path's last segment. A directory's worth of files, two artifacts of the name, an unreadable or over-cap zip: silent. The "names one file" extension heuristic stays as a pre-filter but no longer decides anything.
  • Fix unchanged from round thirteen: destination first, drop the artifact; artifact first, move it behind the destination under if: failure().

Collection

Only where a sampled workflow's file raises a candidate, and only for that workflow's newest sampled successful run: the matched jobs' logs through the job-logs redirect (the file from its host, no token, 8 MB cap), the run's artifact listing (one call) and the candidate artifact's zip through its redirect (same terms). collect takes artifacts: false; logs: false now covers sizes and logs. The client gains fileOf (text or bytes, capped); src/zip.mjs reads a central directory and inflates nothing. README's "what it reads" and "hostile input" sections say so; the log is no longer "never downloaded".

Receipts

  • 262 tests (244 before): every Parked rules with their spec: tests run twice, an artifact that duplicates another upload #34 case as a repeatedTests / duplicateUploads negative, the log readers against a real job log's shape (timestamps, colour codes, a forged header), the zip reader against hand-built zips and a truncated / zip64 one, collect reading only the candidate job's log and only the candidate artifact's zip, and both rules silent without the record.
  • Live, askalf/pitlane --only Scorecard: the finding fires with "the newest sampled run's artifact holds that one file (16 KB zipped)", 2 API calls more than before (29). The zip reader on the real artifact: [{"name":"results.sarif","size":71521}].
  • Live, askalf/pitlane-box, all workflows: no candidate, 4 findings as before, no error.

… an artifact that duplicates another upload

A workflow file cannot establish that a second copy is the first (ci-wizard#34: every case the
review of #33 raised). The file now only picks the candidates, with every one of those cases as a
negative test, and the newest sampled successful run confirms them or the rule is silent:

- tests run twice: the two steps' sections of the job log (across jobs, one section from each
  job's log) list the same tests, by name where the runner prints them (TAP, go test -v,
  pytest -v, a jest or vitest listing), else by count. Priced as before, the shorter run.
- an artifact that duplicates another upload: the run's artifact of the step's name exists,
  once, and its zip's directory lists exactly the one file the other step sent elsewhere.

collect reads the record only where a file raises a candidate: the matched jobs' logs through
the job-logs redirect, the run's artifact listing and the candidate artifact's zip through its
redirect, each from the file's host without the token, 8 MB caps; artifacts: false skips the
latter. The client gains fileOf (text or bytes, capped); zip.mjs reads a central directory and
nothing else. Docs, README (what it reads, hostile input), CHANGELOG, dist rebuilt. 262 tests.
Live: askalf/pitlane's Scorecard artifact is confirmed from its zip (one entry, results.sarif);
askalf/pitlane-box raises no candidate.
Comment thread test/evidence.test.mjs Fixed

@sprayberry-redline sprayberry-redline left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: request changes. Changes are needed before these recommendations are safe to publish. The candidate matching, log and artifact evidence, collection limits, tests, documentation, and bundled paths were reviewed; several cases still recommend removing distinct work, and the push example does not enforce its stated branch restriction.

1. Blocking: src/trim.mjs:184

  const jobKey = [stepKey, text(job['runs-on']), text(job.env), text(job.container), text(job.services), text(job.defaults), cond(job), before].join('\u0000');

The cross-job identity omits job.environment. Two otherwise identical jobs targeting staging and production can both set the same step env value from secrets.API_URL, run the same integration-test command, and print identical test names. Their environment-scoped secrets resolve to different endpoints, but this key groups them and the log comparison confirms a repeat. The report then recommends dropping one environment's test coverage.

Suggested fix:

Include the job's environment in candidate identity, and conservatively reject unresolved context-dependent differences. Add an end-to-end negative test with two environments and identical test listings.

2. Blocking: src/trim.mjs:255

    if (j > i + 1 && !survivesFailure(guard(other))) return;

Only a later destination is checked for intervening steps. A job can upload results.sarif to code scanning, run a second scanner that overwrites results.sarif, then upload that path as an artifact. The earlier destination qualifies, and the artifact's single entry named results.sarif passes artifactOf, despite containing a different report. The resulting advice drops the only retained copy of the second report.

Suggested fix:

Do not match an earlier destination across intervening steps that can change the file. Conservatively require adjacency in that direction too unless content equivalence is established, and add a negative test with an intervening overwrite and valid single-file artifact evidence.

3. Blocking: src/trim.mjs:339

    count = m.length > 2 && m[2] !== undefined ? Number(m[1]) + Number(m[2]) : Number(m[1]);

Every summary overwrites the previous count. Cargo prints a summary per test binary and a separate summary for doctests, not a final aggregate. Two cargo test sections with 10 versus 20 unit tests followed by the same one passing doctest both become count 1 and confirm a repeat. The existing test at test/evidence.test.mjs:44 explicitly accepts losing the earlier binary's count. Thus the evidence does not even establish equal total test counts for this supported runner.

Suggested fix:

Parse summaries by runner and aggregate per-binary counts where there is no overall total, or leave multi-summary evidence unknown. Cover differing earlier binaries followed by an identical final summary.

4. Blocking: src/analyse.mjs:548

    example: `- name: Tests, second run\n  if: github.event_name == 'push'   # once, on main; the pull request ran them above\n  run: ${repeats[0].command}`,

The example says the second run is restricted to main, but its condition accepts pushes to every branch. In a workflow triggered by feature-branch pushes and pull requests, copying this example retains the extra test run on feature-branch pushes rather than restricting it to the default branch as the surrounding advice describes.

Suggested fix:

Require both a push event and the repository's actual default-branch ref, with a placeholder when that branch is unknown, or remove the branch claim and describe the broader behavior accurately.

rule:none

…s identity, a destination before the artifact counts only next to it, summaries add up per runner, and the push example names the default branch

- repeatedTests: job.environment joins the cross-job key; staging against production, alike in
  every line, resolves one secret to two endpoints. Negative test with identical logs.
- duplicateUploads: a destination before the artifact step counts only from the step just before
  it, since a step between them can rewrite the file the artifact keeps. Negative tests, static
  and with valid single-file artifact evidence.
- testSignature: a runner's one total line counts once (two in a section are two suites: unknown),
  per-unit lines add up where the runner prints no total (cargo per binary and doctests, dotnet
  per project), pytest counts every outcome, maven's total is told from its per-class lines; a
  section mixing the kinds is unknown. Tests cover a differing earlier cargo binary.
- The second-run example gates on push and refs/heads/<default branch>, with the repository's
  default branch when known.
- A test regex CodeQL read as half-anchored is now a plain match.
Comment thread src/trim.mjs Fixed

@sprayberry-redline sprayberry-redline left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: request changes. The four earlier findings are addressed, including environment identity, upload adjacency, summary aggregation, and the default-branch gate. Review of candidate matching, evidence collection, tests, documentation, and bundled logic found two remaining cases where the recommendations remove necessary work.

1. Blocking: src/trim.mjs:187

  const jobKey = [stepKey, text(job['runs-on']), text(job.environment), text(job.env), text(job.container), text(job.services), text(job.defaults), cond(job), before].join('\u0000');

Comparing expression text does not establish equal environments across jobs. For example, jobs named staging and production can both declare environment: ${{ github.job }}, set API_URL from secrets.API_URL, and run identical integration-test steps. This key is identical for both jobs, although GitHub selects different environments and secrets. Identical test listings then confirm a repeat, and the report recommends removing one endpoint's coverage.

Suggested fix:

Conservatively exclude cross-job candidates with context-dependent expressions whose equivalence cannot be established, including job.environment expressions using github.job. Add an end-to-end negative test with these two environments and identical test listings.

2. Blocking: src/trim.mjs:273

  if (alsoTo.size) found.push({ job: id, step: stepName(st, i), path, name: artifactNameOf(st), alsoTo: [...alsoTo], after });

The candidate is accepted without checking whether another job consumes the artifact. A producer can upload results.sarif to code scanning and immediately retain it as an artifact named report, while a dependent job uses actions/download-artifact with name: report for further processing. The single-file zip confirms this candidate and the report tells the user to drop the artifact step. The dependent job then fails to download its input. Code scanning is not a replacement for the workflow's artifact transport; moving the artifact under failure() breaks successful downstream runs too.

Suggested fix:

Exclude artifacts consumed by download-artifact steps in the workflow, treating unresolved download names conservatively. Add a negative test with a dependent consumer and valid single-file evidence, and qualify the advice for consumers outside the workflow that cannot be inspected.

rule:none

… in its identity, no candidate for an artifact the workflow downloads

- repeatedTests: a cross-job key with any `${{ }}` in it (environment, env, runs-on, container,
  services, defaults, condition, a preparation step) reads alike in two jobs and can resolve
  apart (`environment: ${{ github.job }}`), so such a job is no cross-job candidate; one job's
  own two steps still compare. Negative tests, static and with identical logs.
- duplicateUploads: an artifact a download-artifact step of the workflow names is never a
  candidate; a download with no name, a pattern or an expression may take any artifact, so none
  is. The trade-off says a consumer outside the file (another workflow, a script) is not visible
  and must be checked. Tests with a consumer and valid single-file evidence.
- pytest's summary regex rewritten without the overlapping repeat CodeQL flagged.

@sprayberry-redline sprayberry-redline left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: request changes. The earlier findings are addressed by the environment and expression guards, upload adjacency and consumer checks, summary aggregation, and default-branch gate. Review of the candidate matching, evidence readers, collection, tests, documentation, and bundled paths found one remaining case where distinct test coverage is recommended for removal.

1. Blocking: src/trim.mjs:47

const HARMLESS = new Set(['set', 'echo', 'true', ':', 'pwd']);

set can change the test arguments, not just shell options. For example, two adjacent steps can run set -- --base-url=https://staging.example followed by pytest "$@", and set -- --base-url=https://production.example followed by pytest "$@". With pytest's base-url plugin these exercise different endpoints. onlyCommand accepts both set commands, and repeatedTests gives both steps the same normalized test command and identity. Their distinct first-line log headers still match their respective sections, and identical test names confirm a repeat. The report consequently recommends removing one endpoint's coverage.

Suggested fix:

Reject positional-parameter assignments and other state-changing uses of set from repeat candidates, or conservatively exclude set entirely unless its exact supported form is known to preserve test inputs. Add a negative end-to-end test with different set arguments and identical test listings, and rebuild the bundles.

rule:none

`set -- --base-url=…` assigns the positional parameters a following `pytest "$@"` reads, and a
bare `set` prints the environment: neither is an aside, so a step carrying one is no repeat
candidate. `set -e`, `set -euo pipefail`, `set -o pipefail` and `set +x` still are. Negative
tests, static and with identical listings against two endpoints; docs; bundles rebuilt.

@sprayberry-redline sprayberry-redline left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: request changes. The earlier findings are addressed, including positional-parameter assignments, environment identity, upload consumers and adjacency, summary aggregation, and the default-branch gate. Review of the candidate matching, evidence readers, collection, tests, documentation, and bundled logic found another case where distinct test coverage is recommended for removal.

1. Blocking: src/trim.mjs:111

if (!HARMLESS.has(ws[0]) || ws.some((w) => /^(<|>|>>|\|)/.test(w) || /[<>|]/.test(w) && !/^['"]/.test(w) && w !== ws[0])) return false;

An echo argument can execute a command substitution that changes the test configuration. For example, two adjacent steps can run echo "$(cp fixtures/sqlite.conf backend.conf)" followed by npm test, and echo "$(cp fixtures/postgres.conf backend.conf)" followed by npm test. shellCommands leaves each quoted substitution inside the echo argument, so this check accepts both as harmless. Both steps then have the same test identity, and identical test listings in their respective log sections confirm a repeat, even though they exercise different backends. The report recommends removing one backend's coverage.

Suggested fix:

Do not treat echo or other aside commands as harmless when their arguments contain executable shell expansions. Conservatively reject command substitutions and other unsupported state-changing shell syntax before comparing candidates. Add an end-to-end negative test with these two configuration-copying substitutions and identical test listings, and rebuild the bundles.

rule:none

…ript is not an aside

`echo "$(cp fixtures/postgres.conf backend.conf)"` runs a command inside an echo's argument and
picks the backend under test; a backtick or a process substitution does the same. A script with
any of them is not one command, so the step is no repeat candidate. Negative tests, static and
with identical listings against two backends; docs; bundles rebuilt.

@sprayberry-redline sprayberry-redline left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: request changes. The earlier findings are addressed, including the new rejection of command and process substitutions. Review of candidate matching, evidence collection, tests, documentation and bundled logic found one remaining input-handling case that can recommend removing distinct test coverage.

1. Blocking: src/trim.mjs:94

const commands = shellCommands(script);

Here-document input is lost before the repeat comparison. For example, two adjacent steps can each run npm test <<EOF, with sqlite in the first here-document body and postgres in the second. A test harness that reads its backend from stdin exercises different backends. shellCommands drops both the here-document operator and its body, returning npm test for both scripts; onlyCommand accepts them and repeatedTests gives them the same identity. Both log headers are npm test <<EOF, so two sections listing the same test names satisfy the evidence check and the report recommends removing one backend's coverage. The new substitution guard does not reject <<.

Suggested fix:

Reject here-document and other unsupported input redirections before calling shellCommands for repeat candidates, or preserve their input semantics in the comparison. Add an end-to-end negative test with different here-document inputs and identical test listings, and rebuild the bundles.

rule:none

… aside

`npm test <<EOF` with the backend in the here-document, a here-string, or `< file` feeds the
tests their input, and shellCommands keeps none of them apart: a script with one is not one
command, so the step is no repeat candidate. Negative tests, static and with identical listings
fed two inputs; docs; bundles rebuilt.

@sprayberry-redline sprayberry-redline left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: request changes. The earlier positional-parameter, substitution and input-redirection findings are addressed. Review of candidate matching, evidence readers, collection, tests and bundled logic found two remaining paths that can recommend removing work without establishing equivalence.

1. Blocking: src/trim.mjs:116

if (!HARMLESS.has(ws[0]) || ws.some((w) => /^(<|>|>>|\|)/.test(w) || /[<>|]/.test(w) && !/^['"]/.test(w) && w !== ws[0])) return false;

Parameter expansion can still change test inputs inside an accepted echo. Two adjacent scripts can run echo "${BACKEND:=sqlite}" followed by npm test -- --backend "$BACKEND", and echo "${BACKEND:=postgres}" followed by the same command. With BACKEND initially unset in each step's shell, these run against different backends. Neither expansion hits the script guard, both echoes pass this check, and both test commands have the same identity. Identical test listings in their respective first-line log sections then confirm a repeat and produce advice to remove one backend's coverage.

Suggested fix:

Reject assignment-bearing parameter expansions in aside commands, or conservatively reject unsupported shell expansions before candidate comparison. Add a negative end-to-end test with these scripts and identical test listings, and rebuild the bundles.

2. Blocking: src/collect.mjs:273

      const listed = listing && Array.isArray(listing.artifacts) ? listing.artifacts : null;

The artifact listing is treated as complete after only the first 100 records. On a run with more than 100 artifacts, a matching artifact on page one and another artifact of the same name on a later page become a single record in run_artifacts. artifactOf consequently accepts the first zip as unique and can recommend dropping the upload, despite the ambiguity that the duplicate-name check is intended to reject. This affects runs using artifact versions that allow separate jobs to upload under the same name.

Suggested fix:

Read all artifact pages before establishing uniqueness, or leave artifact evidence unknown when total_count exceeds the records read. Add a collection test with a duplicate name beyond the first page.

rule:none

… artifact listing is no record

- onlyCommand: `echo "${BACKEND:=postgres}"` assigns what the tests then read, so an aside with
  any `$` in it is not an aside and the step is no repeat candidate. Negative tests, static and
  with identical listings against two assigned backends.
- collect: a run's artifact listing with more artifacts than the page read may hold a second
  artifact of a candidate's name on a page not read, so the record is left unknown and no zip is
  read. Collection test with total_count past the page.
- Docs; bundles rebuilt.

@sprayberry-redline sprayberry-redline left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: request changes. The earlier expansion and artifact-pagination findings are addressed. Review of the candidate matching, evidence readers, collection, tests, documentation and bundled logic found another command-identity collision that can recommend removing distinct test coverage.

1. Blocking: src/trim.mjs:179

  const stepKey = [command, cond(st), dirOf(st, job), text(st.env), text(st.shell)].join('\u0000');

The command used here has lost its shell argument boundaries: shellCommands removes quotes and testCommands joins the resulting words with spaces. Two adjacent steps running node --test "test/a b.mjs" and node --test test/a b.mjs therefore receive the same identity, and onlyCommand accepts both. The first runs one file named test/a b.mjs; the second runs the two files test/a and b.mjs. If the first file contains tests named A and B and the other two files each contain one of those names, both logs list A and B and the evidence check confirms a repeat. The report then recommends removing coverage of different files.

Suggested fix:

Preserve argument boundaries and relevant quoting or expansion semantics in repeat identities, or conservatively exclude commands whose equivalence the shell parser cannot establish. Add an end-to-end negative test with these distinct file arguments and matching test listings, and rebuild the bundles.

rule:none

…, quoting kept

shellCommands splits a script as the shell does; the identity now keeps those boundaries instead
of joining the words with spaces, so `node --test "test/a b.mjs"` (one file) and `node --test
test/a b.mjs` (two files) are two commands, while the same file quoted either way is one.
Reporting flags are removed word by word. Negative test with the two argument shapes and matching
listings; docs; bundles rebuilt.

@sprayberry-redline sprayberry-redline left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: request changes. The earlier input-redirection, aside-expansion, pagination and argument-boundary cases are addressed. Review of candidate matching, evidence collection, tests, documentation and bundled logic found that shell quoting still loses semantics in repeat identities and in the suggested replacement command.

1. Blocking: src/trim.mjs:103

for (const words of shellCommands(script)) {
const ws = testWords(words);
if (ws) out.push(toKey(ws));

Keeping word boundaries does not preserve expansion semantics: shellCommands removes both single and double quotes. With job env FILE=other.mjs, adjacent steps running node --test '$FILE' and node --test "$FILE" receive the same key and pass onlyCommand. The first runs a file literally named $FILE; the second runs other.mjs. If those files print the same test names, their distinct log headers both match and the report recommends removing coverage of a different file. The latest fix resolves the space-separated argument case but not this quoting case.

Suggested fix:

Preserve quoting and expansion semantics in repeat keys, or conservatively exclude commands with expansions whose equivalence cannot be established. Also account for shell options such as set -f that affect argument expansion rather than treating all option changes as asides. Add an end-to-end negative test for the two quoted commands with identical listings and rebuild the bundles.

2. Blocking: src/trim.mjs:67

return String(key).split(KEY_SEP).join(' ');

The command reconstructed here is also inserted into the report's runnable YAML example. For two identical steps running node --test "test/a b.mjs", the candidate is valid, but its command becomes node --test test/a b.mjs. The suggested gated step therefore runs two different paths instead of the original single file, and fails when those paths do not exist.

Suggested fix:

Keep a shell-safe executable command separately from the display label, preferably retaining the original script for the example rather than reconstructing it from unquoted words. Add a report assertion that the example preserves a quoted filename containing spaces.

rule:none

…et` options limited to those that stop or trace, and the example repeats the step's own script

- onlyCommand: the words compare with their quotes gone, so `node --test '$FILE'` (a file of
  that name) and `node --test "$FILE"` (the file FILE names) would read alike: a script with any
  `$` is no repeat candidate. `set` is an aside only with -e, -u, -x (and +) or -o pipefail,
  errexit, nounset, xtrace: `set -f` changes what a glob hands the tests.
- The example gates the second run and repeats the matched step's own script as written, as a
  line or a block, never a line rebuilt from the key's words; the evidence label stays the
  command as a line.
- Negative tests for both quotings with identical listings and for set -f; example assertions
  with a quoted file name and a multi-line script; docs; bundles rebuilt.

@sprayberry-redline sprayberry-redline left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: request changes. The earlier expansion, pagination, argument-boundary and example-quoting cases are addressed. Candidate matching, evidence readers, collection, tests and report examples still have paths that can discard distinct coverage or produce invalid workflow YAML.

1. Blocking: src/trim.mjs:227

  const stepKey = [key, cond(st), dirOf(st, job), text(st.env), text(st.shell)].join('\u0000');

Equal expressions do not establish equal environments even within one job. For example, give the first of two adjacent npm test steps id first, and give both steps env: { SEED: "${{ steps.first.outputs.seed || 'initial' }}" }. If the first test run publishes a seed through GITHUB_OUTPUT, the first step uses initial and the second uses the published seed. Both receive this same key; matching test names in the two log sections confirm a repeat even though the tests exercise different inputs. The expression rejection below applies only to cross-job candidates.

Suggested fix:

Reject unresolved step environment and directory expressions for within-job comparisons too, or establish that their referenced values cannot change between the steps. Add an end-to-end negative test with a first-step output changing the second step's input and identical test listings.

2. Blocking: src/trim.mjs:139

if (/||$|`|<|>(/.test(String(script))) return false;

The expansion guard still permits quote-sensitive pathname and brace expansion. Under bash, node --test test/{a,b}.mjs runs test/a.mjs and test/b.mjs, while node --test 'test/{a,b}.mjs' runs the single file literally named test/{a,b}.mjs. shellCommands removes the quotes, both scripts pass onlyCommand, and both receive the same word key. If the literal file and the two expanded files list the same test names, the log evidence confirms a repeat and recommends removing distinct coverage.

Suggested fix:

Preserve quote and expansion semantics in command identities, or conservatively exclude pathname, tilde and brace expansion syntax until equivalence can be established. Add a negative end-to-end test for quoted versus unquoted brace expansion with matching listings.

3. Blocking: src/analyse.mjs:554

    example: `- name: Tests, second run\n  if: github.event_name == 'push' && github.ref == 'refs/heads/${defaultBranch || '<default branch>'}'   # once, after the merge; the pull request ran them above\n  run: ${repeats[0].script.includes('\n') ? `|\n    ${repeats[0].script.split('\n').join('\n    ')}` : repeats[0].script}`,

A single-line shell script is not necessarily a valid YAML plain scalar. Two matching steps originally expressed as block scalars containing node --test "test/a: b.mjs" produce run: node --test "test/a: b.mjs" here. The colon followed by a space is YAML mapping syntax even inside those shell quotes, so the suggested workflow cannot be parsed. Retaining the script fixes shell quoting but does not preserve its YAML encoding.

Suggested fix:

Emit every script as a correctly indented YAML block scalar, including single-line scripts, or serialize the scalar with a YAML serializer. Add a test that parses the example and verifies the recovered run value for a filename containing a colon and space.

rule:none

…a job either, quotes beside an expanding character, and the example as a block scalar

- repeatedTests: a step whose condition, directory, env or shell carries `${{ }}` is no
  candidate within its job either: `SEED: ${{ steps.first.outputs.seed || 'initial' }}` reads
  alike on two steps and resolves apart once the first has run.
- onlyCommand: a script with a quote beside a brace, glob or tilde is not established as one
  command: the quote decides whether `test/{a,b}.mjs` is two files or one, and the words read
  alike without it.
- The example writes the script as a YAML block scalar always, so a `: ` inside it stays text;
  a test parses the example back and reads the script and the gate.
- Negative tests for each with identical listings; docs; bundles rebuilt.
…ars, and the within-job candidate under an expression-free env

The previous commit's two test failures: the colon test wrote its own workflow file with a plain
scalar that YAML reads as a mapping (the case the example now avoids), so it is built from the
block-scalar variant; and the one-job candidate in the expression test now carries a literal
env, since a step env with `${{ }}` is no candidate within a job either.

@sprayberry-redline sprayberry-redline left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: request changes. The earlier expression and YAML-example findings are addressed, but command matching still conflates escaped brace literals with shell expansion. Review of the evidence readers, collection, report logic and tests also found an assertion that contradicts the new expression exclusion.

1. Blocking: src/trim.mjs:143

if (/['"]/.test(String(script)) && /[{}*?[]~]/.test(String(script))) return false;

The guard covers quote characters but not backslash quoting. In bash, adjacent steps running node --test test/{a,b}.mjs and node --test test/\{a,b\}.mjs run different files: the first expands to a.mjs and b.mjs, while the second names the literal {a,b}.mjs. shellCommands removes the backslashes, so both receive the same word key and pass onlyCommand. Matching test listings then confirm a repeat and recommend removing distinct file coverage.

Suggested fix:

Preserve backslash escape semantics in the identity, or conservatively exclude scripts combining backslashes with expansion characters. Add an end-to-end negative case with these commands and matching listings, and rebuild the bundles.

2. Blocking: test/evidence.test.mjs:580

assert.deepEqual(repeatedTests(doc(file)).map(brief), [['job', ['production'], ['Run npm test', 'Run npm test']]]);

Every test step in this fixture has env.API_URL: ${{ secrets.API_URL }}. The new stepKey expression check returns before adding any of those steps to inJob, including both production steps. repeatedTests therefore returns [], but this assertion still expects a within-job candidate. The test no longer agrees with the implemented exclusion.

Suggested fix:

Update this expectation to [] and, if the test should still demonstrate that job-level expressions allow within-job repeats, use literal step environment values in a separate fixture while retaining the expression on job.environment.

rule:none

@sprayberry-redline sprayberry-redline left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: request changes. The expression fixture and block-scalar example fixes address the earlier test and YAML issues. Candidate matching, evidence collection, log and zip readers, report logic and regression tests still leave the earlier backslash-escaping defect unresolved.

1. Blocking: src/trim.mjs:143

if (/['"]/.test(String(script)) && /[{}*?[]~]/.test(String(script))) return false;

The earlier backslash-escaping finding remains unresolved: this guard still checks only quote characters. In bash, adjacent steps running node --test test/{a,b}.mjs and node --test test/\{a,b\}.mjs run different files, but shellCommands removes the backslashes and gives both the same word key. Both pass onlyCommand. With matching test listings under their respective log headers, the report recommends removing distinct file coverage.

Suggested fix:

Preserve backslash escape semantics in repeat identities, or conservatively exclude scripts combining backslashes with expansion characters. Add an end-to-end negative test for these two commands with matching listings and rebuild the bundles.

rule:none

…is read like a quote

`node --test test/\{a,b\}.mjs` runs one file where the bare line runs two, and the backslashes
are gone from the words as the quotes are: a script with a brace, glob or tilde beside a quote or
a backslash is not established as one command. Negative test with matching listings; docs;
bundles rebuilt.

@sprayberry-redline sprayberry-redline left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: request changes. The earlier backslash-expansion, expression-fixture and block-scalar fixes are addressed. Review of candidate matching, collection, evidence readers, bundles and tests still found command identities that conflate different files, upload conditions that do not guarantee a second copy, and invalid artifact examples.

1. Blocking: src/trim.mjs:144

if (/['"\]/.test(String(script)) && /[{}*?[]~]/.test(String(script))) return false;

Backslashes inside double quotes still differ from those inside single quotes even without an expansion character. In bash, node --test "test/a\\b.mjs" passes a filename containing one backslash, whereas node --test 'test/a\\b.mjs' passes a filename containing two. shellCommands retains the contents of both quoted strings unchanged, so these scripts get the same key and pass this guard. If the two files list the same test names, the report recommends removing distinct file coverage.

Suggested fix:

Preserve shell escape semantics inside double quotes, or conservatively exclude scripts containing backslashes inside quoted arguments from repeat candidates. Add an end-to-end negative test with these two filenames and matching test listings, and rebuild the bundles.

2. Blocking: src/trim.mjs:359

    if (guard(other) !== mine && guard(other) !== '') return;

Equal condition text does not guarantee both uploads run. For example, put the artifact first with id: saved, and give both it and the adjacent SARIF upload if: steps.saved.outputs.artifact-id == ''. The artifact runs because its output is initially absent, then publishes artifact-id, causing the destination to skip. The run succeeds and its single-file artifact passes artifactOf, so the report recommends moving the only copy behind a destination that did not run. Conditions without ${{ }} are valid expressions too.

Suggested fix:

Reject conditions whose values can change between these steps, or establish their equivalence at their respective execution points. Confirm the destination actually executed successfully in the sampled run as well. Add a regression with the output-dependent condition and valid single-file artifact evidence.

3. Blocking: src/analyse.mjs:579

      ? `# after the step that sends ${dupes[0].path} to ${dupes[0].alsoTo[0]}\n- name: ${dupes[0].step}\n  if: failure()   # only when that upload failed\n  uses: actions/upload-artifact@...\n  with:\n    path: ${dupes[0].path}`

Artifact examples interpolate names and paths as YAML plain scalars. A valid source step with name: 'Upload: report' or path: 'report: final.sarif' can pass candidate matching and single-file evidence, but the example emits - name: Upload: report or path: report: final.sarif. The colon followed by a space makes the suggested YAML invalid. Both the move and remove example branches have this problem.

Suggested fix:

Serialize the step name and artifact path as YAML string scalars in both branches. Add tests that parse the examples and verify recovered values containing a colon and space.

rule:none

…o condition on either upload step, and YAML-quoted names and paths in the artifact examples

- onlyCommand: a backslash escapes differently inside double and single quotes ("test/a\b.mjs" is one backslash, 'test/a\b.mjs' two) and the words keep none of it, so a script with any backslash is no repeat candidate.
- duplicateUploads: the artifact step carries no condition, and a destination counts only with none (in a run that succeeded, a step without a condition ran and passed) or, later than next, with exactly always() or !cancelled(); the same text on both steps can hold on one and turn on the next (steps.saved.outputs.artifact-id == ''). Regression with that condition and valid single-file evidence.
- The artifact examples write the step name and path as double-quoted YAML scalars; tests parse both branches back with a ': ' in each.
- Docs; bundles rebuilt.

@sprayberry-redline sprayberry-redline left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: request changes. The earlier backslash, expression-fixture, upload-condition and YAML-example findings are addressed. Candidate matching, evidence collection, log and zip readers, report logic, bundles and regression tests still leave one unsafe artifact-removal case: a later destination can receive a rewritten file rather than the saved artifact's contents.

1. Blocking: src/trim.mjs:371

    if (j > i + 1 && !survivesFailure(guard(other))) return;

A later destination with always() or !cancelled() bypasses the adjacency requirement even though intervening steps can replace the file. For example, upload results.sarif as an artifact, run a second scanner that overwrites results.sarif, then upload results.sarif to code scanning under always(). All steps can succeed. The artifact's zip contains exactly results.sarif, so artifactOf confirms this candidate and the report recommends moving the artifact behind the destination under failure(). That loses the first scanner's report: the destination received only the second scanner's contents. The zip entry name establishes neither content equality nor that the file stayed unchanged.

Suggested fix:

Require the destination to be adjacent in both directions unless intervening steps are proven not to modify the file. Add a regression with artifact upload, a rewriting scanner, and an always() destination, including valid single-file artifact evidence; assert that no duplicate-upload finding is emitted and rebuild the bundles.

rule:none

…tifact step, in either direction

A later destination under always() past a rewriting step receives a report the artifact never held, as a destination before one past a rewriting step sent one the artifact does not hold: the zip's entry name says what the artifact is called, not that the file stayed the same. Regression with the rewriting scanner, an always() destination and valid single-file evidence; the adjacent always() destination still counts under the order fix. Docs; bundles rebuilt.

@sprayberry-redline sprayberry-redline left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: request changes. The earlier escaping, upload-condition, YAML-example and upload-adjacency findings are addressed. Review of candidate matching, evidence collection, log and zip readers, report logic, bundles and regression tests found one remaining command-identity defect that can recommend removing distinct test coverage.

1. Blocking: src/trim.mjs:55

if (REPORTING_FLAG.test(w)) continue;

Reporting flags are removed even after the runner's -- option terminator. For example, adjacent steps running pytest -- -v and pytest -- -q can test two different directories named -v and -q. Both commands become the key pytest after this line removes the path and the trailing separator is dropped. Both pass onlyCommand, and successful logs with the same summary count confirm the candidate, so the report recommends removing distinct directory coverage.

Suggested fix:

Respect option terminators when normalising runner arguments, keeping all following path operands unchanged. Account separately for package-manager argument forwarding rather than treating every `--` alike. Add an end-to-end negative test for these two pytest commands with matching summary counts, and rebuild the bundles.

rule:none

After a runner's -- every word is an operand and stays: pytest -- -v tests a directory named -v, and pytest -- -q another, so the two are two commands. A package manager's -- hands the rest to the test script, flags included, so npm test -- --silent is still npm test. Negative test with matching counts; docs; bundles rebuilt.

@sprayberry-redline sprayberry-redline left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: request changes. The earlier escaping, upload-condition, adjacency and YAML-example findings are addressed, and direct runner option terminators now stay intact. Review of command matching, evidence collection, report logic and regression tests found that package-manager forwarding still conflates distinct test operands.

1. Blocking: src/trim.mjs:62

  if (!forwards) literal = true;

For package-manager commands, every -- leaves flag stripping enabled, including a second separator intended for the underlying runner. With package.json containing "test": "pytest", adjacent npm test -- -- -v and npm test -- -- -q execute pytest against different directories named -v and -q. npm consumes the first separator and forwards the second to pytest. Here both path operands are stripped, both keys become npm test -- after the trailing separator is removed, and both pass onlyCommand. Matching successful summary counts then confirm a repeat and recommend removing distinct directory coverage.

Suggested fix:

Distinguish the package manager's forwarding separator from subsequent runner option terminators. Preserve operands after the forwarded terminator, or conservatively exclude commands whose forwarding semantics are unknown. Add an end-to-end negative test for these two npm commands with matching pytest counts and rebuild the bundles.

rule:none

…script's own terminator

npm consumes the first -- and forwards the rest to the test script, so flags after it are still reporting flags; a second -- reaches the runner as its terminator, and every word after it is an operand that stays: npm test -- -- -v and npm test -- -- -q test two directories. Negative test with matching counts; docs; bundles rebuilt.

@sprayberry-redline sprayberry-redline left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: request changes. The earlier upload-adjacency and npm option-terminator findings are addressed. Review of candidate matching, evidence collection, log and zip readers, report logic, bundles and regression tests found that treating every package manager like npm still conflates distinct test operands.

1. Blocking: src/trim.mjs:53

const forwards = /^(npm|pnpm|yarn|bun)$/.test(ws[0]);

pnpm does not consume the forwarding separator like npm: arguments after the script name are passed to the script, including --. With a test script of pytest, adjacent pnpm run test -- -v and pnpm run test -- -q test different directories named -v and -q. This branch keeps flag stripping enabled after their first separator, so both commands become pnpm run test and pass onlyCommand. Matching successful pytest summary counts then confirm a repeat and recommend removing distinct directory coverage. The new regression covers npm's second separator but not pnpm's first.

Suggested fix:

Handle forwarding semantics per package manager, or conservatively exclude ambiguous forwarded commands from repeat candidates. Preserve operands after pnpm's forwarded first separator, add an end-to-end negative test with these two commands and equal pytest counts, and rebuild the bundles.

rule:none

pnpm, yarn and bun pass a -- through to the test script, so for them the first one already ends the flags and pnpm run test -- -v hands pytest a directory named -v; npm alone consumes its first separator. Negative test with matching counts; docs; bundles rebuilt.

@sprayberry-redline sprayberry-redline left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: request changes. The earlier runner and package-manager separator findings are addressed, including pnpm's first separator. Candidate matching, evidence collection, log and zip readers, report logic, bundles and regression tests still leave a release-upload case where the recommendation can remove the only copy of a file.

1. Blocking: src/trim.mjs:394

    const hit = (/^github\/codeql-action\/upload-sarif@/.test(uses) && same(w.sarif_file)) ? 'code scanning' : (/^softprops\/action-gh-release@/.test(uses) && same(w.files)) ? 'the release' : null;

The release match ignores overwrite_files: false. If the release already has an asset named dist.tar.gz, action-gh-release with that input skips replacing it and can succeed, while the adjacent upload-artifact step stores the newly built dist.tar.gz. Its zip has exactly the expected entry, so the evidence confirms this candidate even though the release still holds the older bytes. With the destination first, the report recommends dropping the artifact, losing the only copy of the new build; with the artifact first, the failure-only replacement also loses it because the skipped replacement succeeds.

Suggested fix:

Exclude release destinations that disable overwriting, including unresolved expressions for that input, unless evidence establishes that this execution actually uploaded the file. Add a negative regression with `overwrite_files: false` and valid single-file artifact evidence, and rebuild the bundles.

rule:none

…is no destination

action-gh-release with overwrite_files off keeps the asset the release already has and still succeeds, so the file it was given never reached the release and the artifact is the only copy of the new build; only the default or a literal true counts, an expression does not. Negative regression with valid single-file evidence; docs; bundles rebuilt.

@sprayberry-redline sprayberry-redline left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: approve. The earlier separator and release-overwrite findings are addressed, with regressions covering matching log counts and valid single-file artifact evidence. Candidate matching, evidence collection, report logic, zip parsing and the rebuilt bundles consistently keep these recommendations silent when the required evidence is unavailable.

@askalf
askalf merged commit 2ba774c into main Nov 8, 2026
10 checks passed
@askalf
askalf deleted the rules/parked-two-with-evidence branch November 8, 2026 17:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

3 participants