From 7954c687d6a21fd3a372b71e05bebe42d2f5b1f7 Mon Sep 17 00:00:00 2001 From: askalf <263217947+askalf@users.noreply.github.com> Date: Wed, 11 Nov 2026 21:55:34 -0500 Subject: [PATCH 1/5] A new action, tree-tested: skip a push's tests when its pull request already passed that tree On a push to the default branch it says whether the pull request merged as that commit passed the same workflow, with every job green on its latest attempt, at a head with the same git tree. A caller guards its build and test steps on it. Anything short of that, and any failed read, runs the job in full. --- .github/workflows/redline-pin-bump.yml | 6 +- .github/workflows/redline-self-test.yml | 1 + CHANGELOG.md | 7 ++ README.md | 1 + actions/tree-tested/action.yml | 38 ++++++ actions/tree-tested/main.mjs | 15 +++ scripts/redline/pin.mjs | 3 +- scripts/redline/review.test.mjs | 2 +- scripts/tree-tested/check.mjs | 147 ++++++++++++++++++++++++ scripts/tree-tested/check.test.mjs | 126 ++++++++++++++++++++ test-all.mjs | 1 + 11 files changed, 344 insertions(+), 3 deletions(-) create mode 100644 actions/tree-tested/action.yml create mode 100644 actions/tree-tested/main.mjs create mode 100644 scripts/tree-tested/check.mjs create mode 100644 scripts/tree-tested/check.test.mjs diff --git a/.github/workflows/redline-pin-bump.yml b/.github/workflows/redline-pin-bump.yml index 7826344..bcecfc0 100644 --- a/.github/workflows/redline-pin-bump.yml +++ b/.github/workflows/redline-pin-bump.yml @@ -6,7 +6,8 @@ # whichever workflow file it lives: uses: askalf/ci/actions/drift-report@ (a watcher's report # step), uses: askalf/ci/actions/checkout-with-retry@ (any job's checkout), # uses: askalf/ci/actions/fleet-status@ and fleet-status-check@ (the fleet lanes), and -# uses: askalf/ci/actions/scorecard-publish-check@ (the check after a Scorecard run). +# uses: askalf/ci/actions/scorecard-publish-check@ (the check after a Scorecard run), and +# uses: askalf/ci/actions/tree-tested@ (the skip for a tree a pull request already tested). # Each pins by SHA so a change reaches it only through a reviewed bump there; this writes those bumps # (scripts/redline/pin.mjs), and each repo's own gate reviews and merges them. # @@ -123,6 +124,9 @@ jobs: :(exclude)scripts/scorecard-publish/*.test.mjs :(exclude)scripts/scorecard-publish/README.md actions/scorecard-publish-check + scripts/tree-tested + :(exclude)scripts/tree-tested/*.test.mjs + actions/tree-tested .github/workflows/redline-review.yml .github/workflows/redline-fix-run.yml .github/workflows/public-text.yml diff --git a/.github/workflows/redline-self-test.yml b/.github/workflows/redline-self-test.yml index 36dc883..0ebf023 100644 --- a/.github/workflows/redline-self-test.yml +++ b/.github/workflows/redline-self-test.yml @@ -41,6 +41,7 @@ jobs: - run: node scripts/fleet-status/check.test.mjs # The check after a Scorecard run, and its action's entry, against a stubbed network. - run: node scripts/scorecard-publish/check.test.mjs + - run: node scripts/tree-tested/check.test.mjs # The pin bump workflow's bump step, run as the workflow runs it, against a fake gh. - run: node scripts/redline/pin-bump.test.mjs # Every fuzz seed through its target, so a target that rots fails here and not only in the weekly fuzz run. diff --git a/CHANGELOG.md b/CHANGELOG.md index 40789ed..91a4b38 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -13,6 +13,13 @@ names the commit's date and the pull request it came from. ## November 2026 +- A new action, `actions/tree-tested`: on a push to the default branch it says whether the + pull request merged as that commit already passed the same workflow, on its latest attempt and + with every job green, at a head whose tree is the pushed tree byte for byte. A caller skips its + build and test steps when it says so. A direct push, a moved main, a skipped or failed job, or a + read that fails runs the job in full, as before. Replayed against the last twelve pushes of five + repositories on 2026-11-12, it would have skipped 33 of 60 runs: claude-re 11, platform 8, + restore-desk 5, unscathed-site 5, dario 4. The pin bump moves it like the other actions. - A second lens, off unless a caller turns it on. A caller on a self-hosted runner that passes `second-reviewer-login` gets every head the review approves read once more in the same job, as that account, with a second env file and rubric the host keeps for it, and diff --git a/README.md b/README.md index 609f2d1..284c7d8 100644 --- a/README.md +++ b/README.md @@ -217,6 +217,7 @@ full commit sha of `main`, so a change here reaches it only through a reviewed b | [`actions/fleet-status`](actions/fleet-status/action.yml) | Posts the `fleet/verify` and `fleet/review` commit statuses a merge waits on; the lane rules are in [scripts/fleet-status/README.md](scripts/fleet-status/README.md). | | [`actions/scorecard-publish-check`](actions/scorecard-publish-check/action.yml) | After a Scorecard run: checks that its result reached scorecard.dev, and dispatches Scorecard once more when it did not ([the Scorecard publish check](#the-scorecard-publish-check)). | | [`actions/fleet-status-check`](actions/fleet-status-check/action.yml) | Checks a repository's fleet-status workflows, its self-hosted `runs-on` labels and its review relay, as that repository's self-test job. | +| [`actions/tree-tested`](actions/tree-tested/action.yml) | On a push to the default branch: says whether the pull request that merged it already passed this workflow, with every job, on exactly this tree, so the job can skip its build and test steps instead of running them twice. Any doubt, or any read that fails, runs the job in full. | | This repository's own | What it does | |---|---| diff --git a/actions/tree-tested/action.yml b/actions/tree-tested/action.yml new file mode 100644 index 0000000..bf3e0c6 --- /dev/null +++ b/actions/tree-tested/action.yml @@ -0,0 +1,38 @@ +# Says whether a push to the default branch carries a tree that this workflow already tested green +# on the pull request that merged it, so the job can skip its build and test steps instead of +# running them a second time on the same files. The rules are scripts/tree-tested/check.mjs (its +# header says each one), tested without the network by check.test.mjs. +# +# Pinned like the other actions here: a caller names one full commit sha of this repository's main, +# - id: tree +# uses: askalf/ci/actions/tree-tested@ +# as an early step of a job, and puts `if: steps.tree.outputs.tested != 'true'` on the steps that +# install, build and test. On anything but a push to the default branch it answers false without a +# request. Every read that fails, and every rule that does not hold, answers false too: the job then +# runs in full, as it did before. The step never fails the job. +# +# It checks out nothing and runs on the Node the runner ships. The job's token needs actions: read, +# contents: read and pull-requests: read, which a workflow's default token has unless its +# permissions block leaves one out. +name: Tree tested +description: Say whether this push's tree already passed this workflow on the pull request that merged it. + +inputs: + token: + description: A token that reads the repository's commits, pull requests and Actions runs. + required: false + default: ${{ github.token }} + default-branch: + description: The repository's default branch. A caller passes none. + required: false + default: ${{ github.event.repository.default_branch }} + +outputs: + tested: + description: "true when this tree already passed this workflow, with every job, at the merged pull request's head; else false." + run: + description: The run that tested it, when tested is true. + +runs: + using: node24 + main: main.mjs diff --git a/actions/tree-tested/main.mjs b/actions/tree-tested/main.mjs new file mode 100644 index 0000000..4effcac --- /dev/null +++ b/actions/tree-tested/main.mjs @@ -0,0 +1,15 @@ +// The action's entry. The runner runs this file with the Node it ships (runs.using in action.yml) +// and hands each input over as INPUT_. The event's facts (repository, event, ref, sha, the +// workflow's file) are the runner's own GITHUB_* variables; the script reads those. +import { appendFileSync } from 'node:fs'; +import { main } from '../../scripts/tree-tested/check.mjs'; + +const input = (name) => (process.env[`INPUT_${name.toUpperCase()}`] ?? '').trim(); + +try { + process.exitCode = await main({ ...process.env, GITHUB_TOKEN: input('token'), DEFAULT_BRANCH: input('default-branch') }); +} catch (e) { + // A failure of the check runs the job in full; it never fails the job. + console.log(`tested in full: ${e.message}`); + if (process.env.GITHUB_OUTPUT) appendFileSync(process.env.GITHUB_OUTPUT, 'tested=false\nrun=\n'); +} diff --git a/scripts/redline/pin.mjs b/scripts/redline/pin.mjs index b664d68..f673816 100644 --- a/scripts/redline/pin.mjs +++ b/scripts/redline/pin.mjs @@ -55,8 +55,9 @@ export const CHECKOUT_RETRY_ACTION = `${CI_REPO}/actions/checkout-with-retry`; export const FLEET_STATUS_ACTION = `${CI_REPO}/actions/fleet-status`; export const FLEET_STATUS_CHECK_ACTION = `${CI_REPO}/actions/fleet-status-check`; export const SCORECARD_PUBLISH_CHECK_ACTION = `${CI_REPO}/actions/scorecard-publish-check`; +export const TREE_TESTED_ACTION = `${CI_REPO}/actions/tree-tested`; /** Every action of this repository a workflow pins as a step; `--action` moves them all. */ -export const ACTIONS = [DRIFT_REPORT_ACTION, CHECKOUT_RETRY_ACTION, FLEET_STATUS_ACTION, FLEET_STATUS_CHECK_ACTION, SCORECARD_PUBLISH_CHECK_ACTION]; +export const ACTIONS = [DRIFT_REPORT_ACTION, CHECKOUT_RETRY_ACTION, FLEET_STATUS_ACTION, FLEET_STATUS_CHECK_ACTION, SCORECARD_PUBLISH_CHECK_ACTION, TREE_TESTED_ACTION]; /** Every reusable workflow a caller pins, and the caller file that pins it. */ export const CALLERS = [ { workflow: REVIEW_WORKFLOW, path: '.github/workflows/redline.yml' }, diff --git a/scripts/redline/review.test.mjs b/scripts/redline/review.test.mjs index a364c45..cc1c40d 100644 --- a/scripts/redline/review.test.mjs +++ b/scripts/redline/review.test.mjs @@ -311,7 +311,7 @@ console.log('\n a fork: ci-repo and reviewer-login'); const fork = probe({ REDLINE_CI_REPO: 'someone/ci', REDLINE_REVIEWER_LOGIN: 'Fork-Bot' }); const got = fork.status === 0 ? JSON.parse(fork.stdout) : null; check('in a fork every pinned workflow and action is the fork\'s, and the poster trusts the fork alone', - got !== null && got.paths.length === 9 && got.paths.every((p) => p.startsWith('someone/ci/')) && JSON.stringify(got.tooling) === '["someone/ci"]' && got.login === 'fork-bot'); + got !== null && got.paths.length === 10 && got.paths.every((p) => p.startsWith('someone/ci/')) && JSON.stringify(got.tooling) === '["someone/ci"]' && got.login === 'fork-bot'); const ours = probe({}); const today = ours.status === 0 ? JSON.parse(ours.stdout) : null; check('unset, it is today\'s: askalf/ci, sprayberry-redline, and askalf/askalf still accepted', today !== null && today.paths.every((p) => p.startsWith('askalf/ci/')) diff --git a/scripts/tree-tested/check.mjs b/scripts/tree-tested/check.mjs new file mode 100644 index 0000000..621d2cd --- /dev/null +++ b/scripts/tree-tested/check.mjs @@ -0,0 +1,147 @@ +#!/usr/bin/env node +// Says whether the tree a push to the default branch carries was already tested green by this same +// workflow, on the pull request that merged it. The tree-tested action runs it as the first step of +// a job; the steps that build and test then take `if: steps..outputs.tested != 'true'`. +// +// Why. A pull request runs its workflow, merges, and the push to main runs the same workflow again +// on, most often, exactly the same files: a squash or merge of a branch that already had main in it +// leaves main with the branch head's tree. On the askalf runner host on 2026-11-11 that second run +// was about 300 of a day's CPU-heavy runner minutes (restore-desk, claude-re, platform, dario), on a +// host whose runner slice was waiting on CPU a quarter of the time. +// +// The rule, all of it read from the GitHub API with the job's own token: +// 1. The event is a push to the default branch. Anything else is tested=false. +// 2. A pull request of this repository was merged as this commit (its merge_commit_sha is the +// pushed sha). A direct push has none, and is tested=false. +// 3. The pull request's head commit has the same tree as the pushed commit: the same files, byte +// for byte, git's own hash of them. A pull request run tests its head merged into the base it +// had then; when the merged result on main is the head's own tree, the head already held +// everything main had, so that merge was this tree too. +// 4. The newest completed pull_request run of this workflow file at that head concluded success, +// on its latest attempt, and every job in it concluded success. A job that was skipped did not +// test anything, so a run with one is not taken: a workflow that picks its jobs by what changed +// runs them all on main, as before. +// Every read that fails, and every rule that does not hold, is tested=false: the job runs in full. +// Nothing here can turn a red run green; at worst it runs a test it could have skipped. +// +// CLI: GITHUB_REPOSITORY=owner/name GITHUB_EVENT_NAME=push GITHUB_REF=refs/heads/main GITHUB_SHA= \ +// GITHUB_WORKFLOW_REF=owner/name/.github/workflows/ci.yml@refs/heads/main DEFAULT_BRANCH=main \ +// GITHUB_TOKEN=... [GITHUB_OUTPUT=file] node check.mjs + +import { appendFileSync } from 'node:fs'; +import { pathToFileURL } from 'node:url'; + +const REPO_RE = /^[A-Za-z0-9-]+\/[A-Za-z0-9._-]+$/; +const SHA_RE = /^[0-9a-f]{40}$/; +/** owner/name/.github/workflows/@: the file this run's workflow came from. */ +const WORKFLOW_REF_RE = /^[A-Za-z0-9-]+\/[A-Za-z0-9._-]+\/\.github\/workflows\/([A-Za-z0-9._-]+\.ya?ml)@/; + +/** The facts the decision starts from, from the runner's environment. Pure. */ +export function factsOf(env) { + const text = (k) => String(env?.[k] ?? '').trim(); + return { + repo: text('GITHUB_REPOSITORY'), + event: text('GITHUB_EVENT_NAME'), + ref: text('GITHUB_REF'), + sha: text('GITHUB_SHA'), + workflowFile: WORKFLOW_REF_RE.exec(text('GITHUB_WORKFLOW_REF'))?.[1] ?? '', + defaultBranch: text('DEFAULT_BRANCH'), + }; +} + +/** Why this run is not one to look up at all, or '' when it is. Pure. */ +export function notEligible(f) { + if (!REPO_RE.test(f.repo)) return `GITHUB_REPOSITORY is not owner/name (${JSON.stringify(f.repo)})`; + if (f.event !== 'push') return `a ${f.event || 'run with no event'} is tested in full`; + if (!f.defaultBranch || f.ref !== `refs/heads/${f.defaultBranch}`) return `a push to ${f.ref || 'no ref'} is tested in full; only a push to the default branch repeats a pull request's run`; + if (!SHA_RE.test(f.sha)) return 'the pushed commit is not a full sha'; + if (!f.workflowFile) return 'the workflow file could not be read from GITHUB_WORKFLOW_REF'; + return ''; +} + +/** The pull request merged as this commit, from the commit's pull request list, or null. Pure. */ +export function mergedAs(pulls, repo, sha) { + return (Array.isArray(pulls) ? pulls : []).find((p) => p?.merged_at && p?.merge_commit_sha === sha + && p?.base?.repo?.full_name === repo && SHA_RE.test(String(p?.head?.sha ?? ''))) ?? null; +} + +/** + * The run that tested the head, from the workflow's runs at that head sha: the newest completed + * pull_request run for this pull request. Null when there is none. Pure. + */ +export function runAtHead(runs, { head, number }) { + return (Array.isArray(runs) ? runs : []) + .filter((r) => r?.event === 'pull_request' && r?.status === 'completed' && r?.head_sha === head + && (!Array.isArray(r?.pull_requests) || r.pull_requests.length === 0 || r.pull_requests.some((p) => p?.number === number))) + .sort((a, b) => Date.parse(b.created_at ?? 0) - Date.parse(a.created_at ?? 0))[0] ?? null; +} + +/** Why the run's jobs do not show this tree tested, or '' when every job passed. Pure. */ +export function jobsProblem(jobs) { + const list = Array.isArray(jobs) ? jobs : []; + if (list.length === 0) return 'the run lists no jobs'; + const off = list.filter((j) => j?.conclusion !== 'success'); + if (off.length) return `not every job passed there (${off.slice(0, 4).map((j) => `${j?.name ?? '?'}: ${j?.conclusion ?? j?.status ?? '?'}`).join(', ')})`; + return ''; +} + +/** GET from the API as the token. Throws on a status that is not 2xx. */ +async function get(io, env, path) { + const api = (env.GITHUB_API_URL || 'https://api.github.com').replace(/\/+$/, ''); + const res = await io.fetch(`${api}${path}`, { + headers: { authorization: `Bearer ${env.GITHUB_TOKEN ?? ''}`, accept: 'application/vnd.github+json', 'x-github-api-version': '2022-11-28' }, + signal: AbortSignal.timeout(20_000), + }); + if (!res.ok) throw new Error(`GET ${path.split('?')[0]}: HTTP ${res.status}`); + return res.json(); +} + +/** + * The decision: { tested, why, run? }. tested is true only when every rule in the header holds. + * io: { fetch }. + */ +export async function decide(env, io = { fetch: globalThis.fetch }) { + const f = factsOf(env); + const skip = notEligible(f); + if (skip) return { tested: false, why: skip }; + try { + const pulls = await get(io, env, `/repos/${f.repo}/commits/${f.sha}/pulls`); + const pr = mergedAs(pulls, f.repo, f.sha); + if (!pr) return { tested: false, why: `no pull request of ${f.repo} was merged as ${f.sha.slice(0, 7)} (a direct push is tested in full)` }; + const head = pr.head.sha; + const [pushed, headed] = await Promise.all([ + get(io, env, `/repos/${f.repo}/git/commits/${f.sha}`), + get(io, env, `/repos/${f.repo}/git/commits/${head}`), + ]); + const tree = String(pushed?.tree?.sha ?? ''), headTree = String(headed?.tree?.sha ?? ''); + if (!SHA_RE.test(tree) || tree !== headTree) { + return { tested: false, why: `#${pr.number}'s head ${head.slice(0, 7)} has another tree than ${f.sha.slice(0, 7)}: main had moved, so the merged files were never tested together` }; + } + const listed = await get(io, env, `/repos/${f.repo}/actions/workflows/${f.workflowFile}/runs?head_sha=${head}&event=pull_request&per_page=50`); + const run = runAtHead(listed?.workflow_runs, { head, number: pr.number }); + if (!run) return { tested: false, why: `${f.workflowFile} has no finished pull_request run at #${pr.number}'s head ${head.slice(0, 7)}` }; + if (run.conclusion !== 'success') return { tested: false, why: `${f.workflowFile} ended ${run.conclusion} at #${pr.number}'s head (${run.html_url ?? run.id})` }; + const jobs = await get(io, env, `/repos/${f.repo}/actions/runs/${run.id}/jobs?filter=latest&per_page=100`); + const bad = jobsProblem(jobs?.jobs); + if (bad) return { tested: false, why: `${bad} in ${run.html_url ?? run.id}` }; + return { tested: true, why: `#${pr.number}'s head ${head.slice(0, 7)} has this tree, and ${f.workflowFile} passed there with every job`, run: run.html_url ?? String(run.id) }; + } catch (e) { + return { tested: false, why: `${e.message}; tested in full` }; + } +} + +export async function main(env, io = {}) { + const log = io.log ?? ((l) => console.log(l)); + const d = await decide(env, { fetch: io.fetch ?? globalThis.fetch }); + if (env.GITHUB_OUTPUT) appendFileSync(env.GITHUB_OUTPUT, `tested=${d.tested ? 'true' : 'false'}\nrun=${d.run ?? ''}\n`); + log(d.tested ? `::notice::this tree already passed: ${d.why} (${d.run}). The steps that test it are skipped.` : `tested in full: ${d.why}`); + return 0; +} + +if (import.meta.url === pathToFileURL(process.argv[1] ?? '').href) { + main(process.env).then((code) => { process.exitCode = code; }, (e) => { + // Even a failure of this script runs the job in full: it never fails the job. + console.log(`tested in full: ${e.message}`); + if (process.env.GITHUB_OUTPUT) appendFileSync(process.env.GITHUB_OUTPUT, 'tested=false\nrun=\n'); + }); +} diff --git a/scripts/tree-tested/check.test.mjs b/scripts/tree-tested/check.test.mjs new file mode 100644 index 0000000..621fbdf --- /dev/null +++ b/scripts/tree-tested/check.test.mjs @@ -0,0 +1,126 @@ +// Tests for scripts/tree-tested/check.mjs. Run: node scripts/tree-tested/check.test.mjs +// GitHub is stubbed through io.fetch; nothing leaves the machine. + +import { mkdtempSync, readFileSync, rmSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { factsOf, notEligible, mergedAs, runAtHead, jobsProblem, decide, main } from './check.mjs'; + +let pass = 0; +let fail = 0; +function check(name, cond) { + if (cond) { console.log(` ok ${name}`); pass++; } + else { console.log(` FAIL ${name}`); fail++; } +} + +const REPO = 'askalf/demo'; +const SHA = 'a'.repeat(40); +const HEAD = 'b'.repeat(40); +const TREE = 'c'.repeat(40); +const OTHER = 'd'.repeat(40); +const env = (o = {}) => ({ + GITHUB_REPOSITORY: REPO, GITHUB_EVENT_NAME: 'push', GITHUB_REF: 'refs/heads/main', GITHUB_SHA: SHA, + GITHUB_WORKFLOW_REF: `${REPO}/.github/workflows/ci.yml@refs/heads/main`, DEFAULT_BRANCH: 'main', GITHUB_TOKEN: 't', ...o, +}); +const pr = (o = {}) => ({ number: 7, merged_at: '2026-11-12T00:00:00Z', merge_commit_sha: SHA, base: { repo: { full_name: REPO } }, head: { sha: HEAD }, ...o }); +const run = (o = {}) => ({ id: 900, event: 'pull_request', status: 'completed', conclusion: 'success', head_sha: HEAD, created_at: '2026-11-11T23:00:00Z', html_url: 'https://x/run/900', pull_requests: [{ number: 7 }], ...o }); +const job = (name, conclusion = 'success') => ({ name, conclusion, status: 'completed' }); + +/** A fake GitHub. routes: path (no query) -> body, an HTTP status, or a function of the url. */ +function world(over = {}) { + const routes = { + [`/repos/${REPO}/commits/${SHA}/pulls`]: [pr()], + [`/repos/${REPO}/git/commits/${SHA}`]: { tree: { sha: TREE } }, + [`/repos/${REPO}/git/commits/${HEAD}`]: { tree: { sha: TREE } }, + [`/repos/${REPO}/actions/workflows/ci.yml/runs`]: { workflow_runs: [run()] }, + [`/repos/${REPO}/actions/runs/900/jobs`]: { jobs: [job('unit'), job('e2e')] }, + ...over, + }; + const calls = []; + const fetch = async (url) => { + const u = new URL(url); + calls.push(u.pathname + u.search); + let body = routes[u.pathname]; + if (typeof body === 'function') body = body(u); + if (body === undefined) return { ok: false, status: 404, json: async () => ({}) }; + if (typeof body === 'number') return { ok: body < 300, status: body, json: async () => ({}) }; + return { ok: true, status: 200, json: async () => body }; + }; + return { fetch, calls }; +} + +console.log('\n the facts and who is looked up'); +{ + const f = factsOf(env()); + check('the workflow file is read from GITHUB_WORKFLOW_REF', f.workflowFile === 'ci.yml' && f.sha === SHA); + check('a push to the default branch is eligible', notEligible(f) === ''); + check('a pull request run is tested in full', /tested in full/.test(notEligible(factsOf(env({ GITHUB_EVENT_NAME: 'pull_request' }))))); + check('a push to another branch is tested in full', /only a push to the default branch/.test(notEligible(factsOf(env({ GITHUB_REF: 'refs/heads/feature' }))))); + check('a schedule, a dispatch and a missing event are tested in full', + ['schedule', 'workflow_dispatch', ''].every((e) => notEligible(factsOf(env({ GITHUB_EVENT_NAME: e }))) !== '')); + check('no default branch, a short sha, a bad repository or no workflow ref is tested in full', + [{ DEFAULT_BRANCH: '' }, { GITHUB_SHA: 'abc' }, { GITHUB_REPOSITORY: 'nope' }, { GITHUB_WORKFLOW_REF: 'x' }].every((o) => notEligible(factsOf(env(o))) !== '')); +} + +console.log('\n the pure rules'); +{ + check('the pull request merged as this commit is found', mergedAs([pr({ number: 3, merge_commit_sha: OTHER }), pr()], REPO, SHA)?.number === 7); + check('an open, unmerged or other repository\'s pull request is not', + mergedAs([pr({ merged_at: null }), pr({ base: { repo: { full_name: 'x/y' } } }), pr({ head: { sha: 'short' } })], REPO, SHA) === null && mergedAs(null, REPO, SHA) === null); + const older = run({ id: 1, created_at: '2026-11-11T20:00:00Z', conclusion: 'failure' }); + check('the newest finished pull_request run at the head is the one read', runAtHead([older, run()], { head: HEAD, number: 7 })?.id === 900); + check('a push run, a run in progress, another head and another pull request\'s run are not', + runAtHead([run({ event: 'push' }), run({ status: 'in_progress' }), run({ head_sha: OTHER }), run({ pull_requests: [{ number: 8 }] })], { head: HEAD, number: 7 }) === null); + check('every job passed reads as tested', jobsProblem([job('a'), job('b')]) === ''); + check('a skipped, failed or cancelled job, or no job at all, does not', + /e2e: skipped/.test(jobsProblem([job('a'), job('e2e', 'skipped')])) && jobsProblem([job('a', 'failure')]) !== '' && jobsProblem([job('a', 'cancelled')]) !== '' && jobsProblem([]) !== ''); +} + +console.log('\n the decision against a stubbed GitHub'); +{ + const w = world(); + const d = await decide(env(), w); + check('same tree, a green run with every job green: tested', d.tested === true && d.run === 'https://x/run/900'); + check('it asks for the runs at the head, of pull_request, of this workflow file', + w.calls.some((c) => c.startsWith(`/repos/${REPO}/actions/workflows/ci.yml/runs?head_sha=${HEAD}&event=pull_request`))); + check('and for the latest attempt\'s jobs', w.calls.some((c) => c === `/repos/${REPO}/actions/runs/900/jobs?filter=latest&per_page=100`)); +} +for (const [name, over, re] of [ + ['a direct push (no merged pull request)', { [`/repos/${REPO}/commits/${SHA}/pulls`]: [] }, /direct push/], + ['main had moved: another tree', { [`/repos/${REPO}/git/commits/${HEAD}`]: { tree: { sha: OTHER } } }, /another tree/], + ['no run at the head', { [`/repos/${REPO}/actions/workflows/ci.yml/runs`]: { workflow_runs: [] } }, /no finished pull_request run/], + ['the run at the head failed', { [`/repos/${REPO}/actions/workflows/ci.yml/runs`]: { workflow_runs: [run({ conclusion: 'failure' })] } }, /ended failure/], + ['a job there was skipped', { [`/repos/${REPO}/actions/runs/900/jobs`]: { jobs: [job('unit'), job('e2e ui', 'skipped')] } }, /e2e ui: skipped/], + ['the pull request list cannot be read', { [`/repos/${REPO}/commits/${SHA}/pulls`]: 500 }, /HTTP 500; tested in full/], + ['a commit cannot be read', { [`/repos/${REPO}/git/commits/${HEAD}`]: 502 }, /HTTP 502/], + ['the jobs cannot be read', { [`/repos/${REPO}/actions/runs/900/jobs`]: 403 }, /HTTP 403/], + ['a commit with no tree', { [`/repos/${REPO}/git/commits/${SHA}`]: {} }, /another tree/], +]) { + const d = await decide(env(), world(over)); + check(`${name}: tested in full`, d.tested === false && re.test(d.why)); +} +{ + const w = world(); + const d = await decide(env({ GITHUB_EVENT_NAME: 'pull_request' }), w); + check('a pull request run makes no request at all', d.tested === false && w.calls.length === 0); + const thrown = await decide(env(), { fetch: async () => { throw new Error('network down'); } }); + check('a fetch that throws is tested in full', thrown.tested === false && /network down; tested in full/.test(thrown.why)); +} + +console.log('\n main: the step output and the line it prints'); +{ + const dir = mkdtempSync(join(tmpdir(), 'tree-tested-')); + const out = join(dir, 'out'); + const lines = []; + const code = await main({ ...env(), GITHUB_OUTPUT: out }, { fetch: world().fetch, log: (l) => lines.push(l) }); + check('tested: the output says so and names the run, and the step passes', + code === 0 && readFileSync(out, 'utf8') === 'tested=true\nrun=https://x/run/900\n' && /^::notice::this tree already passed/.test(lines[0])); + const out2 = join(dir, 'out2'); + const code2 = await main({ ...env({ GITHUB_EVENT_NAME: 'pull_request' }), GITHUB_OUTPUT: out2 }, { fetch: world().fetch, log: (l) => lines.push(l) }); + check('not tested: tested=false, an empty run, and the step still passes', + code2 === 0 && readFileSync(out2, 'utf8') === 'tested=false\nrun=\n' && /^tested in full: a pull_request/.test(lines[1])); + rmSync(dir, { recursive: true, force: true }); +} + +console.log(`\n ${pass} passed, ${fail} failed`); +process.exit(fail ? 1 : 0); diff --git a/test-all.mjs b/test-all.mjs index a1f1659..ad1f07d 100644 --- a/test-all.mjs +++ b/test-all.mjs @@ -12,6 +12,7 @@ const RUNS = [ ['scripts/fleet-status/fleet-status.test.mjs'], ['scripts/fleet-status/check.test.mjs'], ['scripts/scorecard-publish/check.test.mjs'], + ['scripts/tree-tested/check.test.mjs'], ['scripts/redline/pin-bump.test.mjs'], // Every fuzz seed through its target, and the helpers that keep a broken contract apart from a refusal. ['fuzz/seeds.test.mjs'], From b573eda2bfd2f1cd383022a15364fc7a4d7693da Mon Sep 17 00:00:00 2001 From: askalf <263217947+askalf@users.noreply.github.com> Date: Wed, 11 Nov 2026 22:01:27 -0500 Subject: [PATCH 2/5] tree-tested: prove the tested tree, read every page of jobs, require every step A head with the pushed tree did not prove which tree its pull_request run tested: main could gain a change and revert it between the run and the merge. The check now also requires main as it was before the merge to be an ancestor of the head, so every base the run could have merged is inside it. It reads all pages of the run's jobs, and requires every step of every job to have run and passed, so a test step that runs only on push is never skipped on the word of a run that skipped it. --- CHANGELOG.md | 11 +++--- scripts/tree-tested/check.mjs | 58 ++++++++++++++++++++++++------ scripts/tree-tested/check.test.mjs | 44 +++++++++++++++++++---- 3 files changed, 90 insertions(+), 23 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 91a4b38..ffeb825 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -15,11 +15,12 @@ names the commit's date and the pull request it came from. - A new action, `actions/tree-tested`: on a push to the default branch it says whether the pull request merged as that commit already passed the same workflow, on its latest attempt and - with every job green, at a head whose tree is the pushed tree byte for byte. A caller skips its - build and test steps when it says so. A direct push, a moved main, a skipped or failed job, or a - read that fails runs the job in full, as before. Replayed against the last twelve pushes of five - repositories on 2026-11-12, it would have skipped 33 of 60 runs: claude-re 11, platform 8, - restore-desk 5, unscathed-site 5, dario 4. The pin bump moves it like the other actions. + with every job and every step green, at a head whose tree is the pushed tree byte for byte and + that holds main as it was before the merge. A caller skips its build and test steps when it says + so. A direct push, a moved main, a skipped or failed job or step, or a read that fails runs the + job in full, as before. Replayed against the last twelve pushes of five repositories on + 2026-11-12, it would have skipped 31 of 60 runs: claude-re 9, platform 7, unscathed-site 6, + restore-desk 5, dario 4. The pin bump moves it like the other actions. - A second lens, off unless a caller turns it on. A caller on a self-hosted runner that passes `second-reviewer-login` gets every head the review approves read once more in the same job, as that account, with a second env file and rubric the host keeps for it, and diff --git a/scripts/tree-tested/check.mjs b/scripts/tree-tested/check.mjs index 621d2cd..a9cba16 100644 --- a/scripts/tree-tested/check.mjs +++ b/scripts/tree-tested/check.mjs @@ -14,13 +14,17 @@ // 2. A pull request of this repository was merged as this commit (its merge_commit_sha is the // pushed sha). A direct push has none, and is tested=false. // 3. The pull request's head commit has the same tree as the pushed commit: the same files, byte -// for byte, git's own hash of them. A pull request run tests its head merged into the base it -// had then; when the merged result on main is the head's own tree, the head already held -// everything main had, so that merge was this tree too. -// 4. The newest completed pull_request run of this workflow file at that head concluded success, -// on its latest attempt, and every job in it concluded success. A job that was skipped did not -// test anything, so a run with one is not taken: a workflow that picks its jobs by what changed -// runs them all on main, as before. +// for byte, git's own hash of them. +// 4. Main as it was just before the merge (the pushed commit's first parent) is an ancestor of +// that head, by GitHub's compare. A pull_request run tests its head merged with the base main +// had then, an older main than that parent; when the parent is inside the head, so is every +// base the run could have used, and the merge it tested is the head's own tree. Without this, +// the tree a run tested is not known: main could have gained a change and reverted it between +// the run and the merge, leaving the head's tree on main but a different tree tested. +// 5. The newest completed pull_request run of this workflow file at that head concluded success, +// on its latest attempt, and every job in it, and every step of every job, concluded success, +// all pages of them. A job or a step that was skipped did not test anything (a step that runs +// only on push, a job a change did not need), so a run with one is not taken. // Every read that fails, and every rule that does not hold, is tested=false: the job runs in full. // Nothing here can turn a red run green; at worst it runs a test it could have skipped. // @@ -76,15 +80,30 @@ export function runAtHead(runs, { head, number }) { .sort((a, b) => Date.parse(b.created_at ?? 0) - Date.parse(a.created_at ?? 0))[0] ?? null; } -/** Why the run's jobs do not show this tree tested, or '' when every job passed. Pure. */ +/** + * Why the run's jobs do not show this tree tested, or '' when every job, and every step of every job, + * concluded success. A job with no steps listed is not taken either: nothing shows what it ran. Pure. + */ export function jobsProblem(jobs) { const list = Array.isArray(jobs) ? jobs : []; if (list.length === 0) return 'the run lists no jobs'; const off = list.filter((j) => j?.conclusion !== 'success'); if (off.length) return `not every job passed there (${off.slice(0, 4).map((j) => `${j?.name ?? '?'}: ${j?.conclusion ?? j?.status ?? '?'}`).join(', ')})`; + const bare = list.find((j) => !Array.isArray(j?.steps) || j.steps.length === 0); + if (bare) return `the job ${bare?.name ?? '?'} lists no steps`; + for (const j of list) { + const step = j.steps.find((st) => st?.conclusion !== 'success'); + if (step) return `not every step ran and passed there (${j?.name ?? '?'}: ${step?.name ?? '?'} ${step?.conclusion ?? step?.status ?? '?'})`; + } return ''; } +/** Whether a compare of base...head says base is inside head (head is ahead of it, or the same). Pure. */ +export const contains = (compare) => compare?.status === 'ahead' || compare?.status === 'identical'; + +/** How many pages of jobs (100 a page) a run may have before it is not read at all. */ +export const MAX_JOB_PAGES = 5; + /** GET from the API as the token. Throws on a status that is not 2xx. */ async function get(io, env, path) { const api = (env.GITHUB_API_URL || 'https://api.github.com').replace(/\/+$/, ''); @@ -117,14 +136,31 @@ export async function decide(env, io = { fetch: globalThis.fetch }) { if (!SHA_RE.test(tree) || tree !== headTree) { return { tested: false, why: `#${pr.number}'s head ${head.slice(0, 7)} has another tree than ${f.sha.slice(0, 7)}: main had moved, so the merged files were never tested together` }; } + const parent = String(pushed?.parents?.[0]?.sha ?? ''); + if (!SHA_RE.test(parent)) return { tested: false, why: `${f.sha.slice(0, 7)} names no parent to compare` }; + const cmp = await get(io, env, `/repos/${f.repo}/compare/${parent}...${head}`); + if (!contains(cmp)) { + return { tested: false, why: `main before the merge (${parent.slice(0, 7)}) is not inside #${pr.number}'s head ${head.slice(0, 7)} (${cmp?.status ?? 'no status'}), so the tree its run tested is not known` }; + } const listed = await get(io, env, `/repos/${f.repo}/actions/workflows/${f.workflowFile}/runs?head_sha=${head}&event=pull_request&per_page=50`); const run = runAtHead(listed?.workflow_runs, { head, number: pr.number }); if (!run) return { tested: false, why: `${f.workflowFile} has no finished pull_request run at #${pr.number}'s head ${head.slice(0, 7)}` }; if (run.conclusion !== 'success') return { tested: false, why: `${f.workflowFile} ended ${run.conclusion} at #${pr.number}'s head (${run.html_url ?? run.id})` }; - const jobs = await get(io, env, `/repos/${f.repo}/actions/runs/${run.id}/jobs?filter=latest&per_page=100`); - const bad = jobsProblem(jobs?.jobs); + const jobs = []; + for (let page = 1; ; page++) { + if (page > MAX_JOB_PAGES) return { tested: false, why: `the run has more than ${MAX_JOB_PAGES * 100} jobs; not read` }; + const got = await get(io, env, `/repos/${f.repo}/actions/runs/${run.id}/jobs?filter=latest&per_page=100&page=${page}`); + const batch = Array.isArray(got?.jobs) ? got.jobs : []; + jobs.push(...batch); + const total = Number(got?.total_count); + if (batch.length < 100 || (Number.isFinite(total) && jobs.length >= total)) { + if (Number.isFinite(total) && jobs.length !== total) return { tested: false, why: `read ${jobs.length} of the run's ${total} jobs` }; + break; + } + } + const bad = jobsProblem(jobs); if (bad) return { tested: false, why: `${bad} in ${run.html_url ?? run.id}` }; - return { tested: true, why: `#${pr.number}'s head ${head.slice(0, 7)} has this tree, and ${f.workflowFile} passed there with every job`, run: run.html_url ?? String(run.id) }; + return { tested: true, why: `#${pr.number}'s head ${head.slice(0, 7)} has this tree and holds main as it was before the merge, and ${f.workflowFile} passed there with every job and every step`, run: run.html_url ?? String(run.id) }; } catch (e) { return { tested: false, why: `${e.message}; tested in full` }; } diff --git a/scripts/tree-tested/check.test.mjs b/scripts/tree-tested/check.test.mjs index 621fbdf..736f3a6 100644 --- a/scripts/tree-tested/check.test.mjs +++ b/scripts/tree-tested/check.test.mjs @@ -4,7 +4,7 @@ import { mkdtempSync, readFileSync, rmSync } from 'node:fs'; import { tmpdir } from 'node:os'; import { join } from 'node:path'; -import { factsOf, notEligible, mergedAs, runAtHead, jobsProblem, decide, main } from './check.mjs'; +import { factsOf, notEligible, mergedAs, runAtHead, jobsProblem, contains, decide, main } from './check.mjs'; let pass = 0; let fail = 0; @@ -18,22 +18,25 @@ const SHA = 'a'.repeat(40); const HEAD = 'b'.repeat(40); const TREE = 'c'.repeat(40); const OTHER = 'd'.repeat(40); +const PARENT = 'e'.repeat(40); const env = (o = {}) => ({ GITHUB_REPOSITORY: REPO, GITHUB_EVENT_NAME: 'push', GITHUB_REF: 'refs/heads/main', GITHUB_SHA: SHA, GITHUB_WORKFLOW_REF: `${REPO}/.github/workflows/ci.yml@refs/heads/main`, DEFAULT_BRANCH: 'main', GITHUB_TOKEN: 't', ...o, }); const pr = (o = {}) => ({ number: 7, merged_at: '2026-11-12T00:00:00Z', merge_commit_sha: SHA, base: { repo: { full_name: REPO } }, head: { sha: HEAD }, ...o }); const run = (o = {}) => ({ id: 900, event: 'pull_request', status: 'completed', conclusion: 'success', head_sha: HEAD, created_at: '2026-11-11T23:00:00Z', html_url: 'https://x/run/900', pull_requests: [{ number: 7 }], ...o }); -const job = (name, conclusion = 'success') => ({ name, conclusion, status: 'completed' }); +const step = (name, conclusion = 'success') => ({ name, conclusion, status: 'completed' }); +const job = (name, conclusion = 'success', steps = [step('Set up job'), step('Test'), step('Complete job')]) => ({ name, conclusion, status: 'completed', steps }); /** A fake GitHub. routes: path (no query) -> body, an HTTP status, or a function of the url. */ function world(over = {}) { const routes = { [`/repos/${REPO}/commits/${SHA}/pulls`]: [pr()], - [`/repos/${REPO}/git/commits/${SHA}`]: { tree: { sha: TREE } }, + [`/repos/${REPO}/git/commits/${SHA}`]: { tree: { sha: TREE }, parents: [{ sha: PARENT }] }, + [`/repos/${REPO}/compare/${PARENT}...${HEAD}`]: { status: 'ahead' }, [`/repos/${REPO}/git/commits/${HEAD}`]: { tree: { sha: TREE } }, [`/repos/${REPO}/actions/workflows/ci.yml/runs`]: { workflow_runs: [run()] }, - [`/repos/${REPO}/actions/runs/900/jobs`]: { jobs: [job('unit'), job('e2e')] }, + [`/repos/${REPO}/actions/runs/900/jobs`]: { total_count: 2, jobs: [job('unit'), job('e2e')] }, ...over, }; const calls = []; @@ -71,7 +74,12 @@ console.log('\n the pure rules'); check('the newest finished pull_request run at the head is the one read', runAtHead([older, run()], { head: HEAD, number: 7 })?.id === 900); check('a push run, a run in progress, another head and another pull request\'s run are not', runAtHead([run({ event: 'push' }), run({ status: 'in_progress' }), run({ head_sha: OTHER }), run({ pull_requests: [{ number: 8 }] })], { head: HEAD, number: 7 }) === null); - check('every job passed reads as tested', jobsProblem([job('a'), job('b')]) === ''); + check('every job and every step passed reads as tested', jobsProblem([job('a'), job('b')]) === ''); + check('a green job with a skipped step does not: a step that runs only on push did not run there', + /integration skipped/.test(jobsProblem([job('a', 'success', [step('Set up job'), step('integration', 'skipped'), step('Complete job')])]))); + check('a green job with no steps listed does not', /lists no steps/.test(jobsProblem([{ name: 'a', conclusion: 'success' }])) && /lists no steps/.test(jobsProblem([job('a', 'success', [])]))); + check('main before the merge is inside the head when the head is ahead of it or the same', + contains({ status: 'ahead' }) && contains({ status: 'identical' }) && !contains({ status: 'behind' }) && !contains({ status: 'diverged' }) && !contains(null)); check('a skipped, failed or cancelled job, or no job at all, does not', /e2e: skipped/.test(jobsProblem([job('a'), job('e2e', 'skipped')])) && jobsProblem([job('a', 'failure')]) !== '' && jobsProblem([job('a', 'cancelled')]) !== '' && jobsProblem([]) !== ''); } @@ -83,14 +91,23 @@ console.log('\n the decision against a stubbed GitHub'); check('same tree, a green run with every job green: tested', d.tested === true && d.run === 'https://x/run/900'); check('it asks for the runs at the head, of pull_request, of this workflow file', w.calls.some((c) => c.startsWith(`/repos/${REPO}/actions/workflows/ci.yml/runs?head_sha=${HEAD}&event=pull_request`))); - check('and for the latest attempt\'s jobs', w.calls.some((c) => c === `/repos/${REPO}/actions/runs/900/jobs?filter=latest&per_page=100`)); + check('and for the latest attempt\'s jobs', w.calls.some((c) => c === `/repos/${REPO}/actions/runs/900/jobs?filter=latest&per_page=100&page=1`)); + check('it compares main before the merge with the head', w.calls.includes(`/repos/${REPO}/compare/${PARENT}...${HEAD}`)); } for (const [name, over, re] of [ ['a direct push (no merged pull request)', { [`/repos/${REPO}/commits/${SHA}/pulls`]: [] }, /direct push/], ['main had moved: another tree', { [`/repos/${REPO}/git/commits/${HEAD}`]: { tree: { sha: OTHER } } }, /another tree/], ['no run at the head', { [`/repos/${REPO}/actions/workflows/ci.yml/runs`]: { workflow_runs: [] } }, /no finished pull_request run/], ['the run at the head failed', { [`/repos/${REPO}/actions/workflows/ci.yml/runs`]: { workflow_runs: [run({ conclusion: 'failure' })] } }, /ended failure/], - ['a job there was skipped', { [`/repos/${REPO}/actions/runs/900/jobs`]: { jobs: [job('unit'), job('e2e ui', 'skipped')] } }, /e2e ui: skipped/], + ['a job there was skipped', { [`/repos/${REPO}/actions/runs/900/jobs`]: { total_count: 2, jobs: [job('unit'), job('e2e ui', 'skipped')] } }, /e2e ui: skipped/], + ['a step there was skipped in a green job', { [`/repos/${REPO}/actions/runs/900/jobs`]: { total_count: 1, jobs: [job('unit', 'success', [step('Set up job'), step('Integration tests', 'skipped')])] } }, /Integration tests skipped/], + // Main gained a change and reverted it between the run and the merge: the head's tree is on main, + // but the head never held main as it was, so the run tested the head merged with that change. + ['main changed and reverted after the run (the head does not hold main)', { [`/repos/${REPO}/compare/${PARENT}...${HEAD}`]: { status: 'diverged' } }, /is not inside #7's head .*diverged.*not known/], + ['the head is behind main before the merge', { [`/repos/${REPO}/compare/${PARENT}...${HEAD}`]: { status: 'behind' } }, /not inside/], + ['the compare cannot be read', { [`/repos/${REPO}/compare/${PARENT}...${HEAD}`]: 500 }, /HTTP 500/], + ['the pushed commit names no parent', { [`/repos/${REPO}/git/commits/${SHA}`]: { tree: { sha: TREE }, parents: [] } }, /names no parent/], + ['fewer jobs read than the run counts', { [`/repos/${REPO}/actions/runs/900/jobs`]: { total_count: 3, jobs: [job('unit'), job('e2e')] } }, /read 2 of the run's 3 jobs/], ['the pull request list cannot be read', { [`/repos/${REPO}/commits/${SHA}/pulls`]: 500 }, /HTTP 500; tested in full/], ['a commit cannot be read', { [`/repos/${REPO}/git/commits/${HEAD}`]: 502 }, /HTTP 502/], ['the jobs cannot be read', { [`/repos/${REPO}/actions/runs/900/jobs`]: 403 }, /HTTP 403/], @@ -107,6 +124,19 @@ for (const [name, over, re] of [ check('a fetch that throws is tested in full', thrown.tested === false && /network down; tested in full/.test(thrown.why)); } +console.log('\n every page of jobs is read'); +{ + const page1 = Array.from({ length: 100 }, (_, i) => job(`shard ${i}`)); + const jobsRoute = (rest) => (u) => (u.searchParams.get('page') === '1' ? { total_count: 100 + rest.length, jobs: page1 } : { total_count: 100 + rest.length, jobs: rest }); + const skipped = await decide(env(), world({ [`/repos/${REPO}/actions/runs/900/jobs`]: jobsRoute([job('last', 'skipped')]) })); + check('a skipped job past the first hundred is found: tested in full', skipped.tested === false && /last: skipped/.test(skipped.why)); + const w = world({ [`/repos/${REPO}/actions/runs/900/jobs`]: jobsRoute([job('last')]) }); + const green = await decide(env(), w); + check('a hundred and one green jobs, read over two pages: tested', green.tested === true && w.calls.some((c) => c.endsWith('&page=2'))); + const endless = await decide(env(), world({ [`/repos/${REPO}/actions/runs/900/jobs`]: () => ({ jobs: page1 }) })); + check('a run that never ends its pages is not read past the cap', endless.tested === false && /more than 500 jobs/.test(endless.why)); +} + console.log('\n main: the step output and the line it prints'); { const dir = mkdtempSync(join(tmpdir(), 'tree-tested-')); From 7883b6f7d2fd30bee22ee149494a3a6d5edf2663 Mon Sep 17 00:00:00 2001 From: askalf <263217947+askalf@users.noreply.github.com> Date: Wed, 11 Nov 2026 22:14:46 -0500 Subject: [PATCH 3/5] tree-tested: prove the merge a job tested from its own log, and require the same job there Main's parent inside the head did not prove the base a pull_request run merged: a pull request tested against a release branch and then retargeted to main keeps its old green run. The check now reads, from the log of the job of the same name in that run, the merge its checkout made, and requires it to be the head merged into a base the head holds, which is the head's own tree. It also requires the job asking to be a job of that run, so a matrix entry a push expands and a pull request does not is never skipped. The parent compare is gone; the merge in the log replaces it. --- CHANGELOG.md | 14 ++-- README.md | 2 +- actions/tree-tested/action.yml | 2 +- scripts/tree-tested/check.mjs | 107 +++++++++++++++++++++-------- scripts/tree-tested/check.test.mjs | 101 ++++++++++++++++++--------- 5 files changed, 157 insertions(+), 69 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index ffeb825..c132c56 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,13 +14,13 @@ names the commit's date and the pull request it came from. ## November 2026 - A new action, `actions/tree-tested`: on a push to the default branch it says whether the - pull request merged as that commit already passed the same workflow, on its latest attempt and - with every job and every step green, at a head whose tree is the pushed tree byte for byte and - that holds main as it was before the merge. A caller skips its build and test steps when it says - so. A direct push, a moved main, a skipped or failed job or step, or a read that fails runs the - job in full, as before. Replayed against the last twelve pushes of five repositories on - 2026-11-12, it would have skipped 31 of 60 runs: claude-re 9, platform 7, unscathed-site 6, - restore-desk 5, dario 4. The pin bump moves it like the other actions. + pull request merged as that commit already passed the same workflow, with every job and every + step green, at a head whose tree is the pushed tree byte for byte, and whether the job asking + ran there too, with a checkout its log shows was the head merged into a base the head holds. A + caller skips its build and test steps when it says so. A direct push, a moved main, a skipped + or failed job or step, a job the pull request did not run, or a read that fails runs the job in + full, as before. Replayed job by job against the last eight pushes of four repositories on + 2026-11-12, it would have skipped 83 of 162 runner minutes. The pin bump moves it like the other actions. - A second lens, off unless a caller turns it on. A caller on a self-hosted runner that passes `second-reviewer-login` gets every head the review approves read once more in the same job, as that account, with a second env file and rubric the host keeps for it, and diff --git a/README.md b/README.md index 284c7d8..2b44714 100644 --- a/README.md +++ b/README.md @@ -217,7 +217,7 @@ full commit sha of `main`, so a change here reaches it only through a reviewed b | [`actions/fleet-status`](actions/fleet-status/action.yml) | Posts the `fleet/verify` and `fleet/review` commit statuses a merge waits on; the lane rules are in [scripts/fleet-status/README.md](scripts/fleet-status/README.md). | | [`actions/scorecard-publish-check`](actions/scorecard-publish-check/action.yml) | After a Scorecard run: checks that its result reached scorecard.dev, and dispatches Scorecard once more when it did not ([the Scorecard publish check](#the-scorecard-publish-check)). | | [`actions/fleet-status-check`](actions/fleet-status-check/action.yml) | Checks a repository's fleet-status workflows, its self-hosted `runs-on` labels and its review relay, as that repository's self-test job. | -| [`actions/tree-tested`](actions/tree-tested/action.yml) | On a push to the default branch: says whether the pull request that merged it already passed this workflow, with every job, on exactly this tree, so the job can skip its build and test steps instead of running them twice. Any doubt, or any read that fails, runs the job in full. | +| [`actions/tree-tested`](actions/tree-tested/action.yml) | On a push to the default branch: says whether the pull request that merged it already passed this workflow, with every job and step, and ran this same job on exactly this tree, so the job can skip its build and test steps instead of running them twice. Any doubt, or any read that fails, runs the job in full. | | This repository's own | What it does | |---|---| diff --git a/actions/tree-tested/action.yml b/actions/tree-tested/action.yml index bf3e0c6..251d7ec 100644 --- a/actions/tree-tested/action.yml +++ b/actions/tree-tested/action.yml @@ -29,7 +29,7 @@ inputs: outputs: tested: - description: "true when this tree already passed this workflow, with every job, at the merged pull request's head; else false." + description: "true when this tree already passed this workflow, with every job and step, and this job ran there on exactly this tree; else false." run: description: The run that tested it, when tested is true. diff --git a/scripts/tree-tested/check.mjs b/scripts/tree-tested/check.mjs index a9cba16..d0e2bd3 100644 --- a/scripts/tree-tested/check.mjs +++ b/scripts/tree-tested/check.mjs @@ -15,16 +15,20 @@ // pushed sha). A direct push has none, and is tested=false. // 3. The pull request's head commit has the same tree as the pushed commit: the same files, byte // for byte, git's own hash of them. -// 4. Main as it was just before the merge (the pushed commit's first parent) is an ancestor of -// that head, by GitHub's compare. A pull_request run tests its head merged with the base main -// had then, an older main than that parent; when the parent is inside the head, so is every -// base the run could have used, and the merge it tested is the head's own tree. Without this, -// the tree a run tested is not known: main could have gained a change and reverted it between -// the run and the merge, leaving the head's tree on main but a different tree tested. -// 5. The newest completed pull_request run of this workflow file at that head concluded success, +// 4. The newest completed pull_request run of this workflow file at that head concluded success, // on its latest attempt, and every job in it, and every step of every job, concluded success, // all pages of them. A job or a step that was skipped did not test anything (a step that runs // only on push, a job a change did not need), so a run with one is not taken. +// 5. This job, the one asking (found in this run by its runner's name), has the name of a job in +// that run. A job the push runs and the pull request did not, such as a matrix entry only a +// push expands, is never in that run, so it runs in full. Steps that run only on push are a +// skipped step there, which rule 4 does not take. +// 6. That job's log shows what it checked out: GitHub's merge of the head into the base, written +// by the checkout as "Merge into ". The head there is the pull request's head, +// and that base is inside the head by GitHub's compare. A merge of a base the head already +// holds is the head's own tree, so the job tested exactly the pushed tree, whatever branch +// the pull request pointed at then and whatever main did after. With no such line, or more +// than one, what was tested is not known. // Every read that fails, and every rule that does not hold, is tested=false: the job runs in full. // Nothing here can turn a red run green; at worst it runs a test it could have skipped. // @@ -50,6 +54,8 @@ export function factsOf(env) { sha: text('GITHUB_SHA'), workflowFile: WORKFLOW_REF_RE.exec(text('GITHUB_WORKFLOW_REF'))?.[1] ?? '', defaultBranch: text('DEFAULT_BRANCH'), + runId: text('GITHUB_RUN_ID'), + runnerName: text('RUNNER_NAME'), }; } @@ -60,6 +66,8 @@ export function notEligible(f) { if (!f.defaultBranch || f.ref !== `refs/heads/${f.defaultBranch}`) return `a push to ${f.ref || 'no ref'} is tested in full; only a push to the default branch repeats a pull request's run`; if (!SHA_RE.test(f.sha)) return 'the pushed commit is not a full sha'; if (!f.workflowFile) return 'the workflow file could not be read from GITHUB_WORKFLOW_REF'; + if (!/^[1-9][0-9]*$/.test(f.runId)) return 'GITHUB_RUN_ID is not a run id'; + if (!f.runnerName) return 'RUNNER_NAME is not set, so this job cannot be found in its run'; return ''; } @@ -103,6 +111,22 @@ export const contains = (compare) => compare?.status === 'ahead' || compare?.sta /** How many pages of jobs (100 a page) a run may have before it is not read at all. */ export const MAX_JOB_PAGES = 5; +/** The largest job log read for its checkout line. */ +export const MAX_LOG_BYTES = 32 * 1024 * 1024; + +/** This job in its own run: the one in progress on this runner. Null when there is not exactly one. Pure. */ +export function thisJob(jobs, runnerName) { + const mine = (Array.isArray(jobs) ? jobs : []).filter((j) => j?.status === 'in_progress' && j?.runner_name === runnerName); + return mine.length === 1 ? mine[0] : null; +} + +const MERGE_LINE = /HEAD is now at [0-9a-f]{7,40} Merge ([0-9a-f]{40}) into ([0-9a-f]{40})/g; +/** The merges a job's log says it checked out, as { head, base }, once each. Pure. */ +export function mergesIn(log) { + const seen = new Map(); + for (const m of String(log ?? '').matchAll(MERGE_LINE)) seen.set(`${m[1]} ${m[2]}`, { head: m[1], base: m[2] }); + return [...seen.values()]; +} /** GET from the API as the token. Throws on a status that is not 2xx. */ async function get(io, env, path) { @@ -115,6 +139,35 @@ async function get(io, env, path) { return res.json(); } +/** A job's log as text; the API answers with a redirect to storage, which fetch follows. */ +async function getLog(io, env, repo, jobId) { + const api = (env.GITHUB_API_URL || 'https://api.github.com').replace(/\/+$/, ''); + const res = await io.fetch(`${api}/repos/${repo}/actions/jobs/${jobId}/logs`, { + headers: { authorization: `Bearer ${env.GITHUB_TOKEN ?? ''}`, accept: 'application/vnd.github+json', 'x-github-api-version': '2022-11-28' }, + signal: AbortSignal.timeout(60_000), + }); + if (!res.ok) throw new Error(`GET the log of job ${jobId}: HTTP ${res.status}`); + const text = await res.text(); + if (text.length > MAX_LOG_BYTES) throw new Error(`the log of job ${jobId} is over ${MAX_LOG_BYTES} bytes`); + return text; +} + +/** Every job of a run's latest attempt, all pages, or { why } when they cannot all be read. */ +async function allJobs(io, env, repo, runId) { + const jobs = []; + for (let page = 1; ; page++) { + if (page > MAX_JOB_PAGES) return { why: `run ${runId} has more than ${MAX_JOB_PAGES * 100} jobs; not read` }; + const got = await get(io, env, `/repos/${repo}/actions/runs/${runId}/jobs?filter=latest&per_page=100&page=${page}`); + const batch = Array.isArray(got?.jobs) ? got.jobs : []; + jobs.push(...batch); + const total = Number(got?.total_count); + if (batch.length < 100 || (Number.isFinite(total) && jobs.length >= total)) { + if (Number.isFinite(total) && jobs.length !== total) return { why: `read ${jobs.length} of run ${runId}'s ${total} jobs` }; + return { jobs }; + } + } +} + /** * The decision: { tested, why, run? }. tested is true only when every rule in the header holds. * io: { fetch }. @@ -136,31 +189,31 @@ export async function decide(env, io = { fetch: globalThis.fetch }) { if (!SHA_RE.test(tree) || tree !== headTree) { return { tested: false, why: `#${pr.number}'s head ${head.slice(0, 7)} has another tree than ${f.sha.slice(0, 7)}: main had moved, so the merged files were never tested together` }; } - const parent = String(pushed?.parents?.[0]?.sha ?? ''); - if (!SHA_RE.test(parent)) return { tested: false, why: `${f.sha.slice(0, 7)} names no parent to compare` }; - const cmp = await get(io, env, `/repos/${f.repo}/compare/${parent}...${head}`); - if (!contains(cmp)) { - return { tested: false, why: `main before the merge (${parent.slice(0, 7)}) is not inside #${pr.number}'s head ${head.slice(0, 7)} (${cmp?.status ?? 'no status'}), so the tree its run tested is not known` }; - } const listed = await get(io, env, `/repos/${f.repo}/actions/workflows/${f.workflowFile}/runs?head_sha=${head}&event=pull_request&per_page=50`); const run = runAtHead(listed?.workflow_runs, { head, number: pr.number }); if (!run) return { tested: false, why: `${f.workflowFile} has no finished pull_request run at #${pr.number}'s head ${head.slice(0, 7)}` }; if (run.conclusion !== 'success') return { tested: false, why: `${f.workflowFile} ended ${run.conclusion} at #${pr.number}'s head (${run.html_url ?? run.id})` }; - const jobs = []; - for (let page = 1; ; page++) { - if (page > MAX_JOB_PAGES) return { tested: false, why: `the run has more than ${MAX_JOB_PAGES * 100} jobs; not read` }; - const got = await get(io, env, `/repos/${f.repo}/actions/runs/${run.id}/jobs?filter=latest&per_page=100&page=${page}`); - const batch = Array.isArray(got?.jobs) ? got.jobs : []; - jobs.push(...batch); - const total = Number(got?.total_count); - if (batch.length < 100 || (Number.isFinite(total) && jobs.length >= total)) { - if (Number.isFinite(total) && jobs.length !== total) return { tested: false, why: `read ${jobs.length} of the run's ${total} jobs` }; - break; - } - } - const bad = jobsProblem(jobs); + const theirs = await allJobs(io, env, f.repo, run.id); + if (theirs.why) return { tested: false, why: theirs.why }; + const bad = jobsProblem(theirs.jobs); if (bad) return { tested: false, why: `${bad} in ${run.html_url ?? run.id}` }; - return { tested: true, why: `#${pr.number}'s head ${head.slice(0, 7)} has this tree and holds main as it was before the merge, and ${f.workflowFile} passed there with every job and every step`, run: run.html_url ?? String(run.id) }; + // This job, and the job of the same name there. + const ours = await allJobs(io, env, f.repo, f.runId); + if (ours.why) return { tested: false, why: ours.why }; + const me = thisJob(ours.jobs, f.runnerName); + if (!me) return { tested: false, why: `this job was not found in run ${f.runId} by its runner ${f.runnerName}` }; + const twins = theirs.jobs.filter((j) => j?.name === me.name); + if (twins.length !== 1) return { tested: false, why: `the job "${me.name}" is ${twins.length ? 'not one job' : 'not a job'} in ${run.html_url ?? run.id}: the pull request did not run it` }; + // What that job checked out. + const merges = mergesIn(await getLog(io, env, f.repo, twins[0].id)); + if (merges.length !== 1) return { tested: false, why: `the log of "${me.name}" there shows ${merges.length ? 'more than one merge' : 'no merge'} checked out, so what it tested is not known` }; + const { head: tHead, base: tBase } = merges[0]; + if (tHead !== head) return { tested: false, why: `"${me.name}" there checked out a merge of ${tHead.slice(0, 7)}, not of #${pr.number}'s head ${head.slice(0, 7)}` }; + const cmp = await get(io, env, `/repos/${f.repo}/compare/${tBase}...${head}`); + if (!contains(cmp)) { + return { tested: false, why: `"${me.name}" there tested #${pr.number}'s head merged into ${tBase.slice(0, 7)}, which the head does not hold (${cmp?.status ?? 'no status'}), so it tested another tree` }; + } + return { tested: true, why: `#${pr.number}'s head ${head.slice(0, 7)} has this tree, "${me.name}" tested it there (merged into ${tBase.slice(0, 7)}, which the head holds), and every job and step of that run passed`, run: run.html_url ?? String(run.id) }; } catch (e) { return { tested: false, why: `${e.message}; tested in full` }; } diff --git a/scripts/tree-tested/check.test.mjs b/scripts/tree-tested/check.test.mjs index 736f3a6..08cf04a 100644 --- a/scripts/tree-tested/check.test.mjs +++ b/scripts/tree-tested/check.test.mjs @@ -4,7 +4,7 @@ import { mkdtempSync, readFileSync, rmSync } from 'node:fs'; import { tmpdir } from 'node:os'; import { join } from 'node:path'; -import { factsOf, notEligible, mergedAs, runAtHead, jobsProblem, contains, decide, main } from './check.mjs'; +import { factsOf, notEligible, mergedAs, runAtHead, jobsProblem, contains, thisJob, mergesIn, decide, main } from './check.mjs'; let pass = 0; let fail = 0; @@ -18,51 +18,63 @@ const SHA = 'a'.repeat(40); const HEAD = 'b'.repeat(40); const TREE = 'c'.repeat(40); const OTHER = 'd'.repeat(40); -const PARENT = 'e'.repeat(40); +const BASE = 'e'.repeat(40); +const RELEASE = 'f'.repeat(40); +const RUN_ID = '555'; const env = (o = {}) => ({ GITHUB_REPOSITORY: REPO, GITHUB_EVENT_NAME: 'push', GITHUB_REF: 'refs/heads/main', GITHUB_SHA: SHA, - GITHUB_WORKFLOW_REF: `${REPO}/.github/workflows/ci.yml@refs/heads/main`, DEFAULT_BRANCH: 'main', GITHUB_TOKEN: 't', ...o, + GITHUB_WORKFLOW_REF: `${REPO}/.github/workflows/ci.yml@refs/heads/main`, DEFAULT_BRANCH: 'main', GITHUB_TOKEN: 't', + GITHUB_RUN_ID: RUN_ID, RUNNER_NAME: 'pool-e3', ...o, }); const pr = (o = {}) => ({ number: 7, merged_at: '2026-11-12T00:00:00Z', merge_commit_sha: SHA, base: { repo: { full_name: REPO } }, head: { sha: HEAD }, ...o }); -const run = (o = {}) => ({ id: 900, event: 'pull_request', status: 'completed', conclusion: 'success', head_sha: HEAD, created_at: '2026-11-11T23:00:00Z', html_url: 'https://x/run/900', pull_requests: [{ number: 7 }], ...o }); +const run = (o = {}) => ({ id: 900, event: 'pull_request', status: 'completed', conclusion: 'success', head_sha: HEAD, created_at: '2026-11-11T23:00:00Z', html_url: 'https://x/run/900', pull_requests: [], ...o }); const step = (name, conclusion = 'success') => ({ name, conclusion, status: 'completed' }); -const job = (name, conclusion = 'success', steps = [step('Set up job'), step('Test'), step('Complete job')]) => ({ name, conclusion, status: 'completed', steps }); +let jobIds = 1000; +const job = (name, conclusion = 'success', steps = [step('Set up job'), step('Test'), step('Complete job')]) => ({ id: jobIds++, name, conclusion, status: 'completed', steps }); +const mine = (name, runner = 'pool-e3') => ({ id: 1, name, status: 'in_progress', runner_name: runner }); +const checkoutLog = (head = HEAD, base = BASE) => `2026-11-12T02:58:14Z Fetching the repository\n2026-11-12T02:58:14Z HEAD is now at 25a09da Merge ${head} into ${base}\n2026-11-12T02:58:20Z tests pass\n`; -/** A fake GitHub. routes: path (no query) -> body, an HTTP status, or a function of the url. */ +/** A fake GitHub. routes: path (no query) -> body, an HTTP status, text, or a function of the url. */ function world(over = {}) { + const prJobs = [job('unit'), job('e2e')]; const routes = { [`/repos/${REPO}/commits/${SHA}/pulls`]: [pr()], - [`/repos/${REPO}/git/commits/${SHA}`]: { tree: { sha: TREE }, parents: [{ sha: PARENT }] }, - [`/repos/${REPO}/compare/${PARENT}...${HEAD}`]: { status: 'ahead' }, + [`/repos/${REPO}/git/commits/${SHA}`]: { tree: { sha: TREE } }, [`/repos/${REPO}/git/commits/${HEAD}`]: { tree: { sha: TREE } }, [`/repos/${REPO}/actions/workflows/ci.yml/runs`]: { workflow_runs: [run()] }, - [`/repos/${REPO}/actions/runs/900/jobs`]: { total_count: 2, jobs: [job('unit'), job('e2e')] }, + [`/repos/${REPO}/actions/runs/900/jobs`]: { total_count: 2, jobs: prJobs }, + [`/repos/${REPO}/actions/runs/${RUN_ID}/jobs`]: { total_count: 2, jobs: [mine('unit'), { id: 2, name: 'e2e', status: 'queued' }] }, + [`/repos/${REPO}/compare/${BASE}...${HEAD}`]: { status: 'ahead' }, ...over, }; + if (!('LOG' in routes)) routes.LOG = { text: checkoutLog() }; const calls = []; const fetch = async (url) => { const u = new URL(url); calls.push(u.pathname + u.search); let body = routes[u.pathname]; + if (body === undefined && /\/actions\/jobs\/\d+\/logs$/.test(u.pathname)) body = routes.LOG; if (typeof body === 'function') body = body(u); - if (body === undefined) return { ok: false, status: 404, json: async () => ({}) }; - if (typeof body === 'number') return { ok: body < 300, status: body, json: async () => ({}) }; - return { ok: true, status: 200, json: async () => body }; + if (body === undefined) return { ok: false, status: 404, json: async () => ({}), text: async () => '' }; + if (typeof body === 'number') return { ok: body < 300, status: body, json: async () => ({}), text: async () => '' }; + if (body && typeof body.text === 'string') return { ok: true, status: 200, json: async () => ({}), text: async () => body.text }; + return { ok: true, status: 200, json: async () => body, text: async () => JSON.stringify(body) }; }; - return { fetch, calls }; + return { fetch, calls, prJobs }; } console.log('\n the facts and who is looked up'); { const f = factsOf(env()); - check('the workflow file is read from GITHUB_WORKFLOW_REF', f.workflowFile === 'ci.yml' && f.sha === SHA); + check('the workflow file, run id and runner are read from the environment', f.workflowFile === 'ci.yml' && f.sha === SHA && f.runId === RUN_ID && f.runnerName === 'pool-e3'); check('a push to the default branch is eligible', notEligible(f) === ''); check('a pull request run is tested in full', /tested in full/.test(notEligible(factsOf(env({ GITHUB_EVENT_NAME: 'pull_request' }))))); check('a push to another branch is tested in full', /only a push to the default branch/.test(notEligible(factsOf(env({ GITHUB_REF: 'refs/heads/feature' }))))); check('a schedule, a dispatch and a missing event are tested in full', ['schedule', 'workflow_dispatch', ''].every((e) => notEligible(factsOf(env({ GITHUB_EVENT_NAME: e }))) !== '')); - check('no default branch, a short sha, a bad repository or no workflow ref is tested in full', - [{ DEFAULT_BRANCH: '' }, { GITHUB_SHA: 'abc' }, { GITHUB_REPOSITORY: 'nope' }, { GITHUB_WORKFLOW_REF: 'x' }].every((o) => notEligible(factsOf(env(o))) !== '')); + check('no default branch, a short sha, a bad repository, no workflow ref, no run id or no runner is tested in full', + [{ DEFAULT_BRANCH: '' }, { GITHUB_SHA: 'abc' }, { GITHUB_REPOSITORY: 'nope' }, { GITHUB_WORKFLOW_REF: 'x' }, { GITHUB_RUN_ID: 'x' }, { RUNNER_NAME: '' }] + .every((o) => notEligible(factsOf(env(o))) !== '')); } console.log('\n the pure rules'); @@ -78,21 +90,28 @@ console.log('\n the pure rules'); check('a green job with a skipped step does not: a step that runs only on push did not run there', /integration skipped/.test(jobsProblem([job('a', 'success', [step('Set up job'), step('integration', 'skipped'), step('Complete job')])]))); check('a green job with no steps listed does not', /lists no steps/.test(jobsProblem([{ name: 'a', conclusion: 'success' }])) && /lists no steps/.test(jobsProblem([job('a', 'success', [])]))); - check('main before the merge is inside the head when the head is ahead of it or the same', - contains({ status: 'ahead' }) && contains({ status: 'identical' }) && !contains({ status: 'behind' }) && !contains({ status: 'diverged' }) && !contains(null)); check('a skipped, failed or cancelled job, or no job at all, does not', /e2e: skipped/.test(jobsProblem([job('a'), job('e2e', 'skipped')])) && jobsProblem([job('a', 'failure')]) !== '' && jobsProblem([job('a', 'cancelled')]) !== '' && jobsProblem([]) !== ''); + check('a base is inside the head when the head is ahead of it or the same', + contains({ status: 'ahead' }) && contains({ status: 'identical' }) && !contains({ status: 'behind' }) && !contains({ status: 'diverged' }) && !contains(null)); + check('this job is the one in progress on this runner, and only when there is one', + thisJob([mine('a', 'other'), mine('b'), { name: 'c', status: 'queued', runner_name: 'pool-e3' }], 'pool-e3')?.name === 'b' + && thisJob([mine('a'), mine('b')], 'pool-e3') === null && thisJob([], 'pool-e3') === null); + check('the merges a checkout log names are read once each', JSON.stringify(mergesIn(checkoutLog() + checkoutLog())) === JSON.stringify([{ head: HEAD, base: BASE }]) + && mergesIn('no checkout here').length === 0 && mergesIn(checkoutLog() + checkoutLog(HEAD, RELEASE)).length === 2); } console.log('\n the decision against a stubbed GitHub'); { const w = world(); const d = await decide(env(), w); - check('same tree, a green run with every job green: tested', d.tested === true && d.run === 'https://x/run/900'); + check('same tree, every job and step green, this job there, and its checkout a merge of the head into a base it holds: tested', d.tested === true && d.run === 'https://x/run/900'); check('it asks for the runs at the head, of pull_request, of this workflow file', w.calls.some((c) => c.startsWith(`/repos/${REPO}/actions/workflows/ci.yml/runs?head_sha=${HEAD}&event=pull_request`))); - check('and for the latest attempt\'s jobs', w.calls.some((c) => c === `/repos/${REPO}/actions/runs/900/jobs?filter=latest&per_page=100&page=1`)); - check('it compares main before the merge with the head', w.calls.includes(`/repos/${REPO}/compare/${PARENT}...${HEAD}`)); + check('it reads both runs\' jobs, every page, the latest attempt', + w.calls.includes(`/repos/${REPO}/actions/runs/900/jobs?filter=latest&per_page=100&page=1`) && w.calls.includes(`/repos/${REPO}/actions/runs/${RUN_ID}/jobs?filter=latest&per_page=100&page=1`)); + check('it reads the log of the job of the same name there, and compares its base with the head', + w.calls.includes(`/repos/${REPO}/actions/jobs/${w.prJobs[0].id}/logs`) && w.calls.includes(`/repos/${REPO}/compare/${BASE}...${HEAD}`)); } for (const [name, over, re] of [ ['a direct push (no merged pull request)', { [`/repos/${REPO}/commits/${SHA}/pulls`]: [] }, /direct push/], @@ -101,25 +120,37 @@ for (const [name, over, re] of [ ['the run at the head failed', { [`/repos/${REPO}/actions/workflows/ci.yml/runs`]: { workflow_runs: [run({ conclusion: 'failure' })] } }, /ended failure/], ['a job there was skipped', { [`/repos/${REPO}/actions/runs/900/jobs`]: { total_count: 2, jobs: [job('unit'), job('e2e ui', 'skipped')] } }, /e2e ui: skipped/], ['a step there was skipped in a green job', { [`/repos/${REPO}/actions/runs/900/jobs`]: { total_count: 1, jobs: [job('unit', 'success', [step('Set up job'), step('Integration tests', 'skipped')])] } }, /Integration tests skipped/], - // Main gained a change and reverted it between the run and the merge: the head's tree is on main, - // but the head never held main as it was, so the run tested the head merged with that change. - ['main changed and reverted after the run (the head does not hold main)', { [`/repos/${REPO}/compare/${PARENT}...${HEAD}`]: { status: 'diverged' } }, /is not inside #7's head .*diverged.*not known/], - ['the head is behind main before the merge', { [`/repos/${REPO}/compare/${PARENT}...${HEAD}`]: { status: 'behind' } }, /not inside/], - ['the compare cannot be read', { [`/repos/${REPO}/compare/${PARENT}...${HEAD}`]: 500 }, /HTTP 500/], - ['the pushed commit names no parent', { [`/repos/${REPO}/git/commits/${SHA}`]: { tree: { sha: TREE }, parents: [] } }, /names no parent/], - ['fewer jobs read than the run counts', { [`/repos/${REPO}/actions/runs/900/jobs`]: { total_count: 3, jobs: [job('unit'), job('e2e')] } }, /read 2 of the run's 3 jobs/], + ['fewer jobs read than the run counts', { [`/repos/${REPO}/actions/runs/900/jobs`]: { total_count: 3, jobs: [job('unit'), job('e2e')] } }, /read 2 of run 900's 3 jobs/], + // A matrix that a push expands wider than a pull request: this job never ran there. + ['this job is a push-only matrix entry the pull request never ran', + { [`/repos/${REPO}/actions/runs/${RUN_ID}/jobs`]: { total_count: 1, jobs: [mine('test (windows)')] } }, /"test \(windows\)" is not a job in .*did not run it/], + ['this job is not found in its own run', { [`/repos/${REPO}/actions/runs/${RUN_ID}/jobs`]: { total_count: 1, jobs: [mine('unit', 'someone-else')] } }, /not found in run 555/], + ['this run\'s jobs cannot be read', { [`/repos/${REPO}/actions/runs/${RUN_ID}/jobs`]: 500 }, /HTTP 500/], + ['the log shows no merge checked out', { LOG: { text: 'no checkout line\n' } }, /no merge checked out/], + ['the log shows two different merges', { LOG: { text: checkoutLog() + checkoutLog(HEAD, RELEASE) } }, /more than one merge/], + ['the log shows a merge of another head', { LOG: { text: checkoutLog(OTHER, BASE) } }, /merge of ddddddd, not of #7's head/], + // Tested against a release branch, then retargeted to main with no new run: the base it merged + // is not in the head, so the job tested the head plus that branch, not the pushed tree. + ['the run tested a merge into a base the head does not hold (a retarget after the run)', + { LOG: { text: checkoutLog(HEAD, RELEASE) }, [`/repos/${REPO}/compare/${RELEASE}...${HEAD}`]: { status: 'diverged' } }, /merged into fffffff, which the head does not hold \(diverged\)/], + // Main gained a change and reverted it between the run and the merge: the run merged the head + // into main with that change, a base the head does not hold. + ['main changed and reverted after the run', { [`/repos/${REPO}/compare/${BASE}...${HEAD}`]: { status: 'diverged' } }, /does not hold/], + ['the log cannot be read', { LOG: 410 }, /HTTP 410/], + ['the compare cannot be read', { [`/repos/${REPO}/compare/${BASE}...${HEAD}`]: 500 }, /HTTP 500/], ['the pull request list cannot be read', { [`/repos/${REPO}/commits/${SHA}/pulls`]: 500 }, /HTTP 500; tested in full/], ['a commit cannot be read', { [`/repos/${REPO}/git/commits/${HEAD}`]: 502 }, /HTTP 502/], - ['the jobs cannot be read', { [`/repos/${REPO}/actions/runs/900/jobs`]: 403 }, /HTTP 403/], ['a commit with no tree', { [`/repos/${REPO}/git/commits/${SHA}`]: {} }, /another tree/], ]) { const d = await decide(env(), world(over)); check(`${name}: tested in full`, d.tested === false && re.test(d.why)); } { - const w = world(); - const d = await decide(env({ GITHUB_EVENT_NAME: 'pull_request' }), w); - check('a pull request run makes no request at all', d.tested === false && w.calls.length === 0); + const same = await decide(env(), world({ LOG: { text: checkoutLog(HEAD, HEAD) }, [`/repos/${REPO}/compare/${HEAD}...${HEAD}`]: { status: 'identical' } })); + check('a base that is the head itself counts as held', same.tested === true); + const w2 = world(); + const p = await decide(env({ GITHUB_EVENT_NAME: 'pull_request' }), w2); + check('a pull request run makes no request at all', p.tested === false && w2.calls.length === 0); const thrown = await decide(env(), { fetch: async () => { throw new Error('network down'); } }); check('a fetch that throws is tested in full', thrown.tested === false && /network down; tested in full/.test(thrown.why)); } @@ -130,7 +161,11 @@ console.log('\n every page of jobs is read'); const jobsRoute = (rest) => (u) => (u.searchParams.get('page') === '1' ? { total_count: 100 + rest.length, jobs: page1 } : { total_count: 100 + rest.length, jobs: rest }); const skipped = await decide(env(), world({ [`/repos/${REPO}/actions/runs/900/jobs`]: jobsRoute([job('last', 'skipped')]) })); check('a skipped job past the first hundred is found: tested in full', skipped.tested === false && /last: skipped/.test(skipped.why)); - const w = world({ [`/repos/${REPO}/actions/runs/900/jobs`]: jobsRoute([job('last')]) }); + const w = world({ + [`/repos/${REPO}/actions/runs/900/jobs`]: jobsRoute([job('last')]), + [`/repos/${REPO}/actions/runs/${RUN_ID}/jobs`]: { total_count: 1, jobs: [mine('shard 0')] }, + LOG: { text: checkoutLog() }, + }); const green = await decide(env(), w); check('a hundred and one green jobs, read over two pages: tested', green.tested === true && w.calls.some((c) => c.endsWith('&page=2'))); const endless = await decide(env(), world({ [`/repos/${REPO}/actions/runs/900/jobs`]: () => ({ jobs: page1 }) })); From 38bd567c88a99f633d23364ae5d7c2afe0b98e28 Mon Sep 17 00:00:00 2001 From: askalf <263217947+askalf@users.noreply.github.com> Date: Wed, 11 Nov 2026 22:28:30 -0500 Subject: [PATCH 4/5] tree-tested: the caller's declaration, unique job names, and the merge bound to the one checkout Two jobs under one display name, a matrix a push widens without changing the name, matched the pull request's single job; and a merge line anywhere in the log was taken, even with a later checkout of another commit or a test printing the same text. The job asking must now have a name no other job of its run has. The log must hold exactly one actions/checkout step and exactly one "HEAD is now at" line, inside that step, naming the pull request merge. What GitHub cannot show, that a job runs the same commands on push as on a pull request, the caller declares with same-on-pull-request: true; without it the answer is always false. --- CHANGELOG.md | 11 ++--- actions/tree-tested/action.yml | 12 ++++- actions/tree-tested/main.mjs | 2 +- scripts/tree-tested/check.mjs | 72 ++++++++++++++++++++++-------- scripts/tree-tested/check.test.mjs | 40 ++++++++++++----- 5 files changed, 101 insertions(+), 36 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index c132c56..571857d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,11 +16,12 @@ names the commit's date and the pull request it came from. - A new action, `actions/tree-tested`: on a push to the default branch it says whether the pull request merged as that commit already passed the same workflow, with every job and every step green, at a head whose tree is the pushed tree byte for byte, and whether the job asking - ran there too, with a checkout its log shows was the head merged into a base the head holds. A - caller skips its build and test steps when it says so. A direct push, a moved main, a skipped - or failed job or step, a job the pull request did not run, or a read that fails runs the job in - full, as before. Replayed job by job against the last eight pushes of four repositories on - 2026-11-12, it would have skipped 83 of 162 runner minutes. The pin bump moves it like the other actions. + ran there too, with one checkout its log shows was the head merged into a base the head holds. A + caller declares the job the same on push and on pull request (`same-on-pull-request: true`) and + skips its build and test steps when the action says so. A direct push, a moved main, a skipped + or failed job or step, a job the pull request did not run, two checkouts, or a read that fails + runs the job in full, as before. Replayed job by job against the last eight pushes of four + repositories on 2026-11-12, it would have skipped 75 of 170 runner minutes. The pin bump moves it like the other actions. - A second lens, off unless a caller turns it on. A caller on a self-hosted runner that passes `second-reviewer-login` gets every head the review approves read once more in the same job, as that account, with a second env file and rubric the host keeps for it, and diff --git a/actions/tree-tested/action.yml b/actions/tree-tested/action.yml index 251d7ec..0c4eb47 100644 --- a/actions/tree-tested/action.yml +++ b/actions/tree-tested/action.yml @@ -6,8 +6,12 @@ # Pinned like the other actions here: a caller names one full commit sha of this repository's main, # - id: tree # uses: askalf/ci/actions/tree-tested@ +# with: +# same-on-pull-request: true # as an early step of a job, and puts `if: steps.tree.outputs.tested != 'true'` on the steps that -# install, build and test. On anything but a push to the default branch it answers false without a +# install, build and test. same-on-pull-request is the caller's word that the job runs the same steps, +# commands and inputs on a push as on a pull request (nothing that runs only on push, no matrix a +# push widens, one checkout of the event's own commit); without it the answer is always false. On anything but a push to the default branch it answers false without a # request. Every read that fails, and every rule that does not hold, answers false too: the job then # runs in full, as it did before. The step never fails the job. # @@ -22,6 +26,12 @@ inputs: description: A token that reads the repository's commits, pull requests and Actions runs. required: false default: ${{ github.token }} + same-on-pull-request: + description: >- + true: the caller declares that this job runs the same steps, commands and inputs on a push as + on a pull request. Anything else, and the answer is false. + required: false + default: 'false' default-branch: description: The repository's default branch. A caller passes none. required: false diff --git a/actions/tree-tested/main.mjs b/actions/tree-tested/main.mjs index 4effcac..08e6fc3 100644 --- a/actions/tree-tested/main.mjs +++ b/actions/tree-tested/main.mjs @@ -7,7 +7,7 @@ import { main } from '../../scripts/tree-tested/check.mjs'; const input = (name) => (process.env[`INPUT_${name.toUpperCase()}`] ?? '').trim(); try { - process.exitCode = await main({ ...process.env, GITHUB_TOKEN: input('token'), DEFAULT_BRANCH: input('default-branch') }); + process.exitCode = await main({ ...process.env, GITHUB_TOKEN: input('token'), DEFAULT_BRANCH: input('default-branch'), SAME_ON_PULL_REQUEST: input('same-on-pull-request') }); } catch (e) { // A failure of the check runs the job in full; it never fails the job. console.log(`tested in full: ${e.message}`); diff --git a/scripts/tree-tested/check.mjs b/scripts/tree-tested/check.mjs index d0e2bd3..b2393bb 100644 --- a/scripts/tree-tested/check.mjs +++ b/scripts/tree-tested/check.mjs @@ -9,7 +9,15 @@ // was about 300 of a day's CPU-heavy runner minutes (restore-desk, claude-re, platform, dario), on a // host whose runner slice was waiting on CPU a quarter of the time. // +// The caller's declaration. GitHub does not record what a job's steps did with their inputs, so the +// one thing this cannot read is whether a job runs the same commands on a push as on a pull request. +// The caller says so, with the action's same-on-pull-request input set to true, and with nothing set +// the answer is always false. It holds when the job's steps, their commands and their inputs do not +// depend on the event: no step or command that runs only on push, no matrix that a push widens, no +// checkout of anything but the event's own commit. What the rules below can check, they check. +// // The rule, all of it read from the GitHub API with the job's own token: +// 0. The caller declared the job the same on push and on pull request (above). // 1. The event is a push to the default branch. Anything else is tested=false. // 2. A pull request of this repository was merged as this commit (its merge_commit_sha is the // pushed sha). A direct push has none, and is tested=false. @@ -19,16 +27,18 @@ // on its latest attempt, and every job in it, and every step of every job, concluded success, // all pages of them. A job or a step that was skipped did not test anything (a step that runs // only on push, a job a change did not need), so a run with one is not taken. -// 5. This job, the one asking (found in this run by its runner's name), has the name of a job in -// that run. A job the push runs and the pull request did not, such as a matrix entry only a -// push expands, is never in that run, so it runs in full. Steps that run only on push are a -// skipped step there, which rule 4 does not take. -// 6. That job's log shows what it checked out: GitHub's merge of the head into the base, written -// by the checkout as "Merge into ". The head there is the pull request's head, -// and that base is inside the head by GitHub's compare. A merge of a base the head already -// holds is the head's own tree, so the job tested exactly the pushed tree, whatever branch -// the pull request pointed at then and whatever main did after. With no such line, or more -// than one, what was tested is not known. +// 5. This job, the one asking (found in this run by its runner's name), has a name no other job +// of this run has, and exactly one job of that run has it. A job the push runs and the pull +// request did not is not there, and two jobs under one display name (a matrix whose entries +// share a name) cannot be told apart, so neither is taken. +// 6. That job's log shows what its tests ran on. It has exactly one actions/checkout step, and +// exactly one "HEAD is now at" line in the whole log (git prints one for every checkout, +// switch or hard reset, so a later one, or a test that prints one, makes two), and that line +// is inside the checkout step and reads "Merge into ": GitHub's merge of the pull +// request. The head there is the pull request's head, and that base is inside the head by +// GitHub's compare. A merge of a base the head already holds is the head's own tree, so the +// job tested exactly the pushed tree, whatever branch the pull request pointed at then and +// whatever main did after. Anything else, and what was tested is not known. // Every read that fails, and every rule that does not hold, is tested=false: the job runs in full. // Nothing here can turn a red run green; at worst it runs a test it could have skipped. // @@ -56,11 +66,13 @@ export function factsOf(env) { defaultBranch: text('DEFAULT_BRANCH'), runId: text('GITHUB_RUN_ID'), runnerName: text('RUNNER_NAME'), + declared: text('SAME_ON_PULL_REQUEST'), }; } /** Why this run is not one to look up at all, or '' when it is. Pure. */ export function notEligible(f) { + if (f.declared !== 'true') return 'the caller has not declared this job the same on a push as on a pull request (same-on-pull-request: true)'; if (!REPO_RE.test(f.repo)) return `GITHUB_REPOSITORY is not owner/name (${JSON.stringify(f.repo)})`; if (f.event !== 'push') return `a ${f.event || 'run with no event'} is tested in full`; if (!f.defaultBranch || f.ref !== `refs/heads/${f.defaultBranch}`) return `a push to ${f.ref || 'no ref'} is tested in full; only a push to the default branch repeats a pull request's run`; @@ -120,12 +132,32 @@ export function thisJob(jobs, runnerName) { return mine.length === 1 ? mine[0] : null; } -const MERGE_LINE = /HEAD is now at [0-9a-f]{7,40} Merge ([0-9a-f]{40}) into ([0-9a-f]{40})/g; -/** The merges a job's log says it checked out, as { head, base }, once each. Pure. */ -export function mergesIn(log) { - const seen = new Map(); - for (const m of String(log ?? '').matchAll(MERGE_LINE)) seen.set(`${m[1]} ${m[2]}`, { head: m[1], base: m[2] }); - return [...seen.values()]; +const STEP_START = /##\[group\]Run (.*)$/; +const CHECKOUT_STEP = /^actions\/checkout@/; +const HEAD_LINE = /HEAD is now at [0-9a-f]{7,40} /; +const MERGE_LINE = /HEAD is now at [0-9a-f]{7,40} Merge ([0-9a-f]{40}) into ([0-9a-f]{40})\s*$/; +/** + * What a job's log shows its tests ran on: { head, base } of the pull request merge its one + * checkout made, or { why }. The log is cut into steps at each "##[group]Run " line; there + * must be exactly one actions/checkout step, exactly one "HEAD is now at" line in the whole log, + * and that line inside the checkout step, naming a merge. Pure. + */ +export function checkoutMerge(log) { + const lines = String(log ?? '').split(/\r?\n/); + let step = ''; + let checkouts = 0; + const heads = []; + for (const line of lines) { + const s = STEP_START.exec(line); + if (s) { step = s[1].trim(); if (CHECKOUT_STEP.test(step)) checkouts++; continue; } + if (HEAD_LINE.test(line)) heads.push({ line, step }); + } + if (checkouts !== 1) return { why: `the job has ${checkouts} actions/checkout steps, not one` }; + if (heads.length !== 1) return { why: `the log has ${heads.length} "HEAD is now at" lines, not one, so a later checkout, switch or reset may have moved what the tests ran on` }; + if (!CHECKOUT_STEP.test(heads[0].step)) return { why: 'its one "HEAD is now at" line is not in the checkout step' }; + const m = MERGE_LINE.exec(heads[0].line); + if (!m) return { why: 'the checkout did not check out a pull request merge' }; + return { head: m[1], base: m[2] }; } /** GET from the API as the token. Throws on a status that is not 2xx. */ @@ -202,12 +234,14 @@ export async function decide(env, io = { fetch: globalThis.fetch }) { if (ours.why) return { tested: false, why: ours.why }; const me = thisJob(ours.jobs, f.runnerName); if (!me) return { tested: false, why: `this job was not found in run ${f.runId} by its runner ${f.runnerName}` }; + const namesakes = ours.jobs.filter((j) => j?.name === me.name).length; + if (namesakes !== 1) return { tested: false, why: `${namesakes} jobs of this run are named "${me.name}", so the one the pull request ran cannot be told apart` }; const twins = theirs.jobs.filter((j) => j?.name === me.name); if (twins.length !== 1) return { tested: false, why: `the job "${me.name}" is ${twins.length ? 'not one job' : 'not a job'} in ${run.html_url ?? run.id}: the pull request did not run it` }; // What that job checked out. - const merges = mergesIn(await getLog(io, env, f.repo, twins[0].id)); - if (merges.length !== 1) return { tested: false, why: `the log of "${me.name}" there shows ${merges.length ? 'more than one merge' : 'no merge'} checked out, so what it tested is not known` }; - const { head: tHead, base: tBase } = merges[0]; + const seen = checkoutMerge(await getLog(io, env, f.repo, twins[0].id)); + if (seen.why) return { tested: false, why: `in the log of "${me.name}" there, ${seen.why}; what it tested is not known` }; + const { head: tHead, base: tBase } = seen; if (tHead !== head) return { tested: false, why: `"${me.name}" there checked out a merge of ${tHead.slice(0, 7)}, not of #${pr.number}'s head ${head.slice(0, 7)}` }; const cmp = await get(io, env, `/repos/${f.repo}/compare/${tBase}...${head}`); if (!contains(cmp)) { diff --git a/scripts/tree-tested/check.test.mjs b/scripts/tree-tested/check.test.mjs index 08cf04a..2e88e93 100644 --- a/scripts/tree-tested/check.test.mjs +++ b/scripts/tree-tested/check.test.mjs @@ -4,7 +4,7 @@ import { mkdtempSync, readFileSync, rmSync } from 'node:fs'; import { tmpdir } from 'node:os'; import { join } from 'node:path'; -import { factsOf, notEligible, mergedAs, runAtHead, jobsProblem, contains, thisJob, mergesIn, decide, main } from './check.mjs'; +import { factsOf, notEligible, mergedAs, runAtHead, jobsProblem, contains, thisJob, checkoutMerge, decide, main } from './check.mjs'; let pass = 0; let fail = 0; @@ -24,7 +24,7 @@ const RUN_ID = '555'; const env = (o = {}) => ({ GITHUB_REPOSITORY: REPO, GITHUB_EVENT_NAME: 'push', GITHUB_REF: 'refs/heads/main', GITHUB_SHA: SHA, GITHUB_WORKFLOW_REF: `${REPO}/.github/workflows/ci.yml@refs/heads/main`, DEFAULT_BRANCH: 'main', GITHUB_TOKEN: 't', - GITHUB_RUN_ID: RUN_ID, RUNNER_NAME: 'pool-e3', ...o, + GITHUB_RUN_ID: RUN_ID, RUNNER_NAME: 'pool-e3', SAME_ON_PULL_REQUEST: 'true', ...o, }); const pr = (o = {}) => ({ number: 7, merged_at: '2026-11-12T00:00:00Z', merge_commit_sha: SHA, base: { repo: { full_name: REPO } }, head: { sha: HEAD }, ...o }); const run = (o = {}) => ({ id: 900, event: 'pull_request', status: 'completed', conclusion: 'success', head_sha: HEAD, created_at: '2026-11-11T23:00:00Z', html_url: 'https://x/run/900', pull_requests: [], ...o }); @@ -32,7 +32,12 @@ const step = (name, conclusion = 'success') => ({ name, conclusion, status: 'com let jobIds = 1000; const job = (name, conclusion = 'success', steps = [step('Set up job'), step('Test'), step('Complete job')]) => ({ id: jobIds++, name, conclusion, status: 'completed', steps }); const mine = (name, runner = 'pool-e3') => ({ id: 1, name, status: 'in_progress', runner_name: runner }); -const checkoutLog = (head = HEAD, base = BASE) => `2026-11-12T02:58:14Z Fetching the repository\n2026-11-12T02:58:14Z HEAD is now at 25a09da Merge ${head} into ${base}\n2026-11-12T02:58:20Z tests pass\n`; +// A job log as GitHub writes it: each step opens with "##[group]Run ". +const T = '2026-11-12T02:58:14.1195074Z '; +const checkoutStep = (head = HEAD, base = BASE) => [`${T}##[group]Run actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1`, `${T}with:`, `${T}##[endgroup]`, + `${T}##[group]Checking out the ref`, `${T}HEAD is now at 25a09da Merge ${head} into ${base}`, `${T}##[endgroup]`]; +const testStep = (out = 'tests pass') => [`${T}##[group]Run npm test`, `${T}npm test`, `${T}##[endgroup]`, `${T}${out}`]; +const checkoutLog = (head = HEAD, base = BASE) => [`${T}##[group]Run askalf/ci/actions/checkout-with-retry@${'1'.repeat(40)}`, `${T}##[endgroup]`, ...checkoutStep(head, base), ...testStep()].join('\n') + '\n'; /** A fake GitHub. routes: path (no query) -> body, an HTTP status, text, or a function of the url. */ function world(over = {}) { @@ -66,7 +71,9 @@ function world(over = {}) { console.log('\n the facts and who is looked up'); { const f = factsOf(env()); - check('the workflow file, run id and runner are read from the environment', f.workflowFile === 'ci.yml' && f.sha === SHA && f.runId === RUN_ID && f.runnerName === 'pool-e3'); + check('the workflow file, run id, runner and the caller\'s word are read from the environment', f.workflowFile === 'ci.yml' && f.sha === SHA && f.runId === RUN_ID && f.runnerName === 'pool-e3' && f.declared === 'true'); + check('without the caller\'s word that the job is the same on push, nothing is skipped', + ['', 'false', 'yes', 'TRUE'].every((v) => /has not declared this job the same/.test(notEligible(factsOf(env({ SAME_ON_PULL_REQUEST: v })))))); check('a push to the default branch is eligible', notEligible(f) === ''); check('a pull request run is tested in full', /tested in full/.test(notEligible(factsOf(env({ GITHUB_EVENT_NAME: 'pull_request' }))))); check('a push to another branch is tested in full', /only a push to the default branch/.test(notEligible(factsOf(env({ GITHUB_REF: 'refs/heads/feature' }))))); @@ -97,8 +104,17 @@ console.log('\n the pure rules'); check('this job is the one in progress on this runner, and only when there is one', thisJob([mine('a', 'other'), mine('b'), { name: 'c', status: 'queued', runner_name: 'pool-e3' }], 'pool-e3')?.name === 'b' && thisJob([mine('a'), mine('b')], 'pool-e3') === null && thisJob([], 'pool-e3') === null); - check('the merges a checkout log names are read once each', JSON.stringify(mergesIn(checkoutLog() + checkoutLog())) === JSON.stringify([{ head: HEAD, base: BASE }]) - && mergesIn('no checkout here').length === 0 && mergesIn(checkoutLog() + checkoutLog(HEAD, RELEASE)).length === 2); + check('the merge the one checkout made is read from its step', JSON.stringify(checkoutMerge(checkoutLog())) === JSON.stringify({ head: HEAD, base: BASE })); + const lines = (...parts) => parts.flat().join('\n'); + // A merge checkout, then a step that checks out an older ordinary commit before the tests. + check('a later checkout of another commit, even by hand, is caught: two "HEAD is now at" lines', + /2 "HEAD is now at" lines/.test(checkoutMerge(lines(checkoutStep(), [`${T}##[group]Run git checkout ${OTHER}`, `${T}##[endgroup]`, `${T}HEAD is now at ddddddd an older commit`], testStep())).why)); + check('a second actions/checkout step is not taken', /2 actions\/checkout steps/.test(checkoutMerge(lines(checkoutStep(), checkoutStep(), testStep())).why)); + check('a merge line printed by a test, outside the checkout, is not taken', + /not in the checkout step/.test(checkoutMerge(lines([`${T}##[group]Run actions/checkout@${'2'.repeat(40)}`, `${T}##[endgroup]`], testStep(`HEAD is now at 25a09da Merge ${HEAD} into ${BASE}`))).why)); + check('a checkout of the head itself, not a merge, is not taken', + /did not check out a pull request merge/.test(checkoutMerge(lines([`${T}##[group]Run actions/checkout@${'2'.repeat(40)}`, `${T}HEAD is now at bbbbbbb the head commit`], testStep())).why)); + check('no checkout at all is not taken', /0 actions\/checkout steps/.test(checkoutMerge(lines(testStep())).why) && /0 actions\/checkout steps/.test(checkoutMerge('').why)); } console.log('\n the decision against a stubbed GitHub'); @@ -113,7 +129,7 @@ console.log('\n the decision against a stubbed GitHub'); check('it reads the log of the job of the same name there, and compares its base with the head', w.calls.includes(`/repos/${REPO}/actions/jobs/${w.prJobs[0].id}/logs`) && w.calls.includes(`/repos/${REPO}/compare/${BASE}...${HEAD}`)); } -for (const [name, over, re] of [ +for (const [name, over, re, envOver] of [ ['a direct push (no merged pull request)', { [`/repos/${REPO}/commits/${SHA}/pulls`]: [] }, /direct push/], ['main had moved: another tree', { [`/repos/${REPO}/git/commits/${HEAD}`]: { tree: { sha: OTHER } } }, /another tree/], ['no run at the head', { [`/repos/${REPO}/actions/workflows/ci.yml/runs`]: { workflow_runs: [] } }, /no finished pull_request run/], @@ -126,8 +142,12 @@ for (const [name, over, re] of [ { [`/repos/${REPO}/actions/runs/${RUN_ID}/jobs`]: { total_count: 1, jobs: [mine('test (windows)')] } }, /"test \(windows\)" is not a job in .*did not run it/], ['this job is not found in its own run', { [`/repos/${REPO}/actions/runs/${RUN_ID}/jobs`]: { total_count: 1, jobs: [mine('unit', 'someone-else')] } }, /not found in run 555/], ['this run\'s jobs cannot be read', { [`/repos/${REPO}/actions/runs/${RUN_ID}/jobs`]: 500 }, /HTTP 500/], - ['the log shows no merge checked out', { LOG: { text: 'no checkout line\n' } }, /no merge checked out/], - ['the log shows two different merges', { LOG: { text: checkoutLog() + checkoutLog(HEAD, RELEASE) } }, /more than one merge/], + ['the log shows no checkout', { LOG: { text: 'no checkout line\n' } }, /0 actions\/checkout steps, not one; what it tested is not known/], + ['the log shows a merge checkout and then another checkout', { LOG: { text: checkoutLog() + checkoutLog(HEAD, RELEASE) } }, /2 actions\/checkout steps/], + // A matrix whose entries share one display name, wider on push: the job asking cannot be matched. + ['two jobs of this run share the display name of the job asking (a wider push matrix under name: test)', + { [`/repos/${REPO}/actions/runs/${RUN_ID}/jobs`]: { total_count: 2, jobs: [mine('unit'), { id: 3, name: 'unit', status: 'queued' }] } }, /2 jobs of this run are named "unit"/], + ['the caller has not declared the job the same on push', {}, /has not declared/, { SAME_ON_PULL_REQUEST: '' }], ['the log shows a merge of another head', { LOG: { text: checkoutLog(OTHER, BASE) } }, /merge of ddddddd, not of #7's head/], // Tested against a release branch, then retargeted to main with no new run: the base it merged // is not in the head, so the job tested the head plus that branch, not the pushed tree. @@ -142,7 +162,7 @@ for (const [name, over, re] of [ ['a commit cannot be read', { [`/repos/${REPO}/git/commits/${HEAD}`]: 502 }, /HTTP 502/], ['a commit with no tree', { [`/repos/${REPO}/git/commits/${SHA}`]: {} }, /another tree/], ]) { - const d = await decide(env(), world(over)); + const d = await decide(env(envOver ?? {}), world(over)); check(`${name}: tested in full`, d.tested === false && re.test(d.why)); } { From fb26fc04516a008ef477d0e49173b13a1a2a9775 Mon Sep 17 00:00:00 2001 From: askalf <263217947+askalf@users.noreply.github.com> Date: Wed, 11 Nov 2026 22:32:59 -0500 Subject: [PATCH 5/5] tree-tested: say what the log check can and cannot see Git does not print "HEAD is now at" for every change of a checkout: a switch to a branch prints "Switched to branch", and a quiet checkout or reset prints nothing. The comment no longer claims it does. The log check catches a second visible line; that nothing after the checkout moves the tests onto other files is part of the caller's declaration, and the comment and description now say so. --- scripts/tree-tested/check.mjs | 25 +++++++++++++++---------- 1 file changed, 15 insertions(+), 10 deletions(-) diff --git a/scripts/tree-tested/check.mjs b/scripts/tree-tested/check.mjs index b2393bb..3ae97bf 100644 --- a/scripts/tree-tested/check.mjs +++ b/scripts/tree-tested/check.mjs @@ -14,7 +14,9 @@ // The caller says so, with the action's same-on-pull-request input set to true, and with nothing set // the answer is always false. It holds when the job's steps, their commands and their inputs do not // depend on the event: no step or command that runs only on push, no matrix that a push widens, no -// checkout of anything but the event's own commit. What the rules below can check, they check. +// checkout of anything but the event's own commit, and nothing after that checkout that moves the +// tests onto other files (a branch switch, a reset, a second checkout). What the rules below can +// check, they check; the rest is the caller's word. // // The rule, all of it read from the GitHub API with the job's own token: // 0. The caller declared the job the same on push and on pull request (above). @@ -31,14 +33,17 @@ // of this run has, and exactly one job of that run has it. A job the push runs and the pull // request did not is not there, and two jobs under one display name (a matrix whose entries // share a name) cannot be told apart, so neither is taken. -// 6. That job's log shows what its tests ran on. It has exactly one actions/checkout step, and -// exactly one "HEAD is now at" line in the whole log (git prints one for every checkout, -// switch or hard reset, so a later one, or a test that prints one, makes two), and that line -// is inside the checkout step and reads "Merge into ": GitHub's merge of the pull -// request. The head there is the pull request's head, and that base is inside the head by -// GitHub's compare. A merge of a base the head already holds is the head's own tree, so the -// job tested exactly the pushed tree, whatever branch the pull request pointed at then and -// whatever main did after. Anything else, and what was tested is not known. +// 6. That job's log shows what its checkout made. It has exactly one actions/checkout step, and +// exactly one "HEAD is now at" line in the whole log, and that line is inside the checkout +// step and reads "Merge into ": GitHub's merge of the pull request. The head +// there is the pull request's head, and that base is inside the head by GitHub's compare. A +// merge of a base the head already holds is the head's own tree, so the checkout was exactly +// the pushed tree, whatever branch the pull request pointed at then and whatever main did +// after. A second visible "HEAD is now at" line (a later detached checkout or hard reset, or +// a test that prints one) is caught. Not everything that moves a checkout prints that line: +// a switch to a branch prints "Switched to branch", and a quiet checkout or reset prints +// nothing. That the tests ran on what the checkout made is the caller's declaration above, +// not something this log can show. // Every read that fails, and every rule that does not hold, is tested=false: the job runs in full. // Nothing here can turn a red run green; at worst it runs a test it could have skipped. // @@ -153,7 +158,7 @@ export function checkoutMerge(log) { if (HEAD_LINE.test(line)) heads.push({ line, step }); } if (checkouts !== 1) return { why: `the job has ${checkouts} actions/checkout steps, not one` }; - if (heads.length !== 1) return { why: `the log has ${heads.length} "HEAD is now at" lines, not one, so a later checkout, switch or reset may have moved what the tests ran on` }; + if (heads.length !== 1) return { why: `the log has ${heads.length} "HEAD is now at" lines, not one, so something after the checkout may have moved what the tests ran on` }; if (!CHECKOUT_STEP.test(heads[0].step)) return { why: 'its one "HEAD is now at" line is not in the checkout step' }; const m = MERGE_LINE.exec(heads[0].line); if (!m) return { why: 'the checkout did not check out a pull request merge' };