diff --git a/.github/workflows/redline-pin-bump.yml b/.github/workflows/redline-pin-bump.yml index 7826344..bcecfc0 100644 --- a/.github/workflows/redline-pin-bump.yml +++ b/.github/workflows/redline-pin-bump.yml @@ -6,7 +6,8 @@ # whichever workflow file it lives: uses: askalf/ci/actions/drift-report@ (a watcher's report # step), uses: askalf/ci/actions/checkout-with-retry@ (any job's checkout), # uses: askalf/ci/actions/fleet-status@ and fleet-status-check@ (the fleet lanes), and -# uses: askalf/ci/actions/scorecard-publish-check@ (the check after a Scorecard run). +# uses: askalf/ci/actions/scorecard-publish-check@ (the check after a Scorecard run), and +# uses: askalf/ci/actions/tree-tested@ (the skip for a tree a pull request already tested). # Each pins by SHA so a change reaches it only through a reviewed bump there; this writes those bumps # (scripts/redline/pin.mjs), and each repo's own gate reviews and merges them. # @@ -123,6 +124,9 @@ jobs: :(exclude)scripts/scorecard-publish/*.test.mjs :(exclude)scripts/scorecard-publish/README.md actions/scorecard-publish-check + scripts/tree-tested + :(exclude)scripts/tree-tested/*.test.mjs + actions/tree-tested .github/workflows/redline-review.yml .github/workflows/redline-fix-run.yml .github/workflows/public-text.yml diff --git a/.github/workflows/redline-self-test.yml b/.github/workflows/redline-self-test.yml index 36dc883..0ebf023 100644 --- a/.github/workflows/redline-self-test.yml +++ b/.github/workflows/redline-self-test.yml @@ -41,6 +41,7 @@ jobs: - run: node scripts/fleet-status/check.test.mjs # The check after a Scorecard run, and its action's entry, against a stubbed network. - run: node scripts/scorecard-publish/check.test.mjs + - run: node scripts/tree-tested/check.test.mjs # The pin bump workflow's bump step, run as the workflow runs it, against a fake gh. - run: node scripts/redline/pin-bump.test.mjs # Every fuzz seed through its target, so a target that rots fails here and not only in the weekly fuzz run. diff --git a/CHANGELOG.md b/CHANGELOG.md index 40789ed..571857d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -13,6 +13,15 @@ names the commit's date and the pull request it came from. ## November 2026 +- A new action, `actions/tree-tested`: on a push to the default branch it says whether the + pull request merged as that commit already passed the same workflow, with every job and every + step green, at a head whose tree is the pushed tree byte for byte, and whether the job asking + ran there too, with one checkout its log shows was the head merged into a base the head holds. A + caller declares the job the same on push and on pull request (`same-on-pull-request: true`) and + skips its build and test steps when the action says so. A direct push, a moved main, a skipped + or failed job or step, a job the pull request did not run, two checkouts, or a read that fails + runs the job in full, as before. Replayed job by job against the last eight pushes of four + repositories on 2026-11-12, it would have skipped 75 of 170 runner minutes. The pin bump moves it like the other actions. - A second lens, off unless a caller turns it on. A caller on a self-hosted runner that passes `second-reviewer-login` gets every head the review approves read once more in the same job, as that account, with a second env file and rubric the host keeps for it, and diff --git a/README.md b/README.md index 609f2d1..2b44714 100644 --- a/README.md +++ b/README.md @@ -217,6 +217,7 @@ full commit sha of `main`, so a change here reaches it only through a reviewed b | [`actions/fleet-status`](actions/fleet-status/action.yml) | Posts the `fleet/verify` and `fleet/review` commit statuses a merge waits on; the lane rules are in [scripts/fleet-status/README.md](scripts/fleet-status/README.md). | | [`actions/scorecard-publish-check`](actions/scorecard-publish-check/action.yml) | After a Scorecard run: checks that its result reached scorecard.dev, and dispatches Scorecard once more when it did not ([the Scorecard publish check](#the-scorecard-publish-check)). | | [`actions/fleet-status-check`](actions/fleet-status-check/action.yml) | Checks a repository's fleet-status workflows, its self-hosted `runs-on` labels and its review relay, as that repository's self-test job. | +| [`actions/tree-tested`](actions/tree-tested/action.yml) | On a push to the default branch: says whether the pull request that merged it already passed this workflow, with every job and step, and ran this same job on exactly this tree, so the job can skip its build and test steps instead of running them twice. Any doubt, or any read that fails, runs the job in full. | | This repository's own | What it does | |---|---| diff --git a/actions/tree-tested/action.yml b/actions/tree-tested/action.yml new file mode 100644 index 0000000..0c4eb47 --- /dev/null +++ b/actions/tree-tested/action.yml @@ -0,0 +1,48 @@ +# Says whether a push to the default branch carries a tree that this workflow already tested green +# on the pull request that merged it, so the job can skip its build and test steps instead of +# running them a second time on the same files. The rules are scripts/tree-tested/check.mjs (its +# header says each one), tested without the network by check.test.mjs. +# +# Pinned like the other actions here: a caller names one full commit sha of this repository's main, +# - id: tree +# uses: askalf/ci/actions/tree-tested@ +# with: +# same-on-pull-request: true +# as an early step of a job, and puts `if: steps.tree.outputs.tested != 'true'` on the steps that +# install, build and test. same-on-pull-request is the caller's word that the job runs the same steps, +# commands and inputs on a push as on a pull request (nothing that runs only on push, no matrix a +# push widens, one checkout of the event's own commit); without it the answer is always false. On anything but a push to the default branch it answers false without a +# request. Every read that fails, and every rule that does not hold, answers false too: the job then +# runs in full, as it did before. The step never fails the job. +# +# It checks out nothing and runs on the Node the runner ships. The job's token needs actions: read, +# contents: read and pull-requests: read, which a workflow's default token has unless its +# permissions block leaves one out. +name: Tree tested +description: Say whether this push's tree already passed this workflow on the pull request that merged it. + +inputs: + token: + description: A token that reads the repository's commits, pull requests and Actions runs. + required: false + default: ${{ github.token }} + same-on-pull-request: + description: >- + true: the caller declares that this job runs the same steps, commands and inputs on a push as + on a pull request. Anything else, and the answer is false. + required: false + default: 'false' + default-branch: + description: The repository's default branch. A caller passes none. + required: false + default: ${{ github.event.repository.default_branch }} + +outputs: + tested: + description: "true when this tree already passed this workflow, with every job and step, and this job ran there on exactly this tree; else false." + run: + description: The run that tested it, when tested is true. + +runs: + using: node24 + main: main.mjs diff --git a/actions/tree-tested/main.mjs b/actions/tree-tested/main.mjs new file mode 100644 index 0000000..08e6fc3 --- /dev/null +++ b/actions/tree-tested/main.mjs @@ -0,0 +1,15 @@ +// The action's entry. The runner runs this file with the Node it ships (runs.using in action.yml) +// and hands each input over as INPUT_. The event's facts (repository, event, ref, sha, the +// workflow's file) are the runner's own GITHUB_* variables; the script reads those. +import { appendFileSync } from 'node:fs'; +import { main } from '../../scripts/tree-tested/check.mjs'; + +const input = (name) => (process.env[`INPUT_${name.toUpperCase()}`] ?? '').trim(); + +try { + process.exitCode = await main({ ...process.env, GITHUB_TOKEN: input('token'), DEFAULT_BRANCH: input('default-branch'), SAME_ON_PULL_REQUEST: input('same-on-pull-request') }); +} catch (e) { + // A failure of the check runs the job in full; it never fails the job. + console.log(`tested in full: ${e.message}`); + if (process.env.GITHUB_OUTPUT) appendFileSync(process.env.GITHUB_OUTPUT, 'tested=false\nrun=\n'); +} diff --git a/scripts/redline/pin.mjs b/scripts/redline/pin.mjs index b664d68..f673816 100644 --- a/scripts/redline/pin.mjs +++ b/scripts/redline/pin.mjs @@ -55,8 +55,9 @@ export const CHECKOUT_RETRY_ACTION = `${CI_REPO}/actions/checkout-with-retry`; export const FLEET_STATUS_ACTION = `${CI_REPO}/actions/fleet-status`; export const FLEET_STATUS_CHECK_ACTION = `${CI_REPO}/actions/fleet-status-check`; export const SCORECARD_PUBLISH_CHECK_ACTION = `${CI_REPO}/actions/scorecard-publish-check`; +export const TREE_TESTED_ACTION = `${CI_REPO}/actions/tree-tested`; /** Every action of this repository a workflow pins as a step; `--action` moves them all. */ -export const ACTIONS = [DRIFT_REPORT_ACTION, CHECKOUT_RETRY_ACTION, FLEET_STATUS_ACTION, FLEET_STATUS_CHECK_ACTION, SCORECARD_PUBLISH_CHECK_ACTION]; +export const ACTIONS = [DRIFT_REPORT_ACTION, CHECKOUT_RETRY_ACTION, FLEET_STATUS_ACTION, FLEET_STATUS_CHECK_ACTION, SCORECARD_PUBLISH_CHECK_ACTION, TREE_TESTED_ACTION]; /** Every reusable workflow a caller pins, and the caller file that pins it. */ export const CALLERS = [ { workflow: REVIEW_WORKFLOW, path: '.github/workflows/redline.yml' }, diff --git a/scripts/redline/review.test.mjs b/scripts/redline/review.test.mjs index a364c45..cc1c40d 100644 --- a/scripts/redline/review.test.mjs +++ b/scripts/redline/review.test.mjs @@ -311,7 +311,7 @@ console.log('\n a fork: ci-repo and reviewer-login'); const fork = probe({ REDLINE_CI_REPO: 'someone/ci', REDLINE_REVIEWER_LOGIN: 'Fork-Bot' }); const got = fork.status === 0 ? JSON.parse(fork.stdout) : null; check('in a fork every pinned workflow and action is the fork\'s, and the poster trusts the fork alone', - got !== null && got.paths.length === 9 && got.paths.every((p) => p.startsWith('someone/ci/')) && JSON.stringify(got.tooling) === '["someone/ci"]' && got.login === 'fork-bot'); + got !== null && got.paths.length === 10 && got.paths.every((p) => p.startsWith('someone/ci/')) && JSON.stringify(got.tooling) === '["someone/ci"]' && got.login === 'fork-bot'); const ours = probe({}); const today = ours.status === 0 ? JSON.parse(ours.stdout) : null; check('unset, it is today\'s: askalf/ci, sprayberry-redline, and askalf/askalf still accepted', today !== null && today.paths.every((p) => p.startsWith('askalf/ci/')) diff --git a/scripts/tree-tested/check.mjs b/scripts/tree-tested/check.mjs new file mode 100644 index 0000000..3ae97bf --- /dev/null +++ b/scripts/tree-tested/check.mjs @@ -0,0 +1,275 @@ +#!/usr/bin/env node +// Says whether the tree a push to the default branch carries was already tested green by this same +// workflow, on the pull request that merged it. The tree-tested action runs it as the first step of +// a job; the steps that build and test then take `if: steps..outputs.tested != 'true'`. +// +// Why. A pull request runs its workflow, merges, and the push to main runs the same workflow again +// on, most often, exactly the same files: a squash or merge of a branch that already had main in it +// leaves main with the branch head's tree. On the askalf runner host on 2026-11-11 that second run +// was about 300 of a day's CPU-heavy runner minutes (restore-desk, claude-re, platform, dario), on a +// host whose runner slice was waiting on CPU a quarter of the time. +// +// The caller's declaration. GitHub does not record what a job's steps did with their inputs, so the +// one thing this cannot read is whether a job runs the same commands on a push as on a pull request. +// The caller says so, with the action's same-on-pull-request input set to true, and with nothing set +// the answer is always false. It holds when the job's steps, their commands and their inputs do not +// depend on the event: no step or command that runs only on push, no matrix that a push widens, no +// checkout of anything but the event's own commit, and nothing after that checkout that moves the +// tests onto other files (a branch switch, a reset, a second checkout). What the rules below can +// check, they check; the rest is the caller's word. +// +// The rule, all of it read from the GitHub API with the job's own token: +// 0. The caller declared the job the same on push and on pull request (above). +// 1. The event is a push to the default branch. Anything else is tested=false. +// 2. A pull request of this repository was merged as this commit (its merge_commit_sha is the +// pushed sha). A direct push has none, and is tested=false. +// 3. The pull request's head commit has the same tree as the pushed commit: the same files, byte +// for byte, git's own hash of them. +// 4. The newest completed pull_request run of this workflow file at that head concluded success, +// on its latest attempt, and every job in it, and every step of every job, concluded success, +// all pages of them. A job or a step that was skipped did not test anything (a step that runs +// only on push, a job a change did not need), so a run with one is not taken. +// 5. This job, the one asking (found in this run by its runner's name), has a name no other job +// of this run has, and exactly one job of that run has it. A job the push runs and the pull +// request did not is not there, and two jobs under one display name (a matrix whose entries +// share a name) cannot be told apart, so neither is taken. +// 6. That job's log shows what its checkout made. It has exactly one actions/checkout step, and +// exactly one "HEAD is now at" line in the whole log, and that line is inside the checkout +// step and reads "Merge into ": GitHub's merge of the pull request. The head +// there is the pull request's head, and that base is inside the head by GitHub's compare. A +// merge of a base the head already holds is the head's own tree, so the checkout was exactly +// the pushed tree, whatever branch the pull request pointed at then and whatever main did +// after. A second visible "HEAD is now at" line (a later detached checkout or hard reset, or +// a test that prints one) is caught. Not everything that moves a checkout prints that line: +// a switch to a branch prints "Switched to branch", and a quiet checkout or reset prints +// nothing. That the tests ran on what the checkout made is the caller's declaration above, +// not something this log can show. +// Every read that fails, and every rule that does not hold, is tested=false: the job runs in full. +// Nothing here can turn a red run green; at worst it runs a test it could have skipped. +// +// CLI: GITHUB_REPOSITORY=owner/name GITHUB_EVENT_NAME=push GITHUB_REF=refs/heads/main GITHUB_SHA= \ +// GITHUB_WORKFLOW_REF=owner/name/.github/workflows/ci.yml@refs/heads/main DEFAULT_BRANCH=main \ +// GITHUB_TOKEN=... [GITHUB_OUTPUT=file] node check.mjs + +import { appendFileSync } from 'node:fs'; +import { pathToFileURL } from 'node:url'; + +const REPO_RE = /^[A-Za-z0-9-]+\/[A-Za-z0-9._-]+$/; +const SHA_RE = /^[0-9a-f]{40}$/; +/** owner/name/.github/workflows/@: the file this run's workflow came from. */ +const WORKFLOW_REF_RE = /^[A-Za-z0-9-]+\/[A-Za-z0-9._-]+\/\.github\/workflows\/([A-Za-z0-9._-]+\.ya?ml)@/; + +/** The facts the decision starts from, from the runner's environment. Pure. */ +export function factsOf(env) { + const text = (k) => String(env?.[k] ?? '').trim(); + return { + repo: text('GITHUB_REPOSITORY'), + event: text('GITHUB_EVENT_NAME'), + ref: text('GITHUB_REF'), + sha: text('GITHUB_SHA'), + workflowFile: WORKFLOW_REF_RE.exec(text('GITHUB_WORKFLOW_REF'))?.[1] ?? '', + defaultBranch: text('DEFAULT_BRANCH'), + runId: text('GITHUB_RUN_ID'), + runnerName: text('RUNNER_NAME'), + declared: text('SAME_ON_PULL_REQUEST'), + }; +} + +/** Why this run is not one to look up at all, or '' when it is. Pure. */ +export function notEligible(f) { + if (f.declared !== 'true') return 'the caller has not declared this job the same on a push as on a pull request (same-on-pull-request: true)'; + if (!REPO_RE.test(f.repo)) return `GITHUB_REPOSITORY is not owner/name (${JSON.stringify(f.repo)})`; + if (f.event !== 'push') return `a ${f.event || 'run with no event'} is tested in full`; + if (!f.defaultBranch || f.ref !== `refs/heads/${f.defaultBranch}`) return `a push to ${f.ref || 'no ref'} is tested in full; only a push to the default branch repeats a pull request's run`; + if (!SHA_RE.test(f.sha)) return 'the pushed commit is not a full sha'; + if (!f.workflowFile) return 'the workflow file could not be read from GITHUB_WORKFLOW_REF'; + if (!/^[1-9][0-9]*$/.test(f.runId)) return 'GITHUB_RUN_ID is not a run id'; + if (!f.runnerName) return 'RUNNER_NAME is not set, so this job cannot be found in its run'; + return ''; +} + +/** The pull request merged as this commit, from the commit's pull request list, or null. Pure. */ +export function mergedAs(pulls, repo, sha) { + return (Array.isArray(pulls) ? pulls : []).find((p) => p?.merged_at && p?.merge_commit_sha === sha + && p?.base?.repo?.full_name === repo && SHA_RE.test(String(p?.head?.sha ?? ''))) ?? null; +} + +/** + * The run that tested the head, from the workflow's runs at that head sha: the newest completed + * pull_request run for this pull request. Null when there is none. Pure. + */ +export function runAtHead(runs, { head, number }) { + return (Array.isArray(runs) ? runs : []) + .filter((r) => r?.event === 'pull_request' && r?.status === 'completed' && r?.head_sha === head + && (!Array.isArray(r?.pull_requests) || r.pull_requests.length === 0 || r.pull_requests.some((p) => p?.number === number))) + .sort((a, b) => Date.parse(b.created_at ?? 0) - Date.parse(a.created_at ?? 0))[0] ?? null; +} + +/** + * Why the run's jobs do not show this tree tested, or '' when every job, and every step of every job, + * concluded success. A job with no steps listed is not taken either: nothing shows what it ran. Pure. + */ +export function jobsProblem(jobs) { + const list = Array.isArray(jobs) ? jobs : []; + if (list.length === 0) return 'the run lists no jobs'; + const off = list.filter((j) => j?.conclusion !== 'success'); + if (off.length) return `not every job passed there (${off.slice(0, 4).map((j) => `${j?.name ?? '?'}: ${j?.conclusion ?? j?.status ?? '?'}`).join(', ')})`; + const bare = list.find((j) => !Array.isArray(j?.steps) || j.steps.length === 0); + if (bare) return `the job ${bare?.name ?? '?'} lists no steps`; + for (const j of list) { + const step = j.steps.find((st) => st?.conclusion !== 'success'); + if (step) return `not every step ran and passed there (${j?.name ?? '?'}: ${step?.name ?? '?'} ${step?.conclusion ?? step?.status ?? '?'})`; + } + return ''; +} + +/** Whether a compare of base...head says base is inside head (head is ahead of it, or the same). Pure. */ +export const contains = (compare) => compare?.status === 'ahead' || compare?.status === 'identical'; + +/** How many pages of jobs (100 a page) a run may have before it is not read at all. */ +export const MAX_JOB_PAGES = 5; +/** The largest job log read for its checkout line. */ +export const MAX_LOG_BYTES = 32 * 1024 * 1024; + +/** This job in its own run: the one in progress on this runner. Null when there is not exactly one. Pure. */ +export function thisJob(jobs, runnerName) { + const mine = (Array.isArray(jobs) ? jobs : []).filter((j) => j?.status === 'in_progress' && j?.runner_name === runnerName); + return mine.length === 1 ? mine[0] : null; +} + +const STEP_START = /##\[group\]Run (.*)$/; +const CHECKOUT_STEP = /^actions\/checkout@/; +const HEAD_LINE = /HEAD is now at [0-9a-f]{7,40} /; +const MERGE_LINE = /HEAD is now at [0-9a-f]{7,40} Merge ([0-9a-f]{40}) into ([0-9a-f]{40})\s*$/; +/** + * What a job's log shows its tests ran on: { head, base } of the pull request merge its one + * checkout made, or { why }. The log is cut into steps at each "##[group]Run " line; there + * must be exactly one actions/checkout step, exactly one "HEAD is now at" line in the whole log, + * and that line inside the checkout step, naming a merge. Pure. + */ +export function checkoutMerge(log) { + const lines = String(log ?? '').split(/\r?\n/); + let step = ''; + let checkouts = 0; + const heads = []; + for (const line of lines) { + const s = STEP_START.exec(line); + if (s) { step = s[1].trim(); if (CHECKOUT_STEP.test(step)) checkouts++; continue; } + if (HEAD_LINE.test(line)) heads.push({ line, step }); + } + if (checkouts !== 1) return { why: `the job has ${checkouts} actions/checkout steps, not one` }; + if (heads.length !== 1) return { why: `the log has ${heads.length} "HEAD is now at" lines, not one, so something after the checkout may have moved what the tests ran on` }; + if (!CHECKOUT_STEP.test(heads[0].step)) return { why: 'its one "HEAD is now at" line is not in the checkout step' }; + const m = MERGE_LINE.exec(heads[0].line); + if (!m) return { why: 'the checkout did not check out a pull request merge' }; + return { head: m[1], base: m[2] }; +} + +/** GET from the API as the token. Throws on a status that is not 2xx. */ +async function get(io, env, path) { + const api = (env.GITHUB_API_URL || 'https://api.github.com').replace(/\/+$/, ''); + const res = await io.fetch(`${api}${path}`, { + headers: { authorization: `Bearer ${env.GITHUB_TOKEN ?? ''}`, accept: 'application/vnd.github+json', 'x-github-api-version': '2022-11-28' }, + signal: AbortSignal.timeout(20_000), + }); + if (!res.ok) throw new Error(`GET ${path.split('?')[0]}: HTTP ${res.status}`); + return res.json(); +} + +/** A job's log as text; the API answers with a redirect to storage, which fetch follows. */ +async function getLog(io, env, repo, jobId) { + const api = (env.GITHUB_API_URL || 'https://api.github.com').replace(/\/+$/, ''); + const res = await io.fetch(`${api}/repos/${repo}/actions/jobs/${jobId}/logs`, { + headers: { authorization: `Bearer ${env.GITHUB_TOKEN ?? ''}`, accept: 'application/vnd.github+json', 'x-github-api-version': '2022-11-28' }, + signal: AbortSignal.timeout(60_000), + }); + if (!res.ok) throw new Error(`GET the log of job ${jobId}: HTTP ${res.status}`); + const text = await res.text(); + if (text.length > MAX_LOG_BYTES) throw new Error(`the log of job ${jobId} is over ${MAX_LOG_BYTES} bytes`); + return text; +} + +/** Every job of a run's latest attempt, all pages, or { why } when they cannot all be read. */ +async function allJobs(io, env, repo, runId) { + const jobs = []; + for (let page = 1; ; page++) { + if (page > MAX_JOB_PAGES) return { why: `run ${runId} has more than ${MAX_JOB_PAGES * 100} jobs; not read` }; + const got = await get(io, env, `/repos/${repo}/actions/runs/${runId}/jobs?filter=latest&per_page=100&page=${page}`); + const batch = Array.isArray(got?.jobs) ? got.jobs : []; + jobs.push(...batch); + const total = Number(got?.total_count); + if (batch.length < 100 || (Number.isFinite(total) && jobs.length >= total)) { + if (Number.isFinite(total) && jobs.length !== total) return { why: `read ${jobs.length} of run ${runId}'s ${total} jobs` }; + return { jobs }; + } + } +} + +/** + * The decision: { tested, why, run? }. tested is true only when every rule in the header holds. + * io: { fetch }. + */ +export async function decide(env, io = { fetch: globalThis.fetch }) { + const f = factsOf(env); + const skip = notEligible(f); + if (skip) return { tested: false, why: skip }; + try { + const pulls = await get(io, env, `/repos/${f.repo}/commits/${f.sha}/pulls`); + const pr = mergedAs(pulls, f.repo, f.sha); + if (!pr) return { tested: false, why: `no pull request of ${f.repo} was merged as ${f.sha.slice(0, 7)} (a direct push is tested in full)` }; + const head = pr.head.sha; + const [pushed, headed] = await Promise.all([ + get(io, env, `/repos/${f.repo}/git/commits/${f.sha}`), + get(io, env, `/repos/${f.repo}/git/commits/${head}`), + ]); + const tree = String(pushed?.tree?.sha ?? ''), headTree = String(headed?.tree?.sha ?? ''); + if (!SHA_RE.test(tree) || tree !== headTree) { + return { tested: false, why: `#${pr.number}'s head ${head.slice(0, 7)} has another tree than ${f.sha.slice(0, 7)}: main had moved, so the merged files were never tested together` }; + } + const listed = await get(io, env, `/repos/${f.repo}/actions/workflows/${f.workflowFile}/runs?head_sha=${head}&event=pull_request&per_page=50`); + const run = runAtHead(listed?.workflow_runs, { head, number: pr.number }); + if (!run) return { tested: false, why: `${f.workflowFile} has no finished pull_request run at #${pr.number}'s head ${head.slice(0, 7)}` }; + if (run.conclusion !== 'success') return { tested: false, why: `${f.workflowFile} ended ${run.conclusion} at #${pr.number}'s head (${run.html_url ?? run.id})` }; + const theirs = await allJobs(io, env, f.repo, run.id); + if (theirs.why) return { tested: false, why: theirs.why }; + const bad = jobsProblem(theirs.jobs); + if (bad) return { tested: false, why: `${bad} in ${run.html_url ?? run.id}` }; + // This job, and the job of the same name there. + const ours = await allJobs(io, env, f.repo, f.runId); + if (ours.why) return { tested: false, why: ours.why }; + const me = thisJob(ours.jobs, f.runnerName); + if (!me) return { tested: false, why: `this job was not found in run ${f.runId} by its runner ${f.runnerName}` }; + const namesakes = ours.jobs.filter((j) => j?.name === me.name).length; + if (namesakes !== 1) return { tested: false, why: `${namesakes} jobs of this run are named "${me.name}", so the one the pull request ran cannot be told apart` }; + const twins = theirs.jobs.filter((j) => j?.name === me.name); + if (twins.length !== 1) return { tested: false, why: `the job "${me.name}" is ${twins.length ? 'not one job' : 'not a job'} in ${run.html_url ?? run.id}: the pull request did not run it` }; + // What that job checked out. + const seen = checkoutMerge(await getLog(io, env, f.repo, twins[0].id)); + if (seen.why) return { tested: false, why: `in the log of "${me.name}" there, ${seen.why}; what it tested is not known` }; + const { head: tHead, base: tBase } = seen; + if (tHead !== head) return { tested: false, why: `"${me.name}" there checked out a merge of ${tHead.slice(0, 7)}, not of #${pr.number}'s head ${head.slice(0, 7)}` }; + const cmp = await get(io, env, `/repos/${f.repo}/compare/${tBase}...${head}`); + if (!contains(cmp)) { + return { tested: false, why: `"${me.name}" there tested #${pr.number}'s head merged into ${tBase.slice(0, 7)}, which the head does not hold (${cmp?.status ?? 'no status'}), so it tested another tree` }; + } + return { tested: true, why: `#${pr.number}'s head ${head.slice(0, 7)} has this tree, "${me.name}" tested it there (merged into ${tBase.slice(0, 7)}, which the head holds), and every job and step of that run passed`, run: run.html_url ?? String(run.id) }; + } catch (e) { + return { tested: false, why: `${e.message}; tested in full` }; + } +} + +export async function main(env, io = {}) { + const log = io.log ?? ((l) => console.log(l)); + const d = await decide(env, { fetch: io.fetch ?? globalThis.fetch }); + if (env.GITHUB_OUTPUT) appendFileSync(env.GITHUB_OUTPUT, `tested=${d.tested ? 'true' : 'false'}\nrun=${d.run ?? ''}\n`); + log(d.tested ? `::notice::this tree already passed: ${d.why} (${d.run}). The steps that test it are skipped.` : `tested in full: ${d.why}`); + return 0; +} + +if (import.meta.url === pathToFileURL(process.argv[1] ?? '').href) { + main(process.env).then((code) => { process.exitCode = code; }, (e) => { + // Even a failure of this script runs the job in full: it never fails the job. + console.log(`tested in full: ${e.message}`); + if (process.env.GITHUB_OUTPUT) appendFileSync(process.env.GITHUB_OUTPUT, 'tested=false\nrun=\n'); + }); +} diff --git a/scripts/tree-tested/check.test.mjs b/scripts/tree-tested/check.test.mjs new file mode 100644 index 0000000..2e88e93 --- /dev/null +++ b/scripts/tree-tested/check.test.mjs @@ -0,0 +1,211 @@ +// Tests for scripts/tree-tested/check.mjs. Run: node scripts/tree-tested/check.test.mjs +// GitHub is stubbed through io.fetch; nothing leaves the machine. + +import { mkdtempSync, readFileSync, rmSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { factsOf, notEligible, mergedAs, runAtHead, jobsProblem, contains, thisJob, checkoutMerge, decide, main } from './check.mjs'; + +let pass = 0; +let fail = 0; +function check(name, cond) { + if (cond) { console.log(` ok ${name}`); pass++; } + else { console.log(` FAIL ${name}`); fail++; } +} + +const REPO = 'askalf/demo'; +const SHA = 'a'.repeat(40); +const HEAD = 'b'.repeat(40); +const TREE = 'c'.repeat(40); +const OTHER = 'd'.repeat(40); +const BASE = 'e'.repeat(40); +const RELEASE = 'f'.repeat(40); +const RUN_ID = '555'; +const env = (o = {}) => ({ + GITHUB_REPOSITORY: REPO, GITHUB_EVENT_NAME: 'push', GITHUB_REF: 'refs/heads/main', GITHUB_SHA: SHA, + GITHUB_WORKFLOW_REF: `${REPO}/.github/workflows/ci.yml@refs/heads/main`, DEFAULT_BRANCH: 'main', GITHUB_TOKEN: 't', + GITHUB_RUN_ID: RUN_ID, RUNNER_NAME: 'pool-e3', SAME_ON_PULL_REQUEST: 'true', ...o, +}); +const pr = (o = {}) => ({ number: 7, merged_at: '2026-11-12T00:00:00Z', merge_commit_sha: SHA, base: { repo: { full_name: REPO } }, head: { sha: HEAD }, ...o }); +const run = (o = {}) => ({ id: 900, event: 'pull_request', status: 'completed', conclusion: 'success', head_sha: HEAD, created_at: '2026-11-11T23:00:00Z', html_url: 'https://x/run/900', pull_requests: [], ...o }); +const step = (name, conclusion = 'success') => ({ name, conclusion, status: 'completed' }); +let jobIds = 1000; +const job = (name, conclusion = 'success', steps = [step('Set up job'), step('Test'), step('Complete job')]) => ({ id: jobIds++, name, conclusion, status: 'completed', steps }); +const mine = (name, runner = 'pool-e3') => ({ id: 1, name, status: 'in_progress', runner_name: runner }); +// A job log as GitHub writes it: each step opens with "##[group]Run ". +const T = '2026-11-12T02:58:14.1195074Z '; +const checkoutStep = (head = HEAD, base = BASE) => [`${T}##[group]Run actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1`, `${T}with:`, `${T}##[endgroup]`, + `${T}##[group]Checking out the ref`, `${T}HEAD is now at 25a09da Merge ${head} into ${base}`, `${T}##[endgroup]`]; +const testStep = (out = 'tests pass') => [`${T}##[group]Run npm test`, `${T}npm test`, `${T}##[endgroup]`, `${T}${out}`]; +const checkoutLog = (head = HEAD, base = BASE) => [`${T}##[group]Run askalf/ci/actions/checkout-with-retry@${'1'.repeat(40)}`, `${T}##[endgroup]`, ...checkoutStep(head, base), ...testStep()].join('\n') + '\n'; + +/** A fake GitHub. routes: path (no query) -> body, an HTTP status, text, or a function of the url. */ +function world(over = {}) { + const prJobs = [job('unit'), job('e2e')]; + const routes = { + [`/repos/${REPO}/commits/${SHA}/pulls`]: [pr()], + [`/repos/${REPO}/git/commits/${SHA}`]: { tree: { sha: TREE } }, + [`/repos/${REPO}/git/commits/${HEAD}`]: { tree: { sha: TREE } }, + [`/repos/${REPO}/actions/workflows/ci.yml/runs`]: { workflow_runs: [run()] }, + [`/repos/${REPO}/actions/runs/900/jobs`]: { total_count: 2, jobs: prJobs }, + [`/repos/${REPO}/actions/runs/${RUN_ID}/jobs`]: { total_count: 2, jobs: [mine('unit'), { id: 2, name: 'e2e', status: 'queued' }] }, + [`/repos/${REPO}/compare/${BASE}...${HEAD}`]: { status: 'ahead' }, + ...over, + }; + if (!('LOG' in routes)) routes.LOG = { text: checkoutLog() }; + const calls = []; + const fetch = async (url) => { + const u = new URL(url); + calls.push(u.pathname + u.search); + let body = routes[u.pathname]; + if (body === undefined && /\/actions\/jobs\/\d+\/logs$/.test(u.pathname)) body = routes.LOG; + if (typeof body === 'function') body = body(u); + if (body === undefined) return { ok: false, status: 404, json: async () => ({}), text: async () => '' }; + if (typeof body === 'number') return { ok: body < 300, status: body, json: async () => ({}), text: async () => '' }; + if (body && typeof body.text === 'string') return { ok: true, status: 200, json: async () => ({}), text: async () => body.text }; + return { ok: true, status: 200, json: async () => body, text: async () => JSON.stringify(body) }; + }; + return { fetch, calls, prJobs }; +} + +console.log('\n the facts and who is looked up'); +{ + const f = factsOf(env()); + check('the workflow file, run id, runner and the caller\'s word are read from the environment', f.workflowFile === 'ci.yml' && f.sha === SHA && f.runId === RUN_ID && f.runnerName === 'pool-e3' && f.declared === 'true'); + check('without the caller\'s word that the job is the same on push, nothing is skipped', + ['', 'false', 'yes', 'TRUE'].every((v) => /has not declared this job the same/.test(notEligible(factsOf(env({ SAME_ON_PULL_REQUEST: v })))))); + check('a push to the default branch is eligible', notEligible(f) === ''); + check('a pull request run is tested in full', /tested in full/.test(notEligible(factsOf(env({ GITHUB_EVENT_NAME: 'pull_request' }))))); + check('a push to another branch is tested in full', /only a push to the default branch/.test(notEligible(factsOf(env({ GITHUB_REF: 'refs/heads/feature' }))))); + check('a schedule, a dispatch and a missing event are tested in full', + ['schedule', 'workflow_dispatch', ''].every((e) => notEligible(factsOf(env({ GITHUB_EVENT_NAME: e }))) !== '')); + check('no default branch, a short sha, a bad repository, no workflow ref, no run id or no runner is tested in full', + [{ DEFAULT_BRANCH: '' }, { GITHUB_SHA: 'abc' }, { GITHUB_REPOSITORY: 'nope' }, { GITHUB_WORKFLOW_REF: 'x' }, { GITHUB_RUN_ID: 'x' }, { RUNNER_NAME: '' }] + .every((o) => notEligible(factsOf(env(o))) !== '')); +} + +console.log('\n the pure rules'); +{ + check('the pull request merged as this commit is found', mergedAs([pr({ number: 3, merge_commit_sha: OTHER }), pr()], REPO, SHA)?.number === 7); + check('an open, unmerged or other repository\'s pull request is not', + mergedAs([pr({ merged_at: null }), pr({ base: { repo: { full_name: 'x/y' } } }), pr({ head: { sha: 'short' } })], REPO, SHA) === null && mergedAs(null, REPO, SHA) === null); + const older = run({ id: 1, created_at: '2026-11-11T20:00:00Z', conclusion: 'failure' }); + check('the newest finished pull_request run at the head is the one read', runAtHead([older, run()], { head: HEAD, number: 7 })?.id === 900); + check('a push run, a run in progress, another head and another pull request\'s run are not', + runAtHead([run({ event: 'push' }), run({ status: 'in_progress' }), run({ head_sha: OTHER }), run({ pull_requests: [{ number: 8 }] })], { head: HEAD, number: 7 }) === null); + check('every job and every step passed reads as tested', jobsProblem([job('a'), job('b')]) === ''); + check('a green job with a skipped step does not: a step that runs only on push did not run there', + /integration skipped/.test(jobsProblem([job('a', 'success', [step('Set up job'), step('integration', 'skipped'), step('Complete job')])]))); + check('a green job with no steps listed does not', /lists no steps/.test(jobsProblem([{ name: 'a', conclusion: 'success' }])) && /lists no steps/.test(jobsProblem([job('a', 'success', [])]))); + check('a skipped, failed or cancelled job, or no job at all, does not', + /e2e: skipped/.test(jobsProblem([job('a'), job('e2e', 'skipped')])) && jobsProblem([job('a', 'failure')]) !== '' && jobsProblem([job('a', 'cancelled')]) !== '' && jobsProblem([]) !== ''); + check('a base is inside the head when the head is ahead of it or the same', + contains({ status: 'ahead' }) && contains({ status: 'identical' }) && !contains({ status: 'behind' }) && !contains({ status: 'diverged' }) && !contains(null)); + check('this job is the one in progress on this runner, and only when there is one', + thisJob([mine('a', 'other'), mine('b'), { name: 'c', status: 'queued', runner_name: 'pool-e3' }], 'pool-e3')?.name === 'b' + && thisJob([mine('a'), mine('b')], 'pool-e3') === null && thisJob([], 'pool-e3') === null); + check('the merge the one checkout made is read from its step', JSON.stringify(checkoutMerge(checkoutLog())) === JSON.stringify({ head: HEAD, base: BASE })); + const lines = (...parts) => parts.flat().join('\n'); + // A merge checkout, then a step that checks out an older ordinary commit before the tests. + check('a later checkout of another commit, even by hand, is caught: two "HEAD is now at" lines', + /2 "HEAD is now at" lines/.test(checkoutMerge(lines(checkoutStep(), [`${T}##[group]Run git checkout ${OTHER}`, `${T}##[endgroup]`, `${T}HEAD is now at ddddddd an older commit`], testStep())).why)); + check('a second actions/checkout step is not taken', /2 actions\/checkout steps/.test(checkoutMerge(lines(checkoutStep(), checkoutStep(), testStep())).why)); + check('a merge line printed by a test, outside the checkout, is not taken', + /not in the checkout step/.test(checkoutMerge(lines([`${T}##[group]Run actions/checkout@${'2'.repeat(40)}`, `${T}##[endgroup]`], testStep(`HEAD is now at 25a09da Merge ${HEAD} into ${BASE}`))).why)); + check('a checkout of the head itself, not a merge, is not taken', + /did not check out a pull request merge/.test(checkoutMerge(lines([`${T}##[group]Run actions/checkout@${'2'.repeat(40)}`, `${T}HEAD is now at bbbbbbb the head commit`], testStep())).why)); + check('no checkout at all is not taken', /0 actions\/checkout steps/.test(checkoutMerge(lines(testStep())).why) && /0 actions\/checkout steps/.test(checkoutMerge('').why)); +} + +console.log('\n the decision against a stubbed GitHub'); +{ + const w = world(); + const d = await decide(env(), w); + check('same tree, every job and step green, this job there, and its checkout a merge of the head into a base it holds: tested', d.tested === true && d.run === 'https://x/run/900'); + check('it asks for the runs at the head, of pull_request, of this workflow file', + w.calls.some((c) => c.startsWith(`/repos/${REPO}/actions/workflows/ci.yml/runs?head_sha=${HEAD}&event=pull_request`))); + check('it reads both runs\' jobs, every page, the latest attempt', + w.calls.includes(`/repos/${REPO}/actions/runs/900/jobs?filter=latest&per_page=100&page=1`) && w.calls.includes(`/repos/${REPO}/actions/runs/${RUN_ID}/jobs?filter=latest&per_page=100&page=1`)); + check('it reads the log of the job of the same name there, and compares its base with the head', + w.calls.includes(`/repos/${REPO}/actions/jobs/${w.prJobs[0].id}/logs`) && w.calls.includes(`/repos/${REPO}/compare/${BASE}...${HEAD}`)); +} +for (const [name, over, re, envOver] of [ + ['a direct push (no merged pull request)', { [`/repos/${REPO}/commits/${SHA}/pulls`]: [] }, /direct push/], + ['main had moved: another tree', { [`/repos/${REPO}/git/commits/${HEAD}`]: { tree: { sha: OTHER } } }, /another tree/], + ['no run at the head', { [`/repos/${REPO}/actions/workflows/ci.yml/runs`]: { workflow_runs: [] } }, /no finished pull_request run/], + ['the run at the head failed', { [`/repos/${REPO}/actions/workflows/ci.yml/runs`]: { workflow_runs: [run({ conclusion: 'failure' })] } }, /ended failure/], + ['a job there was skipped', { [`/repos/${REPO}/actions/runs/900/jobs`]: { total_count: 2, jobs: [job('unit'), job('e2e ui', 'skipped')] } }, /e2e ui: skipped/], + ['a step there was skipped in a green job', { [`/repos/${REPO}/actions/runs/900/jobs`]: { total_count: 1, jobs: [job('unit', 'success', [step('Set up job'), step('Integration tests', 'skipped')])] } }, /Integration tests skipped/], + ['fewer jobs read than the run counts', { [`/repos/${REPO}/actions/runs/900/jobs`]: { total_count: 3, jobs: [job('unit'), job('e2e')] } }, /read 2 of run 900's 3 jobs/], + // A matrix that a push expands wider than a pull request: this job never ran there. + ['this job is a push-only matrix entry the pull request never ran', + { [`/repos/${REPO}/actions/runs/${RUN_ID}/jobs`]: { total_count: 1, jobs: [mine('test (windows)')] } }, /"test \(windows\)" is not a job in .*did not run it/], + ['this job is not found in its own run', { [`/repos/${REPO}/actions/runs/${RUN_ID}/jobs`]: { total_count: 1, jobs: [mine('unit', 'someone-else')] } }, /not found in run 555/], + ['this run\'s jobs cannot be read', { [`/repos/${REPO}/actions/runs/${RUN_ID}/jobs`]: 500 }, /HTTP 500/], + ['the log shows no checkout', { LOG: { text: 'no checkout line\n' } }, /0 actions\/checkout steps, not one; what it tested is not known/], + ['the log shows a merge checkout and then another checkout', { LOG: { text: checkoutLog() + checkoutLog(HEAD, RELEASE) } }, /2 actions\/checkout steps/], + // A matrix whose entries share one display name, wider on push: the job asking cannot be matched. + ['two jobs of this run share the display name of the job asking (a wider push matrix under name: test)', + { [`/repos/${REPO}/actions/runs/${RUN_ID}/jobs`]: { total_count: 2, jobs: [mine('unit'), { id: 3, name: 'unit', status: 'queued' }] } }, /2 jobs of this run are named "unit"/], + ['the caller has not declared the job the same on push', {}, /has not declared/, { SAME_ON_PULL_REQUEST: '' }], + ['the log shows a merge of another head', { LOG: { text: checkoutLog(OTHER, BASE) } }, /merge of ddddddd, not of #7's head/], + // Tested against a release branch, then retargeted to main with no new run: the base it merged + // is not in the head, so the job tested the head plus that branch, not the pushed tree. + ['the run tested a merge into a base the head does not hold (a retarget after the run)', + { LOG: { text: checkoutLog(HEAD, RELEASE) }, [`/repos/${REPO}/compare/${RELEASE}...${HEAD}`]: { status: 'diverged' } }, /merged into fffffff, which the head does not hold \(diverged\)/], + // Main gained a change and reverted it between the run and the merge: the run merged the head + // into main with that change, a base the head does not hold. + ['main changed and reverted after the run', { [`/repos/${REPO}/compare/${BASE}...${HEAD}`]: { status: 'diverged' } }, /does not hold/], + ['the log cannot be read', { LOG: 410 }, /HTTP 410/], + ['the compare cannot be read', { [`/repos/${REPO}/compare/${BASE}...${HEAD}`]: 500 }, /HTTP 500/], + ['the pull request list cannot be read', { [`/repos/${REPO}/commits/${SHA}/pulls`]: 500 }, /HTTP 500; tested in full/], + ['a commit cannot be read', { [`/repos/${REPO}/git/commits/${HEAD}`]: 502 }, /HTTP 502/], + ['a commit with no tree', { [`/repos/${REPO}/git/commits/${SHA}`]: {} }, /another tree/], +]) { + const d = await decide(env(envOver ?? {}), world(over)); + check(`${name}: tested in full`, d.tested === false && re.test(d.why)); +} +{ + const same = await decide(env(), world({ LOG: { text: checkoutLog(HEAD, HEAD) }, [`/repos/${REPO}/compare/${HEAD}...${HEAD}`]: { status: 'identical' } })); + check('a base that is the head itself counts as held', same.tested === true); + const w2 = world(); + const p = await decide(env({ GITHUB_EVENT_NAME: 'pull_request' }), w2); + check('a pull request run makes no request at all', p.tested === false && w2.calls.length === 0); + const thrown = await decide(env(), { fetch: async () => { throw new Error('network down'); } }); + check('a fetch that throws is tested in full', thrown.tested === false && /network down; tested in full/.test(thrown.why)); +} + +console.log('\n every page of jobs is read'); +{ + const page1 = Array.from({ length: 100 }, (_, i) => job(`shard ${i}`)); + const jobsRoute = (rest) => (u) => (u.searchParams.get('page') === '1' ? { total_count: 100 + rest.length, jobs: page1 } : { total_count: 100 + rest.length, jobs: rest }); + const skipped = await decide(env(), world({ [`/repos/${REPO}/actions/runs/900/jobs`]: jobsRoute([job('last', 'skipped')]) })); + check('a skipped job past the first hundred is found: tested in full', skipped.tested === false && /last: skipped/.test(skipped.why)); + const w = world({ + [`/repos/${REPO}/actions/runs/900/jobs`]: jobsRoute([job('last')]), + [`/repos/${REPO}/actions/runs/${RUN_ID}/jobs`]: { total_count: 1, jobs: [mine('shard 0')] }, + LOG: { text: checkoutLog() }, + }); + const green = await decide(env(), w); + check('a hundred and one green jobs, read over two pages: tested', green.tested === true && w.calls.some((c) => c.endsWith('&page=2'))); + const endless = await decide(env(), world({ [`/repos/${REPO}/actions/runs/900/jobs`]: () => ({ jobs: page1 }) })); + check('a run that never ends its pages is not read past the cap', endless.tested === false && /more than 500 jobs/.test(endless.why)); +} + +console.log('\n main: the step output and the line it prints'); +{ + const dir = mkdtempSync(join(tmpdir(), 'tree-tested-')); + const out = join(dir, 'out'); + const lines = []; + const code = await main({ ...env(), GITHUB_OUTPUT: out }, { fetch: world().fetch, log: (l) => lines.push(l) }); + check('tested: the output says so and names the run, and the step passes', + code === 0 && readFileSync(out, 'utf8') === 'tested=true\nrun=https://x/run/900\n' && /^::notice::this tree already passed/.test(lines[0])); + const out2 = join(dir, 'out2'); + const code2 = await main({ ...env({ GITHUB_EVENT_NAME: 'pull_request' }), GITHUB_OUTPUT: out2 }, { fetch: world().fetch, log: (l) => lines.push(l) }); + check('not tested: tested=false, an empty run, and the step still passes', + code2 === 0 && readFileSync(out2, 'utf8') === 'tested=false\nrun=\n' && /^tested in full: a pull_request/.test(lines[1])); + rmSync(dir, { recursive: true, force: true }); +} + +console.log(`\n ${pass} passed, ${fail} failed`); +process.exit(fail ? 1 : 0); diff --git a/test-all.mjs b/test-all.mjs index a1f1659..ad1f07d 100644 --- a/test-all.mjs +++ b/test-all.mjs @@ -12,6 +12,7 @@ const RUNS = [ ['scripts/fleet-status/fleet-status.test.mjs'], ['scripts/fleet-status/check.test.mjs'], ['scripts/scorecard-publish/check.test.mjs'], + ['scripts/tree-tested/check.test.mjs'], ['scripts/redline/pin-bump.test.mjs'], // Every fuzz seed through its target, and the helpers that keep a broken contract apart from a refusal. ['fuzz/seeds.test.mjs'],