Skip to content

fix: the release lane tests the pull-request path at v7, not v7-next - #749

Merged
soydiloreto merged 12 commits into
mainfrom
fix/748-the-release-lane-tests-the
Nov 14, 2026
Merged

soydiloreto merged 12 commits into
mainfrom
fix/748-the-release-lane-tests-the

Conversation

@dilux-bot

@dilux-bot dilux-bot Bot commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

📝 What changes

Today the required workflow is the workflows rule of the organisation ruleset "org: main". That rule runs org-pull-request.yml at v7 in every adopted repository, the test repository too, and its inner calls are hard-wired to @v7. So in the test repository a scenario pull request's review, auto-merge decision and fix rounds always run v7, whatever the lane moves. The fix has three parts. First, the test repository stops getting the required workflow. Second, it runs the same pipeline from a caller of its own, which the lane's point_callers already moves to @v7-next and back like any other caller there. Third, the lane refuses to play when a pipeline there would still run v7.

  1. Ruleset (C2), scripts/sync-repos.py rulesets. GitHub takes only one repository condition per org ruleset (name, id or property), so the test repository cannot be dropped by name from a ruleset that targets by property. Recommended: option A.

    • Option A (recommended). Move the workflows rule out of "org: main" into a new ruleset, "org: required workflow". It is created first and copies "org: main"'s workflows rule unchanged (the same org-pull-request.yml, at the same v7 ref), its bypass actors, its target and its ref_name (~DEFAULT_BRANCH). Only then is the rule removed from "org: main", so no repository is ever left without the required workflow. Its repository_property condition includes dx-adopted=true and excludes a second custom property of the organisation: test-property in org.yml, dx-engine-test by default, a true_false property the sync creates and sets to true on org.yml test-repo alone. "org: main" keeps its other rules on the test repository: pull request, deletion and signed commits. The PUT body always carries both ref_name and repository_property ("complete conditions": with_property_condition and the new builder are tested for it).
      • The exclusion is written only once the test repository has its pull-request caller (item 2) on its default branch. Until then the sync says so and leaves the test repository covered, because otherwise its pull requests would run no pipeline. For that short time both pipelines run there at v7.
      • ORG_RULESET_NAMES, by_property and covers learn the new ruleset and its exclusion.
      • sync-repos.py check (the daily drift check) flags four things: a workflows rule still in "org: main", the required-workflow ruleset covering the test repository, a ruleset with no ref_name, and a required-workflow ref other than v7.
      • This widens the organisation's settings by one custom property and one ruleset. The pull request will say so.
    • Option B. Same split, but the new ruleset targets by repository_name: it includes every repository repos.yml adopts and excludes test-repo. It needs no new property, but it brings back the list of names that feat: adopting a repository is data and one command #562 removed, and every adoption has to extend it.
    • Option C. No split: exclude the test repository from "org: main" as a whole, and give it repository rulesets that copy "org: main"'s other rules. This means more rulesets to keep in step, and superseded() would have to stop deleting them.
  2. Caller (C3). A new template, workflow-templates/test-pull-request.yml (plus its .properties.json). Its on: and its jobs are org-pull-request.yml's, job for job: the same ids and uses:, so the same check names. It adds one job, pipeline, which needs every other job, runs if: always() on both pull_request and merge_group, and fails when any of them failed or was cancelled.

    • Why pipeline: a required workflow has no named checks; it requires the whole run to succeed. pipeline is that run's result as one check, and it reports in the merge queue too. Requiring each job's check instead would block the queue for good, because the jobs skip on merge_group under another name.
    • The pack (kinds/engine-test/pack.yml callers) lists the new template. new_callers also writes a kind's pack callers, so sync-repos.py callers writes it to the test repository and holds it job for job.
    • The sync writes, in the test repository only, a repository ruleset "required pipeline" that requires the pipeline check on the default branch. That gives the test repository the same gate the required workflow gives everywhere else.
    • The lane's move needs no new code: point_callers rewrites every engine call under .github/workflows/, so the new caller moves to @v7-next and back with the others (shown by a test).
    • The template sits in workflow-templates/ with the other shipped callers, so the organisation's template picker also offers it to other repositories. Its description says it belongs to the test repository alone.
  3. The check (C4), scripts/release-scenarios.py. test_callers() reads the pack's callers (now including test-pull-request.yml) in place of org-pull-request.yml. The pairs are unchanged, so the scenarios' expectations are untouched. Two new checks, each naming the file or rule it refuses:

    • problems (static, coverage) refuses three things: a pack with no caller that wires the pipeline's pull_request and merge_group jobs, org-pull-request.yml among the test repository's callers, and an engine call in a test caller that at_ref would not move (one that does not match USES).
    • move (run time, before any group plays) refuses two things, each as a failing callers row: the test repository's default-branch rules (repos/{repo}/rules/branches/{default}, readable with its token) still carrying a workflows rule for org-pull-request.yml, and any .github/workflows/*.yml there still calling the engine off the ref after the move.
  4. Docs (C5). docs/versioning.md (Scenarios step) and docs/workflows.md (the org-pull-request.yml and release-lane rows) say that the test repository runs the candidate's pipeline from test-pull-request.yml at @v7-next, and why: the required workflow is read at v7 with its calls hard-wired. To stay in step, the same goes into docs/adopting.md (the ruleset list), kinds/README.md (pack callers are now written), the pack's comments, and the header comments of org-pull-request.yml and release-lane.yml.

💡 Why

Closes #748. The autopilot works it here in the open: one commit per criterion, each box ticked once its commit is pushed.

🎯 Acceptance criteria

  • C1 Given the release lane playing a candidate, when a scenario pull request in the test repository runs its pipeline, then its review, auto-merge decision and fix rounds check out the candidate's engine (v7-next), not v7.
    Test: scripts/release-scenarios.py (--test: after point_callers, every engine call of test-pull-request.yml's review, auto-merge and autopilot-fix jobs is at @v7-next), tests/workflows/test_test_repo_pipeline.py
  • C2 Given the organisation's rulesets after the sync, when they are read, then the required-workflow ruleset covers every adopted repository except org.yml test-repo, its conditions are complete (ref and repository conditions), and every other repository keeps the required workflow at v7.
    Test: scripts/sync-repos.py (--test: the split ruleset's PUT/POST bodies carry ref_name and repository_property with the test property excluded, the same v7 workflows rule, "org: main" kept for the test repository, created before the rule is removed, the drift check's four findings)
  • C3 Given the test repository, when its callers are synced, then it has a pull-request caller with the same required checks the required workflow gives, and the lane moves it to @v7-next and back with the others.
    Test: tests/workflows/test_test_repo_pipeline.py (the template's jobs and on: equal org-pull-request.yml's plus pipeline), scripts/sync-repos.py (--test: new_callers writes it to the engine-test repository only, and "required pipeline" requires pipeline), scripts/release-scenarios.py (--test: moved to @v7-next and put back)
  • C4 Given a release plan in which a pipeline of the test repository would not run at the candidate, when release-scenarios.py checks the plan, then it refuses it and says which one.
    Test: scripts/release-scenarios.py (--test: problems names a missing pull-request caller, org-pull-request.yml among the callers, an unmovable call; move names a remaining workflows rule and a caller off the ref)
  • C5 Given docs/versioning.md and docs/workflows.md, when a reader looks for how the lane tests the pull-request path, then they say the test repository runs the candidate's pipeline and why.
    Test: tests/workflows/test_test_repo_pipeline.py (docs/versioning.md and docs/workflows.md name test-pull-request.yml at @v7-next and say that the required workflow runs at v7)

🧭 Decisions

  • 2026-11-14: The ruleset follows the Spec's option A, the one it recommends and whose files it lists (org.yml test-property, org.py test_property): a new org ruleset 'org: required workflow' by property, excluding the test property.
  • 2026-11-14: A template a kind's pack lists in callers that every-repository does not (test-pull-request.yml) belongs to that kind alone (sync-repos.py kinds_only): its caller is only the file of its own name, so the drift check never compares it with another repository's workflows or with the test repository's issues.yml.
  • 2026-11-14: The run-time check (release-scenarios.py at_candidate) runs in move once the callers moved, so it reads the test repository's workflows as they will play; a refusal is a failing callers row, no group plays, and run puts back what moved (move leaves it to back, which already reads the record).
  • 2026-11-14: "required pipeline" is written by sync-repos.py rulesets, in the same step as the exclusion and just before it, and only once the test repository has test-pull-request.yml on its default branch: before that, its required check would never report and would block every merge there.

🙌 Credits

🧪 How I tested it

  • python3 scripts/sync-repos.py --test: the split against a stand-in for gh ("org: required workflow" created first with "org: main"'s workflows rule at v7, its ref_name and repository_property with dx-engine-test=true excluded, then the rule taken out of "org: main"; the test repository left covered until it has test-pull-request.yml, then excluded on a later run; "required pipeline" requiring pipeline written before the exclusion; a ruleset with no ref_name never written; the drift check's four findings), new_callers writing test-pull-request.yml to the engine-test repository only, and the template compared nowhere else.
  • python3 scripts/release-scenarios.py --test: the shipped template moved by point_callers (review, auto-merge and autopilot-fix at @v7-next) and put back; pipeline_problems naming a pack with no pull-request caller, org-pull-request.yml among the callers and an unmovable call; move refusing a remaining workflows rule and a caller off the ref, playing nothing and putting the callers back. coverage passes on the repository (24 scenarios).
  • python3 scripts/org.py --test, python3 scripts/install.py --test, python3 scripts/caller-contract.py check, python3 scripts/settings.py check.
  • pytest on tests/ (273 passed, 1 skipped), with the new tests/workflows/test_test_repo_pipeline.py: the template is org-pull-request.yml's on: and jobs plus pipeline, whose step fails only on a failed or cancelled job (run in bash), and the docs say what runs at @v7-next and why.
  • Every run: step of pull-request.yml's scripts jobs, run locally: all pass but plugin-i18n.sh (no gettext here), review-brief.sh (no pull request number) and runner-vm/dx-runner-slot.sh (the runner VM's paths), which fail the same way on the base commit.

Not tested live: nothing here touched GitHub. Before the next release lane run, a maintainer runs sync-repos.py callers --repo diluxone-test (merge its pull request) and then sync-repos.py rulesets; until then move refuses to play, because the test repository still requires org-pull-request.yml.

🤖 AI-generated · Claude Opus 5.5 (Anthropic)

The draft pull request opens before any code: the Spec as its checklist.
@dilux-bot dilux-bot Bot added autopilot Opened by the autopilot from an accepted issue autopilot:run-1 The autopilot's runs on this pull request labels Nov 14, 2026
@dilux-bot

dilux-bot Bot commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor Author

The pull request's board

State
Waits on nobody

🤖 AI-generated · Claude Opus 5.5 (Anthropic)

@dilux-bot dilux-bot Bot mentioned this pull request Nov 14, 2026
dilux-bot Bot added 7 commits November 14, 2026 05:04
workflow-templates/test-pull-request.yml is the required workflow's on: and jobs, job for job, plus a pipeline job that is the whole run's result as one check. The engine-test pack lists it, so the release lane's point_callers moves its review, auto-merge and fix-round calls to @v7-next with the other callers of the test repository and back after. A template a pack lists that every-repository does not belongs to that kind alone: the sync compares it with the file of its own name in a repository of that kind, and nowhere else.

Assisted-by: Claude Opus 5.5 (Anthropic)
sync-repos.py rulesets moves the workflows rule of "org: main" into a ruleset of its own, "org: required workflow", created first with the same org-pull-request.yml at v7, target, bypass actors and ref_name, and only then takes the rule out of "org: main", which keeps its other rules on every repository, the test repository included. The new ruleset covers the repositories with dx-adopted=true and excludes those with org.yml test-property (dx-engine-test), which the sync creates and sets to true on org.yml test-repo alone, once that repository has its own pull-request caller on its default branch. A ruleset body whose conditions lack ref_name or repository_property is never written, and the daily drift check names a workflows rule left in "org: main", the required workflow covering the test repository, a ruleset with no ref_name and a required workflow at another ref than the current major.

Assisted-by: Claude Opus 5.5 (Anthropic)
new_callers also writes the callers a kind's pack lists that every repository does not have, so sync-repos.py callers writes test-pull-request.yml to the engine-test repository alone and holds it there job for job; no other template takes that file for its caller. Once the test repository has the caller on its default branch, sync-repos.py rulesets writes there a ruleset of its own, "required pipeline", that requires the caller's pipeline check on the default branch, before the required workflow leaves it: the gate the required workflow gives everywhere else. The template's tests hold its on: and jobs to org-pull-request.yml's plus pipeline, and the release lane's self-test moves it to @v7-next and puts it back.

Assisted-by: Claude Opus 5.5 (Anthropic)
org.yml test-property, which C2 adds, is a setting: policy/settings-schema.yml lists it, and docs/configuration.md is written again from it, so settings.py check passes.

Assisted-by: Claude Opus 5.5 (Anthropic)
release-scenarios.py reads the test repository's callers from its pack alone, test-pull-request.yml among them, in place of org-pull-request.yml; the pairs are the same, so the scenarios are unchanged. coverage refuses a pack with no caller that wires the pull-request pipeline on pull_request and merge_group, org-pull-request.yml among its callers, and an engine call the lane cannot move, each named. move, once the callers moved, refuses as failing callers rows the test repository's default branch still requiring org-pull-request.yml and any workflow there still calling the engine off the ref, naming the rule or the file and line; no group plays, and what moved is put back.

Assisted-by: Claude Opus 5.5 (Anthropic)
docs/versioning.md (the Scenarios step) and docs/workflows.md (the org-pull-request.yml and release-lane.yml rows) say that the test repository runs its pull-request pipeline from test-pull-request.yml at @v7-next, and why: the required workflow is read at v7 with its calls hard-wired to @v7, so it could never try a candidate there. docs/adopting.md lists the new ruleset, kinds/README.md says the sync now writes the pack's callers, and the pack's comments and the header comments of org-pull-request.yml and release-lane.yml say the same. A test holds the two docs to it.

Assisted-by: Claude Opus 5.5 (Anthropic)
install.py writes the engine-test pack's callers to a new test repository, and the pack now lists test-pull-request.yml as well: its self-test counts the pack's callers instead of a fixed two, and checks that the pull-request pipeline is among them.

Assisted-by: Claude Opus 5.5 (Anthropic)
@dilux-bot
dilux-bot Bot marked this pull request as ready for review November 14, 2026 05:05
@dilux-bot
dilux-bot Bot requested a review from soydiloreto as a code owner November 14, 2026 05:05
@dilux-bot dilux-bot Bot added the autopilot:free-2 The autopilot's runs on this pull request that do not count towards its cap label Nov 14, 2026
…ane-tests-the

# Conflicts:
#	docs/workflows.md
@dilux-bot dilux-bot Bot added the autopilot:round-1 The autopilot's review rounds on this pull request label Nov 14, 2026
main's #746 changed org-pull-request.yml's autopilot-fix (needs the conventions too, fires on a failed gate or a review without a verdict, never on a draft) after test-pull-request.yml copied it, so the template no longer matched the required workflow job for job and tests/workflows/test_test_repo_pipeline.py failed the scripts job. The template's autopilot-fix now matches the required workflow's again.

Assisted-by: Claude Opus 5.5 (Anthropic)
@dilux-bot dilux-bot Bot added the autopilot:round-2 The autopilot's review rounds on this pull request label Nov 14, 2026
The Scripts job failed at #749's head: org_drift now calls current_major(), which lists the tags through gh, and the C4 (#739) self-test stood in for every other call to GitHub but that one, so on a runner with no token the self-test reached the API and failed. org_drift reads the rulesets once and asks for the major only when there is a ruleset to hold against it; the self-test refuses gh outright, so a call it does not stand in for fails locally too, and a new case checks that the org's rulesets are held against the current major.

Refs #748

Assisted-by: Claude Opus 5.5 (Anthropic)
@dilux-bot dilux-bot Bot added risk:high Set by the Claude review complexity:high Set by the Claude review type:fix The kind of change, read from the diff by the Claude review labels Nov 14, 2026
@dilux-bot

dilux-bot Bot commented Nov 14, 2026

Copy link
Copy Markdown
Contributor Author

Claude review · risk high · complexity high · type fix

Nothing that must be fixed · 5 minor

First review, so there are no earlier commits to credit. I read the diff only and did not run the tests: the sandbox refused the --test commands, so my reading is static, and the description's pass claims are unverified.

The design holds up on a close read:

  • The ruleset split creates "org: required workflow" before it takes the rule out of "org: main".
  • The test repository is left covered until its own caller exists on the default branch.
  • The test-pull-request.yml pipeline job fails only on a failed or cancelled job.
  • move fails closed with a "could not tell" row when the rules or the files cannot be read, and run_all puts the callers back.

I found no blocker or major problem, only the minor points below.

The description matches the diff. It says plainly that nothing was tested live and that a maintainer must run sync-repos.py callers and then rulesets before the next lane run. Until then the lane refuses to play, by design. This is not a new capability, so no version:major suggestion.

  • minor docs/adopting.md:104: The rewritten sentence now attaches 'which sync-repos.py rulesets writes' to the wrong subject
  • minor workflow-templates/test-pull-request.yml:1: The test repository's gate is now a file in the repository, not an org-pinned required workflow
  • minor scripts/sync-repos.py:3016: The drift check now calls repo_kind for DiluxOne/.github itself
  • minor scripts/sync-repos.py:4697: Stray formatting change: block =f"... lost its space
  • minor scripts/sync-repos.py:1366: kinds_only re-reads every pack from disk on each call

Policy floor: high (touches high-risk paths: .github/workflows/org-pull-request.yml, .github/workflows/release-lane.yml, kinds/README.md, kinds/engine-test/pack.yml, org.yml …). Reviewed 74c6b77 (whole pull request; review 1 of 5 automatic). Author trusted for auto-merge: false.

🤖 AI review · claude-sonnet-5-5 (Anthropic) · $0.71, 21 turns

@soydiloreto
soydiloreto merged commit 4622926 into main Nov 14, 2026
33 checks passed
@soydiloreto
soydiloreto deleted the fix/748-the-release-lane-tests-the branch November 14, 2026 05:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

autopilot:free-2 The autopilot's runs on this pull request that do not count towards its cap autopilot:round-1 The autopilot's review rounds on this pull request autopilot:round-2 The autopilot's review rounds on this pull request autopilot:run-1 The autopilot's runs on this pull request autopilot Opened by the autopilot from an accepted issue complexity:high Set by the Claude review risk:high Set by the Claude review type:fix The kind of change, read from the diff by the Claude review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix: the release lane tests the pull-request path at v7, not v7-next

1 participant