Skip to content

fix: an autopilot pull request whose checks fail is never fixed - #746

Merged
soydiloreto merged 9 commits into
mainfrom
fix/741-an-autopilot-pull-request-whose-43555621171
Nov 14, 2026
Merged

soydiloreto merged 9 commits into
mainfrom
fix/741-an-autopilot-pull-request-whose-43555621171

Conversation

@dilux-bot

@dilux-bot dilux-bot Bot commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

📝 What changes

The PRD passes G1: the problem is clear and C1 to C5 can each be tested from the pipelines' YAML and the scripts' pure functions.

  1. Trigger (C1, C2, C5). Each pipeline's autopilot-fix gets needs: on every job the review waits on (pull-request.yml: [conventions, actionlint, scripts, review]; org-pull-request.yml: [conventions, review]). Its if: keeps every guard it has now: always(), pull_request, not edited, the autopilot label, dilux-bot as author, and the repository clause. Only the trigger part becomes needs.review.outputs.blocking == 'true' || needs.review.result == 'failure' || needs.<gate>.result == 'failure' (one clause for each gate). That covers a gate that failed (the review is then skipped and has no outputs) and a review that ended without a verdict, whether its verdict step said so (blocking=true, "did not finish") or the job died before that step (fix: the bug reproduced from #720 #723: no outputs). The two conditions stay the same apart from the repository and the gate list, and the existing fix: the autopilot answers a review in the organisation's repository #504 test checks this.

  2. Fix mode reads the failed jobs' log (C1, C2). No new input: org-pull-request.yml calls autopilot.yml@v7, and the queue round (ci: test before the pull request, a faster queue, an unstuck autopilot #479) avoided a new input for the same reason. In fix mode, autopilot.py gets a checks-failed command and inputs fix gets a matching branch. They read the jobs of the calling run (GITHUB_RUN_ID; a reusable workflow's jobs belong to it). Two cases make a check round: a failed job other than the review's, or a failed review job with no verdict at the head (the bot's dx-review summary says "did not finish", or its record is not at the head SHA). The failed jobs' gh run view --job <id> --log-failed goes into .dx-io/checks.md, scrubbed and cut like queue.md: queue_log_text is generalised and keeps the queue's text. prompt_fix(..., checks=True) points the agent at that file instead of review.md, and the round writes no review ids. A review that blocks with a verdict keeps today's review round. A round that fails as the round before did still stops on the existing dx-prepr-sig.

  3. Cap (C3). The rounds check in "May it start" stays where it is (ROUND > autopilot-rounds → needs-human, a comment, and the issue opened to claims). Its comment now comes from a pure rounds_spent(max, checks) in autopilot.py, so it says "its checks still fail" on a check round instead of always "the review still blocks". next_step gets a checks_failed state, so the sweep and the board's next step agree with the trigger: one round below the cap, the person at the cap.

  4. Board (C4). waiting.py blocker: an autopilot pull request whose failed checks include at least one the pipeline fixes (check-run names conventions /, review /, actionlint, Scripts (self-tests)) waits on "the autopilot's fix round", with rounds left. Rounds left means the highest autopilot:round-N label is below autopilot-rounds, read from policy/autonomy.yml. At the cap, or when only other checks failed (a repository's own workflow, which nothing re-runs), it waits on the maintainer, with the action to take it over (needs-human already comes first). The board then never tells the autopilot to "fix the failed check".

Options a maintainer may care about:

  • (A, recommended, within scope) The round starts only on the jobs of the organisation's pipeline (conventions, the review and, here, actionlint and scripts). On the board, a repository's own failing workflow goes to the maintainer.
  • (B) Also start rounds on a repository's own required workflows, such as a WordPress plugin's suites. That needs a new workflow_run/check_suite trigger outside the two pipelines, which is beyond the PRD's scope, so it is listed as a finding instead.

What this widens: a fix round (the agent's job, with the Claude subscription and a read-only token) now also starts on a failed gate or a review without a verdict. It stays within the same autopilot-rounds and runs caps. No new permission or secret is needed: reading the run's jobs needs actions: read, which autopilot.yml's prep job already holds or will be given, and the pull request says which. In repositories other than this one it takes effect once v7 is moved by a release.

💡 Why

Closes #741. The autopilot works it here in the open: one commit per criterion, each box ticked once its commit is pushed.

🎯 Acceptance criteria

  • C1 Given an autopilot pull request whose scripts job fails, when its checks complete, then a fix round starts with the failed jobs' log, without a person.
    Test: scripts/autopilot.py (--test: pull-request.yml's autopilot-fix needs the gates and fires on needs.scripts.result == 'failure'; checks-failed/checks_log_text select the failed jobs and their log; prompt_fix(checks=True) reads .dx-io/checks.md)
  • C2 Given an autopilot pull request whose review ends without a verdict, when its checks complete, then a fix round starts the same way.
    Test: scripts/autopilot.py (--test: the condition fires on needs.review.result == 'failure' and on a blocking review; a review job that failed without a verdict at the head is a check round, one with a verdict a review round)
  • C3 Given an autopilot pull request whose rounds reached autopilot-rounds, when a check fails again, then no round starts and it is handed to a person with needs-human.
    Test: scripts/autopilot.py (--test: next_step with checks_failed past autopilot-rounds waits on a person; rounds_spent text; the fix branch of "May it start" adds needs-human above the cap)
  • C4 Given either state, when the board is written, then it names the fix round that is running, or the person, never a fix nothing will start.
    Test: scripts/waiting.py (--test with scripts/fixtures/waiting.json: an autopilot pull request whose scripts or review check failed with rounds left waits on the autopilot's fix round; at the cap, or with only a repository's own check failed, it waits on the maintainer; never "fix the failed check" for the autopilot)
  • C5 Given a repository of the organisation other than DiluxOne/.github, when the same happens there (org-pull-request.yml), then C1 to C4 hold too.
    Test: scripts/autopilot.py (--test: org-pull-request.yml's autopilot-fix carries the same trigger as pull-request.yml's apart from the repository and its gate list, which is conventions and review)

🧭 Decisions

  • 2026-11-14: autopilot-f​ix also requires a ready pull request (!github.event.pull_request.draft): a gate can fail on a draft while its work run is still going, and the review, the only trigger before, never ran on a draft, so nothing that ran before stops running.
  • 2026-11-14: A check round reads only the jobs of the calling run named as the pipeline's gates (conventions /, review /, actionlint, Scripts (self-tests)), the same names the board uses, so a waiting or auto-merge job that fails never starts one; when the run's jobs cannot be read, the round reads the review as before.
  • 2026-11-14: No permission is added: the jobs of the calling run are read with prep's App token, which already holds actions: read (permission-actions: read), the token the queue round reads its merge group's runs with.
  • 2026-11-14: On the board, an autopilot draft whose check failed waits on the autopilot's work, not a f​ix round (a draft gets none), and actionlint and Scripts (self-tests) count as checks a round f​ixes only in DiluxOne/.github, whose pipeline runs them.

🙌 Credits

🧪 How I tested it

  • python3 scripts/autopilot.py --test: the new checks for each criterion pass (fix: an autopilot pull request whose checks fail is never fixed #741 C1 to C5). They cover the trigger read from both pipelines' YAML, checks_failed, review_verdict and checks_log_text, and inputs fix and prompt fix run with gh stood in (a failed scripts job writes .dx-io/checks.md; a review that died gives its job's log; a review that blocks with its verdict is a review round, as before). They also cover next_step with checks_failed, rounds_spent, and the fix branch of "May it start" run as a script: past the cap it adds needs-human and says the checks still fail. The existing fix: the autopilot answers a review in the organisation's repository #504 and ci: test before the pull request, a faster queue, an unstuck autopilot #479 checks pass too.
  • python3 scripts/waiting.py --test with the new fixtures in scripts/fixtures/waiting.json: with rounds left the board names the autopilot's fix round. At the cap, or when only a repository's own check failed, it names the maintainer. A draft waits on its own work. No board tells the autopilot to "fix the failed check".
  • I ran every step of pull-request.yml's scripts job locally, plus the pytest suite under tests/ (268 passed; I updated test_autopilot_rounds.py and test_queue_drop_fix.py for the new needs and the comment from rounds-spent) and actionlint 1.7.12. Two steps fail here, and they fail the same way on the base commit: review-brief.sh --test, because of a pull request number in this environment, and runner-vm/dx-runner-slot.sh --test, because there is no docker.
  • This repository has no make pre-pr.

🤖 AI-generated · Claude Opus 5.5 (Anthropic)

The draft pull request opens before any code: the Spec as its checklist.
@dilux-bot dilux-bot Bot added autopilot Opened by the autopilot from an accepted issue autopilot:run-1 The autopilot's runs on this pull request labels Nov 14, 2026
@dilux-bot

dilux-bot Bot commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor Author

The pull request's board

State
Waits on nobody

🤖 AI-generated · Claude Opus 5.5 (Anthropic)

@dilux-bot dilux-bot Bot mentioned this pull request Nov 14, 2026
dilux-bot Bot added 7 commits November 14, 2026 04:38
An autopilot pull request whose conventions, actionlint or scripts job fails now gets a fix round, as one whose review blocks does: pull-request.yml's autopilot-fix waits for every gate the review waits on and starts when one of them failed, on a ready pull request. In fix mode, `inputs fix` reads the jobs of the calling run, and when a gate of the pipeline failed it writes their failed log to .dx-io/checks.md (scrubbed and cut as the queue's log, through the shared failed_log_text) instead of the review, and `prompt fix` points the agent at it. `autopilot.py checks-failed` says whether a round is a check round.

Assisted-by: Claude Opus 5.5 (Anthropic)
A review that ended without a verdict, whether its verdict step said "did not finish" or its job died before that step and left no outputs, now starts the same check round: pull-request.yml's autopilot-fix also fires on `needs.review.result == 'failure'`, and a failed review job whose bot summary has no record at the head (none, "did not finish", an older head's) is read as a check round, with the review job's log. A review that blocks with its verdict at the head stays a review round, from its threads.

Assisted-by: Claude Opus 5.5 (Anthropic)
A check that fails again once the rounds reached `autopilot-rounds` starts no round: "May it start" hands the pull request to a person with `needs-human`, as for a review round, and its comment now comes from `autopilot.py rounds-spent`, which says its checks still fail on a check round (and on the merge queue's) instead of always that the review still blocks. `next_step` gets a `checks_failed` state, one round below the cap and a person at it, so the sweep and the board agree with the trigger.

Assisted-by: Claude Opus 5.5 (Anthropic)
The board of an autopilot pull request whose check failed no longer tells the autopilot to "fix the failed check", which nothing would start. When a check the pipeline fixes failed (the conventions, the review, actionlint, the scripts) and its highest `autopilot:round-N` is below policy/autonomy.yml's `autopilot-rounds`, it waits on the autopilot's fix round. At the cap, or when only a check no round runs again failed (a repository's own workflow), it waits on the maintainer, with the action to read it and take it over. The fixtures carry each case.

Assisted-by: Claude Opus 5.5 (Anthropic)
org-pull-request.yml's autopilot-fix now carries the trigger pull-request.yml's has, apart from the repository and its gate list: it waits for the conventions and the review, and starts on a ready autopilot pull request whose review blocks, whose review failed before its verdict, or whose conventions failed. The self-test checks that the two conditions keep the same guards and the same trigger but for each pipeline's gates. docs/workflows.md says what starts a round, a check round from the failed jobs' log, and what the board says.

Assisted-by: Claude Opus 5.5 (Anthropic)
The board of an autopilot draft whose check failed waits on the autopilot's work, since a draft gets no fix round (its own run checks it), and actionlint and the scripts count as checks a round fixes only in the central repository, whose pipeline runs them, so a repository's own job of the same name goes to the maintainer. The fixtures name each pull request's repository.

Assisted-by: Claude Opus 5.5 (Anthropic)
docs/pull-requests.md says that the autopilot's fix rounds start on a gate that failed or a review that ended without a verdict, from the failed jobs' log, as they do on a review that blocks.

Assisted-by: Claude Opus 5.5 (Anthropic)
@dilux-bot
dilux-bot Bot marked this pull request as ready for review November 14, 2026 04:39
@dilux-bot
dilux-bot Bot requested a review from soydiloreto as a code owner November 14, 2026 04:39
@dilux-bot dilux-bot Bot added risk:high Set by the Claude review complexity:high Set by the Claude review type:fix The kind of change, read from the diff by the Claude review labels Nov 14, 2026
@dilux-bot

dilux-bot Bot commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor Author

Claude review · risk high · complexity high · type fix

Nothing that must be fixed · 5 minor

Incremental review: the new commit (abbc9d0, "chore: ready for review") changes nothing against c50db52, so none of the earlier findings were fixed or newly introduced. I could not run the self-tests in this session (the shell refused them), so the criteria rest on the tests I found in the code and on the description's report. The description's Decisions still hide zero-width characters inside "fix" (autopilot-f​ix, f​ix round), which makes the text hard to grep. checks_failed is read in next_step (scripts/autopilot.py:1078) and set only in tests (grep shows no production caller). The sweep's pr state never sets it, so the sweep and the board's next step can disagree with the workflow trigger. The rest of the earlier minors stand: the new trigger fires in other repositories only once v7 moves, the board reads the rounds cap from the central policy file only, and unreadable jobs at the cap give the "review still blocks" comment.

✍️ The description does not match the code (see the summary): no auto-merge until it does.

  • minor .github/workflows/org-pull-request.yml:162: New trigger runs before autopilot.yml@v7 has check-round support
  • minor scripts/autopilot.py:1078: checks_failed is read by next_step but no production code sets it
  • minor scripts/waiting.py:1449: Board reads the rounds cap from the central policy file only
  • minor .github/workflows/autopilot.yml:621: Unreadable jobs at the cap give the 'review still blocks' comment
  • minor scripts/autopilot.py:1: Description has zero-width characters inside 'fix' in its Decisions

Policy floor: high (touches high-risk paths: .github/workflows/autopilot.yml, .github/workflows/org-pull-request.yml, .github/workflows/pull-request.yml, scripts/autopilot.py, scripts/fixtures/waiting.json …). Reviewed abbc9d0 (since c50db52; review 2 of 5 automatic). Author trusted for auto-merge: false.

🤖 AI review · claude-sonnet-5-5 (Anthropic) · $0.15, 8 turns

@dilux-bot

dilux-bot Bot commented Nov 14, 2026

Copy link
Copy Markdown
Contributor Author

@soydiloreto: review it and merge it #746

🤖 AI-generated · Claude Opus 5.5 (Anthropic)

@soydiloreto
soydiloreto merged commit 545af68 into main Nov 14, 2026
44 of 46 checks passed
@soydiloreto
soydiloreto deleted the fix/741-an-autopilot-pull-request-whose-43555621171 branch November 14, 2026 05:06
dilux-bot Bot added a commit that referenced this pull request Nov 14, 2026
main's #746 changed org-pull-request.yml's autopilot-fix (needs the conventions too, fires on a failed gate or a review without a verdict, never on a draft) after test-pull-request.yml copied it, so the template no longer matched the required workflow job for job and tests/workflows/test_test_repo_pipeline.py failed the scripts job. The template's autopilot-fix now matches the required workflow's again.

Assisted-by: Claude Opus 5.5 (Anthropic)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

autopilot:run-1 The autopilot's runs on this pull request autopilot Opened by the autopilot from an accepted issue complexity:high Set by the Claude review risk:high Set by the Claude review type:fix The kind of change, read from the diff by the Claude review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix: an autopilot pull request whose checks fail is never fixed

1 participant