Skip to content

fix: an autopilot pull request whose checks fail is never fixed - #744

Closed
dilux-bot[bot] wants to merge 9 commits into
mainfrom
fix/741-an-autopilot-pull-request-whose
Closed

dilux-bot[bot] wants to merge 9 commits into
mainfrom
fix/741-an-autopilot-pull-request-whose

Conversation

@dilux-bot

@dilux-bot dilux-bot Bot commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

📝 What changes

The PRD passes G1: the problem is clear and C1 to C5 can each be tested from the pipelines' YAML and the scripts' pure functions.

  1. Trigger (C1, C2, C5). Each pipeline's autopilot-fix gets needs: on every job the review waits on (pull-request.yml: [conventions, actionlint, scripts, review]; org-pull-request.yml: [conventions, review]). Its if: keeps every guard it has now: always(), pull_request, not edited, the autopilot label, dilux-bot as author, and the repository clause. Only the trigger part becomes needs.review.outputs.blocking == 'true' || needs.review.result == 'failure' || needs.<gate>.result == 'failure' (one clause for each gate). That covers a gate that failed (the review is then skipped and has no outputs) and a review that ended without a verdict, whether its verdict step said so (blocking=true, "did not finish") or the job died before that step (fix: the bug reproduced from #720 #723: no outputs). The two conditions stay the same apart from the repository and the gate list, and the existing fix: the autopilot answers a review in the organisation's repository #504 test checks this.

  2. Fix mode reads the failed jobs' log (C1, C2). No new input: org-pull-request.yml calls autopilot.yml@v7, and the queue round (ci: test before the pull request, a faster queue, an unstuck autopilot #479) avoided a new input for the same reason. In fix mode, autopilot.py gets a checks-failed command and inputs fix gets a matching branch. They read the jobs of the calling run (GITHUB_RUN_ID; a reusable workflow's jobs belong to it). Two cases make a check round: a failed job other than the review's, or a failed review job with no verdict at the head (the bot's dx-review summary says "did not finish", or its record is not at the head SHA). The failed jobs' gh run view --job <id> --log-failed goes into .dx-io/checks.md, scrubbed and cut like queue.md: queue_log_text is generalised and keeps the queue's text. prompt_fix(..., checks=True) points the agent at that file instead of review.md, and the round writes no review ids. A review that blocks with a verdict keeps today's review round. A round that fails as the round before did still stops on the existing dx-prepr-sig.

  3. Cap (C3). The rounds check in "May it start" stays where it is (ROUND > autopilot-rounds → needs-human, a comment, and the issue opened to claims). Its comment now comes from a pure rounds_spent(max, checks) in autopilot.py, so it says "its checks still fail" on a check round instead of always "the review still blocks". next_step gets a checks_failed state, so the sweep and the board's next step agree with the trigger: one round below the cap, the person at the cap.

  4. Board (C4). waiting.py blocker: an autopilot pull request whose failed checks include at least one the pipeline fixes (check-run names conventions /, review /, actionlint, Scripts (self-tests)) waits on "the autopilot's fix round", with rounds left. Rounds left means the highest autopilot:round-N label is below autopilot-rounds, read from policy/autonomy.yml. At the cap, or when only other checks failed (a repository's own workflow, which nothing re-runs), it waits on the maintainer, with the action to take it over (needs-human already comes first). The board then never tells the autopilot to "fix the failed check".

Options a maintainer may care about:

  • (A, recommended, within scope) The round starts only on the jobs of the organisation's pipeline (conventions, the review and, here, actionlint and scripts). On the board, a repository's own failing workflow goes to the maintainer.
  • (B) Also start rounds on a repository's own required workflows, such as a WordPress plugin's suites. That needs a new workflow_run/check_suite trigger outside the two pipelines, which is beyond the PRD's scope, so it is listed as a finding instead.

What this widens: a fix round (the agent's job, with the Claude subscription and a read-only token) now also starts on a failed gate or a review without a verdict. It stays within the same autopilot-rounds and runs caps. No new permission or secret is needed: reading the run's jobs needs actions: read, which autopilot.yml's prep job already holds or will be given, and the pull request says which. In repositories other than this one it takes effect once v7 is moved by a release.

💡 Why

Closes #741. The autopilot works it here in the open: one commit per criterion, each box ticked once its commit is pushed.

🎯 Acceptance criteria

  • C1 Given an autopilot pull request whose scripts job fails, when its checks complete, then a fix round starts with the failed jobs' log, without a person.
    Test: scripts/autopilot.py (--test: pull-request.yml's autopilot-fix needs the gates and fires on needs.scripts.result == 'failure'; checks-failed/checks_log_text select the failed jobs and their log; prompt_fix(checks=True) reads .dx-io/checks.md)
  • C2 Given an autopilot pull request whose review ends without a verdict, when its checks complete, then a fix round starts the same way.
    Test: scripts/autopilot.py (--test: the condition fires on needs.review.result == 'failure' and on a blocking review; a review job that failed without a verdict at the head is a check round, one with a verdict a review round)
  • C3 Given an autopilot pull request whose rounds reached autopilot-rounds, when a check fails again, then no round starts and it is handed to a person with needs-human.
    Test: scripts/autopilot.py (--test: next_step with checks_failed past autopilot-rounds waits on a person; rounds_spent text; the fix branch of "May it start" adds needs-human above the cap)
  • C4 Given either state, when the board is written, then it names the fix round that is running, or the person, never a fix nothing will start.
    Test: scripts/waiting.py (--test with scripts/fixtures/waiting.json: an autopilot pull request whose scripts or review check failed with rounds left waits on the autopilot's fix round; at the cap, or with only a repository's own check failed, it waits on the maintainer; never "fix the failed check" for the autopilot)
  • C5 Given a repository of the organisation other than DiluxOne/.github, when the same happens there (org-pull-request.yml), then C1 to C4 hold too.
    Test: scripts/autopilot.py (--test: org-pull-request.yml's autopilot-fix carries the same trigger as pull-request.yml's apart from the repository and its gate list, which is conventions and review)

🧭 Decisions

  • 2026-11-14: The run's jobs and their logs are read in prep with dilux-bot's App token, which already reads Actions for the queue round (#​479): no job's permissions change. A job's log is gh run view --job --log-failed, falling back to the job's whole log from the API (its end is what the agent reads), because the calling run is still running while the f​ix round is one of its jobs.
  • 2026-11-14: When the run's jobs cannot be read, no round starts: the pull request gets needs-human and a comment, so the board never names a round nothing runs.
  • 2026-11-14: A draft gets no f​ix round from a failed gate (May it start stops it) and its board says the autopilot's work: the review never runs on a draft, so no round ran beside a draft's own run before, and its make pre-pr already makes the checks pass before it is ready.
  • 2026-11-14: Credits: nobody. The issue's author, soydiloreto, is org.yml's only maintainer, who accepts and merges here, and the rule names no co-author among them; the only commenter is dilux-bot. A maintainer corrects it with /coauthor.

🙌 Credits

🧪 How I tested it

python3 scripts/autopilot.py --test and python3 scripts/waiting.py --test pass, with the new checks for C1 to C5: the two pipelines' autopilot-fix trigger and needs read from their YAML, checks_failed on job lists (a failed gate, a review that failed with or without a verdict at the head, other workflows', cancelled and running jobs), the log text cut and scrubbed, the check round's prompt, next_step and rounds_spent at and below the cap, "May it start"'s fix branch run in bash with gh and the check answer stubbed (below the cap, at it, unread jobs, a draft), and the board on the new fixtures 15 to 19 in scripts/fixtures/waiting.json. The tests under tests/ pass with pytest (268 passed); test_autopilot_rounds.py and test_queue_drop_fix.py now expect the gate's checks= output. I ran every step of pull-request.yml's scripts job, and actionlint 1.7.12 is clean. review-brief.sh and dx-runner-slot.sh failed only with this run's PR and DX_BASE variables set, and pass without them. The repository has no make pre-pr. Nothing ran against GitHub: reading a real run's jobs and logs (checks-failed, job_log) is tested only through stubs.

🤖 AI-generated · Claude Opus 5.5 (Anthropic)

The draft pull request opens before any code: the Spec as its checklist.
@dilux-bot dilux-bot Bot added autopilot Opened by the autopilot from an accepted issue autopilot:run-1 The autopilot's runs on this pull request labels Nov 14, 2026
@dilux-bot

dilux-bot Bot commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor Author

The pull request's board

State
Waits on nobody

🤖 AI-generated · Claude Opus 5.5 (Anthropic)

@dilux-bot dilux-bot Bot mentioned this pull request Nov 14, 2026
dilux-bot Bot added 6 commits November 14, 2026 03:40
An autopilot pull request whose conventions, actionlint or scripts job fails gets a fix round with the failed jobs' log in .dx-io/checks.md, read from the calling run's jobs by the new checks-failed command, instead of waiting for a person nobody told.

Assisted-by: Claude Opus 5.5 (Anthropic)
The pipeline's fix round also starts when the review's job fails, and checks-failed counts that job as a check round when no verdict of the review is at the head (its did-not-finish record carries no sha, a job that died before its verdict left none), so the agent reads its log; a review that blocks with a verdict keeps the review's round.

Assisted-by: Claude Opus 5.5 (Anthropic)
At autopilot-rounds a check round starts no more: May it start adds needs-human and comments from the new rounds_spent, which says the checks still fail rather than that the review blocks. next_step reads a checks_failed state the same way: one round below the cap, a person at it.

Assisted-by: Claude Opus 5.5 (Anthropic)
An autopilot pull request whose failed checks include one the pipeline fixes waits on the autopilot's fix round while its highest autopilot:round-N is below policy/autonomy.yml's autopilot-rounds; at the cap, or with only a repository's own check failed, it waits on the maintainer with the action to take it over; an autopilot draft's failed check is its own run's work. No board tells the autopilot to fix the failed check any more. autopilot.py reads the same check names from waiting.py, and no fix round runs beside a draft's own run.

Assisted-by: Claude Opus 5.5 (Anthropic)
The rounds and the queue tests of May it start stub checks-failed and rounds-spent, expect its checks output, and let autopilot-fix wait on the conventions beside the review, as C1 to C3 made them.

Assisted-by: Claude Opus 5.5 (Anthropic)
org-pull-request.yml's autopilot-fix waits on the conventions and the review and starts on the same trigger as this repository's, its gate list the conventions alone: a failed conventions job or a review with no verdict gets its round, its cap and its board in every repository once v7 moves. docs/workflows.md and the pipelines' comments say so.

Assisted-by: Claude Opus 5.5 (Anthropic)
@dilux-bot
dilux-bot Bot marked this pull request as ready for review November 14, 2026 03:41
@dilux-bot
dilux-bot Bot requested a review from soydiloreto as a code owner November 14, 2026 03:41
@dilux-bot dilux-bot Bot added risk:high Set by the Claude review complexity:high Set by the Claude review type:fix The kind of change, read from the diff by the Claude review labels Nov 14, 2026
@dilux-bot

dilux-bot Bot commented Nov 14, 2026

Copy link
Copy Markdown
Contributor Author

Claude review · risk high · complexity high · type fix · blocking

1 to fix (1 major)

First review. The description says next_step gets a checks_failed state "so the sweep and the board's next step agree with the trigger". Nothing sets checks_failed in production. The state JSON built in autopilot.yml's what step and sweep_step never carry it, so only the tests exercise that branch. Nothing here looks like a new capability needing version:major. Credits are empty and match the issue, with no model listed. Minor points: (1) run_checks_failed reads the run's jobs with per_page=100 and no --paginate, so a larger run would silently drop jobs and could look like "no check failed". (2) docs/configuration.md (autopilot-rounds: "Review rounds … on a blocked pull request") and docs/pull-requests.md:141 still describe rounds as review-only. (3) A review job that fails after a non-blocking verdict at the head (for example a later step erroring) yields no check round and no blocking review, so the review round runs with an empty review.md and spends a round. (4) job_log's fallback pulls the whole log of the review job, which is mostly agent transcript noise; it is cut to 300 lines, but the useful part may fall outside that window.

✍️ The description does not match the code (see the summary): no auto-merge until it does.

  • major scripts/autopilot.py:2007: A capped review (stale blocking verdict) is misread as a review with no verdict

Policy floor: high (touches high-risk paths: .github/workflows/autopilot.yml, .github/workflows/org-pull-request.yml, .github/workflows/pull-request.yml, scripts/autopilot.py, scripts/fixtures/waiting.json …). Reviewed 3234a4b (whole pull request; review 1 of 5 automatic). Author trusted for auto-merge: false.

🤖 AI review · claude-sonnet-5-5 (Anthropic) · $0.32, 9 turns

Comment thread scripts/autopilot.py
#723). Empty: a review round, as before, on the verdict at the head.
`record` is the review's last dx-review-record, `head` the pull
request's head. Pure."""
verdict = isinstance(record, dict) and bool(head) and record.get("sha") == head

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

major: A capped review (stale blocking verdict) is misread as a review with no verdict

checks_failed treats a failed review / … job as "no verdict" unless the record's sha equals the head. In claude-review.yml the capped mode (max-auto-reviews reached) reuses the previous record, which keeps the older sha and is still blocking. It then does exit 1, so the review job fails with a valid blocking verdict that is simply not at this head. The record check fails, so the job counts as a check round. The agent then reads only the review job's log ("The last review found a blocking problem; see the summary comment") instead of the review's threads in review.md. Before this PR the same state ran the review round with the threads. At the cap the comment also says "its checks still fail" instead of "the review still blocks". It is reachable wherever max-auto-reviews is below autopilot-rounds, and after round 5 with the defaults (6th push is capped). Fix: count a review job as "no verdict" only when the record is missing or has no verdict fields. The "did not finish" record written by the workflow carries only count and findings, with no risk, model_risk or blocking, while a reused or capped record has them. Add a --test fixture for a record with an older sha and blocking: true (expected: empty, review round) next to the existing "did not finish" fixture.

🤖 AI review · claude-sonnet-5-5 (Anthropic)

@dilux-bot dilux-bot Bot added the needs-human The autopilot handed this over to a person label Nov 14, 2026
@dilux-bot

dilux-bot Bot commented Nov 14, 2026

Copy link
Copy Markdown
Contributor Author

The autopilot could not read which checks failed (), so no fix round starts: a person takes it from here.

🤖 AI-generated · Claude Opus 5.5 (Anthropic)

@soydiloreto

Copy link
Copy Markdown
Member

No check failed here: what blocks is the review (#744 (comment) from 03:48, one major). Please fix the major at scripts/autopilot.py:2007 (a capped review with a stale blocking verdict is misread), make the production state actually carry checks_failed (autopilot.yml's what step and sweep_step) or drop it from the description, and take the cheap minors: --paginate in run_checks_failed, and docs/configuration.md and docs/pull-requests.md:141 saying rounds also start from a failed check.

🤖 AI-assisted · Claude Opus 5.5 (Anthropic)

@dilux-bot

dilux-bot Bot commented Nov 14, 2026

Copy link
Copy Markdown
Contributor Author

@soydiloreto: a person takes it from here (needs-human): read why and decide #744

🤖 AI-generated · Claude Opus 5.5 (Anthropic)

@soydiloreto

Copy link
Copy Markdown
Member

Answering the stop: what blocks is the review's one major, not a failed check. Please fix the major at scripts/autopilot.py:2007 (a capped review with a stale blocking verdict is misread as a fresh one), make the production state carry checks_failed (autopilot.yml's what step and sweep_step) or drop it from the description, and take the cheap minors: --paginate in run_checks_failed, and docs/configuration.md and docs/pull-requests.md:141 saying rounds also start from a failed check.

🤖 AI-assisted · Claude Opus 5.5 (Anthropic)

@soydiloreto

Copy link
Copy Markdown
Member

Closing: this pull request's own new fix trigger stopped it ("could not read which checks failed ()"), so no round can bring it back. #741 starts again with the review's findings written on the issue.

🤖 AI-assisted · Claude Opus 5.5 (Anthropic)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

autopilot:run-1 The autopilot's runs on this pull request autopilot Opened by the autopilot from an accepted issue complexity:high Set by the Claude review needs-human The autopilot handed this over to a person risk:high Set by the Claude review type:fix The kind of change, read from the diff by the Claude review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix: an autopilot pull request whose checks fail is never fixed

1 participant