Skip to content

fix(agents): accept errors an earlier look showed, and mark killed reviews - #219

Open
cjber wants to merge 5 commits into
mainfrom
cb/bugpatrol-reports
Open

cjber wants to merge 5 commits into
mainfrom
cb/bugpatrol-reports

Conversation

@cjber

@cjber cjber commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Two fixes to how Bugpatrol reads what happened in a run.

flowchart LR
  A[look shows console error] --> B[next look drains it]
  B --> C{report_bug error}
  C -->|before: last observation only| D[refused]
  C -->|now: errors seen since the report's steps began| E[accepted]
Loading

report_bug accepts an error an earlier look showed

report_bug checked its error only against the console and network errors of the last observation. A web observation drains the errors it reports, so a look after a crash returned an empty list and the explorer's error was refused even though it had just been shown. The session now keeps every error seen since the start of what a report replays, and a report checks its error against all of them. They are cleared only where that replay starts over: a replayed routine, a claim flow, or a change of control of the app (AgentSession.startFlow). Saving a routine continues the flow, since the routine holds the steps before it.

A killed review no longer reads as running

A review stopped by SIGKILL left pr-N.json at status: running for good. The review report now records the pid of its process, and the dashboard reader shows a running review whose process is gone as failed with Interrupted. Stopped: its process ended., matching how it already treats sessions. The next run of the review overwrites the record as before.

Verification

  • pnpm gate:changed (biome, tsc, vitest for changed files): 41 files, 373 tests pass.
  • New tests fail without the fix: agents-flow.test.ts (error shown, then a look and a saved routine, then a report with that error is accepted; after a new flow it is refused) and dashboard/src/agents.test.ts (running review with a dead pid reads as failed; a live pid stays running). review.test.ts checks the pid is recorded.

Closes #217
Closes #218

🤖 Generated with Claude Code

cjber and others added 2 commits November 19, 2026 22:25
report_bug matched its error only against the console and network errors of
the last observation. A web observation drains the errors it reports, so a
look after the crash left that list empty and an error the explorer had just
been shown was refused. The session now keeps every error it saw with the
trail position, and a report checks against those seen since the start of
its steps, which is what the replay of those steps sees.

Closes #217

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A review killed mid-run (SIGKILL) left its report at status running with no
way to tell it from a live one. The report now records the pid of the
process that runs it, and the dashboard reader shows a running review whose
process ended as failed and interrupted, as it already does for sessions.

Closes #218

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vercel

vercel Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated
bugpatrol Skipped Skipped Nov 19, 2026 11:57pm UTC

Request Review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-19T22:29:04.160174Z 9235f1b PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

Bugpatrol review: fixture-app

No problem introduced · tested 81e41d4 against main 2c57911

Tested

  • Home page (index.html) loads
  • Settings page (settings) loads

Not tested

  • The rest of the change (agents session, explorer tools, dashboard reader, review pid): outside the fixture app

Updated on each push. Tests only what the diff can affect. Does not block the merge.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9235f1b085

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread packages/agents/src/tools/explorer.ts Outdated
...(redactedObservation.networkErrors ?? []),
]);
// The replay of the report's steps sees every error from the start of those steps.
const errors = session.errorsSeen.filter((seen) => seen.step >= session.anchor.index).map((seen) => seen.text);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve errors from the newly saved anchor routine

When an action produces a console or network error and the explorer calls save_routine before report_bug, capture() records the error at the trail length before that action is appended, while saveRoutine() advances anchor.index to the length after it. This filter consequently drops the error and rejects the report, even though the candidate's routineId would replay the saved action and reproduce it, silently losing a valid finding.

AGENTS.md reference: AGENTS.md:L17-L17

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Accepted, fixed in 7697fea. The session no longer filters errors by trail position. It keeps every error since the start of what a report replays, and clears them only where that replay starts over: run_routine, a claim flow, or a control change (new AgentSession.startFlow). save_routine continues the flow, since the saved routine holds the earlier steps. The test in agents-flow.test.ts now saves a routine between the error and the report (accepted), then starts a new flow (refused).

@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

Bugpatrol review: dashboard

No problem introduced · tested 81e41d4 against main 2c57911

Next run: The explorer used all 150 steps. Raise agents.explorer.maxSteps, or the Action's steps input.

Tested

Not tested

  • Dead-pid 'running' review reading as failed: Fixture has no running review record; the dashboard reader change is only reachable with one
  • report_bug error-acceptance change in agents/session.ts: Explorer tool logic runs in agent runs, not rendered by the dashboard; outside what this app can show
  • The rest of the change (roles/claims.ts, tools/explorer.ts, core types): outside the dashboard

Updated on each push. Tests only what the diff can affect. Does not block the merge.

@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

Bugpatrol review: cli

No problem introduced · tested 81e41d4 against main 2c57911

Next run: The explorer used all 150 steps. Raise agents.explorer.maxSteps, or the Action's steps input.

Tested

  • bugpatrol --help/--version and issue list on the fixture workspace
  • bugpatrol replay fixture-open in /tmp/qa219 (replayed ok, exit 0)
  • replay open-home against a locally started fixture app (after installing the playwright browser)

Not tested

  • report_bug accepting an error an earlier look showed (session.errorsSeen flow): needs an explorer agent session with report_bug; the CLI commands I may run do not expose it
  • killed review reads as failed via pid in the dashboard reader: review/dashboard commands are off-limits per the app guide
  • review records its pid in pr-N.json: the review command is banned; verified only via the unit-test diff

Updated on each push. Tests only what the diff can affect. Does not block the merge.

The error check filtered by trail position, but an action's error is
recorded before the action joins the trail, and save_routine moves the
anchor past it. A saved routine holds those steps, so its replay still sees
the error. The session now clears its errors only where a report's replay
starts over: a replayed routine, a claim flow or a control change
(startFlow), and a report checks against all errors seen since.

Refs #217

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A report checks its error against every error seen since its flow
started, but a saved routine drops wait steps and detours. An error
shown on such a step was accepted though the replayed routine never
reproduces it. Each seen error now keeps the trail step that showed
it, saveRoutine records the steps compactSteps leaves out, and
report_bug accepts only errors on steps the replay still takes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A failed action adds no trail step, so its observation's errors are
not part of any replay. Saving over the routine a flow continues
replaces it, so errors from before the slice no longer replay. An error
seen before a flow's first step (a routine replay) belongs to no old
trail step. Saving now filters the seen errors to what the new routine
replays instead of marking left-out steps.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@cjber

cjber commented Nov 20, 2026

Copy link
Copy Markdown
Contributor Author

Review record

Head: 81e41d4

Codex (gpt-reserve, xhigh) review result and outcomes:

  • packages/agents/src/session.ts: errorsSeen kept errors from steps that compactSteps later drops from the saved routine (wait steps, detours), so report_bug could accept an error the replayed routine never reproduces. Fixed in 4b87a3b: each seen error keeps the trail step that showed it, and saveRoutine filters the errors through compactSteps' own result. Test: "accepts only errors on steps that the saved routine keeps".
  • Re-review of 4b87a3b, P1: saving over the routine a flow continues kept errors from the replaced routine. Fixed in 81e41d4.
  • Re-review of 4b87a3b, P1: errors seen after run_routine were attributed to the last old trail step, which could be a dropped one. Fixed in 81e41d4: an error seen before a flow's first step belongs to no step.
  • Re-review of 4b87a3b, P2: a failed action's errors were attributed to the step before it and accepted. Fixed in 81e41d4: a failed action drops its observation's errors. Test: "drops the errors of a failed action and of a routine that a save replaces".
  • Re-review of 81e41d4, P2: an action interrupted by a control change could keep its first observation's errors. Rejected: every controlVersion bump calls Control.changed, which emits control-change (packages/drivers/src/cua/control.ts), and AgentSession clears errorsSeen and the trail on that event.

Fast checks: biome, tsc --build, vitest agents-flow.test.ts and api-evidence.test.ts pass. CI green at this head.

Muse review not run: subscription quota exhausted until 2026-11-23T00:00:00Z (API error 429); a reviewer lost to quota is not replaced.

This branch was previously deployed

1 inactive deployment
Preview — 81e41d43 Deployed Nov 19, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A killed review leaves its report at status running report_bug refuses an error that is in the console list the explorer was shown

1 participant