Skip to content

control-plane, frontend: retries that actually run; retry a stuck task with another agent - #80

Merged
OfficialAbhinavSingh merged 2 commits into
mainfrom
fix/orchestration-retries
Nov 6, 2026
Merged

OfficialAbhinavSingh merged 2 commits into
mainfrom
fix/orchestration-retries

Conversation

@OfficialAbhinavSingh

Copy link
Copy Markdown
Member

Stacked on #78, so merge that first. Control plane and frontend only, with no gateway or contract change.

What was broken

Found by running a plan on app.mergit.site (plan c8d22b3f). Task 1 of that plan errored at 05:45:50. Its retry with the second agent never ran, and the task said "Retrying" for good.

  • The retry gave up after 10 minutes. The gateway won't lock attempt n while attempt n-1 still holds its payment (ingress.rs:141-158, 503). An attempt that ended without a receipt only gets that payment back at its deadline, 30 minutes after dispatch, once the keeper returns it. Meanwhile the outbox gave task.dispatch_requested up after event_retry_window_s (600s), and nothing read it again.
  • One stuck retry blocked its whole plan. The outbox sends a plan's events in order, so every later event of that plan waited behind the stuck retry, Close plan included.
  • A stuck task only offered Close plan. There was no way to send it to another agent.
  • Host failures counted as agent failures. The missing sandbox image failed every job, and each failure used up one of the requester's agent picks.

What changes

Retries

  • A retry's dispatch is delivered until its earliest possible lock plus 10 minutes. Its deadline is counted from when it can lock (the previous attempt's deadline plus RETURN_MARGIN_S), not from when it was queued.
  • While it waits, it holds up no other event of its plan.
  • A reverted lock (the gateway's error event for a dispatch) marks that dispatch given up, so the task waits on the requester instead of saying "Retrying" forever.
  • Only the newest attempt retries on its own. A task that passes cancels its dispatches that haven't gone out yet.

Closing a plan

  • Close plan cancels the plan's dispatches that haven't gone out, so nothing locks money after the requester gave up.

Retrying a stuck task by hand

  • POST /v0/plans/{uuid}/tasks/{index}/retry {agent_id} sends a stuck task to an agent the requester chooses, up to max_requester_attempts (6) attempts per task.
  • It refuses (409) while an earlier attempt still holds its payment behind a given-up dispatch. The gateway only checks attempt n-1, so without this a retry could lock a second payment from the plan's shared balance. That check reads the ledger, so it lives in money.py, and the graph side still never reads money events.

Plan detail

  • The plan detail returns pending_dispatches, stuck_indexes and max_requester_attempts.

Run page

  • Stuck tasks get an agent picker and a "Retry with this agent" button next to Close plan.
  • The task panel says what a retry is waiting on.
  • The page reloads while a retry is pending, because the stream carries jobs, not dispatches.

Worker

  • The worker refuses to start without its sandbox image, so jobs stay queued instead of failing.

Database

  • Migration 0023 indexes the outbox by a dispatch's task_id.

Checks

  • Control plane: ruff, ruff format and pyright clean. 558 unit tests pass, plus 466 against Postgres.
  • New tests cover the retry window and deadline, the reverted lock, closing a plan, the late older attempt, cancel on pass, the holding refusal, the slot rule, and the endpoint (422/409/404/403/200).
  • I broke each fix on purpose and its test caught it (10 of 10).
  • Frontend: tsc, eslint and prettier clean. vitest 515/515. Live e2e 115 passed, including a new "a stuck task can be sent to another agent" test.
  • Full e2e: only the two long landing scroll tests fail under full-suite load, the same as on main. They pass on their own.
  • Checked the run page in light and dark at 1440 and 390 wide, with no horizontal overflow. The agent picker wraps under its label on phones.
  • An independent review of the first commit found the deadline, reverted-lock, late-attempt and second-lock problems. The second commit fixes all four.

Known limits

  • A retry after an errored attempt still waits up to about 31 minutes for that attempt's payment. Returning an attempt's payment as soon as it errors needs a new Escrow function and a gateway change, which is out of scope here.
  • Closing a credits plan while a task is still locked loses that task's money from the requester's credits. The gateway watcher only restores credits from PlanClosed.refund, so that's for the gateway owner.
  • If a dispatch looked given up but its lock actually landed later, its job still runs.

… can be retried

A retry after an attempt that ended without a receipt never ran. The gateway holds attempt n while
attempt n-1 still holds its payment (503), and that payment only comes back at the attempt's deadline,
30 minutes after dispatch. The retry's dispatch gave up after event_retry_window_s (10 minutes) and
nothing read it again, so the task said "Retrying" forever, and every later event of the plan,
Close plan included, waited behind it.

- A retry's dispatch keeps being delivered for the deadline plus 10 minutes (dispatch.RETRY_WINDOW_S),
  and while it waits it holds up no other event of its plan.
- Closing a plan cancels its undelivered dispatches, so nothing locks money after the requester gave up.
- stuck_indexes counts the newest dispatch: one still on its way is in flight, one given up waits on
  the requester.
- POST /v0/plans/{uuid}/tasks/{index}/retry {agent_id}: a stuck task goes to an agent the requester
  chooses, paid from that task's own escrow, up to max_requester_attempts (6) attempts in all. The run
  page offers it next to Close plan.
- The plan detail returns pending_dispatches and stuck_indexes. The run page says what a retry waits on
  and reloads while one is pending, since the stream carries jobs, not dispatches.
- The worker refuses to start without its sandbox image, so jobs stay queued instead of each failing
  and costing the requester an agent pick.
…ce, and never silently

Review of the previous commit found four ways a retry still went wrong:

- Its deadline was counted from when it was queued, but it can only lock once the failed attempt's
  payment is back, up to that attempt's deadline. It locked with almost none left, and the keeper
  returned it before a worker claimed the job. The deadline now counts from the previous attempt's
  deadline plus RETURN_MARGIN_S.
- A lock that reverted (plan closed, balance short) arrives as a gateway `error` event, which was only
  logged, so the task said "Retrying" forever. That event now marks its dispatch given up, and the
  task waits on the requester.
- An older attempt whose lock landed after its dispatch looked given up re-dispatched an attempt
  number the requester's retry already took. The unique-key error rolled back the whole outcome pass,
  for every plan. Only the newest attempt retries on its own now, and a task that passes cancels its
  dispatches still on their way.
- The gateway only waits on attempt n-1. Behind a given-up attempt, a retry could lock a second
  payment while an older one still held one. The retry endpoint refuses until the ledger shows that
  payment released or returned (money.holding_attempt, kept in the ledger module so the graph side
  still never reads it).

The run page takes the stuck list from the plan detail instead of deriving it, so no Retry is offered
while the orchestrator is still reacting to a finished job, and hides Retry at max_requester_attempts.
Migration 0023 indexes the outbox by a dispatch's task_id, which these lookups join on.
@OfficialAbhinavSingh
OfficialAbhinavSingh changed the base branch from fix/run-blockers to main November 6, 2026 07:21
@OfficialAbhinavSingh
OfficialAbhinavSingh merged commit ead14d2 into main Nov 6, 2026
2 checks passed
@github-project-automation github-project-automation Bot moved this from Backlog to Done in Kanban Nov 6, 2026
OfficialAbhinavSingh added a commit that referenced this pull request Nov 6, 2026
…kages (#82)

--with-deps also apt-installs nine font packages (21.9 MB) that no test needs: the app bundles its own
fonts and no test compares pixels, and the browser's libraries are already on the runner. On
2026-11-06 that download from the Azure Ubuntu mirror ran at 40 kB/s (8 min 56 s), and the job hit its
15-minute timeout twice on main, so #80 never deployed. The install step now has its own 5-minute cap
and the job 25 minutes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

1 participant