Repository navigation
control-plane, frontend: retries that actually run; retry a stuck task with another agent - #80
Merged
Merged
Conversation
… can be retried
A retry after an attempt that ended without a receipt never ran. The gateway holds attempt n while
attempt n-1 still holds its payment (503), and that payment only comes back at the attempt's deadline,
30 minutes after dispatch. The retry's dispatch gave up after event_retry_window_s (10 minutes) and
nothing read it again, so the task said "Retrying" forever, and every later event of the plan,
Close plan included, waited behind it.
- A retry's dispatch keeps being delivered for the deadline plus 10 minutes (dispatch.RETRY_WINDOW_S),
and while it waits it holds up no other event of its plan.
- Closing a plan cancels its undelivered dispatches, so nothing locks money after the requester gave up.
- stuck_indexes counts the newest dispatch: one still on its way is in flight, one given up waits on
the requester.
- POST /v0/plans/{uuid}/tasks/{index}/retry {agent_id}: a stuck task goes to an agent the requester
chooses, paid from that task's own escrow, up to max_requester_attempts (6) attempts in all. The run
page offers it next to Close plan.
- The plan detail returns pending_dispatches and stuck_indexes. The run page says what a retry waits on
and reloads while one is pending, since the stream carries jobs, not dispatches.
- The worker refuses to start without its sandbox image, so jobs stay queued instead of each failing
and costing the requester an agent pick.
…ce, and never silently Review of the previous commit found four ways a retry still went wrong: - Its deadline was counted from when it was queued, but it can only lock once the failed attempt's payment is back, up to that attempt's deadline. It locked with almost none left, and the keeper returned it before a worker claimed the job. The deadline now counts from the previous attempt's deadline plus RETURN_MARGIN_S. - A lock that reverted (plan closed, balance short) arrives as a gateway `error` event, which was only logged, so the task said "Retrying" forever. That event now marks its dispatch given up, and the task waits on the requester. - An older attempt whose lock landed after its dispatch looked given up re-dispatched an attempt number the requester's retry already took. The unique-key error rolled back the whole outcome pass, for every plan. Only the newest attempt retries on its own now, and a task that passes cancels its dispatches still on their way. - The gateway only waits on attempt n-1. Behind a given-up attempt, a retry could lock a second payment while an older one still held one. The retry endpoint refuses until the ledger shows that payment released or returned (money.holding_attempt, kept in the ledger module so the graph side still never reads it). The run page takes the stuck list from the plan detail instead of deriving it, so no Retry is offered while the orchestrator is still reacting to a finished job, and hides Retry at max_requester_attempts. Migration 0023 indexes the outbox by a dispatch's task_id, which these lookups join on.
OfficialAbhinavSingh
force-pushed
the
fix/orchestration-retries
branch
from
November 6, 2026 07:21
f45c41e to
15fb703
Compare
OfficialAbhinavSingh
added a commit
that referenced
this pull request
Nov 6, 2026
…kages (#82) --with-deps also apt-installs nine font packages (21.9 MB) that no test needs: the app bundles its own fonts and no test compares pixels, and the browser's libraries are already on the runner. On 2026-11-06 that download from the Azure Ubuntu mirror ran at 40 kB/s (8 min 56 s), and the job hit its 15-minute timeout twice on main, so #80 never deployed. The install step now has its own 5-minute cap and the job 25 minutes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #78, so merge that first. Control plane and frontend only, with no gateway or contract change.
What was broken
Found by running a plan on app.mergit.site (plan
c8d22b3f). Task 1 of that plan errored at 05:45:50. Its retry with the second agent never ran, and the task said "Retrying" for good.ingress.rs:141-158, 503). An attempt that ended without a receipt only gets that payment back at its deadline, 30 minutes after dispatch, once the keeper returns it. Meanwhile the outbox gavetask.dispatch_requestedup afterevent_retry_window_s(600s), and nothing read it again.What changes
Retries
RETURN_MARGIN_S), not from when it was queued.errorevent for a dispatch) marks that dispatch given up, so the task waits on the requester instead of saying "Retrying" forever.Closing a plan
Retrying a stuck task by hand
POST /v0/plans/{uuid}/tasks/{index}/retry {agent_id}sends a stuck task to an agent the requester chooses, up tomax_requester_attempts(6) attempts per task.money.py, and the graph side still never reads money events.Plan detail
pending_dispatches,stuck_indexesandmax_requester_attempts.Run page
Worker
Database
task_id.Checks
Known limits
PlanClosed.refund, so that's for the gateway owner.