Skip to content

Replay provider streams interrupted after partial output - #78

Merged
vanedaltro merged 3 commits into
mainfrom
feat/retry-interrupted-streams
Oct 30, 2026
Merged

vanedaltro merged 3 commits into
mainfrom
feat/retry-interrupted-streams

Conversation

@leonardoaraujosantos

@leonardoaraujosantos leonardoaraujosantos commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

Current merge review at ee5c453d: This head integrates merged #77/#81. The retry API and usage-limit clock exports are both retained, and the dashboard bundle is regenerated from combined sources. Local integration checks pass: eight real-HTTP transport-fault tests, fourteen usage-limit tests, the runtime replay/history regression, 48 dashboard tests and typecheck, Rust formatting and strict OpenSpec validation. New-head hosted CI is pending; the earlier full-workspace results below apply to the preceding implementation head.

A provider stream that fails mid-answer with a retryable error (transport, the stream idle timeout, in-stream overload, rate limit) now discards the partial answer and re-sends the same request within the existing retry budget. Before, any output ended retries and the Turn settled as interrupted.

Motivation: in the same-model pilot against Codex CLI, build-inventory-cli was lost to transport: error decoding response body before any edits (pilot). The task passes 3/3 on a fresh run; a mid-stream break should cost one retry, not the task.

Rules (OpenSpec change retry-interrupted-streams)

  • What is retried: a retryable failure after partial text, reasoning, streamed tool arguments or complete tool calls discards that attempt and replays the same request.
  • Budget: retries before and after output share one budget of 4, with the same backoff and retry-after cap.
  • When replay stops: never after protocol completion, and never once a tool call of that answer has been dispatched.
  • Completed calls don't block a replay: calls are dispatched (execute_tools records the dispatch first) only after the stream completes. A completed but undispatched call has no effect yet, so discarding it is safe. The spec now states that dispatching a call early ends replay for that answer.
  • Nothing from a discarded attempt reaches history: the drain now records reasoning items (with opaque state) and tool.called events together with the text when the stream ends, instead of while it is open. This updates the eager-execution durability wording in tool-registry and session-runtime: calls are recorded no later than stream completion and before they start.
  • Usage:
    • A discarded attempt's usage is recorded as a failed usage.recorded.1 (purpose turn, same message id), unless a budget reservation already billed it.
    • It counts toward totals and budgets, but not toward the step's usage or the context meter.
    • Each attempt is checked against budgets and gets its own reservation, so nothing is double-reserved and hard caps still apply.
  • Turn state: the turn-state header is kept across the replay, and reasoning item numbering restarts at 0.

Clients

  • session.retry carries message_id and discarded.
  • The TUI and the web dashboard clear the partial answer.
  • cyber exec stream-json prints a retry line.
  • Exec text output and the SDK render only recorded events, so they never show a discarded attempt.

Tests

  • crates/cyber-llm/tests/transport_faults.rs (real HTTP faults):
    • partial text, reasoning, partial tool arguments and a completed tool call are each replayed and succeed;
    • retries before and after output share the budget, and exhaustion returns the last attempt's partial output;
    • a stalled stream is replayed after the idle timeout.
  • adapters.rs: an Anthropic in-stream overload is replayed, and a discarded attempt's usage is kept.
  • cyber-server/tests/runtime.rs:
    • a_mid_stream_retry_records_only_the_replayed_attempt: no stale data in the event log, one tool.called, the tool runs once, and history survives a restart;
    • the exhaustion test still settles as interrupted;
    • a_complete_tool_call_is_recorded_when_its_stream_completes replaces the old "durable while the stream is open" test.
  • Clients: TUI a_discarded_attempt_clears_its_streamed_text, the exec retry line, and a web model.test.ts case.

Earlier implementation checks

  • fmt and workspace clippy -D warnings: clean.
  • Full nextest --workspace --profile ci: 5249 of 5252 passed (2 flaky), 21 skipped.
    • The 3 failures came while machine load was around 70 and passed on rerun: reserved_budgets::cancellation_during_opening_and_wall_deadline_keep_unknown_allocations, review_command::a_budget_stopped_review_workflow_exits_4_with_partial_findings and uninstall_cli::manager_known_from_listings_only_is_not_run.
    • The reserved-budget test uses a 20 ms timer on a path this change doesn't touch. Under that load it failed 2 times in 40 on this branch and 0 in 40 on main; at normal load it passed 40 of 40.
  • OpenAPI current, SDK check and docs/cli.md: unchanged.
  • Web: typecheck passes and 47 tests pass.
  • spec_lint: 0 errors, 25 warnings (baseline). openspec validate --all --strict: passes.

Trade-off

If the process crashes mid-stream, that step's undispatched tool calls are now lost. Before, they were recorded and settled as "never dispatched". Neither version ran them.

Docs: docs/local-core-guide.md, a dated note in docs/measurements/provider-stream-reliability-follow-up.md (measured results unchanged), ROADMAP. The "Stalled stream" scenario of improve-model-protocol-fidelity now describes a replay.

leonardoaraujosantos and others added 3 commits October 29, 2026 21:30
A retryable failure (transport, idle timeout, in-stream overload, rate
limit) after partial text, reasoning or tool calls now discards the
attempt and re-sends the same request within the existing retry budget,
instead of settling the Turn as interrupted. Replay ends at protocol
completion; tool calls only dispatch after the stream completes.

- cyber-llm: open_with_retry returns a replaying stream that yields
  LlmEvent::Discarded before the new attempt; retries before and after
  output share one budget. collect drops discarded output, keeps usage.
- Drain records reasoning items and tool calls with the text when the
  stream ends, so nothing of a discarded attempt reaches history.
  Usage a discarded attempt reported is recorded as a failed turn
  usage.recorded.1 unless a reservation already billed it.
- session.retry carries message_id and discarded; the TUI and web
  dashboard clear the partial answer, exec stream-json prints a retry line.
- OpenSpec change retry-interrupted-streams (Retry policy, eager
  execution durability, exec stream-json schema), docs and ROADMAP.
@vanedaltro
vanedaltro merged commit 3ecf075 into main Oct 30, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants