Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2950,6 +2950,18 @@ step; each entry says what to use instead.

### Fixed

- **After a failed turn, a call that returned keeps its result on ADK too.**
ADK stores a step's parallel results together, after the last, so a turn
that failed in between (its one `ERROR`: a sibling call's permission check
that raised, a store that refused the write) left a call that had returned
without a stored result, and the next turn answered it "Interrupted before
it finished", though it had run; a delegation's run ended `completed`
while its coordinator read the interruption. ADK now keeps the failed
turn's record of the calls that returned (`OpenCalls.failed`), as Deep
Agents does: the next turn answers each with its result and the state it
changed (`left_open`), any other as interrupted, and forgets the record
once those answers are stored, so it answers no later call of the same id.

- **A timed-out `execute_python` stops the code and what it started.** The
timeout killed only the process the executor started (the shell, or
bubblewrap's monitor), so the code ran on: on Linux the sandbox's init,
Expand Down
6 changes: 3 additions & 3 deletions CLAUDE.md

Large diffs are not rendered by default.

4 changes: 3 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -1573,7 +1573,9 @@ is answered "Interrupted by the user before it finished; it may have partly
run…", one waiting at its permission prompt or at a question "Not run: the
user cancelled it.", and a request stopped before its reply leaves
"(Interrupted by the user.)" as the agent's reply. A call a crash left open
is answered the same way on the next turn. Background jobs the request
is answered the same way on the next turn. After a turn that fails (its one
`ERROR`), a call that had returned keeps its result at the next turn, and
any other call the turn left open is answered as interrupted. Background jobs the request
started keep running, and a line after the cancel names each still queued
or running, with how to stop it: "Still running in the background: build
(j1). Stop one with /jobs cancel <id>."
Expand Down
106 changes: 57 additions & 49 deletions src/agentic_cli/workflow/adk/interrupted_calls_plugin.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,10 +20,12 @@
keeps its result, one at its prompt was not run.

``left_open`` answers, on the next turn, the calls nothing answered: a
crash's, a cancel's from before this plugin, or a stop's whose answers
could not be written as it stopped. ADK would keep a long-running one
without its result in every later request, which Claude and OpenAI refuse,
and drop any other one, logging "Dropping function calls" on every request.
crash's, a cancel's from before this plugin, a stop's whose answers could
not be written as it stopped, or a failed turn's, whose record keeps the
calls that returned (ADK had not stored their results yet). ADK would keep
a long-running one without its result in every later request, which Claude
and OpenAI refuse, and drop any other one, logging "Dropping function
calls" on every request.

Each call is recorded and answered as the run's that made it, the active run
(a sub-agent run is the active one in the task its runner runs in,
Expand Down Expand Up @@ -54,7 +56,6 @@
MARKER,
OpenCalls,
interrupted,
unrecorded,
)
from agentic_cli.workflow.sessions import (
ToolCallRef,
Expand Down Expand Up @@ -206,32 +207,26 @@ def _invocation(tool_context: Any) -> str | None:
return getattr(tool_context, "invocation_id", None)


def _from_record(open_calls: OpenCalls, call_id: str | None, run: str | None, actions: Any) -> Any:
"""A stopped turn's call answered from the record, as ``run``'s.
def _from_record(
open_calls: OpenCalls, call_id: str | None, run: str | None, actions: Any,
*, by_user: bool = True,
) -> Any:
"""A call left open answered from the record, as ``run``'s.

A finished call keeps what the model would have read (``_Finished.read``);
the state it changed goes into ``actions``, stored with the answer as
with its result. Any other is answered as ``OpenCalls.answer`` says.
with its result. Any other is answered as ``OpenCalls.answer`` says:
``by_user`` whether the user stopped its turn (a stopped turn's call, by
default).
"""
answer = open_calls.answer(call_id, by_user=True, run=run)
answer = open_calls.answer(call_id, by_user=by_user, run=run)
if not isinstance(answer, _Finished):
return answer
actions.state_delta.update(answer.state_delta)
actions.artifact_delta.update(answer.artifact_delta)
return answer.read


def returned(open_calls: OpenCalls, call_id: str | None, run: str | None) -> dict[str, Any] | None:
"""What the call ``call_id`` of ``run`` returned, from the record; None if it did not finish.

As ADK stores the result (text under ``result``), and as the model reads
it (the cap's cut): what the manager ends a delegation's run with when
no event brought its result (``ADKEventProcessor.end_returned_run``).
"""
finished = open_calls.recorded(call_id, run=run)
return _as_response(finished.read) if isinstance(finished, _Finished) else None


def _as_response(result: Any) -> dict[str, Any]:
"""A result as ADK stores it: a dict as it is, anything else under ``result``."""
return result if isinstance(result, dict) else {"result": result}
Expand Down Expand Up @@ -317,47 +312,60 @@ def unsealed(events: list[Event], open_calls: OpenCalls) -> list[Event]:


def left_open(events: list[Event], open_calls: OpenCalls) -> list[Event]:
"""The events that answer the calls nothing answered: a crash's, or a stop's left over.
"""The events that answer the calls nothing answered: a crash's, a failed
turn's, or a stop's left over.

A result answers only a call on its own branch: a sub-agent's result
never answers its parent's call of the same id. Each call left open is
answered as no record holds it (``interrupts.unrecorded``: a job's state,
else ``INTERRUPTED_BY_USER`` when the user stopped the conversation's
last turn, ``OpenCalls.take_cancel``, which this asks once, else
``INTERRUPTED``): every turn clears its conversation's record as it ends
(``OpenCalls.clear``), and a new process has none. Only the root's own
calls (and those of an agent it transferred to) are matched to a job: a
conversation stored while sub-agents ran on delegation branches of its
own session (``<agent>@<call id>``) may hold a sub-agent's call there,
whose run is numbered no more, so its call is answered interrupted, never
with another run's job of the same id. A stopped turn
answers what ADK's seals missed itself (``unsealed``), so a stop leaves
a call here only when those answers could not be written. The answer is
in the author, branch, scope and invocation of the event that made the
call, so it pairs with its call in that agent's view; calls that share
those are answered in one event, marked as ADK marks a seal.
never answers its parent's call of the same id. The root's own call left
open (and that of an agent it transferred to) is answered from the
record (``OpenCalls.answer``, as the root's). A failed turn keeps there
the calls that returned (``OpenCalls.failed``), whose results ADK had
not stored (it stores a step's parallel results together, after the
last): each is answered with its result and the state it changed. A call
no record holds gets its job's state, else ``INTERRUPTED_BY_USER`` when
the user stopped the conversation's last turn (``OpenCalls.take_cancel``,
which this asks once), else ``INTERRUPTED``: a turn that ends otherwise
clears its conversation's record (``OpenCalls.clear``), and a new
process has none. A conversation stored while sub-agents ran on
delegation branches of its own session (``<agent>@<call id>``) may hold
a sub-agent's call there, whose run is numbered no more: it is answered
interrupted, never with another run's result or job of the same id. A
stopped turn answers what ADK's seals missed itself (``unsealed``), so a
stop leaves a call here only when those answers could not be written.
The answer is in the author, branch, scope and invocation of the event
that made the call, so it pairs with its call in that agent's view;
calls that share those are answered in one event, marked as ADK marks a
seal.
"""
by_user = open_calls.take_cancel(get_active_turn())
return [
_answer_event(place, [
answers: list[Event] = []
for place, calls in _unanswered(events).items():
actions = EventActions()
answers.append(_answer_event(place, [
types.FunctionResponse(
id=call.id, name=call.name,
response=_as_response(_left_open_answer(call.id, place[2], by_user)),
response=_as_response(
_left_open_answer(open_calls, call.id, place[2], by_user, actions)
),
)
for call in calls
], INTERRUPTED)
for place, calls in _unanswered(events).items()
]
], INTERRUPTED, actions))
return answers


def _left_open_answer(call_id: str | None, branch: str | None, by_user: bool) -> Any:
def _left_open_answer(
open_calls: OpenCalls, call_id: str | None, branch: str | None, by_user: bool,
actions: Any,
) -> Any:
"""What ``left_open`` answers the call ``call_id`` made on ``branch`` with.

The root's own call (no delegation on its branch) gets its job's state
when it submitted one (``unrecorded``, the root's: ``started_by`` None);
a sub-agent's on a delegation's branch (``<agent>@<call id>``) cannot be
named any more, so its call gets no job.
The root's own call (no delegation on its branch) is answered from the
record (``_from_record``, as the root's: a call that returned keeps its
result, the state it changed going into ``actions``; one no record holds
gets its job's state when it submitted one, ``started_by`` None); a
sub-agent's on a delegation's branch (``<agent>@<call id>``) cannot be
named any more, so its call gets neither.
"""
if any("@" in segment for segment in (branch or "").split(".")):
return interrupted(INTERRUPTED_BY_USER if by_user else INTERRUPTED)
return unrecorded(call_id, by_user=by_user)
return _from_record(open_calls, call_id, None, actions, by_user=by_user)
28 changes: 22 additions & 6 deletions src/agentic_cli/workflow/adk/manager.py
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,12 @@
JOB_RESULT,
OUTSIDE_CONTENT_KEY,
)
from agentic_cli.workflow.sessions import ConversationChangedError, SessionInfo, get_active_run
from agentic_cli.workflow.sessions import (
ConversationChangedError,
SessionInfo,
get_active_run,
get_active_turn,
)
from agentic_cli.workflow.tool_output import tool_output_dir
from agentic_cli.workflow.usage import adk_usage_event

Expand Down Expand Up @@ -1212,7 +1217,12 @@ async def _answer_left_open(self, session: Session) -> Session:
them, and so does a stop whose own answers to the calls ADK's seals
missed could not be written (``_answer_unsealed``); ADK would keep a
long-running one in every request, which Claude and OpenAI refuse,
and drop any other one with a warning.
and drop any other one with a warning. So does a failed turn, for
the calls whose results it had not stored; its record of those that
returned (``OpenCalls.failed``) answers them with their results.
Once the answers are stored the record is forgotten, so it answers
no later call of the same id (a provider that numbers its call ids
repeats them).

ADK keeps two kinds of call open on purpose. A task-mode agent's
delegation cannot be built (a native ADK YAML app with one is
Expand All @@ -1237,6 +1247,7 @@ async def _answer_left_open(self, session: Session) -> Session:
logger.debug("left_open_calls_answered", count=sum(
len(event.content.parts) for event in answers
))
self._open_calls.clear(get_active_turn())
return session

async def _answer_unsealed(self, ref: SessionRef) -> None:
Expand Down Expand Up @@ -1420,17 +1431,22 @@ async def _run_and_stream(
# Every call of the turn is answered now, sealed or returned,
# but for one a stop's seals took for answered (they pair by
# id alone): it is answered here, from the record, before the
# record is cleared. One a failure left open is answered at
# the next turn (left_open) as interrupted: the state it
# changed, which ADK stores with its result, is gone.
# record is cleared. A failure keeps the record of the calls
# that returned (OpenCalls.failed), as Deep Agents does: ADK
# stores a step's parallel results together, after the last,
# so one that returned beside a call the failure stopped has
# no stored result, and the next turn answers it from the
# record, with the state it changed (left_open).
ref = self.session_ref(session_id, user_id)
if aborted.is_set():
# Should those answers not be written, left_open gives
# them as the user's.
self._open_calls.cancelled(ref)
await asyncio.shield(self._answer_unsealed(ref))
else:
elif failure is None:
self._open_calls.clear(ref)
else:
self._open_calls.failed(ref)

# Drain any remaining LLM events after processing completes
if self._llm_logging_plugin:
Expand Down
22 changes: 13 additions & 9 deletions src/agentic_cli/workflow/interrupts.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,11 @@
tool results) leaves ``MARKER`` as the main agent's reply, so the next
turn answers the new message rather than resuming the old one;
- a call a crash left open (nothing ran on the way out) is answered
``INTERRUPTED`` on the next turn.
``INTERRUPTED`` on the next turn;
- a turn that fails (its one ``ERROR``) keeps the record of its calls that
returned (``OpenCalls.failed``): at the next turn each of them that has no
stored result is answered with the result it returned, and any other
call left open ``INTERRUPTED``.

Background jobs a long-running tool started are not part of the turn: they
go on, and their results arrive later. A call whose job was submitted is
Expand All @@ -29,9 +33,10 @@
own, or a sub-agent's: ``workflow.sessions.get_active_run``), since a
provider that numbers its call ids per agent gives a sub-agent's call the
id of its parent's delegation: the backends answer from it (ADK as the
cancelled turn stops, through its ``abort_signal``; Deep Agents before the
first model call of that conversation's next turn, however many turns ran
in others since).
cancelled turn stops, through its ``abort_signal``, and a failed turn's
calls as that conversation's next turn starts; Deep Agents before the first
model call of that conversation's next turn, however many turns ran in
others since).
"""

from __future__ import annotations
Expand Down Expand Up @@ -115,11 +120,10 @@ def unrecorded(
"""The answer for an open call no record holds: its job's state, else why it has none.

``OpenCalls.answer`` gives it for a call it holds nothing for, matching
the job by the instance of the call's run (``started_by``). ADK's
``left_open`` gives it for the root's own calls it answers (``started_by``
None), since each turn clears its conversation's record as it ends, and
a new process has none; a sub-agent's call there gets no job, its run
being numbered no more.
the job by the instance of the call's run (``started_by``); ADK's
``left_open`` answers the root's own calls through it (``started_by``
None), and a sub-agent's call with no job, its run being numbered no
more.
"""
record = job_for_call(call_id, started_by=started_by)
if record is not None:
Expand Down
75 changes: 75 additions & 0 deletions tests/integration/test_interrupted_calls.py
Original file line number Diff line number Diff line change
Expand Up @@ -937,6 +937,81 @@ async def test_on_deep_agents_a_call_left_open_across_a_restart_is_answered(
assert INTERRUPTED in results(claude.last("Answer"))["toolu_w"]


# --- a failed turn ----------------------------------------------------------------------


async def test_on_adk_calls_whose_results_the_store_refused_keep_them(hermetic, monkeypatch):
"""The store refuses the step's results (another process changed the
conversation): the turn fails, and the next turn answers each call with
the result it returned, not as interrupted."""
from google.adk.errors import StaleSessionError

claude = serve(monkeypatch, Claude(Answer=[
say("First."),
calls(("ic_fast", {"topic": "X"}, "toolu_a"), ("ic_fast", {"topic": "Y"}, "toolu_b")),
say("Back again."),
]), "adk")
manager = solo("adk", ic_fast)
try:
await turn(manager, "Hello.")
service = manager._session_service
real = service.append_event
refused: list = []

async def stale_once(session, event):
if event.get_function_responses() and not refused:
refused.append(event)
raise StaleSessionError("The session has been modified in storage.")
return await real(session, event)

monkeypatch.setattr(service, "append_event", stale_once)
failed = await turn(manager, "Look X and Y up.")
await turn(manager, "Hello again.")
finally:
await manager.cleanup()

assert claude.refused == []
assert [e.metadata["error_code"] for e in failed if e.type is EventType.ERROR] == [
"ConversationChangedError",
]
sent = results(claude.last("Answer"))
assert sent["toolu_a"] == json.dumps({"success": True, "fact": "X is 42"})
assert sent["toolu_b"] == json.dumps({"success": True, "fact": "Y is 42"})


async def test_on_adk_a_failed_turns_record_answers_no_later_call(hermetic, monkeypatch):
"""A failed turn's record answers its own calls left open at the next
turn, and nothing after: a later call of the same id (a provider that
numbers its call ids repeats them), left running by a stop, is the
user's, not the earlier call's result."""
from agentic_cli.workflow.adk.interrupted_calls_plugin import ABORTED
from agentic_cli.workflow.interrupts import interrupted

claude = serve(monkeypatch, Claude(Answer=[
calls(("ic_fast", {"topic": "X"}, "call_0")),
bad_request("Overloaded."),
calls(("ic_wait", {}, "call_0")),
]), "adk")
manager = solo("adk", ic_fast, ic_wait)
try:
failed = await turn(manager, "Look X up.")
await cancelled(manager, "Wait for it.")
session = await manager._session_service.get_session(
app_name=manager.app_name, user_id="u", session_id="s")
finally:
await manager.cleanup()

assert claude.refused == []
assert [e.metadata["recoverable"] for e in failed if e.type is EventType.ERROR] == [False]
stopped = session.events[-1].invocation_id
(answer,) = [
response.response for event in session.events
if event.invocation_id == stopped and event.error_code == ABORTED
for response in event.get_function_responses() if response.id == "call_0"
]
assert answer == interrupted(INTERRUPTED_BY_USER)


# --- the ADK it rests on ----------------------------------------------------------------


Expand Down
Loading
Loading