diff --git a/CHANGELOG.md b/CHANGELOG.md index 53871992..82043a67 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -177,10 +177,10 @@ the start of the entry's title. - On ADK `TASK_PROGRESS` comes only when a tool call changed the task list, and a list whose tasks are all done stays: `get_tasks` answers it, where 0.6 cleared it. (Changed: **On ADK, task progress is reported…**) -- A sub-agent run past 100 model calls ends with a partial answer, and a turn - starts at most 20 sub-agent runs; an app sets other numbers with - `subagents=SubAgentPolicy(...)`. (Added: **`SubAgentPolicy` and - `AgentLimits`…**) +- A sub-agent run past 100 model calls ends with a partial answer, more than + 4 parallel delegations queue, and a turn starts at most 20 sub-agent runs; + an app sets other numbers with `subagents=SubAgentPolicy(...)`. (Added: + **`SubAgentPolicy` and `AgentLimits`…**) #### Settings that moved or changed meaning @@ -575,7 +575,17 @@ the start of the entry's title. run's limits, enforced on both backends** (`agentic_cli.workflow`, and `agentic_cli.workflow.agents` with `DEFAULT_LIMITS`). Frozen dataclasses. `SubAgentPolicy` holds an app's numbers for its sub-agents: `max_active` - (4, the runs working at once in a conversation, at any depth), + (4: at most that many runs of a conversation work at once, at any depth; + a run past them waits `queued` for a free working slot, the time it waits + not counted against its `timeout`, and a run that waits on its children + or on the user (a question, a permission prompt) lends its slot + meanwhile, taking one back after ahead of the runs that have not + started, so nested delegations never deadlock (a prompt no one can be + shown lends nothing: `BaseWorkflowManager.can_request_user_input()`, an + input callback set for the turn, which has not ended); `task`'s + description says "At most 4 sub-agents work at once; more calls in one + response wait for a free slot."; provisional: `agentic_cli.workflow.agents.slots.SlotPool`, + the manager's delegations' `slots(conversation)`), `max_runs_per_turn` (20: a turn starts at most that many runs, at any depth; a `task` call past it starts no run, and its caller reads `{"success": false, "error": "This turn has already started 20 sub-agent @@ -670,8 +680,9 @@ the start of the entry's title. to the user, never to its parent (contract-tested on both backends). - **`/agents`: the conversation's sub-agent runs.** A table of the open conversation's sub-agent runs, each followed by the runs it started: - instance, type, status (`running`, `completed`, `completed (partial)` for - an answer cut at its output limit, `refused`, `failed`, `interrupted`), + instance, type, status (`running`, `queued` waiting for a free working + slot, `needs you` waiting on the user, `completed`, `completed (partial)` + for an answer cut at its output limit, `refused`, `failed`, `interrupted`), run, title, tokens in/out (from the usage ledger, so a run still going shows what it has used so far) and time. `/agents ` shows one instance (`researcher-2`) or run (`researcher-2#1`): its type, parent and title, @@ -724,10 +735,12 @@ the start of the entry's title. (`WorkflowEvent.agent_run`; `metadata["status"]` `started`, with its brief's title, the brief's filled fields (`brief`), who started it (`started_by`: `agent:`, `agent:root` for the main agent) and - how it runs (`mode`: `blocking`, its caller waiting for it), then - `completed` (`partial` for an answer cut at its output limit), `refused`, - `failed` or `interrupted`, with the first 200 characters of its result; a - run fails when it cannot finish (a model call of its fails, its request no + how it runs (`mode`: `blocking`, its caller waiting for it), then, while + it works, `queued` (waiting for a free working slot), `running` (once it + has one, and once the user has answered) and `input_required` (waiting + on the user), then `completed` (`partial` for an answer cut at its output + limit), `refused`, `failed` or `interrupted`, with the first 200 + characters of its result; a run fails when it cannot finish (a model call of its fails, its request no longer fits its model's window, or its model gives no answer), and the sub-agent's answer is never read for success, so an answer that is JSON saying it failed completes), the same on both backends: the framework @@ -737,13 +750,15 @@ the start of the entry's title. provisional: `records(ref)`, `record(ref, id)`, `number_from(ref, counts)` to number a conversation's instances past the ones its store keeps; a record has the brief, - `started_by`, `mode` and, for a cancelled run, `cancelled_by`); a run a + `started_by`, `mode` and, for a cancelled run, `cancelled_by`, and its + status is `running`, `queued` or `input_required` while it works); a run a stopped or failed turn leaves open is recorded `interrupted`. `get_active_run()` (exported from `agentic_cli.workflow` with `RunRef` and `root_run`) is the sub-agent run at work, None in the root's own: a sub-agent's tools, permission checks and questions see it. A consumer that does not - know `AGENT_RUN` ignores it; the CLI ignores it. + know `AGENT_RUN` ignores it; the CLI's events box says when a run waits + for a slot or for the user. - **A backend's conversations, read without its manager: `agentic_cli.workflow.session_readers`.** `await diff --git a/CLAUDE.md b/CLAUDE.md index 527f63c3..d4fd02f8 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -36,27 +36,27 @@ agentic-cli/ │ │ ├── conversation.py # ConversationCore: the conversation the turns run in (its id, turned, its lock at its Place: the backend and store, the store the run's for its whole life (Place.kept: not the memory store, where the CLI keeps one conversation at a time and NOT_KEPT says why nothing else opens), the backend recorded as the app starts, for a new conversation on another backend, and on a /settings change of the orchestrator while the workflow is down, its next start builds afresh and nothing of the conversation is stored (SettingsUI._move_here, WorkflowController.next_start_builds), never read from the settings in between; deferred: an id this run made (the start's, /new's, an orchestrator change's) is in no store, so it is held from its first write, hold_to_write under the turn lock before a turn, a job's notice or /compact (_held_to_write; a run in which no turn runs makes no lock file: a line typed while the workflow is down takes the hold before its retried start; a lock that cannot be taken, an OSError, leaves it unheld, said once, and the write goes on); fresh: nothing of it stored, opened_empty (an id this run made, or one a load found empty) and not written (no turn of it ran on a ready workflow), the one condition for another backend); switch: the one protocol (/new on any backend, /resume, startup, the same id in another place while the workflow is down): the target's lock taken (another process's refused), memories proposed, the workflow loads it (a stored one gone: not made, or, made already in another place, opened empty: gone), then made at once (lock, id, task list, usage counters) or let go, a workflow that took it given the old one back; made before the load when there is no other conversation to stay in (the same id, or another place: a new or stored conversation on another backend); a Ctrl+C at any await leaves one conversation held; switch_now: the deprecated session_id setter's part; turn_lock: what writes to the conversation and every switch take it; run_turn, compact and run_job_resume: a turn (echoed, held to write, turned, written once it reached a ready workflow; run_turn says whether it ran, whatever the processor returns), /compact and a finished job's notice under it, on TurnParts (the app's parts as they are); deliver_finished_jobs: each job that waits here (cli/waiting_jobs.py's rule, which waiting_jobs names them by, the other backends' look through TurnParts.kept_elsewhere), every JobManager call on a worker thread, a delivery claimed only once it can reach the agent (process_resume(claim=), after the readiness check and can_resume; _claim_delivery: a claim a Ctrl+C stopped is handed back, JobManager.release_resume), a failure before the claim said once per job and conversation when the CLI tells jobs on its own (telling_on_its_own); to_drop/drop_left: the conversations /new left on the memory store that the store may still hold (the workflow down, a lookup or delete that raised), dropped through TurnParts.drop under the turn lock once a workflow is ready, before a turn, /compact or a delivery uses it (after a turn or /compact that started it), one try per point, done once the store has it no more (a workflow without supports_sessions has none), the open one skipped; a delivery with none to drop takes no turn lock │ │ ├── conversation_lock.py # ConversationLock: one process per stored conversation (flock under /locks/; ConversationCore holds it and refuses one held elsewhere; /sessions --delete too, through conversation_held) │ │ ├── builtin_commands.py # /help (by category; /help : its usage and examples), /status (the main agent's context rows: context_budget; why the workflow did not start: app.workflow_error; tokens by agent: the main agent, each sub-agent with its runs, the side calls by purpose, from the manager's usage ledger), /models (ModelsCommand: the list, each model's facts and sources, Status for an entry left out, the providers in use; /models searches the full listings; refresh), /sessions (every installed backend's, a Backend column; --delete= on the backend that keeps it, --backend= for an id two keep; on the memory store NOT_KEPT, no table, --delete refused), /resume [] (sessions; refused on the memory store), /jobs (resume: finished jobs), /compact [focus], /new [backend] (an unknown backend or a missing extra refused: check_backend), ... -│ │ ├── agents_command.py # /agents: the open conversation's sub-agent runs (agent_runs.records under workflow.session_ref(app.session_id)) in a table, each followed by the runs it started (instance, type, status: running, completed (partial) for a cut answer, the end as recorded; run, title, tokens in/out from the usage ledger, usage_of, live for a running run; time); /agents : one instance or run (type, parent, title; each run's status and time, its brief's other fields, its result's first lines, its error, its tokens by model); the records live in memory with the manager, so a conversation resumed after a restart or a backend switch lists none; a module of its own, as /memory's +│ │ ├── agents_command.py # /agents: the open conversation's sub-agent runs (agent_runs.records under workflow.session_ref(app.session_id)) in a table, each followed by the runs it started (instance, type, status: running, queued (waiting for a free working slot), needs you (input_required: waiting on the user), completed (partial) for a cut answer, the end as recorded; run, title, tokens in/out from the usage ledger, usage_of, live for a running run; time); /agents : one instance or run (type, parent, title; each run's status and time, its brief's other fields, its result's first lines, its error, its tokens by model); the records live in memory with the manager, so a conversation resumed after a restart or a backend switch lists none; a module of its own, as /memory's │ │ ├── memory_command.py # /memory: the list (a picker like /resume's), one memory (Delete, History), /memory , history (Restore), inbox (Accept/Discard/Skip), stale; proposals_line; Enter's button is Close or Skip, and Delete acts only on the memory as shown │ │ ├── workflow_controller.py # WorkflowController (lazy/background init; subagents=: the SubAgentPolicy every manager it builds gets, a switch_backend's too; the session store of every manager of the run, store, fixed at its first start; the backend it runs, backend: the settings' orchestrator at its first start, then switch_backend's (a conversation on another backend, new or opened: the new manager started first on the run's services lent, given over by the old one and published at once, the old one released in the controller's task, a cancel stops only the wait; a failed start or give leaves the old one with its services; while down, a manager a failed reinitialization kept lends its services to the next build, _lender, and is released once they are given); reinitialize, a model switch, always in place; the status bar names the main agent's model) │ │ ├── session_resume.py # What /resume shows: the picker (picker: each row's backend once one is another's; PickedConversation, the id a str naming its backend), the "Resumed:" line ("Resumed on :" on another backend), the replay of the last 10 messages │ │ ├── stored_conversations.py # StoredConversations: one look at every installed backend's stored conversations (the running workflow's own, each other backend's SessionReader, opened and closed with the look): rows merged newest first, each naming its backend; where(id), delete; a backend not read (its extra missing, its store failing) left out and named in notes; a failing store warned of in the log at each look the user asks for, once a run for the app's own (warned: the job monitor's, a finished job's) │ │ ├── session_args.py # --continue / --resume → SessionStart (BaseCLIApp.main() adds them; start: new, continue, resume; SessionArgumentParser: --resume without its id names --continue and /resume) -│ │ ├── message_processor.py # WorkflowEvent → ThinkingPromptSession rendering (a sub-agent run's events box lines and transcript tool results under its name, "researcher-2 › …": the event's run, _at_work; the main agent's unchanged); Ctrl+C stops the request being processed (its boxes end with it; "Operation cancelled..." once, then the background jobs it submitted that still run, named; CancelledError re-raised); tool questions as dialogs, one at a time (a sub-agent run's titled with its instance, "researcher-2 · Permission" or "researcher-2 asks", a question's text opening with its run_label; one whose asking task alone is cancelled, a run stopped by its limit, its parent or the user, withdrawn and the events box back, the turn going on); process_compact: /compact run as a turn (compact() as its event source, live events merged in); a success line for each memory write (save_memory, update_memory, delete_memory); a failed turn said once, from its ERROR as from a raise ("Rate limited: …" for rate_limited, "Send your message again" for a ConversationChangedError) +│ │ ├── message_processor.py # WorkflowEvent → ThinkingPromptSession rendering (a sub-agent run's events box lines and transcript tool results under its name, "researcher-2 › …": the event's run, _at_work; the main agent's unchanged; a run's AGENT_RUN queued, input_required and running each write its line: "Queued: waiting for a free slot", "Waiting for your answer", "Working..."); Ctrl+C stops the request being processed (its boxes end with it; "Operation cancelled..." once, then the background jobs it submitted that still run, named; CancelledError re-raised); tool questions as dialogs, one at a time (a sub-agent run's titled with its instance, "researcher-2 · Permission" or "researcher-2 asks", a question's text opening with its run_label; one whose asking task alone is cancelled, a run stopped by its limit, its parent or the user, withdrawn and the events box back, the turn going on); process_compact: /compact run as a turn (compact() as its event source, live events merged in); a success line for each memory write (save_memory, update_memory, delete_memory); a failed turn said once, from its ERROR as from a raise ("Rate limited: …" for rate_limited, "Send your message again" for a ConversationChangedError) │ │ ├── settings_command.py # /settings: a Dialog with Save and Cancel; the fields that changed → apply_settings, then save_settings(keys=changed) (an override that cannot take `keys` fails there: "Settings applied but not saved: …") │ │ ├── settings_introspection.py # Pydantic field → thinking-prompt form field (built complete: they are frozen) │ │ ├── settings_ui.py # SettingsUI(app), what BaseCLIApp's _build_ui_items/apply_settings/save_settings delegate to (it calls the app's hooks: get_ui_setting_keys, get_theme, start_new_session; keeps nothing): the fields (the model pickers: the list, catalog.offered with model_services.role_models, as /models has it; a role's model the list leaves out offered first as " (current)", "Not offered ()" starting the description, so a Save changes no model untouched); a Save all or nothing, each key as settings_persistence declares (an orchestrator change, check_backend first, starts a new conversation on the new backend through the app's start_new_session() when the open one is fresh, ConversationCore.fresh read under the turn lock, else "(applies to the next /new)"; while the workflow is down a fresh one moves there, _move_here, decided again under the turn lock); Ctrl+C wherever a Save waits says "Operation cancelled..." once (_says_so_when_stopped; a new conversation's own, as /new says it) and undoes it; the save on a worker thread │ │ ├── waiting_jobs.py # Which finished jobs wait to be told in the open conversation: one rule (JobRecord.awaiting_resume, then its own, no conversation's, or a gone conversation's: can_resume, then the other installed backends' readers, opens_anywhere: one another backend keeps is not gone, its job waits for it) for the deliveries, ↻N and the startup line │ │ └── usage_tracker.py # Token usage / status bar: ctx = the main agent's last prompt over its context_trigger (a call with a purpose, a sub-agent's, a summary's or a side call's, never moves it); the token segment the open conversation's total, every call of its turns; format_usage: /status's rows by agent and /agents 's tokens by model │ ├── workflow/ -│ │ ├── base_manager.py # BaseWorkflowManager (abstract; subagents=: its SubAgentPolicy, subagent_policy, read-only, kept for its life; _delegations: its Delegations (agents/delegate.py), made on first use over agent_runs, the policy, the agents' configs, the usage ledgers, _run_agent, the backend's hook that runs one sub-agent run, and _stored_instances, the one that says how many instances of each type a conversation's store keeps (the base: none); _load_models: load the catalog, then resolve every model (the configs', then _resolve_extra_models: a backend's agents built without configs, leniently), once, no retry (the background refresh started after a successful init checks the models the settings name); _compute_budgets right after it, before any service is built: each agent's context numbers, ContextBudgetError for a window too small to work in, for a backend that says _manages_context (ADK, whoever built its agents, and Deep Agents; LangGraph has none); the lifecycle: cleanup, reinitialize and the one release, _release_resources: the base's state reset, the backend's hooks _detach_backend/_detach_store (skipped while _keeping keeps the store), the services detached and their closes started on a worker thread, then every closer and the services waited for last (shielded), in one shielded task; the turn: _turn, every turn's entry (log context, _turn_admission, _workflow_context; the sub-agent runs it leaves open, a cancel's or a failure's, recorded interrupted as it ends: agent_runs, workflow/agents/runs.py), and the templates over the backend's hooks: resume_with_job_result (the job's result read on a worker thread, job_notice → _deliver_job_notice), can_resume (_record_resumable, _conversation_exists), compact (→ _compact_turn), supports_job_resume/supports_compact derived from them; services (services.py's ServiceSet behind the forwards: _services, _build_services(_into), _get_service_tool_map, _close_services, _keeping) and the hand-over: lend_services → initialize_services(services=loan) → give_services (no turn while a manager holds services it does not own); service detection, tool assembly; context_budget/root_agent_name; the usage ledgers (usage: the open conversation's, usage_of; _turn counts each turn in its conversation's and gives a run a cancel leaves open what it used) and side calls (_report_side_call: generate_simple's usage, its purpose from usage.side_call, summarize's tool, on_session_end's memory_extraction outside any turn, recorded in its conversation's ledger and, in a turn, reported through its live events, never to a turn that has ended: turn_has_ended); summarize and fact extraction: a prompt over the default model's room, at the resolved max_tokens, cut in the middle, never under MIN_PROMPT_ROOM, _fit_prompt; the memory index behind the forwards _memory_index_now/_memory_index_cached/_memory_index_cap/_forget_memory_index and _index_snapshots (memory_index.py's MemoryIndex; the release clears it)) +│ │ ├── base_manager.py # BaseWorkflowManager (abstract; subagents=: its SubAgentPolicy, subagent_policy, read-only, kept for its life; _delegations: its Delegations (agents/delegate.py), made on first use over agent_runs, the policy, the agents' configs, the usage ledgers, _run_agent, the backend's hook that runs one sub-agent run, and _stored_instances, the one that says how many instances of each type a conversation's store keeps (the base: none); request_user_input: a sub-agent run waits on the user with its working slot lent (agents/delegate.py::waiting_on_user), and can_request_user_input says whether anyone could answer now (no callback, or its turn ended: nothing is lent); _load_models: load the catalog, then resolve every model (the configs', then _resolve_extra_models: a backend's agents built without configs, leniently), once, no retry (the background refresh started after a successful init checks the models the settings name); _compute_budgets right after it, before any service is built: each agent's context numbers, ContextBudgetError for a window too small to work in, for a backend that says _manages_context (ADK, whoever built its agents, and Deep Agents; LangGraph has none); the lifecycle: cleanup, reinitialize and the one release, _release_resources: the base's state reset, the backend's hooks _detach_backend/_detach_store (skipped while _keeping keeps the store), the services detached and their closes started on a worker thread, then every closer and the services waited for last (shielded), in one shielded task; the turn: _turn, every turn's entry (log context, _turn_admission, _workflow_context; the sub-agent runs it leaves open, a cancel's or a failure's, recorded interrupted as it ends: agent_runs, workflow/agents/runs.py), and the templates over the backend's hooks: resume_with_job_result (the job's result read on a worker thread, job_notice → _deliver_job_notice), can_resume (_record_resumable, _conversation_exists), compact (→ _compact_turn), supports_job_resume/supports_compact derived from them; services (services.py's ServiceSet behind the forwards: _services, _build_services(_into), _get_service_tool_map, _close_services, _keeping) and the hand-over: lend_services → initialize_services(services=loan) → give_services (no turn while a manager holds services it does not own); service detection, tool assembly; context_budget/root_agent_name; the usage ledgers (usage: the open conversation's, usage_of; _turn counts each turn in its conversation's and gives a run a cancel leaves open what it used) and side calls (_report_side_call: generate_simple's usage, its purpose from usage.side_call, summarize's tool, on_session_end's memory_extraction outside any turn, recorded in its conversation's ledger and, in a turn, reported through its live events, never to a turn that has ended: turn_has_ended); summarize and fact extraction: a prompt over the default model's room, at the resolved max_tokens, cut in the middle, never under MIN_PROMPT_ROOM, _fit_prompt; the memory index behind the forwards _memory_index_now/_memory_index_cached/_memory_index_cap/_forget_memory_index and _index_snapshots (memory_index.py's MemoryIndex; the release clears it)) │ │ ├── factory.py # create_workflow_manager_from_settings (_BACKENDS: orchestrator → "module:Class" + extra, imported when chosen; orchestrator=: another backend than the settings'; session_store=: the store a manager is built for, BaseWorkflowManager.session_store, the settings' by default; subagents=: the SubAgentPolicy the manager gets, checked before a backend is imported); orchestrator_of; check_backend (an unknown name, a missing extra) │ │ ├── session_readers.py # SessionReader: a backend's stored conversations read without its manager (list/info/exists/delete/aclose, rows naming the backend; delete deletes what the manager's does, ADK's sub-agent transcripts with the conversation), opened from the settings and a store (open_session_reader: the backend's own session code, adk/sessions.py::open_sessions and deepagents/sessions.py::open_sessions, imported on a worker thread; no agents, models or services; a sqlite store not made yet reads empty and is not made, Deep Agents not even imported for it; a database without its backend's tables, looked at read-only, reads empty, nothing written to it: no table, no journal mode); none for the memory store or LangGraph; ImportError (the install hint) for a missing extra; reader_backends │ │ ├── service_registry.py # get_service/require_service + ContextVar registry; the registry keys and KNOWN_SERVICE_KEYS, read from services.py's table │ │ ├── services.py # The services a manager builds: SERVICES, one ServiceSpec each (key, build, close, tools, declarable, always, built_with, rebind; builders import lazily); ServiceSet: one manager's (build_into, publish, carried, release, tool_map, not_owned: borrowed or given away, never closed there); ServiceLoan: a hand-over between managers (rebind: the engine's workflow/per_call, the KB's summarizer) │ │ ├── memory_index.py # MemoryIndex: the memory index the root's requests carry, one snapshot per conversation (now: taken once, off the event loop; cached: never reading; forget: /resume, a summary and /compact; clear: the release; cap: memory_index_max_bytes, or a tenth of the root's budget: MEMORY_INDEX_BUDGET_SHARE, MEMORY_INDEX_MIN_BYTES); reads the manager's settings, memory store, the root's memory tools (_root_memory_tools) and its room live, through callables, and takes a snapshot at the manager's _memory_index_cap (limit=), so a subclass's cap is the one asked; imports no backend │ │ ├── outside_content.py # OutsideContentMarks (per conversation, first tool wins), brings_outside_content, the note -│ │ ├── events.py # WorkflowEvent, EventType; WorkflowEvent.turn_failed: a failed turn's one ERROR (rate_limited, error_type); WorkflowEvent.run: the agent run that wrote it (RunRef; None: the manager's own); AGENT_RUN (WorkflowEvent.agent_run): a sub-agent run's start (brief=, started_by=, mode=) and end (an end with what the run used, by model: usage=; cancelled_by= for a cancelled run); UserInputRequest.run: the sub-agent run that asks (request_user_input fills it from get_active_run()) +│ │ ├── events.py # WorkflowEvent, EventType; WorkflowEvent.turn_failed: a failed turn's one ERROR (rate_limited, error_type); WorkflowEvent.run: the agent run that wrote it (RunRef; None: the manager's own); AGENT_RUN (WorkflowEvent.agent_run): a sub-agent run's start (brief=, started_by=, mode=), how it works meanwhile (queued, running, input_required) and end (an end with what the run used, by model: usage=; cancelled_by= for a cancelled run); UserInputRequest.run: the sub-agent run that asks (request_user_input fills it from get_active_run()) │ │ ├── live_events.py # LiveEvents, merged: events reported inside a turn (reporting/report) reach its stream as they happen; on_stop: a consumer that stops early, before the source is cancelled; emit gives an event that names no run the active one (get_active_run) │ │ ├── usage.py # Token usage: normalized_tokens, the one place a provider's report becomes the counts every LLM_USAGE carries (prompt_tokens every input token, cache included; completion_tokens every output token, thinking included): adk_tokens (ADK's, by provider kind: Gemini's and ADK's Claude adapter's thinking apart from the candidates, OpenAI's inside; an unknown kind's apart), adk_usage_event (the one place ADK's usage becomes an LLM_USAGE: the event processor, the summarizer, generate_simple, the raw log), langchain_tokens (every partner package's, langchain-openai's priority and flex tiers' parts too), llm_usage_event; PURPOSE_* (none: the root's step; subagent, summarization, tool, memory_extraction, side_call) and side_call (what a caller's side calls are for); UsageLedger (a conversation's calls, a row each, each with its latency_ms: totals by run, agent, model, purpose and turn; a row's turn its conversation's while a turn of it runs, as the manager counts them, None in another conversation's turn or a task its turn outlived; by_agent, by_purpose, by_model, run_usage), UsageLedgers (a manager's, by conversation, filed under the turn a call is made in; for the manager's life: a backend switch's new manager starts afresh) │ │ ├── interrupts.py # What a call a cancel or a crash left open is answered with; OpenCalls (each manager's record, kept by conversation: only that conversation's turns clear it; within it by the run that made each call, run=, the active run by default); unrecorded: a call no record holds; job_for_call: a job of the turn's conversation and of the run whose call it answers (started_by) only @@ -78,11 +78,11 @@ agentic-cli/ │ │ ├── sessions.py # SessionRef; SessionInfo (a session list's row; backend: a reader's rows and the CLI's name theirs) and session_title; ConversationChangedError (ADK's stale session); the active turn; the active sub-agent run (RunRef, root_run, get_active_run: None in the root's own calls); the active tool call (ToolCallRef); the turn's span (open_turn_scope/close_turn_scope, turn_has_ended: a task that outlived its turn; current_turn_scope: the span itself, the identity of what is kept for one turn; runs_this_turn/count_run_this_turn: the sub-agent runs it started, at any depth); instance_of (a run's instance, None for the root's: a job's started_by) │ │ ├── mcp.py # MCPServerConfig → to_adk_toolset / to_langchain_connection │ │ ├── permissions/ # Framework-independent capability engine -│ │ │ ├── engine.py # PermissionEngine (deny-wins, default-ASK; uncontained=, per_call=: capabilities approved for each call) +│ │ │ ├── engine.py # PermissionEngine (deny-wins, default-ASK; uncontained=, per_call=: capabilities approved for each call; one prompt at a time, _ask_lock, a sub-agent run waiting with its working slot lent from before the lock until after it: waiting_on_user; nothing lent for a prompt no one can be shown: can_request_user_input) │ │ │ ├── capabilities.py # Capability, EXEMPT │ │ │ ├── matchers.py # PathMatcher, URLMatcher, ShellMatcher, StringGlobMatcher │ │ │ └── rules.py, store.py, prompt.py (build_request(run=): a sub-agent's prompt opens with its run; is_permission_request: by its perm- id) -│ │ ├── agents/ # Sub-agents: runs.py: the RUN_* statuses, RunRecord, AgentRuns (a manager's runs by conversation: run_for names - and its run #1 per (conversation, parent run, call id), a new one once it ended; number_from: past the instances a conversation's store keeps, a count never lowered; observe records AGENT_RUN under the turn it is reported in; interrupt_open; a record's usage: what the run used, by model, from its end event or the ledger), run_title, result_head, run_instance and run_number (the one reader of a run id: researcher-2#1's instance and number), run_label (how a dialog names a run: researcher-2 (researcher, run 1)), SUBAGENT_ERROR_PREFIX (ADK's old words for a failed delegation, kept for callers), show_as_delegation(result, envelope) (a delegation's TOOL_RESULT shown alike on both backends: the result's first line, or Failed: ; metadata["result"] the envelope); a record's brief, started_by, mode and cancelled_by; delegate.py: Delegations (delegate(DelegationRequest) → Delegated: the one place a run starts and ends, on both backends: the brief read against the caller's own sub-agents, the run named (a turn's first run past the instances its conversation's store keeps: stored=, asked once per conversation per turn, again after a failed or stopped ask), its AGENT_RUN start and end observed and reported, the backend's hook (RunHook: AgentRun → RunAnswer) run in a task of its own with the run active and no tool call, the end by the answer (completed, partial for a cut or the step limit, refused, failed for an error or a raise, with one recoverable ERROR failed: , or RunFailed's own words, cancelled by its control with what the hook had), interrupted on its turn's cancel, which wins; the envelope as the result; each run under its RunControl, the type's limits over the policy's, kept while it works: control(run), cancel(run_id, by=, error=), the run alone; a call past max_runs_per_turn the error result RUNS_PER_TURN_ERROR, no run), DelegationRequest, AgentRun (control), RunAnswer, RunFailed, started_by; control.py: RunControl (one run's: cancel(by=, error=) its task, the first cancel winning, own_cancel for the hook that takes it and returns, timing: the time limit while the hook runs; model_call/at_step_limit/final_call: the step limit, STEP_LIMIT_NOTE on the last allowed call; used: the token limit, an answer past it that makes calls the run's last, cancelled by it, one that makes none its result (calls=); model_call once final stays final; a cancel leaves a task its turn is cancelling alone; steer/take_steering), STEP_LIMIT_NOTE; outcome.py: a sub-agent's refused or cut answer (outcome_of, the refusal notice as Deep Agents 0.7.34 words it, cut_off, cut_off_warning, OPENAI_REFUSAL_PREFIX: how ADK's OpenAI adapters word a refusal); policy.py: SubAgentPolicy (max_active, max_runs_per_turn, limits) and AgentLimits (max_model_calls, timeout, max_tokens; over: a type's limits over the policy's, field by field, None taking the base's), DEFAULT_LIMITS (100 model calls), frozen and checked as they are made (a ValueError naming the field), as_policy (the one reading of a subagents= argument: a TypeError for anything but a policy or None); brief.py: BRIEF_FIELDS, Brief, BriefError, read_brief (a task call's agent type and brief, against the types its caller may start: the error texts), task_message (a run's first message, a section per filled field; an inheriting type's ends with INHERITED_CONTEXT_NOTE), task_parameters and task_description (task's schema and description, the same for both backends; limit_sentences: the limits it states, "At most runs per turn."); envelope.py: envelope (a delegation run's result: success only for completed, a one-line error, cancelled_by for a cancelled run), read_envelope (a dict or its JSON text, else None; never raises), the limit and interrupted error texts +│ │ ├── agents/ # Sub-agents: runs.py: the RUN_* statuses (a working run's: RUN_QUEUED, RUN_RUNNING, RUN_INPUT_REQUIRED, RUN_WORKING_STATUSES; a record says running from its start), RunRecord, AgentRuns (a manager's runs by conversation: run_for names - and its run #1 per (conversation, parent run, call id), a new one once it ended; number_from: past the instances a conversation's store keeps, a count never lowered; observe records AGENT_RUN under the turn it is reported in; interrupt_open; a record's usage: what the run used, by model, from its end event or the ledger), run_title, result_head, run_instance and run_number (the one reader of a run id: researcher-2#1's instance and number), run_label (how a dialog names a run: researcher-2 (researcher, run 1)), SUBAGENT_ERROR_PREFIX (ADK's old words for a failed delegation, kept for callers), show_as_delegation(result, envelope) (a delegation's TOOL_RESULT shown alike on both backends: the result's first line, or Failed: ; metadata["result"] the envelope); a record's brief, started_by, mode and cancelled_by; delegate.py: Delegations (delegate(DelegationRequest) → Delegated: the one place a run starts and ends, on both backends: the brief read against the caller's own sub-agents, the run named (a turn's first run past the instances its conversation's store keeps: stored=, asked once per conversation per turn, again after a failed or stopped ask), its AGENT_RUN start and end observed and reported, the backend's hook (RunHook: AgentRun → RunAnswer) run in a task of its own with the run active and no tool call, the end by the answer (completed, partial for a cut or the step limit, refused, failed for an error or a raise, with one recoverable ERROR failed: , or RunFailed's own words, cancelled by its control with what the hook had), interrupted on its turn's cancel, which wins; the envelope as the result; each run under its RunControl, the type's limits over the policy's, kept while it works: control(run), cancel(run_id, by=, error=), the run alone; a call past max_runs_per_turn the error result RUNS_PER_TURN_ERROR, no run; each run in a working slot of its conversation's SlotPool, slots(conversation): policy.max_active slots, made on first use and kept, the root taking none: taken right before its hook and its time limit, queued then running while it waits for one, given back at its end, straight back to a caller that waits on it alone; a caller that delegates lends its slot while it waits for its children), DelegationRequest, AgentRun (control), RunAnswer, RunFailed, started_by, waiting_on_user (a sub-agent run waiting on the user: input_required, its slot lent, then running; nested waits once; request_user_input and the permission engine); slots.py: SlotPool (acquire(priority): a free slot, else a future in a heap by priority, RETURNING before NEW, then arrival; release; lent(slot): the slot lent while its run waits, nested lends once, taken back at RETURNING after, at most one take-back waiting for a slot and none once it is held (a wait that starts while one is queued takes it over, and one handed the slot withdraws any queued since), nothing taken back after a cancel; hand_back(slot, to): a child's slot to its parent waiting on it alone; free, waiting; a cancelled waiter leaves the heap, one handed a slot as it is cancelled hands it on; not asyncio.Semaphore), Slot (held, lent), RETURNING, NEW; control.py: RunControl (one run's: cancel(by=, error=) its task, the first cancel winning, own_cancel for the hook that takes it and returns, timing: the time limit while the hook runs; model_call/at_step_limit/final_call: the step limit, STEP_LIMIT_NOTE on the last allowed call; used: the token limit, an answer past it that makes calls the run's last, cancelled by it, one that makes none its result (calls=); model_call once final stays final; a cancel leaves a task its turn is cancelling alone; steer/take_steering), STEP_LIMIT_NOTE; outcome.py: a sub-agent's refused or cut answer (outcome_of, the refusal notice as Deep Agents 0.7.34 words it, cut_off, cut_off_warning, OPENAI_REFUSAL_PREFIX: how ADK's OpenAI adapters word a refusal); policy.py: SubAgentPolicy (max_active, max_runs_per_turn, limits) and AgentLimits (max_model_calls, timeout, max_tokens; over: a type's limits over the policy's, field by field, None taking the base's), DEFAULT_LIMITS (100 model calls), frozen and checked as they are made (a ValueError naming the field), as_policy (the one reading of a subagents= argument: a TypeError for anything but a policy or None); brief.py: BRIEF_FIELDS, Brief, BriefError, read_brief (a task call's agent type and brief, against the types its caller may start: the error texts), task_message (a run's first message, a section per filled field; an inheriting type's ends with INHERITED_CONTEXT_NOTE), task_parameters and task_description (task's schema and description, the same for both backends; limit_sentences: the limits it states, "At most sub-agents work at once; more calls in one response wait for a free slot. At most runs per turn."); envelope.py: envelope (a delegation run's result: success only for completed, a one-line error, cancelled_by for a cancelled run), read_envelope (a dict or its JSON text, else None; never raises), the limit and interrupted error texts │ │ ├── adk/ # ADK orchestrator │ │ │ ├── manager.py # GoogleADKWorkflowManager (the agents built from configs or, adk_config_path, by ADK from a native YAML (library-only: no BaseCLIApp starts one; loaded once per manager at the first _validate_agent_graph (_native_root: the settings' keys exported first; a retry, a model switch and a start after cleanup build on the same tree, whose toolsets the closed runner left reusable), its agents' models resolved leniently in _resolve_extra_models, the base's hook, its registered tools' services detected; configs passed with it ignored, adk_native_configs_ignored): the only difference; _do_initialize builds the app from the tree alone, _build_app → ADKBuild, through the hooks _plugin_chain (the whole chain, top to bottom, its head _init_plugins), _compaction_config (None for a root without a budget), _context_cache_config, _agent_model_refs; the event processor built from the build; a config-built agent's prompt as an instruction provider, sent as written: no `{name}` state injection; _compact_turn: ADK's compaction once, in the base's compact turn; supports_compact False once built for a root without a budget (_no_compact_support says why); the job resume hooks: _conversation_exists (the session read without its events, num_recent_events=0), _deliver_job_notice (the notice, for every app); _run_config: the agents' synchronous tools run in ADK's tool thread pool, 8 workers; _run_and_stream: a run that fails yields the turn's one ERROR, last, none after an ERROR already yielded; the agents: _create_agents builds each config's agent on its own (_agents, by name; no ADK sub_agents), a delegating one with a TaskTool (its own tool named task: AgentGraphError); a runner per sub-agent type (_agent_runners: NoAfterTurnCompactionRunner on App(name=child_app(app_name), root_agent=the type's agent, plugins=_agent_plugin_chain(agent), the app's context cache, no compaction) over the session service, closed with the main runner by _detach_backend, rebuilt by every initialization); _run_agent, the base's hook: the run's child session (_child_session: run.state, an inheriting run's copied events; never one the store keeps, whose instance numbering goes past: ADK's create_session refuses it, and the run fails), _stored_instances (the base's hook: the highest instance number of each type the conversation's child sessions keep), the type's runner on the task message with an abort signal set before it is closed, a run-mode ADKEventProcessor, the raw log's records reported as they come (_report_raw_log); a model's failure raised as the provider's (_plugin_cause), an overflow RunFailed, a reply that is no answer RunAnswer(error=), the shared keys it changed read back) │ │ │ ├── build.py # ADKBuild (root, reply_agents, plugins, compaction, cache, models); load_native_root (from_config at validation; what ADK cannot load, wrapped as AgentGraphError naming the file, ADK's error the cause; a root that is not an LlmAgent, a name used twice in the tree and a task-mode agent refused, AgentGraphError naming it; ensure_direct_anthropic); what the run reads of any tree: reply_agents (the root and the chat-mode agents it can transfer to), own_after_tool_agents (left out by the cap) @@ -251,7 +251,7 @@ Workflow: - **Tool error handling**: All tools return `{"success": bool, ...}` dicts. Never raise `ToolError`. An exception that still escapes a tool is a bug, but it costs one call rather than the turn: on ADK, `workflow/adk/tool_error_plugin.py::ToolErrorPlugin` turns it into an error dict; on Deep Agents, `workflow/deepagents/middleware.py::ToolCallErrorMiddleware`, inside the permission gate, gives the same result, `{"success": false, "error": " failed: : "}` cut to 500 characters (both use `workflow/tool_errors.py`). A call to a tool name the agent does not have gets ADK's own text, its first two lines (`unknown_tool_result`: "nope failed: ValueError: Tool 'nope' not found." and the agent's tools by name), logged `tool_not_found` at info (the model's mistake): on ADK the permission gate lets ADK's stand-in through (`is_unknown_tool`: a bare `BaseTool` described "Tool not found", which runs nothing; contract-tested) and `ToolErrorPlugin` answers its lookup error; on Deep Agents the gate answers it. It catches `Exception`, never `BaseException`, so cancelling still cancels. `task` raises nothing into it: a sub-agent's model or run failing (its summary or overflow included: its own tools' errors are results already) fails the run, whose envelope its parent reads (`error`: the provider's reason; on ADK out of ADK's plugin wrapper), reported at once through the turn's live events as one recoverable `ERROR`, ` failed: `, the run's: the coordinator carries on. A sub-agent's overflow (still too long after its summary or trim) fails its run, on both backends, in the words of ADK's overflow handler, `: the request no longer fits 's context window.` with `error_code` `CONTEXT_OVERFLOW` (`context.py::subagent_overflow_message`): ADK's `OverflowPlugin` answers the run's model call with it, Deep Agents' manager raises it (`RunFailed`). A refusal is no exception here (the framework adds no `ModelRefusalMiddleware`): a sub-agent whose final answer a model refused ends its run `refused`, and its parent reads, as its envelope's result, Deep Agents 0.7.34's notice ("The subagent stopped because a model refused the request. … Retrying the same request is unlikely to succeed; rephrase it or continue without it."), on both backends and whichever deepagents the pin installs (0.7.28 has no notice of its own; 0.7.34+ finds no refusal block left to word again); an answer cut at the output limit keeps its text, followed by `[cut off at the output limit]`, and its run ends `completed` and `partial`, with one recoverable warning ("'s answer was cut off at the output limit"); the answer's thoughts or reasoning are kept. ADK's `SubAgentOutcomePlugin` and Deep Agents' `SubAgentOutcomeMiddleware` rewrite and mark the answer, by `workflow/agents/outcome.py`. On ADK the plugin reads the answer's `finish_reason` (Claude's `refusal` and a Chat Completions `content_filter` arrive as `SAFETY`, `max_tokens` and `length` as `MAX_TOKENS`, Gemini's as themselves), the `error_code` ADK sets on an answer with no content (a Gemini prompt's `block_reason`), the `reason` an `OTHER` stop names in its JSON error message (ADK's OpenAI Responses adapter's `incomplete_details`, `content_filter` among them) and a text part `OpenAI refusal: ` (that adapter's refusal, and `OpenAICompatibleLlm`'s of a Chat Completions `refusal`, which ADK's class drops). On Deep Agents the middleware reads `response_metadata`'s `stop_reason` (ChatAnthropic), `finish_reason` (ChatOpenAI, ChatGoogleGenerativeAI), `incomplete_details.reason` (the Responses API) and `prompt_feedback.block_reason` (a Gemini prompt block), and a refusal the answer holds (a refusal block, standard or `non_standard`, or `additional_kwargs["refusal"]`). The root's own answers reach the user as their model gave them: on ADK a root's refusal with no answer (Claude's, Gemini's) is still the turn's one `ERROR` (" returned no answer"), and a root on OpenRouter or an OpenAI-compatible server replies "OpenAI refusal: ". On both backends the root's own failure leaves the run: the turn's one `ERROR`. A gate or error result goes back as the call's `ToolMessage` with `status="error"`. LangGraph's `ToolNode` runs with `handle_tool_errors=True`. - **Tool result cap**: every tool result passes `workflow/tool_output.py::cap_tool_result` (registered, MCP and skill tools, `job_result`, a delegation's envelope on either backend): one over `tool_result_max_chars` is cut to its shape and as much of its head and tail as fits (the kept length is bisected, since JSON escapes vary), with a note (under `truncated`, or appended to a string) pointing at its full output in `./.{app_name}/tool-output/`, written for `read_file` to page by lines (lines over 2,000 characters split). A string that holds a JSON object (`json_object`: a delegation's envelope, which both task tools write unescaped, `ensure_ascii=False`; a framework tool's dict in a Deep Agents `ToolMessage`) is cut as that dict and given back as its JSON text, so the parent reads JSON still, on both backends. On ADK, `workflow/adk/tool_result_cap_plugin.py::ToolResultCapPlugin` is registered after `InterruptedCallsPlugin`, since ADK stops at the first `after_tool_callback` that returns a value, and it leaves out an agent with `after_tool_callback`s of its own (a native YAML app's, `build.own_after_tool_agents`), which ADK would skip once the cap answered. ADK's Claude and OpenAI Responses adapters send only a result's `content`, else its `result`, so the plugin also puts the note there (`_note_where_sent`). On Deep Agents, `ToolResultCapMiddleware` sits between the permission gate and `ToolCallErrorMiddleware`, so an error result is capped too, and reads a `ToolMessage` (its text cut by the cap's string rule, text blocks joined into the first) and each `ToolMessage` of a `Command`. A call of the framework's `read_file` (by identity, as each gate resolves tools) on a saved output is cut without a second save. Saving never fails the call (`tool_output_not_saved` warns once per folder, then debug), and a cap that raises leaves the result uncut (`tool_result_cap_failed` at warning, its traceback at debug). - **Summarizing on ADK (main agent only)**: every app's App (a native ADK YAML app's too, when its root's model has a context budget) has ADK's `EventsCompactionConfig` with `workflow/adk/compaction.py::AgentCliSummarizer` at the root's trigger (`context_budget`, as on Deep Agents); ADK checks before every agent's model call, and `MainAgentPlugin` (the first plugin) records the running agent in a ContextVar, so the summarizer writes only for the root (a sub-agent's run, on its type's runner, has no compaction config at all: a sub-agent compacting the shared session lost its brief and looped, 100 requests in one turn, offline). The summarizer picks its own range before paying for anything (`summary_range`: the newest `keep` tokens and the current turn kept, a cut only where a user turn starts, never parting a call from its result, long-running calls excepted since ADK re-injects them, nor leaving a kept event stamped inside the range; none for a saving under 20% of the trigger, the seed left out of it), writes on the root's own model (`_simple_model(SUMMARY_MAX_TOKENS, agent=root)`, thinking off) with `workflow/summary_prompt.py`, the rendering sized to that model's window (the middle cut, `MIDDLE_OMITTED`, the seed kept whole), and returns **user-role** text starting `SUMMARY_PREFIX` (a model-role first message is one Claude refuses), which the fit guard folds, after the memory index when the request has one, into the next user text (`fold_summary`), so no request holds two user messages in a row. `last_kept_turns` and `last_error` are for its callers: a manual summarizer warns of nothing itself. It reports through the turn's live events (`_run_and_stream` iterates the runner through `live_events.merged` under `reporting(live)`); any failure is one recoverable `ERROR` a turn, never raised to ADK. `NoAfterTurnCompactionRunner` overrides ADK's private `Runner._run_post_invocation_compaction` (contract-tested), so the prompt never waits for a summary after the answer. `FitGuardPlugin` is the last plugin (but `SentRequestLoggingPlugin`, which logs a request as sent, and `CallSentPlugin`, which starts the call's clock) and `OverflowPlugin` the one before it (`MemoryIndexPlugin`, when the root has a memory tool and `memory_index_enabled` is on, the one before that: it puts the conversation's memory index snapshot first in the root's requests, never in the session): the guard leaves a request's oldest turns out when it is over its agent's budget (keeping the newest turn and what opens the conversation, the memory index then a summary: `protected_head`) and keeps each request as it came in (`request_as_seen`: one copy per model call, under the call's `EventActions`, which ADK passes to all three of its callbacks: each sub-agent run is an invocation of its own on a child session, but two runs of one type share that type's runner and its plugins, and a run drops only what its own cancelled calls left, as it ends), and the overflow handler, on a provider's overflow, summarizes now (the root, through `compact_session`, ADK's private `_run_compaction_for_token_threshold_config`; `_summarize_now` returns `WrittenSummary(text, kept_turns)`, and the retry is the index, the summary and those turns) or trims to half the budget (a sub-agent, or nothing to summarize), builds the retry from the guard's copy without the context cache, retries once on the agent's `canonical_model`, then answers the root with `LlmResponse(error_code="CONTEXT_OVERFLOW")`, never model text (ADK would store it as the reply), which `ADKEventProcessor` maps to the turn's `ERROR`; a retry that fails otherwise propagates, and `_run_and_stream` unwraps ADK's plugin-manager `RuntimeError` to the provider's error; a sub-agent's call that still does not fit is answered with an error response too, `CONTEXT_OVERFLOW` and `subagent_overflow_message`, which its run's processor keeps as the run's failure: the run ends `failed` with those words (`RunFailed`), its parent reads them in the envelope, and the user is warned once, by the framework's delegation (`agents/delegate.py`), as on Deep Agents. A native ADK YAML app gets all of it: only the building of its agents differs (`workflow/adk/build.py`). Its chat-mode sub-agents keep transferring, and an agent the root transfers to replies to the user (`build.reply_agents`: its text is the reply, recent_messages reads it, its overflow is the turn's clear error, its calls write the outside-content mark); the summary and the memory index stay the root's, so such an agent is trimmed by the fit guard, not summarized. An agent's own callbacks (a native app's) run after every plugin, and only when none answered: its before-model callbacks see the request after the fit guard, and the cap leaves out an agent with after-tool callbacks. A main agent whose model does not resolve through the catalog has no budget and no summary. -- **Sub-agents delegate and return**: on both backends a sub-agent run is a call of its parent's `task` tool (the framework's: `tools/_core/delegation.py` declares it, `workflow/agents/delegate.py` runs it) that returns in the same turn; only the root replies to the user (a sub-agent's `TEXT` carries `metadata["subagent"]`, and the CLI shows it as the events box's status line, under its run's name: `researcher-2 › …`; `/agents` lists a conversation's runs), and the next turn starts at the root; fact extraction reads only the user's and the root's text. `task` takes `agent_type` (one of the caller's own sub-agents) and a brief (`workflow/agents/brief.py`: `title`, `objective`, `deliverable`, `constraints`, `known`, `done_when`), its schema and description the same on both backends (`task_parameters`, `task_description`: a line per type with its description, "The agent." with a warning, `subagent_without_description`, for one without, and the fields `AgentConfig.brief_required` asks for); `read_brief` checks a call against them, and one that cannot start a run is the call's `{"success": false, "error": …}`, no run started or numbered. `Delegations.delegate` (the manager's `_delegations`) does the rest, alike on both: it names the run (`AgentRuns.run_for`), reports its `AGENT_RUN` start (title, brief, `started_by`, `mode`) and end through the turn's live events and the records, and runs the backend's hook, `_run_agent(AgentRun)`, in a task of its own, where the run is the active run and no tool call is, so each run ends at its own end. The run's first message is the task message (`task_message`); it starts from the conversation's `plan` and `tasks` (`AgentRun.state`), and the keys it changed come back to its caller's state (`RunAnswer.state`). It ends as the `RunAnswer` says: its last text (a reply that also called a tool included; none at all, an empty result shown `No answer`), `refused` or `partial` by the outcome mark, `failed` for an error or a raise (one recoverable `ERROR`, ` failed: `, or the hook's own words, `RunFailed`: an overflow's); and the call's result is the run's envelope as JSON (`workflow/agents/envelope.py`), shown by `show_as_delegation`. A cancel of the caller's task stops the run's task first and ends the run `interrupted`. Each run works under its `RunControl` (`workflow/agents/control.py`, `AgentRun.control`), with its type's `AgentConfig.limits` over the policy's (`SubAgentPolicy.limits`, `DEFAULT_LIMITS`: 100 model calls): `Delegations.cancel(run_id, by=, error=)` cancels the run alone (its parent, the user or a limit), and the hook, seeing its own cancel (`own_cancel`), stops the run as a stopped turn's (ADK: its abort signal, its open calls sealed in its child session; Deep Agents: its stream closed) and returns what it has: the run ends `cancelled` with `cancelled_by` and that text, `partial`, while its siblings and the turn go on; the turn's own cancel still wins (`interrupted`). Past its `timeout` (counted from when its hook starts, `timing`) the run is cancelled by its limit (`time limit reached`); each model call is counted before it is made (`RunControlPlugin` in each sub-agent chain, `RunControlMiddleware` on each sub-agent's model side) and the last allowed one gets `STEP_LIMIT_NOTE` after its last message, its tools still declared (Claude refuses tool calls in a request that defines no tools); its answer's calls are dropped, never run, so the run completes `partial`, `step limit reached`; each call's tokens are counted after it, and past `max_tokens` an answer that makes tool calls is the run's last, its calls dropped, `cancelled` by its limit (`token limit reached`), while one that makes none ends the run as it would, `completed`, its result whole (a model that reports no usage is not limited by tokens). A turn starts at most `max_runs_per_turn` runs, at any depth (`sessions.runs_this_turn`): a call past it is the error result, no run started; `task`'s description says so (`limit_sentences`). A message steered to a working run (`RunControl.steer`, through `Delegations.control(run)`) is added after its last message, as the user's, before its next model call, and stays in its transcript (ADK: appended to its child session, the run's later events stamped after it so a durable store reads them back in order; Deep Agents: added to its state); a sub-agent run's task message is never trimmed, on ADK neither by the fit guard nor by the overflow handler's retry, whatever was steered after it. A dialog whose run is stopped is withdrawn, the turn going on (`MessageProcessor`). `AgentConfig.inherit_context` sets what a run sees: False (the default) its task alone, True its parent's conversation too, read once as the run starts (the conversation after any summary, less the message that called `task`), before the task message, which then ends with `INHERITED_CONTEXT_NOTE` (whose conversation it reads, which message is its task); `validate_agent_graph` refuses a value that is not True or False, `inherit_context=True` on the root and, in 0.7.0, on an agent with sub-agents or below another sub-agent. On ADK (`workflow/adk/task_tool.py::TaskTool`) every config-built agent is built on its own, with no ADK `sub_agents`, so ADK offers none `transfer_to_agent`, and gets ADK's identity line ("You are an agent. Your internal name is …"); a delegating agent has one `TaskTool`. Each sub-agent type runs on its own runner (`App(name=child_app(app_name))`, the type's agent its root, `_agent_plugin_chain`: the main chain configured for a sub-agent, no memory index, no summary, `RunControlPlugin` and `SubAgentOutcomePlugin`; every plugin's per-call state keyed by invocation) over the manager's session service, on a child session per run (`child_session_id`: `/`; an inheriting run's gets the conversation's events copied, `task_tool.inherited_events`), under an app no conversation list reads. On Deep Agents (`workflow/deepagents/task_tool.py::build_task_tool`) each type's graph is built on its own, and a run `astream`s it in its task with `var_child_runnable_config` cleared, so its chunks reach its own loop alone, never its parent's stream; an inheriting run's history is the parent's messages with Deep Agents' summary event applied (`SummarizationMiddleware._apply_event_to_messages`, private: `tests/workflow/deepagents/test_compact.py` pins it), and `ForeignReasoningMiddleware(kind, agent=...)` leaves out the reasoning of AI messages another agent wrote. A native ADK YAML app keeps transferring (chat-mode sub-agents); a task-mode agent in one is refused at validation, since ADK runs its delegation outside the tool pipeline and keeps the call open across turns. +- **Sub-agents delegate and return**: on both backends a sub-agent run is a call of its parent's `task` tool (the framework's: `tools/_core/delegation.py` declares it, `workflow/agents/delegate.py` runs it) that returns in the same turn; only the root replies to the user (a sub-agent's `TEXT` carries `metadata["subagent"]`, and the CLI shows it as the events box's status line, under its run's name: `researcher-2 › …`; `/agents` lists a conversation's runs), and the next turn starts at the root; fact extraction reads only the user's and the root's text. `task` takes `agent_type` (one of the caller's own sub-agents) and a brief (`workflow/agents/brief.py`: `title`, `objective`, `deliverable`, `constraints`, `known`, `done_when`), its schema and description the same on both backends (`task_parameters`, `task_description`: a line per type with its description, "The agent." with a warning, `subagent_without_description`, for one without, and the fields `AgentConfig.brief_required` asks for); `read_brief` checks a call against them, and one that cannot start a run is the call's `{"success": false, "error": …}`, no run started or numbered. `Delegations.delegate` (the manager's `_delegations`) does the rest, alike on both: it names the run (`AgentRuns.run_for`), reports its `AGENT_RUN` start (title, brief, `started_by`, `mode`) and end through the turn's live events and the records, and runs the backend's hook, `_run_agent(AgentRun)`, in a task of its own, where the run is the active run and no tool call is, so each run ends at its own end. The run's first message is the task message (`task_message`); it starts from the conversation's `plan` and `tasks` (`AgentRun.state`), and the keys it changed come back to its caller's state (`RunAnswer.state`). It ends as the `RunAnswer` says: its last text (a reply that also called a tool included; none at all, an empty result shown `No answer`), `refused` or `partial` by the outcome mark, `failed` for an error or a raise (one recoverable `ERROR`, ` failed: `, or the hook's own words, `RunFailed`: an overflow's); and the call's result is the run's envelope as JSON (`workflow/agents/envelope.py`), shown by `show_as_delegation`. A cancel of the caller's task stops the run's task first and ends the run `interrupted`. Each run works under its `RunControl` (`workflow/agents/control.py`, `AgentRun.control`), with its type's `AgentConfig.limits` over the policy's (`SubAgentPolicy.limits`, `DEFAULT_LIMITS`: 100 model calls): `Delegations.cancel(run_id, by=, error=)` cancels the run alone (its parent, the user or a limit), and the hook, seeing its own cancel (`own_cancel`), stops the run as a stopped turn's (ADK: its abort signal, its open calls sealed in its child session; Deep Agents: its stream closed) and returns what it has: the run ends `cancelled` with `cancelled_by` and that text, `partial`, while its siblings and the turn go on; the turn's own cancel still wins (`interrupted`). Past its `timeout` (counted from when its hook starts, `timing`) the run is cancelled by its limit (`time limit reached`); each model call is counted before it is made (`RunControlPlugin` in each sub-agent chain, `RunControlMiddleware` on each sub-agent's model side) and the last allowed one gets `STEP_LIMIT_NOTE` after its last message, its tools still declared (Claude refuses tool calls in a request that defines no tools); its answer's calls are dropped, never run, so the run completes `partial`, `step limit reached`; each call's tokens are counted after it, and past `max_tokens` an answer that makes tool calls is the run's last, its calls dropped, `cancelled` by its limit (`token limit reached`), while one that makes none ends the run as it would, `completed`, its result whole (a model that reports no usage is not limited by tokens). A turn starts at most `max_runs_per_turn` runs, at any depth (`sessions.runs_this_turn`): a call past it is the error result, no run started; `task`'s description says so (`limit_sentences`). At most `max_active` runs of a conversation work at once, at any depth (`workflow/agents/slots.py::SlotPool`, one per conversation, `Delegations.slots`; the root takes no slot): a run takes a slot right before its hook and its time limit start and gives it back at its end, however it ends; one that finds none waits `queued` (an `AGENT_RUN` and its record say so, then `running`), not counted against its `timeout`. A run lends its slot while it waits on its children (one lend for all it waits on at once; the child that ends last hands its slot straight back, so the tree in progress goes on ahead of the runs that have not started) or on the user (`waiting_on_user`: `input_required`, then `running`, in `request_user_input` and in the permission engine from before its one-prompt lock until after it: a run waiting for a slot while holding the lock would hold every other run's prompt), so nested fan-out never deadlocks; a cancel, queued or lending, leaves no waiter in the heap and no slot taken. A message steered to a working run (`RunControl.steer`, through `Delegations.control(run)`) is added after its last message, as the user's, before its next model call, and stays in its transcript (ADK: appended to its child session, the run's later events stamped after it so a durable store reads them back in order; Deep Agents: added to its state); a sub-agent run's task message is never trimmed, on ADK neither by the fit guard nor by the overflow handler's retry, whatever was steered after it. A dialog whose run is stopped is withdrawn, the turn going on (`MessageProcessor`). `AgentConfig.inherit_context` sets what a run sees: False (the default) its task alone, True its parent's conversation too, read once as the run starts (the conversation after any summary, less the message that called `task`), before the task message, which then ends with `INHERITED_CONTEXT_NOTE` (whose conversation it reads, which message is its task); `validate_agent_graph` refuses a value that is not True or False, `inherit_context=True` on the root and, in 0.7.0, on an agent with sub-agents or below another sub-agent. On ADK (`workflow/adk/task_tool.py::TaskTool`) every config-built agent is built on its own, with no ADK `sub_agents`, so ADK offers none `transfer_to_agent`, and gets ADK's identity line ("You are an agent. Your internal name is …"); a delegating agent has one `TaskTool`. Each sub-agent type runs on its own runner (`App(name=child_app(app_name))`, the type's agent its root, `_agent_plugin_chain`: the main chain configured for a sub-agent, no memory index, no summary, `RunControlPlugin` and `SubAgentOutcomePlugin`; every plugin's per-call state keyed by invocation) over the manager's session service, on a child session per run (`child_session_id`: `/`; an inheriting run's gets the conversation's events copied, `task_tool.inherited_events`), under an app no conversation list reads. On Deep Agents (`workflow/deepagents/task_tool.py::build_task_tool`) each type's graph is built on its own, and a run `astream`s it in its task with `var_child_runnable_config` cleared, so its chunks reach its own loop alone, never its parent's stream; an inheriting run's history is the parent's messages with Deep Agents' summary event applied (`SummarizationMiddleware._apply_event_to_messages`, private: `tests/workflow/deepagents/test_compact.py` pins it), and `ForeignReasoningMiddleware(kind, agent=...)` leaves out the reasoning of AI messages another agent wrote. A native ADK YAML app keeps transferring (chat-mode sub-agents); a task-mode agent in one is refused at validation, since ADK runs its delegation outside the tool pipeline and keeps the call open across turns. - **Token usage**: every model call the framework makes is one `LLM_USAGE`, on both backends, side calls included (`generate_simple`: `summarize`'s, `purpose` `tool`; `on_session_end`'s, `memory_extraction`, outside any turn; an app's own, `side_call`, or what its caller names with `workflow/usage.py::side_call`), its counts normalized where it is made (`normalized_tokens`: `prompt_tokens` every input token, cache reads and writes included; `completion_tokens` every output token, thinking included; ADK's read by its model's provider kind, `adk_usage_event`/`adk_tokens`, since Gemini and ADK's Claude adapter keep the thinking apart from the candidates and OpenAI's count it inside, an agent of no known kind read as apart; Deep Agents' by LangChain's one convention, `langchain_tokens`; the raw log's usage records alike), and timed (`latency_ms`: on ADK from the request to the answer, `workflow/adk/call_clock.py`'s plugins marking the answer's `custom_metadata`; on Deep Agents by the turn's `RunCallback`, which marks the answer's `response_metadata["agentic_cli"]`, beside the outcome mark; a side call, a summary and the overflow retry around their own request; no adapter sends either mark). Each is recorded in its conversation's ledger (`UsageLedger`; `BaseWorkflowManager.usage`/`usage_of`, in memory for the manager's life: a backend switch's new manager starts afresh, as with `agent_runs`) where it is made, before `on_event` (the event processors, ADK's summarizer, as soon as its call answers, Deep Agents' `RunCallback`, `_report_side_call`), under the run it is made in and its conversation's turn while one runs (as the manager counts them, `/compact` and job notices included: not `SessionInfo.turns`; none in another conversation's turn or from a task its turn outlived, which `_report_side_call` reports to no one); a sub-agent run's `AGENT_RUN` end carries what it used, by model (`metadata["usage"]`, `RunRecord.usage`). The CLI's status bar sums the events it sees (the open conversation's total, on any backend); `/status` shows the ledger by agent. LangGraph (deprecated) records no ledger. - **Tool registration**: Use `@register_tool(category=..., capabilities=..., description=...)` decorator. `capabilities=` is required — pass `EXEMPT` for tools that need no permission check or a list of `Capability(name, target_arg=...)` tuples the engine matches against rules. Tools are auto-discovered via the global `ToolRegistry`. A tool in `AgentConfig.tools` that the registry did not issue is denied at every call; building the agents names it once (`tool_not_registered`, per agent and tool: `BaseWorkflowManager._name_unregistered`): a plain callable, a `FunctionTool`/`StructuredTool` judged by the function it runs (as the gates unwrap it), an app's own tool class; ADK's own tool objects excepted (a built-in runs inside the model). `outside_content=True` (or a function of the call's arguments) declares that a call brings content from outside the user and the project into the conversation (a page, a document, a program's output): once the gate allows one, a memory write in that conversation is approved per call (see Permissions); an application's tools are unmarked unless they declare it, and a variant must declare the same flag as its tool. - **Async tools never block the event loop**: a synchronous tool runs on a worker thread on both backends; an async tool runs on the event loop itself, where a blocking call (DNS, a connect, a file lock's wait, a sleep, a subprocess wait, PDF parsing, page conversion, embedding) freezes the whole app, Ctrl+C included. An async tool runs its slow work with `asyncio.to_thread`. `tests/tools/test_async_tools_never_block.py` lists every async tool (the registry's, the factories' service-bound variants, Deep Agents' skill tools), so a new one fails until it is listed, and calls each through the loop guard (`tests/loop_guard.py`). diff --git a/README.md b/README.md index a052023a..0a19e2a5 100644 --- a/README.md +++ b/README.md @@ -1374,16 +1374,19 @@ reviewer = AgentConfig(name="reviewer", prompt="Check what was said.", - **A run's limits.** `subagents=SubAgentPolicy(...)` (given to the app, a manager or the factory) sets every run's `limits` (`AgentLimits(max_model_calls=100)` by default; a `timeout` in seconds and - `max_tokens` too) and the runs one turn may start (`max_runs_per_turn`, - 20, at any depth); an agent's own `limits=AgentLimits(...)` lies over the - policy's, field by field. A run's last allowed model call is told to - answer with what it has, and the run completes `partial` (`step limit - reached`); past its time, or past its tokens with an answer that asks for - tools, it is `cancelled` (`time limit reached`, `token limit reached`) - with the text it wrote so far, a prompt it waited at withdrawn (an answer - past its tokens that asks for none is its result, and the run completes; - a model that reports no usage is not limited by tokens); a `task` call - past the runs a turn may start starts no run, and the parent reads why. + `max_tokens` too), the runs one turn may start (`max_runs_per_turn`, 20, + at any depth) and the runs that work at once (`max_active`, 4, at any + depth: a run past them waits for a free slot, and one waiting on its + children or the user lends its slot meanwhile); an agent's own + `limits=AgentLimits(...)` lies over the policy's, field by field. A run's + last allowed model call is told to answer with what it has, and the run + completes `partial` (`step limit reached`); past its time, or past its + tokens with an answer that asks for tools, it is `cancelled` (`time limit + reached`, `token limit reached`) with the text it wrote so far, a prompt + it waited at withdrawn (an answer past its tokens that asks for none is + its result, and the run completes; a model that reports no usage is not + limited by tokens); a `task` call past the runs a turn may start starts + no run, and the parent reads why. - **A sub-agent's job.** A long-running tool on a sub-agent (a job starter) returns the job's id at once, and the sub-agent answers its parent in the same turn. When the job finishes, the main agent is told, in a message of @@ -2383,7 +2386,7 @@ agentic-cli/ │ │ ├── service_registry.py # Unified service ContextVar for tools │ │ ├── events.py # WorkflowEvent, EventType │ │ ├── config.py # AgentConfig -│ │ ├── agents/ # Sub-agents: delegate.py (the framework's task: the brief read, the run started and ended, its envelope), control.py (a run's limits, its cancel alone, steering), brief.py, envelope.py, runs.py, policy.py, outcome.py +│ │ ├── agents/ # Sub-agents: delegate.py (the framework's task: the brief read, the run started and ended, its envelope), control.py (a run's limits, its cancel alone, steering), slots.py (the working slots: SlotPool), brief.py, envelope.py, runs.py, policy.py, outcome.py │ │ ├── settings.py # WorkflowSettingsMixin │ │ ├── retry.py # is_rate_limit_error: a provider's 429 (the provider clients retry) │ │ ├── tool_summaries.py # Tool result one-liner summaries diff --git a/src/agentic_cli/cli/agents_command.py b/src/agentic_cli/cli/agents_command.py index a43b76b5..ec257595 100644 --- a/src/agentic_cli/cli/agents_command.py +++ b/src/agentic_cli/cli/agents_command.py @@ -2,8 +2,9 @@ ``/agents`` lists them in a table, each run followed by the runs it started, a level deeper: its instance (``researcher-2``), its type, its -status (``running`` while it runs; a completion cut at its output limit -``completed (partial)``), its run, its title, the tokens it used, and how +status (while it works ``running``, ``queued`` waiting for a free working +slot, or ``needs you`` waiting on the user; a completion cut at its output +limit ``completed (partial)``), its run, its title, the tokens it used, and how long it ran (or has been running). ``/agents `` shows one instance (``researcher-2``) or one run (``researcher-2#1``): its type, its parent, its title, then each run's status, the fields of its brief its parent filled @@ -30,7 +31,7 @@ from agentic_cli.cli.commands import Command, CommandCategory from agentic_cli.cli.usage_tracker import format_tokens, format_usage -from agentic_cli.workflow.agents import RUN_COMPLETED, RUN_STARTED +from agentic_cli.workflow.agents import RUN_COMPLETED, RUN_INPUT_REQUIRED, RUN_STARTED from agentic_cli.workflow.agents.brief import BRIEF_FIELDS from agentic_cli.workflow.agents.runs import run_number @@ -181,10 +182,13 @@ def follow(parent: str | None, level: int) -> None: def _status(record: "RunRecord") -> str: - """A run's status as ``/agents`` says it: ``running`` while it runs, and + """A run's status as ``/agents`` says it: as recorded (``running``, + ``queued``, an end), but ``needs you`` while it waits on the user, and a completion cut at its output limit ``completed (partial)``.""" if record.status == RUN_STARTED: return "running" + if record.status == RUN_INPUT_REQUIRED: + return "needs you" if record.status == RUN_COMPLETED and record.partial: return "completed (partial)" return record.status diff --git a/src/agentic_cli/cli/message_processor.py b/src/agentic_cli/cli/message_processor.py index fcfc059a..14cf1777 100644 --- a/src/agentic_cli/cli/message_processor.py +++ b/src/agentic_cli/cli/message_processor.py @@ -17,6 +17,7 @@ from agentic_cli.constants import truncate from agentic_cli.logging import Loggers, bind_context +from agentic_cli.workflow.agents.runs import RUN_INPUT_REQUIRED, RUN_QUEUED, RUN_RUNNING from agentic_cli.workflow.sessions import ConversationChangedError, instance_of if TYPE_CHECKING: @@ -338,6 +339,16 @@ def get_status(self) -> str: return f"{self.line_by} › {self.status_line}" +#: The events box's line for a sub-agent run that says how it works now +#: (``AGENT_RUN``), under its name: waiting for a free working slot, waiting +#: on the user, working again. +_RUN_LINES = { + RUN_QUEUED: "Queued: waiting for a free slot", + RUN_INPUT_REQUIRED: "Waiting for your answer", + RUN_RUNNING: "Working...", +} + + def _at_work(event: "WorkflowEvent") -> str | None: """The sub-agent run ``event`` is the work of (its instance), or None. @@ -1315,6 +1326,25 @@ async def _handle_context_summarizing( """ state.status_line = event.content + async def _handle_agent_run( + self, + event: "WorkflowEvent", + state: _EventProcessingState, + ui: "ThinkingPromptSession", + settings: "BaseSettings", + workflow: object, + ) -> None: + """Handle AGENT_RUN events — a run that waits, or works again, says so in the events box. + + Under the run's name: ``queued`` (waiting for a free working slot), + ``input_required`` (waiting on the user) and ``running`` each write + the line (``_RUN_LINES``); a run's start and end write none, nor is + anything printed (``/agents`` lists the runs). + """ + line = _RUN_LINES.get(event.metadata.get("status")) + if line is not None: + state.status_line = line + async def _handle_llm_usage( self, event: "WorkflowEvent", @@ -1361,5 +1391,6 @@ def _get_event_dispatch(cls) -> dict: EventType.CONTEXT_TRIMMED: cls._handle_context_trimmed, EventType.CONTEXT_SUMMARIZING: cls._handle_context_summarizing, EventType.LLM_USAGE: cls._handle_llm_usage, + EventType.AGENT_RUN: cls._handle_agent_run, } return cls._EVENT_DISPATCH diff --git a/src/agentic_cli/workflow/agents/__init__.py b/src/agentic_cli/workflow/agents/__init__.py index 3996b458..7db606fa 100644 --- a/src/agentic_cli/workflow/agents/__init__.py +++ b/src/agentic_cli/workflow/agents/__init__.py @@ -3,7 +3,8 @@ A sub-agent run is one piece of work a sub-agent does for its parent: one delegation (its parent's tool call). ``runs`` names each run within its conversation and records it from its ``AGENT_RUN`` events; ``policy`` holds -an app's sub-agent policy and a run's limits. +an app's sub-agent policy and a run's limits; ``slots`` how many runs work +at once. """ from agentic_cli.workflow.agents.policy import DEFAULT_LIMITS, AgentLimits, SubAgentPolicy @@ -13,9 +14,13 @@ RUN_COMPLETED, RUN_END_STATUSES, RUN_FAILED, + RUN_INPUT_REQUIRED, RUN_INTERRUPTED, + RUN_QUEUED, RUN_REFUSED, + RUN_RUNNING, RUN_STARTED, + RUN_WORKING_STATUSES, SUBAGENT_ERROR_PREFIX, TITLE_MAX, AgentRuns, @@ -34,9 +39,13 @@ "RUN_COMPLETED", "RUN_END_STATUSES", "RUN_FAILED", + "RUN_INPUT_REQUIRED", "RUN_INTERRUPTED", + "RUN_QUEUED", "RUN_REFUSED", + "RUN_RUNNING", "RUN_STARTED", + "RUN_WORKING_STATUSES", "RunRecord", "SUBAGENT_ERROR_PREFIX", "SubAgentPolicy", diff --git a/src/agentic_cli/workflow/agents/brief.py b/src/agentic_cli/workflow/agents/brief.py index a34c2823..e76b4480 100644 --- a/src/agentic_cli/workflow/agents/brief.py +++ b/src/agentic_cli/workflow/agents/brief.py @@ -233,10 +233,15 @@ def limit_sentences(policy: SubAgentPolicy) -> list[str]: """The sentences of ``task``'s description that state the limits a delegation enforces. One sentence for each limit the framework enforces on a delegation, with - the number ``policy`` sets, in the order they apply: the runs one turn - may start (``max_runs_per_turn``). + the number ``policy`` sets, in the order they apply: the runs that work + at once (``max_active``: more wait for a free working slot), then the + runs one turn may start (``max_runs_per_turn``). """ - return [f"At most {policy.max_runs_per_turn} runs per turn."] + return [ + f"At most {policy.max_active} sub-agents work at once; more calls in one response " + "wait for a free slot.", + f"At most {policy.max_runs_per_turn} runs per turn.", + ] def _type_line(name: str, config: AgentConfig, *, warn: bool = False) -> str: diff --git a/src/agentic_cli/workflow/agents/delegate.py b/src/agentic_cli/workflow/agents/delegate.py index 680a5c86..e5a38641 100644 --- a/src/agentic_cli/workflow/agents/delegate.py +++ b/src/agentic_cli/workflow/agents/delegate.py @@ -31,6 +31,24 @@ limit running while the hook does, and a cancel of the run alone (``Delegations.cancel``), which its limits use too. +Each run works in a slot of its conversation's ``SlotPool`` +(``agents/slots.py``, ``SubAgentPolicy.max_active`` slots; the root takes +none): it takes one before its hook runs, waiting ``queued`` (then +``running``) when every slot is held, and gives it back at its end. A run +that delegates lends its slot while it waits for its child (one lend for +all the children it waits on at once), and the child that ends last hands +it straight back, so a parent goes on ahead of the runs that have not +started, and nested delegations never wait on each other's slots. A cancel +of a queued run ends it as a cancel of a working run does, and its place +in the queue goes with it. The time a run waits for its slot is not +counted against its time limit; the time it lends its slot is. + +A run that waits on the user (``waiting_on_user``: a question, a permission +prompt, ``BaseWorkflowManager.request_user_input`` and the permission +engine) says ``input_required``, lends its slot for the wait, and says +``running`` once it has its slot back; a prompt no one can be shown (no +input callback) lends nothing. + Each ``AGENT_RUN`` is recorded (``AgentRuns.observe``) as it is made and reported to the turn's live events, so it reaches the turn's stream while the run works. A run ends: @@ -58,7 +76,9 @@ from __future__ import annotations import asyncio -from collections.abc import Awaitable, Callable, Mapping +from collections.abc import AsyncIterator, Awaitable, Callable, Mapping +from contextlib import asynccontextmanager +from contextvars import ContextVar from dataclasses import dataclass, field from typing import TYPE_CHECKING, Any @@ -79,13 +99,17 @@ RUN_CANCELLED, RUN_COMPLETED, RUN_FAILED, + RUN_INPUT_REQUIRED, RUN_INTERRUPTED, + RUN_QUEUED, RUN_REFUSED, + RUN_RUNNING, RUN_STARTED, AgentRuns, result_head, run_title, ) +from agentic_cli.workflow.agents.slots import NEW, Slot, SlotPool from agentic_cli.workflow.events import WorkflowEvent from agentic_cli.workflow.sessions import ( RunRef, @@ -219,6 +243,72 @@ class Delegated: StoredInstances = Callable[[SessionRef], Awaitable[Mapping[str, int]]] +@dataclass +class _Working: + """A run at work, as its own task sees it: its slot in its conversation's + pool, the child runs it waits on (by run id), and its waits on the user.""" + + ref: RunRef + call_id: str + pool: SlotPool + slot: Slot + emit: Callable[[WorkflowEvent], None] + children: set[str] = field(default_factory=set) + asking: int = 0 + + def waits_on_only(self, child: str) -> bool: + """Whether ``child`` is all the run waits on: its last child, the user not asked.""" + return self.asking == 0 and self.children <= {child} + + @asynccontextmanager + async def waiting_on_user(self) -> AsyncIterator[None]: + """``input_required``, its slot lent for the block, then ``running``; nested waits once.""" + self.asking += 1 + if self.asking == 1: + self.emit(WorkflowEvent.agent_run(self.ref, RUN_INPUT_REQUIRED, call_id=self.call_id)) + stopped = False + try: + async with self.pool.lent(self.slot): + yield + except asyncio.CancelledError: + stopped = True + raise + finally: + self.asking -= 1 + if self.asking == 0 and not stopped: + self.emit(WorkflowEvent.agent_run(self.ref, RUN_RUNNING, call_id=self.call_id)) + + +#: The run at work in this task (``_in_run`` sets it in the run's own task, +#: whose copies its tool calls run in); None in the root's own work. +_working: ContextVar[_Working | None] = ContextVar("agentic_cli_working_run", default=None) + + +@asynccontextmanager +async def waiting_on_user() -> AsyncIterator[None]: + """The block waits on the user (a question, a permission prompt): the run lends its slot. + + In a sub-agent run's own work, the run says ``input_required``, its slot + works for another run meanwhile, and once the block ends it takes the + slot back, ahead of the runs that have not started, and says + ``running``; nested waits (a prompt waiting its turn behind another's, + then the prompt) count once. A wait a cancel ends takes nothing back: + the run is being stopped. Nothing in the root's own work, nor outside a + run. + + Only a wait that can block belongs in it: a lent slot goes to the first + run queued for one, which may work its whole run in it before this run + takes it back. ``request_user_input`` and the permission engine enter it + only when someone can answer (``can_request_user_input``). + """ + working = _working.get() + if working is None: + yield + return + async with working.waiting_on_user(): + yield + + class RunFailed(Exception): """A run's failure in its backend's words: what its parent and its user read. @@ -278,6 +368,16 @@ def __init__( self._numbering = asyncio.Lock() # The working runs' controls, by (conversation, run id). self._controls: dict[tuple[SessionRef | None, str], RunControl] = {} + # Each conversation's working slots, made on first use and kept: a + # slot lent while its run waits must come back to the pool it left. + self._pools: dict[SessionRef | None, SlotPool] = {} + + def slots(self, conversation: SessionRef | None) -> SlotPool: + """``conversation``'s working slots: ``policy.max_active`` of them, made on first use.""" + pool = self._pools.get(conversation) + if pool is None: + pool = self._pools[conversation] = SlotPool(self.policy.max_active) + return pool def sub_agents(self, agent: str) -> dict[str, AgentConfig]: """The types ``agent`` may start, by name, in its ``sub_agents``' order.""" @@ -289,7 +389,8 @@ async def delegate(self, request: DelegationRequest) -> Delegated: """Run the sub-agent ``request`` asks for, to its end; its envelope. A brief that cannot start a run, or a call past the runs its turn - may start, is the error result, and nothing else happens. A cancel + may start, is the error result, and nothing else happens. A caller + that is a run lends its slot while it waits for this one. A cancel of the caller's task (the turn's) ends the run ``interrupted`` once it has stopped, and goes on. """ @@ -321,8 +422,17 @@ async def delegate(self, request: DelegationRequest) -> Delegated: control.bind(inner) key = (conversation, ref.run or "") self._controls[key] = control + caller = _working.get() try: - return await self._ended(ref, request.call_id, inner, control) + if caller is None: + return await self._ended(ref, request.call_id, inner, control) + # The caller waits: its slot works for another run meanwhile. + caller.children.add(ref.run or "") + try: + async with caller.pool.lent(caller.slot): + return await self._ended(ref, request.call_id, inner, control) + finally: + caller.children.discard(ref.run or "") finally: self._controls.pop(key, None) @@ -466,9 +576,14 @@ async def _in_run( """The run, in its own task: the active run, and no tool call active. The task's context is a copy of its caller's, so what this sets stays - here. An inheriting type's history is read here, once. The run's - time limit runs from when its hook starts (``RunControl.timing``). + here. An inheriting type's history is read here, once. The hook runs + in a working slot of the conversation's pool, the run ``queued`` + while it waits for one; the slot goes back at the run's end, to its + caller when the caller waits on it alone (``SlotPool.hand_back``). + The run's time limit runs from when its hook starts + (``RunControl.timing``), not while it waits for its slot. """ + caller = _working.get() set_active_run(ref) set_active_tool_call(None) history = ( @@ -484,8 +599,28 @@ async def _in_run( history=history, control=control, ) - with control.timing(): - return await self._hook(run) + pool = self.slots(conversation) + queued = pool.free == 0 + if queued: + self._emit(WorkflowEvent.agent_run(ref, RUN_QUEUED, call_id=request.call_id)) + slot = await pool.acquire(NEW) + try: + if queued: + self._emit(WorkflowEvent.agent_run(ref, RUN_RUNNING, call_id=request.call_id)) + _working.set(_Working(ref, request.call_id, pool, slot, emit=self._emit)) + with control.timing(): + return await self._hook(run) + finally: + if caller is None: + pool.release(slot) + else: + last = caller.waits_on_only(ref.run or "") + caller.children.discard(ref.run or "") + if last: + # The caller's last wait: its slot goes straight back to it. + pool.hand_back(slot, to=caller.slot) + else: + pool.release(slot) def _cancelled( self, diff --git a/src/agentic_cli/workflow/agents/policy.py b/src/agentic_cli/workflow/agents/policy.py index dff67f9b..fba3e0b0 100644 --- a/src/agentic_cli/workflow/agents/policy.py +++ b/src/agentic_cli/workflow/agents/policy.py @@ -95,7 +95,8 @@ class SubAgentPolicy: Attributes: max_active: How many sub-agent runs may work at once in a - conversation, at any depth. + conversation, at any depth; the others wait for a free + working slot (``agents/slots.py``). max_runs_per_turn: How many sub-agent runs one turn may start. limits: Every run's limits (``AgentLimits``), which an agent type's own lie over. A policy's own replace ``DEFAULT_LIMITS`` whole: diff --git a/src/agentic_cli/workflow/agents/runs.py b/src/agentic_cli/workflow/agents/runs.py index f03ee8dd..2b6cfc1c 100644 --- a/src/agentic_cli/workflow/agents/runs.py +++ b/src/agentic_cli/workflow/agents/runs.py @@ -10,14 +10,14 @@ numbers its call ids repeats them every turn). The records are written from the runs' own events (``AGENT_RUN``: started, -then one end status), which ``Delegations`` hands to ``observe`` as it makes -them, under the turn they are reported in. The one other writer is -``interrupt_open``, for the runs a turn leaves open when it is stopped or -fails, when no event can reach its consumer any more. A conversation is a -``SessionRef``, or None for runs started outside a manager's turn. All of it -is in memory, for as long as its manager lives: a backend switch builds a -new manager, and the runs recorded before it do not follow (as with its -usage ledgers). +then how it works while it does, then one end status), which +``Delegations`` hands to ``observe`` as it makes them, under the turn they +are reported in. The one other writer is ``interrupt_open``, for the runs a +turn leaves open when it is stopped or fails, when no event can reach its +consumer any more. A conversation is a ``SessionRef``, or None for runs +started outside a manager's turn. All of it is in memory, for as long as +its manager lives: a backend switch builds a new manager, and the runs +recorded before it do not follow (as with its usage ledgers). """ from __future__ import annotations @@ -31,8 +31,16 @@ from agentic_cli.workflow.sessions import RunRef, SessionRef, get_active_turn from agentic_cli.workflow.tool_summaries import delegation_summary -#: A run's statuses: started, then one end. +#: A run's statuses: started, then how it works while it does, then one end. RUN_STARTED = "started" +#: A working run's status: waiting for a free working slot, working, or +#: waiting on the user (a question, a permission prompt). A record says +#: ``running`` from its start; its events say ``queued`` when it must wait +#: for a slot, and ``running`` once it has one, and after the user answered. +RUN_QUEUED = "queued" +RUN_RUNNING = "running" +RUN_INPUT_REQUIRED = "input_required" +RUN_WORKING_STATUSES = frozenset({RUN_QUEUED, RUN_RUNNING, RUN_INPUT_REQUIRED}) RUN_COMPLETED = "completed" RUN_FAILED = "failed" RUN_REFUSED = "refused" @@ -109,7 +117,9 @@ class RunRecord: ref: The run. call_id: The parent's call that started it. title: Its brief's title. - status: ``started`` while it runs, then its end status. + status: While it works, ``running``, ``queued`` (waiting for a + free working slot) or ``input_required`` (waiting on the user); + then its end status. started: When it started. ended: When it ended; None while it runs. result: The first ``RESULT_MAX`` characters of its result. @@ -222,10 +232,13 @@ def observe(self, event: WorkflowEvent) -> None: """Record what a run's ``AGENT_RUN`` event says, under the turn it is reported in. A start makes the run's record (its call, title, brief, who started - it and its mode); an end gives it its status, time, result, error, - whether it is partial, what it used and who cancelled it (an end - whose start was not seen makes the record too). Any other event - changes nothing. + it and its mode), ``running``; a working status (``queued``, + ``running``, ``input_required``) becomes a working record's status; + an end gives it its status, time, result, error, whether it is + partial, what it used and who cancelled it (an end whose start was + not seen makes the record too). Any other event changes nothing, and + neither does a working status of a run that has ended or was never + started. """ ref = event.run if event.type is not EventType.AGENT_RUN or ref is None or ref.run is None: @@ -238,13 +251,18 @@ def observe(self, event: WorkflowEvent) -> None: ref=ref, call_id=metadata.get("call_id"), title=metadata.get("title") or "", - status=RUN_STARTED, + status=RUN_RUNNING, started=event.timestamp, brief=dict(metadata.get("brief") or {}), started_by=metadata.get("started_by"), mode=metadata.get("mode") or "blocking", ) return + if status in RUN_WORKING_STATUSES: + working = records.get(ref.run) + if working is not None and working.status not in RUN_END_STATUSES: + working.status = status + return if status not in RUN_END_STATUSES: return record = records.get(ref.run) diff --git a/src/agentic_cli/workflow/agents/slots.py b/src/agentic_cli/workflow/agents/slots.py new file mode 100644 index 00000000..5a2f0f17 --- /dev/null +++ b/src/agentic_cli/workflow/agents/slots.py @@ -0,0 +1,291 @@ +"""Working slots: how many sub-agent runs work at once. + +A conversation's sub-agent runs share a ``SlotPool`` of +``SubAgentPolicy.max_active`` slots (``agents/delegate.py::Delegations``). A +run takes one (``acquire``) before it starts to work and gives it back +(``release``) when it ends, however it ends. A run that finds every slot +held waits for one, in a heap ordered by priority, then arrival: +``RETURNING`` (a run taking its slot back) before ``NEW`` (a run that has +not started). + +A run that waits on its child runs or on the user lends its slot meanwhile +(``async with pool.lent(slot):``): another run works with it, and the run +takes one back at ``RETURNING`` when the wait is over. A child that ends +while its parent waits on it alone hands its slot straight to the parent +(``hand_back``: whether the parent waits on anything else is its caller's +to know), ahead of every waiter. Everything holding a slot is working, so +nested delegations never deadlock, and the trees in progress finish before +new runs start. + +A cancel never loses or leaks a slot: a waiter cancelled leaves the heap +(one handed a slot in the tick it was cancelled hands it to the next); a +holder releases in its ``finally``; a run cancelled while its slot is lent +does not take it back, and its release is then nothing. + +Not ``asyncio.Semaphore``: it wakes its waiters in arrival order, with no +priority. All of it runs on the event loop; nothing here blocks. +""" + +from __future__ import annotations + +import asyncio +import heapq +import itertools +from collections.abc import AsyncIterator +from contextlib import asynccontextmanager + +#: A run taking its slot back, after waiting on its children or the user. +RETURNING = 0 +#: A run that has not started. +NEW = 1 + +_HELD = "held" +_LENT = "lent" +_RELEASED = "released" + +#: A waiting run's entry in the heap: its priority, its arrival, and the +#: future a slot handed to it sets (True), or a withdrawn take-back (False). +_Waiter = tuple[int, int, "asyncio.Future[bool]"] + + +class Slot: + """One run's working slot, from a ``SlotPool``. + + Held while its run works; lent while its run waits on its children or + the user (``SlotPool.lent``); released once the run has given it back. + """ + + __slots__ = ("_pool", "_state", "_lends", "_back") + + def __init__(self, pool: SlotPool) -> None: + self._pool = pool + self._state = _HELD + # The lends in progress (``SlotPool.lent``, nested): the slot is lent + # on the first and taken back after the last. + self._lends = 0 + # Its run's take-back waiting in the heap for a slot, while there + # is one (``SlotPool._take_back``). + self._back: _Waiter | None = None + + @property + def held(self) -> bool: + """Whether its run holds it: it works with it.""" + return self._state == _HELD + + @property + def lent(self) -> bool: + """Whether its run lends it, while it waits.""" + return self._state == _LENT + + +class SlotPool: + """``size`` working slots, and the runs waiting for one (see the module docstring). + + Args: + size: How many runs work at once. + + Raises: + ValueError: ``size`` is not a positive int. + """ + + def __init__(self, size: int) -> None: + if isinstance(size, bool) or not isinstance(size, int) or size < 1: + raise ValueError(f"A slot pool's size is a positive int, not {size!r}.") + self.size = size + self._free = size + # One (priority, arrival, future) per waiting run, a heap: a slot + # handed to it sets its future's result. Arrivals are unique, so two + # entries never compare their futures. + self._waiters: list[_Waiter] = [] + self._arrivals = itertools.count() + + @property + def free(self) -> int: + """How many slots no run holds (a run acquiring one waits only while this is 0).""" + return self._free + + @property + def waiting(self) -> int: + """How many runs wait for a slot.""" + return len(self._waiters) + + async def acquire(self, priority: int = NEW) -> Slot: + """A slot for a run: at once when one is free, else once one is handed to it. + + The waiters are served by ``priority`` (``RETURNING`` before ``NEW``), + then in the order they came. A cancel while it waits takes it out of + the heap; a slot handed to it in the same tick goes to the next. + """ + await self._take(priority) + return Slot(self) + + def release(self, slot: Slot) -> None: + """Give ``slot`` back: to the first waiter, else it is free. + + Nothing for a slot released already, nor for a lent one (its run + was cancelled while it waited: the slot went back as it was lent). + + Raises: + ValueError: ``slot`` is another pool's. + """ + self._own(slot) + state, slot._state = slot._state, _RELEASED + if state == _HELD: + self._give() + else: + self._withdraw(slot) + + def hand_back(self, slot: Slot, to: Slot) -> None: + """Release a child's ``slot`` at its end to ``to``, its parent's, which waits on it alone. + + A lent ``to`` is held again with the child's slot, ahead of every + waiter: its lends then end without waiting, and one that starts + before they have ended lends it again. A ``to`` that is not lent + takes nothing: as ``release``. The caller knows that the parent + waits on nothing else (no other child, not the user). + + Raises: + ValueError: either slot is another pool's. + """ + self._own(slot) + self._own(to) + if slot._state == _HELD and to._state == _LENT: + slot._state = _RELEASED + to._state = _HELD + self._withdraw(to) + return + self.release(slot) + + @asynccontextmanager + async def lent(self, slot: Slot) -> AsyncIterator[None]: + """Lend ``slot`` for the block, while its run waits on its children or the user. + + A lend of a held slot frees it for another run (of nested lends, the + first); when the last ends, it is taken back at ``RETURNING`` (at + once when a slot is free, else ahead of the runs that have not + started), unless a child handed it back already. A lend that starts + while another of the run's waits is taking the slot back keeps it + lent: that wait ends without it, and this one takes it back. A lend + that ends in a cancel takes nothing back: its run is being stopped, + and its ``release`` is then nothing. An error is no cancel: the run + goes on, with its slot back. + + Raises: + ValueError: ``slot`` is another pool's. + """ + self._own(slot) + slot._lends += 1 + if slot._state == _HELD: + slot._state = _LENT + self._give() + else: + # Lent already: an earlier wait of the run that is taking it back + # no longer needs it, since the run waits again; this lend takes + # it back when it ends. + self._withdraw(slot) + try: + yield + except asyncio.CancelledError: + slot._lends -= 1 + if slot._lends == 0 and slot._state == _LENT: + slot._state = _RELEASED + raise + except BaseException: + slot._lends -= 1 + if slot._lends == 0: + await self._take_back(slot) + raise + slot._lends -= 1 + if slot._lends == 0: + await self._take_back(slot) + + async def _take_back(self, slot: Slot) -> None: + """A lent ``slot`` held again, at ``RETURNING``; nothing for one handed back or released. + + A take-back in the heap the slot no longer needs (a child handed it + back, its run released it, or the run lent it again) is withdrawn + and takes nothing. One handed a slot before it could be withdrawn + gives that slot to the next waiter when the slot is lent again, + handed back or released by the time it goes on; else it holds the + slot, and a take-back queued meanwhile (a wait of the run that + started and ended in between) is withdrawn. So a run's slot has at + most one take-back waiting for a slot, and none once it is held. + """ + if slot._state != _LENT: + return + try: + took = await self._take(RETURNING, slot) + except asyncio.CancelledError: + if slot._state == _LENT and slot._lends == 0: + slot._state = _RELEASED + raise + if not took: + return + if slot._state != _LENT or slot._lends > 0: + self._give() + return + slot._state = _HELD + # A take-back another wait of the run queued while this one was + # being handed the slot: the slot is back, so it ends without one. + self._withdraw(slot) + + async def _take(self, priority: int, slot: Slot | None = None) -> bool: + """Count one slot taken: a free one, else wait in the heap until one is handed over. + + False when the wait is ``slot``'s take-back and was withdrawn + (``_withdraw``): no slot was taken. + """ + if self._free > 0: + self._free -= 1 + return True + future: asyncio.Future[bool] = asyncio.get_running_loop().create_future() + entry = (priority, next(self._arrivals), future) + heapq.heappush(self._waiters, entry) + if slot is not None: + slot._back = entry + try: + return await future + except asyncio.CancelledError: + if future.done() and not future.cancelled(): + if future.result(): + # Handed a slot in the tick it was cancelled: the next one's. + self._give() + else: + self._leave(entry) + raise + finally: + if slot is not None and slot._back is entry: + slot._back = None + + def _give(self) -> None: + """A slot came back: hand it to the first waiter still waiting, else it is free.""" + while self._waiters: + _, _, future = heapq.heappop(self._waiters) + if not future.done(): + future.set_result(True) + return + self._free += 1 + + def _withdraw(self, slot: Slot) -> None: + """End ``slot``'s take-back waiting in the heap, if any, without a slot.""" + entry, slot._back = slot._back, None + if entry is None: + return + self._leave(entry) + if not entry[2].done(): + entry[2].set_result(False) + + def _leave(self, entry: _Waiter) -> None: + """Take a cancelled or withdrawn waiter's ``entry`` out of the heap. + + ``_give`` may have popped it already. + """ + for index, waiting in enumerate(self._waiters): + if waiting is entry: + self._waiters.pop(index) + heapq.heapify(self._waiters) + return + + def _own(self, slot: Slot) -> None: + if slot._pool is not self: + raise ValueError("A slot goes back to the pool it came from.") diff --git a/src/agentic_cli/workflow/base_manager.py b/src/agentic_cli/workflow/base_manager.py index c0b6bae0..9baf83d6 100644 --- a/src/agentic_cli/workflow/base_manager.py +++ b/src/agentic_cli/workflow/base_manager.py @@ -49,7 +49,12 @@ resolve_model, ) from agentic_cli.workflow.config import AgentConfig -from agentic_cli.workflow.agents.delegate import AgentRun, Delegations, RunAnswer +from agentic_cli.workflow.agents.delegate import ( + AgentRun, + Delegations, + RunAnswer, + waiting_on_user, +) from agentic_cli.workflow.agents.policy import SubAgentPolicy, as_policy from agentic_cli.workflow.agents.runs import AgentRuns from agentic_cli.workflow.usage import ( @@ -524,6 +529,18 @@ def clear_input_callback(self, token: "Token | None" = None) -> None: pass self._user_input_callback.set(None) + def can_request_user_input(self) -> bool: + """Whether ``request_user_input`` could reach the user from this context now. + + False when no input callback is set for this context, or the turn + this context comes from has ended: ``request_user_input`` then raises + ``UserInputUnavailable`` at once. The permission engine asks this + before a sub-agent run lends its working slot for a prompt: a prompt + no one can be shown is denied without waiting, and the run keeps its + slot. + """ + return self._user_input_callback.get() is not None and not turn_has_ended() + @property def agent_configs(self) -> list[AgentConfig]: """Get the agent configurations.""" @@ -2508,6 +2525,15 @@ async def request_user_input(self, request: UserInputRequest) -> str: (``get_active_run()``; None in the root's own calls), so the consumer can say which agent asks, whoever built the request. + A sub-agent run waits for the answer with its working slot lent + (``agents/delegate.py::waiting_on_user``): it is ``input_required`` + meanwhile, another run may work in its slot, and it takes a slot + back, ahead of the runs that have not started, once the user has + answered. A request no one can answer (``can_request_user_input``) + raises before the run lends anything. A callback that answers + without waiting still lends the slot: a run queued for one may then + work in it before the asking run takes it back. + Args: request: The user input request. @@ -2544,10 +2570,11 @@ async def request_user_input(self, request: UserInputRequest) -> str: # A call cancelled at its question was not run (workflow/interrupts.py). call = get_active_tool_call() - if call is None: - return await callback(request) - with self._open_calls.asking(call.call_id): - return await callback(request) + async with waiting_on_user(): + if call is None: + return await callback(request) + with self._open_calls.asking(call.call_id): + return await callback(request) # Async context manager support diff --git a/src/agentic_cli/workflow/permissions/engine.py b/src/agentic_cli/workflow/permissions/engine.py index 009b2653..cc5c4afe 100644 --- a/src/agentic_cli/workflow/permissions/engine.py +++ b/src/agentic_cli/workflow/permissions/engine.py @@ -13,6 +13,7 @@ from __future__ import annotations import asyncio +import contextlib import ipaddress from collections.abc import Callable, Collection, Mapping from enum import Enum @@ -577,6 +578,15 @@ def _fmt_rule_reason(rule: Rule, cap: ResolvedCapability) -> str: return f"rule: {rule.source.value}/{rule.effect.value} {cap.name} {rule.target}" + def _can_ask(self) -> bool: + """Whether a prompt now could reach the user. + + The workflow says (``BaseWorkflowManager.can_request_user_input``); + one that cannot say is taken to be able to, and is asked. + """ + can_ask = getattr(self._workflow, "can_request_user_input", None) + return can_ask is None or bool(can_ask()) + async def _ask_and_apply( self, tool_name: str, @@ -586,12 +596,22 @@ async def _ask_and_apply( uncontained: Collection[str] = frozenset(), per_call: Mapping[str, str] | None = None, ) -> CheckResult: + from agentic_cli.workflow.agents.delegate import waiting_on_user from agentic_cli.workflow.permissions.prompt import build_request, parse_response from agentic_cli.workflow.permissions.store import append_project_rule from agentic_cli.workflow.sessions import get_active_run unmatched = [cap for cap, r in outcomes if r is None] - async with self._ask_lock: + # A sub-agent run waits on the user from here, its working slot lent + # (request_user_input's own wait counts once with this one): while + # another run's prompt shows, then while its own does. It takes the + # slot back only once the lock is released: a run that waited for + # its slot holding the lock would hold every other run's prompt, and + # with them the slots their runs lent. A prompt no one can be shown + # (no input callback for this turn) is denied at once: the run lends + # nothing for it, and keeps its slot. + waits = waiting_on_user() if self._can_ask() else contextlib.nullcontext() + async with waits, self._ask_lock: # The run whose call is gated: both gates run inside it, so its # prompt says which sub-agent asks. request = build_request( diff --git a/tests/cli/test_agents_command.py b/tests/cli/test_agents_command.py index c86d57bc..fda4ad6e 100644 --- a/tests/cli/test_agents_command.py +++ b/tests/cli/test_agents_command.py @@ -80,6 +80,10 @@ def start(self, agent: str, call_id: str, request: str, *, at: int, ), at) return ref + def now(self, ref: RunRef, status: str, *, at: int) -> None: + """``ref``'s run says how it works now: ``queued``, ``running`` or ``input_required``.""" + self._observe(WorkflowEvent.agent_run(ref, status), at) + def end(self, ref: RunRef, status: str, *, at: int, result: str = "", error: str | None = None, partial: bool = False) -> None: self._observe(WorkflowEvent.agent_run( @@ -153,6 +157,23 @@ async def test_every_run_each_followed_by_the_runs_it_started(self, manager): ("fetcher-1", "fetcher", "failed", "#1", "Fetch Z.", "—", "2s"), ] + async def test_a_working_run_says_whether_it_works_waits_for_a_slot_or_needs_the_user( + self, manager + ): + runs = Runs(manager, manager.session_ref("s1")) + runs.start("researcher", "c1", "Find X.", at=0) + queued = runs.start("researcher", "c2", "Find Y.", at=1) + runs.now(queued, "queued", at=1) + asking = runs.start("researcher", "c3", "Find Z.", at=2) + runs.now(asking, "input_required", at=30) + + session = await agents(manager) + + (table,) = session.add_rich.call_args.args + assert [(row[0], row[2]) for row in rows_of(table)] == [ + ("researcher-1", "running"), ("researcher-2", "queued"), ("researcher-3", "needs you"), + ] + async def test_a_conversation_without_runs_says_so(self, manager): session = await agents(manager) diff --git a/tests/permissions/test_engine.py b/tests/permissions/test_engine.py index 552c8657..3b967324 100644 --- a/tests/permissions/test_engine.py +++ b/tests/permissions/test_engine.py @@ -318,6 +318,111 @@ async def fake_input(request): assert ask_peak == 1 # never two asks in flight simultaneously +class TestASubAgentRunsWaitForThePrompt: + """A sub-agent run whose prompt waits behind another's lends its working + slot from the moment it waits, and takes it back only once the lock is + released: with one slot, two runs that both ask finish.""" + + @pytest.mark.asyncio + async def test_two_runs_with_one_slot_both_ask_and_both_finish(self, ctx): + import asyncio + + from agentic_cli.workflow.agents import ( + RUN_INPUT_REQUIRED, + AgentRuns, + SubAgentPolicy, + ) + from agentic_cli.workflow.agents.delegate import RunAnswer + from tests.workflow.test_agent_delegate import ( + CONVERSATION, + Turn, + delegations, + request, + until, + ) + + runs = AgentRuns() + + async def fake_input(asked) -> str: + if asked.run.instance == "researcher-1": + # Answered once researcher-2 waits for its own prompt: it got + # researcher-1's slot, lent while researcher-1 asks. + await until(lambda: runs.record(CONVERSATION, "researcher-2").status + == RUN_INPUT_REQUIRED) + return ALLOW_ONCE_CHOICE + + w = MagicMock() + w.request_user_input = fake_input + engine = PermissionEngine(settings=_stub_settings(), workflow=w, ctx=ctx) + + async def hook(run) -> RunAnswer: + checked = await engine.check( + "web_fetch", [Capability("http.read", target_arg="url")], + {"url": f"https://{run.ref.instance}.test/"}, + ) + return RunAnswer(text=checked.reason) + + made = delegations(hook, runs=runs, policy=SubAgentPolicy(max_active=1)) + turn = Turn(lambda: asyncio.gather( + made.delegate(request(call_id="c1")), made.delegate(request(call_id="c2")) + )) + await asyncio.wait_for(turn.task, 10) + + assert [one.result["result"] for one in turn.result] == [ + "no rule + user allowed (once)", "no rule + user allowed (once)", + ] + # researcher-2 started in the slot researcher-1 lent at its prompt. + assert turn.of("researcher-1") == turn.of("researcher-2") == [ + "started", "input_required", "running", "completed", + ] + assert (made.slots(CONVERSATION).free, made.slots(CONVERSATION).waiting) == (1, 0) + + @pytest.mark.asyncio + async def test_a_prompt_no_one_can_answer_lends_nothing(self, ctx): + """Headless (no input callback), a sub-agent run's prompt is denied + at once: the run keeps its slot and goes on ahead of the run queued + for one, and never says it waits on the user.""" + import asyncio + + from agentic_cli.workflow.agents import SubAgentPolicy + from agentic_cli.workflow.agents.delegate import RunAnswer + from tests.test_input_callback import _manager + from tests.workflow.test_agent_delegate import CONVERSATION, Turn, delegations, request + + engine = PermissionEngine(settings=_stub_settings(), workflow=_manager(), ctx=ctx) + order: list[str] = [] + + async def hook(run) -> RunAnswer: + me = run.ref.instance + order.append(f"{me} starts") + await asyncio.sleep(0) # a model call first: the other run waits for a slot + checked = await engine.check( + "web_fetch", [Capability("http.read", target_arg="url")], + {"url": f"https://{me}.test/"}, + ) + order.append(f"{me} goes on") + await asyncio.sleep(0) # more work after the prompt + order.append(f"{me} ends") + return RunAnswer(text=checked.reason) + + made = delegations(hook, policy=SubAgentPolicy(max_active=1)) + turn = Turn(lambda: asyncio.gather( + made.delegate(request(call_id="c1")), made.delegate(request(call_id="c2")) + )) + await asyncio.wait_for(turn.task, 10) + + assert [one.result["result"] for one in turn.result] == [ + "no rule + no interactive approver to ask", + ] * 2 + assert order == [ + "researcher-1 starts", "researcher-1 goes on", "researcher-1 ends", + "researcher-2 starts", "researcher-2 goes on", "researcher-2 ends", + ] + assert turn.of("researcher-1") == ["started", "completed"] + assert turn.of("researcher-2") == ["started", "queued", "running", "completed"] + assert (made.slots(CONVERSATION).free, made.slots(CONVERSATION).waiting) == (1, 0) + + class TestResolveListTargetArg: def test_list_target_arg_resolves_per_item(self, ctx): from agentic_cli.workflow.permissions.capabilities import ResolvedCapability diff --git a/tests/test_input_callback.py b/tests/test_input_callback.py index e4d7e66b..afa17c72 100644 --- a/tests/test_input_callback.py +++ b/tests/test_input_callback.py @@ -130,3 +130,42 @@ async def test_a_span_closed_from_another_context_ends_nothing_here(self): assert await manager.request_user_input(_request()) == "answer" finally: close_turn_scope(span) + + +class TestASubAgentRunWaitingOnTheUser: + """A question or prompt of a sub-agent run that holds a working slot + lends the slot while the user answers (the run ``input_required``), and + takes it back after (``running``); the root's asks as before.""" + + async def test_it_lends_its_slot_while_the_user_answers(self): + from agentic_cli.workflow.agents import RUN_INPUT_REQUIRED, AgentRuns, SubAgentPolicy + from tests.workflow.test_agent_delegate import CONVERSATION, Turn, delegations, request + + manager = _manager() + runs = AgentRuns() + seen: list[tuple[int, str]] = [] + + async def answer(asked: UserInputRequest) -> str: + seen.append((made.slots(CONVERSATION).free, runs.records(CONVERSATION)[0].status)) + return "B" + + async def hook(run): + from agentic_cli.workflow.agents.delegate import RunAnswer + + return RunAnswer(text=await manager.request_user_input(_request())) + + manager.set_input_callback(answer) + made = delegations(hook, runs=runs, policy=SubAgentPolicy(max_active=1)) + turn = Turn(lambda: made.delegate(request())) + await asyncio.wait_for(turn.task, 5) + + assert seen == [(1, RUN_INPUT_REQUIRED)] # its one slot free while the user answers + assert turn.of("researcher-1") == ["started", "input_required", "running", "completed"] + assert turn.result.result["result"] == "B" + assert made.slots(CONVERSATION).free == 1 + + async def test_the_roots_own_question_lends_nothing(self): + manager = _manager() + manager.set_input_callback(AsyncMock(return_value="answer")) + + assert await manager.request_user_input(_request()) == "answer" diff --git a/tests/test_message_processor_render.py b/tests/test_message_processor_render.py index 1bd8f0c3..a39096a2 100644 --- a/tests/test_message_processor_render.py +++ b/tests/test_message_processor_render.py @@ -563,6 +563,33 @@ async def test_each_line_names_the_sub_agent_run_whose_work_it_is(self): ] assert ui.responses() == ["Done."] + async def test_a_run_that_waits_says_so_under_its_name(self): + """A run waiting for a free working slot, or for the user, says so; + once it works again, its line says that too. Its start and end write + no line (``/agents`` lists them).""" + events = [ + _by(RESEARCHER_1, WorkflowEvent.tool_call("lookup", {"q": "x"})), + WorkflowEvent.agent_run(RESEARCHER_2, "started", title="Find Y.", call_id="c2"), + WorkflowEvent.agent_run(RESEARCHER_2, "queued", call_id="c2"), + WorkflowEvent.agent_run(RESEARCHER_2, "running", call_id="c2"), + WorkflowEvent.agent_run(RESEARCHER_2, "input_required", call_id="c2"), + WorkflowEvent.agent_run(RESEARCHER_2, "running", call_id="c2"), + WorkflowEvent.agent_run(RESEARCHER_2, "completed", call_id="c2", result="Y."), + ] + + ui, lines = await _watch(events) + + assert lines == [ + "researcher-1 › Calling: lookup", + "researcher-1 › Calling: lookup", + "researcher-2 › Queued: waiting for a free slot", + "researcher-2 › Working...", + "researcher-2 › Waiting for your answer", + "researcher-2 › Working...", + "researcher-2 › Working...", + ] + assert ui.rich() == [] and ui.responses() == [] + async def test_a_sub_agents_text_without_a_run_keeps_its_agents_name(self): """An event that names no run (an app's own event processor) is shown as before: under its agent's name.""" diff --git a/tests/workflow/test_agent_brief.py b/tests/workflow/test_agent_brief.py index 07785791..fe0e88d7 100644 --- a/tests/workflow/test_agent_brief.py +++ b/tests/workflow/test_agent_brief.py @@ -274,7 +274,8 @@ def test_the_description(self, logged): "response run in parallel.\n\n" "A sub-agent sees only the brief you write, unless its line below says it sees " "this conversation: put in the brief everything it needs to know.\n\n" - "At most 20 runs per turn.\n\n" + "At most 4 sub-agents work at once; more calls in one response wait for a free " + "slot. At most 20 runs per turn.\n\n" "The agents you can hand a task to:\n" "- researcher: Finds facts.\n" "- writer (its brief also needs: deliverable, done_when): Writes the report." @@ -300,11 +301,19 @@ def test_a_blank_description_is_none(self, logged): ) assert [e["agent"] for e in logged] == ["researcher"] - def test_it_states_the_runs_a_turn_may_start_with_the_policys_number(self): + def test_it_states_the_working_slots_then_the_runs_a_turn_may_start_with_the_policys_numbers( + self, + ): policy = SubAgentPolicy(max_active=2, max_runs_per_turn=3) - assert brief_module.limit_sentences(policy) == ["At most 3 runs per turn."] - assert task_description(TYPES, policy).split("\n\n")[2] == "At most 3 runs per turn." + assert brief_module.limit_sentences(policy) == [ + "At most 2 sub-agents work at once; more calls in one response wait for a free slot.", + "At most 3 runs per turn.", + ] + assert task_description(TYPES, policy).split("\n\n")[2] == ( + "At most 2 sub-agents work at once; more calls in one response wait for a free " + "slot. At most 3 runs per turn." + ) def test_the_limits_in_force_are_one_paragraph_before_the_agents(self, monkeypatch): seen: list[SubAgentPolicy] = [] diff --git a/tests/workflow/test_agent_delegate.py b/tests/workflow/test_agent_delegate.py index c7d9a46e..5e55cee8 100644 --- a/tests/workflow/test_agent_delegate.py +++ b/tests/workflow/test_agent_delegate.py @@ -10,6 +10,7 @@ from __future__ import annotations import asyncio +import collections import contextlib from collections.abc import Awaitable, Callable, Iterator from typing import Any @@ -22,8 +23,11 @@ RUN_CANCELLED, RUN_COMPLETED, RUN_FAILED, + RUN_INPUT_REQUIRED, RUN_INTERRUPTED, + RUN_QUEUED, RUN_REFUSED, + RUN_RUNNING, RUN_STARTED, AgentLimits, AgentRuns, @@ -38,6 +42,7 @@ RunAnswer, RunFailed, started_by, + waiting_on_user, ) from agentic_cli.workflow.agents.envelope import ( INTERRUPTED_ERROR, @@ -1161,3 +1166,482 @@ async def test_outside_a_turn_nothing_is_counted(self): await delegated(made, request(call_id="c2")) assert len(asked) == 2 + + +# -- working slots: at most max_active runs of a conversation work at once ---------------- + +MIDDLE = {"agent_type": "middle", "title": "Plan", "objective": "Plan X."} +LEAF = {"agent_type": "leaf", "title": "Look", "objective": "Look X up."} + + +async def until(condition: Callable[[], bool], *, within: float = 5.0) -> None: + """Wait until ``condition()`` holds, letting the other tasks run.""" + loop = asyncio.get_running_loop() + deadline = loop.time() + within + while not condition(): + if loop.time() > deadline: + raise AssertionError(f"not within {within} s") + await asyncio.sleep(0) + + +class Turn: + """``work()`` in one turn of ``CONVERSATION`` (its span open), in a task of + its own, to stop it as Ctrl+C does: what it returned, and every event + reported meanwhile.""" + + def __init__(self, work: Callable[[], Awaitable[Any]]) -> None: + self.events: list[WorkflowEvent] = [] + self.result: Any = None + self.task = asyncio.create_task(self._run(work)) + + async def _run(self, work: Callable[[], Awaitable[Any]]) -> None: + live = LiveEvents() + + async def source(): + token = set_active_turn(CONVERSATION) + scope = open_turn_scope() + try: + with live_events.reporting(live): + self.result = await work() + finally: + close_turn_scope(scope) + reset_active_turn(token) + if False: # pragma: no cover - a source that yields nothing itself + yield + + async for event in merged(source(), live): + self.events.append(event) + + def of(self, instance: str) -> list[str]: + """The statuses ``instance``'s run reported, in order.""" + return [ + event.metadata["status"] for event in self.events + if event.type is EventType.AGENT_RUN and event.run.instance == instance + ] + + def order(self, *wanted: str) -> list[tuple[str, str]]: + """Each run's ``wanted`` statuses as reported, in order: (instance, status).""" + return [ + (event.run.instance, event.metadata["status"]) for event in self.events + if event.type is EventType.AGENT_RUN and event.metadata["status"] in wanted + ] + + +class Observed(AgentRuns): + """``AgentRuns`` that also keep each ``AGENT_RUN`` they observe, in order: + the events a stopped turn reports after its stop reach no consumer.""" + + def __init__(self) -> None: + super().__init__() + self.seen: list[tuple[str, str]] = [] + + def observe(self, event: WorkflowEvent) -> None: + self.seen.append((event.run.instance, event.metadata["status"])) + super().observe(event) + + def of(self, instance: str) -> list[str]: + """The statuses ``instance``'s run reported, in order.""" + return [status for name, status in self.seen if name == instance] + + +class Holding: + """A hook whose runs each work until let go (``go[instance].set()``); + ``working`` names the runs whose hook ran, in order.""" + + def __init__(self) -> None: + self.working: list[str] = [] + self.go: dict[str, asyncio.Event] = collections.defaultdict(asyncio.Event) + + async def __call__(self, run: AgentRun) -> RunAnswer: + self.working.append(run.ref.instance) + await self.go[run.ref.instance].wait() + return RunAnswer(text=f"{run.ref.instance} done.") + + +def fanning_out(made: Callable[[], Delegations], worked: list[str]) -> Hook: + """A hook whose ``middle`` runs each delegate two ``leaf`` runs at once and + answer with their results; any other run answers at once. ``worked`` + gets each run's work in the order it was done, each with its slot: a + leaf's answer, a middle's after its leaves (``middle-1 back``).""" + + async def hook(run: AgentRun) -> RunAnswer: + if run.ref.agent != "middle": + worked.append(run.ref.instance) + return RunAnswer(text=f"{run.ref.instance} found it.") + found = await asyncio.gather(*( + made().delegate(request( + LEAF, caller=run.ref, caller_agent="middle", call_id=f"{run.ref.instance}/{n}", + )) + for n in (1, 2) + )) + worked.append(f"{run.ref.instance} back") + return RunAnswer(text=" ".join(one.result["result"] for one in found)) + + return hook + + +class TestWorkingSlots: + async def test_a_run_past_the_slots_waits_queued_then_runs(self): + hook = Holding() + runs = AgentRuns() + made = delegations(hook, runs=runs, policy=SubAgentPolicy(max_active=2)) + pool = made.slots(CONVERSATION) + + turn = Turn(lambda: asyncio.gather( + *(made.delegate(request(call_id=call)) for call in ("c1", "c2", "c3")) + )) + await until(lambda: len(hook.working) == 2 and pool.waiting == 1) + assert hook.working == ["researcher-1", "researcher-2"] + assert runs.record(CONVERSATION, "researcher-3").status == RUN_QUEUED + hook.go["researcher-1"].set() + await until(lambda: "researcher-3" in hook.working) + assert runs.record(CONVERSATION, "researcher-3").status == RUN_RUNNING + hook.go["researcher-2"].set() + hook.go["researcher-3"].set() + await asyncio.wait_for(turn.task, 5) + + assert turn.of("researcher-3") == [RUN_STARTED, RUN_QUEUED, RUN_RUNNING, RUN_COMPLETED] + assert turn.of("researcher-1") == [RUN_STARTED, RUN_COMPLETED] + assert [one.result["result"] for one in turn.result] == [ + "researcher-1 done.", "researcher-2 done.", "researcher-3 done.", + ] + assert (pool.free, pool.waiting) == (2, 0) + + async def test_a_run_holds_a_slot_while_it_works_and_the_root_none(self): + seen: list[int] = [] + made: Delegations + + async def hook(run: AgentRun) -> RunAnswer: + seen.append(made.slots(CONVERSATION).free) + return RunAnswer(text="Done.") + + made = delegations(hook, policy=SubAgentPolicy(max_active=1)) + turn = Turn(lambda: made.delegate(request())) + await asyncio.wait_for(turn.task, 5) + + assert seen == [0] # its own; the root's work took none + assert turn.of("researcher-1") == [RUN_STARTED, RUN_COMPLETED] + assert made.slots(CONVERSATION).free == 1 + + async def test_each_conversation_has_its_own_slots(self): + hook, _ = answering() + made = delegations(hook, policy=SubAgentPolicy(max_active=3)) + + one, other = made.slots(CONVERSATION), made.slots(SessionRef("app", "u", "other")) + + assert one is made.slots(CONVERSATION) and one is not other + assert (one.size, other.size) == (3, 3) + + async def test_nested_fan_out_with_one_slot_completes(self): + """Spec 13: a parent lends its slot while it waits on its children, so + two runs that each delegate two at once finish with one slot.""" + made = delegations(fanning_out(lambda: made, []), policy=SubAgentPolicy(max_active=1)) + + turn = Turn(lambda: asyncio.gather( + made.delegate(request(MIDDLE, call_id="c1")), + made.delegate(request(MIDDLE, call_id="c2")), + )) + await asyncio.wait_for(turn.task, 5) + + assert [one.result["result"] for one in turn.result] == [ + "leaf-1 found it. leaf-2 found it.", "leaf-3 found it. leaf-4 found it.", + ] + pool = made.slots(CONVERSATION) + assert (pool.free, pool.waiting) == (1, 0) + + async def test_a_parent_whose_children_are_done_goes_ahead_of_runs_that_have_not_started( + self, + ): + """middle-1's last leaf hands its slot back to middle-1 while leaf-3 + and leaf-4 wait: middle-1 does its work before they start.""" + worked: list[str] = [] + made = delegations(fanning_out(lambda: made, worked), policy=SubAgentPolicy(max_active=1)) + + turn = Turn(lambda: asyncio.gather( + made.delegate(request(MIDDLE, call_id="c1")), + made.delegate(request(MIDDLE, call_id="c2")), + )) + await asyncio.wait_for(turn.task, 5) + + # middle-2 takes middle-1's slot, lent while middle-1 waits on its + # leaves; leaf-1 takes middle-2's; each queued run says when it runs. + assert turn.order(RUN_RUNNING) == [ + ("middle-2", RUN_RUNNING), ("leaf-1", RUN_RUNNING), ("leaf-2", RUN_RUNNING), + ("leaf-3", RUN_RUNNING), ("leaf-4", RUN_RUNNING), + ] + # Taking its slot back is no new status: the work shows it. + assert worked == [ + "leaf-1", "leaf-2", "middle-1 back", "leaf-3", "leaf-4", "middle-2 back", + ] + + async def test_time_spent_queued_is_not_counted_against_the_timeout(self): + hook = Holding() + types = { + **TYPES, + "researcher": AgentConfig(name="researcher", prompt="p", description="Finds.", + limits=AgentLimits(timeout=0.2)), + } + made = delegations(hook, types=types, policy=SubAgentPolicy(max_active=1)) + hook.go["researcher-1"].set() # answers as soon as it has a slot + + turn = Turn(lambda: asyncio.gather( + made.delegate(request(MIDDLE, call_id="c1")), + made.delegate(request(call_id="c2")), + )) + await until(lambda: made.slots(CONVERSATION).waiting == 1) + await asyncio.sleep(0.3) # past the researcher's time limit, queued + hook.go["middle-1"].set() + await asyncio.wait_for(turn.task, 5) + + assert turn.of("researcher-1") == [RUN_STARTED, RUN_QUEUED, RUN_RUNNING, RUN_COMPLETED] + + async def test_queued_runs_block_no_event_loop(self): + hook = Holding() + made = delegations(hook, policy=SubAgentPolicy(max_active=1)) + hook.go["researcher-2"].set() + + with no_blocking_calls(): + turn = Turn(lambda: asyncio.gather( + made.delegate(request(call_id="c1")), made.delegate(request(call_id="c2")) + )) + await until(lambda: made.slots(CONVERSATION).waiting == 1) + hook.go["researcher-1"].set() + await asyncio.wait_for(turn.task, 5) + + assert turn.of("researcher-2") == [RUN_STARTED, RUN_QUEUED, RUN_RUNNING, RUN_COMPLETED] + + +class TestACancelWhileQueued: + async def test_the_turns_leaves_the_heap_and_the_run_never_starts(self): + hook = Holding() + runs = AgentRuns() + made = delegations(hook, runs=runs, policy=SubAgentPolicy(max_active=1)) + pool = made.slots(CONVERSATION) + + turn = Turn(lambda: asyncio.gather( + made.delegate(request(call_id="c1")), made.delegate(request(call_id="c2")) + )) + await until(lambda: hook.working == ["researcher-1"] and pool.waiting == 1) + turn.task.cancel() + with pytest.raises(asyncio.CancelledError): + await turn.task + + assert hook.working == ["researcher-1"] + assert [(r.ref.instance, r.status) for r in runs.records(CONVERSATION)] == [ + ("researcher-1", RUN_INTERRUPTED), ("researcher-2", RUN_INTERRUPTED), + ] + assert (pool.free, pool.waiting) == (1, 0) + + async def test_its_controls_ends_it_cancelled_with_nothing_and_the_rest_go_on(self): + hook = Holding() + made = delegations(hook, policy=SubAgentPolicy(max_active=1)) + pool = made.slots(CONVERSATION) + + turn = Turn(lambda: asyncio.gather( + made.delegate(request(call_id="c1")), made.delegate(request(call_id="c2")) + )) + await until(lambda: hook.working == ["researcher-1"] and pool.waiting == 1) + token = set_active_turn(CONVERSATION) + try: + assert made.cancel("researcher-2#1", by="parent", error="no longer needed") is True + finally: + reset_active_turn(token) + await until(lambda: pool.waiting == 0) + hook.go["researcher-1"].set() + await asyncio.wait_for(turn.task, 5) + + first, second = turn.result + assert first.result["status"] == RUN_COMPLETED + assert (second.result["status"], second.result["cancelled_by"], second.result["result"], + second.result["partial"]) == (RUN_CANCELLED, "parent", "", False) + assert turn.of("researcher-2") == [RUN_STARTED, RUN_QUEUED, RUN_CANCELLED] + assert hook.working == ["researcher-1"] + assert (pool.free, pool.waiting) == (1, 0) + + async def test_ctrl_c_with_a_run_queued_and_another_lending_leaves_no_waiter(self): + """middle-1 lends its slot to researcher-1 while leaf-1 + waits for one; Ctrl+C stops all three, leaf-1 never starts, and the next + turn's delegation gets a slot at once.""" + hook = Holding() + runs = AgentRuns() + made: Delegations + + async def runs_or_delegates(run: AgentRun) -> RunAnswer: + if run.ref.agent == "middle": + await made.delegate(request( + LEAF, caller=run.ref, caller_agent="middle", call_id="m1", + )) + return await hook(run) + + made = delegations(runs_or_delegates, runs=runs, policy=SubAgentPolicy(max_active=1)) + pool = made.slots(CONVERSATION) + + turn = Turn(lambda: asyncio.gather( + made.delegate(request(MIDDLE, call_id="c1")), made.delegate(request(call_id="c2")) + )) + await until(lambda: "researcher-1" in hook.working and pool.waiting == 1) + assert runs.record(CONVERSATION, "leaf-1").status == RUN_QUEUED + turn.task.cancel() + with pytest.raises(asyncio.CancelledError): + await turn.task + + assert "leaf-1" not in hook.working + assert {r.ref.instance: r.status for r in runs.records(CONVERSATION)} == { + "middle-1": RUN_INTERRUPTED, "researcher-1": RUN_INTERRUPTED, + "leaf-1": RUN_INTERRUPTED, + } + assert (pool.free, pool.waiting) == (1, 0) + + hook.go["researcher-2"].set() + after = Turn(lambda: made.delegate(request(call_id="c3"))) + await asyncio.wait_for(after.task, 5) + + assert after.of("researcher-2") == [RUN_STARTED, RUN_COMPLETED] + + +# -- a run waiting on the user lends its slot ------------------------------------------------ + + +class TestWaitingOnTheUser: + """A sub-agent run that waits on the user (``waiting_on_user``: a + question, a permission prompt) says ``input_required``, lends its slot + meanwhile, and takes it back ahead of the runs that have not started + (``running``).""" + + async def test_a_run_lends_its_slot_and_takes_it_back_ahead_of_new_runs(self): + hook = Holding() + runs = AgentRuns() + answered = asyncio.Event() + + async def asking(run: AgentRun) -> RunAnswer: + if run.ref.instance != "researcher-1": + return await hook(run) + async with waiting_on_user(): + await answered.wait() + hook.working.append("researcher-1 back") + return RunAnswer(text="Asked.") + + made = delegations(asking, runs=runs, policy=SubAgentPolicy(max_active=1)) + pool = made.slots(CONVERSATION) + hook.go["researcher-3"].set() + + turn = Turn(lambda: asyncio.gather( + *(made.delegate(request(call_id=call)) for call in ("c1", "c2", "c3")) + )) + await until(lambda: hook.working == ["researcher-2"] and pool.waiting == 1) + assert runs.record(CONVERSATION, "researcher-1").status == RUN_INPUT_REQUIRED + answered.set() + await until(lambda: pool.waiting == 2) # researcher-1 wants it back; researcher-3 waits + hook.go["researcher-2"].set() + await asyncio.wait_for(turn.task, 5) + + assert hook.working == ["researcher-2", "researcher-1 back", "researcher-3"] + assert turn.of("researcher-1") == [ + RUN_STARTED, RUN_INPUT_REQUIRED, RUN_RUNNING, RUN_COMPLETED, + ] + assert (pool.free, pool.waiting) == (1, 0) + + async def test_nested_waits_count_once(self): + async def hook(run: AgentRun) -> RunAnswer: + async with waiting_on_user(): + async with waiting_on_user(): # a prompt, its turn behind another's + pass + return RunAnswer(text="Asked.") + + made = delegations(hook) + turn = Turn(lambda: made.delegate(request())) + await asyncio.wait_for(turn.task, 5) + + assert turn.of("researcher-1") == [ + RUN_STARTED, RUN_INPUT_REQUIRED, RUN_RUNNING, RUN_COMPLETED, + ] + + async def test_the_roots_own_wait_lends_nothing_and_says_nothing(self): + live = LiveEvents() + + async def source(): + with live_events.reporting(live): + async with waiting_on_user(): + pass + if False: # pragma: no cover - a source that yields nothing itself + yield + + assert [event async for event in merged(source(), live)] == [] + + async def test_a_child_ending_while_its_parent_waits_on_the_user_gives_its_slot_to_the_pool( + self, + ): + answered = asyncio.Event() + runs = AgentRuns() + made: Delegations + + async def hook(run: AgentRun) -> RunAnswer: + if run.ref.agent != "middle": + return RunAnswer(text="Found.") + + async def ask() -> None: + async with waiting_on_user(): + await answered.wait() + + await asyncio.gather( + made.delegate(request(LEAF, caller=run.ref, caller_agent="middle", call_id="m1")), + ask(), + ) + return RunAnswer(text="Planned.") + + made = delegations(hook, runs=runs, policy=SubAgentPolicy(max_active=1)) + pool = made.slots(CONVERSATION) + turn = Turn(lambda: made.delegate(request(MIDDLE))) + await until(lambda: runs.record(CONVERSATION, "leaf-1") is not None + and runs.record(CONVERSATION, "leaf-1").status == RUN_COMPLETED) + + assert pool.free == 1 # not handed back: middle-1 still waits on the user + answered.set() + await asyncio.wait_for(turn.task, 5) + + assert turn.of("middle-1") == [RUN_STARTED, RUN_INPUT_REQUIRED, RUN_RUNNING, RUN_COMPLETED] + assert (pool.free, pool.waiting) == (1, 0) + + async def test_ctrl_c_while_a_run_waits_on_the_user_and_another_is_queued(self): + """researcher-1 lends its slot to researcher-2 while + the user is slow to answer and researcher-3 waits for a slot; Ctrl+C + stops all three, researcher-3 never starts, nothing waits, and + researcher-1 takes nothing back.""" + hook = Holding() + runs = Observed() + + async def asking(run: AgentRun) -> RunAnswer: + if run.ref.instance == "researcher-1": + async with waiting_on_user(): + await asyncio.Event().wait() # the user is slow to answer + return await hook(run) + + made = delegations(asking, runs=runs, policy=SubAgentPolicy(max_active=1)) + pool = made.slots(CONVERSATION) + + turn = Turn(lambda: asyncio.gather( + *(made.delegate(request(call_id=call)) for call in ("c1", "c2", "c3")) + )) + await until(lambda: hook.working == ["researcher-2"] and pool.waiting == 1) + turn.task.cancel() + with pytest.raises(asyncio.CancelledError): + await turn.task + + assert hook.working == ["researcher-2"] + assert runs.of("researcher-1") == [RUN_STARTED, RUN_INPUT_REQUIRED, RUN_INTERRUPTED] + assert runs.of("researcher-3") == [RUN_STARTED, RUN_QUEUED, RUN_INTERRUPTED] + assert [r.status for r in runs.records(CONVERSATION)] == [RUN_INTERRUPTED] * 3 + assert (pool.free, pool.waiting) == (1, 0) + + async def test_a_wait_on_the_user_blocks_no_event_loop(self): + async def hook(run: AgentRun) -> RunAnswer: + async with waiting_on_user(): + await asyncio.sleep(0) + return RunAnswer(text="Asked.") + + made = delegations(hook, policy=SubAgentPolicy(max_active=1)) + with no_blocking_calls(): + turn = Turn(lambda: made.delegate(request())) + await asyncio.wait_for(turn.task, 5) + + assert turn.of("researcher-1")[-1] == RUN_COMPLETED diff --git a/tests/workflow/test_agent_runs.py b/tests/workflow/test_agent_runs.py index 935012e0..d9e0f419 100644 --- a/tests/workflow/test_agent_runs.py +++ b/tests/workflow/test_agent_runs.py @@ -19,8 +19,12 @@ from agentic_cli.workflow.agents import ( RUN_COMPLETED, RUN_FAILED, + RUN_INPUT_REQUIRED, RUN_INTERRUPTED, + RUN_QUEUED, + RUN_RUNNING, RUN_STARTED, + RUN_WORKING_STATUSES, AgentRuns, RunRecord, result_head, @@ -189,9 +193,34 @@ def test_a_start_is_recorded(self): in_turn(ONE, lambda: runs.observe(event)) assert runs.records(ONE) == [RunRecord( - ref=ref, call_id="c1", title="Find X.", status=RUN_STARTED, started=event.timestamp, + ref=ref, call_id="c1", title="Find X.", status=RUN_RUNNING, started=event.timestamp, )] + def test_a_working_runs_status_is_the_latest_it_reported(self): + runs = AgentRuns() + ref = started(runs, ONE, "c1", "researcher") + seen: list[str] = [] + + for status in (RUN_QUEUED, RUN_RUNNING, RUN_INPUT_REQUIRED, RUN_RUNNING): + in_turn(ONE, lambda: runs.observe(WorkflowEvent.agent_run(ref, status, call_id="c1"))) + seen.append(runs.records(ONE)[0].status) + + assert seen == [RUN_QUEUED, RUN_RUNNING, RUN_INPUT_REQUIRED, RUN_RUNNING] + assert RUN_WORKING_STATUSES == {RUN_QUEUED, RUN_RUNNING, RUN_INPUT_REQUIRED} + + def test_a_working_status_after_the_end_or_without_a_start_changes_nothing(self): + runs = AgentRuns() + ended = started(runs, ONE, "c1", "researcher") + in_turn(ONE, lambda: runs.observe( + WorkflowEvent.agent_run(ended, RUN_COMPLETED, call_id="c1", result="ok") + )) + unknown = runs.run_for(ONE, None, "c2", "researcher") + + in_turn(ONE, lambda: runs.observe(WorkflowEvent.agent_run(ended, RUN_RUNNING))) + in_turn(ONE, lambda: runs.observe(WorkflowEvent.agent_run(unknown, RUN_QUEUED))) + + assert [(r.ref, r.status) for r in runs.records(ONE)] == [(ended, RUN_COMPLETED)] + def test_an_end_ends_the_record(self): runs = AgentRuns() ref = started(runs, ONE, "c1", "researcher") diff --git a/tests/workflow/test_agent_slots.py b/tests/workflow/test_agent_slots.py new file mode 100644 index 00000000..93e02a41 --- /dev/null +++ b/tests/workflow/test_agent_slots.py @@ -0,0 +1,420 @@ +"""Working slots (``workflow/agents/slots.py``), alone. + +A ``SlotPool`` lets ``size`` runs work at once; the others wait in a heap +ordered by priority (``RETURNING`` before ``NEW``), then arrival. A run that +waits on its children or the user lends its slot and takes it back after, +and every cancel leaves the pool as it found it: no waiter left in the heap, +no slot lost or counted twice. Driven here with plain asyncio tasks. +""" + +from __future__ import annotations + +import asyncio +from collections.abc import Callable + +import pytest + +from agentic_cli.workflow.agents.slots import NEW, RETURNING, Slot, SlotPool +from tests.loop_guard import no_blocking_calls + + +async def until(condition: Callable[[], bool], *, within: float = 5.0) -> None: + """Wait until ``condition()`` holds, letting the other tasks run.""" + loop = asyncio.get_running_loop() + deadline = loop.time() + within + while not condition(): + if loop.time() > deadline: + raise AssertionError(f"not within {within} s") + await asyncio.sleep(0) + + +def waiter(pool: SlotPool, name: str, served: list[str], priority: int = NEW) -> asyncio.Task[Slot]: + """A task that acquires a slot at ``priority`` and writes ``name`` once it has one.""" + + async def wait() -> Slot: + slot = await pool.acquire(priority) + served.append(name) + return slot + + return asyncio.create_task(wait(), name=name) + + +class TestAcquireAndRelease: + async def test_a_run_gets_a_free_slot_at_once_and_gives_it_back(self): + pool = SlotPool(2) + + first, second = await pool.acquire(), await pool.acquire() + + assert (pool.free, pool.waiting, first.held, second.held) == (0, 0, True, True) + pool.release(first) + assert (pool.free, first.held) == (1, False) + pool.release(first) # given back once: a second release is nothing + assert pool.free == 1 + + async def test_waiters_are_served_by_priority_then_arrival(self): + pool = SlotPool(1) + holder = await pool.acquire() + served: list[str] = [] + a = waiter(pool, "a", served) + b = waiter(pool, "b", served) + back = waiter(pool, "back", served, RETURNING) + await until(lambda: pool.waiting == 3) + + for task in (back, a, b): + pool.release(holder) + holder = await task + + assert served == ["back", "a", "b"] + assert (pool.free, pool.waiting) == (0, 0) + + def test_its_size_is_a_positive_int(self): + for size in (0, -1, True, 1.5, "2"): + with pytest.raises(ValueError, match="positive int"): + SlotPool(size) + + async def test_a_slot_goes_back_to_its_own_pool(self): + one, other = SlotPool(1), SlotPool(1) + slot = await one.acquire() + + with pytest.raises(ValueError, match="pool it came from"): + other.release(slot) + + +class TestCancellation: + async def test_a_cancelled_waiter_leaves_the_heap_and_the_next_is_served(self): + pool = SlotPool(1) + holder = await pool.acquire() + served: list[str] = [] + a, b = waiter(pool, "a", served), waiter(pool, "b", served) + await until(lambda: pool.waiting == 2) + + a.cancel() + with pytest.raises(asyncio.CancelledError): + await a + assert pool.waiting == 1 + pool.release(holder) + await b + + assert served == ["b"] + assert (pool.free, pool.waiting) == (0, 0) + + async def test_a_waiter_handed_a_slot_and_cancelled_in_the_same_tick_hands_it_on(self): + pool = SlotPool(1) + holder = await pool.acquire() + served: list[str] = [] + a, b = waiter(pool, "a", served), waiter(pool, "b", served) + await until(lambda: pool.waiting == 2) + + pool.release(holder) # a's: handed over, a not yet resumed + a.cancel() # the same tick + with pytest.raises(asyncio.CancelledError): + await a + slot = await b + + assert served == ["b"] + pool.release(slot) + assert (pool.free, pool.waiting) == (1, 0) + + async def test_a_cancelled_holder_releases_in_its_finally(self): + pool = SlotPool(1) + working = asyncio.Event() + + async def run() -> None: + slot = await pool.acquire() + try: + working.set() + await asyncio.Event().wait() + finally: + pool.release(slot) + + task = asyncio.create_task(run()) + await working.wait() + task.cancel() + with pytest.raises(asyncio.CancelledError): + await task + + assert (pool.free, pool.waiting) == (1, 0) + + +class TestLending: + async def test_a_lent_slot_works_for_another_run_and_is_taken_back_after(self): + pool = SlotPool(1) + parent = await pool.acquire() + served: list[str] = [] + + async with pool.lent(parent): + assert (parent.lent, parent.held, pool.free) == (True, False, 1) + child = await waiter(pool, "child", served) + pool.release(child) + + assert (parent.held, pool.free, served) == (True, 0, ["child"]) + pool.release(parent) + assert pool.free == 1 + + async def test_a_returning_run_goes_ahead_of_runs_that_have_not_started(self): + pool = SlotPool(1) + parent = await pool.acquire() + served: list[str] = [] + first = waiter(pool, "first", served) + await until(lambda: pool.waiting == 1) + + async def lends() -> None: + async with pool.lent(parent): + await until(lambda: served == ["first"]) + waiter(pool, "second", served) + await until(lambda: pool.waiting == 1) + served.append("parent") + + lending = asyncio.create_task(lends()) + holding = await first # the parent's lent slot + await until(lambda: pool.waiting == 2) # the parent waits to take it back, and "second" + pool.release(holding) + await lending + pool.release(parent) + await until(lambda: len(served) == 3) + + assert served == ["first", "parent", "second"] + + async def test_nested_lends_of_one_slot_count_once(self): + pool = SlotPool(1) + parent = await pool.acquire() + + async with pool.lent(parent): + async with pool.lent(parent): + assert pool.free == 1 + assert (parent.lent, pool.free) == (True, 1) # still waiting on the other + assert (parent.held, pool.free) == (True, 0) + + async def test_a_run_cancelled_while_its_slot_is_lent_does_not_take_it_back(self): + pool = SlotPool(1) + lending = asyncio.Event() + + async def run() -> None: + slot = await pool.acquire() + try: + async with pool.lent(slot): + lending.set() + await asyncio.Event().wait() + finally: + pool.release(slot) + + task = asyncio.create_task(run()) + await lending.wait() + other = await pool.acquire() # the lent slot, at work for another run + task.cancel() + with pytest.raises(asyncio.CancelledError): + await task + + assert (pool.free, pool.waiting) == (0, 0) + pool.release(other) + assert pool.free == 1 + + async def test_a_run_cancelled_while_it_waits_to_take_its_slot_back_leaves_the_heap(self): + pool = SlotPool(1) + lending = asyncio.Event() + + async def run() -> None: + slot = await pool.acquire() + try: + async with pool.lent(slot): + lending.set() + await asyncio.sleep(0) + finally: + pool.release(slot) + + task = asyncio.create_task(run()) + await lending.wait() + other = await pool.acquire() # holds the slot while the run wants it back + await until(lambda: pool.waiting == 1) + task.cancel() + with pytest.raises(asyncio.CancelledError): + await task + + assert pool.waiting == 0 + pool.release(other) + assert pool.free == 1 + + async def test_an_error_while_lent_is_no_cancel_the_slot_comes_back(self): + pool = SlotPool(1) + slot = await pool.acquire() + + with pytest.raises(RuntimeError): + async with pool.lent(slot): + raise RuntimeError("no one to ask") + + assert (slot.held, pool.free) == (True, 0) + + async def test_a_wait_that_starts_while_the_slot_waits_to_come_back_lends_it_on(self): + """Two waits of one run that do not overlap (two calls of one + response): the second starts while the first waits for the slot to + come back. The first ends without it, since the run waits again, and + the slot comes back once, after the second.""" + pool = SlotPool(1) + parent = await pool.acquire() + first_over, second_over = asyncio.Event(), asyncio.Event() + + async def waits(over: asyncio.Event) -> None: + async with pool.lent(parent): + await over.wait() + + first = asyncio.create_task(waits(first_over)) + await until(lambda: parent.lent) + other = await pool.acquire() # works in the parent's lent slot + first_over.set() + await until(lambda: pool.waiting == 1) # the first wants it back + second = asyncio.create_task(waits(second_over)) + try: + await until(first.done, within=1) + assert (parent.lent, pool.waiting) == (True, 0) + second_over.set() + await until(lambda: pool.waiting == 1) # the second wants it back + pool.release(other) + await until(second.done, within=1) + finally: + first.cancel() + second.cancel() + + assert (parent.held, pool.free, pool.waiting) == (True, 0, 0) + pool.release(parent) + assert (pool.free, pool.waiting) == (1, 0) + + async def test_a_wait_over_at_once_as_the_slot_comes_back_leaves_no_wait_behind(self): + """The run's first wait has been handed its slot back but has not yet + gone on when another of its waits starts and ends at once (a prompt + answered without waiting): the run holds its slot once, and nothing + of it is left waiting in the heap.""" + pool = SlotPool(1) + parent = await pool.acquire() + first_over = asyncio.Event() + + async def waits() -> None: + async with pool.lent(parent): + await first_over.wait() + + async def over_at_once() -> None: + async with pool.lent(parent): + pass + + first = asyncio.create_task(waits()) + await until(lambda: parent.lent) + other = await pool.acquire() # works in the parent's lent slot + first_over.set() + await until(lambda: pool.waiting == 1) # the first wants it back + second = asyncio.create_task(over_at_once()) # runs before the first wakes + pool.release(other) # the first is handed the slot + try: + await asyncio.wait_for(asyncio.gather(first, second), 1) + finally: + first.cancel() + second.cancel() + + assert (parent.held, pool.free, pool.waiting) == (True, 0, 0) + pool.release(parent) + assert (pool.free, pool.waiting) == (1, 0) + + async def test_a_slot_released_while_it_waits_to_come_back_is_not_taken(self): + pool = SlotPool(1) + slot = await pool.acquire() + over = asyncio.Event() + + async def waits() -> None: + async with pool.lent(slot): + await over.wait() + + lending = asyncio.create_task(waits()) + await until(lambda: slot.lent) + other = await pool.acquire() + over.set() + await until(lambda: pool.waiting == 1) + pool.release(slot) # its run ended meanwhile + try: + await until(lending.done, within=1) + finally: + lending.cancel() + + assert pool.waiting == 0 + pool.release(other) + assert (pool.free, pool.waiting) == (1, 0) + + +class TestHandBack: + async def test_a_child_hands_its_slot_to_its_parent_waiting_on_it_alone(self): + pool = SlotPool(1) + parent = await pool.acquire() + served: list[str] = [] + + async with pool.lent(parent): + child = await pool.acquire() + late = waiter(pool, "late", served) + await until(lambda: pool.waiting == 1) + pool.hand_back(child, to=parent) + assert (parent.held, child.held) == (True, False) + + assert (parent.held, pool.free, pool.waiting, served) == (True, 0, 1, []) + pool.release(parent) + await late + assert served == ["late"] + + async def test_to_a_parent_waiting_to_take_its_slot_back_ends_that_wait(self): + """The parent's wait is over and it waits in the heap for its slot + when its child hands it back: that take-back ends without a slot, + and the parent holds the child's.""" + pool = SlotPool(1) + parent = await pool.acquire() + over = asyncio.Event() + + async def waits() -> None: + async with pool.lent(parent): + await over.wait() + + lending = asyncio.create_task(waits()) + await until(lambda: parent.lent) + child = await pool.acquire() # works in the parent's lent slot + over.set() + await until(lambda: pool.waiting == 1) # the parent wants it back + pool.hand_back(child, to=parent) + try: + await until(lending.done, within=1) + finally: + lending.cancel() + + assert (parent.held, child.held, pool.free, pool.waiting) == (True, False, 0, 0) + pool.release(parent) + assert (pool.free, pool.waiting) == (1, 0) + + async def test_to_a_parent_that_does_not_lend_it_is_a_release(self): + pool = SlotPool(2) + parent, child = await pool.acquire(), await pool.acquire() + + pool.hand_back(child, to=parent) + + assert (parent.held, child.held, pool.free) == (True, False, 1) + + async def test_a_parent_handed_its_slot_lends_it_again_for_a_new_wait(self): + """A wait that starts before the lend its child ended has closed + lends the slot again; the last lend to end takes it back.""" + pool = SlotPool(1) + parent = await pool.acquire() + + async with pool.lent(parent): + child = await pool.acquire() + pool.hand_back(child, to=parent) + assert (parent.held, pool.free) == (True, 0) + async with pool.lent(parent): # the user, asked meanwhile + assert (parent.lent, pool.free) == (True, 1) + assert (parent.lent, pool.free) == (True, 1) + assert (parent.held, pool.free) == (True, 0) + + +async def test_no_path_blocks_the_loop(): + """Spec 13: every new async path once under the loop guard.""" + pool = SlotPool(1) + with no_blocking_calls(): + holder = await pool.acquire() + served: list[str] = [] + queued = waiter(pool, "queued", served) + async with pool.lent(holder): + pool.release(await queued) + pool.release(holder) + + assert (pool.free, pool.waiting, served) == (1, 0, ["queued"]) diff --git a/tests/workflow/test_subagents_contract.py b/tests/workflow/test_subagents_contract.py index cdb23374..20ecfe3b 100644 --- a/tests/workflow/test_subagents_contract.py +++ b/tests/workflow/test_subagents_contract.py @@ -38,14 +38,18 @@ RUN_CANCELLED, RUN_COMPLETED, RUN_FAILED, + RUN_INPUT_REQUIRED, RUN_INTERRUPTED, + RUN_QUEUED, RUN_REFUSED, + RUN_RUNNING, RUN_STARTED, AgentLimits, RunRecord, SubAgentPolicy, ) from agentic_cli.workflow.agents.control import STEP_LIMIT_NOTE, RunControl +from agentic_cli.workflow.agents.slots import SlotPool from agentic_cli.workflow.agents.envelope import ( STEP_LIMIT_ERROR, TIME_LIMIT_ERROR, @@ -467,6 +471,10 @@ def control(self, run: str) -> RunControl | None: finally: reset_active_turn(token) + def slots(self) -> SlotPool: + """The conversation's working slots (``Delegations.slots``).""" + return self.manager._delegations.slots(self.conversation) + def recorded(self, call_id: str, run: str | None = None) -> bool: """Whether the record holds ``call_id`` of ``run`` (None: the root's own call) as finished: its result is kept for a stop to answer with.""" @@ -1028,7 +1036,7 @@ async def test_a_stopped_turn_leaves_its_open_run_interrupted(self, make_crew): turn = crew.start() try: await asyncio.wait_for(BLOCKING["started"].wait(), 10) - await until(lambda: [r.status for r in crew.records()] == [RUN_STARTED]) + await until(lambda: [r.status for r in crew.records()] == [RUN_RUNNING]) finally: # Stopped whatever happened: a turn left running holds the turn # lock, and the fixture's cleanup would wait for it forever. @@ -2824,3 +2832,331 @@ async def test_a_message_steered_before_the_last_allowed_call_comes_before_the_n last = crew.bodies("researcher")[1] assert last.index("Be brief.") < last.index(STEP_LIMIT_NOTE) assert crew.last_message("researcher", 1).endswith(STEP_LIMIT_NOTE) + + +# -- working slots: at most max_active runs work at once, a waiting parent lending its slot ----- + + +def working(events: list[WorkflowEvent], *statuses: str) -> list[tuple[str, str]]: + """The turn's ``AGENT_RUN`` events of ``statuses``, in order: (instance, status).""" + return [ + (event.run.instance, event.metadata["status"]) + for event in runs_of(events) if event.metadata["status"] in statuses + ] + + +def fanning_out(make_crew) -> Crew: + """With one slot: a coordinator delegating to two middles in one response, + each delegating to two leaves in one response. Each leaf, and each + middle after its leaves, says who it is (``SEEN``) before it answers: + the order the runs worked in.""" + return make_crew( + [agent("coordinator", sub_agents=("middle",)), + agent("middle", sub_agents=("leaf",), tools=(contract_whoami,)), + agent("leaf", tools=(contract_whoami,))], + subagents=SubAgentPolicy(max_active=1), + coordinator=[calls(delegate("middle", "Do A.", "c1"), delegate("middle", "Do B.", "c2")), + says("Both planned.")], + middle=[calls(delegate("leaf", "Look A1 up.", "m1"), delegate("leaf", "Look A2 up.", "m2")), + calls(delegate("leaf", "Look B1 up.", "m3"), delegate("leaf", "Look B2 up.", "m4")), + calls(use("contract_whoami", "w1")), says("A planned."), + calls(use("contract_whoami", "w2")), says("B planned.")], + leaf=[calls(use("contract_whoami", "l1")), says("A1 found."), + calls(use("contract_whoami", "l2")), says("A2 found."), + calls(use("contract_whoami", "l3")), says("B1 found."), + calls(use("contract_whoami", "l4")), says("B2 found.")], + ) + + +class TestWorkingSlots: + """Spec 5.5 and 13: a conversation's runs work in at most ``max_active`` + slots, at any depth; a run past them waits ``queued``, and a run that + waits on its children lends its slot and takes it back ahead of the + runs that have not started.""" + + async def test_a_third_parallel_call_waits_queued_then_runs(self, make_crew): + crew = make_crew( + [agent("coordinator", sub_agents=("researcher",)), + agent("researcher", tools=(contract_hold,))], + subagents=SubAgentPolicy(max_active=2), + coordinator=[calls(delegate("researcher", "Find X.", "c1"), + delegate("researcher", "Find Y.", "c2"), + delegate("researcher", "Find Z.", "c3")), + says("Done.")], + researcher=[calls(use("contract_hold", "r1")), calls(use("contract_hold", "r2")), + says("Found."), says("Found."), says("Found.")], + ) + + with no_blocking_calls(): + turn = crew.start() + try: + await until( + lambda: crew.requests("researcher") == 2 and crew.slots().waiting == 1 + ) + assert [r.status for r in crew.records()] == [ + RUN_RUNNING, RUN_RUNNING, RUN_QUEUED, + ] + finally: + HELD["go"].set() # never a turn left holding: the cleanup would wait on it + events = await asyncio.wait_for(turn, 10) + + assert working(events, RUN_QUEUED, RUN_RUNNING) == [ + ("researcher-3", RUN_QUEUED), ("researcher-3", RUN_RUNNING), + ] + read = crew.results_read("coordinator") + assert [envelope_read(read[c])["status"] for c in ("c1", "c2", "c3")] == [RUN_COMPLETED] * 3 + assert (crew.slots().free, crew.slots().waiting) == (2, 0) + + async def test_nested_fan_out_with_one_slot_completes(self, make_crew): + crew = fanning_out(make_crew) + + events = await asyncio.wait_for(crew.turn(), 20) + + read = crew.results_read("coordinator") + assert [envelope_read(read[c])["result"] for c in ("c1", "c2")] == [ + "A planned.", "B planned.", + ] + first = crew.results_read("middle", 2) # middle-1's request after its leaves + assert [envelope_read(first[c])["result"] for c in ("m1", "m2")] == [ + "A1 found.", "A2 found.", + ] + assert {r.ref.instance: r.status for r in crew.records()} == { + name: RUN_COMPLETED + for name in ("middle-1", "middle-2", "leaf-1", "leaf-2", "leaf-3", "leaf-4") + } + assert [e for e in events if e.type is EventType.ERROR] == [] + assert (crew.slots().free, crew.slots().waiting) == (1, 0) + + async def test_a_parent_whose_children_are_done_goes_ahead_of_new_runs(self, make_crew): + """middle-1's last leaf hands its slot back to middle-1 while leaf-3 + and leaf-4 wait for one: middle-1 works before they start.""" + crew = fanning_out(make_crew) + + events = await asyncio.wait_for(crew.turn(), 20) + + # middle-2 runs in middle-1's slot, lent while middle-1 waits on its + # leaves, and leaf-1 in middle-2's; every leaf waited for a slot. + assert working(events, RUN_RUNNING) == [ + ("middle-2", RUN_RUNNING), ("leaf-1", RUN_RUNNING), ("leaf-2", RUN_RUNNING), + ("leaf-3", RUN_RUNNING), ("leaf-4", RUN_RUNNING), + ] + # Taking its slot back is no new status; the order the runs worked + # in shows it. + assert [(run.instance, call) for run, call in SEEN] == [ + ("leaf-1", "l1"), ("leaf-2", "l2"), ("middle-1", "w1"), + ("leaf-3", "l3"), ("leaf-4", "l4"), ("middle-2", "w2"), + ] + + async def test_ctrl_c_with_a_run_queued_and_another_lending_leaves_no_waiter( + self, make_crew + ): + """middle-1 lends its slot to researcher-1 while its + leaf waits for one; Ctrl+C stops the turn, the leaf never starts, + nothing waits in the pool, and the next turn's run starts at once.""" + crew = make_crew( + [agent("coordinator", sub_agents=("middle", "researcher")), + agent("middle", sub_agents=("leaf",)), + agent("leaf"), + agent("researcher", tools=(contract_block,))], + subagents=SubAgentPolicy(max_active=1), + coordinator=[calls(delegate("middle", "Plan X.", "c1"), + delegate("researcher", "Find X.", "c2")), + calls(delegate("researcher", "Find Y.", "c3")), says("Y is 7.")], + middle=[calls(delegate("leaf", "Look X up.", "m1"))], + leaf=[says("[never asked]")], + researcher=[calls(use("contract_block", "r1")), says("Y is 7.")], + ) + + with no_blocking_calls(): + turn = crew.start() + try: + await asyncio.wait_for(BLOCKING["started"].wait(), 10) + await until(lambda: crew.slots().waiting == 1) + finally: + await crew.stop(turn) + + assert crew.requests("leaf") == 0 + assert {r.ref.instance: r.status for r in crew.records()} == { + "middle-1": RUN_INTERRUPTED, "researcher-1": RUN_INTERRUPTED, + "leaf-1": RUN_INTERRUPTED, + } + assert (crew.slots().free, crew.slots().waiting) == (1, 0) + + events = await asyncio.wait_for(crew.turn("Now Y."), 10) + + assert working(events, RUN_STARTED, RUN_QUEUED, RUN_RUNNING, RUN_COMPLETED) == [ + ("researcher-2", RUN_STARTED), ("researcher-2", RUN_COMPLETED), + ] + assert envelope_read(crew.results_read("coordinator")["c3"])["result"] == "Y is 7." + + +def asking_crew(make_crew, *, holder: Step, **scripts: list[Step]) -> Crew: + """With one slot: a coordinator delegating, in one response, to an asker + (who asks the user, ``contract_ask``), a holder (whose first answer is + ``holder``) and a researcher, in that order.""" + return make_crew( + [agent("coordinator", sub_agents=("asker", "holder", "researcher")), + agent("asker", tools=(contract_ask,)), + agent("holder", tools=(contract_hold, contract_block)), + agent("researcher")], + subagents=SubAgentPolicy(max_active=1), + coordinator=[calls(delegate("asker", "Ask which.", "c1"), + delegate("holder", "Hold on.", "c2"), + delegate("researcher", "Find X.", "c3")), + *scripts.pop("coordinator", [says("Done.")])], + asker=[calls(use("contract_ask", "a1")), says("Asked.")], + holder=[holder, says("Held.")], + researcher=[says("Found.")], + **scripts, + ) + + +class TestWaitingOnTheUser: + """Spec 5.5 and 6.2: a run waiting on the user (a question, a permission + prompt) is ``input_required`` and lends its slot meanwhile; once the user + has answered it takes a slot back ahead of the runs that have not + started, and is ``running`` again.""" + + async def test_a_run_at_a_question_lends_its_slot_and_takes_it_back_first(self, make_crew): + crew = asking_crew(make_crew, holder=calls(use("contract_hold", "h1"))) + while_asked: list[str] = [] + + async def answer(request: UserInputRequest) -> str: + await HELD["started"].wait() # the holder works in the asker's slot + while_asked.extend(r.status for r in crew.records()) + return "B" + + crew.answer_with(answer) + with no_blocking_calls(): + turn = crew.start() + try: + # The asker waits for its slot back, ahead of the researcher. + await until(lambda: crew.slots().waiting == 2) + finally: + HELD["go"].set() + events = await asyncio.wait_for(turn, 10) + + assert while_asked == [RUN_INPUT_REQUIRED, RUN_RUNNING, RUN_QUEUED] + assert working(events, RUN_INPUT_REQUIRED, RUN_RUNNING) == [ + ("asker-1", RUN_INPUT_REQUIRED), ("holder-1", RUN_RUNNING), + ("asker-1", RUN_RUNNING), ("researcher-1", RUN_RUNNING), + ] + # The asker took its slot back and answered before the researcher ran. + ended = working(events, RUN_COMPLETED) + assert ended.index(("asker-1", RUN_COMPLETED)) < ended.index( + ("researcher-1", RUN_COMPLETED) + ) + assert envelope_read(crew.results_read("coordinator")["c1"])["result"] == "Asked." + assert (crew.slots().free, crew.slots().waiting) == (1, 0) + + async def test_two_runs_at_permission_prompts_with_one_slot_both_finish( + self, make_crew, project + ): + """The permission engine shows one prompt at a time: a run waiting + for its turn to ask lends its slot too, or the run asking first could + never take its slot back.""" + crew = make_crew( + [agent("coordinator", sub_agents=("researcher",)), + agent("researcher", tools=("write_file",))], + subagents=SubAgentPolicy(max_active=1), + coordinator=[calls(delegate("researcher", "Write one.", "c1"), + delegate("researcher", "Write two.", "c2")), + says("Both written.")], + researcher=[calls(use("write_file", "r1", path="one.txt", content="1")), + calls(use("write_file", "r2", path="two.txt", content="2")), + says("Written."), says("Written.")], + ) + + async def allow_once(request: UserInputRequest) -> str: + if request.run.instance == "researcher-1": + # Answered once researcher-2 waits for its own prompt, in the + # slot researcher-1 lent. + await until(lambda: crew.records()[1].status == RUN_INPUT_REQUIRED) + return ALLOW_ONCE_CHOICE + + crew.answer_with(allow_once) + events = await asyncio.wait_for(crew.turn(), 15) + + assert (project / "one.txt").read_text() == "1" + assert (project / "two.txt").read_text() == "2" + read = crew.results_read("coordinator") + assert [envelope_read(read[c])["status"] for c in ("c1", "c2")] == [RUN_COMPLETED] * 2 + assert working(events, RUN_QUEUED, RUN_INPUT_REQUIRED, RUN_RUNNING) == [ + ("researcher-2", RUN_QUEUED), + ("researcher-1", RUN_INPUT_REQUIRED), ("researcher-2", RUN_RUNNING), + ("researcher-2", RUN_INPUT_REQUIRED), ("researcher-1", RUN_RUNNING), + ("researcher-2", RUN_RUNNING), + ] + assert (crew.slots().free, crew.slots().waiting) == (1, 0) + + async def test_ctrl_c_at_a_question_with_a_run_queued_leaves_no_waiter(self, make_crew): + """The asker lends its slot to the holder while the + user is slow to answer and the researcher waits for a slot; Ctrl+C + stops the turn, the researcher never starts, nothing waits, and the + next turn's run starts at once.""" + crew = asking_crew( + make_crew, holder=calls(use("contract_block", "h1")), + coordinator=[calls(delegate("researcher", "Find Y.", "c4")), says("Y is 7.")], + ) + withdrawn: list[str] = [] + + async def slow(request: UserInputRequest) -> str: + try: + await asyncio.Event().wait() + except asyncio.CancelledError: + withdrawn.append(request.run.instance) + raise + return "B" # pragma: no cover + + crew.answer_with(slow) + with no_blocking_calls(): + turn = crew.start() + try: + await asyncio.wait_for(BLOCKING["started"].wait(), 10) + await until(lambda: crew.slots().waiting == 1) + assert [r.status for r in crew.records()] == [ + RUN_INPUT_REQUIRED, RUN_RUNNING, RUN_QUEUED, + ] + finally: + await crew.stop(turn) + + assert withdrawn == ["asker-1"] + assert crew.requests("researcher") == 0 + assert [r.status for r in crew.records()] == [RUN_INTERRUPTED] * 3 + assert (crew.slots().free, crew.slots().waiting) == (1, 0) + + events = await asyncio.wait_for(crew.turn("Now Y."), 10) + + assert working(events, RUN_STARTED, RUN_QUEUED, RUN_RUNNING, RUN_COMPLETED) == [ + ("researcher-2", RUN_STARTED), ("researcher-2", RUN_COMPLETED), + ] + + async def test_a_prompt_no_one_can_answer_lends_nothing(self, make_crew, project): + """Headless (no input callback), a sub-agent run's permission prompt + is denied at once: the run keeps its slot and finishes before the + run queued for one starts, and never says it waits on the user.""" + crew = make_crew( + [agent("coordinator", sub_agents=("researcher",)), + agent("researcher", tools=("write_file", contract_whoami))], + subagents=SubAgentPolicy(max_active=1), + coordinator=[calls(delegate("researcher", "Write one.", "c1"), + delegate("researcher", "Write two.", "c2")), + says("Neither written.")], + researcher=[calls(use("write_file", "r1", path="one.txt", content="1")), + calls(use("contract_whoami", "w1")), says("Not written."), + calls(use("write_file", "r2", path="two.txt", content="2")), + calls(use("contract_whoami", "w2")), says("Not written.")], + ) + + events = await asyncio.wait_for(crew.turn(), 15) + + assert not (project / "one.txt").exists() and not (project / "two.txt").exists() + read = crew.results_read("coordinator") + assert [envelope_read(read[c])["result"] for c in ("c1", "c2")] == ["Not written."] * 2 + assert working(events, RUN_QUEUED, RUN_INPUT_REQUIRED, RUN_RUNNING) == [ + ("researcher-2", RUN_QUEUED), ("researcher-2", RUN_RUNNING), + ] + assert [(run.instance, call) for run, call in SEEN] == [ + ("researcher-1", "w1"), ("researcher-2", "w2"), + ] + assert (crew.slots().free, crew.slots().waiting) == (1, 0)