Skip to content

Tests: live checks of sub-agent delegation and limits - #325

Merged
shoom1 merged 2 commits into
developfrom
test/subagents-live
Nov 18, 2026
Merged

shoom1 merged 2 commits into
developfrom
test/subagents-live

Conversation

@shoom1

@shoom1 shoom1 commented Nov 18, 2026

Copy link
Copy Markdown
Owner

Adds live checks of sub-agent delegation against real providers: tests/integration/test_subagents_live.py, marked llm. Each check runs on every keyed provider (Google, Anthropic, OpenAI, OpenRouter where configured), on both ADK and Deep Agents:

  • Delegation through the framework's task tool. The provider accepts task's declaration (on ADK as parameters_json_schema), the run completes, and the coordinator is sent the envelope.
  • The step limit's last call. With max_model_calls=2, the provider accepts the last request (tools still declared, the note last). The run ends completed, partial, "step limit reached".
  • The token limit. It ends the run cancelled by the limit, or completes it when the crossing answer calls no tool.

Assertions read the raw request log and the framework's records, not the model's wording, and a task call the framework refuses (a plain error result) is tolerated. A full run with every provider keyed costs about 107 model calls, under US$0.25. Spend is capped with max_runs_per_turn=3.

Run on 2026-11-17: 18 passed (Gemini 2.5 Flash, Claude Haiku 4.5, OpenRouter Qwen 3.7 Plus, each on both backends); OpenAI skipped (no key). Offline the module is deselected (24 cases).

Run a coordinator and one sub-agent type on each provider that has a key
(Gemini, Claude, OpenAI, OpenRouter), on ADK and on Deep Agents, and check
what the offline contract suite cannot: that the real API takes what the
framework sends.

- delegation through task: the provider accepts its declaration, the run
  completes, and the coordinator reads the envelope's success;
- the step limit's last call: its request, tools still declared and
  ending with STEP_LIMIT_NOTE (read back from the raw LLM log), is
  accepted, and the run completes partial, "step limit reached";
- the token limit: the run is cancelled by the limit, or completes when
  the answer that crosses it makes no tool calls.

Each test keeps HOME and the working directory in its temporary folder,
so the raw log never lands under the user's home.
The review found checks that failed on legitimate model behaviour:

- a task result that is no envelope (a brief that cannot start a run, a
  call past max_runs_per_turn) failed the test; only envelopes are read
  now, and the first one is checked;
- the replies had to copy a field (SUCCESS, the status); the raw LLM log
  already shows the coordinator was sent each envelope, so only "42" is
  read from a reply;
- the step limit's note is matched as the last text part, unescaped;
- each record is the envelope's run's, not the first;
- an ERROR is reported before any envelope is unpacked;
- a token-limit case whose first answer makes no tool call skips: the
  limit had nothing to cancel.

A turn starts at most 3 runs (Qwen re-delegates), and the docstring
states what a full run costs.
@shoom1
shoom1 merged commit e244284 into develop Nov 18, 2026
6 checks passed
@shoom1
shoom1 deleted the test/subagents-live branch November 18, 2026 03:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant