Repository navigation
Tests: live checks of sub-agent delegation and limits - #325
Merged
Merged
Conversation
Run a coordinator and one sub-agent type on each provider that has a key (Gemini, Claude, OpenAI, OpenRouter), on ADK and on Deep Agents, and check what the offline contract suite cannot: that the real API takes what the framework sends. - delegation through task: the provider accepts its declaration, the run completes, and the coordinator reads the envelope's success; - the step limit's last call: its request, tools still declared and ending with STEP_LIMIT_NOTE (read back from the raw LLM log), is accepted, and the run completes partial, "step limit reached"; - the token limit: the run is cancelled by the limit, or completes when the answer that crosses it makes no tool calls. Each test keeps HOME and the working directory in its temporary folder, so the raw log never lands under the user's home.
The review found checks that failed on legitimate model behaviour: - a task result that is no envelope (a brief that cannot start a run, a call past max_runs_per_turn) failed the test; only envelopes are read now, and the first one is checked; - the replies had to copy a field (SUCCESS, the status); the raw LLM log already shows the coordinator was sent each envelope, so only "42" is read from a reply; - the step limit's note is matched as the last text part, unescaped; - each record is the envelope's run's, not the first; - an ERROR is reported before any envelope is unpacked; - a token-limit case whose first answer makes no tool call skips: the limit had nothing to cancel. A turn starts at most 3 runs (Qwen re-delegates), and the docstring states what a full run costs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds live checks of sub-agent delegation against real providers:
tests/integration/test_subagents_live.py, markedllm. Each check runs on every keyed provider (Google, Anthropic, OpenAI, OpenRouter where configured), on both ADK and Deep Agents:tasktool. The provider acceptstask's declaration (on ADK asparameters_json_schema), the run completes, and the coordinator is sent the envelope.max_model_calls=2, the provider accepts the last request (tools still declared, the note last). The run ends completed, partial, "step limit reached".Assertions read the raw request log and the framework's records, not the model's wording, and a
taskcall the framework refuses (a plain error result) is tolerated. A full run with every provider keyed costs about 107 model calls, under US$0.25. Spend is capped withmax_runs_per_turn=3.Run on 2026-11-17: 18 passed (Gemini 2.5 Flash, Claude Haiku 4.5, OpenRouter Qwen 3.7 Plus, each on both backends); OpenAI skipped (no key). Offline the module is deselected (24 cases).