Promote round-robin model rotation to a RoundRobinRequests capability - #849
Open
mpfaffenberger wants to merge 4 commits into
Open
Promote round-robin model rotation to a RoundRobinRequests capability#849mpfaffenberger wants to merge 4 commits into
mpfaffenberger wants to merge 4 commits into
Conversation
…bility Route each model request to the next RoundRobinModel leaf on pydantic-ai 2.31.0's wrap_model_request seam instead of inside the Model subclass's request/request_stream methods. The Model class stays intact as the guest fallback (explicit run/override models, instrumentation-wrapped models), gated by object identity; both paths share one rotation state so every request advances the rotation exactly once. - new code_puppy/agents/_round_robin.py: RoundRobinRequests capability + build_round_robin_requests conditional splice - round_robin_model.py: next_model/record_span_attributes made public so the capability and the eager path share rotation + span fix-up - wired into both capabilities=[...] sites (builder + sub-agent invocation) - 18 contract tests in tests/agents/test_round_robin_capability.py
…pan custody) Reviewer findings, both verified empirically: - BLOCKING 1: continuation segments (Anthropic pause_turn / OpenAI background polls) re-invoke req_ctx.model inside ONE wrapped request; the eager terminal was the round-robin wrapper so every segment rotated - stitching one merged response from two different models and re-polling a different provider for a suspended job id. The owned path pins the whole chain to the opening leaf: documented as a deliberate bounded divergence (strictly saner; eager behaviour survives on the guest path) and pinned both ways by new continuation tests. - BLOCKING 2: the streamed span fix-up documentation understated the divergence - a mid-stream cancel/teardown records nothing where eager custody had already recorded at stream-open. Now documented honestly with its materiality bound (the fix-up only matches instrumented spans, and instrumentation re-wraps the model, failing the identity gate into the eager path anyway) and pinned by a teardown test pair. - NON-BLOCKING 1: prepare_request parity now proven against a leaf with an observable (non-idempotent) customize_request_parameters counter - both paths deliver with the identical application count. - NON-BLOCKING 2: sub-agent wiring now covered end-to-end through the real _invoke_agent_impl (capability serves the streamed request, eager Model methods spied silent), alongside the source pin. - NIT: typing.List -> list.
- BLOCKING: the streamed-span materiality bound claimed instrumentation re-wraps the model and routes to the eager path. Backwards on 2.31.0: instrumentation is capability-based and explicit InstrumentedModels are unwrapped before the run, so instrumented requests carry the bare round-robin model and take the OWNED path - reachable under the shipped logfire.instrument_pydantic_ai() (observability.py). Docstring rewritten honestly (scope of the loss: only the leaf-attribute refinement on a torn-down stream's chat span) and pinned by a new test proving the identity gate HOLDS under real InstrumentationSettings. - NON-BLOCKING: 'every request advances exactly once' custody wording disambiguated (wrapped request vs continuation segment) and the guest list corrected (arbitrary WrapperModel, explicitly NOT instrumentation). - Split the test file (750 lines > 600 cap) into seam-contract + runs/wiring files sharing round_robin_capability_harness.py.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Promotes round-robin model rotation — previously implemented entirely inside the
RoundRobinModel(Model)subclass'srequest/request_streammethods — to a first-class pydantic-ai capability,RoundRobinRequests, on thewrap_model_requestseam. Nineteenth in the capability series (#828–#836, #838–#842, #844, #845, #847, #848).Second, disjoint claim of
wrap_model_request(#830PluginMessageTransformmutates only the context's messages; this router swaps only the context's model — nesting order is inert, documented at both wiring sites).Why this seam
Per-request model rotation is a request-routing decision wearing a
Modelcostume.wrap_model_requesthands every request (streamed and non-streamed — the seam spans both via the cooperative hand-off) aModelRequestContextwhosemodelfield the terminal handler honours:model_request(...)/model_request_stream(...)issue againstreq_ctx.model, and upstream's own durable-execution capabilities swaprequest_context.modelat exactly this seam (documented in_agent_graph.py).Custody split (explicit-when-ours, fallback-for-guests — #838/#841/#842 pattern)
request_context.model is self.model. Owned requests advance the shared rotation, mirror the eager leaf-sideprepare_requestmerge byte-for-byte, route straight to the leaf, and run the same span-attribute fix-up.RoundRobinModel.requestnever executes (pinned by spy).run(model=...)/override(model=...)models pass through untouched (parity: the eager kwarg was replaced wholesale too); an instrumentation-wrapped round-robin model (InstrumentedModel,WrapperModel) fails the gate and theModelsubclass — kept intact — rotates eagerly exactly as before.RoundRobinModel.next_model, now public) is shared by both paths, so every request advances the rotation exactly once no matter which path serves it (pinned by the shared-state test).Bounded divergences (documented in the module docstring)
_ensure_model_supports_streamingnow checks the routed leaf on owned streamed requests instead of the wrapper — strictly more precise; vacuous for real provider leaves.model_request/model_request_streamresolve suspended → complete continuations (Anthropicpause_turn, OpenAI background polls) by re-invokingreq_ctx.modelinside ONE wrapped request; eagerly the terminal model was the round-robin wrapper, so every segment rotated — stitching one merged response from two different models and re-polling a different provider for a suspended job id. Owned requests advance the rotation once and serve the whole chain from the selected leaf — a deliberate divergence, strictly saner than eager (which survives unchanged on the guest path). Pinned both ways by continuation tests.InstrumentedModels are unwrapped before the run, and the shippedlogfire.instrument_pydantic_ai()therefore reaches the owned path). Honest scope: only the leaf-attribute refinement on a torn-down stream's chat span is lost; the span itself, its round-robingen_ai.request.model, and completed streams are unaffected. Pinned by the teardown test pair plus a new test proving the identity gate HOLDS under realInstrumentationSettings.ModelRequestContext.model_idis invalidated by the swap for durable-execution consumers — upstream-documented semantics of any model-swapping hook, including upstream's own.Wiring
_builder.py: one instance hoisted across probe+final passes (probe never runs; rotation state lives on the model regardless), conditional splice*build_round_robin_requests(model)— thebuild_tool_output_limitsshape, so non-round-robin agents carry no inert capability.subagent_invocation.py: same conditional splice — a round-robin-pinned sub-agent is capability-owned too (the eager path covered sub-agents, so scope parity requires both sites).get_serialization_name() → None(liveModelwith provider HTTP clients — Promote model delivery to a ResolvedModel capability #833 precedent).Tests
25 contract tests across
tests/agents/test_round_robin_capability.py(seam contract) andtests/agents/test_round_robin_capability_runs.py(end-to-end + wiring), sharinground_robin_capability_harness.py— split to respect the 600-line cap (this feature had zero prior test coverage):prepare_requestmerge parity, span-attr fix-up custody, error propagation with rotation-already-advanced parityAgentruns: non-streamed + streamed capability ownership (eager methods spied silent),rotate_everycadence, explicit-model override, wrapped-model eager fallback, shared rotation state across paths, coexistence withPluginMessageTransformapplyvisitor), sub-agent source pin, spec-constructibility opt-outPost-review additions: continuation pinning (both custodies), streamed-teardown span custody (both custodies), non-idempotent
customize_request_parametersapplication-count parity, and a real end-to-end sub-agent invocation drive (capability serves the streamed request; eager Model methods spied silent).Full suite: 7622 passed, 0 failed (28 skipped, 1 xpassed).
ruff check+ruff formatclean.Review
typing.List). Addressed in d481a0e: divergences documented honestly + pinned by tests rather than papered over.