Skip to content

Promote round-robin model rotation to a RoundRobinRequests capability - #849

Open
mpfaffenberger wants to merge 4 commits into
mainfrom
feature/round-robin-requests-capability
Open

Promote round-robin model rotation to a RoundRobinRequests capability#849
mpfaffenberger wants to merge 4 commits into
mainfrom
feature/round-robin-requests-capability

Conversation

@mpfaffenberger

@mpfaffenberger mpfaffenberger commented Aug 22, 2026

Copy link
Copy Markdown
Owner

What

Promotes round-robin model rotation — previously implemented entirely inside the RoundRobinModel(Model) subclass's request/request_stream methods — to a first-class pydantic-ai capability, RoundRobinRequests, on the wrap_model_request seam. Nineteenth in the capability series (#828#836, #838#842, #844, #845, #847, #848).

Second, disjoint claim of wrap_model_request (#830 PluginMessageTransform mutates only the context's messages; this router swaps only the context's model — nesting order is inert, documented at both wiring sites).

Why this seam

Per-request model rotation is a request-routing decision wearing a Model costume. wrap_model_request hands every request (streamed and non-streamed — the seam spans both via the cooperative hand-off) a ModelRequestContext whose model field the terminal handler honours: model_request(...) / model_request_stream(...) issue against req_ctx.model, and upstream's own durable-execution capabilities swap request_context.model at exactly this seam (documented in _agent_graph.py).

Custody split (explicit-when-ours, fallback-for-guests — #838/#841/#842 pattern)

  • Capability-owned: identity gate request_context.model is self.model. Owned requests advance the shared rotation, mirror the eager leaf-side prepare_request merge byte-for-byte, route straight to the leaf, and run the same span-attribute fix-up. RoundRobinModel.request never executes (pinned by spy).
  • Guest: explicit run(model=...)/override(model=...) models pass through untouched (parity: the eager kwarg was replaced wholesale too); an instrumentation-wrapped round-robin model (InstrumentedModel, WrapperModel) fails the gate and the Model subclass — kept intact — rotates eagerly exactly as before.
  • One rotation state (RoundRobinModel.next_model, now public) is shared by both paths, so every request advances the rotation exactly once no matter which path serves it (pinned by the shared-state test).

Bounded divergences (documented in the module docstring)

  1. _ensure_model_supports_streaming now checks the routed leaf on owned streamed requests instead of the wrapper — strictly more precise; vacuous for real provider leaves.
  2. Continuation segments stay pinned to the leaf that opened the chain (reviewer pass-1 blocking find, verified empirically). model_request/model_request_stream resolve suspended → complete continuations (Anthropic pause_turn, OpenAI background polls) by re-invoking req_ctx.model inside ONE wrapped request; eagerly the terminal model was the round-robin wrapper, so every segment rotated — stitching one merged response from two different models and re-polling a different provider for a suspended job id. Owned requests advance the rotation once and serve the whole chain from the selected leaf — a deliberate divergence, strictly saner than eager (which survives unchanged on the guest path). Pinned both ways by continuation tests.
  3. The streamed span-attribute fix-up moves from stream-open to handler-return, which parks until the stream drains — a mid-stream cancel/teardown records nothing where eager custody had already recorded at open (reviewer pass-1 blocking find: originally understated as a pure timing shift; pass-2 blocking find killed the follow-up "instrumentation routes to the eager path anyway" materiality claim — backwards on 2.31.0, where instrumentation is capability-based, explicit InstrumentedModels are unwrapped before the run, and the shipped logfire.instrument_pydantic_ai() therefore reaches the owned path). Honest scope: only the leaf-attribute refinement on a torn-down stream's chat span is lost; the span itself, its round-robin gen_ai.request.model, and completed streams are unaffected. Pinned by the teardown test pair plus a new test proving the identity gate HOLDS under real InstrumentationSettings.
  4. ModelRequestContext.model_id is invalidated by the swap for durable-execution consumers — upstream-documented semantics of any model-swapping hook, including upstream's own.

Wiring

  • _builder.py: one instance hoisted across probe+final passes (probe never runs; rotation state lives on the model regardless), conditional splice *build_round_robin_requests(model) — the build_tool_output_limits shape, so non-round-robin agents carry no inert capability.
  • subagent_invocation.py: same conditional splice — a round-robin-pinned sub-agent is capability-owned too (the eager path covered sub-agents, so scope parity requires both sites).
  • get_serialization_name() → None (live Model with provider HTTP clients — Promote model delivery to a ResolvedModel capability #833 precedent).

Tests

25 contract tests across tests/agents/test_round_robin_capability.py (seam contract) and tests/agents/test_round_robin_capability_runs.py (end-to-end + wiring), sharing round_robin_capability_harness.py — split to respect the 600-line cap (this feature had zero prior test coverage):

  • Seam contract: alternating leaf routing, copy isolation (Promote plugin message transform to a first-class capability #830 shallow-copy shape), guest pass-through identity, prepare_request merge parity, span-attr fix-up custody, error propagation with rotation-already-advanced parity
  • End-to-end real Agent runs: non-streamed + streamed capability ownership (eager methods spied silent), rotate_every cadence, explicit-model override, wrapped-model eager fallback, shared rotation state across paths, coexistence with PluginMessageTransform
  • Wiring: builder attaches/skips (both polarities, capability found via the public apply visitor), sub-agent source pin, spec-constructibility opt-out

Post-review additions: continuation pinning (both custodies), streamed-teardown span custody (both custodies), non-idempotent customize_request_parameters application-count parity, and a real end-to-end sub-agent invocation drive (capability serves the streamed request; eager Model methods spied silent).

Full suite: 7622 passed, 0 failed (28 skipped, 1 xpassed). ruff check + ruff format clean.

Review

  • Pass 1 (code-puppy-clone-1): REQUEST_CHANGES — 2 blocking (continuation-segment rotation parity; streamed span-custody loss on teardown), 2 non-blocking (non-idempotent prepare parity probe; sub-agent wiring as source-pin only), 1 nit (typing.List). Addressed in d481a0e: divergences documented honestly + pinned by tests rather than papered over.
  • Pass 2: REQUEST_CHANGES — 1 blocking (the pass-1 fix's "instrumentation forces eager custody" materiality bound was backwards under 2.31.0 capability-based instrumentation), 1 non-blocking (custody wording ambiguity). Addressed in 7045e17 with corrected docs + an instrumentation-gate test.

…bility

Route each model request to the next RoundRobinModel leaf on pydantic-ai
2.31.0's wrap_model_request seam instead of inside the Model subclass's
request/request_stream methods. The Model class stays intact as the guest
fallback (explicit run/override models, instrumentation-wrapped models),
gated by object identity; both paths share one rotation state so every
request advances the rotation exactly once.

- new code_puppy/agents/_round_robin.py: RoundRobinRequests capability +
  build_round_robin_requests conditional splice
- round_robin_model.py: next_model/record_span_attributes made public so
  the capability and the eager path share rotation + span fix-up
- wired into both capabilities=[...] sites (builder + sub-agent invocation)
- 18 contract tests in tests/agents/test_round_robin_capability.py
…pan custody)

Reviewer findings, both verified empirically:

- BLOCKING 1: continuation segments (Anthropic pause_turn / OpenAI
  background polls) re-invoke req_ctx.model inside ONE wrapped request;
  the eager terminal was the round-robin wrapper so every segment
  rotated - stitching one merged response from two different models and
  re-polling a different provider for a suspended job id. The owned path
  pins the whole chain to the opening leaf: documented as a deliberate
  bounded divergence (strictly saner; eager behaviour survives on the
  guest path) and pinned both ways by new continuation tests.
- BLOCKING 2: the streamed span fix-up documentation understated the
  divergence - a mid-stream cancel/teardown records nothing where eager
  custody had already recorded at stream-open. Now documented honestly
  with its materiality bound (the fix-up only matches instrumented
  spans, and instrumentation re-wraps the model, failing the identity
  gate into the eager path anyway) and pinned by a teardown test pair.
- NON-BLOCKING 1: prepare_request parity now proven against a leaf with
  an observable (non-idempotent) customize_request_parameters counter -
  both paths deliver with the identical application count.
- NON-BLOCKING 2: sub-agent wiring now covered end-to-end through the
  real _invoke_agent_impl (capability serves the streamed request, eager
  Model methods spied silent), alongside the source pin.
- NIT: typing.List -> list.
- BLOCKING: the streamed-span materiality bound claimed instrumentation
  re-wraps the model and routes to the eager path. Backwards on 2.31.0:
  instrumentation is capability-based and explicit InstrumentedModels are
  unwrapped before the run, so instrumented requests carry the bare
  round-robin model and take the OWNED path - reachable under the shipped
  logfire.instrument_pydantic_ai() (observability.py). Docstring rewritten
  honestly (scope of the loss: only the leaf-attribute refinement on a
  torn-down stream's chat span) and pinned by a new test proving the
  identity gate HOLDS under real InstrumentationSettings.
- NON-BLOCKING: 'every request advances exactly once' custody wording
  disambiguated (wrapped request vs continuation segment) and the guest
  list corrected (arbitrary WrapperModel, explicitly NOT instrumentation).
- Split the test file (750 lines > 600 cap) into seam-contract +
  runs/wiring files sharing round_robin_capability_harness.py.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant