libsy: RoutingOutcome API proposal

[grahamking]

Plan: Algorithm::route returns RoutingOutcome

Goal

An algorithm can finish in one of three ways:

1. It chose an ordered list of candidate models and cares about nothing after that
   (passthrough, stage, random, and llm_class on their normal answer path).
2. It chose candidates and rewrote the request (stage with SystemPromptProcessor).
3. It made a call for routing, then discovered that the same response can answer the turn
   (llm_class escalation when the efficient answer is accepted).

Today Algorithm::route returns Result<Response>, which makes normal answer calls travel from
the host into the algorithm task and back. Introduce RoutingOutcome so an algorithm returns its
selected candidates and rewritten request directly. libsy-llm-client can then make the terminal
answer call itself and pass the stream straight to the caller's SSE connection.

The resulting boundary is:

Algorithm task
  Driver::call_model(...)          routing-only classifier/judge calls, when needed
  return RoutingOutcome            selected candidates + rewritten request + optional response
             │
             ▼
libsy-llm-client
  response present                 return it; no second call
  response absent                  run backend retries and ordered candidate fallback

This project is pre-production. The public Rust, PyO3, and Python APIs may change directly in this
work; no compatibility shim or deprecation period is required.

Retry and fallback ownership

libsy does not retry model calls. FallThrough only classifies the request and constructs an
ordered candidate list: the selected target first, followed by the other eligible targets.

libsy-llm-client owns both recovery layers:

• run::call_first_available moves to the next candidate after an eligible candidate failure.
• TranslatingLlmClient exhausts a backend's configured HTTP retry budget before candidate
  fallback begins.

RoutingOutcome must therefore preserve the selected model and its ordered fallbacks. No failure
callback, retry state, or second routing pass belongs in libsy.

Contract simplification

Make the terminal outcome the only representation of a routing selection:

• Remove Driver::decide.
• Remove Step::Decision; run_stream yields routing-only CallModel steps followed by one
  Step::Done(RoutingOutcome).
• Remove Decision from switchyard-protocol, libsy exports, PyO3, Python stubs, and tests.
• Remove the decision trace. RoutingOutcome.selected_model_id is the one algorithm selection.
• Remove CallModel::into_parts. Its routing-outcome escape hatch is replaced by the terminal
  RoutingOutcome itself.
• Remove DriverError::Abandoned and the abandoned-run observability paths, which exist only to
  support into_parts.
• Remove is_answer_call everywhere. A Driver::call_model call is always routing work. If an
  algorithm knows that a response answers the turn, it returns that response in
  RoutingOutcome; if it does not have the response, the client makes the terminal call from the
  outcome.

Processor callbacks that need the chosen model receive the ModelId directly. Do not replace
Decision with another one-field wrapper.

AffinityRouter finding

Keep AffinityRouter. It implements libsy routing policy, not libsy-llm-client retry or fallback
behavior:

• As a Processor, it stores the model selected for a stable session/task/subagent identity.
• As an early Classifier, it replays that model on later turns, often avoiding another judge
  call entirely.
• It stores the algorithm-selected model before any client attempt. If the client temporarily
  falls back to a later candidate, affinity still prefers the original selected model on the next
  turn; it does not pin the transient fallback that happened to serve one request.
• session_affinity, message_hash_fallback, subagent affinity, and latch-only behavior are all
  configured and tested as algorithm behavior. libsy-llm-client does not read the assignment
  map or select from it.

Removing the Decision type does not remove the signal affinity needs. FallThrough already
knows the selected ModelId and replays Event::Decision after classification. Keep the event
name and change only its payload to Event::Decision { request, selected_model_id }; delete the
external step publication and its wrapper type.

Success criteria for affinity:

• A first selection is still retained and reused for the same identity.
• Reuse still bypasses a later judge/classifier in the cascade.
• A client-side fallback does not overwrite the retained algorithm selection.
• Existing session, message-hash, latch-only, and subagent tests remain behaviorally unchanged.

RoutingOutcome

Add to crates/libsy/src/core/algorithm.rs:

/// The terminal result of routing: ordered serving candidates, the rewritten request, and an
/// optional response when routing work already produced the answer.
pub struct RoutingOutcome {
    /// The model selected by the algorithm and tried first by the client.
    pub selected_model_id: ModelId,
    /// Additional models the client may try, in order, after an eligible selected-model failure.
    pub fallback_models: Vec<ModelId>,
    /// The request after all routing-time rewrites, stamped with `selected_model_id`.
    pub request: Request,
    /// A response already produced while routing, or `None` when the client must make the
    /// terminal call.
    pub response: Option<Response>,
}

impl RoutingOutcome {
    /// Leave the terminal call to the client and stamp `selected_model_id` onto the request.
    pub fn route_to(
        selected_model_id: ModelId,
        fallback_models: Vec<ModelId>,
        request: Request,
    ) -> Self { .. }

    /// Return a response that routing work already produced. Stamp `selected_model_id` and leave
    /// `fallback_models` empty.
    pub fn answered(
        selected_model_id: ModelId,
        request: Request,
        response: Response,
    ) -> Self { .. }
}

route_to is structurally non-empty: selected_model_id is required, while
fallback_models may be empty. answered leaves fallback_models empty because fallback is
irrelevant after a response exists. libsy-llm-client serves candidates by chaining
selected_model_id with fallback_models; it never indexes into a possibly empty vector.

Success criteria

• cargo test --workspace passes and cargo clippy --workspace --all-targets is clean.
• Algorithm::route returns Result<RoutingOutcome>.
• Driver exposes call_model(request, models) but no decide method.
• CallModel has no is_answer_call field and no into_parts method.
• The Decision type, decision trace, and their Rust/PyO3/Python exports are removed.
• Rust observations, PyO3 bindings, Python stubs, tests, and docs contain no is_answer_call
  member.
• Step has only CallModel and Done variants.
• drive serves only routing-time model calls and returns the terminal RoutingOutcome without
  turning it back into CallModel.
• libsy-llm-client::run consumes the outcome, preserves ordered candidate fallback and backend
  retries, and returns (ModelId, Response): the algorithm-selected model plus the final
  response. The response continues to carry the candidate that actually served it.
• FallThrough, Passthrough, StageRouter, Random, and LlmTaskClassifier no longer make
  their normal answer calls through Driver.
• AffinityRouter retains algorithm selections and continues to avoid repeated judge calls.
• No compatibility or deprecation scaffolding is added for the removed APIs.

Step-by-step

Step 1 — remove Decision

In crates/protocol/src/client.rs:

• Delete Decision, its constructor/accessors, and its export.
• Update the module docs, protocol tests, and imports.
• Remove the PyO3/Python Decision class rather than replacing it with another wrapper.

The algorithm selection now lives at RoutingOutcome.selected_model_id. General prose may still
use the word "decision," but there should be no Rust or Python Decision API.

Verify: cargo test -p switchyard-protocol.

Step 2 — add RoutingOutcome and change the algorithm contract

In crates/libsy/src/core/algorithm.rs:

• Add RoutingOutcome and its route_to / answered constructors.
• Change Algorithm::route, Driver::finish, and Step::Done to use RoutingOutcome.
• Make the run observability helpers generic over their successful payload.
• Update trait and stream documentation to distinguish routing-time model calls from the terminal
  outcome.

Add constructor tests covering selected-model stamping, fallback order, an empty fallback list,
and an already answered outcome.

Step 3 — remove selection-step publication and traces

Remove Driver::decide and Step::Decision from crates/libsy/src/core/algorithm.rs.

When a route completes successfully:

• Read outcome.selected_model_id directly.
• Record the existing routing-decision log and metric once from that model.
• Emit only Step::Done(outcome).

Keep the existing telemetry names: observability::record_decision, switchyard.decisions, and
the routing-decision log message. Change record_decision to accept the selected ModelId
directly instead of a Decision wrapper; this contract change does not rename the telemetry.

drive no longer collects or returns a trace. Its result becomes:

Result<RoutingOutcome>

It continues to serve concurrent Step::CallModel items and propagate their failures, but it does
not serve the terminal outcome. Remove tests for publishing, dropping, or collecting decision
steps; replace them with tests that a successful outcome records exactly one selection.

Step 4 — make CallModel routing-only

Remove is_answer_call from CallModel and from Driver::call_model:

pub async fn call_model(&self, request: Request, models: Vec<ModelId>) -> Result<Response>

Update all classifier, judge, escalation, hedge, fan-out, and test calls. No caller passes true
or false; every call represents model work required before the algorithm can return its outcome.

Simplify libsy call observability accordingly:

• libsy.llm_call spans and switchyard.llm_calls measure routing-time calls requested through
  Driver.
• Remove answer-only counters, gauges, branches, and comments from record_llm_call.
• Terminal request/error accounting moves to the client path that consumes RoutingOutcome.

Do not preserve the boolean under another name.

Step 5 — remove CallModel::into_parts and abandonment

Delete:

• CallModel::into_parts and its documentation;
• DriverError::Abandoned;
• observability::is_abandoned;
• the abandoned run/call outcome label and special logging branches;
• Rust and Python tests that distinguish abandonment from a dropped response promise.

Dropping a CallModel without calling respond remains DriverError::ResponseDropped. A host
that only wants the routing result waits for Step::Done(RoutingOutcome) instead of taking apart
an answer-generating CallModel; answer-generating CallModels no longer exist.

Step 6 — convert production algorithms

Passthrough no longer publishes a separate selection step or calls the target:

async fn route(
    self: Arc<Self>,
    _driver: Driver,
    request: Request,
) -> Result<RoutingOutcome> {
    tracing::info!(target = %self.target, "passthrough selected target");
    Ok(RoutingOutcome::route_to(
        self.target.clone(),
        Vec::new(),
        request,
    ))
}

FallThrough keeps classification and post-decision processing, but returns the result:

• Remove driver.decide(...).
• Replay Event::Decision { request, selected_model_id: &target } through processors so
  affinity and target-specific request rewrites still see the selection.
• If the deciding classifier returned Some(response), return
  Ok(RoutingOutcome::answered(target, request, response)).
• Otherwise compute let fallback_models = self.fallbacks(&target) before moving target, then
  return Ok(RoutingOutcome::route_to(target, fallback_models, request)).

Rename candidates to fallbacks and have it return only the other eligible models in their
existing order. The selected model plus those fallbacks still reaches libsy-llm-client in the
same order; FallThrough performs no retry or fallback execution.

StageRouter, Random, and LlmTaskClassifier change their return types and continue to
delegate to FallThrough::execute.

Noop returns Ok(RoutingOutcome::answered(model_id, request, response)) without publishing
a separate selection step.

Test-only algorithms that call models before choosing a response wrap their final response in
RoutingOutcome::answered. Retain request/model copies only where needed to construct the
outcome.

Step 7 — retain and adapt AffinityRouter

Do not remove crates/libsy/src/algorithms/util/affinity.rs, its export, configuration fields, or
documentation.

Adapt it to the direct model payload:

• Keep Event::Decision in the processor API, docs, prompt processors, affinity, and tests.
• Replace its decision: &Decision field with selected_model_id: &ModelId.
• Affinity stores that model directly; test helpers no longer construct a wrapper.
• FallThrough continues replaying the event after it selects a target and before returning the
  outcome.

Add an integration assertion that when the selected model fails and libsy-llm-client serves a
fallback, the affinity assignment remains the algorithm's selected model. On a later healthy turn,
the original selection is still tried first. This locks down the separation between algorithm
affinity and client fallback.

Step 8 — make libsy-llm-client consume the terminal outcome

Refactor crates/libsy-llm-client/src/run.rs around the new drive result:

1. Use the existing serve callback only for routing-time CallModels.
2. Receive outcome from drive.
3. If outcome.response is present, return it without another model call.
4. Otherwise pass outcome.request, outcome.selected_model_id, and
   outcome.fallback_models through the existing call_first_available path, trying the selected
   model first and then each fallback in order.
5. Return (selected_model_id, response). The server uses response.served_model() when a later
   candidate actually served the request and falls back to selected_model_id for synthetic or
   already answered responses.

Do not duplicate retry policy. The existing retryable error classes, backend retry budget,
Retry-After handling, and candidate-order loop remain the single implementation.

Use type-level observation variants instead of a boolean:

pub enum RunObservation {
    LlmCall(LlmCallObservation),
    AnswerCall(LlmCallObservation),
    RoutingOverhead(Duration),
}

• Calls produced by Driver are collected until the outcome reveals their final role. Remaining
  classifier and judge calls emit the existing LlmCall variant.
• A terminal call made from an unresolved RoutingOutcome emits AnswerCall.
• When an answered outcome reuses a routing response, that physical call emits AnswerCall
  instead of LlmCall; it is not observed twice and no additional call is made.

Remove LlmCallObservation::is_answer_call. Update the server stats observer to match the enum
variant: LlmCall feeds classifier/judge statistics, while AnswerCall feeds backend
statistics. The server's final response observation remains responsible for terminal usage.

Move switchyard.requests, switchyard.errors, switchyard.model_call_latency_ms, and the
compatibility total request/error gauges from libsy's boolean branch to terminal run accounting in
libsy-llm-client:

• Once run receives an outcome, count exactly one routed request under
  outcome.selected_model_id.
• Record success or error after resolving the optional terminal call. An answered outcome is a
  successful routed request even though the client makes no new call.
• Record switchyard.model_call_latency_ms only when the client actually makes the terminal call;
  do not invent a call duration for a response produced during routing.
• A routing failure before any outcome exists remains a failed algorithm run, not a routed request
  attributed to a guessed model.

Update metric initialization ownership or exports accordingly; do not leave a public libsy helper
whose only consumer is libsy-llm-client.

For an answered outcome, all Driver calls remain routing work because that was their role when
issued; the server still observes the returned response exactly once as the final answer. Add a
focused escalation test to prevent response usage from being appended twice to the same routing
log totals.

Step 9 — update PyO3 and Python APIs

The Python stream should mirror the new Rust stream rather than emulate the removed API:

• Remove Step.Decision.
• Remove the Python Decision class, ModelCall.decision, ModelCall.into_parts, and all
  is_answer_call properties.
• Keep ModelCall.request, ModelCall.models, respond, and fail for routing-time calls.
• Add a Python RoutingOutcome value exposed by Step.Done, with selected_model_id,
  fallback_models, rewritten request, and optional aggregate response.
• Update switchyard_rust/libsy.py and binding tests. A Python host serves Step.CallModel values
  needed for routing; on Step.Done, it either uses outcome.response or calls the ordered
  sequence of outcome.selected_model_id followed by outcome.fallback_models itself.
• Update test/example helpers that currently return list[Decision] to return the selected model
  directly alongside the response.

Because the API is pre-production, remove the old shapes directly. Do not retain aliases or
deprecated properties.

Step 10 — exports, docs, and remaining call sites

• Re-export RoutingOutcome from crates/libsy/src/lib.rs.
• Remove the Decision export and stale CallModel members from Rust and Python docs.
• Update crates/libsy/README.md and docs/getting_started.md so Step::CallModel is explicitly
  routing-only and Step::Done carries the terminal routing outcome.
• Update crates/libsy-llm-client/README.md to show that run owns the terminal answer call,
  backend retries, and ordered candidate fallback, and that it returns
  (selected_model_id, response) rather than a decision trace.
• Keep affinity documentation, but clarify that it pins the algorithm selection rather than the
  candidate that happened to serve after client fallback.
• Update the server call site to consume (selected_model_id, response). Continue preferring
  response.served_model() for routing headers and usage, falling back to selected_model_id for
  synthetic or already answered responses.
• Update server observation tests for LlmCall / AnswerCall variants.
• Remove every stale Decision type/API, decision trace, is_answer_call, into_parts,
  Step::Decision, and Driver::decide reference from Rust, PyO3, Python stubs, tests, and
  documentation.

Tests to add or adapt

Test	Asserts
RoutingOutcome::route_to	the required selection is stamped and fallback order survives, including an empty fallback list
RoutingOutcome::answered	an existing response and its selected model are retained without a second call
stream contract	zero or more routing-only CallModel steps are followed by exactly one Done; there is no Decision step
decision telemetry	a successful outcome records one decision directly from selected_model_id, using the existing names
dropped call	dropping an unanswered routing call reports ResponseDropped; there is no abandoned outcome
Passthrough / Random	no CallModel is emitted before their route-only terminal outcome
request rewrite	post-decision processor changes survive in RoutingOutcome.request
escalation accepted	routing calls may produce an answered outcome and no terminal client call is made
escalation rejected	the capable candidate is returned and the client makes the terminal call
client fallback	backend retries are exhausted before the next outcome candidate is tried
affinity vs fallback	affinity retains selected_model_id, not a transient client fallback
affinity reuse	later matching turns bypass the judge/classifier as before
observations	routing and answer calls use distinct enum variants with no purpose boolean
accounting	one external request increments request/error totals once; accepted escalation does not double-count final usage
Python stream	routing calls are fulfilled, then Done(RoutingOutcome) supplies either a response or a selected model plus fallbacks

Verification

cargo test -p switchyard-protocol
cargo test -p switchyard-libsy
cargo test -p switchyard-llm-client
cargo test -p switchyard-py
cargo test -p switchyard-server
cargo test --workspace
cargo clippy --workspace --all-targets
cargo doc -p switchyard-libsy
cargo doc -p switchyard-llm-client
uv run pytest tests/ -v
uv run ruff check .
uv run mypy switchyard

Before considering the work complete, verify these searches return no stale API references:

rg 'Driver::decide|\.decide\(' crates switchyard switchyard_rust tests docs
rg 'is_answer_call|into_parts|Step::Decision|Step\.Decision' \
  crates switchyard switchyard_rust tests docs
rg 'struct Decision|Decision::|Vec<Decision>|Step[.:]Decision' \
  crates switchyard switchyard_rust tests docs

Review the final diff specifically for configuration and docs: affinity remains supported, while
the removed stream/call APIs have no compatibility scaffolding.

Commit

One commit, signed off, single-line Conventional Commits message:

refactor(libsy): return routing outcomes directly

Show the diff and get approval before committing.

Out of scope

1. Changing retry or fallback policy. Error classes, retry budgets, backoff, and candidate
   fallback stay in libsy-llm-client.
2. Removing affinity. It is confirmed algorithm policy and remains supported through the
   direct ModelId payload on Event::Decision.
3. Removing buffered escalation. The efficient response must still be buffered for the judge;
   when accepted, it is returned as an answered outcome.
4. Adding API compatibility layers. The old stream variants, flags, and escape hatch are
   removed directly because the project is not yet in production.