Skip to content

feat(libsy): prepare requests for routed candidates - #463

Draft
afourniernv wants to merge 2 commits into
NVIDIA-NeMo:mainfrom
afourniernv:afournier/switch-1253-libsy-target-prompts
Draft

feat(libsy): prepare requests for routed candidates#463
afourniernv wants to merge 2 commits into
NVIDIA-NeMo:mainfrom
afourniernv:afournier/switch-1253-libsy-target-prompts

Conversation

@afourniernv

@afourniernv afourniernv commented Aug 17, 2026

Copy link
Copy Markdown
Contributor

Summary

Moves target-prompt policy into libsy, where the routing decision and ordered fallback candidates are known.

This is 2 of 3 for SWITCH-1253. It builds on #455, which provides the provider-safe request operation.

Before

A direct libsy host received a RoutingOutcome and prepared fallbacks itself, commonly by copying the published request and replacing the model:

candidate_request = {**outcome.request, "model": target}

That works while every candidate receives the same request. It cannot safely select a different prompt per candidate, and preparing the first candidate too early would make a fallback inherit the first target's prompt.

Routing-time calls had the same split: libsy could ask a host to call a model, but the host had no candidate-aware request operation.

After

The host asks libsy to prepare the candidate it is about to call:

candidate_request = outcome.request_for(target)

For routing-time calls, the same contract is available on CallModel:

candidate_request = call.request_for(target)

The built-in Rust client uses the corresponding Rust methods. For fallback, libsy starts from the request before a target prompt was applied, stamps the new model, and applies only the new target's prompt. Custom Rust, Python, Relay, and other hosts no longer need to reproduce this policy.

What changes

  • Adds router-independent with_target_prompts(...) policy in libsy.
  • Adds RoutingOutcome::request_for(...) and CallModel::request_for(...), with Python bindings for both.
  • Applies target prompts only to answer calls. Classifier and judge calls remain unchanged.
  • Moves Stage Router's existing tier prompts onto the same Driver path.
  • Migrates switchyard-llm-client and the direct Python libsy host example/test path.

Ownership and async behavior

  • Prompt maps are immutable and shared with Arc; requests do not copy the map per call.
  • A pristine normalized request is retained only when the selected target has a prompt and fallbacks exist.
  • Fallback requests are materialized one at a time when request_for(...) is called.
  • There are no new locks in libsy core and no borrow is held across provider I/O.
  • The Python wrapper uses a short synchronous mutex only to clone/prepare the Rust value; the lock is released before Python conversion or any await.

API compatibility

The Python changes and the CallModel methods are additive.

RoutingOutcome itself landed on main after the last tagged release. This PR adds private preparation state and marks the struct #[non_exhaustive], so code tracking unreleased main that constructs it with a struct literal or destructures every field will need to use its constructors/public fields instead. RoutingOutcome::route_to(...), RoutingOutcome::answered(...), field reads, and existing tagged APIs remain available.

Not in this PR

  • Native TOML targets.*.system_prompt configuration
  • Count-tokens handling
  • Server documentation and integration tests

Those are isolated in #464.

Validation

  • First and fallback candidates receive only their own prompt; unconfigured candidates remain unchanged.
  • Classifier and judge calls do not receive answer-target prompts.
  • Stage Router's existing tier prompts still apply to answer calls.
  • Built-in client retry, fallback, authentication-stop, and streaming-commit boundaries remain covered.
  • Direct Python Algorithm.run_stream hosting covers prompted first and fallback candidates.
  • Workspace Clippy, full non-PyO3 Rust tests, rebuilt-extension Python tests, ruff, mypy, and strict docs passed on the complete stack.
  • Direct Python libsy first-candidate and fallback calls both reached NVIDIA in the 19-scenario live matrix.

Suggested review order

  1. crates/libsy/src/core/algorithm.rsRoutingOutcome, Driver, and candidate request contracts
  2. crates/libsy/src/core/target_prompts.rs — immutable prompt lookup policy
  3. crates/libsy-llm-client/src/run.rs — built-in retry/fallback consumer
  4. crates/libsy/src/algorithms/stage.rs — legacy Stage prompt compatibility
  5. crates/switchyard-py/src/libsy_bindings.rs and tests/test_libsy_minimal_bindings.py — direct Python host API

Stack

PR Layer Responsibility
#455 Translation Mutate normalized and exact provider requests safely
#463 (this PR) libsy Prepare the request for each routed candidate
#464 Native server Expose targets.*.system_prompt, compatibility, docs, and integration tests

This PR's unique change is one signed commit, fb27c6d5 (18 files, +518/-151). GitHub currently compares the draft with main, so it also displays PR1 below that commit. After #455 merges, this branch will be rebased onto the updated main to leave only the libsy layer in the displayed diff.

Signed-off-by: Alex Fournier <afournier@nvidia.com>
Signed-off-by: Alex Fournier <afournier@nvidia.com>
@afourniernv
afourniernv force-pushed the afournier/switch-1253-libsy-target-prompts branch from 16dfb7c to fb27c6d Compare August 18, 2026 18:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant