Skip to content

feat(server): configure system prompts by target - #464

Draft
afourniernv wants to merge 3 commits into
NVIDIA-NeMo:mainfrom
afourniernv:afournier/switch-1253-server-target-prompts
Draft

feat(server): configure system prompts by target#464
afourniernv wants to merge 3 commits into
NVIDIA-NeMo:mainfrom
afourniernv:afournier/switch-1253-server-target-prompts

Conversation

@afourniernv

@afourniernv afourniernv commented Aug 17, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds system_prompt to native-server targets so the prompt follows the model that actually serves an answer.

This is 3 of 3 for SWITCH-1253. It builds on the translation operation in #455 and the libsy candidate contract in #463.

Use case

Different models can need different standing instructions or model-specific prompt tuning. The caller addresses a Switchyard route and may not know which target will answer, especially after fallback. Switchyard therefore selects the prompt with the answer candidate rather than requiring the caller to predict the route.

Before

Only Stage Router could configure different prompts, and those prompts were tied to its capable and efficient roles:

[routes.stage]
capable_system_prompt = "diagnose before editing"
efficient_system_prompt = "follow the existing plan"

Other routers could not use target-specific prompts. Prompt selection also happened before a client fallback knew which target would ultimately answer.

After

Prompts live on the targets they describe:

[targets.weak]
id = "model/weak"
llm_client = "provider"
system_prompt = "follow the existing plan"

[targets.strong]
id = "model/strong"
llm_client = "provider"
system_prompt = "diagnose before editing"

Any router can select those targets. Switchyard applies the prompt when it prepares each answer candidate, so a fallback receives its own prompt rather than the first target's.

The legacy Stage fields remain supported. If both forms configure the same target, the target-level system_prompt wins.

Call boundary

Call purpose Target prompt
Passthrough, random, fallback, Stage, or classifier final answer Applied
Escalation's provisional efficient answer Applied
Advisor executor answer Applied
Classifier, judge, or Advisor reviewer call Not applied
Noop No model call
Anthropic count-tokens Applied explicitly because this endpoint bypasses the algorithm

This uses libsy's answer-call distinction rather than a router allowlist. Routers that return a terminal RoutingOutcome inherit the behavior; routers that produce an answer while routing use Driver::call_answer_model(...), as escalation and Advisor do here.

Configuration checks

  • Only answer targets contribute to the effective prompt map; judge-only targets are excluded.
  • Configuration fails early if two answer aliases resolve to one model ID with different effective prompts. After alias resolution, the prompt identity would otherwise be ambiguous.
  • Existing client-provided system content is retained after the target prompt.
  • No HTTP endpoint is added or changed.

Validation

  • Target prompts covered through random routing, Stage, classifier, escalation, Advisor, and fallback paths.
  • Legacy Stage behavior and target-level precedence covered.
  • Classifier/judge and Advisor reviewer exclusion covered.
  • Anthropic count-tokens uses the same effective target prompt and rejects unsupported target formats.
  • Conflicting answer aliases are rejected during configuration.
  • cargo fmt --all --check, workspace Clippy, and the full non-PyO3 Rust workspace passed.
  • PyO3 rebuilt successfully; 143 Python tests, ruff, mypy, and strict docs passed.
  • The final squashed stack passed 19 live NVIDIA scenarios: Chat, Responses, and Anthropic buffered/streaming requests; same-target retry; pre-commit fallback; post-commit no-fallback; Stage precedence; classifier/judge isolation; escalation; and direct Python libsy hosting.

Suggested review order

  1. crates/switchyard-server/src/config.rs — target field, effective-prompt resolution, and conflict validation
  2. crates/switchyard-server/src/lib.rs — explicit count-tokens preparation
  3. crates/switchyard-server/tests/server.rs — routing, fallback, judge, and count-tokens behavior
  4. crates/switchyard-server/CONFIGURATION.md and docs/ — user-facing schema and Stage compatibility

Stack

PR Layer Responsibility
#455 Translation Mutate normalized and exact provider requests safely
#463 libsy Prepare the request for each routed candidate
#464 (this PR) Native server Expose targets.*.system_prompt, compatibility, docs, and integration tests

This PR's unique change is one signed commit, 87f9f541 (6 files, +190/-52). GitHub currently compares the draft with main, so it also displays PRs 1 and 2 below that commit. After the parent PRs merge, this branch will be rebased onto the updated main to leave only the server layer in the displayed diff.

Signed-off-by: Alex Fournier <afournier@nvidia.com>
Signed-off-by: Alex Fournier <afournier@nvidia.com>
Signed-off-by: Alex Fournier <afournier@nvidia.com>
@afourniernv
afourniernv force-pushed the afournier/switch-1253-server-target-prompts branch from 6731a11 to 87f9f54 Compare August 18, 2026 18:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant