Skip to content

feat(translation): prepare requests for routed targets - #455

Draft
afourniernv wants to merge 1 commit into
NVIDIA-NeMo:mainfrom
afourniernv:afournier/switch-1253-target-prompt-translation
Draft

feat(translation): prepare requests for routed targets#455
afourniernv wants to merge 1 commit into
NVIDIA-NeMo:mainfrom
afourniernv:afournier/switch-1253-target-prompt-translation

Conversation

@afourniernv

@afourniernv afourniernv commented Aug 17, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds one translation-layer operation that selects a routed target and optionally prepends that target's system prompt without dropping provider-specific request fields.

This is 1 of 3 for SWITCH-1253. It establishes the wire-format boundary only; it does not add target configuration or change router behavior by itself.

Before

Routing hosts changed the normalized request when selecting a candidate:

request.llm_request.model = Some(target.to_string());

The normalized request can also retain an exact provider body so same-format forwarding preserves fields Switchyard does not model. Changing only the normalized request could leave that exact body with the route alias and no target prompt. Dropping the exact body avoided stale replay, but also dropped provider-specific fields.

After

Target selection and prompt insertion are one provider-aware operation:

prepare_request_for_target(&mut request.llm_request, target, prompt);

The helper updates the normalized request and exact preserved OpenAI Chat, OpenAI Responses, and Anthropic Messages bodies together. Existing system content and provider-only fields remain intact, while a same-format encode cannot restore the route alias or omit the target prompt.

For a custom exact format, an unprompted request remains untouched. If a configured prompt cannot be patched safely, the stale exact replay is discarded rather than silently forwarding a request without the prompt.

Why this layer owns the operation

Libsy decides which target prompt applies; the translation layer knows how each provider represents model and system content. Keeping provider mutation here prevents libsy, the native server, and custom hosts from implementing separate OpenAI and Anthropic patching rules.

Scope and compatibility

  • Adds prepare_request_for_target(...) as an exported translation helper.
  • Preserves OpenAI Chat message fields, OpenAI Responses metadata, Anthropic structured system blocks, and embedded fallback bodies.
  • Refreshes embedded preservation metadata without recursively nesting stale snapshots.
  • Does not change a Python API, server API, router, or TOML schema.

Validation

  • cargo test -p switchyard-translation
  • cargo clippy -p switchyard-translation --all-targets -- -D warnings
  • cargo fmt --all -- --check
  • Tests cover all three built-in request formats, provider-field preservation, embedded fallback bodies, malformed exact bodies, and custom-format behavior.
  • Full-stack workspace Clippy, non-PyO3 Rust tests, rebuilt-extension Python tests, ruff, mypy, strict docs, and live NVIDIA scenarios passed on the complete stack.

Suggested review order

  1. crates/switchyard-translation/src/util.rs — helper contract and exact-body patching rules
  2. crates/switchyard-translation/tests/request_translation.rs — provider preservation and failure boundaries
  3. crates/switchyard-translation/src/lib.rs — public export

Stack

PR Layer Responsibility
#455 (this PR) Translation Mutate normalized and exact provider requests safely
#463 libsy Prepare the request for each routed candidate
#464 Native server Expose targets.*.system_prompt, compatibility, docs, and integration tests

The later two PRs are drafts and currently show cumulative diffs against main. Their descriptions identify the commit and line count unique to each layer; those diffs will shrink as the parent PRs merge.

Signed-off-by: Alex Fournier <afournier@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant