feat(operations): add release soak and routing benchmarks

[elyasmnvidian]

cargo run --release -p switchyard-soak -- \
  --model switchyard/general \
  --scenario tool-call-burst

error: package(s) `switchyard-soak` not found in workspace

Switchyard had no release soak runner and no repeatable report for comparing routing algorithms
under short, long-context, agentic, burst, and failure workloads. Short tests could pass while a
release still had memory growth, incomplete streams, routing drift, or failures that appeared only
under sustained traffic.

What changed

• The new switchyard-soak command keeps a fixed number of requests active for a set time. It
  checks /health, /metrics, invalid-request recovery, optional process memory and CPU, and the
  server request counters. It exits with status 1 when a release gate fails.
• crates/switchyard-soak/src/scenarios/ is the only scenario source. Each request shape has one
  Rust file, and the crate exports the same ordered sessions as AIPerf inputs-json data.
• The standard set covers short interactions, long context, long output, prefix reuse, mixed
  traffic, growing conversations, large tool catalogs, tool-call bursts, stage transitions, and
  deterministic classifier mixes. Resilience scenarios cover context fallback, bounded 429/500
  pressure, malformed classifier output, truncated streams, and client cancellation.
• Session IDs and growing payload order are preserved per worker, including the multi-turn tool and
  stage scenarios.
• scripts/benchmark_routing_algorithms.py runs algorithms back-to-back for each identical
  scenario and load point. It writes Markdown, CSV, and JSON with request throughput, TTFT, ITL,
  output-token throughput, confidence intervals, selected-target shares and errors, classifier
  calls and latency, routing overhead, and explicit resilience gates.
• The local test uses a request-aware Axum backend instead of VidaiMock. It returns valid easy and
  hard classifier verdicts, injects the named failure cases, and resets transient-failure counters
  before each algorithm receives the workload.
• The short baseline still covers Chat Completions, Anthropic Messages, and OpenAI Responses with
  streaming on and off. Stream validation rejects malformed events, in-band errors, missing
  terminal events, and data after completion.

Scenario catalog

Group	Scenarios
Core	short-interactive, long-context, decode-heavy, prefix-reuse, mixed-traffic
Agentic	growing-conversation, large-tool-catalog, tool-call-burst, stage-transitions, classifier-mix
Resilience	context-overflow, failure-pressure, client-cancellation

short-interactive also defines fixed, concurrency-knee, and 10x traffic-burst schedules. Load
schedules remain separate from request shapes, so a burst can be compared without duplicating the
scenario payload.

How oha and AIPerf fit together

They measure different layers and run sequentially:

• oha measures the non-streaming fixed-body HTTP ceiling for short-interactive.
• AIPerf replays every selected streaming session and measures LLM latency and token behavior.
• switchyard-soak runs the long release gate across public endpoint formats while watching
  health, counters, process use, and bounded result files.

Use the local scenario backend first for deterministic routing correctness and proxy overhead. Run
the same exported scenarios against routes backed by one real deployment when the question is
end-to-end TTFT, tokenization, generated-token throughput, or provider queuing. Real-model runs are
billable and variable, so use a dedicated deployment and repeat them before treating a small
difference as an algorithm effect.

Compare routing algorithms at high throughput

cargo build --release -p switchyard-server -p switchyard-soak \
  --bins --example switchyard-soak-mock

python3.12 scripts/benchmark_routing_algorithms.py \
  --base-url http://127.0.0.1:4000 \
  --model random=switchyard/random \
  --model classifier=switchyard/classifier \
  --model stage=switchyard/stage \
  --scenario short-interactive \
  --load-profile all \
  --concurrency 128 \
  --request-rate 20 \
  --request-count 1000 \
  --profile-runs 3 \
  --backend-label "release model deployment"

The command fails after writing the report when a scenario misses its expected client error-rate
range. The local runner also performs the backend reset needed to give every algorithm the same
transient 429/500 attempt sequence.

Run the local test

Install oha and AIPerf, then run every configured route without provider credentials or inference
cost:

cargo install oha
uv tool install --python 3.12 aiperf

python3.12 scripts/run_local_soak_test.py \
  --duration 10s \
  --concurrency 4 \
  --request-count 100

The request-aware local smoke selected the weak target for the easy classifier case, the strong
target for the hard case, recovered a weak-target context overflow through the strong target, and
recovered an injected 429 through the configured retries. AIPerf also consumed the exported
short-interactive session and produced request throughput, TTFT, and output-token metrics.

Run the release gate

target/release/switchyard-soak \
  --base-url http://127.0.0.1:4000 \
  --model RELEASE_MODEL_ID \
  --scenario-set standard \
  --duration 48h \
  --concurrency 16 \
  --server-pid "$SWITCHYARD_SERVER_PID" \
  --max-rss-growth-mib 512

Each run writes its inputs, interval measurements, up to 10,000 error records, and a final summary
to a new results directory. The command retains at most 100,000 response-time samples.

Existing valid requests keep their behavior. Chat Completions and Anthropic Messages requests whose
messages field is present but is not an array now return an error instead of being treated as if
no messages were sent.

[github-actions]

PR Preview Action v1.8.1
🚀 View preview at https://NVIDIA-NeMo.github.io/Switchyard/pr-preview/pr-176/
Built to branch gh-pages at 2026-08-19 21:38 UTC. Preview will be ready when the GitHub Pages deployment is complete.

[coderabbitai]

Walkthrough

Changes

Soak Test

Layer / File(s)	Summary
Request contracts and result recording scripts/soak_test.py, tests/test_soak_test.py	Adds endpoint payload construction, prompt and metric helpers, statistics tracking, bounded error recording, and corresponding unit tests.
Preflight and closed-loop traffic scripts/soak_test.py, tests/test_soak_test.py	Adds model discovery, streaming and non-streaming request validation, concurrent endpoint workers, and request-path tests.
Monitoring, canary, and release gates scripts/soak_test.py, tests/test_soak_test.py	Adds health, metrics, process, invalid-request, restart, and RSS checks with summary gating and failure-case coverage.
CLI orchestration and operational support scripts/soak_test.py, tests/test_soak_test.py, docs/operations/soak_test.md, mkdocs.yml, .gitignore	Adds CLI lifecycle orchestration, integration coverage, the soak-test runbook and navigation entry, and ignores generated soak results.

Estimated code review effort: 4 (Complex) | ~60 minutes

Poem

I’m a rabbit with a workload to run,
Through streams and messages, one by one.
Health checks hop, metrics gleam,
Errors rest in a bounded stream.
Green gates bloom when the soak is done!

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly identifies the primary change: adding a release soak test, with routing-related coverage also present in the test suite.

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 5

🧹 Nitpick comments (4)

scripts/soak_test.py (3)

132-149: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick win

process_sample blocks the event loop.

It's called from reporter (Line 442) on the running loop, so the ps fork/exec stalls all in-flight workers and skews the latency samples they're recording. Cheap to make non-blocking.

As per coding guidelines: "Use async-first Python APIs".

♻️ Proposed fix

-        rss_mib, cpu_percent = process_sample(server_pid)
+        rss_mib, cpu_percent = await asyncio.to_thread(process_sample, server_pid)

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@scripts/soak_test.py` around lines 132 - 149, Make process_sample
asynchronous and replace the blocking subprocess.run call with an async-first
subprocess API, awaiting completion while preserving the existing RSS/CPU
parsing and None-return behavior. Update reporter to await process_sample
wherever it samples process metrics.

Source: Coding guidelines

---

247-261: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Add docstrings to the public writer methods.

write_interval, write_error, and close are public API entry points; note the MAX_ERROR_RECORDS drop invariant in write_error.

As per coding guidelines: "Add concise triple-quoted docstrings for public functions, classes, methods, and API entry points, documenting behavior, important invariants, and relevant errors."

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@scripts/soak_test.py` around lines 247 - 261, Add concise triple-quoted
docstrings to the public methods write_interval, write_error, and close in the
writer class, documenting their behavior; explicitly note that write_error drops
records after MAX_ERROR_RECORDS and tracks the dropped count.

Source: Coding guidelines

---

876-903: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick win

Task failure discards summary.json for the whole run.

Raising here exits before the summary block, so a single failed worker after 47 hours leaves only intervals.csv/errors.jsonl and a stderr line. Consider recording the failure into the summary (completed_duration is already false, so it will be marked FAIL) and returning 1/2 instead of skipping artifact generation.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@scripts/soak_test.py` around lines 876 - 903, Update the task-result handling
loop around run_tasks and worker_results/background_results so failed tasks are
recorded for the run and do not raise before the build_summary and summary.json
artifact generation. Preserve the failure details in the summary’s
failure_reasons, ensure the resulting summary is marked failed, and return the
appropriate nonzero status after writing the summary.

tests/test_soak_test.py (1)

96-147: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Byte-matching the serialized body is brittle; prefer respx per guidelines.

request.read().find(b'"stream":true') depends on httpx's compact JSON separators; a serializer change silently flips these tests to the wrong branch without failing. Decode the body instead. Separately, these handlers are the case respx is meant for.

As per coding guidelines: "Use respx for HTTP mocking".

♻️ Sturdier branch condition

     def handler(request: httpx.Request) -> httpx.Response:
-        if request.read().find(b'"stream":true') >= 0:
+        if json.loads(request.read())["stream"]:

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/test_soak_test.py` around lines 96 - 147, Replace the byte-level
request-body checks in the handlers for
test_send_request_accepts_json_and_streaming_success and
test_send_request_rejects_missing_fields_and_non_sse_stream with decoded JSON
inspection of the stream field, and migrate these HTTP mocks to respx as
required by the project guidelines. Preserve the existing JSON-success,
SSE-success, invalid-response, and invalid-stream outcomes for stream=False and
stream=True.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/operations/soak_test.md`:
- Line 44: Update the health readiness instruction in the soak test
documentation to require both HTTP 200 and a JSON response containing
{"status":"ok"} from GET /health. Make clear that both conditions must be
satisfied before proceeding.
- Around line 45-46: Update the soak-test instructions to select and pass the
model id returned in each `/v1/models` response, rather than a release route
identifier. Apply this correction to all referenced command examples and
preserve the existing `--model` usage.
- Around line 97-110: Update the pass-criteria checklist in the soak test
documentation to replace the “no inference request fails” requirement with a
requirement that the inference error rate is at or below --max-error-rate, whose
default is 0. Keep the remaining criteria unchanged and align the wording with
the runner’s configured error-budget gate.

In `@scripts/soak_test.py`:
- Around line 403-407: Update the health-check logic around the response JSON
parsing so non-dict JSON bodies are treated as unhealthy rather than calling
.get() on them. Validate that the parsed body is a mapping before reading its
"status" field, while preserving the existing healthy condition and failure
handling in the surrounding try/except.

In `@tests/test_soak_test.py`:
- Around line 179-204: Increase the live soak duration configured in the test
invocation around soak_test.run, and adjust the report and invalid-canary
intervals if needed to preserve multiple metric and canary checks. Keep the
existing success, endpoint, and invalid_request_canaries assertions unchanged
while providing sufficient timing headroom for slower CI runners.

---

Nitpick comments:
In `@scripts/soak_test.py`:
- Around line 132-149: Make process_sample asynchronous and replace the blocking
subprocess.run call with an async-first subprocess API, awaiting completion
while preserving the existing RSS/CPU parsing and None-return behavior. Update
reporter to await process_sample wherever it samples process metrics.
- Around line 247-261: Add concise triple-quoted docstrings to the public
methods write_interval, write_error, and close in the writer class, documenting
their behavior; explicitly note that write_error drops records after
MAX_ERROR_RECORDS and tracks the dropped count.
- Around line 876-903: Update the task-result handling loop around run_tasks and
worker_results/background_results so failed tasks are recorded for the run and
do not raise before the build_summary and summary.json artifact generation.
Preserve the failure details in the summary’s failure_reasons, ensure the
resulting summary is marked failed, and return the appropriate nonzero status
after writing the summary.

In `@tests/test_soak_test.py`:
- Around line 96-147: Replace the byte-level request-body checks in the handlers
for test_send_request_accepts_json_and_streaming_success and
test_send_request_rejects_missing_fields_and_non_sse_stream with decoded JSON
inspection of the stream field, and migrate these HTTP mocks to respx as
required by the project guidelines. Preserve the existing JSON-success,
SSE-success, invalid-response, and invalid-stream outcomes for stream=False and
stream=True.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: c51ae66a-bea9-400b-b5b3-fb971f696e24

📥 Commits

Reviewing files that changed from the base of the PR and between 61f41ff and eaa62d7.

📒 Files selected for processing (5)

• .gitignore
• docs/operations/soak_test.md
• mkdocs.yml
• scripts/soak_test.py
• tests/test_soak_test.py