feat(proxy+analytics): routing header injector and LiteLLM metadata capture - #513
Merged
Merged
Conversation
…isions in transcripts The upstream router (Switchyard or LiteLLM) reports routing decisions in HTTP response headers: tier selection, decision rationale, classifier model, cost, etc. Headers are forwarded downstream but agents do not persist them, so the decision is lost when the turn ends. This plugin captures routing headers and injects them into the response body at the top level (JSON) or into the first `message_start` SSE event's `message` object (streaming). The agent then persists these fields in its native transcript alongside `usage`, making the routing metadata auditable and queryable. Routing header families supported: - x-litellm-*: LiteLLM router complexity tier, classifier model, routed model, score - x-codemie-*: Switchyard tier, decision source, confidence, judge cost and tokens Headers are captured by prefix matching, so new fields in either family are automatically recorded without code changes. Priority: 55 (after logging at 50, before session-sync at 100) - JSON path: buffers and merges fields at top level - SSE path: stores headers in context.metadata, injects into first message_start chunk - Malformed inputs (parse errors, split events, non-JSON) are passed through unchanged Tested against real session data from both Switchyard and LiteLLM deployments. Verified: both families' vocabularies captured correctly, authorization headers not leaked, other SSE events preserved, robustness against edge cases. Co-Authored-By: codemie-ai <codemie.ai@gmail.com>
…ting cost observability
Extend the analytics pipeline to fully capture LiteLLM router headers alongside
Switchyard ones, treating both families equally in the cost and timeline views.
Changes to cost/usage-readers.ts:
- Hoist parseOptInt and parseOptFloat to module scope (used by LiteLLM extraction)
- Add ClaudeRawMessage fields for the 6 previously-undeclared LiteLLM headers:
x-litellm-router-score, x-litellm-model-name (the actual deployment target)
- Introduce routingFamily discriminator ('switchyard' | 'litellm') so consumers can
distinguish between a zero routing cost (Switchyard reports it explicitly) and an
unmeasurable one (LiteLLM never emits cost headers despite running classifiers)
- Extract routingTierRaw to preserve both 2-tier (efficient/capable) and 4-tier
(SIMPLE/MEDIUM/COMPLEX/REASONING) vocabularies before normalization, allowing
future refinement of the mapping without data loss
- Populate routedModel (LiteLLM: router-routed-model), classifierModel (which LLM
decided), routerType (e.g. 'complexity'), routerScore, and requestedModel
now falls back to x-litellm-router-model-name when Switchyard isn't present
Changes to cost-enricher.ts:
- Pass the new fields through to ModelTimelinePoint
- Track routingCostKnown at session level: true when all routed turns came from a
family that reports cost (Switchyard), false when any turn used LiteLLM
(reading an absent judgeCostUSD then means 'unmeasured', not '/bin/zsh')
Changes to report types and payload-builder:
- Carry routingCostKnown through to the rendered report
Changes to report/client/app.js:
- Routing (judge) cost row now shows '— unmeasured (LiteLLM)' instead of being
absent when cost is unknown
- KPI panel includes both measured and unmeasured routed sessions, surfacing the
count and noting where cost data is incomplete
Result: LiteLLM and Switchyard deployments now render equivalent routing timelines
and KPI views, with the caveat that LiteLLM routing overhead is visible in tier
distribution and decision source but not in cost totals (missing upstream headers).
Co-Authored-By: codemie-ai <codemie.ai@gmail.com>
…se body and headers Show the model the LiteLLM router or CodeMie Switchyard actually dispatched to, not just the nominal model from Claude Code's configuration. Detect routing via: 1. Explicit routing headers (x_codemie_routing_capable_model, x-litellm-router-routed-model) 2. Response body's own model field, normalized to strip Bedrock qualifiers Use a normalized comparison (family matching: opus/sonnet/haiku/fable) to avoid false-positive arrows when aliases resolve to their concrete provider snapshots. Display format when routing differs: [Claude Sonnet 5 → claude-opus-5] - Export normalizeModelId() for use in display path (was private) - Add modelId to extractBasicInfo() to compare against candidate - Layers response-body detection on top of header detection - Never throws on missing/unreadable transcript; gracefully degrades Fixes: claude-only-expensive-no-aff showing spurious arrows to bedrock backend ids
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Continuation of PR #503
Adds two components to complete the routing decisions capture and reporting:
feat(proxy): add routing header injector plugin (037da99)
feat(analytics): capture LiteLLM routing metadata (c5c1627)
Result
Completes the routing decisions flow: headers → plugin body injection → analytics extraction → report timeline and KPIs.
Both LiteLLM and Switchyard deployments now render equivalent routing views.