Feat/sglang endpoint support - #5
Open
oferki wants to merge 3 commits into
Open
Conversation
DisplayHitRate() falls back to the TTFT heuristic when the server never
sends usage.prompt_tokens_details.cached_tokens (e.g. vLLM without
--enable-prompt-tokens-details), but the existing cacheWarning flag meant
to surface exactly this ("server may not support prompt caching") was
never actually set anywhere in the codebase. Wire it up: once neither
signal has fired over a large-enough window, latch cacheWarning so the
existing "server may not support prompt caching" note in the run summary
fires instead of a bare, unexplained 0%.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds type=openai_sglang alongside the existing openai_vllm, covering the two concrete ways SGLang diverges from vLLM's OpenAI-compatible surface: - Like vLLM, it wants max_tokens rather than max_completion_tokens (UseCompatMaxTokens). - Unlike vLLM (a server-launch flag), SGLang only reports usage.prompt_tokens_details.cached_tokens when a request asks for it via return_cached_tokens_details — set automatically for type=openai_sglang in both the live Chat client (llm/chat_clients.go, via the existing ExtraBodyParams mechanism) and the replay/router-post path (benchmark/replay_router_wire.go / replay_router_post.go, which builds its own request body independently of the Chat client). Without this, cache hit rate reads as 0 against every SGLang response. Also adds benchmark/sglang_metrics.go, a sampler for SGLang's sglang:cache_hit_rate Prometheus gauge. It deliberately does not reuse vllm_metrics.go's delta-accumulation scheme: that scheme assumes a monotonic counter (vllm:prompt_tokens_by_source), while SGLang's metric is a gauge already expressed as a ratio, so it is sampled as a current value instead. Router-side, router/internal/vllmmetrics.DefaultNames gets a comment documenting that SGLang's counter-shaped metrics could be added there but the gauge must not be, for the same reason. Chart/report rendering for the new sglang_metrics_sample record type is left for a followup — readJSONLFileWithParams's reader/report path only recognizes vllm_metrics_sample and run_params today, and by design silently skips anything else (forward-compat with older readers), so the samples land in --save-request-data output without yet appearing in the HTML report. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Confirmed against sglang/srt/entrypoints/openai/serving_chat.py: SGLang gates cached_tokens reporting on --enable-cache-report at server launch, the same two-sided requirement as vLLM's --enable-prompt-tokens-details (the client must ask per-request AND the server must be launched with the flag). wekai already handles the client half for type=openai_sglang, but the generic "server may not support prompt caching" warning gave no way to tell that apart from a server that genuinely never caches. Name the flag for whichever backend type= identifies. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Added support for sglang openai endpoint.
type=openai_sglang