[MI355X][SGLang][AgentX] Improve DeepSeek V4.1 Flash performance - #3343
Conversation
|
Thanks for the contribution!
中文感谢你的贡献！
|
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=35943469224 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=35943469224 |
cquil11
added
all-evals
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Beyond the inline all-evals finding, I checked three other candidates and ruled them out: the spec-decoding: mtp label on this DSpark recipe matches the existing convention (the sibling dsv41flash-fp4-mi355x-vllm-agentic-dspark recipe right above it does the same); the MAX_RUNNING_REQUESTS(128)/CUDA_GRAPH_MAX_BS(64) gap is explicitly documented in the script as intentional eager-decode overflow, mirroring the cookbook's high-throughput cell; and --disable-radix-cache is required alongside kv-offloading: none per the script's own comment, not an oversight.
Extended reasoning...
The diff adds a new MI355X SGLang+DSpark AgentX benchmark lane (launcher script, amd-master.yaml recipe, append-only changelog entry); no auth/crypto/injection surface, purely benchmark config and shell scripting. A confirmed finding (all-evals: true silently suppressing the throughput sweep this PR exists to add) is already posted inline, so approval is withheld; I separately verified three other candidate concerns (spec-decoding label naming, decode-graph batch-size cap vs max running requests, and disable-radix-cache under kv-offloading none) against the diff and existing sibling recipes and found each consistent with established patterns and the script's own documented rationale.
| - dsv41flash-fp4-mi355x-sglang-agentic-dspark | ||
| scenario-type: | ||
| - agentic-coding | ||
| all-evals: true |
There was a problem hiding this comment.
🔴 The new changelog entry's all-evals: true (perf-changelog.yaml:8539) will silently suppress the throughput sweep for dsv41flash-fp4-mi355x-sglang-agentic-dspark, so the CI-generated sweep for this PR only runs evals, never the TP4/EP4 conc 1-32 throughput points the PR exists to add. infx/matrix/plan.py ORs entry.all_evals into suppress_throughput (plan.py:523) independent of the PR's all-evals label, and .github/AGENT_OPERATIONS.md:89 documents "Changelog all-evals: true suppresses throughput for that entry." This contradicts the PR description's own claim of validating "six throughput points" and its stated goal of publishing a vLLM-comparable throughput number. …
Why this was flagged
…Fix: remove all-evals: true (or use evals-only/nothing, matching every other new-recipe entry in perf-changelog.yaml, none of which sets all-evals) so the throughput matrix generates for this config.
When CI (run-sweep.yml / e2e-tests.yml) calls infx.workflows.validate_perf_changelog -> infx.matrix.plan.build_plan on this PR's new changelog entry, ChangelogEntry.all_evals is True (validation.py:1014, alias all-evals). In build_plan (plan.py:522-523) expand_all_evals/suppress_throughput both become True for this entry solely because entry.all_evals is set, so the if not suppress_throughput: block (plan.py:522) that generates all_benchmark_results via generate_current(... 'none' ...) is skipped entirely for dsv41flash-fp4-mi355x-sglang-agentic-dspark. Only the eval matrix (generate_current(..., 'all', ...)) is produced. On base branch behavior (no all-evals field), a new recipe's changelog entry generates both throughput and a subset of evals. AGENT_OPERATIONS.md:89 documents this suppression as intended for reused/eval-focused entries, not for a…
Verification: normal. The candidate's mechanism is real and confirmed in code. perf-changelog.yaml:8539 sets all-evals: true on the new dsv41flash-fp4-mi355x-sglang-agentic-dspark entry (config-keys 8535-8536, scenario-type agentic-coding 8537-8538). validation.py:1014 maps this to ChangelogEntry.all_evals (alias all-evals). In plan.py:523 `suppress_throughput = evals_only or entry.evals_only or…
cquil11
changed the title
…STP qualification
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
…41-rocm10-20260923 # Conflicts: # perf-changelog.yaml
|
/stage-results 35873541294 |
|
@cquil11 staged run 35873541294: https://inferencemax-app-git-staging-semianalysisai.vercel.app/inference?i_dates=2026-09-23~r35873541294 This run remains available across future |
cquil11
removed
the
engine-patch
|
/stage-results 35943469224 |
|
@cquil11 staged run 35943469224: https://inferencemax-app-git-staging-semianalysisai.vercel.app/inference?i_dates=2026-09-24~r35943469224 This run remains available across future |
|
/reuse-sweep-run 35943469224 |
1 participant
Use the explicitly selected native
lmsysorg/sglang:dev-dsv41-mi35x@sha256:c609f74f01b206af9664fb66da8ebef47642a7328a2478eabeebb352e014661aimage for DeepSeek V4.1 Flash: TP4/EP4, GPU Engram, shipped-default DSpark5 precision, radix caching enabled, and native 1M context. The recipe uses 4,096-token prefill chunks, static memory fraction 0.70, admission 32, supported allocator GC, eager prefill and native decode graphs. No engine patch or draft-precision override is included.Earlier C32 trials failed during long prefill. The serialized diagnostic localized the failure to native compressed-KV storage; actual gfx950 compiled code independently confirms signed 32-bit byte-offset arithmetic can overflow with the default pool. The supported
--max-total-tokens 3145728cap keeps ratio-1 storage below 2 GiB while preserving model context. The corrected runtime completed the official sweep; the compiled overflow hazard is proven independently, while a corrupt live index was not sampled. Reduced cache capacity remains a performance tradeoff.The official full sweep 35943469224, attempt 1, is SUCCESS at selected head
a21462389f433eb025596386c8f74a8745f321a6. All six performance and six full GSM cells passed: TP4/EP4 at C1, 2, 4, 8, 16 and 32. Runtime checkout was pinned event merge37b2ba7dd6a4f229cb76b89c7d0ce58b0274f1bewith this selected head as its second parent.All six raw performance exports have
submission_valid=true, no request errors, normal warmup and 3,600-second profiling. Each meets the required 95% duration coverage using TTFT OR ITL; observed TTFT coverage is 99.54–100% and ITL coverage is 99.98–100%. All six accuracy artifacts contain all 1,319 GSM questions withlimit=None; strict accuracy ranges from 97.1190% to 97.5739%, above the 90% gate. The pinned image and TP4/EP4 match. Prefix reuse and native 1M context are retained; the image's separate chunk-boundary caching compatibility restriction remains unchanged. This qualifies the six-cell grid only; parity with the published TP2/TP4 C<=128 frontier was not assessed.Power is optional by explicit user instruction. C4 retains raw
power_valid=0; no power metric was fabricated and shared validation is unchanged./stage-results 35943469224 completed. The bot acknowledgement names this exact run and links InferenceX-app 35952609482, independently verified SUCCESS, including ingest, chart refresh and callback. View staged results.
Previously completed Bash syntax, real 6+6 matrix and strict changelog validation remain applicable: exactly one new entry with historical bytes preserved. PR was merged as
d0a81b61ac514439ed39494b0929f20a14372e44. The Sep22 staging and earlier screening runs remain historical evidence.The main publication run reused source sweep
35943469224and passed. Production ingestion passed database verification and publication checks: six benchmark rows, six evaluations, 7,914 evaluation samples and one changelog entry. The public API independently confirms all six TP4/EP4 concurrency points match source throughput/image and preserve source-run links. Published results.