Repository navigation
[AgentX] DeepSeek-V4-Pro-0813 llm-d on GB200 - #3741
Conversation
|
Thanks for the contribution!
中文感谢你的贡献！
|
cquil11
added
the
full-sweep-enabled
cquil11
changed the title
|
Generally most InferenceX submissions don't need to add changes to the infx package. On the off chance they do, generally the diff is small. Please prompt your agent to clean your PR to keep your diff to only necessary & minimal changes too. We have an increased code quality bar for changes to the infx package. 中文大多数 InferenceX 提交通常不需要修改 infx 包。即使确实需要，改动通常也很小。请让你的 agent 清理 PR，使改动只包含必要且最小的内容。infx 包的改动有更高的代码质量要求。 |
|
Claude finished @cquil11's task in 1m 2s —— View job Review of PR #3741
I found one blocking issue and posted it as an inline comment.
The other checks passed:
|
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Additional findings (outside the current diff — GitHub can't attach inline comments there):
-
🔴
inferencex-e2e/infx/launch/drivers/srt/lanes.py— New gb200-nv dsv4 'vllm' (llm-d) multi-node AgentX jobs never qualify for shared_run_root, so their srt-slurm checkout lands in the plain runner workspace instead of the Lustre shared-run-root. check_request (lanes.py:69) now lets FRAMEWORK=vllm dsv4 jobs onto this lane, but shared_run_root's Matches at lines 82-83 only list frameworks=any_of("dynamo-vllm") and dynamo-sglang+agentic, so shared stays False in run_multinode (srt/init.py:134) and compute_workspace (checkout.py:195) skips staging a node-visible copy. If GITHUB_WORKSPACE isn't visible to the extra GB200 compute nodes a job allocates, the worker nodes can't find the installed venv/checkout and the job fails at startup. Fix: add a dsv4+frameworks=any_of("vllm") (agentic) Match to gb200-nv's shared_run_root tuple alongside the existing dynamo-vllm/dynamo-sglang rows.Why this was flagged
Trigger: a multinode AgentX launch with MODEL_PREFIX=dsv4, FRAMEWORK=vllm on cluster gb200-nv (the new agg-llm-d.yaml/disagg-llm-d.yaml recipes added in configs/nvidia-master.yaml, e.g. dsv4-fp4-gb200-vllm-llm-d-agentic-mtp-disagg using 2P-DEP8/1D-DEP16 over 8 nodes). lanes.py:69 now accepts framework 'vllm' for gb200-nv, but the shared_run_root Matches at lanes.py:82-83 require frameworks dynamo-vllm or dynamo-sglang exactly, so Match.call (policy.py:42) returns False for 'vllm'. run_multinode computes shared=False (srt/init.py:134), so checkout_dir (checkout.py:52) places the checkout under run.workspace rather than runners.yaml's gb200-nv shared-run-root (/mnt/lustre01/users-public/sa-shared/gha-runs, line 721). The existing dsv4+dynamo-vllm/dynamo-sglang rows were added specifically because this same cluster+model combo at multi-node scale needed the Lustre root; nothing in the new code gives 'vllm' that same treatment.
Verification: The new dsv4-fp4-gb200-vllm-llm-d-agentic-mtp-disagg/agg jobs (nvidia-master.yaml, framework: vllm, runner: cluster:gb200-nv) are admitted to the gb200-nv SRT_MULTI lane because lanes.py:69 added "vllm" to frameworks. shared is computed at srt/init.py:134, and gb200-nv's shared_run_root matches (lanes.py:81-83) cover only dynamo-vllm and dynamo-sglang+agentic, not vllm, so shared=False.
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=38275856282 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=38275856282 |
ec5ed23 to
67c1918
Compare
cquil11
added
non-canary-full-sweep-enabled
|
/stage-results 38029231836 |
|
@cquil11 staged run 38029231836: https://inferencemax-app-git-staging-semianalysisai.vercel.app/inference?i_dates=2026-10-10~r38029231836 This run remains available across future |
|
@cquil11 staged run 38275856282. This run remains available across future |
ilmarkov
added
non-canary-full-sweep-enabled
|
/use 38275856282 |
|
@cquil11 staged run 38275856282. This run remains available across future |
Signed-off-by: Cam Quilici <cjquilici@gmail.com>
Port #2719's llm-d submission to srt-slurm's llm-d frontend: disagg 1P1D DEP8/DEP8 and 2P1D DEP8/DEP16 with Mooncake on prefill, and agg TP8, DEP8, and DEP8 with Mooncake. Drop the MiniMax-M3 smoke keys. Co-authored-by: Ilya Markov <markovilya197@gmail.com> Signed-off-by: Cam Quilici <cjquilici@gmail.com>
Signed-off-by: Cam Quilici <cjquilici@gmail.com>
Signed-off-by: Cam Quilici <cjquilici@gmail.com>
Signed-off-by: Cam Quilici <cjquilici@gmail.com>
Signed-off-by: Cam Quilici <cjquilici@gmail.com>
Signed-off-by: Cam Quilici <cjquilici@gmail.com>
… timeout Signed-off-by: Cam Quilici <cjquilici@gmail.com>
Signed-off-by: Cam Quilici <cjquilici@gmail.com>
Match Sep 28 speculative policy on GB200 llm-d recipes (adaptive disagg/DEP8, golden AL only on agg TP8) and skip golden injection when adaptive is enabled. 在 GB200 llm-d AgentX 配方中恢复 9 月 28 日的投机策略（disagg/DEP8 自适应验证、 仅 agg TP8 使用黄金 AL），并在启用自适应时跳过黄金接受长度注入。
Align AgentX llm-d recipes with Sep 28 rank-level EPP routing and stop sweeping 2P×1D concurrency 576 after warmup stalls. 为 GB200 AgentX llm-d 启用每 DP rank 的外部负载均衡与 Rust 前端，并从主配置搜索空间中移除反复 warmup 失败的 2P×1D 并发 576。
llmd-vllm now launches through srt-slurm, which reads SRT_RECIPE rather than a CONFIG_FILE additional-setting. Signed-off-by: Cam Quilici <cjquilici@gmail.com>
a372834 to
5a436aa
Compare
Changes
frontend.type: llm-d, feat(frontends): add the llm-d router NVIDIA/srt-slurm#588), which the submodule includes as of v2.49.0 (chore: bump srt-slurm to v2.49.0 #3882). The standalone llm-d launcher is removed in chore: remove the standalone llm-d launcher #3883, which this PR is stacked on. Each DEP worker is onevllm servethat balances its DP ranks itself, so the EPP routes to the worker.