Repository navigation
Conversation
…DEP8) Ports the agentX (llm-manifesto/agentx-mvp) DeepSeek-V4-Pro 1P-EP8/1D-EP8 K8s deployment to InferenceX's SLURM-based llmd-vllm framework on B200 (8 GPUs/node, so each DEP8 role fits on one node vs GB200's 4 GPUs/node needing two). New master-config key dsv4-fp4-b200-llmd-vllm-agentx and recipe carry the agentX EPP scheduling config (disagg-profile-handler's `deciders:` shape, prefix-cache-affinity-filter, token-load-scorer, etc., ported verbatim from a real agentx-mvp results snapshot) and per-role vLLM tuning, dropping the Kubernetes/manifesto-specific Mooncake KV chain in favor of server.sh's existing plain NixlConnector. Wires a self-contained FRAMEWORK==llmd-vllm branch into runners/launch_b200-dgxc.sh (ahead of the srtctl/srt-slurm path used by the other B200 multinode frameworks, which llm-d bypasses entirely), mirroring launch_gb200-nv.sh's pyxis/enroot + slurm_utils.sh helpers pattern. Pins a B200-only EPP/pd-sidecar v0.10.0 binary override (LLMD_BIN_DIR) for the newer EPP plugin shape, isolated from GB200's shared v0.9.0 binaries/path. Co-Authored-By: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Cursor <cursoragent@cursor.com>
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
- dsv4-fp4-b200-llmd-vllm-agentx: drop c256 from the 1P-DEP8/1D-DEP8 conc-list per request. - extract-binaries.sh: support BINARIES_ENV_FILE to source an alternate pins file, so a cluster-scoped router-version bump doesn't require editing the shared binaries.env (whose EPP_FROM_IMAGE/ ROUTING_SIDECAR_IMAGE are plain assignments and can't be overridden via pre-exported env vars). - Add binaries-b200-v0.10.0.env: B200-only v0.10.0 EPP/pd-sidecar pins, linux/amd64 platform (B200 is x86_64, unlike GB200's arm64 default), dedicated LLMD_BIN_DIR. - launch_b200-dgxc.sh: fix the extraction command documented in the LLMD_BIN_DIR comment, which previously relied on the non-overridable env vars above and would have silently extracted v0.9.0 binaries. Co-authored-by: Cursor <cursoragent@cursor.com>
Sibling of the P/D disagg benchmark: one engine per topology does both
prefill and decode (1 node / 8 GPUs each), no P/D KV handoff.
- server.sh/job.slurm: add an aggregated mode (DECODE_NODES=0) - no
decode role, no pd-sidecar, no --kv_transfer_config (there is nothing
to hand KV off to), and the sole engine's rank 0 takes over the
EPP/Envoy/bench coordinator duties normally owned by the decode
leader. Also fixes the underlying DECODE_NODES=0 bugs: job.slurm's
out-of-bounds IPS[] read for DECODE_LEADER_IP, and server.sh emitting
a bogus empty-address decode-0 endpoint in endpoints.yaml.
- New recipes dsv4-fp4-b200-agentx-agg-{tp8,dep8}.yaml: per-role vLLM
tuning ported from agentX's ix-agg-base/agg-tp8 and
ix-agg-dep-base/agg-dep8, plus a minimal single-schedulingProfile EPP
config (no disagg-profile-handler needed - llm-d auto-selects
single-profile-handler for one profile). Needs no EPP router bump
since every plugin used is already in the default v0.9.0 binaries.
- New dsv4_fp4_b200_llmd-vllm-agg.sh wrapper + launch_b200-dgxc.sh now
branches its llmd-vllm wrapper-script choice on $DISAGG instead of
hardcoding the disagg wrapper.
- New dsv4-fp4-b200-llmd-vllm-agentx-agg master-config key (disagg:
false, prefill num-worker=1 / decode num-worker=0 per the
process_result.py / dynamo-vllm aggregated convention). conc-lists
mirror the agentX dspark historical runs (results_ilmarkov-2-ix-agg-
tp8-dspark-v3-long, results_ilmarkov-ix-agg-dep8-dspark-1308-long).
- README.md + perf-changelog.yaml updated.
Co-authored-by: Cursor <cursoragent@cursor.com>
|
Claude finished @ilmarkov's task in 13m 58s —— View job Review of PR #2719 — DeepSeek-V4-Pro llm-d B200 (AgentX)
SummaryFound 3 blocking and 2 warning issues (posted as inline comments):
Doc-consistency nits (no inline comments): many comments reference Positive notes: server.sh's aggregated-mode gating ( |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| if [[ "$registry" == "registry-1.docker.io" && "$repository" != */* ]]; then | ||
| repository="library/$repository" | ||
| fi | ||
| printf 'docker://%s#%s:%s\n' "$registry" "$repository" "$digest" |
There was a problem hiding this comment.
🟡 WARNING: The digest branch keeps the tag inside repository, producing a malformed enroot URI for repo:tag@sha256:... images.
Why it matters: For an image like the sibling GB200 llm-d pin quay.io/rh-ee-imarkov/llm-d-nokube-vllm:vllm0.26@sha256:a909..., image_without_digest still contains :vllm0.26, so this prints docker://quay.io#rh-ee-imarkov/llm-d-nokube-vllm:vllm0.26:sha256:a909... — exactly the tag@digest ambiguity this function exists to avoid (per its own header comment). It's latent today only because the new B200 image is tag-only; the moment this image gets digest-pinned like its GB200 counterpart, the import breaks.
Fix:
| if [[ "$registry" == "registry-1.docker.io" && "$repository" != */* ]]; then | |
| repository="library/$repository" | |
| fi | |
| printf 'docker://%s#%s:%s\n' "$registry" "$repository" "$digest" | |
| if [[ "$registry" == "registry-1.docker.io" && "$repository" != */* ]]; then | |
| repository="library/$repository" | |
| fi | |
| # tag@digest form: drop the tag; enroot wants registry#repo:digest. | |
| repository="${repository%%:*}" | |
| printf 'docker://%s#%s:%s\n' "$registry" "$repository" "$digest" |
| # benchmarks/llm-d/extract-binaries.sh | ||
| # before this recipe can actually run; job.slurm's pyxis mount loop is | ||
| # a no-op (falls back to the image's baked-in v0.9.0) until then. | ||
| export LLMD_BIN_DIR="${LLMD_BIN_DIR:-/home/sa-shared/llm-d-bins-v0.10.0}" |
There was a problem hiding this comment.
🟡 WARNING: The v0.10.0 LLMD_BIN_DIR override is applied to every B200 llmd-vllm run, including the aggregated configs that declare router: { version: "0.9.0" }.
Why it matters: This export happens before the DISAGG branch, and job.slurm's mount loop mounts whatever executables it finds there over the image's baked-in binaries. Once /home/sa-shared/llm-d-bins-v0.10.0 is populated (a prerequisite for the disagg recipe to work at all), the dsv4-fp4-b200-llmd-vllm-agentx-agg runs will silently execute the v0.10.0 EPP/pd-sidecar while their master-config metadata, the recipe comments, and the perf-changelog all state they use the default v0.9.0 binaries — the published router version won't match what actually ran.
Fix: Either gate the override on the disagg path (move the export LLMD_BIN_DIR=... inside the DISAGG == true branch, added after the wrapper-selection if), or update the agg key's router: version/comments to 0.10.0 so the metadata matches reality.
There was a problem hiding this comment.
Beyond the inline findings, I checked the ruled-out changelog entry more closely: the Mooncake entry's config-keys (dsv4-fp4-b200-llmd-vllm-agentx) does match its "disagg DEP8 arm" description, since that key names the disagg config (not the -agg aggregated one), so there's no additional config-keys mismatch there.
Extended reasoning...
This run's bug hunt already surfaced five confirmed, non-trivial issues (env-var export bug breaking SLURM submission, append-only violation and invalid PLACEHOLDER pr-links in the byte-sensitive perf-changelog.yaml, a misnamed gb200 prefix on B200-only recipe files, and a malformed enroot URI for tag@ digest images), which on their own are sufficient to require human attention and block approval. I independently re-verified the one ruled-out candidate (the Mooncake entry's config-keys pointing at the disagg key dsv4-fp4-b200-llmd-vllm-agentx versus its "disagg DEP8 arm" description) by reading the surrounding perf-changelog.yaml entries directly, and confirmed the key correctly refers to the disagg config (the aggregated variant is the separate -agg key), so there is no mismatch there. Given the confirmed findings already speak for themselves as inline comments, this defer note is limited to recording that one additional, distinct concern was checked and ruled out rather than restating the existing findings.
Findings marked 🟡 are optional suggestions and need no follow-up push.
| if [[ "$FRAMEWORK" == "llmd-vllm" ]]; then | ||
| # Self-contained: bypasses the srtctl/srt-slurm path entirely (the | ||
| # "Validate framework" guard and the dsv4-only-dynamo-vllm guard | ||
| # further below are srtctl-specific and are never reached from here). | ||
| # MODEL_PATH is already resolved above (the dsv4/fp4 branch); llm-d | ||
| # additionally needs MODEL_NAME (served-model-name), which this file | ||
| # doesn't otherwise set outside the srtctl block - reuse $MODEL (the | ||
| # master-config `model:` field), matching this file's own | ||
| # `export SERVED_MODEL_NAME=$MODEL` convention below. | ||
| if [[ ! ( "$MODEL_PREFIX" == "dsv4" && "$PRECISION" == "fp4" ) ]]; then | ||
| echo "Unsupported MODEL_PREFIX/PRECISION for llmd-vllm on B200: $MODEL_PREFIX/$PRECISION" >&2 | ||
| exit 1 | ||
| fi | ||
| export MODEL_NAME="$MODEL" |
There was a problem hiding this comment.
🔴 The new llmd-vllm block never exports SLURM_PARTITION/SLURM_ACCOUNT before handing off to bash "$BENCH_SCRIPT", which forks submit.sh (a separate process) whose check_env SLURM_ACCOUNT/check_env SLURM_PARTITION require them in the environment; every B200 llmd-vllm job submission fails immediately with "Error: SLURM_ACCOUNT not set". Fix: add export SLURM_PARTITION SLURM_ACCOUNT in the llmd-vllm block, mirroring the sibling tilert branch a few lines below (line 249) which does exactly this before its own subprocess handoff.
Extended reasoning...
SLURM_PARTITION/SLURM_ACCOUNT are set as plain (non-exported) shell vars at the top of the file (lines 5-6) and no workflow sets them as job-level env, so they exist only in this script's local scope. The llmd-vllm branch runs JOB_ID=$(bash "$BENCH_SCRIPT") which execs dsv4_fp4_b200_llmd-vllm-agg.sh/disagg.sh, which runs bash ./submit.sh ... - a brand-new process that only inherits exported vars. submit.sh's check_env SLURM_ACCOUNT / check_env SLURM_PARTITION (both required first) then exit 1 before sbatch ever runs. Contrast with the tilert branch immediately below, which does export SLURM_PARTITION SLURM_ACCOUNT right before its own exec bash "$TILERT_DISAGG" handoff - the same pattern the new block omits.
Verification: normal — the new llmd-vllm block forks a subprocess without exporting the SLURM credentials that subprocess requires. runners/launch_b200-nscale-compat.sh:5-6 set SLURM_PARTITION="${SLURM_PARTITION:-batch_1}" / SLURM_ACCOUNT="${SLURM_ACCOUNT:-benchmark}" as plain (non-exported) vars. The new llmd-vllm branch runs JOB_ID=$(bash "$BENCH_SCRIPT") which execs… | normal — new code…
cquil11
added
the
full-sweep-fail-fast
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33643687916 |
Signed-off-by: Markov Ilya <markovilya197@gmail.com>
Signed-off-by: Markov Ilya <markovilya197@gmail.com>
Lower Mooncake gpu-memory-utilization to 0.87 after c52/c72 OOM, match aggregated DEP8 speculative decoding to disagg decode (3 tokens with adaptive verification), and extend 2P1D conc-list with 576. 将 Mooncake gpu-memory-utilization 降至 0.87 以修复 c52/c72 OOM， 聚合 DEP8 DSpark 与分离式 decode 对齐（3 token + 自适应验证）， 并为 2P1D 新增并发 576。
Signed-off-by: Markov Ilya <markovilya197@gmail.com>
…entic Signed-off-by: Markov Ilya <markovilya197@gmail.com>
|
Heads-up: #3576 (merged) replaced the bash launchers with a Python launcher, so this PR will conflict when you merge |
* feat(launch): route llmd-vllm multinode jobs through infx.launch Add an llm-d driver that submits the job through the llm-d wrapper, attaches to the Slurm job, streams its log, checks its final state and stages results, agentic and eval artifacts. Route llmd-vllm multinode requests to it on gb200-nv and resolve DeepSeek-V4-Pro to the node-local numa1 checkpoint. Restore the GB200 DeepSeek-V4-Pro disagg wrapper. job.slurm no longer scancels its own allocation when the coordinator finishes; it stops the srun step and exits 0, so the job ends COMPLETED instead of CANCELLED, which infx.launch reports as a failure. Driver port by Ilya Markov from #2719. * chore(launch): trim llm-d comments * feat(launch): run llm-d submit.sh directly; re-add GB200 llm-d 8k1k test point Drop the per-model llm-d wrapper script. The driver passes topology, sequence lengths and concurrency to submit.sh and sets GPUS_PER_NODE, TIME_LIMIT, CONTAINER_IMAGE and worker counts itself. llmd-vllm supports only P/D disaggregated points. Re-add the dsv4-fp4-gb200-llmd-vllm low-latency 8k1k point (1P DEP8 + 1D TP8, conc 1) to exercise the path end to end. * refactor(launch): drop the llm-d cluster whitelist Route every multinode llmd-vllm request to the llm-d driver. The driver requires slurm.squash and a staged checkpoint, which is what the whitelist stood in for. * fix(launch): leave node-local llm-d checkpoints to the job /mnt/numa1 exists only on compute nodes, so the runner host cannot see the checkpoint. job.slurm already checks it on every allocated node. * chore: drop the GB200 llm-d 8k1k test point The point validated the llm-d path end to end in e2e run 36906427804; it is not a benchmark to publish.
ba1978d to
a310fea
Compare
Add aggregate and disaggregated AgentX recipes on the Python llm-d launcher from #3613. Preserve explicit checkpoint selection, worker topology, offload metadata, and per-rank metrics collection. Keep successful Slurm completion distinct from failed or premature exits. Use golden synthetic acceptance for every AgentX MTP throughput role and launch accuracy evaluations separately with real verification. Co-authored-by: Cam Quilici <cjquilici@gmail.com>
a310fea to
139313e
Compare
…lysisAI/InferenceX into imarkov/dsv4-pro-agentic Signed-off-by: Markov Ilya <markovilya197@gmail.com>
Integrate main while preserving AgentX llm-d paths: agentic.sh routing, ALL_IPS/IS_AGGREGATED, and adaptive-verification changelog entry. 将 upstream/main 合并入 imarkov/dsv4-pro-agentic，保留 AgentX llm-d 路径、 ALL_IPS/IS_AGGREGATED 以及自适应验证 changelog 条目。
Signed-off-by: Markov Ilya <markovilya197@gmail.com>
Port #2719's llm-d submission to srt-slurm's llm-d frontend: disagg 1P1D DEP8/DEP8 and 2P1D DEP8/DEP16 with Mooncake on prefill, and agg TP8, DEP8, and DEP8 with Mooncake. Drop the MiniMax-M3 smoke keys. Co-authored-by: Ilya Markov <markovilya197@gmail.com> Signed-off-by: Cam Quilici <cjquilici@gmail.com>
|
Closing in favor of #3741, which runs this submission through srt-slurm's llm-d frontend. |
Port #2719's llm-d submission to srt-slurm's llm-d frontend: disagg 1P1D DEP8/DEP8 and 2P1D DEP8/DEP16 with Mooncake on prefill, and agg TP8, DEP8, and DEP8 with Mooncake. Drop the MiniMax-M3 smoke keys. Co-authored-by: Ilya Markov <markovilya197@gmail.com> Signed-off-by: Cam Quilici <cjquilici@gmail.com>
Port #2719's llm-d submission to srt-slurm's llm-d frontend: disagg 1P1D DEP8/DEP8 and 2P1D DEP8/DEP16 with Mooncake on prefill, and agg TP8, DEP8, and DEP8 with Mooncake. Drop the MiniMax-M3 smoke keys. Co-authored-by: Ilya Markov <markovilya197@gmail.com> Signed-off-by: Cam Quilici <cjquilici@gmail.com>
* chore: remove the standalone llm-d launcher Signed-off-by: Cam Quilici <cjquilici@gmail.com> * feat: add DeepSeek-V4-Pro-0813 llm-d AgentX on GB200 through srt-slurm Port #2719's llm-d submission to srt-slurm's llm-d frontend: disagg 1P1D DEP8/DEP8 and 2P1D DEP8/DEP16 with Mooncake on prefill, and agg TP8, DEP8, and DEP8 with Mooncake. Drop the MiniMax-M3 smoke keys. Co-authored-by: Ilya Markov <markovilya197@gmail.com> Signed-off-by: Cam Quilici <cjquilici@gmail.com> * fix: declare the 2P1D decode worker as TP16 so results count 32 GPUs Signed-off-by: Cam Quilici <cjquilici@gmail.com> * fix: stage gb200-nv dsv4 vllm runs on the shared run root Signed-off-by: Cam Quilici <cjquilici@gmail.com> * chore: drop PR references from the llm-d recipes Signed-off-by: Cam Quilici <cjquilici@gmail.com> * fix: label the llm-d keys llmd-vllm Signed-off-by: Cam Quilici <cjquilici@gmail.com> * fix: pin huggingface-hub below 2 when installing lm-eval Signed-off-by: Cam Quilici <cjquilici@gmail.com> * fix: keep decode HTTP connections alive past the llm-d sidecar's idle timeout Signed-off-by: Cam Quilici <cjquilici@gmail.com> * fix: keep agg HTTP connections alive past Envoy's upstream idle timeout Signed-off-by: Cam Quilici <cjquilici@gmail.com> * fix(srt-slurm): restore adaptive DSpark verification for llm-d AgentX Match Sep 28 speculative policy on GB200 llm-d recipes (adaptive disagg/DEP8, golden AL only on agg TP8) and skip golden injection when adaptive is enabled. 在 GB200 llm-d AgentX 配方中恢复 9 月 28 日的投机策略（disagg/DEP8 自适应验证、 仅 agg TP8 使用黄金 AL），并在启用自适应时跳过黄金接受长度注入。 * feat(benchmark): llm-d DEP8 external LB, Rust FE, drop c576 disagg point Align AgentX llm-d recipes with Sep 28 rank-level EPP routing and stop sweeping 2P×1D concurrency 576 after warmup stalls. 为 GB200 AgentX llm-d 启用每 DP rank 的外部负载均衡与 Rust 前端，并从主配置搜索空间中移除反复 warmup 失败的 2P×1D 并发 576。 * fix: select the llm-d recipes with srt-recipe-dir and srt-recipe llmd-vllm now launches through srt-slurm, which reads SRT_RECIPE rather than a CONFIG_FILE additional-setting. Signed-off-by: Cam Quilici <cjquilici@gmail.com> --------- Signed-off-by: Cam Quilici <cjquilici@gmail.com> Co-authored-by: Ilya Markov <markovilya197@gmail.com>
Changes
submit.shdirectly, with topology from the master config and an explicit 0813 checkpoint mapping.