[Klaud Cold] Add GB200 TP2 DeepSeek-V4.1-Flash vLLM AgentX arm with Engram host offload / 新增 Engram 主机卸载的 GB200 TP2 DeepSeek-V4.1-Flash vLLM AgentX 臂 - #3320
Conversation
…t offload / 新增 Engram 主机卸载的 TP2 臂 Add a TP2 search-space line beside the existing TP4 arm. The Engram tables stay in pinned host DRAM via --engram-config cpu_offload; weights rise to ~133 GiB on each 256 GiB GPU, which still leaves room for the sparse-attention indexer's 32 GiB logits buffer at the upstream batched-token default plus ~47 GiB of KV per GPU. Unlike the 180 GB B200 TP2 arm (#3216) this SKU needs no batched-token or graph-capture caps, so no benchmark script changes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
a5771ec to
421ae3e
Compare
|
Thanks for the contribution!
中文感谢你的贡献！
|
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| description: | ||
| - "Update B200 vLLM AgentX to DSpark6 and a new image with TP8 and DEP8 configurations." | ||
| pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/3274 | ||
|
|
||
| - config-keys: | ||
| - dsv41flash-fp4-gb200-vllm-agentic-dspark | ||
| scenario-type: | ||
| - agentic-coding | ||
| description: | ||
| - "Add a TP2 arm to the GB200 vLLM DeepSeek-V4.1-Flash AgentX recipe alongside the existing TP4 arm, keeping the Engram tables in pinned host DRAM via --engram-config cpu_offload; weights rise to ~133 GiB on each 256 GiB GPU, leaving room for the sparse-attention indexer's 32 GiB logits buffer at the upstream batched-token default plus ~47 GiB of KV per GPU, so the arm keeps the upstream scheduler settings" | ||
| - "Unlike the 180 GB B200 TP2 arm the GB200 GPUs need no --max-num-batched-tokens or CUDA-graph capture caps, so no benchmark script changes accompany this entry" | ||
| - "为 GB200 vLLM DeepSeek-V4.1-Flash AgentX 配方在现有 TP4 臂旁新增 TP2 臂，Engram 表继续通过 --engram-config cpu_offload 放在固定页主机 DRAM；每张 256 GiB GPU 的权重升至约 133 GiB，仍可容纳 indexer 在上游 batched-token 默认值下的 32 GiB logits 缓冲区，并为每张 GPU 留出约 47 GiB KV，因此本臂沿用上游调度器设置" | ||
| - "与 180 GB 的 B200 TP2 臂不同，GB200 无需 --max-num-batched-tokens 或 CUDA graph 捕获上限，因此本条目不涉及 benchmark 脚本改动" | ||
| pr-link: PRLINK_PLACEHOLDER |
There was a problem hiding this comment.
🔴 The new changelog entry's pr-link uses the literal string "PRLINK_PLACEHOLDER", which is not a recognized placeholder, so CI validation will reject this PR and the merge-time link rewrite will silently skip it.
Extended reasoning...
infx/workflows/validate_perf_changelog.py defines PR_LINK_PLACEHOLDERS = {"XXX", "https://github.com/SemiAnalysisAI/InferenceX/pull/XXX"}. validate_added_pr_link (line 134) checks: if pr_number is not None, link must be in PR_LINK_PLACEHOLDERS or equal the expected pull/<pr_number> URL, else it raises ChangelogValidationError with message "new PR entry must use ... or an XXX placeholder; found ". The new entry's pr-link is "PRLINK_PLACEHOLDER" (perf-changelog.yaml:8469), which matches neither condition, so this PR's CI check fails. Even if bypassed, infx/workflows/prepare_perf_changelog_merge.py imports PR_LINK_PLACEHOLDERS to detect entries needing their pr-link rewritten to the real PR URL at merge time; since "PRLINK_PLACEHOLDER" isn't in that set, the merge-prep logic won't recognize or replace it, leaving a permanently broken pr-link in the merged changelog.
Verification: normal. The new changelog entry at perf-changelog.yaml:8468 sets pr-link: PRLINK_PLACEHOLDER. The validator (infx/workflows/validate_perf_changelog.py:21-24) only accepts placeholders PR_LINK_PLACEHOLDERS = {"XXX", "https://github.com/SemiAnalysisAI/InferenceX/pull/XXX"}. In validate_added_pr_link (lines 134-145): on a PR run (pr_number not None), since "PRLINK_PLACEHOLDER" is…
GB200 TP2 c128 died in memory profiling with a -5.34 GiB KV budget (run 35528745995): weights are 145 GiB per rank, the indexer's logits buffer is 32 GiB at the upstream 16384 batched tokens, and graph capture at 1024 adds ~22 GiB. Apply the same caps the B200 TP2 arm uses (#3216): batched tokens 4096, scheduler batch bounded to 2x concurrency (16-256), capture stopped at 512. TP4 and TP8 arms keep the upstream defaults. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=35529685706 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=35529685706 |
|
/reuse-sweep-run 35529685706 |
Sync with origin/main after the green sweep run 35529685706; the reuse gate authorizes that run on this head. 在绿色 sweep 运行 35529685706 之后与 origin/main 同步；reuse gate 在此 head 上授权该运行。 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
1 participant
Summary
Adds a TP2 arm to
dsv41flash-fp4-gb200-vllm-agentic-dsparknext to the existing TP4 arm, keeping the Engram n-gram tables in pinned host DRAM through--engram-config '{"cpu_offload":true}'(the shared vLLM AgentX script passes this on every arm). This is the GB200 counterpart of the B200 TP2 arm in #3216.Memory budget, measured in the c1 server log of the merged GB200 TP4 run 35307253872 from #3247:
[batched-tokens, 1M]fp8 logits buffer is 32 GiB at the upstream 16384 batched tokens, and graph capture at 1024 (reached at c128) adds ~22 GiB.dsv41flash_fp4_vllm_mtp.shnow caps--max-num-batched-tokensat 4096, bounds--max-num-seqsto 2x concurrency (16-256) and stops capture at 512 whenever TP is 2, leaving ~49 GiB of KV per GPU. TP4 and TP8 arms keep the upstream defaults.TP2 doubles the per-GPU expert work while removing two GPUs of all-reduce, so per-GPU throughput should improve at low concurrency and the E2EL curve should bend earlier than TP4.
Upstream recipe: the merged vllm-project recipe's
single_node_tpmap defaults every non-H100/B300 NVIDIA SKU to TP4 and documents Engram CPU offload as the memory lever for smaller replica counts (DeepSeek-V4.1-Flash.yaml). TP2 with Engram offload is the same cell B300 already carries there.Test plan
full-sweep-enabledsweep: TP2 c1-c128 AgentX + evals green中文
摘要
在
dsv41flash-fp4-gb200-vllm-agentic-dspark现有 TP4 臂旁新增 TP2 臂，Engram n-gram 表继续通过--engram-config '{"cpu_offload":true}'放在固定页主机 DRAM 中（共享的 vLLM AgentX 脚本在所有臂上都会传入该参数）。本 PR 是 #3216 中 B200 TP2 臂的 GB200 对应版本。显存预算，取自 #3247 已合并的 GB200 TP4 运行 35307253872 的 c1 服务日志：
[batched-tokens, 1M]fp8 logits 缓冲区为 32 GiB，而 c128 时 graph 捕获达 1024，额外占用约 22 GiB。dsv41flash_fp4_vllm_mtp.sh现在在 TP=2 时将--max-num-batched-tokens限为 4096、--max-num-seqs限为并发的两倍（16-256）、graph 捕获上限 512，为每张 GPU 留出约 49 GiB KV。TP4 与 TP8 臂沿用上游默认值。TP2 使每 GPU 专家计算量翻倍，同时减少两张 GPU 的 all-reduce，低并发下每 GPU 吞吐应有提升，E2EL 曲线拐点也会早于 TP4。
上游配方： 已合并的 vllm-project 配方中
single_node_tp映射对 H100/B300 以外的所有 NVIDIA SKU 默认为 TP4，并将 Engram CPU 卸载记录为降低副本 GPU 数时的显存手段（DeepSeek-V4.1-Flash.yaml）。TP2 配合 Engram 卸载正是 B300 已采用的单元。测试计划
full-sweep-enabledsweep：TP2 c1-c128 AgentX 与 evals 全绿🤖 Generated with Claude Code