[Klaud Cold] Add GB300 TP2 DeepSeek-V4.1-Flash vLLM AgentX arm with Engram host offload / 新增 Engram 主机卸载的 GB300 TP2 DeepSeek-V4.1-Flash vLLM AgentX 臂 - #3321
Conversation
…t offload / 新增 Engram 主机卸载的 TP2 臂 Add a TP2 search-space line beside the existing TP4 arm. The Engram tables stay in pinned host DRAM via --engram-config cpu_offload; weights rise to ~163 GiB on each 277 GiB GPU, which still leaves room for the sparse-attention indexer's 32 GiB logits buffer at the upstream batched-token default plus ~43 GiB of KV per GPU. Unlike the 180 GB B200 TP2 arm (#3216) this SKU needs no batched-token or graph-capture caps, so no benchmark script changes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
cb1f014 to
95511a2
Compare
|
Thanks for the contribution!
中文感谢你的贡献！
|
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Beyond the inline pr-link finding, I also checked whether adding this second (TP2) arm weakens the full-sweep eval-coverage gate for the pre-existing TP4 arm — other multi-arm recipes in configs/nvidia-master.yaml already have this same shape, so it's a pre-existing pattern rather than something newly introduced here.
Extended reasoning...
The confirmed inline finding (perf-changelog.yaml pr-link left as the literal PRLINK_PLACEHOLDER instead of a real PR URL) is a real CI-blocking defect per validate_perf_changelog.py, so this PR should not be approved as-is. Given a finding was already reported, I looked for anything additional worth flagging: the ruled-out concern that adding a second search-space arm could dilute per-arm eval coverage under the full-sweep check is a legitimate pattern to consider, but it already exists for other multi-arm recipe entries elsewhere in the same file, so it is not a regression introduced by this diff and doesn't warrant a separate objection.
| - "Unlike the 180 GB B200 TP2 arm the GB300 GPUs need no --max-num-batched-tokens or CUDA-graph capture caps, so no benchmark script changes accompany this entry" | ||
| - "为 GB300 vLLM DeepSeek-V4.1-Flash AgentX 配方在现有 TP4 臂旁新增 TP2 臂，Engram 表继续通过 --engram-config cpu_offload 放在固定页主机 DRAM；每张 277 GiB GPU 的权重升至约 163 GiB，仍可容纳 indexer 在上游 batched-token 默认值下的 32 GiB logits 缓冲区，并为每张 GPU 留出约 43 GiB KV，因此本臂沿用上游调度器设置" | ||
| - "与 180 GB 的 B200 TP2 臂不同，GB300 无需 --max-num-batched-tokens 或 CUDA graph 捕获上限，因此本条目不涉及 benchmark 脚本改动" | ||
| pr-link: PRLINK_PLACEHOLDER |
There was a problem hiding this comment.
🔴 This new entry's pr-link value PRLINK_PLACEHOLDER will fail the perf-changelog CI gate, blocking merge until fixed. validate_added_pr_link in infx/workflows/validate_perf_changelog.py only accepts the literal XXX, https://github.com/SemiAnalysisAI/InferenceX/pull/XXX, or the exact https://github.com/SemiAnalysisAI/InferenceX/pull/<pr_number>; PRLINK_PLACEHOLDER matches none of them. Fix: set pr-link to the real PR URL or the canonical XXX/pull/XXX placeholder so validate_perf_changelog.py's PR_LINK_PLACEHOLDERS/CANONICAL_PR_LINK check passes.
Extended reasoning...
infx/workflows/validate_perf_changelog.py:134 defines validate_added_pr_link(link, pr_number): for a new PR entry it computes expected = f'https://github.com/SemiAnalysisAI/InferenceX/pull/{pr_number}' and raises ChangelogValidationError unless link is in PR_LINK_PLACEHOLDERS ({'XXX', '.../pull/XXX'}) or equals expected. The appended entry at perf-changelog.yaml:8468 has pr-link: PRLINK_PLACEHOLDER, which is neither. When the changelog gate workflow runs validate_perf_changelog on this PR, it raises 'new PR entry must use ... or an XXX placeholder; found PRLINK_PLACEHOLDER', failing CI and blocking merge until an author or maintainer edits the field to the real PR link.
Verification: normal (mechanism corrected). The defect is real: perf-changelog.yaml:8468 pr-link: PRLINK_PLACEHOLDER is neither an accepted placeholder nor a canonical link, so it will raise ChangelogValidationError and block the supported merge path. But the candidate misattributes WHICH gate fires. What the candidate gets right: validate_perf_changelog.py:21-24 defines PR_LINK_PLACEHOLDERS = {"XXX",…
GB200 TP2 c128 died in memory profiling with a -5.34 GiB KV budget (run 35528745995): weights are 145 GiB per rank, the indexer's logits buffer is 32 GiB at the upstream 16384 batched tokens, and graph capture at 1024 adds ~22 GiB. Apply the same caps the B200 TP2 arm uses (#3216): batched tokens 4096, scheduler batch bounded to 2x concurrency (16-256), capture stopped at 512. TP4 and TP8 arms keep the upstream defaults. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=35529688883 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=35529688883 |
|
/reuse-sweep-run 35529688883 |
Sync with origin/main after the green sweep run 35529688883; the reuse gate authorizes that run on this head. 在绿色 sweep 运行 35529688883 之后与 origin/main 同步；reuse gate 在此 head 上授权该运行。 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
1 participant
Summary
Adds a TP2 arm to
dsv41flash-fp4-gb300-vllm-agentic-dsparknext to the existing TP4 arm, keeping the Engram n-gram tables in pinned host DRAM through--engram-config '{"cpu_offload":true}'(the shared vLLM AgentX script passes this on every arm). This is the GB300 counterpart of the B200 TP2 arm in #3216.Memory budget, measured in the c1 server log of the merged GB300 TP4 run 35073891536 from #3176:
[batched-tokens, 1M]fp8 logits buffer is 32 GiB at the upstream 16384 batched tokens, and graph capture at 1024 (reached at c128) adds ~22 GiB.dsv41flash_fp4_vllm_mtp.shnow caps--max-num-batched-tokensat 4096, bounds--max-num-seqsto 2x concurrency (16-256) and stops capture at 512 whenever TP is 2, leaving ~36 GiB of KV per GPU. TP4 and TP8 arms keep the upstream defaults.TP2 doubles the per-GPU expert work while removing two GPUs of all-reduce, so per-GPU throughput should improve at low concurrency and the E2EL curve should bend earlier than TP4.
Upstream recipe: the merged vllm-project recipe's
single_node_tpmap defaults every non-H100/B300 NVIDIA SKU to TP4 and documents Engram CPU offload as the memory lever for smaller replica counts (DeepSeek-V4.1-Flash.yaml). TP2 with Engram offload is the same cell B300 already carries there.Test plan
full-sweep-enabledsweep: TP2 c1-c128 AgentX + evals green中文
摘要
在
dsv41flash-fp4-gb300-vllm-agentic-dspark现有 TP4 臂旁新增 TP2 臂，Engram n-gram 表继续通过--engram-config '{"cpu_offload":true}'放在固定页主机 DRAM 中（共享的 vLLM AgentX 脚本在所有臂上都会传入该参数）。本 PR 是 #3216 中 B200 TP2 臂的 GB300 对应版本。显存预算，取自 #3176 已合并的 GB300 TP4 运行 35073891536 的 c1 服务日志：
[batched-tokens, 1M]fp8 logits 缓冲区为 32 GiB，而 c128 时 graph 捕获达 1024，额外占用约 22 GiB。dsv41flash_fp4_vllm_mtp.sh现在在 TP=2 时将--max-num-batched-tokens限为 4096、--max-num-seqs限为并发的两倍（16-256）、graph 捕获上限 512，为每张 GPU 留出约 36 GiB KV。TP4 与 TP8 臂沿用上游默认值。TP2 使每 GPU 专家计算量翻倍，同时减少两张 GPU 的 all-reduce，低并发下每 GPU 吞吐应有提升，E2EL 曲线拐点也会早于 TP4。
上游配方： 已合并的 vllm-project 配方中
single_node_tp映射对 H100/B300 以外的所有 NVIDIA SKU 默认为 TP4，并将 Engram CPU 卸载记录为降低副本 GPU 数时的显存手段（DeepSeek-V4.1-Flash.yaml）。TP2 配合 Engram 卸载正是 B300 已采用的单元。测试计划
full-sweep-enabledsweep：TP2 c1-c128 AgentX 与 evals 全绿🤖 Generated with Claude Code