[Klaud Cold] Add MI325X TP4 DeepSeek-V4.1-Flash vLLM AgentX arm with Engram host offload / 新增 Engram 主机卸载的 MI325X TP4 DeepSeek-V4.1-Flash vLLM AgentX 臂 - #3333
Conversation
…st offload / 新增 Engram 主机卸载的 TP4 臂 Add a TP4 search-space row beside the existing TP8 arm and repin the image from the expired nightly-eed1f3d0 tag to nightly-3df4ae15, the same commit the MI355X arm moved to. That image carries vllm-project/vllm#57491, which widened the two is_cuda() gates to is_cuda_alike() so engram_config resolves on gfx942; cpu_offload now defaults on via VLLM_PLE_CPU_OFFLOAD, so the script sets it explicitly per TP: off at TP8 to keep that curve comparable, on at TP4 where the tables do not fit beside half the checkpoint. Also add --no-swa-bounded-replay. vllm-project/vllm#56227 turned SWA bounded replay on by default between the two pins and relies on a window clamp the ROCm sparse SWA path lacks, which crashed every gfx950 point with HSA_STATUS_ERROR_MEMORY_FAULT. gfx942 runs the same path. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
687ba22 to
fb18c4b
Compare
|
Thanks for the contribution!
中文感谢你的贡献！
|
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| @@ -1223,7 +1223,7 @@ minimaxm3-fp8-mi325x-vllm-agentic-mtp: | |||
| # golden AL 3.51 for throughput, real block verification for evals, adaptive | |||
There was a problem hiding this comment.
🟡 nit (optional): the header comment above dsv41flash-fp4-mi325x-vllm-agentic-dspark still says "TP8 rather than TP4 because a 256 GB card must hold its share of the 511 GB checkpoint, the GPU-resident Engram tables and a 1M-context KV pool," but this same PR adds a tp: 4 arm right below it at line 1247. A maintainer reading the block header now gets a rationale that directly contradicts the code beneath it. Fix: update or remove the TP8-only rationale in the header comment (lines 1218-1223) now that a TP4 arm exists, e.g. folding it into the new per-arm comments already added at lines 1237-1246.
Extended reasoning...
configs/amd-master.yaml:1218-1223 states TP8 is used instead of TP4 because TP4 can't fit the checkpoint plus Engram tables plus KV pool. The same diff adds { tp: 4, kv-offloading: none, spec-decoding: mtp, conc-list: [1, 2, 4, 8, 16, 32] } at line 1247 within the very entry this comment describes. On base branch there was no TP4 arm so the comment was accurate; after merge it is stale and misleading to anyone maintaining this recipe. No safeguard catches doc/code drift like this; it's comment-only so it doesn't break CI, but it misinforms future edits (e.g. someone might 'fix' the TP4 arm back out based on this stale rationale).
Verification: nit. The header comment at configs/amd-master.yaml:1220-1222 (unchanged by this diff — verified it does not appear in the diff hunks) reads "gfx942 is not in the upstream hardware table ...; TP8 rather than TP4 because a 256 GB card must hold its share of the 511 GB checkpoint, the GPU-resident Engram tables and a 1M-context KV pool." This same PR adds a TP4 arm at line 1246 (`- { tp: 4,…
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=35639762585 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=35639762585 |
|
Revoked. That authorization pointed at the sweep of Struck through rather than deleted so the history stays readable. The reuse gate matches only a standalone command line, so this comment no longer authorizes a skip. 中文已撤销。 该授权指向的是 plain ROCm nightly 渠道镜像 此处采用删除线而非删除评论，以保留记录。reuse gate 仅匹配独立成行的命令，故本评论不再授权跳过扫描。 |
…5x-tp4-engram-offload # Conflicts: # perf-changelog.yaml
Sync with origin/main after the green sweep run 35617152305; the reuse gate authorizes that run on this head. 在绿色 sweep 运行 35617152305 之后与 origin/main 同步；reuse gate 在此 head 上授权该运行。 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…10.0 镜像上重新触发 PR #3333 的扫描 The prior /reuse-sweep-run authorization was revoked: it pointed at a sweep of the plain-channel image this PR no longer ships. 此前的 /reuse-sweep-run 授权已撤销：其指向的是本 PR 不再使用的 plain 渠道镜像的扫描。 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
/reuse-sweep-run 35639762585 |
…5x-tp4-engram-offload # Conflicts: # perf-changelog.yaml
Sync with origin/main after the green sweep run 35639762585; the reuse gate authorizes that run on this head. 在绿色 sweep 运行 35639762585 之后与 origin/main 同步；reuse gate 在此 head 上授权该运行。 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
1 participant
Summary
Adds a TP4 arm with Engram CPU offload to
dsv41flash-fp4-mi325x-vllm-agentic-dspark, beside the existing TP8 arm, and repins the image.tp: 4at concurrency 1-32, mirroring the TP8 grid so the two are directly comparable.dsv41flash_fp4_mi325x_mtp.shnow sets--engram-configexplicitly per TP:cpu_offload:falseat TP8,cpu_offload:trueat TP4.nightly-eed1f3d0…tonightly-3df4ae153eb385e27b52f26c81f8edb9e20b9984.--no-swa-bounded-replayflag.Why the image had to move
The pinned
nightly-eed1f3d0…tag 404s on Docker Hub; it survives only on node caches. The new pin is the same upstream commit the MI355X arm moved to in #3326, on the plain ROCm channel that the gfx942 arms already use rather than therocm100channel, which no gfx942 recipe has exercised.That commit is also what makes this change possible: vllm-project/vllm#57491 widened two
is_cuda()gates tois_cuda_alike(), soengram_configfinally resolves on gfx942. Previously this recipe had to omit the flag entirely.Why
--engram-configis now explicit on both armsOn builds carrying #57491,
cpu_offloaddefaults on throughVLLM_PLE_CPU_OFFLOAD. Bumping the image without setting it would silently move the TP8 tables to host memory and change that curve. Setting it explicitly tofalseat TP8 keeps the existing arm behaviourally identical, andtrueat TP4 is what makes TP4 fit.Why
--no-swa-bounded-replayvllm-project/vllm#56227 turned SWA bounded replay on by default between the two pins. It relies on a window clamp that landed in the FlashInfer and FlashMLA kernels; the ROCm sparse SWA path only gained the
replay_startkwarg. On gfx950 that crashed every point withHSA_STATUS_ERROR_MEMORY_FAULTat the first prefix hit carrying a replay start. gfx942 runs the same ROCm sparse path, so this recipe pre-empts it rather than burning a sweep to rediscover it. Prefix caching stays on.Memory
With the Engram tables offloaded, resident weights are ~81 GiB per rank at TP4, from the per-rank fit across the measured H100 TP8, H200 TP4 and B200 TP2 points (256 GiB sharded plus 17.2 GiB replicated). On a 256 GB card at
--gpu-memory-utilization 0.9that leaves roughly 100 GiB of KV per GPU after the indexer's 32 GiB[batched-tokens, 1M]logits buffer. TP8 with the tables resident is unchanged at roughly 73 GiB per rank.Test plan
full-sweep-enabledsweep: TP4 concurrency 1-32 AgentX + evals green中文
摘要
在
dsv41flash-fp4-mi325x-vllm-agentic-dspark现有 TP8 臂旁新增启用 Engram CPU 卸载的 TP4 臂，并重新固定镜像。tp: 4搜索空间行，并发 1-32，与 TP8 网格一致以便直接对比。dsv41flash_fp4_mi325x_mtp.sh现按 TP 显式设置--engram-config：TP8 为cpu_offload:false，TP4 为cpu_offload:true。nightly-eed1f3d0…重新固定为nightly-3df4ae153eb385e27b52f26c81f8edb9e20b9984。--no-swa-bounded-replay。镜像为何必须更换
原固定的
nightly-eed1f3d0…在 Docker Hub 上已 404 失效，仅依靠节点缓存存活。新 pin 与 #3326 中 MI355X 臂所用为同一上游 commit，但选用 gfx942 臂一贯使用的普通 ROCm 通道，而非尚无 gfx942 配方验证过的rocm100通道。该 commit 也正是本次改动的前提：vllm-project/vllm#57491 将两处
is_cuda()判定放宽为is_cuda_alike()，使engram_config得以在 gfx942 上解析；此前本配方只能完全省略该参数。为何两个臂都显式设置
--engram-config在包含 #57491 的构建中，
cpu_offload会通过VLLM_PLE_CPU_OFFLOAD默认开启。若仅升级镜像而不设置该值，TP8 的 Engram 表会被静默移至主机内存，改变该曲线。TP8 显式设为false可保持现有臂行为完全一致，TP4 设为true则是其得以运行的前提。为何需要
--no-swa-bounded-replayvllm-project/vllm#56227 在两个 pin 之间将 SWA bounded replay 默认开启。其依赖的 window clamp 已进入 FlashInfer 与 FlashMLA kernel，但 ROCm sparse SWA 路径仅获得
replay_start参数。在 gfx950 上，这使所有数据点在首次命中携带 replay start 的 prefix 时以HSA_STATUS_ERROR_MEMORY_FAULT崩溃。gfx942 使用同一 ROCm sparse 路径，因此本配方提前规避，而非再耗费一次 sweep 重新发现。prefix caching 保持开启。显存
Engram 表卸载后，TP4 每 rank 常驻权重约 81 GiB，该数值来自对已实测的 H100 TP8、H200 TP4 与 B200 TP2 数据点的按 rank 拟合（256 GiB 分片 + 17.2 GiB 每 rank 副本）。在 256 GB 卡、
--gpu-memory-utilization 0.9下，扣除 indexer 的 32 GiB[batched-tokens, 1M]logits 缓冲区后，每张 GPU 约剩 100 GiB KV。TP8 表驻留 GPU 的情形不变，约 73 GiB/rank。测试计划
full-sweep-enabledsweep：TP4 并发 1-32 AgentX 与 evals 全绿🤖 Generated with Claude Code