From 803bd730a209d5e6fb3684068396aa4dd190151c Mon Sep 17 00:00:00 2001 From: Fangzhou Ai <31551580+Fangzhou-Ai@users.noreply.github.com> Date: Mon, 21 Sep 2026 04:13:33 +0000 Subject: [PATCH 01/12] config(dsv41flash): sweep MI355X at TP=2 and TP=4 through concurrency 128 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add a TP=2 search space next to TP=4 for the MI355X DSv4.1-Flash AgentX entry, and extend both rows to concurrency 128. TP=2 became feasible once the Engram tables moved to host memory: at 47.2 GiB of device memory per rank they previously forced four GPUs per server, and two GPUs per server doubles the servers per node. The image moves to the ROCm 10 nightly and carries a placeholder tag until that build is published. The digest is pinned before the sweep runs, which is why the PR opens as a draft. 中文：为 MI355X DSv4.1-Flash AgentX 配置在 TP=4 之外新增 TP=2 搜索空间，并将两者 的并发扩展到 128。Engram 表已驻留主机内存，不再占用每 rank 47.2 GiB 设备内存 （此前这迫使每个服务器使用四张 GPU），因此 TP=2 可行，且每节点可容纳的服务器数 量翻倍。镜像切换到 ROCm 10 nightly，在该构建发布前暂用占位 tag；digest 固定后再 运行 sweep，因此本 PR 以 draft 形式提交。 Co-authored-by: Cursor --- configs/amd-master.yaml | 11 +++++++++-- perf-changelog.yaml | 11 +++++++++++ 2 files changed, 20 insertions(+), 2 deletions(-) diff --git a/configs/amd-master.yaml b/configs/amd-master.yaml index 9b4fcadeb..c62cf55b4 100644 --- a/configs/amd-master.yaml +++ b/configs/amd-master.yaml @@ -1458,8 +1458,14 @@ dsv4-fp4-mi355x-sglang-agentic-mtp: # tag predates vllm-project/vllm#56503, which moves the mHC delayed pre block off # the eager Torch reference and onto AITER. MI355X run 34710937012 passed the # full concurrency 1-32 sweep and eval-only concurrency 32. +# TP=2 joins TP=4, and both extend to concurrency 128. TP=2 became feasible on +# this SKU once the Engram tables moved to host memory: they cost 47.2 GiB of +# device memory per rank, which is what previously forced four GPUs per server. +# Two GPUs per server doubles the servers per node and is the layout that +# decides whether DSv4.1-Flash is throughput- or interactivity-bound here. dsv41flash-fp4-mi355x-vllm-agentic-dspark: - image: vllm/vllm-openai-rocm:nightly-eed1f3d0c6043bd494424a22443ee198dd56f657 + # TODO: pin to the ROCm 10 nightly digest once that image is published. + image: vllm/vllm-openai-rocm:nightly-rocm10-TBD model: deepseek-ai/DeepSeek-V4.1-Flash model-prefix: dsv41flash runner: cluster:mi355x-amds @@ -1471,4 +1477,5 @@ dsv41flash-fp4-mi355x-vllm-agentic-dspark: - dram-utilization: 0.60 search-space: # Follow upstream AMD Engram defaults; omit the CUDA-only config flag. - - { tp: 4, kv-offloading: none, spec-decoding: mtp, conc-list: [1, 2, 4, 8, 16, 32] } + - { tp: 4, kv-offloading: none, spec-decoding: mtp, conc-list: [1, 2, 4, 8, 16, 32, 64, 128] } + - { tp: 2, kv-offloading: none, spec-decoding: mtp, conc-list: [1, 2, 4, 8, 16, 32, 64, 128] } diff --git a/perf-changelog.yaml b/perf-changelog.yaml index c5b4309f5..f6ed77c36 100644 --- a/perf-changelog.yaml +++ b/perf-changelog.yaml @@ -8455,3 +8455,14 @@ description: - "Update B200 vLLM AgentX to DSpark6 and a new image with TP8 and DEP8 configurations." pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/3274 + +- config-keys: + - dsv41flash-fp4-mi355x-vllm-agentic-dspark + scenario-type: + - agentic-coding + description: + - "Add a TP=2 search space alongside TP=4 for MI355X DSv4.1-Flash AgentX, and extend both to concurrency 128. TP=2 is feasible now that the Engram tables live in host memory rather than costing 47.2 GiB of device memory per rank, which previously forced four GPUs per server." + - "Move the image to the ROCm 10 nightly; the digest is pinned once that build is published." + - "为 MI355X DSv4.1-Flash AgentX 在 TP=4 之外新增 TP=2 搜索空间，并将两者的并发扩展到 128。Engram 表已驻留主机内存，不再占用每 rank 47.2 GiB 设备内存（此前这迫使每个服务器使用四张 GPU），因此 TP=2 可行。" + - "镜像切换到 ROCm 10 nightly；该构建发布后再固定 digest。" + pr-link: TBD From 0de2d6e8ca887e92b098b715d4823ccea2ebd84c Mon Sep 17 00:00:00 2001 From: Fangzhou Ai <31551580+Fangzhou-Ai@users.noreply.github.com> Date: Mon, 21 Sep 2026 04:14:21 +0000 Subject: [PATCH 02/12] config(dsv41flash): record the sweep PR link in the changelog MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 中文：在 changelog 中记录本次 sweep 的 PR 链接。 Co-authored-by: Cursor --- perf-changelog.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/perf-changelog.yaml b/perf-changelog.yaml index f6ed77c36..95ff22a4b 100644 --- a/perf-changelog.yaml +++ b/perf-changelog.yaml @@ -8465,4 +8465,4 @@ - "Move the image to the ROCm 10 nightly; the digest is pinned once that build is published." - "为 MI355X DSv4.1-Flash AgentX 在 TP=4 之外新增 TP=2 搜索空间，并将两者的并发扩展到 128。Engram 表已驻留主机内存，不再占用每 rank 47.2 GiB 设备内存（此前这迫使每个服务器使用四张 GPU），因此 TP=2 可行。" - "镜像切换到 ROCm 10 nightly；该构建发布后再固定 digest。" - pr-link: TBD + pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/3326 From 99f0859eb23a538389c0f285fd768b9822d4f28d Mon Sep 17 00:00:00 2001 From: Fangzhou Ai <31551580+Fangzhou-Ai@users.noreply.github.com> Date: Mon, 21 Sep 2026 04:18:07 +0000 Subject: [PATCH 03/12] config(dsv41flash): let vLLM pick max_num_seqs for the MI355X AgentX recipe MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A fixed max_num_seqs of 128 caps in-flight sequences at the outer concurrency once the sweep reaches c128, leaving no headroom for AgentX subagent fan-out. Use the MI355X API-server default of 1024 instead, and derive the CUDA graph ceiling from max(2 * CONC, 128): 1024 through c64, 2048 at c128. Supersedes PR #3111, which carried this recipe change and was closed in favour of this sweep. 中文：扫描到 c128 时，固定的 max_num_seqs 128 会把在途序列数限制在外层并发上， 使 AgentX 子代理扇出没有余量。改用 MI355X API server 默认值 1024，并由 max(2 * CONC, 128) 推导 CUDA graph 上限：c64 及以下为 1024，c128 为 2048。 本次提交取代已关闭的 PR #3111（该 PR 原本包含此 recipe 改动）。 Co-authored-by: Cursor --- .../agentic/dsv41flash_fp4_mi355x_vllm_mtp.sh | 15 ++++++++++----- perf-changelog.yaml | 2 ++ 2 files changed, 12 insertions(+), 5 deletions(-) diff --git a/benchmarks/single_node/agentic/dsv41flash_fp4_mi355x_vllm_mtp.sh b/benchmarks/single_node/agentic/dsv41flash_fp4_mi355x_vllm_mtp.sh index fd6da6ab9..e2855035d 100755 --- a/benchmarks/single_node/agentic/dsv41flash_fp4_mi355x_vllm_mtp.sh +++ b/benchmarks/single_node/agentic/dsv41flash_fp4_mi355x_vllm_mtp.sh @@ -41,12 +41,18 @@ export VLLM_ENGINE_READY_TIMEOUT_S=3600 export VLLM_USE_RUST_FRONTEND=1 export PYTHONUNBUFFERED=1 -# Upstream picks 1024 on GPUs with >= 160 GiB, and 2*CONC starves AgentX -# subagent fan-out at low CONC. 128 also keeps CAPTURE_SIZE deterministic. -MAX_NUM_SEQS=128 +# Let vLLM select max_num_seqs; its API-server default is 1024 on MI355X. A +# fixed 128 caps in-flight sequences at the outer concurrency once the sweep +# reaches c128, leaving no headroom for AgentX subagent fan-out. +# Keep graph capture at the #3058 size through c64, then cover twice the outer +# concurrency at c128. NUM_SPEC_TOKENS=5 +GRAPH_NUM_SEQS=$((2 * CONC)) +if (( GRAPH_NUM_SEQS < 128 )); then + GRAPH_NUM_SEQS=128 +fi CAPTURE_SIZE=1 -while (( CAPTURE_SIZE < MAX_NUM_SEQS * (1 + NUM_SPEC_TOKENS) && CAPTURE_SIZE < 2048 )); do +while (( CAPTURE_SIZE < GRAPH_NUM_SEQS * (1 + NUM_SPEC_TOKENS) && CAPTURE_SIZE < 2048 )); do CAPTURE_SIZE=$((CAPTURE_SIZE * 2)) done @@ -82,7 +88,6 @@ VLLM_CMD=( --gpu-memory-utilization 0.9 --speculative-config "$SPEC_CONFIG" --max-model-len 1048576 - --max-num-seqs "$MAX_NUM_SEQS" --max-cudagraph-capture-size "$CAPTURE_SIZE" --max-num-batched-tokens 16384 --disable-uvicorn-access-log diff --git a/perf-changelog.yaml b/perf-changelog.yaml index 95ff22a4b..2783daf50 100644 --- a/perf-changelog.yaml +++ b/perf-changelog.yaml @@ -8463,6 +8463,8 @@ description: - "Add a TP=2 search space alongside TP=4 for MI355X DSv4.1-Flash AgentX, and extend both to concurrency 128. TP=2 is feasible now that the Engram tables live in host memory rather than costing 47.2 GiB of device memory per rank, which previously forced four GPUs per server." - "Move the image to the ROCm 10 nightly; the digest is pinned once that build is published." + - "Drop the fixed max_num_seqs of 128 and use vLLM's MI355X API-server default of 1024, since a fixed 128 caps in-flight sequences at the outer concurrency once the sweep reaches c128. The CUDA graph ceiling stays at 1024 through c64 and rises to 2048 at c128, covering twice the outer concurrency. Supersedes PR #3111." - "为 MI355X DSv4.1-Flash AgentX 在 TP=4 之外新增 TP=2 搜索空间，并将两者的并发扩展到 128。Engram 表已驻留主机内存，不再占用每 rank 47.2 GiB 设备内存（此前这迫使每个服务器使用四张 GPU），因此 TP=2 可行。" - "镜像切换到 ROCm 10 nightly；该构建发布后再固定 digest。" + - "移除固定的 max_num_seqs 128，改用 vLLM 在 MI355X API server 上的默认值 1024：扫描到 c128 时，固定 128 会把在途序列数限制在外层并发上。CUDA graph 上限在 c64 及以下保持 1024，c128 提升至 2048，以覆盖两倍外层并发。本 PR 取代 PR #3111。" pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/3326 From 039194e50d92a80c847d770729e106237e2b36c4 Mon Sep 17 00:00:00 2001 From: Fangzhou Ai <31551580+Fangzhou-Ai@users.noreply.github.com> Date: Mon, 21 Sep 2026 04:24:30 +0000 Subject: [PATCH 04/12] config(dsv41flash): keep the placeholder on the existing ROCm nightly channel MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Per #3215, the MI355X DSv4.1-Flash entry stays on the existing upstream ROCm nightly channel rather than moving to nightly-rocm100. Drop the rocm10 marker from the placeholder tag; the pin is filled in with the nightly this sweep actually runs on before dispatch. 中文：依据 #3215，MI355X DSv4.1-Flash 条目继续使用现有的上游 ROCm nightly 渠道， 不切换到 nightly-rocm100。占位 tag 去掉 rocm10 标记；派发前再填入本次 sweep 实际运行的 nightly。 Co-authored-by: Cursor --- configs/amd-master.yaml | 5 +++-- perf-changelog.yaml | 4 ++-- 2 files changed, 5 insertions(+), 4 deletions(-) diff --git a/configs/amd-master.yaml b/configs/amd-master.yaml index c62cf55b4..fea471d85 100644 --- a/configs/amd-master.yaml +++ b/configs/amd-master.yaml @@ -1464,8 +1464,9 @@ dsv4-fp4-mi355x-sglang-agentic-mtp: # Two GPUs per server doubles the servers per node and is the layout that # decides whether DSv4.1-Flash is throughput- or interactivity-bound here. dsv41flash-fp4-mi355x-vllm-agentic-dspark: - # TODO: pin to the ROCm 10 nightly digest once that image is published. - image: vllm/vllm-openai-rocm:nightly-rocm10-TBD + # TODO: pin the nightly this sweep runs on. Stay on the existing ROCm nightly + # channel rather than nightly-rocm100, per #3215. + image: vllm/vllm-openai-rocm:nightly-TBD model: deepseek-ai/DeepSeek-V4.1-Flash model-prefix: dsv41flash runner: cluster:mi355x-amds diff --git a/perf-changelog.yaml b/perf-changelog.yaml index 2783daf50..68793a83f 100644 --- a/perf-changelog.yaml +++ b/perf-changelog.yaml @@ -8462,9 +8462,9 @@ - agentic-coding description: - "Add a TP=2 search space alongside TP=4 for MI355X DSv4.1-Flash AgentX, and extend both to concurrency 128. TP=2 is feasible now that the Engram tables live in host memory rather than costing 47.2 GiB of device memory per rank, which previously forced four GPUs per server." - - "Move the image to the ROCm 10 nightly; the digest is pinned once that build is published." + - "Re-pin the image to the upstream ROCm nightly this sweep runs on, staying on the existing nightly channel rather than nightly-rocm100; the digest is filled in before dispatch." - "Drop the fixed max_num_seqs of 128 and use vLLM's MI355X API-server default of 1024, since a fixed 128 caps in-flight sequences at the outer concurrency once the sweep reaches c128. The CUDA graph ceiling stays at 1024 through c64 and rises to 2048 at c128, covering twice the outer concurrency. Supersedes PR #3111." - "为 MI355X DSv4.1-Flash AgentX 在 TP=4 之外新增 TP=2 搜索空间，并将两者的并发扩展到 128。Engram 表已驻留主机内存，不再占用每 rank 47.2 GiB 设备内存（此前这迫使每个服务器使用四张 GPU），因此 TP=2 可行。" - - "镜像切换到 ROCm 10 nightly；该构建发布后再固定 digest。" + - "将镜像重新固定到本次 sweep 实际运行的上游 ROCm nightly，继续使用现有 nightly 渠道而非 nightly-rocm100；digest 在派发前填入。" - "移除固定的 max_num_seqs 128，改用 vLLM 在 MI355X API server 上的默认值 1024：扫描到 c128 时，固定 128 会把在途序列数限制在外层并发上。CUDA graph 上限在 c64 及以下保持 1024，c128 提升至 2048，以覆盖两倍外层并发。本 PR 取代 PR #3111。" pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/3326 From a1ddae93053c525047331842fc0afbb706abb660 Mon Sep 17 00:00:00 2001 From: Fangzhou Ai <31551580+Fangzhou-Ai@users.noreply.github.com> Date: Mon, 21 Sep 2026 04:43:16 +0000 Subject: [PATCH 05/12] Revert "config(dsv41flash): keep the placeholder on the existing ROCm nightly channel" MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit This reverts commit 039194e50d92a80c847d770729e106237e2b36c4. The MI355X DSv4.1-Flash AgentX sweep runs on the ROCm 10 nightly after all, so restore the nightly-rocm100 placeholder. The pin is filled in with the published nightly before dispatch. 中文：还原提交 039194e50d92a80c847d770729e106237e2b36c4。 MI355X DSv4.1-Flash AgentX sweep 最终仍在 ROCm 10 nightly 上运行，因此恢复 nightly-rocm100 占位 tag。派发前再填入已发布的 nightly。 Co-authored-by: Cursor --- configs/amd-master.yaml | 5 ++--- perf-changelog.yaml | 4 ++-- 2 files changed, 4 insertions(+), 5 deletions(-) diff --git a/configs/amd-master.yaml b/configs/amd-master.yaml index fea471d85..c62cf55b4 100644 --- a/configs/amd-master.yaml +++ b/configs/amd-master.yaml @@ -1464,9 +1464,8 @@ dsv4-fp4-mi355x-sglang-agentic-mtp: # Two GPUs per server doubles the servers per node and is the layout that # decides whether DSv4.1-Flash is throughput- or interactivity-bound here. dsv41flash-fp4-mi355x-vllm-agentic-dspark: - # TODO: pin the nightly this sweep runs on. Stay on the existing ROCm nightly - # channel rather than nightly-rocm100, per #3215. - image: vllm/vllm-openai-rocm:nightly-TBD + # TODO: pin to the ROCm 10 nightly digest once that image is published. + image: vllm/vllm-openai-rocm:nightly-rocm10-TBD model: deepseek-ai/DeepSeek-V4.1-Flash model-prefix: dsv41flash runner: cluster:mi355x-amds diff --git a/perf-changelog.yaml b/perf-changelog.yaml index 68793a83f..2783daf50 100644 --- a/perf-changelog.yaml +++ b/perf-changelog.yaml @@ -8462,9 +8462,9 @@ - agentic-coding description: - "Add a TP=2 search space alongside TP=4 for MI355X DSv4.1-Flash AgentX, and extend both to concurrency 128. TP=2 is feasible now that the Engram tables live in host memory rather than costing 47.2 GiB of device memory per rank, which previously forced four GPUs per server." - - "Re-pin the image to the upstream ROCm nightly this sweep runs on, staying on the existing nightly channel rather than nightly-rocm100; the digest is filled in before dispatch." + - "Move the image to the ROCm 10 nightly; the digest is pinned once that build is published." - "Drop the fixed max_num_seqs of 128 and use vLLM's MI355X API-server default of 1024, since a fixed 128 caps in-flight sequences at the outer concurrency once the sweep reaches c128. The CUDA graph ceiling stays at 1024 through c64 and rises to 2048 at c128, covering twice the outer concurrency. Supersedes PR #3111." - "为 MI355X DSv4.1-Flash AgentX 在 TP=4 之外新增 TP=2 搜索空间，并将两者的并发扩展到 128。Engram 表已驻留主机内存，不再占用每 rank 47.2 GiB 设备内存（此前这迫使每个服务器使用四张 GPU），因此 TP=2 可行。" - - "将镜像重新固定到本次 sweep 实际运行的上游 ROCm nightly，继续使用现有 nightly 渠道而非 nightly-rocm100；digest 在派发前填入。" + - "镜像切换到 ROCm 10 nightly；该构建发布后再固定 digest。" - "移除固定的 max_num_seqs 128，改用 vLLM 在 MI355X API server 上的默认值 1024：扫描到 c128 时，固定 128 会把在途序列数限制在外层并发上。CUDA graph 上限在 c64 及以下保持 1024，c128 提升至 2048，以覆盖两倍外层并发。本 PR 取代 PR #3111。" pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/3326 From 4db75a62853ff5579027e5e5a206f4311d31fcc3 Mon Sep 17 00:00:00 2001 From: Fangzhou Ai <31551580+Fangzhou-Ai@users.noreply.github.com> Date: Mon, 21 Sep 2026 05:58:26 +0000 Subject: [PATCH 06/12] config(dsv41flash): pin the MI355X AgentX sweep to the ROCm 10 nightly MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Replace the placeholder with vllm/vllm-openai-rocm:nightly-rocm100-3df4ae153eb385e27b52f26c81f8edb9e20b9984, published 2026-09-21T05:50:03Z with digest sha256:eccb72b74b8c04ce7406d9212e200b129f58e24be642795a857f755a94fdb3a1. The tag is active and linux/amd64, and its manifest resolves from registry-1.docker.io at that digest. This sweep qualifies the new image; the previous green run on nightly-eed1f3d0 is not evidence for it. 将占位 tag 替换为 vllm/vllm-openai-rocm:nightly-rocm100-3df4ae153eb385e27b52f26c81f8edb9e20b9984， 该镜像于 2026-09-21T05:50:03Z 发布，digest 为 sha256:eccb72b74b8c04ce7406d9212e200b129f58e24be642795a857f755a94fdb3a1。 该 tag 状态为 active、架构为 linux/amd64，其 manifest 可从 registry-1.docker.io 按该 digest 解析。 本次 sweep 用于验证新镜像；此前在 nightly-eed1f3d0 上的绿色运行不能作为其证据。 Co-authored-by: Cursor --- configs/amd-master.yaml | 4 ++-- perf-changelog.yaml | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/configs/amd-master.yaml b/configs/amd-master.yaml index c62cf55b4..ecc97d7be 100644 --- a/configs/amd-master.yaml +++ b/configs/amd-master.yaml @@ -1464,8 +1464,8 @@ dsv4-fp4-mi355x-sglang-agentic-mtp: # Two GPUs per server doubles the servers per node and is the layout that # decides whether DSv4.1-Flash is throughput- or interactivity-bound here. dsv41flash-fp4-mi355x-vllm-agentic-dspark: - # TODO: pin to the ROCm 10 nightly digest once that image is published. - image: vllm/vllm-openai-rocm:nightly-rocm10-TBD + # ROCm 10.0 nightly channel, shared with kimik3-fp4-mi355x-vllm-agentic-mtp. + image: vllm/vllm-openai-rocm:nightly-rocm100-3df4ae153eb385e27b52f26c81f8edb9e20b9984 model: deepseek-ai/DeepSeek-V4.1-Flash model-prefix: dsv41flash runner: cluster:mi355x-amds diff --git a/perf-changelog.yaml b/perf-changelog.yaml index 2783daf50..9e1e293fa 100644 --- a/perf-changelog.yaml +++ b/perf-changelog.yaml @@ -8462,9 +8462,9 @@ - agentic-coding description: - "Add a TP=2 search space alongside TP=4 for MI355X DSv4.1-Flash AgentX, and extend both to concurrency 128. TP=2 is feasible now that the Engram tables live in host memory rather than costing 47.2 GiB of device memory per rank, which previously forced four GPUs per server." - - "Move the image to the ROCm 10 nightly; the digest is pinned once that build is published." + - "Move the image from vllm/vllm-openai-rocm:nightly-eed1f3d0c6043bd494424a22443ee198dd56f657 on the plain ROCm nightly channel to vllm/vllm-openai-rocm:nightly-rocm100-3df4ae153eb385e27b52f26c81f8edb9e20b9984 on the ROCm 10.0 nightly channel, published 2026-09-21T05:50:03Z with digest sha256:eccb72b74b8c04ce7406d9212e200b129f58e24be642795a857f755a94fdb3a1. kimik3-fp4-mi355x-vllm-agentic-mtp already runs that channel on this cluster. This sweep is the qualification for the new image; the previous green run is not evidence for it." - "Drop the fixed max_num_seqs of 128 and use vLLM's MI355X API-server default of 1024, since a fixed 128 caps in-flight sequences at the outer concurrency once the sweep reaches c128. The CUDA graph ceiling stays at 1024 through c64 and rises to 2048 at c128, covering twice the outer concurrency. Supersedes PR #3111." - "为 MI355X DSv4.1-Flash AgentX 在 TP=4 之外新增 TP=2 搜索空间，并将两者的并发扩展到 128。Engram 表已驻留主机内存，不再占用每 rank 47.2 GiB 设备内存（此前这迫使每个服务器使用四张 GPU），因此 TP=2 可行。" - - "镜像切换到 ROCm 10 nightly；该构建发布后再固定 digest。" + - "镜像从 plain ROCm nightly 渠道的 vllm/vllm-openai-rocm:nightly-eed1f3d0c6043bd494424a22443ee198dd56f657 切换到 ROCm 10.0 nightly 渠道的 vllm/vllm-openai-rocm:nightly-rocm100-3df4ae153eb385e27b52f26c81f8edb9e20b9984，该镜像于 2026-09-21T05:50:03Z 发布，digest 为 sha256:eccb72b74b8c04ce7406d9212e200b129f58e24be642795a857f755a94fdb3a1。本集群上的 kimik3-fp4-mi355x-vllm-agentic-mtp 已在使用该渠道。本次 sweep 即为新镜像的验证；此前的绿色运行不能作为其证据。" - "移除固定的 max_num_seqs 128，改用 vLLM 在 MI355X API server 上的默认值 1024：扫描到 c128 时，固定 128 会把在途序列数限制在外层并发上。CUDA graph 上限在 c64 及以下保持 1024，c128 提升至 2048，以覆盖两倍外层并发。本 PR 取代 PR #3111。" pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/3326 From beec2ef1735311bbe121d107cd5c10d8e019d231 Mon Sep 17 00:00:00 2001 From: Fangzhou Ai <31551580+Fangzhou-Ai@users.noreply.github.com> Date: Mon, 21 Sep 2026 06:12:21 +0000 Subject: [PATCH 07/12] config(dsv41flash): offload Engram to host memory at TP=2 on MI355X MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit vllm-project/vllm#57491 widened the two ROCm is_cuda() gates to is_cuda_alike(), so the pinned nightly resolves an Engram config on gfx950 and offloads the tables to pinned host memory by default. Set cpu_offload explicitly per TP instead of taking that default. TP=2 needs the offload: the tables cost 47.2 GiB per rank at TP=4, so 94.4 GiB at TP=2, which does not fit beside half of the 511 GB checkpoint on a 288 GiB card. TP=4 keeps them resident so it stays comparable with the validated concurrency 1-32 run. vllm-project/vllm#57491 将 ROCm 上的两处 is_cuda() 判断放宽为 is_cuda_alike()，因此所固定的 nightly 在 gfx950 上会解析 Engram 配置， 并默认将表下放到锁页主机内存。这里按 TP 显式设置 cpu_offload，而不是 沿用该默认值。 TP=2 需要该下放：表在 TP=4 时每 rank 占用 47.2 GiB，TP=2 时为 94.4 GiB， 无法与 511 GB 检查点的一半同时放入 288 GiB 的单卡。TP=4 保持表常驻 GPU， 以便与已验证的并发 1-32 运行保持可比。 Co-authored-by: Cursor --- .../agentic/dsv41flash_fp4_mi355x_vllm_mtp.sh | 13 +++++++++++++ configs/amd-master.yaml | 2 +- perf-changelog.yaml | 6 ++++-- 3 files changed, 18 insertions(+), 3 deletions(-) diff --git a/benchmarks/single_node/agentic/dsv41flash_fp4_mi355x_vllm_mtp.sh b/benchmarks/single_node/agentic/dsv41flash_fp4_mi355x_vllm_mtp.sh index e2855035d..f4b5ba6a1 100755 --- a/benchmarks/single_node/agentic/dsv41flash_fp4_mi355x_vllm_mtp.sh +++ b/benchmarks/single_node/agentic/dsv41flash_fp4_mi355x_vllm_mtp.sh @@ -46,6 +46,18 @@ export PYTHONUNBUFFERED=1 # reaches c128, leaving no headroom for AgentX subagent fan-out. # Keep graph capture at the #3058 size through c64, then cover twice the outer # concurrency at c128. +# vllm-project/vllm#57491 widened the two is_cuda() gates to is_cuda_alike(), +# so on gfx950 this image now resolves an Engram config and offloads the tables +# to pinned host memory unless an explicit value says otherwise. The tables cost +# 47.2 GiB per rank at TP=4, so 94.4 GiB at TP=2, which does not fit beside half +# of the 511 GB checkpoint on a 288 GiB card. Offload at TP=2; keep them resident +# at TP=4 so it stays comparable with the validated concurrency 1-32 run. +if (( TP == 2 )); then + ENGRAM_CONFIG='{"cpu_offload":true}' +else + ENGRAM_CONFIG='{"cpu_offload":false}' +fi + NUM_SPEC_TOKENS=5 GRAPH_NUM_SEQS=$((2 * CONC)) if (( GRAPH_NUM_SEQS < 128 )); then @@ -79,6 +91,7 @@ VLLM_CMD=( --tokenizer-mode deepseek_v41 --tool-call-parser deepseek_v41 --enable-auto-tool-choice --reasoning-parser deepseek_v41 + --engram-config "$ENGRAM_CONFIG" # aiter, not aiter_triton_mxfp4_bf16: the plain name opens vLLM's full # priority list and the CK kernel at its head wins. CK quantizes # activations to FP8 internally and dispatches the a8w4 experts diff --git a/configs/amd-master.yaml b/configs/amd-master.yaml index ecc97d7be..cf673c3af 100644 --- a/configs/amd-master.yaml +++ b/configs/amd-master.yaml @@ -1476,6 +1476,6 @@ dsv41flash-fp4-mi355x-vllm-agentic-dspark: agentic-coding: - dram-utilization: 0.60 search-space: - # Follow upstream AMD Engram defaults; omit the CUDA-only config flag. + # The recipe sets engram cpu_offload per TP: off at TP=4, on at TP=2. - { tp: 4, kv-offloading: none, spec-decoding: mtp, conc-list: [1, 2, 4, 8, 16, 32, 64, 128] } - { tp: 2, kv-offloading: none, spec-decoding: mtp, conc-list: [1, 2, 4, 8, 16, 32, 64, 128] } diff --git a/perf-changelog.yaml b/perf-changelog.yaml index 9e1e293fa..2ca21763f 100644 --- a/perf-changelog.yaml +++ b/perf-changelog.yaml @@ -8461,10 +8461,12 @@ scenario-type: - agentic-coding description: - - "Add a TP=2 search space alongside TP=4 for MI355X DSv4.1-Flash AgentX, and extend both to concurrency 128. TP=2 is feasible now that the Engram tables live in host memory rather than costing 47.2 GiB of device memory per rank, which previously forced four GPUs per server." + - "Add a TP=2 search space alongside TP=4 for MI355X DSv4.1-Flash AgentX, and extend both to concurrency 128. TP=2 is feasible now that the Engram tables can live in host memory on ROCm rather than costing 47.2 GiB of device memory per rank, which previously forced four GPUs per server." + - "Set engram cpu_offload per TP in the recipe: on at TP=2, off at TP=4. vllm-project/vllm#57491 widened the two ROCm is_cuda() gates to is_cuda_alike(), so this image resolves an Engram config on gfx950 and offloads the tables to pinned host memory by default. TP=2 needs that, since the tables cost 94.4 GiB per rank at TP=2 against a 288 GiB card; TP=4 keeps them resident so it stays comparable with the validated concurrency 1-32 run." - "Move the image from vllm/vllm-openai-rocm:nightly-eed1f3d0c6043bd494424a22443ee198dd56f657 on the plain ROCm nightly channel to vllm/vllm-openai-rocm:nightly-rocm100-3df4ae153eb385e27b52f26c81f8edb9e20b9984 on the ROCm 10.0 nightly channel, published 2026-09-21T05:50:03Z with digest sha256:eccb72b74b8c04ce7406d9212e200b129f58e24be642795a857f755a94fdb3a1. kimik3-fp4-mi355x-vllm-agentic-mtp already runs that channel on this cluster. This sweep is the qualification for the new image; the previous green run is not evidence for it." - "Drop the fixed max_num_seqs of 128 and use vLLM's MI355X API-server default of 1024, since a fixed 128 caps in-flight sequences at the outer concurrency once the sweep reaches c128. The CUDA graph ceiling stays at 1024 through c64 and rises to 2048 at c128, covering twice the outer concurrency. Supersedes PR #3111." - - "为 MI355X DSv4.1-Flash AgentX 在 TP=4 之外新增 TP=2 搜索空间，并将两者的并发扩展到 128。Engram 表已驻留主机内存，不再占用每 rank 47.2 GiB 设备内存（此前这迫使每个服务器使用四张 GPU），因此 TP=2 可行。" + - "为 MI355X DSv4.1-Flash AgentX 在 TP=4 之外新增 TP=2 搜索空间，并将两者的并发扩展到 128。ROCm 上 Engram 表现在可以驻留主机内存，不再占用每 rank 47.2 GiB 设备内存（此前这迫使每个服务器使用四张 GPU），因此 TP=2 可行。" + - "在 recipe 中按 TP 设置 engram cpu_offload：TP=2 开启，TP=4 关闭。vllm-project/vllm#57491 将 ROCm 上的两处 is_cuda() 判断放宽为 is_cuda_alike()，因此该镜像在 gfx950 上会解析 Engram 配置并默认将表下放到锁页主机内存。TP=2 需要该行为：此时每 rank 的表需 94.4 GiB，而单卡仅 288 GiB；TP=4 则保持表常驻 GPU，以便与已验证的并发 1-32 运行保持可比。" - "镜像从 plain ROCm nightly 渠道的 vllm/vllm-openai-rocm:nightly-eed1f3d0c6043bd494424a22443ee198dd56f657 切换到 ROCm 10.0 nightly 渠道的 vllm/vllm-openai-rocm:nightly-rocm100-3df4ae153eb385e27b52f26c81f8edb9e20b9984，该镜像于 2026-09-21T05:50:03Z 发布，digest 为 sha256:eccb72b74b8c04ce7406d9212e200b129f58e24be642795a857f755a94fdb3a1。本集群上的 kimik3-fp4-mi355x-vllm-agentic-mtp 已在使用该渠道。本次 sweep 即为新镜像的验证；此前的绿色运行不能作为其证据。" - "移除固定的 max_num_seqs 128，改用 vLLM 在 MI355X API server 上的默认值 1024：扫描到 c128 时，固定 128 会把在途序列数限制在外层并发上。CUDA graph 上限在 c64 及以下保持 1024，c128 提升至 2048，以覆盖两倍外层并发。本 PR 取代 PR #3111。" pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/3326 From a9ab48e77f597c63722722984061a07019e1f496 Mon Sep 17 00:00:00 2001 From: Fangzhou Ai <31551580+Fangzhou-Ai@users.noreply.github.com> Date: Mon, 21 Sep 2026 07:41:02 +0000 Subject: [PATCH 08/12] config(dsv41flash): disable SWA bounded replay on MI355X MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Every TP=2 and TP=4 point of run 35567570539 died with HSA_STATUS_ERROR_MEMORY_FAULT, taking the worker and then the engine core down and aborting AgentX warmup. The fault follows the first launch of _pad_replayed_slots_kernel in all 16 server logs, which is the first prefix hit carrying a replay start. vllm-project/vllm#56227 added SWA bounded replay, default on, after the eed1f3d0 pin and before nightly-rocm100-3df4ae153. It pads the replayed tokens' slots in the prefix-cacheable groups, but the window clamp it relies on landed in the FlashInfer and FlashMLA kernels; the ROCm sparse SWA path only gained the replay_start kwarg, and consumes it on the decode path while the fault is in prefill. Pass --no-swa-bounded-replay until ROCm clamps too. Prefix caching itself stays on. 运行 35567570539 的所有 TP=2 与 TP=4 数据点都以 HSA_STATUS_ERROR_MEMORY_FAULT 崩溃，先后带崩 worker 与 engine core， 并中止 AgentX warmup。16 份 server 日志中，该故障均紧随 _pad_replayed_slots_kernel 的首次启动，即首个带 replay start 的前缀命中。 vllm-project/vllm#56227 在 eed1f3d0 与 nightly-rocm100-3df4ae153 之间 引入了默认开启的 SWA bounded replay。它会填充被重放 token 在可前缀缓存 分组中的 slot，但其依赖的窗口钳制只落在 FlashInfer 与 FlashMLA 内核中； ROCm 稀疏 SWA 路径仅新增了 replay_start 参数，且只在 decode 路径消费它， 而故障发生在 prefill。在 ROCm 同样实现钳制之前传入 --no-swa-bounded-replay。前缀缓存本身保持开启。 Co-authored-by: Cursor --- .../single_node/agentic/dsv41flash_fp4_mi355x_vllm_mtp.sh | 8 ++++++++ perf-changelog.yaml | 2 ++ 2 files changed, 10 insertions(+) diff --git a/benchmarks/single_node/agentic/dsv41flash_fp4_mi355x_vllm_mtp.sh b/benchmarks/single_node/agentic/dsv41flash_fp4_mi355x_vllm_mtp.sh index f4b5ba6a1..d9e1189e2 100755 --- a/benchmarks/single_node/agentic/dsv41flash_fp4_mi355x_vllm_mtp.sh +++ b/benchmarks/single_node/agentic/dsv41flash_fp4_mi355x_vllm_mtp.sh @@ -103,6 +103,14 @@ VLLM_CMD=( --max-model-len 1048576 --max-cudagraph-capture-size "$CAPTURE_SIZE" --max-num-batched-tokens 16384 + # vllm-project/vllm#56227 added SWA bounded replay (default on) after the + # eed1f3d0 pin and before this one. It pads the replayed tokens' slots in the + # prefix-cacheable groups, but the window clamp it relies on landed in the + # FlashInfer and FlashMLA kernels; the ROCm sparse SWA path only gained the + # replay_start kwarg. On gfx950 every TP=2 and TP=4 point of run 35567570539 + # died with HSA_STATUS_ERROR_MEMORY_FAULT at the first prefix hit carrying a + # replay start. Drop this once ROCm clamps too; prefix caching stays on. + --no-swa-bounded-replay --disable-uvicorn-access-log ) printf '%q ' "${VLLM_CMD[@]}" | tee "$RESULT_DIR/vllm_command.txt" diff --git a/perf-changelog.yaml b/perf-changelog.yaml index 2ca21763f..421b30ac0 100644 --- a/perf-changelog.yaml +++ b/perf-changelog.yaml @@ -8465,8 +8465,10 @@ - "Set engram cpu_offload per TP in the recipe: on at TP=2, off at TP=4. vllm-project/vllm#57491 widened the two ROCm is_cuda() gates to is_cuda_alike(), so this image resolves an Engram config on gfx950 and offloads the tables to pinned host memory by default. TP=2 needs that, since the tables cost 94.4 GiB per rank at TP=2 against a 288 GiB card; TP=4 keeps them resident so it stays comparable with the validated concurrency 1-32 run." - "Move the image from vllm/vllm-openai-rocm:nightly-eed1f3d0c6043bd494424a22443ee198dd56f657 on the plain ROCm nightly channel to vllm/vllm-openai-rocm:nightly-rocm100-3df4ae153eb385e27b52f26c81f8edb9e20b9984 on the ROCm 10.0 nightly channel, published 2026-09-21T05:50:03Z with digest sha256:eccb72b74b8c04ce7406d9212e200b129f58e24be642795a857f755a94fdb3a1. kimik3-fp4-mi355x-vllm-agentic-mtp already runs that channel on this cluster. This sweep is the qualification for the new image; the previous green run is not evidence for it." - "Drop the fixed max_num_seqs of 128 and use vLLM's MI355X API-server default of 1024, since a fixed 128 caps in-flight sequences at the outer concurrency once the sweep reaches c128. The CUDA graph ceiling stays at 1024 through c64 and rises to 2048 at c128, covering twice the outer concurrency. Supersedes PR #3111." + - "Pass --no-swa-bounded-replay. vllm-project/vllm#56227 added SWA bounded replay, default on, between the eed1f3d0 pin and this one. The window clamp it relies on landed in the FlashInfer and FlashMLA kernels, while the ROCm sparse SWA path only gained the replay_start kwarg, so on gfx950 the replayed tokens' slots are padded but the prefill kernels never stop at replay_start. Every TP=2 and TP=4 point of run 35567570539 died with HSA_STATUS_ERROR_MEMORY_FAULT at the first prefix hit carrying a replay start. Prefix caching itself stays on; drop this once ROCm clamps too." - "为 MI355X DSv4.1-Flash AgentX 在 TP=4 之外新增 TP=2 搜索空间，并将两者的并发扩展到 128。ROCm 上 Engram 表现在可以驻留主机内存，不再占用每 rank 47.2 GiB 设备内存（此前这迫使每个服务器使用四张 GPU），因此 TP=2 可行。" - "在 recipe 中按 TP 设置 engram cpu_offload：TP=2 开启，TP=4 关闭。vllm-project/vllm#57491 将 ROCm 上的两处 is_cuda() 判断放宽为 is_cuda_alike()，因此该镜像在 gfx950 上会解析 Engram 配置并默认将表下放到锁页主机内存。TP=2 需要该行为：此时每 rank 的表需 94.4 GiB，而单卡仅 288 GiB；TP=4 则保持表常驻 GPU，以便与已验证的并发 1-32 运行保持可比。" - "镜像从 plain ROCm nightly 渠道的 vllm/vllm-openai-rocm:nightly-eed1f3d0c6043bd494424a22443ee198dd56f657 切换到 ROCm 10.0 nightly 渠道的 vllm/vllm-openai-rocm:nightly-rocm100-3df4ae153eb385e27b52f26c81f8edb9e20b9984，该镜像于 2026-09-21T05:50:03Z 发布，digest 为 sha256:eccb72b74b8c04ce7406d9212e200b129f58e24be642795a857f755a94fdb3a1。本集群上的 kimik3-fp4-mi355x-vllm-agentic-mtp 已在使用该渠道。本次 sweep 即为新镜像的验证；此前的绿色运行不能作为其证据。" - "移除固定的 max_num_seqs 128，改用 vLLM 在 MI355X API server 上的默认值 1024：扫描到 c128 时，固定 128 会把在途序列数限制在外层并发上。CUDA graph 上限在 c64 及以下保持 1024，c128 提升至 2048，以覆盖两倍外层并发。本 PR 取代 PR #3111。" + - "传入 --no-swa-bounded-replay。vllm-project/vllm#56227 在 eed1f3d0 与本次固定的镜像之间引入了默认开启的 SWA bounded replay。它所依赖的窗口钳制只落在 FlashInfer 与 FlashMLA 内核中，而 ROCm 稀疏 SWA 路径仅新增了 replay_start 参数，因此在 gfx950 上被重放 token 的 slot 会被填充，但 prefill 内核并不会在 replay_start 处停下。运行 35567570539 的所有 TP=2 与 TP=4 数据点都在首次带 replay start 的前缀命中时以 HSA_STATUS_ERROR_MEMORY_FAULT 崩溃。前缀缓存本身保持开启；待 ROCm 同样实现钳制后即可移除。" pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/3326 From 42bb4f568739e8c0f024f1312faec6d53f446d12 Mon Sep 17 00:00:00 2001 From: Fangzhou Ai <31551580+Fangzhou-Ai@users.noreply.github.com> Date: Mon, 21 Sep 2026 19:13:34 +0000 Subject: [PATCH 09/12] config(dsv41flash): fix the MI355X KV split at high concurrency Concurrency 64 collapsed on both arms of run 35574132719 because the KV pool was too small to hold the AgentX working set. TP=2 fell to a 17.6% prefix cache hit rate, 187 s TTFT and 150 tok/s, against 94.8%, 1.3 s and 957 tok/s at c32. Two settings were spending device memory that the KV pool needed. The Engram tables stayed resident at TP=4. vllm-project/vllm#57491 widened the two ROCm is_cuda() gates to is_cuda_alike(), so gfx950 can now offload them to pinned host memory as every NVIDIA arm has since #2963. Measured here at TP=4 with 16384 batched tokens, resident tables leave 37.96 GiB of KV cache and 14.15x maximum concurrency at 1M context, against 84.54 GiB and 31.52x offloaded. The sparse-attention indexer and its companion per-rank buffers scale at roughly 4.4 MiB per batched token, so the upstream 16384 was the larger cost. Size it per TP instead, 4096 at TP=2 and 8192 at TP=4, and cap max_num_seqs at the shape graph capture already covers rather than the MI355X API-server default of 1024. TP=2 then keeps 79.34 GiB (39.44x) and TP=4 keeps 121.03 GiB (54.15x), against the B300 arm's 132.48 GiB (60.17x). B300 runs 8192 at TP=4 and the Blackwell TP=2 arms run 4096 (#3320, #3321). Sweep concurrency 64 and 128 to confirm the high end first. Concurrency 1-32 is backfilled once those land, since every point now runs the new memory split. Co-authored-by: Cursor Agent Signed-off-by: Fangzhou Ai --- .../agentic/dsv41flash_fp4_mi355x_vllm_mtp.sh | 49 ++++++++++++------- configs/amd-master.yaml | 10 ++-- docs/configuration-procedures.md | 5 +- docs/configuration-procedures_zh.md | 5 +- perf-changelog.yaml | 8 ++- 5 files changed, 51 insertions(+), 26 deletions(-) diff --git a/benchmarks/single_node/agentic/dsv41flash_fp4_mi355x_vllm_mtp.sh b/benchmarks/single_node/agentic/dsv41flash_fp4_mi355x_vllm_mtp.sh index d9e1189e2..995ab6a81 100755 --- a/benchmarks/single_node/agentic/dsv41flash_fp4_mi355x_vllm_mtp.sh +++ b/benchmarks/single_node/agentic/dsv41flash_fp4_mi355x_vllm_mtp.sh @@ -41,23 +41,17 @@ export VLLM_ENGINE_READY_TIMEOUT_S=3600 export VLLM_USE_RUST_FRONTEND=1 export PYTHONUNBUFFERED=1 -# Let vLLM select max_num_seqs; its API-server default is 1024 on MI355X. A -# fixed 128 caps in-flight sequences at the outer concurrency once the sweep -# reaches c128, leaving no headroom for AgentX subagent fan-out. -# Keep graph capture at the #3058 size through c64, then cover twice the outer -# concurrency at c128. -# vllm-project/vllm#57491 widened the two is_cuda() gates to is_cuda_alike(), -# so on gfx950 this image now resolves an Engram config and offloads the tables -# to pinned host memory unless an explicit value says otherwise. The tables cost -# 47.2 GiB per rank at TP=4, so 94.4 GiB at TP=2, which does not fit beside half -# of the 511 GB checkpoint on a 288 GiB card. Offload at TP=2; keep them resident -# at TP=4 so it stays comparable with the validated concurrency 1-32 run. -if (( TP == 2 )); then - ENGRAM_CONFIG='{"cpu_offload":true}' -else - ENGRAM_CONFIG='{"cpu_offload":false}' -fi +# vllm-project/vllm#57491 widened the two is_cuda() gates to is_cuda_alike(), so +# on gfx950 this image resolves an Engram config and an explicit value is needed +# rather than the VLLM_PLE_CPU_OFFLOAD default. Offload on every arm, as every +# NVIDIA DSv4.1-Flash arm has since #2963. Measured on gfx950 at TP=4, batched +# 16384: resident leaves 37.96 GiB of KV (14.15x max concurrency at 1M context), +# offloaded leaves 84.54 GiB (31.52x). +ENGRAM_CONFIG='{"cpu_offload":true}' +# Graph capture covers twice the outer concurrency, floored at the #3058 size of +# 128 sequences, across the 1+5 DSpark token shape. Twice leaves headroom for +# AgentX subagent fan-out above the outer concurrency. NUM_SPEC_TOKENS=5 GRAPH_NUM_SEQS=$((2 * CONC)) if (( GRAPH_NUM_SEQS < 128 )); then @@ -67,6 +61,26 @@ CAPTURE_SIZE=1 while (( CAPTURE_SIZE < GRAPH_NUM_SEQS * (1 + NUM_SPEC_TOKENS) && CAPTURE_SIZE < 2048 )); do CAPTURE_SIZE=$((CAPTURE_SIZE * 2)) done +# Cap in-flight sequences at the shape graph capture already covers, so the +# largest decode batch stays on a captured graph. This also drops max_num_seqs +# from the MI355X API-server default of 1024, which sized scheduler state for +# four times the sequences this sweep can actually run. +MAX_NUM_SEQS="$GRAPH_NUM_SEQS" + +# The sparse-attention indexer and its companion per-rank buffers scale with +# --max-num-batched-tokens at roughly 4.4 MiB per token, measured on gfx950. +# The upstream 16384 is what separates a KV pool that survives concurrency 64 +# from one that collapses: at TP=2 it leaves 20.06 GiB (7.48x), and TP=2 c64 of +# run 35574132719 fell to a 17.6% prefix cache hit rate, 187 s TTFT and 150 +# tok/s. At 4096 the same arm keeps 79.34 GiB (39.44x). TP=4 has twice the +# per-rank room, so 8192 is enough there: 121.03 GiB (54.15x), against the B300 +# arm's 132.48 GiB (60.17x). B300 runs 8192 at TP=4 and the Blackwell TP=2 arms +# run 4096 (#3320, #3321). +if (( TP == 2 )); then + BATCHED_TOKENS=4096 +else + BATCHED_TOKENS=8192 +fi # Use the runner-specific port assigned by launch_mi355x-amds.sh. export AIPERF_SERVER_URL="http://localhost:${PORT}" @@ -102,7 +116,8 @@ VLLM_CMD=( --speculative-config "$SPEC_CONFIG" --max-model-len 1048576 --max-cudagraph-capture-size "$CAPTURE_SIZE" - --max-num-batched-tokens 16384 + --max-num-seqs "$MAX_NUM_SEQS" + --max-num-batched-tokens "$BATCHED_TOKENS" # vllm-project/vllm#56227 added SWA bounded replay (default on) after the # eed1f3d0 pin and before this one. It pads the replayed tokens' slots in the # prefix-cacheable groups, but the window clamp it relies on landed in the diff --git a/configs/amd-master.yaml b/configs/amd-master.yaml index cf673c3af..9fa49ac14 100644 --- a/configs/amd-master.yaml +++ b/configs/amd-master.yaml @@ -1476,6 +1476,10 @@ dsv41flash-fp4-mi355x-vllm-agentic-dspark: agentic-coding: - dram-utilization: 0.60 search-space: - # The recipe sets engram cpu_offload per TP: off at TP=4, on at TP=2. - - { tp: 4, kv-offloading: none, spec-decoding: mtp, conc-list: [1, 2, 4, 8, 16, 32, 64, 128] } - - { tp: 2, kv-offloading: none, spec-decoding: mtp, conc-list: [1, 2, 4, 8, 16, 32, 64, 128] } + # The recipe offloads the engram tables on both arms and sizes + # --max-num-batched-tokens per TP, which together take TP=4 from 14.15x to + # 54.15x and TP=2 from 7.48x to 39.44x maximum concurrency at 1M context. + # Confirm the high end first; concurrency 1-32 is backfilled once c64 and + # c128 land, since every point now runs the new memory split. + - { tp: 4, kv-offloading: none, spec-decoding: mtp, conc-list: [64, 128] } + - { tp: 2, kv-offloading: none, spec-decoding: mtp, conc-list: [64, 128] } diff --git a/docs/configuration-procedures.md b/docs/configuration-procedures.md index 4c6abfccf..6b4f97f0c 100644 --- a/docs/configuration-procedures.md +++ b/docs/configuration-procedures.md @@ -519,14 +519,15 @@ A configuration is ready for sweep only when the executable files agree, the exa The draft `dsv41flash-fp4-mi355x-vllm-agentic-dspark` recipe extends [#2958](https://github.com/SemiAnalysisAI/InferenceX/pull/2958) to MI355X AgentX: TP4, concurrency 1–32, native five-token DSpark. Throughput uses the [committed golden AL](../golden_al_distribution/dsv41flash_dspark.yaml) of 3.51 for thinking on and five draft tokens, with synthetic rejection sampling and adaptive verification disabled. Accuracy evals retain real block rejection but, unlike the CUDA arms, also keep adaptive verification disabled: it trims verification requests on device, which the ROCm `DeepseekV4IndexerBackend` does not support, and the engine refused to start with it enabled ([run 34651830283](https://github.com/SemiAnalysisAI/InferenceX/actions/runs/34651830283)). FP4 describes the MXFP4 experts; the checkpoint also contains MXFP8 weights. -Follow the AMD overrides in the merged [upstream recipe #968](https://github.com/vllm-project/recipes/pull/968): `VLLM_ROCM_USE_AITER=1`, `VLLM_ROCM_USE_AITER_MOE=1`, and `--moe-backend aiter`. The generic AITER selector lets vLLM pick the CK a8w4 experts, matching the DSV4-Pro MI355X recipe. The recipe pins `semianalysis_cc_traces_weka_062126` (the unfiltered corpus) via `WEKA_LOADER_OVERRIDE`. KV stays GPU-resident; Engram follows upstream AMD defaults. Do not copy the NVIDIA `--engram-config` option: upstream currently rejects it on ROCm. The MI355X launcher uses the shared HF cache and mounts this model's repository at `/ix`, and exports `INFMAX_CONTAINER_WORKSPACE=/ix` so AgentX dependencies and outputs resolve inside that mount. +Follow the AMD overrides in the merged [upstream recipe #968](https://github.com/vllm-project/recipes/pull/968): `VLLM_ROCM_USE_AITER=1`, `VLLM_ROCM_USE_AITER_MOE=1`, and `--moe-backend aiter`. The generic AITER selector lets vLLM pick the CK a8w4 experts, matching the DSV4-Pro MI355X recipe. The recipe pins `semianalysis_cc_traces_weka_062126` (the unfiltered corpus) via `WEKA_LOADER_OVERRIDE`. KV stays GPU-resident. Engram stayed on GPU under the upstream AMD defaults until [vllm-project/vllm#57491](https://github.com/vllm-project/vllm/pull/57491) widened the two `is_cuda()` gates to `is_cuda_alike()`. From that commit on, ROCm resolves an `EngramConfig` and `cpu_offload` defaults to on through `VLLM_PLE_CPU_OFFLOAD`, so the recipe sets `--engram-config` to `{"cpu_offload":true}` explicitly on both arms rather than leaning on that default, matching every NVIDIA DSv4.1-Flash arm since [#2963](https://github.com/SemiAnalysisAI/InferenceX/pull/2963). Offloading is what makes the KV pool viable: measured on this cluster at TP=4 with 16384 batched tokens, resident tables leave 37.96 GiB of KV cache against 84.54 GiB offloaded. The recipe also sizes `--max-num-batched-tokens` per TP, 4096 at TP=2 and 8192 at TP=4, because the sparse-attention indexer and its companion per-rank buffers grow at roughly 4.4 MiB per batched token and the upstream 16384 starves the KV pool at high concurrency. Images built before that merge still reject the option on ROCm. The MI355X launcher uses the shared HF cache and mounts this model's repository at `/ix`, and exports `INFMAX_CONTAINER_WORKSPACE=/ix` so AgentX dependencies and outputs resolve inside that mount. **GPU validation:** [Run 34710937012](https://github.com/SemiAnalysisAI/InferenceX/actions/runs/34710937012) passed the exact pinned image for throughput at concurrency 1, 2, 4, 8, 16, and 32, plus eval-only concurrency 32. The recipe uses `vllm/vllm-openai-rocm:nightly-eed1f3d0c6043bd494424a22443ee198dd56f657` (digest `sha256:960228cf…`, published 2026-09-12). The earlier `deepseekv41-flash-0909` tag predates [vllm-project/vllm#56503](https://github.com/vllm-project/vllm/pull/56503), which moves the mHC delayed pre block off the eager Torch reference and onto AITER; the merged [upstream recipe #968](https://github.com/vllm-project/recipes/pull/968) pins the same nightly and records the complete InferenceX command. Follow the [AgentX procedure](./eval-agentx-procedures.md#7-run-agentx-fast-feedback-versus-canonical-evidence) for future runtime evidence; local generation and registry metadata alone are not GPU proof. ## DeepSeek-V4.1-Flash on MI300X and MI325X `dsv41flash-fp4-mi300x-vllm-agentic-dspark` and `dsv41flash-fp4-mi325x-vllm-agentic-dspark` -copy the validated MI355X vLLM arm onto gfx942, on the same ROCm nightly and with the same +copy the validated MI355X vLLM arm onto gfx942, on the `nightly-eed1f3d0` ROCm nightly the +MI355X arm used before it moved to the `nightly-rocm100` channel, and with the same AMD overrides (`VLLM_ROCM_USE_AITER=1`, `VLLM_ROCM_USE_AITER_MOE=1`, `VLLM_USE_BREAKABLE_CUDAGRAPH=1`, `--moe-backend aiter`, adaptive verification off). gfx942 is not in the upstream hardware table, and it has no FP4 MFMA: the plain `aiter` MoE diff --git a/docs/configuration-procedures_zh.md b/docs/configuration-procedures_zh.md index f877035f1..94b33dd92 100644 --- a/docs/configuration-procedures_zh.md +++ b/docs/configuration-procedures_zh.md @@ -465,14 +465,15 @@ python -m pytest utils/matrix_logic/ -v 草案配方 `dsv41flash-fp4-mi355x-vllm-agentic-dspark` 将 [#2958](https://github.com/SemiAnalysisAI/InferenceX/pull/2958) 扩展至 MI355X AgentX：TP4、并发 1–32、原生五 token DSpark。吞吐测试使用[已提交的黄金 AL](../golden_al_distribution/dsv41flash_dspark.yaml)：thinking 开启、五个草稿 token 对应 3.51，采用合成拒绝采样并关闭自适应验证。准确率 eval 保留真实块拒绝采样，但与 CUDA 分支不同，同样关闭自适应验证：它会在设备端裁剪验证请求，而 ROCm 的 `DeepseekV4IndexerBackend` 不支持该操作，启用后引擎拒绝启动（[运行 34651830283](https://github.com/SemiAnalysisAI/InferenceX/actions/runs/34651830283)）。FP4 表示 MXFP4 专家权重；检查点还包含 MXFP8 权重。 -遵循已合并的[上游配方 #968](https://github.com/vllm-project/recipes/pull/968) 中的 AMD 设置：`VLLM_ROCM_USE_AITER=1`、`VLLM_ROCM_USE_AITER_MOE=1` 和 `--moe-backend aiter`。通用 AITER 选择器允许 vLLM 选择 CK a8w4 专家内核，与 DSV4-Pro MI355X 配方一致。配方通过 `WEKA_LOADER_OVERRIDE` 固定使用完整语料 `semianalysis_cc_traces_weka_062126`。KV 驻留 GPU；Engram 沿用上游 AMD 默认设置。不要复制 NVIDIA 的 `--engram-config` 选项：上游目前在 ROCm 上拒绝该选项。MI355X launcher 使用共享 HF 缓存，并将此模型的仓库挂载至 `/ix`，同时导出 `INFMAX_CONTAINER_WORKSPACE=/ix`，确保 AgentX 依赖与输出路径位于该挂载中。 +遵循已合并的[上游配方 #968](https://github.com/vllm-project/recipes/pull/968) 中的 AMD 设置：`VLLM_ROCM_USE_AITER=1`、`VLLM_ROCM_USE_AITER_MOE=1` 和 `--moe-backend aiter`。通用 AITER 选择器允许 vLLM 选择 CK a8w4 专家内核，与 DSV4-Pro MI355X 配方一致。配方通过 `WEKA_LOADER_OVERRIDE` 固定使用完整语料 `semianalysis_cc_traces_weka_062126`。KV 驻留 GPU。在 [vllm-project/vllm#57491](https://github.com/vllm-project/vllm/pull/57491) 将两处 `is_cuda()` 判断放宽为 `is_cuda_alike()` 之前，Engram 按上游 AMD 默认设置常驻 GPU。自该提交起，ROCm 会解析 `EngramConfig`，且 `cpu_offload` 经由 `VLLM_PLE_CPU_OFFLOAD` 默认开启，因此配方在两个搜索空间上都显式将 `--engram-config` 设为 `{"cpu_offload":true}`，而不依赖该默认值，与自 [#2963](https://github.com/SemiAnalysisAI/InferenceX/pull/2963) 起所有 NVIDIA DSv4.1-Flash 搜索空间保持一致。下放是 KV 池可用的前提：本集群实测，TP=4、批量 16384 时，表常驻 GPU 仅剩 37.96 GiB KV cache，下放后为 84.54 GiB。配方同时按 TP 设置 `--max-num-batched-tokens`：TP=2 为 4096，TP=4 为 8192，因为稀疏注意力 indexer 及其配套的每 rank 缓冲区按每个批量 token 约 4.4 MiB 增长，上游的 16384 会在高并发下耗尽 KV 池。早于该合并的镜像在 ROCm 上仍会拒绝该选项。MI355X launcher 使用共享 HF 缓存，并将此模型的仓库挂载至 `/ix`，同时导出 `INFMAX_CONTAINER_WORKSPACE=/ix`，确保 AgentX 依赖与输出路径位于该挂载中。 **GPU 验证：** [运行 34710937012](https://github.com/SemiAnalysisAI/InferenceX/actions/runs/34710937012) 使用精确固定的镜像，通过了并发 1、2、4、8、16、32 的吞吐测试以及仅评测并发 32。配方使用 `vllm/vllm-openai-rocm:nightly-eed1f3d0c6043bd494424a22443ee198dd56f657`（摘要 `sha256:960228cf…`，发布于 2026-09-12）。较早的 `deepseekv41-flash-0909` 标签早于 [vllm-project/vllm#56503](https://github.com/vllm-project/vllm/pull/56503)，该 PR 将 mHC delayed pre 块从 eager Torch 参考实现切换到 AITER；已合并的[上游配方 #968](https://github.com/vllm-project/recipes/pull/968) 固定使用同一 nightly，并记录了完整的 InferenceX 命令。后续运行时证据请遵循 [AgentX 流程](./eval-agentx-procedures_zh.md)；仅有本地矩阵生成和镜像元数据不能证明 GPU 验证完成。 ## MI300X 与 MI325X 上的 DeepSeek-V4.1-Flash `dsv41flash-fp4-mi300x-vllm-agentic-dspark` 与 `dsv41flash-fp4-mi325x-vllm-agentic-dspark` -将已验证的 MI355X vLLM 配方复制到 gfx942，使用同一 ROCm nightly 与相同的 AMD 设置 +将已验证的 MI355X vLLM 配方复制到 gfx942，使用 MI355X 迁移到 `nightly-rocm100` 通道之前所用的 +`nightly-eed1f3d0` ROCm nightly，以及相同的 AMD 设置 （`VLLM_ROCM_USE_AITER=1`、`VLLM_ROCM_USE_AITER_MOE=1`、`VLLM_USE_BREAKABLE_CUDAGRAPH=1`、 `--moe-backend aiter`、关闭自适应验证）。gfx942 不在上游硬件表中，且没有 FP4 MFMA：通用的 `aiter` MoE 后端允许 vLLM 选择器跳过仅 gfx950 可用的 CK a8w4 专家内核；若启动时所有候选均被 diff --git a/perf-changelog.yaml b/perf-changelog.yaml index 421b30ac0..654303ed0 100644 --- a/perf-changelog.yaml +++ b/perf-changelog.yaml @@ -8462,12 +8462,16 @@ - agentic-coding description: - "Add a TP=2 search space alongside TP=4 for MI355X DSv4.1-Flash AgentX, and extend both to concurrency 128. TP=2 is feasible now that the Engram tables can live in host memory on ROCm rather than costing 47.2 GiB of device memory per rank, which previously forced four GPUs per server." - - "Set engram cpu_offload per TP in the recipe: on at TP=2, off at TP=4. vllm-project/vllm#57491 widened the two ROCm is_cuda() gates to is_cuda_alike(), so this image resolves an Engram config on gfx950 and offloads the tables to pinned host memory by default. TP=2 needs that, since the tables cost 94.4 GiB per rank at TP=2 against a 288 GiB card; TP=4 keeps them resident so it stays comparable with the validated concurrency 1-32 run." + - "Offload the Engram tables on both arms. vllm-project/vllm#57491 widened the two ROCm is_cuda() gates to is_cuda_alike(), so this image resolves an Engram config on gfx950 and the recipe now sets the value explicitly instead of relying on the default. Every NVIDIA DSv4.1-Flash arm has offloaded since #2963. Measured on this cluster at TP=4 with 16384 batched tokens, resident tables leave 37.96 GiB of KV cache and 14.15x maximum concurrency at 1M context, against 84.54 GiB and 31.52x when offloaded." + - "Size --max-num-batched-tokens per TP, 4096 at TP=2 and 8192 at TP=4, and cap --max-num-seqs at the shape graph capture already covers rather than the MI355X API-server default of 1024. The sparse-attention indexer and its companion per-rank buffers scale at roughly 4.4 MiB per batched token, so the upstream 16384 was spending the KV pool on scratch: TP=2 kept only 20.06 GiB, 7.48x maximum concurrency, and TP=2 c64 of run 35574132719 fell to a 17.6% prefix cache hit rate, 187 s TTFT and 150 tok/s, against 94.8%, 1.3 s and 957 tok/s at c32. With the new split TP=2 keeps 79.34 GiB (39.44x) and TP=4 keeps 121.03 GiB (54.15x), against the B300 arm's 132.48 GiB (60.17x). B300 runs 8192 at TP=4 and the Blackwell TP=2 arms run 4096 (#3320, #3321)." + - "Run concurrency 64 and 128 in this sweep to confirm the high end first. Concurrency 1-32 is backfilled once those land, since every point now runs the new memory split rather than the one the earlier 1-32 results were taken on." - "Move the image from vllm/vllm-openai-rocm:nightly-eed1f3d0c6043bd494424a22443ee198dd56f657 on the plain ROCm nightly channel to vllm/vllm-openai-rocm:nightly-rocm100-3df4ae153eb385e27b52f26c81f8edb9e20b9984 on the ROCm 10.0 nightly channel, published 2026-09-21T05:50:03Z with digest sha256:eccb72b74b8c04ce7406d9212e200b129f58e24be642795a857f755a94fdb3a1. kimik3-fp4-mi355x-vllm-agentic-mtp already runs that channel on this cluster. This sweep is the qualification for the new image; the previous green run is not evidence for it." - "Drop the fixed max_num_seqs of 128 and use vLLM's MI355X API-server default of 1024, since a fixed 128 caps in-flight sequences at the outer concurrency once the sweep reaches c128. The CUDA graph ceiling stays at 1024 through c64 and rises to 2048 at c128, covering twice the outer concurrency. Supersedes PR #3111." - "Pass --no-swa-bounded-replay. vllm-project/vllm#56227 added SWA bounded replay, default on, between the eed1f3d0 pin and this one. The window clamp it relies on landed in the FlashInfer and FlashMLA kernels, while the ROCm sparse SWA path only gained the replay_start kwarg, so on gfx950 the replayed tokens' slots are padded but the prefill kernels never stop at replay_start. Every TP=2 and TP=4 point of run 35567570539 died with HSA_STATUS_ERROR_MEMORY_FAULT at the first prefix hit carrying a replay start. Prefix caching itself stays on; drop this once ROCm clamps too." - "为 MI355X DSv4.1-Flash AgentX 在 TP=4 之外新增 TP=2 搜索空间，并将两者的并发扩展到 128。ROCm 上 Engram 表现在可以驻留主机内存，不再占用每 rank 47.2 GiB 设备内存（此前这迫使每个服务器使用四张 GPU），因此 TP=2 可行。" - - "在 recipe 中按 TP 设置 engram cpu_offload：TP=2 开启，TP=4 关闭。vllm-project/vllm#57491 将 ROCm 上的两处 is_cuda() 判断放宽为 is_cuda_alike()，因此该镜像在 gfx950 上会解析 Engram 配置并默认将表下放到锁页主机内存。TP=2 需要该行为：此时每 rank 的表需 94.4 GiB，而单卡仅 288 GiB；TP=4 则保持表常驻 GPU，以便与已验证的并发 1-32 运行保持可比。" + - "两个搜索空间均将 Engram 表下放到主机内存。vllm-project/vllm#57491 将 ROCm 上的两处 is_cuda() 判断放宽为 is_cuda_alike()，因此该镜像在 gfx950 上会解析 Engram 配置，recipe 现在显式设置该值而不再依赖默认行为。自 #2963 起所有 NVIDIA DSv4.1-Flash 搜索空间都已下放。本集群实测：TP=4、批量 16384 时，表常驻 GPU 仅剩 37.96 GiB KV cache（1M 上下文下最大并发 14.15x），下放后为 84.54 GiB（31.52x）。" + - "按 TP 设置 --max-num-batched-tokens：TP=2 为 4096，TP=4 为 8192；并将 --max-num-seqs 限制为 CUDA graph 已捕获的规模，而非 MI355X API server 默认的 1024。稀疏注意力 indexer 及其配套的每 rank 缓冲区按每个批量 token 约 4.4 MiB 增长，因此上游的 16384 实际上把 KV 池消耗在了暂存空间上：TP=2 仅剩 20.06 GiB（最大并发 7.48x），运行 35574132719 的 TP=2 c64 前缀缓存命中率跌至 17.6%，TTFT 187 秒、吞吐 150 tok/s，而 c32 为 94.8%、1.3 秒、957 tok/s。采用新的内存划分后，TP=2 保有 79.34 GiB（39.44x），TP=4 保有 121.03 GiB（54.15x），对比 B300 搜索空间的 132.48 GiB（60.17x）。B300 在 TP=4 使用 8192，Blackwell 的 TP=2 搜索空间使用 4096（#3320、#3321）。" + - "本次 sweep 只运行并发 64 与 128，优先确认高并发端。并发 1-32 将在其落地后补跑，因为现在所有数据点都采用新的内存划分，与此前 1-32 结果所基于的配置不同。" - "镜像从 plain ROCm nightly 渠道的 vllm/vllm-openai-rocm:nightly-eed1f3d0c6043bd494424a22443ee198dd56f657 切换到 ROCm 10.0 nightly 渠道的 vllm/vllm-openai-rocm:nightly-rocm100-3df4ae153eb385e27b52f26c81f8edb9e20b9984，该镜像于 2026-09-21T05:50:03Z 发布，digest 为 sha256:eccb72b74b8c04ce7406d9212e200b129f58e24be642795a857f755a94fdb3a1。本集群上的 kimik3-fp4-mi355x-vllm-agentic-mtp 已在使用该渠道。本次 sweep 即为新镜像的验证；此前的绿色运行不能作为其证据。" - "移除固定的 max_num_seqs 128，改用 vLLM 在 MI355X API server 上的默认值 1024：扫描到 c128 时，固定 128 会把在途序列数限制在外层并发上。CUDA graph 上限在 c64 及以下保持 1024，c128 提升至 2048，以覆盖两倍外层并发。本 PR 取代 PR #3111。" - "传入 --no-swa-bounded-replay。vllm-project/vllm#56227 在 eed1f3d0 与本次固定的镜像之间引入了默认开启的 SWA bounded replay。它所依赖的窗口钳制只落在 FlashInfer 与 FlashMLA 内核中，而 ROCm 稀疏 SWA 路径仅新增了 replay_start 参数，因此在 gfx950 上被重放 token 的 slot 会被填充，但 prefill 内核并不会在 replay_start 处停下。运行 35567570539 的所有 TP=2 与 TP=4 数据点都在首次带 replay start 的前缀命中时以 HSA_STATUS_ERROR_MEMORY_FAULT 崩溃。前缀缓存本身保持开启；待 ROCm 同样实现钳制后即可移除。" From 64dab293b5d24a8e2ce5115f96d8f29a3be5b6e7 Mon Sep 17 00:00:00 2001 From: Fangzhou Ai <31551580+Fangzhou-Ai@users.noreply.github.com> Date: Mon, 21 Sep 2026 20:17:00 +0000 Subject: [PATCH 10/12] config(dsv41flash): change the MI355X memory split only where KV ran short The previous commit applied one memory split per TP across every concurrency. Concurrency 1-32 never ran short of KV, so it paid for changes it did not need: offloaded Engram lookups go to pinned host memory over UVA, and a smaller prefill chunk costs TTFT. Scope each change to the points that were actually starved, which also leaves concurrency 1-32 and TP=4 c64 byte-identical to the settings run 35574132719 already measured on this image and flag set, so those points are combined with this sweep rather than re-run. Dividing measured KV tokens by concurrency gives the per-request budget and separates the failures cleanly. Every point that held had 232K or more; TP=2 c64 collapsed at 122K. TP=4 c1-c64 resident, 16384 14.83M tokens 232K and up TP=4 c128 offload, 8192 49.63M 388K TP=2 c1-c32 offload, 16384 7.84M 245K and up TP=2 c64 offload, 8192 24.68M 386K TP=2 c128 offload, 4096 33.34M 260K Probing the configurations directly also turned up two things. TP=2 c128 at 4096 with the API-server default of 1024 sequences faults during profiling with HSA_STATUS_ERROR_EXCEPTION at M=1024, N=129280, K=256, reproduced on two GPU pairs: DSpark verifies 1+5 tokens per sequence, so a decode batch of max_num_seqs needs six times that many token slots and 4096 leaves four. Cap max_num_seqs at the graph-capture shape wherever the chunk falls below that bound, which is TP=2 c128 alone. And the reason c128 never completed before is that at 16384 with capture at 2048, TP=2 c128 held 7.72 GiB, 3.02M tokens, 23.6K per request. Co-authored-by: Cursor Agent Signed-off-by: Fangzhou Ai --- .../agentic/dsv41flash_fp4_mi355x_vllm_mtp.sh | 66 ++++++++++++------- configs/amd-master.yaml | 10 +-- docs/configuration-procedures.md | 2 +- docs/configuration-procedures_zh.md | 2 +- perf-changelog.yaml | 14 ++-- 5 files changed, 59 insertions(+), 35 deletions(-) diff --git a/benchmarks/single_node/agentic/dsv41flash_fp4_mi355x_vllm_mtp.sh b/benchmarks/single_node/agentic/dsv41flash_fp4_mi355x_vllm_mtp.sh index 995ab6a81..e53578124 100755 --- a/benchmarks/single_node/agentic/dsv41flash_fp4_mi355x_vllm_mtp.sh +++ b/benchmarks/single_node/agentic/dsv41flash_fp4_mi355x_vllm_mtp.sh @@ -43,11 +43,20 @@ export PYTHONUNBUFFERED=1 # vllm-project/vllm#57491 widened the two is_cuda() gates to is_cuda_alike(), so # on gfx950 this image resolves an Engram config and an explicit value is needed -# rather than the VLLM_PLE_CPU_OFFLOAD default. Offload on every arm, as every -# NVIDIA DSv4.1-Flash arm has since #2963. Measured on gfx950 at TP=4, batched -# 16384: resident leaves 37.96 GiB of KV (14.15x max concurrency at 1M context), -# offloaded leaves 84.54 GiB (31.52x). -ENGRAM_CONFIG='{"cpu_offload":true}' +# rather than the VLLM_PLE_CPU_OFFLOAD default. +# +# The tables cost 47.2 GiB per rank at TP=4, so 94.4 GiB at TP=2, which does not +# fit beside half of the 511 GB checkpoint on a 288 GiB card: TP=2 always +# offloads. TP=4 keeps them resident, as the validated concurrency 1-32 run +# measured, because offloaded lookups go to pinned host memory over UVA and +# nothing below c128 is short of KV. Resident leaves 14.83M KV tokens, which is +# 232K per request at c64 and healthy, but only 116K at c128, under the 122K at +# which TP=2 c64 collapsed. Offloading lifts it to 33.06M, so 258K at c128. +if (( TP == 2 || CONC >= 128 )); then + ENGRAM_CONFIG='{"cpu_offload":true}' +else + ENGRAM_CONFIG='{"cpu_offload":false}' +fi # Graph capture covers twice the outer concurrency, floored at the #3058 size of # 128 sequences, across the 1+5 DSpark token shape. Twice leaves headroom for @@ -61,25 +70,36 @@ CAPTURE_SIZE=1 while (( CAPTURE_SIZE < GRAPH_NUM_SEQS * (1 + NUM_SPEC_TOKENS) && CAPTURE_SIZE < 2048 )); do CAPTURE_SIZE=$((CAPTURE_SIZE * 2)) done -# Cap in-flight sequences at the shape graph capture already covers, so the -# largest decode batch stays on a captured graph. This also drops max_num_seqs -# from the MI355X API-server default of 1024, which sized scheduler state for -# four times the sequences this sweep can actually run. -MAX_NUM_SEQS="$GRAPH_NUM_SEQS" # The sparse-attention indexer and its companion per-rank buffers scale with -# --max-num-batched-tokens at roughly 4.4 MiB per token, measured on gfx950. -# The upstream 16384 is what separates a KV pool that survives concurrency 64 -# from one that collapses: at TP=2 it leaves 20.06 GiB (7.48x), and TP=2 c64 of -# run 35574132719 fell to a 17.6% prefix cache hit rate, 187 s TTFT and 150 -# tok/s. At 4096 the same arm keeps 79.34 GiB (39.44x). TP=4 has twice the -# per-rank room, so 8192 is enough there: 121.03 GiB (54.15x), against the B300 -# arm's 132.48 GiB (60.17x). B300 runs 8192 at TP=4 and the Blackwell TP=2 arms -# run 4096 (#3320, #3321). -if (( TP == 2 )); then - BATCHED_TOKENS=4096 -else +# --max-num-batched-tokens at roughly 4.4 MiB per token, measured on gfx950, so +# a smaller prefill chunk buys KV room. TP=2 starts from half the per-rank space +# and is the arm that runs short: at the upstream 16384 it holds 7.84M KV +# tokens, 122K per request at c64, where run 35574132719 fell to a 17.6% prefix +# cache hit rate, 187 s TTFT and 150 tok/s against 94.8%, 1.3 s and 957 tok/s at +# c32. Every point that held had 232K per request or more, so keep the upstream +# chunk through c32 (245K at TP=2) and trade it away only above that. B300 runs +# 8192 at TP=4 and the Blackwell TP=2 arms run 4096 (#3320, #3321). +if (( CONC >= 128 )); then + (( TP == 2 )) && BATCHED_TOKENS=4096 || BATCHED_TOKENS=8192 +elif (( TP == 2 && CONC >= 64 )); then BATCHED_TOKENS=8192 +else + BATCHED_TOKENS=16384 +fi + +# DSpark verifies 1+5 tokens per sequence, so a decode batch of max_num_seqs +# needs six times that many token slots. The MI355X API-server default of 1024 +# sequences therefore wants 6144, and below that the engram projection faults +# during profiling: TP=2 c128 at 4096 leaves four slots per sequence and dies +# with HSA_STATUS_ERROR_EXCEPTION at M=1024, N=129280, K=256, reproduced on two +# separate GPU pairs. Where the chunk is that small, cap in-flight sequences at +# the shape graph capture already covers, which also keeps the largest decode +# batch on a captured graph. Leave the default alone everywhere else. +DEFAULT_MAX_NUM_SEQS=1024 +MAX_NUM_SEQS="" +if (( BATCHED_TOKENS < DEFAULT_MAX_NUM_SEQS * (1 + NUM_SPEC_TOKENS) )); then + MAX_NUM_SEQS="$GRAPH_NUM_SEQS" fi # Use the runner-specific port assigned by launch_mi355x-amds.sh. @@ -116,7 +136,6 @@ VLLM_CMD=( --speculative-config "$SPEC_CONFIG" --max-model-len 1048576 --max-cudagraph-capture-size "$CAPTURE_SIZE" - --max-num-seqs "$MAX_NUM_SEQS" --max-num-batched-tokens "$BATCHED_TOKENS" # vllm-project/vllm#56227 added SWA bounded replay (default on) after the # eed1f3d0 pin and before this one. It pads the replayed tokens' slots in the @@ -128,6 +147,9 @@ VLLM_CMD=( --no-swa-bounded-replay --disable-uvicorn-access-log ) +if [[ -n "$MAX_NUM_SEQS" ]]; then + VLLM_CMD+=(--max-num-seqs "$MAX_NUM_SEQS") +fi printf '%q ' "${VLLM_CMD[@]}" | tee "$RESULT_DIR/vllm_command.txt" printf '\n' | tee -a "$RESULT_DIR/vllm_command.txt" SERVER_PID="" diff --git a/configs/amd-master.yaml b/configs/amd-master.yaml index 9bd6d37ee..00e05da36 100644 --- a/configs/amd-master.yaml +++ b/configs/amd-master.yaml @@ -1474,10 +1474,10 @@ dsv41flash-fp4-mi355x-vllm-agentic-dspark: agentic-coding: - dram-utilization: 0.60 search-space: - # The recipe offloads the engram tables on both arms and sizes - # --max-num-batched-tokens per TP, which together take TP=4 from 14.15x to - # 54.15x and TP=2 from 7.48x to 39.44x maximum concurrency at 1M context. - # Confirm the high end first; concurrency 1-32 is backfilled once c64 and - # c128 land, since every point now runs the new memory split. + # The recipe only changes the memory split where a point ran short of KV: + # TP=4 offloads the engram tables at c128, and the prefill chunk shrinks at + # TP=2 c64, TP=4 c128 and TP=2 c128. Concurrency 1-32 and TP=4 c64 keep the + # settings the earlier run on this image already measured, so those points + # are combined with these rather than re-run. - { tp: 4, kv-offloading: none, spec-decoding: mtp, conc-list: [64, 128] } - { tp: 2, kv-offloading: none, spec-decoding: mtp, conc-list: [64, 128] } diff --git a/docs/configuration-procedures.md b/docs/configuration-procedures.md index 9834475b5..ba44a5455 100644 --- a/docs/configuration-procedures.md +++ b/docs/configuration-procedures.md @@ -553,7 +553,7 @@ A configuration is ready for sweep only when the executable files agree, the exa The draft `dsv41flash-fp4-mi355x-vllm-agentic-dspark` recipe extends [#2958](https://github.com/SemiAnalysisAI/InferenceX/pull/2958) to MI355X AgentX: TP4, concurrency 1–32, native five-token DSpark. Throughput uses the [committed golden AL](../golden_al_distribution/dsv41flash_dspark.yaml) of 3.51 for thinking on and five draft tokens, with synthetic rejection sampling and adaptive verification disabled. Accuracy evals retain real block rejection but, unlike the CUDA arms, also keep adaptive verification disabled: it trims verification requests on device, which the ROCm `DeepseekV4IndexerBackend` does not support, and the engine refused to start with it enabled ([run 34651830283](https://github.com/SemiAnalysisAI/InferenceX/actions/runs/34651830283)). FP4 describes the MXFP4 experts; the checkpoint also contains MXFP8 weights. -Follow the AMD overrides in the merged [upstream recipe #968](https://github.com/vllm-project/recipes/pull/968): `VLLM_ROCM_USE_AITER=1`, `VLLM_ROCM_USE_AITER_MOE=1`, and `--moe-backend aiter`. The generic AITER selector lets vLLM pick the CK a8w4 experts, matching the DSV4-Pro MI355X recipe. The recipe pins `semianalysis_cc_traces_weka_062126` (the unfiltered corpus) via `WEKA_LOADER_OVERRIDE`. KV stays GPU-resident. Engram stayed on GPU under the upstream AMD defaults until [vllm-project/vllm#57491](https://github.com/vllm-project/vllm/pull/57491) widened the two `is_cuda()` gates to `is_cuda_alike()`. From that commit on, ROCm resolves an `EngramConfig` and `cpu_offload` defaults to on through `VLLM_PLE_CPU_OFFLOAD`, so the recipe sets `--engram-config` to `{"cpu_offload":true}` explicitly on both arms rather than leaning on that default, matching every NVIDIA DSv4.1-Flash arm since [#2963](https://github.com/SemiAnalysisAI/InferenceX/pull/2963). Offloading is what makes the KV pool viable: measured on this cluster at TP=4 with 16384 batched tokens, resident tables leave 37.96 GiB of KV cache against 84.54 GiB offloaded. The recipe also sizes `--max-num-batched-tokens` per TP, 4096 at TP=2 and 8192 at TP=4, because the sparse-attention indexer and its companion per-rank buffers grow at roughly 4.4 MiB per batched token and the upstream 16384 starves the KV pool at high concurrency. Images built before that merge still reject the option on ROCm. The MI355X launcher uses the shared HF cache and mounts this model's repository at `/ix`, and exports `INFMAX_CONTAINER_WORKSPACE=/ix` so AgentX dependencies and outputs resolve inside that mount. +Follow the AMD overrides in the merged [upstream recipe #968](https://github.com/vllm-project/recipes/pull/968): `VLLM_ROCM_USE_AITER=1`, `VLLM_ROCM_USE_AITER_MOE=1`, and `--moe-backend aiter`. The generic AITER selector lets vLLM pick the CK a8w4 experts, matching the DSV4-Pro MI355X recipe. The recipe pins `semianalysis_cc_traces_weka_062126` (the unfiltered corpus) via `WEKA_LOADER_OVERRIDE`. KV stays GPU-resident. Engram stayed on GPU under the upstream AMD defaults until [vllm-project/vllm#57491](https://github.com/vllm-project/vllm/pull/57491) widened the two `is_cuda()` gates to `is_cuda_alike()`. From that commit on, ROCm resolves an `EngramConfig` and `cpu_offload` defaults to on through `VLLM_PLE_CPU_OFFLOAD`, so the recipe sets `--engram-config` explicitly rather than leaning on that default. TP=2 always offloads, since the tables need 94.4 GiB per rank there; TP=4 keeps them resident through concurrency 64, where the KV pool is not the constraint, and offloads only at 128. The recipe likewise trims `--max-num-batched-tokens` only above concurrency 32, to 8192 at TP=2 c64 and TP=4 c128 and to 4096 at TP=2 c128, because the sparse-attention indexer and its companion per-rank buffers grow at roughly 4.4 MiB per batched token. Where that chunk falls below six times the API-server default of 1024 sequences, `--max-num-seqs` is capped at the graph-capture shape: DSpark verifies 1+5 tokens per sequence, and at 4096 against 1024 sequences the engram projection faults during profiling. The rule in every case is to spend device memory on KV only at the concurrencies that ran short of it, leaving the validated low-concurrency settings alone. Images built before that merge still reject the option on ROCm. The MI355X launcher uses the shared HF cache and mounts this model's repository at `/ix`, and exports `INFMAX_CONTAINER_WORKSPACE=/ix` so AgentX dependencies and outputs resolve inside that mount. **GPU validation:** [Run 34710937012](https://github.com/SemiAnalysisAI/InferenceX/actions/runs/34710937012) passed the exact pinned image for throughput at concurrency 1, 2, 4, 8, 16, and 32, plus eval-only concurrency 32. The recipe uses `vllm/vllm-openai-rocm:nightly-eed1f3d0c6043bd494424a22443ee198dd56f657` (digest `sha256:960228cf…`, published 2026-09-12). The earlier `deepseekv41-flash-0909` tag predates [vllm-project/vllm#56503](https://github.com/vllm-project/vllm/pull/56503), which moves the mHC delayed pre block off the eager Torch reference and onto AITER; the merged [upstream recipe #968](https://github.com/vllm-project/recipes/pull/968) pins the same nightly and records the complete InferenceX command. Follow the [AgentX procedure](./eval-agentx-procedures.md#7-run-agentx-fast-feedback-versus-canonical-evidence) for future runtime evidence; local generation and registry metadata alone are not GPU proof. diff --git a/docs/configuration-procedures_zh.md b/docs/configuration-procedures_zh.md index b1d10e116..a2b81f526 100644 --- a/docs/configuration-procedures_zh.md +++ b/docs/configuration-procedures_zh.md @@ -494,7 +494,7 @@ python -m pytest utils/matrix_logic/ -v 草案配方 `dsv41flash-fp4-mi355x-vllm-agentic-dspark` 将 [#2958](https://github.com/SemiAnalysisAI/InferenceX/pull/2958) 扩展至 MI355X AgentX：TP4、并发 1–32、原生五 token DSpark。吞吐测试使用[已提交的黄金 AL](../golden_al_distribution/dsv41flash_dspark.yaml)：thinking 开启、五个草稿 token 对应 3.51，采用合成拒绝采样并关闭自适应验证。准确率 eval 保留真实块拒绝采样，但与 CUDA 分支不同，同样关闭自适应验证：它会在设备端裁剪验证请求，而 ROCm 的 `DeepseekV4IndexerBackend` 不支持该操作，启用后引擎拒绝启动（[运行 34651830283](https://github.com/SemiAnalysisAI/InferenceX/actions/runs/34651830283)）。FP4 表示 MXFP4 专家权重；检查点还包含 MXFP8 权重。 -遵循已合并的[上游配方 #968](https://github.com/vllm-project/recipes/pull/968) 中的 AMD 设置：`VLLM_ROCM_USE_AITER=1`、`VLLM_ROCM_USE_AITER_MOE=1` 和 `--moe-backend aiter`。通用 AITER 选择器允许 vLLM 选择 CK a8w4 专家内核，与 DSV4-Pro MI355X 配方一致。配方通过 `WEKA_LOADER_OVERRIDE` 固定使用完整语料 `semianalysis_cc_traces_weka_062126`。KV 驻留 GPU。在 [vllm-project/vllm#57491](https://github.com/vllm-project/vllm/pull/57491) 将两处 `is_cuda()` 判断放宽为 `is_cuda_alike()` 之前，Engram 按上游 AMD 默认设置常驻 GPU。自该提交起，ROCm 会解析 `EngramConfig`，且 `cpu_offload` 经由 `VLLM_PLE_CPU_OFFLOAD` 默认开启，因此配方在两个搜索空间上都显式将 `--engram-config` 设为 `{"cpu_offload":true}`，而不依赖该默认值，与自 [#2963](https://github.com/SemiAnalysisAI/InferenceX/pull/2963) 起所有 NVIDIA DSv4.1-Flash 搜索空间保持一致。下放是 KV 池可用的前提：本集群实测，TP=4、批量 16384 时，表常驻 GPU 仅剩 37.96 GiB KV cache，下放后为 84.54 GiB。配方同时按 TP 设置 `--max-num-batched-tokens`：TP=2 为 4096，TP=4 为 8192，因为稀疏注意力 indexer 及其配套的每 rank 缓冲区按每个批量 token 约 4.4 MiB 增长，上游的 16384 会在高并发下耗尽 KV 池。早于该合并的镜像在 ROCm 上仍会拒绝该选项。MI355X launcher 使用共享 HF 缓存，并将此模型的仓库挂载至 `/ix`，同时导出 `INFMAX_CONTAINER_WORKSPACE=/ix`，确保 AgentX 依赖与输出路径位于该挂载中。 +遵循已合并的[上游配方 #968](https://github.com/vllm-project/recipes/pull/968) 中的 AMD 设置：`VLLM_ROCM_USE_AITER=1`、`VLLM_ROCM_USE_AITER_MOE=1` 和 `--moe-backend aiter`。通用 AITER 选择器允许 vLLM 选择 CK a8w4 专家内核，与 DSV4-Pro MI355X 配方一致。配方通过 `WEKA_LOADER_OVERRIDE` 固定使用完整语料 `semianalysis_cc_traces_weka_062126`。KV 驻留 GPU。在 [vllm-project/vllm#57491](https://github.com/vllm-project/vllm/pull/57491) 将两处 `is_cuda()` 判断放宽为 `is_cuda_alike()` 之前，Engram 按上游 AMD 默认设置常驻 GPU。自该提交起，ROCm 会解析 `EngramConfig`，且 `cpu_offload` 经由 `VLLM_PLE_CPU_OFFLOAD` 默认开启，因此配方显式设置 `--engram-config`，而不依赖该默认值。TP=2 始终下放，因为此时表每 rank 需 94.4 GiB；TP=4 在并发 64 及以下保持常驻（此时 KV 池并非瓶颈），仅在并发 128 时下放。同样地，配方仅在并发高于 32 时调低 `--max-num-batched-tokens`：TP=2 c64 与 TP=4 c128 为 8192，TP=2 c128 为 4096，因为稀疏注意力 indexer 及其配套的每 rank 缓冲区按每个批量 token 约 4.4 MiB 增长。当该分块低于 API server 默认 1024 序列所需的六倍时，`--max-num-seqs` 会被限制为 CUDA graph 捕获的规模：DSpark 每序列验证 1+5 个 token，4096 对 1024 序列会使 engram 投影在 profiling 阶段崩溃。所有情况下的原则一致：只在确实出现 KV 不足的并发点上把设备内存让给 KV，保持低并发处已验证的设置不变。早于该合并的镜像在 ROCm 上仍会拒绝该选项。MI355X launcher 使用共享 HF 缓存，并将此模型的仓库挂载至 `/ix`，同时导出 `INFMAX_CONTAINER_WORKSPACE=/ix`，确保 AgentX 依赖与输出路径位于该挂载中。 **GPU 验证：** [运行 34710937012](https://github.com/SemiAnalysisAI/InferenceX/actions/runs/34710937012) 使用精确固定的镜像，通过了并发 1、2、4、8、16、32 的吞吐测试以及仅评测并发 32。配方使用 `vllm/vllm-openai-rocm:nightly-eed1f3d0c6043bd494424a22443ee198dd56f657`（摘要 `sha256:960228cf…`，发布于 2026-09-12）。较早的 `deepseekv41-flash-0909` 标签早于 [vllm-project/vllm#56503](https://github.com/vllm-project/vllm/pull/56503)，该 PR 将 mHC delayed pre 块从 eager Torch 参考实现切换到 AITER；已合并的[上游配方 #968](https://github.com/vllm-project/recipes/pull/968) 固定使用同一 nightly，并记录了完整的 InferenceX 命令。后续运行时证据请遵循 [AgentX 流程](./eval-agentx-procedures_zh.md)；仅有本地矩阵生成和镜像元数据不能证明 GPU 验证完成。 diff --git a/perf-changelog.yaml b/perf-changelog.yaml index b6de91ecf..3c73ab5aa 100644 --- a/perf-changelog.yaml +++ b/perf-changelog.yaml @@ -8490,16 +8490,18 @@ - agentic-coding description: - "Add a TP=2 search space alongside TP=4 for MI355X DSv4.1-Flash AgentX, and extend both to concurrency 128. TP=2 is feasible now that the Engram tables can live in host memory on ROCm rather than costing 47.2 GiB of device memory per rank, which previously forced four GPUs per server." - - "Offload the Engram tables on both arms. vllm-project/vllm#57491 widened the two ROCm is_cuda() gates to is_cuda_alike(), so this image resolves an Engram config on gfx950 and the recipe now sets the value explicitly instead of relying on the default. Every NVIDIA DSv4.1-Flash arm has offloaded since #2963. Measured on this cluster at TP=4 with 16384 batched tokens, resident tables leave 37.96 GiB of KV cache and 14.15x maximum concurrency at 1M context, against 84.54 GiB and 31.52x when offloaded." - - "Size --max-num-batched-tokens per TP, 4096 at TP=2 and 8192 at TP=4, and cap --max-num-seqs at the shape graph capture already covers rather than the MI355X API-server default of 1024. The sparse-attention indexer and its companion per-rank buffers scale at roughly 4.4 MiB per batched token, so the upstream 16384 was spending the KV pool on scratch: TP=2 kept only 20.06 GiB, 7.48x maximum concurrency, and TP=2 c64 of run 35574132719 fell to a 17.6% prefix cache hit rate, 187 s TTFT and 150 tok/s, against 94.8%, 1.3 s and 957 tok/s at c32. With the new split TP=2 keeps 79.34 GiB (39.44x) and TP=4 keeps 121.03 GiB (54.15x), against the B300 arm's 132.48 GiB (60.17x). B300 runs 8192 at TP=4 and the Blackwell TP=2 arms run 4096 (#3320, #3321)." - - "Run concurrency 64 and 128 in this sweep to confirm the high end first. Concurrency 1-32 is backfilled once those land, since every point now runs the new memory split rather than the one the earlier 1-32 results were taken on." + - "Offload the Engram tables only where the KV pool is short, not everywhere. vllm-project/vllm#57491 widened the two ROCm is_cuda() gates to is_cuda_alike(), so this image resolves an Engram config on gfx950 and the recipe sets the value explicitly instead of relying on the default. TP=2 always offloads: the tables cost 94.4 GiB per rank there against a 288 GiB card. TP=4 keeps them resident through c64, exactly as the validated concurrency 1-32 run measured, and offloads only at c128, where resident would leave 14.83M KV tokens, 116K per concurrent request, below the 122K at which TP=2 c64 collapsed. Offloaded lookups go to pinned host memory over UVA, so that cost is paid only where it buys KV room." + - "Trade prefill chunk size for KV room only above c32: --max-num-batched-tokens drops from the upstream 16384 to 8192 at TP=2 c64 and TP=4 c128, and to 4096 at TP=2 c128. The sparse-attention indexer and its companion per-rank buffers scale at roughly 4.4 MiB per batched token. KV tokens per concurrent request, which is what decides whether the prefix cache holds, rises from 116K to 388K at TP=4 c128, from 122K to 386K at TP=2 c64 and from 23.6K to 260K at TP=2 c128. Concurrency 1-32 keeps the upstream chunk untouched, where TP=2 measured 245K per request and TP=4 463K. B300 runs 8192 at TP=4 and the Blackwell TP=2 arms run 4096 (#3320, #3321)." + - "Cap --max-num-seqs at the graph-capture shape wherever the chunk falls below six times the MI355X API-server default of 1024 sequences, which is TP=2 c128 alone. DSpark verifies 1+5 tokens per sequence, so a decode batch of max_num_seqs needs six times that many token slots; at 4096 against 1024 sequences only four remain and the engram projection faults during profiling with HSA_STATUS_ERROR_EXCEPTION at M=1024, N=129280, K=256, reproduced on two separate GPU pairs. The same probes show why c128 never completed before: at the upstream 16384 with capture at 2048, TP=2 c128 held just 7.72 GiB, 3.02M KV tokens, 23.6K per request." + - "Run concurrency 64 and 128 in this sweep. Concurrency 1-32 is untouched by the changes above, so those points keep the settings the earlier run on this image and flag set already measured and are combined with these rather than re-run." - "Move the image from vllm/vllm-openai-rocm:nightly-eed1f3d0c6043bd494424a22443ee198dd56f657 on the plain ROCm nightly channel to vllm/vllm-openai-rocm:nightly-rocm100-3df4ae153eb385e27b52f26c81f8edb9e20b9984 on the ROCm 10.0 nightly channel, published 2026-09-21T05:50:03Z with digest sha256:eccb72b74b8c04ce7406d9212e200b129f58e24be642795a857f755a94fdb3a1. kimik3-fp4-mi355x-vllm-agentic-mtp already runs that channel on this cluster. This sweep is the qualification for the new image; the previous green run is not evidence for it." - "Drop the fixed max_num_seqs of 128 and use vLLM's MI355X API-server default of 1024, since a fixed 128 caps in-flight sequences at the outer concurrency once the sweep reaches c128. The CUDA graph ceiling stays at 1024 through c64 and rises to 2048 at c128, covering twice the outer concurrency. Supersedes PR #3111." - "Pass --no-swa-bounded-replay. vllm-project/vllm#56227 added SWA bounded replay, default on, between the eed1f3d0 pin and this one. The window clamp it relies on landed in the FlashInfer and FlashMLA kernels, while the ROCm sparse SWA path only gained the replay_start kwarg, so on gfx950 the replayed tokens' slots are padded but the prefill kernels never stop at replay_start. Every TP=2 and TP=4 point of run 35567570539 died with HSA_STATUS_ERROR_MEMORY_FAULT at the first prefix hit carrying a replay start. Prefix caching itself stays on; drop this once ROCm clamps too." - "为 MI355X DSv4.1-Flash AgentX 在 TP=4 之外新增 TP=2 搜索空间，并将两者的并发扩展到 128。ROCm 上 Engram 表现在可以驻留主机内存，不再占用每 rank 47.2 GiB 设备内存（此前这迫使每个服务器使用四张 GPU），因此 TP=2 可行。" - - "两个搜索空间均将 Engram 表下放到主机内存。vllm-project/vllm#57491 将 ROCm 上的两处 is_cuda() 判断放宽为 is_cuda_alike()，因此该镜像在 gfx950 上会解析 Engram 配置，recipe 现在显式设置该值而不再依赖默认行为。自 #2963 起所有 NVIDIA DSv4.1-Flash 搜索空间都已下放。本集群实测：TP=4、批量 16384 时，表常驻 GPU 仅剩 37.96 GiB KV cache（1M 上下文下最大并发 14.15x），下放后为 84.54 GiB（31.52x）。" - - "按 TP 设置 --max-num-batched-tokens：TP=2 为 4096，TP=4 为 8192；并将 --max-num-seqs 限制为 CUDA graph 已捕获的规模，而非 MI355X API server 默认的 1024。稀疏注意力 indexer 及其配套的每 rank 缓冲区按每个批量 token 约 4.4 MiB 增长，因此上游的 16384 实际上把 KV 池消耗在了暂存空间上：TP=2 仅剩 20.06 GiB（最大并发 7.48x），运行 35574132719 的 TP=2 c64 前缀缓存命中率跌至 17.6%，TTFT 187 秒、吞吐 150 tok/s，而 c32 为 94.8%、1.3 秒、957 tok/s。采用新的内存划分后，TP=2 保有 79.34 GiB（39.44x），TP=4 保有 121.03 GiB（54.15x），对比 B300 搜索空间的 132.48 GiB（60.17x）。B300 在 TP=4 使用 8192，Blackwell 的 TP=2 搜索空间使用 4096（#3320、#3321）。" - - "本次 sweep 只运行并发 64 与 128，优先确认高并发端。并发 1-32 将在其落地后补跑，因为现在所有数据点都采用新的内存划分，与此前 1-32 结果所基于的配置不同。" + - "仅在 KV 池吃紧处下放 Engram 表，而非全面下放。vllm-project/vllm#57491 将 ROCm 上的两处 is_cuda() 判断放宽为 is_cuda_alike()，因此该镜像在 gfx950 上会解析 Engram 配置，recipe 显式设置该值而不再依赖默认行为。TP=2 始终下放：此时表每 rank 需 94.4 GiB，而单卡仅 288 GiB。TP=4 在 c64 及以下保持常驻，与已验证的并发 1-32 运行完全一致，仅在 c128 下放：此时若保持常驻只有 14.83M KV token，即每并发请求 116K，低于 TP=2 c64 崩溃时的 122K。下放后的查表经 UVA 访问锁页主机内存，因此只在能换来 KV 空间处才付出该代价。" + - "仅在并发高于 c32 时用 prefill 分块大小换取 KV 空间：--max-num-batched-tokens 从上游的 16384 降至 TP=2 c64 与 TP=4 c128 的 8192，以及 TP=2 c128 的 4096。稀疏注意力 indexer 及其配套的每 rank 缓冲区按每个批量 token 约 4.4 MiB 增长。决定前缀缓存能否维持的每并发请求 KV token 数随之提升：TP=4 c128 从 116K 升至 388K，TP=2 c64 从 122K 升至 386K，TP=2 c128 从 23.6K 升至 260K。并发 1-32 完全保留上游分块大小，实测 TP=2 每请求 245K、TP=4 463K。B300 在 TP=4 使用 8192，Blackwell 的 TP=2 搜索空间使用 4096（#3320、#3321）。" + - "当分块大小低于 MI355X API server 默认 1024 序列所需的六倍时，将 --max-num-seqs 限制为 CUDA graph 捕获的规模，实际只影响 TP=2 c128。DSpark 每个序列验证 1+5 个 token，因此 max_num_seqs 规模的 decode 批次需要六倍的 token 槽位；4096 对 1024 序列仅剩四个槽位，profiling 阶段 engram 投影会在 M=1024、N=129280、K=256 处以 HSA_STATUS_ERROR_EXCEPTION 崩溃，并在两组不同 GPU 上复现。同一批探测也解释了此前 c128 始终无法完成的原因：在上游的 16384 且捕获上限为 2048 时，TP=2 c128 仅有 7.72 GiB、3.02M KV token，即每请求 23.6K。" + - "本次 sweep 运行并发 64 与 128。并发 1-32 不受上述改动影响，因此沿用此前在同一镜像与同一 flag 组合下已测得的结果，与本次结果合并，无需重跑。" - "镜像从 plain ROCm nightly 渠道的 vllm/vllm-openai-rocm:nightly-eed1f3d0c6043bd494424a22443ee198dd56f657 切换到 ROCm 10.0 nightly 渠道的 vllm/vllm-openai-rocm:nightly-rocm100-3df4ae153eb385e27b52f26c81f8edb9e20b9984，该镜像于 2026-09-21T05:50:03Z 发布，digest 为 sha256:eccb72b74b8c04ce7406d9212e200b129f58e24be642795a857f755a94fdb3a1。本集群上的 kimik3-fp4-mi355x-vllm-agentic-mtp 已在使用该渠道。本次 sweep 即为新镜像的验证；此前的绿色运行不能作为其证据。" - "移除固定的 max_num_seqs 128，改用 vLLM 在 MI355X API server 上的默认值 1024：扫描到 c128 时，固定 128 会把在途序列数限制在外层并发上。CUDA graph 上限在 c64 及以下保持 1024，c128 提升至 2048，以覆盖两倍外层并发。本 PR 取代 PR #3111。" - "传入 --no-swa-bounded-replay。vllm-project/vllm#56227 在 eed1f3d0 与本次固定的镜像之间引入了默认开启的 SWA bounded replay。它所依赖的窗口钳制只落在 FlashInfer 与 FlashMLA 内核中，而 ROCm 稀疏 SWA 路径仅新增了 replay_start 参数，因此在 gfx950 上被重放 token 的 slot 会被填充，但 prefill 内核并不会在 replay_start 处停下。运行 35567570539 的所有 TP=2 与 TP=4 数据点都在首次带 replay start 的前缀命中时以 HSA_STATUS_ERROR_MEMORY_FAULT 崩溃。前缀缓存本身保持开启；待 ROCm 同样实现钳制后即可移除。" From e913c5c6dab9639ff777089c313565dfb0a76951 Mon Sep 17 00:00:00 2001 From: Chun Fang Date: Tue, 22 Sep 2026 06:27:23 +0000 Subject: [PATCH 11/12] config(dsv41flash): measure the full MI355X concurrency range on the new pin --- configs/amd-master.yaml | 10 +++++----- perf-changelog.yaml | 4 ++-- 2 files changed, 7 insertions(+), 7 deletions(-) diff --git a/configs/amd-master.yaml b/configs/amd-master.yaml index a4d4ea094..bcf8c3787 100644 --- a/configs/amd-master.yaml +++ b/configs/amd-master.yaml @@ -1486,8 +1486,8 @@ dsv41flash-fp4-mi355x-vllm-agentic-dspark: search-space: # The recipe only changes the memory split where a point ran short of KV: # TP=4 offloads the engram tables at c128, and the prefill chunk shrinks at - # TP=2 c64, TP=4 c128 and TP=2 c128. Concurrency 1-32 and TP=4 c64 keep the - # settings the earlier run on this image already measured, so those points - # are combined with these rather than re-run. - - { tp: 4, kv-offloading: none, spec-decoding: mtp, conc-list: [64, 128] } - - { tp: 2, kv-offloading: none, spec-decoding: mtp, conc-list: [64, 128] } + # TP=2 c64, TP=4 c128 and TP=2 c128. Every point is measured here rather + # than combined from the earlier run: the image move leaves concurrency + # 1-32 measured only on the superseded nightly-eed1f3d0 pin. + - { tp: 4, kv-offloading: none, spec-decoding: mtp, conc-list: [1, 2, 4, 8, 16, 32, 64, 128] } + - { tp: 2, kv-offloading: none, spec-decoding: mtp, conc-list: [1, 2, 4, 8, 16, 32, 64, 128] } diff --git a/perf-changelog.yaml b/perf-changelog.yaml index 18a92005f..077f2838d 100644 --- a/perf-changelog.yaml +++ b/perf-changelog.yaml @@ -8493,7 +8493,7 @@ - "Offload the Engram tables only where the KV pool is short, not everywhere. vllm-project/vllm#57491 widened the two ROCm is_cuda() gates to is_cuda_alike(), so this image resolves an Engram config on gfx950 and the recipe sets the value explicitly instead of relying on the default. TP=2 always offloads: the tables cost 94.4 GiB per rank there against a 288 GiB card. TP=4 keeps them resident through c64, exactly as the validated concurrency 1-32 run measured, and offloads only at c128, where resident would leave 14.83M KV tokens, 116K per concurrent request, below the 122K at which TP=2 c64 collapsed. Offloaded lookups go to pinned host memory over UVA, so that cost is paid only where it buys KV room." - "Trade prefill chunk size for KV room only above c32: --max-num-batched-tokens drops from the upstream 16384 to 8192 at TP=2 c64 and TP=4 c128, and to 4096 at TP=2 c128. The sparse-attention indexer and its companion per-rank buffers scale at roughly 4.4 MiB per batched token. KV tokens per concurrent request, which is what decides whether the prefix cache holds, rises from 116K to 388K at TP=4 c128, from 122K to 386K at TP=2 c64 and from 23.6K to 260K at TP=2 c128. Concurrency 1-32 keeps the upstream chunk untouched, where TP=2 measured 245K per request and TP=4 463K. B300 runs 8192 at TP=4 and the Blackwell TP=2 arms run 4096 (#3320, #3321)." - "Cap --max-num-seqs at the graph-capture shape wherever the chunk falls below six times the MI355X API-server default of 1024 sequences, which is TP=2 c128 alone. DSpark verifies 1+5 tokens per sequence, so a decode batch of max_num_seqs needs six times that many token slots; at 4096 against 1024 sequences only four remain and the engram projection faults during profiling with HSA_STATUS_ERROR_EXCEPTION at M=1024, N=129280, K=256, reproduced on two separate GPU pairs. The same probes show why c128 never completed before: at the upstream 16384 with capture at 2048, TP=2 c128 held just 7.72 GiB, 3.02M KV tokens, 23.6K per request." - - "Run concurrency 64 and 128 in this sweep. Concurrency 1-32 is untouched by the changes above, so those points keep the settings the earlier run on this image and flag set already measured and are combined with these rather than re-run." + - "Sweep the full concurrency range 1 through 128 on both TP arms, sixteen points. The changes above leave concurrency 1-32 on the upstream settings, so the recipe there is unchanged, but its only measured points come from run 34710937012 on the superseded nightly-eed1f3d0 pin. Re-measuring them puts the whole MI355X AgentX curve on one image, and TP=2 has no measured point below c64 at all." - "Move the image from vllm/vllm-openai-rocm:nightly-eed1f3d0c6043bd494424a22443ee198dd56f657 on the plain ROCm nightly channel to vllm/vllm-openai-rocm:nightly-rocm100-3df4ae153eb385e27b52f26c81f8edb9e20b9984 on the ROCm 10.0 nightly channel, published 2026-09-21T05:50:03Z with digest sha256:eccb72b74b8c04ce7406d9212e200b129f58e24be642795a857f755a94fdb3a1. kimik3-fp4-mi355x-vllm-agentic-mtp already runs that channel on this cluster. This sweep is the qualification for the new image; the previous green run is not evidence for it." - "Drop the fixed max_num_seqs of 128 and use vLLM's MI355X API-server default of 1024, since a fixed 128 caps in-flight sequences at the outer concurrency once the sweep reaches c128. The CUDA graph ceiling stays at 1024 through c64 and rises to 2048 at c128, covering twice the outer concurrency. Supersedes PR #3111." - "Pass --no-swa-bounded-replay. vllm-project/vllm#56227 added SWA bounded replay, default on, between the eed1f3d0 pin and this one. The window clamp it relies on landed in the FlashInfer and FlashMLA kernels, while the ROCm sparse SWA path only gained the replay_start kwarg, so on gfx950 the replayed tokens' slots are padded but the prefill kernels never stop at replay_start. Every TP=2 and TP=4 point of run 35567570539 died with HSA_STATUS_ERROR_MEMORY_FAULT at the first prefix hit carrying a replay start. Prefix caching itself stays on; drop this once ROCm clamps too." @@ -8501,7 +8501,7 @@ - "仅在 KV 池吃紧处下放 Engram 表，而非全面下放。vllm-project/vllm#57491 将 ROCm 上的两处 is_cuda() 判断放宽为 is_cuda_alike()，因此该镜像在 gfx950 上会解析 Engram 配置，recipe 显式设置该值而不再依赖默认行为。TP=2 始终下放：此时表每 rank 需 94.4 GiB，而单卡仅 288 GiB。TP=4 在 c64 及以下保持常驻，与已验证的并发 1-32 运行完全一致，仅在 c128 下放：此时若保持常驻只有 14.83M KV token，即每并发请求 116K，低于 TP=2 c64 崩溃时的 122K。下放后的查表经 UVA 访问锁页主机内存，因此只在能换来 KV 空间处才付出该代价。" - "仅在并发高于 c32 时用 prefill 分块大小换取 KV 空间：--max-num-batched-tokens 从上游的 16384 降至 TP=2 c64 与 TP=4 c128 的 8192，以及 TP=2 c128 的 4096。稀疏注意力 indexer 及其配套的每 rank 缓冲区按每个批量 token 约 4.4 MiB 增长。决定前缀缓存能否维持的每并发请求 KV token 数随之提升：TP=4 c128 从 116K 升至 388K，TP=2 c64 从 122K 升至 386K，TP=2 c128 从 23.6K 升至 260K。并发 1-32 完全保留上游分块大小，实测 TP=2 每请求 245K、TP=4 463K。B300 在 TP=4 使用 8192，Blackwell 的 TP=2 搜索空间使用 4096（#3320、#3321）。" - "当分块大小低于 MI355X API server 默认 1024 序列所需的六倍时，将 --max-num-seqs 限制为 CUDA graph 捕获的规模，实际只影响 TP=2 c128。DSpark 每个序列验证 1+5 个 token，因此 max_num_seqs 规模的 decode 批次需要六倍的 token 槽位；4096 对 1024 序列仅剩四个槽位，profiling 阶段 engram 投影会在 M=1024、N=129280、K=256 处以 HSA_STATUS_ERROR_EXCEPTION 崩溃，并在两组不同 GPU 上复现。同一批探测也解释了此前 c128 始终无法完成的原因：在上游的 16384 且捕获上限为 2048 时，TP=2 c128 仅有 7.72 GiB、3.02M KV token，即每请求 23.6K。" - - "本次 sweep 运行并发 64 与 128。并发 1-32 不受上述改动影响，因此沿用此前在同一镜像与同一 flag 组合下已测得的结果，与本次结果合并，无需重跑。" + - "在两个 TP 分支上完整扫描并发 1 至 128，共十六个点。上述改动在并发 1-32 区间保留上游设置，recipe 本身没有变化，但该区间仅有的实测点来自运行 34710937012，使用的是已被取代的 nightly-eed1f3d0 镜像。重新测量这些点可使整条 MI355X AgentX 曲线落在同一镜像上；此外 TP=2 在 c64 以下完全没有实测点。" - "镜像从 plain ROCm nightly 渠道的 vllm/vllm-openai-rocm:nightly-eed1f3d0c6043bd494424a22443ee198dd56f657 切换到 ROCm 10.0 nightly 渠道的 vllm/vllm-openai-rocm:nightly-rocm100-3df4ae153eb385e27b52f26c81f8edb9e20b9984，该镜像于 2026-09-21T05:50:03Z 发布，digest 为 sha256:eccb72b74b8c04ce7406d9212e200b129f58e24be642795a857f755a94fdb3a1。本集群上的 kimik3-fp4-mi355x-vllm-agentic-mtp 已在使用该渠道。本次 sweep 即为新镜像的验证；此前的绿色运行不能作为其证据。" - "移除固定的 max_num_seqs 128，改用 vLLM 在 MI355X API server 上的默认值 1024：扫描到 c128 时，固定 128 会把在途序列数限制在外层并发上。CUDA graph 上限在 c64 及以下保持 1024，c128 提升至 2048，以覆盖两倍外层并发。本 PR 取代 PR #3111。" - "传入 --no-swa-bounded-replay。vllm-project/vllm#56227 在 eed1f3d0 与本次固定的镜像之间引入了默认开启的 SWA bounded replay。它所依赖的窗口钳制只落在 FlashInfer 与 FlashMLA 内核中，而 ROCm 稀疏 SWA 路径仅新增了 replay_start 参数，因此在 gfx950 上被重放 token 的 slot 会被填充，但 prefill 内核并不会在 replay_start 处停下。运行 35567570539 的所有 TP=2 与 TP=4 数据点都在首次带 replay start 的前缀命中时以 HSA_STATUS_ERROR_MEMORY_FAULT 崩溃。前缀缓存本身保持开启；待 ROCm 同样实现钳制后即可移除。" From da6d453cadffe8f0f3d6a6fa6b77bce8eed6f1a8 Mon Sep 17 00:00:00 2001 From: Chun Fang Date: Tue, 22 Sep 2026 06:39:14 +0000 Subject: [PATCH 12/12] docs(dsv41flash): retire the MI355X draft status and repoint GPU validation The MI355X section still described a draft recipe at TP4 concurrency 1-32 and cited run 34710937012 on nightly-eed1f3d0 as the pinned image. Both predate this branch: the arm now sweeps TP4 and TP2 at concurrency 1-128 on nightly-rocm100-3df4ae15. Lead with the current pin, keep the earlier run only as the superseded-pin note explaining why its points do not carry onto this image, and add the merged upstream recipe #1006 beside #968. MODELS.md and MODELS_zh.md carried a second DeepSeek-V4.1-Flash row holding the MI355X arm as pending GPU validation. Drop it; the active row already covers the model. --- MODELS.md | 1 - MODELS_zh.md | 1 - docs/configuration-procedures.md | 4 ++-- docs/configuration-procedures_zh.md | 4 ++-- 4 files changed, 4 insertions(+), 6 deletions(-) diff --git a/MODELS.md b/MODELS.md index f845453d5..d461f1657 100644 --- a/MODELS.md +++ b/MODELS.md @@ -159,7 +159,6 @@ Other offloading tiers, including NVMe KV cache offloading, are outside the init | Kimi-K3 | `kimik3` | 2026-07-27 ([#2391](https://github.com/SemiAnalysisAI/InferenceX/pull/2391)) | Agentic coding (DSpark may be disabled for better Pareto points) | Standalone non-DSpark A/B baseline (not required from day 0) | | GLM-5.2 | `glm5.2` | 2026-07-18 ([#2268](https://github.com/SemiAnalysisAI/InferenceX/pull/2268)) | Agentic coding (non-MTP points remain eligible under the Pareto policy; see Deprecation Notice) | | | MiniMax-M3 | `minimaxm3` | 2026-06-12 ([#1724](https://github.com/SemiAnalysisAI/InferenceX/pull/1724)) | Agentic coding | Single-turn 1k1k, Single-turn 8k1k (removed 2026-08-04, [#2493](https://github.com/SemiAnalysisAI/InferenceX/pull/2493)) | -| DeepSeek-V4.1-Flash | `dsv41flash` | Pending | Agentic coding on MI355X (draft; GPU validation pending) | — | | DeepSeek-V4-Pro | `dsv4` | 2026-04-24 ([#1130](https://github.com/SemiAnalysisAI/InferenceX/pull/1130)) | Agentic coding (non-spec-decode points remain eligible under the Pareto policy) | Single-turn 1k1k, Single-turn 8k1k (removed 2026-09-09, [#2921](https://github.com/SemiAnalysisAI/InferenceX/pull/2921)) | | GLM-5 / GLM-5.1 | `glm5`, `glm5.1` | 2026-03-06 ([#762](https://github.com/SemiAnalysisAI/InferenceX/pull/762)), with GLM-5.1 added 2026-04-21 ([#1098](https://github.com/SemiAnalysisAI/InferenceX/pull/1098)) | GLM-5.1 B200 TileRT only: 1k1k and 8k1k added 2026-08-09 ([#2533](https://github.com/SemiAnalysisAI/InferenceX/pull/2533)); Agentic coding added in [#2650](https://github.com/SemiAnalysisAI/InferenceX/pull/2650) | The earlier GLM-5 / GLM-5.1 recipes were retired 2026-07-18 ([#2276](https://github.com/SemiAnalysisAI/InferenceX/pull/2276)) | | MiniMax-M2.5/2.7 | `minimaxm2.5` | 2026-02-18 ([#755](https://github.com/SemiAnalysisAI/InferenceX/pull/755)) | None (retired 2026-06-20, [#1874](https://github.com/SemiAnalysisAI/InferenceX/pull/1874)) | Single-turn 1k1k, Single-turn 1k8k, Single-turn 8k1k | diff --git a/MODELS_zh.md b/MODELS_zh.md index 8215a8de9..dd52db9a2 100644 --- a/MODELS_zh.md +++ b/MODELS_zh.md @@ -159,7 +159,6 @@ InferenceX 支持 SGLang 和 vLLM 双方的维护者，并响应 AI 实验室和 | Kimi-K3 | `kimik3` | 2026-07-27 ([#2391](https://github.com/SemiAnalysisAI/InferenceX/pull/2391)) | 智能体编码（可关闭 DSpark 以获得更优帕累托点） | 独立非 DSpark A/B 基线（自第 0 天起即不要求） | | GLM-5.2 | `glm5.2` | 2026-07-18（[#2268](https://github.com/SemiAnalysisAI/InferenceX/pull/2268)） | 智能体编码（非 MTP 数据点仍可按帕累托策略参与发布；见弃用公告） | | | MiniMax-M3 | `minimaxm3` | 2026-06-12（[#1724](https://github.com/SemiAnalysisAI/InferenceX/pull/1724)） | 智能体编码 | 单轮 1k1k、单轮 8k1k（2026-08-04 移除，[#2493](https://github.com/SemiAnalysisAI/InferenceX/pull/2493)） | -| DeepSeek-V4.1-Flash | `dsv41flash` | 待验证 | MI355X 上的 Agentic coding（草案；等待 GPU 验证） | — | | DeepSeek-V4-Pro | `dsv4` | 2026-04-24（[#1130](https://github.com/SemiAnalysisAI/InferenceX/pull/1130)） | 智能体编码（非投机解码数据点仍可按帕累托策略参与发布） | 单轮 1k1k、单轮 8k1k（已于 2026-09-09 移除，[#2921](https://github.com/SemiAnalysisAI/InferenceX/pull/2921)） | | GLM-5 / GLM-5.1 | `glm5`、`glm5.1` | 2026-03-06（[#762](https://github.com/SemiAnalysisAI/InferenceX/pull/762)），GLM-5.1 于 2026-04-21 加入（[#1098](https://github.com/SemiAnalysisAI/InferenceX/pull/1098)） | 仅 GLM-5.1 B200 TileRT：1k1k 和 8k1k 于 2026-08-09 加入（[#2533](https://github.com/SemiAnalysisAI/InferenceX/pull/2533)）；智能体编码由 [#2650](https://github.com/SemiAnalysisAI/InferenceX/pull/2650) 加入 | 此前的 GLM-5 / GLM-5.1 配方于 2026-07-18 退役（[#2276](https://github.com/SemiAnalysisAI/InferenceX/pull/2276)） | | MiniMax-M2.5/2.7 | `minimaxm2.5` | 2026-02-18（[#755](https://github.com/SemiAnalysisAI/InferenceX/pull/755)） | 无（2026-06-20 退役，[#1874](https://github.com/SemiAnalysisAI/InferenceX/pull/1874)） | 单轮 1k1k、单轮 1k8k、单轮 8k1k | diff --git a/docs/configuration-procedures.md b/docs/configuration-procedures.md index 7a91fcef2..725e427a0 100644 --- a/docs/configuration-procedures.md +++ b/docs/configuration-procedures.md @@ -555,11 +555,11 @@ A configuration is ready for sweep only when the executable files agree, the exa ## DeepSeek-V4.1-Flash on MI355X -The draft `dsv41flash-fp4-mi355x-vllm-agentic-dspark` recipe extends [#2958](https://github.com/SemiAnalysisAI/InferenceX/pull/2958) to MI355X AgentX: TP4, concurrency 1–32, native five-token DSpark. Throughput uses the [committed golden AL](../golden_al_distribution/dsv41flash_dspark.yaml) of 3.51 for thinking on and five draft tokens, with synthetic rejection sampling and adaptive verification disabled. Accuracy evals retain real block rejection but, unlike the CUDA arms, also keep adaptive verification disabled: it trims verification requests on device, which the ROCm `DeepseekV4IndexerBackend` does not support, and the engine refused to start with it enabled ([run 34651830283](https://github.com/SemiAnalysisAI/InferenceX/actions/runs/34651830283)). FP4 describes the MXFP4 experts; the checkpoint also contains MXFP8 weights. +The `dsv41flash-fp4-mi355x-vllm-agentic-dspark` recipe extends [#2958](https://github.com/SemiAnalysisAI/InferenceX/pull/2958) to MI355X AgentX: TP4 and TP2, concurrency 1–128, native five-token DSpark. Throughput uses the [committed golden AL](../golden_al_distribution/dsv41flash_dspark.yaml) of 3.51 for thinking on and five draft tokens, with synthetic rejection sampling and adaptive verification disabled. Accuracy evals retain real block rejection but, unlike the CUDA arms, also keep adaptive verification disabled: it trims verification requests on device, which the ROCm `DeepseekV4IndexerBackend` does not support, and the engine refused to start with it enabled ([run 34651830283](https://github.com/SemiAnalysisAI/InferenceX/actions/runs/34651830283)). FP4 describes the MXFP4 experts; the checkpoint also contains MXFP8 weights. Follow the AMD overrides in the merged [upstream recipe #968](https://github.com/vllm-project/recipes/pull/968): `VLLM_ROCM_USE_AITER=1`, `VLLM_ROCM_USE_AITER_MOE=1`, and `--moe-backend aiter`. The generic AITER selector lets vLLM pick the CK a8w4 experts, matching the DSV4-Pro MI355X recipe. The recipe pins `semianalysis_cc_traces_weka_062126` (the unfiltered corpus) via `WEKA_LOADER_OVERRIDE`. KV stays GPU-resident. Engram stayed on GPU under the upstream AMD defaults until [vllm-project/vllm#57491](https://github.com/vllm-project/vllm/pull/57491) widened the two `is_cuda()` gates to `is_cuda_alike()`. From that commit on, ROCm resolves an `EngramConfig` and `cpu_offload` defaults to on through `VLLM_PLE_CPU_OFFLOAD`, so the recipe sets `--engram-config` explicitly rather than leaning on that default. TP=2 always offloads, since the tables need 94.4 GiB per rank there; TP=4 keeps them resident through concurrency 64, where the KV pool is not the constraint, and offloads only at 128. The recipe likewise trims `--max-num-batched-tokens` only above concurrency 32, to 8192 at TP=2 c64 and TP=4 c128 and to 4096 at TP=2 c128, because the sparse-attention indexer and its companion per-rank buffers grow at roughly 4.4 MiB per batched token. Where that chunk falls below six times the API-server default of 1024 sequences, `--max-num-seqs` is capped at the graph-capture shape: DSpark verifies 1+5 tokens per sequence, and at 4096 against 1024 sequences the engram projection faults during profiling. The rule in every case is to spend device memory on KV only at the concurrencies that ran short of it, leaving the validated low-concurrency settings alone. Images built before that merge still reject the option on ROCm. The MI355X launcher uses the shared HF cache and mounts this model's repository at `/ix`, and exports `INFMAX_CONTAINER_WORKSPACE=/ix` so AgentX dependencies and outputs resolve inside that mount. -**GPU validation:** [Run 34710937012](https://github.com/SemiAnalysisAI/InferenceX/actions/runs/34710937012) passed the exact pinned image for throughput at concurrency 1, 2, 4, 8, 16, and 32, plus eval-only concurrency 32. The recipe uses `vllm/vllm-openai-rocm:nightly-eed1f3d0c6043bd494424a22443ee198dd56f657` (digest `sha256:960228cf…`, published 2026-09-12). The earlier `deepseekv41-flash-0909` tag predates [vllm-project/vllm#56503](https://github.com/vllm-project/vllm/pull/56503), which moves the mHC delayed pre block off the eager Torch reference and onto AITER; the merged [upstream recipe #968](https://github.com/vllm-project/recipes/pull/968) pins the same nightly and records the complete InferenceX command. Follow the [AgentX procedure](./eval-agentx-procedures.md#7-run-agentx-fast-feedback-versus-canonical-evidence) for future runtime evidence; local generation and registry metadata alone are not GPU proof. +**GPU validation:** The recipe uses `vllm/vllm-openai-rocm:nightly-rocm100-3df4ae153eb385e27b52f26c81f8edb9e20b9984` (digest `sha256:eccb72b7…`, published 2026-09-21) on the ROCm 10.0 nightly channel that `kimik3-fp4-mi355x-vllm-agentic-mtp` already runs on this cluster. The sweep in [#3326](https://github.com/SemiAnalysisAI/InferenceX/pull/3326) qualifies that pin across TP4 and TP2 at concurrency 1–128, and is the only evidence for it: [run 34710937012](https://github.com/SemiAnalysisAI/InferenceX/actions/runs/34710937012) covered TP4 concurrency 1–32 plus eval-only concurrency 32 on the superseded `nightly-eed1f3d0c6043bd494424a22443ee198dd56f657`, so its points do not carry onto this image. The merged [upstream recipe #1006](https://github.com/vllm-project/recipes/pull/1006) documents the MI355X TP2 Engram offload and `--no-swa-bounded-replay`, and the merged [#968](https://github.com/vllm-project/recipes/pull/968) records the original AMD overrides and the complete InferenceX command. Follow the [AgentX procedure](./eval-agentx-procedures.md#7-run-agentx-fast-feedback-versus-canonical-evidence) for future runtime evidence; local generation and registry metadata alone are not GPU proof. ## DeepSeek-V4.1-Flash on MI300X and MI325X diff --git a/docs/configuration-procedures_zh.md b/docs/configuration-procedures_zh.md index 5bed0462c..612a997b6 100644 --- a/docs/configuration-procedures_zh.md +++ b/docs/configuration-procedures_zh.md @@ -496,11 +496,11 @@ python -m pytest utils/matrix_logic/ -v ## MI355X 上的 DeepSeek-V4.1-Flash -草案配方 `dsv41flash-fp4-mi355x-vllm-agentic-dspark` 将 [#2958](https://github.com/SemiAnalysisAI/InferenceX/pull/2958) 扩展至 MI355X AgentX：TP4、并发 1–32、原生五 token DSpark。吞吐测试使用[已提交的黄金 AL](../golden_al_distribution/dsv41flash_dspark.yaml)：thinking 开启、五个草稿 token 对应 3.51，采用合成拒绝采样并关闭自适应验证。准确率 eval 保留真实块拒绝采样，但与 CUDA 分支不同，同样关闭自适应验证：它会在设备端裁剪验证请求，而 ROCm 的 `DeepseekV4IndexerBackend` 不支持该操作，启用后引擎拒绝启动（[运行 34651830283](https://github.com/SemiAnalysisAI/InferenceX/actions/runs/34651830283)）。FP4 表示 MXFP4 专家权重；检查点还包含 MXFP8 权重。 +配方 `dsv41flash-fp4-mi355x-vllm-agentic-dspark` 将 [#2958](https://github.com/SemiAnalysisAI/InferenceX/pull/2958) 扩展至 MI355X AgentX：TP4 与 TP2、并发 1–128、原生五 token DSpark。吞吐测试使用[已提交的黄金 AL](../golden_al_distribution/dsv41flash_dspark.yaml)：thinking 开启、五个草稿 token 对应 3.51，采用合成拒绝采样并关闭自适应验证。准确率 eval 保留真实块拒绝采样，但与 CUDA 分支不同，同样关闭自适应验证：它会在设备端裁剪验证请求，而 ROCm 的 `DeepseekV4IndexerBackend` 不支持该操作，启用后引擎拒绝启动（[运行 34651830283](https://github.com/SemiAnalysisAI/InferenceX/actions/runs/34651830283)）。FP4 表示 MXFP4 专家权重；检查点还包含 MXFP8 权重。 遵循已合并的[上游配方 #968](https://github.com/vllm-project/recipes/pull/968) 中的 AMD 设置：`VLLM_ROCM_USE_AITER=1`、`VLLM_ROCM_USE_AITER_MOE=1` 和 `--moe-backend aiter`。通用 AITER 选择器允许 vLLM 选择 CK a8w4 专家内核，与 DSV4-Pro MI355X 配方一致。配方通过 `WEKA_LOADER_OVERRIDE` 固定使用完整语料 `semianalysis_cc_traces_weka_062126`。KV 驻留 GPU。在 [vllm-project/vllm#57491](https://github.com/vllm-project/vllm/pull/57491) 将两处 `is_cuda()` 判断放宽为 `is_cuda_alike()` 之前，Engram 按上游 AMD 默认设置常驻 GPU。自该提交起，ROCm 会解析 `EngramConfig`，且 `cpu_offload` 经由 `VLLM_PLE_CPU_OFFLOAD` 默认开启，因此配方显式设置 `--engram-config`，而不依赖该默认值。TP=2 始终下放，因为此时表每 rank 需 94.4 GiB；TP=4 在并发 64 及以下保持常驻（此时 KV 池并非瓶颈），仅在并发 128 时下放。同样地，配方仅在并发高于 32 时调低 `--max-num-batched-tokens`：TP=2 c64 与 TP=4 c128 为 8192，TP=2 c128 为 4096，因为稀疏注意力 indexer 及其配套的每 rank 缓冲区按每个批量 token 约 4.4 MiB 增长。当该分块低于 API server 默认 1024 序列所需的六倍时，`--max-num-seqs` 会被限制为 CUDA graph 捕获的规模：DSpark 每序列验证 1+5 个 token，4096 对 1024 序列会使 engram 投影在 profiling 阶段崩溃。所有情况下的原则一致：只在确实出现 KV 不足的并发点上把设备内存让给 KV，保持低并发处已验证的设置不变。早于该合并的镜像在 ROCm 上仍会拒绝该选项。MI355X launcher 使用共享 HF 缓存，并将此模型的仓库挂载至 `/ix`，同时导出 `INFMAX_CONTAINER_WORKSPACE=/ix`，确保 AgentX 依赖与输出路径位于该挂载中。 -**GPU 验证：** [运行 34710937012](https://github.com/SemiAnalysisAI/InferenceX/actions/runs/34710937012) 使用精确固定的镜像，通过了并发 1、2、4、8、16、32 的吞吐测试以及仅评测并发 32。配方使用 `vllm/vllm-openai-rocm:nightly-eed1f3d0c6043bd494424a22443ee198dd56f657`（摘要 `sha256:960228cf…`，发布于 2026-09-12）。较早的 `deepseekv41-flash-0909` 标签早于 [vllm-project/vllm#56503](https://github.com/vllm-project/vllm/pull/56503)，该 PR 将 mHC delayed pre 块从 eager Torch 参考实现切换到 AITER；已合并的[上游配方 #968](https://github.com/vllm-project/recipes/pull/968) 固定使用同一 nightly，并记录了完整的 InferenceX 命令。后续运行时证据请遵循 [AgentX 流程](./eval-agentx-procedures_zh.md)；仅有本地矩阵生成和镜像元数据不能证明 GPU 验证完成。 +**GPU 验证：** 配方使用 `vllm/vllm-openai-rocm:nightly-rocm100-3df4ae153eb385e27b52f26c81f8edb9e20b9984`（摘要 `sha256:eccb72b7…`，发布于 2026-09-21），位于 ROCm 10.0 nightly 渠道，本集群上的 `kimik3-fp4-mi355x-vllm-agentic-mtp` 已在使用该渠道。[#3326](https://github.com/SemiAnalysisAI/InferenceX/pull/3326) 的 sweep 在 TP4 与 TP2、并发 1–128 下验证该镜像，且是其唯一证据：[运行 34710937012](https://github.com/SemiAnalysisAI/InferenceX/actions/runs/34710937012) 覆盖的是已被取代的 `nightly-eed1f3d0c6043bd494424a22443ee198dd56f657` 上 TP4 并发 1–32 的吞吐测试与仅评测并发 32，其数据点不能沿用到本镜像。已合并的[上游配方 #1006](https://github.com/vllm-project/recipes/pull/1006) 记录了 MI355X 的 TP2 Engram 卸载与 `--no-swa-bounded-replay`，已合并的 [#968](https://github.com/vllm-project/recipes/pull/968) 则记录了最初的 AMD 设置与完整的 InferenceX 命令。后续运行时证据请遵循 [AgentX 流程](./eval-agentx-procedures_zh.md)；仅有本地矩阵生成和镜像元数据不能证明 GPU 验证完成。 ## MI300X 与 MI325X 上的 DeepSeek-V4.1-Flash