config(dsv41flash): sweep MI355X at TP=2 and TP=4 through concurrency 128 / MI355X DSv4.1-Flash 在 TP=2 与 TP=4 下扫描至并发 128

[Fangzhou-Ai]

Description

Adds a TP=2 search space next to TP=4 for dsv41flash-fp4-mi355x-vllm-agentic-dspark, and extends both rows to concurrency 128 (previously TP=4 only, c1–c32).

TP=2 is what this sweep is really for. The Engram tables cost 47.2 GiB of device memory per rank at TP=4, which is what forced four GPUs per server on this SKU; with those tables held in host memory the model fits on two, doubling the servers per node. Whether DSv4.1-Flash on MI355X is throughput-bound or interactivity-bound at agentic concurrencies depends on that choice, and today we have no TP=2 point to compare against.

That offload only became available on ROCm in vllm-project/vllm#57491, merged 2026-09-21T02:17:12Z, which widened the two is_cuda() gates to is_cuda_alike() so engram_config resolves on gfx950. The pinned nightly is six commits ahead of that merge, so it carries the change. Because cpu_offload now defaults to on via VLLM_PLE_CPU_OFFLOAD, the recipe sets it explicitly per TP rather than inheriting the default: {"cpu_offload":true} at TP=2, {"cpu_offload":false} at TP=4. TP=4 stays GPU-resident so its numbers remain comparable with the validated c1–c32 run; TP=2 needs the offload, since the tables cost 94.4 GiB per rank there against a 288 GiB card.

Supersedes #3111, which extended TP=4 to c128 and was closed in favour of this sweep. Its recipe change is carried here: benchmarks/single_node/agentic/dsv41flash_fp4_mi355x_vllm_mtp.sh drops the fixed max_num_seqs=128 in favour of the MI355X API-server default of 1024, because a fixed 128 caps in-flight sequences at the outer concurrency exactly when the sweep reaches c128, leaving no headroom for AgentX subagent fan-out. The CUDA graph ceiling now derives from max(2 * CONC, 128): 1024 through c64, 2048 at c128.

Image pinned. The entry runs on vllm/vllm-openai-rocm:nightly-rocm100-3df4ae153eb385e27b52f26c81f8edb9e20b9984, the ROCm 10.0 nightly published 2026-09-21T05:50:03Z. Docker Hub reports it active and linux/amd64 at digest sha256:eccb72b74b8c04ce7406d9212e200b129f58e24be642795a857f755a94fdb3a1, and a registry manifest request to registry-1.docker.io returns that same digest for the tag. This sweep is the qualification for the image; the previous green run on nightly-eed1f3d0 is not evidence for it.

The pin is on the nightly-rocm100 channel (ROCm 10.0). #3215 stayed on the plain ROCm nightly channel, but it scoped itself to an image-only bump on the channel already in place and did not reject nightly-rocm100 on technical grounds. kimik3-fp4-mi355x-vllm-agentic-mtp already runs that channel on this same cluster:mi355x-amds fleet, so ROCm 10.0 is proven for vLLM AgentX on MI355X. #3215 also edits this image: line, so whichever lands second rebases onto the other.

Generated matrix for the edited key — 16 points, TP=4 and TP=2 each across c1, 2, 4, 8, 16, 32, 64, 128:

uv run --no-project --exclude-newer PT12H --python 3.12 --with pydantic --with pyyaml \
  python -m infx.matrix.generate test-config \
  --config-files configs/amd-master.yaml \
  --runner-config configs/runners.yaml \
  --config-keys dsv41flash-fp4-mi355x-vllm-agentic-dspark

dsv41flash_tp4_conc{1,2,4,8,16,32,64,128}_kvnone_spec-mtp
dsv41flash_tp2_conc{1,2,4,8,16,32,64,128}_kvnone_spec-mtp

Each point resolves runner: cluster:mi355x-amds, precision: fp4, spec-decoding: mtp, kv-offloading: none, dram-utilization: 0.60, matching the existing TP=4 row. configs/amd-master.yaml, configs/runners.yaml and perf-changelog.yaml all parse. All 16 points carry the pinned image and no TBD placeholder survives anywhere in the changed files. infx.workflows.validate_perf_changelog --base-ref origin/main --head-ref HEAD passes, and the recipe script passes bash -n and git diff --check. GPU sweep and eval results are still pending.

AI model disclosure

• Model/version: Claude Opus 5 (runtime identifier claude-opus-5-thinking-high), via the Cursor agent.
• Role: drafted the config and changelog edits, moved the pin to the nightly-rocm100 channel, verified the published tag against Docker Hub and the registry, ran local YAML parsing, matrix generation and changelog validation, and wrote this description. All changes are reviewed by the submitter.

Related Issue

N/A

Type of Change

• Configuration change

Checklist

• I have completed the AI model disclosure and kept it current
• I have tested my changes locally
• I have updated documentation if necessary
• For every change that can affect benchmark performance and every recipe addition or modification, I have appended a new entry to the physical end of perf-changelog.yaml and have not edited historical entries
• Before merging via reuse, an authorized maintainer (OWNER/MEMBER/COLLABORATOR) has commented /use <run_id> (or the legacy /reuse-sweep-run) on this PR

中文

说明

为 dsv41flash-fp4-mi355x-vllm-agentic-dspark 在 TP=4 之外新增 TP=2 搜索空间，并将两者的并发扩展到 128（此前仅 TP=4，且并发为 c1–c32）。

本次 sweep 的重点是 TP=2。Engram 表在 TP=4 时每 rank 占用 47.2 GiB 设备内存，这正是该 SKU 此前必须每服务器四张 GPU 的原因；将这些表放在主机内存后，模型可在两张 GPU 上运行，每节点的服务器数量随之翻倍。DSv4.1-Flash 在 MI355X 上于 agentic 并发下究竟受吞吐还是受交互性限制，取决于这一选择，而目前我们没有可供对比的 TP=2 数据点。

该下放能力在 ROCm 上直到 vllm-project/vllm#57491（2026-09-21T02:17:12Z 合并）才可用：它将两处 is_cuda() 判断放宽为 is_cuda_alike()，使 engram_config 在 gfx950 上得以解析。所固定的 nightly 比该合并提交领先 6 个提交，因此包含该改动。由于 cpu_offload 现在经由 VLLM_PLE_CPU_OFFLOAD 默认开启，recipe 按 TP 显式设置该值，而不是沿用默认：TP=2 为 {"cpu_offload":true}，TP=4 为 {"cpu_offload":false}。TP=4 保持表常驻 GPU，使其数据与已验证的 c1–c32 运行保持可比；TP=2 则需要该下放，因为此时每 rank 的表需 94.4 GiB，而单卡仅 288 GiB。

本 PR 取代 #3111（该 PR 将 TP=4 扩展到 c128，已为本次 sweep 关闭），并承接其 recipe 改动：benchmarks/single_node/agentic/dsv41flash_fp4_mi355x_vllm_mtp.sh 移除固定的 max_num_seqs=128，改用 MI355X API server 默认值 1024。因为扫描到 c128 时，固定 128 恰好把在途序列数限制在外层并发上，使 AgentX 子代理扇出没有余量。CUDA graph 上限改由 max(2 * CONC, 128) 推导：c64 及以下为 1024，c128 为 2048。

镜像已固定。 该条目运行在 vllm/vllm-openai-rocm:nightly-rocm100-3df4ae153eb385e27b52f26c81f8edb9e20b9984 上，这是 2026-09-21T05:50:03Z 发布的 ROCm 10.0 nightly。Docker Hub 显示其状态为 active、架构为 linux/amd64，digest 为 sha256:eccb72b74b8c04ce7406d9212e200b129f58e24be642795a857f755a94fdb3a1；向 registry-1.docker.io 请求 manifest 也返回相同的 digest。本次 sweep 即为该镜像的验证；此前在 nightly-eed1f3d0 上的绿色运行不能作为其证据。

镜像使用 nightly-rocm100（ROCm 10.0）渠道。#3215 此前继续使用原有的 ROCm nightly 渠道，但它只是把自身范围限定为在既有渠道上做纯镜像更新，并未基于技术原因否定 nightly-rocm100。本 cluster:mi355x-amds 集群上的 kimik3-fp4-mi355x-vllm-agentic-mtp 已在使用该渠道，因此 ROCm 10.0 在 MI355X 上的 vLLM AgentX 场景已得到验证。#3215 同样修改了这一行 image:，因此后合并的一方需要 rebase。

针对所改 key 生成的矩阵共 16 个点，TP=4 与 TP=2 各覆盖 c1、2、4、8、16、32、64、128（命令与输出见上方英文部分）。每个点均解析为 runner: cluster:mi355x-amds、precision: fp4、spec-decoding: mtp、kv-offloading: none、dram-utilization: 0.60，与现有 TP=4 行一致。configs/amd-master.yaml、configs/runners.yaml 与 perf-changelog.yaml 均可正常解析。16 个点全部使用已固定的镜像，改动文件中不再残留任何 TBD 占位。infx.workflows.validate_perf_changelog --base-ref origin/main --head-ref HEAD 通过，recipe 脚本通过 bash -n 与 git diff --check。GPU sweep 与 eval 结果尚未完成。

AI 模型使用说明

• 模型/版本：Claude Opus 5（运行时标识 claude-opus-5-thinking-high），通过 Cursor agent 使用。
• 工作内容：起草配置与 changelog 改动，将镜像固定切换到 nightly-rocm100 渠道，对照 Docker Hub 与 registry 核验已发布的 tag，在本地运行 YAML 解析、矩阵生成与 changelog 校验，并撰写本说明。所有改动均由提交者复核。

关联 Issue

无

改动类型

• 配置改动

[github-actions]

Thanks for the contribution!

• Review: If this PR changes files owned by someone other than a repository admin or @SemiAnalysisAI/core, ask one eligible CODEOWNER to complete the latest PR_REVIEW_CHECKLIST.md before contacting a core maintainer on Slack. Follow the template exactly, including As a PR reviewer and CODEOWNER, I have reviewed this and have, so sign-off verification triggers.
• PR verification: Sweeps only run on labeled PRs. Add full-sweep-fail-fast (strongly recommended); use full-sweep-enabled only when matrix jobs should continue after a failure.
• After merging: PR authors must ensure all GitHub Actions jobs pass. Transient failures often pass on rerun; see how to rerun failed jobs.

中文

感谢你的贡献！

• **审阅：**如果 PR 修改的文件归属于仓库管理员及 @SemiAnalysisAI/core 之外的 CODEOWNER，请先联系一位有资格的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，再通过 Slack 联系核心维护者。必须严格遵循模板，并保留 As a PR reviewer and CODEOWNER, I have reviewed this and have，才能触发签核验证。
• **PR 验证：**扫描仅在带有标签的 PR 上运行。强烈建议添加 full-sweep-fail-fast；仅当需要矩阵任务在失败后继续运行时才使用 full-sweep-enabled。
• **合并后：**PR 作者必须确保所有 GitHub Actions 任务通过。临时性失败通常可以通过重新运行恢复；参见重新运行失败任务的说明。

[claude]

Claude finished @Fangzhou-Ai's task in 4m 2s —— View job

---

Review of PR #3326

• Gather context: read PR diff against origin/main and the full recipe script
• Check perf-changelog.yaml entry position and append-only rules
• Verify image accessibility and matrix generation
• Review recipe script logic (max_num_seqs, cudagraph ceiling, TP=2 path)
• Post review

Summary

One 🔴 BLOCKING issue, posted inline on configs/amd-master.yaml: the TP=2 row rests on the Engram tables living in host memory, but the recipe passes no --engram-config, the config comment still says to follow the upstream AMD (GPU-resident) Engram defaults, and docs/configuration-procedures.md:522 records that ROCm rejects that flag. With GPU-resident Engram, each TP=2 rank must hold ~255 GB of the 511 GB checkpoint inside a ~259 GB budget, leaving no room for the 32 GiB indexer buffer or the 1M-context KV pool, so TP=2 will OOM at engine init and full-sweep-fail-fast will also cancel the new TP=4 c64/c128 points. The new changelog entry repeats the same claim and needs the same correction.

Everything else checks out: the changelog entry is appended at the physical end of the file, the image uses a public Docker Hub reference, the recipe's max(2 * CONC, 128) capture-size derivation yields 1024 through c64 and 2048 at c128 as described, and git diff --check is clean. Not verified in this run: uv run (matrix generation, changelog validator), bash -n, outbound curl/WebFetch to Docker Hub, and the inferencemax-repos MCP server were all denied or unreachable in this sandbox, so the tag reachability and the vLLM ROCm Engram behaviour rely on the repo's own documentation.

中文

摘要

有一个 🔴 阻塞 问题，已在 configs/amd-master.yaml 内联标注：TP=2 行的依据是 Engram 表驻留主机内存，但 recipe 并未传入 --engram-config，配置注释仍写着沿用上游 AMD（GPU 驻留）Engram 默认值，且 docs/configuration-procedures.md:522 记录 ROCm 会拒绝该选项。在 Engram 驻留 GPU 的情况下，TP=2 每个 rank 需在约 259 GB 预算内容纳 511 GB checkpoint 的一半（约 255 GB），已无空间放置 32 GiB 的 indexer 缓冲区和 1M 上下文 KV 池，因此 TP=2 会在引擎初始化时 OOM，full-sweep-fail-fast 也会同时取消新增的 TP=4 c64/c128 点。新的 changelog 条目重复了同一说法，需一并修正。

其余检查通过：changelog 条目追加在文件末尾，镜像使用公开的 Docker Hub 引用，recipe 的 max(2 * CONC, 128) 捕获尺寸推导在 c64 及以下为 1024、c128 为 2048，git diff --check 无问题。本次未能验证：uv run（矩阵生成、changelog 校验）、bash -n、对 Docker Hub 的 curl/WebFetch 以及 inferencemax-repos MCP 服务器在此沙箱中均被拒绝或不可达，因此镜像可达性与 vLLM 在 ROCm 上的 Engram 行为依据仓库自身文档判断。

[claude]

🔴 BLOCKING: The TP=2 row is justified by Engram tables living in host memory, but nothing in this PR moves them there. The recipe (benchmarks/single_node/agentic/dsv41flash_fp4_mi355x_vllm_mtp.sh) passes no --engram-config, the comment two lines below this row still says "Follow upstream AMD Engram defaults; omit the CUDA-only config flag", and docs/configuration-procedures.md:522 records that upstream rejects --engram-config on ROCm, so Engram stays row-sharded on GPU (docs/configuration-procedures.md:537-539, dsv41flash_fp4_mi300x_mtp.sh:7-8).

Why it matters: With GPU-resident Engram, each TP=2 rank holds half of the 511 GB checkpoint (~255 GB), against a --gpu-memory-utilization 0.9 budget of ~259 GB on a 288 GB MI355X. That leaves ~4 GB for the indexer's [16384, 1M] fp8 logits buffer (32 GiB at this recipe's --max-num-batched-tokens 16384, per the MI300X notes), the 1M-context KV pool, and graph capture. The TP=2 points will OOM at engine init, and because the PR runs full-sweep-fail-fast, that failure also cancels the new TP=4 c64/c128 points. The same incorrect claim is repeated in the new perf-changelog.yaml entry (lines 8464 and 8467).

Fix: Either (a) actually add the host-memory Engram path to the recipe and cite evidence that the pinned nightly-rocm100 image accepts it on ROCm (the docs say it currently does not), or (b) drop the TP=2 row and rewrite this comment block and the changelog entry so they do not claim the tables live in host memory:

Suggested change

	# Both TP=4 rows extend to concurrency 128. Engram follows the upstream AMD
	# defaults and stays row-sharded on GPU; TP=2 is not feasible on this SKU
	# until a ROCm host-memory Engram path is available.
	dsv41flash-fp4-mi355x-vllm-agentic-dspark:
	# ROCm 10.0 nightly channel, shared with kimik3-fp4-mi355x-vllm-agentic-mtp.
	image: vllm/vllm-openai-rocm:nightly-rocm100-3df4ae153eb385e27b52f26c81f8edb9e20b9984
	model: deepseek-ai/DeepSeek-V4.1-Flash
	model-prefix: dsv41flash
	runner: cluster:mi355x-amds
	precision: fp4
	framework: vllm
	multinode: false
	scenarios:
	agentic-coding:
	- dram-utilization: 0.60
	search-space:
	# Follow upstream AMD Engram defaults; omit the CUDA-only config flag.
	- { tp: 4, kv-offloading: none, spec-decoding: mtp, conc-list: [1, 2, 4, 8, 16, 32, 64, 128] }

Fix this →

[github-actions]

View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=35574132719

View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=35574132719

[claude]

Beyond the inline finding (CI eval selection silently drops the new TP=2 sweep), I checked two other things and ruled them out: the removed --max-num-seqs flag decouples the CUDA-graph capture ceiling from the server's real cap, but vLLM's own default (1024) already exceeds every configured outer concurrency here, so it doesn't change observed behavior; and the new nightly-rocm100-* image tag, while unused elsewhere at this exact digest, is on the same channel already running for kimik3-fp4-mi355x-vllm-agentic-mtp on this cluster.

Extended reasoning...

A CONFIRMED finding is already queued for inline posting: configs/amd-master.yaml's new TP=2 search-space row shares (model, runner, framework, precision) with the existing TP=4 row, and infx/matrix/generate.py's ag_sn_groups keying (lines ~643-653) omits tp, so an evals-only CI run collapses both into one group and only one TP variant gets RUN_EVAL=True — silently dropping eval coverage for whichever TP the PR's own stated purpose was to add. That is a substantive, non-nit issue affecting the PR's stated goal, so approval is not appropriate here regardless of the other changes' simplicity. I verified two other candidate issues from the ruled-out list against the actual code rather than taking them on faith: sibling recipes like dsv4_fp4_mi355x_vllm_mtp.sh and kimik3_fp4_mi355x_atom_mtp.sh do tie an explicit --max-num-seqs to the same value used for graph-capture sizing, so this PR's script is a genuine departure from that pattern, but since vLLM's default of 1024 exceeds all configured concurrencies in this sweep (max outer CONC is 128), the practical cap is unaffected; and I confirmed via grep that kimik3-fp4-mi355x-vllm-agentic-mtp (configs/amd-master.yaml:580) already pins a nightly-rocm100-* tag, so the new channel is not unprecedented on this cluster even though the exact digest is new. These are worth surfacing briefly since they were concretely checked, not merely asserted, but they don't change the bottom line that the confirmed eval-selection gap needs human attention before merge.

[claude]

🔴 Running CI with the evals-only mode now silently drops the entire new TP=2 sweep for dsv41flash-fp4-mi355x-vllm-agentic-dspark, the whole point of this PR. infx/matrix/generate.py:645-653 groups agentic eval selection by (model, runner, framework, precision) only, omitting tp, so the tp:4 and tp:2 rows added here land in one group; both tie at conc=128 and Python's max() keeps the first (tp:4), leaving every tp:2 row's RUN_EVAL False. --evals-only then filters out all non-RUN_EVAL rows, removing all 8 tp:2 points, not just their eval. Fix: include tp (and any other search-space-distinguishing field) in the ag_sn_groups key so each topology keeps its own eval pick under evals-only.

Extended reasoning...

run-sweep.yml/e2e-tests.yml expose an evals-only label/flag that calls select_matrix_evals(mode='subset'). That path calls mark_eval_entries(rows, include_agentic=True) then filters rows to only RUN_EVAL=True. Inside mark_eval_entries, generate.py:643-650 builds ag_sn_groups[key] with key=(model, runner, framework, precision) for every agentic-coding single-node row, never including tp. Rows are appended in generation order: tp:4 conc 1..128 (from amd-master.yaml:1480) then tp:2 conc 1..128 (line 1481). Both tp:4-c128 and tp:2-c128 have conc_val=128. generate.py:653 does max(entries, key=lambda item: item[1]) — on a tie Python keeps the first-seen max, so the tp:4-c128 index wins and is the only one added to eval_indices for this group. Every tp:2 row gets RUN_EVAL=False. select_matrix_evals then does rows = [row for row in rows if row.get(RUN_EVAL, False)], dropping all 8 tp:2 rows from the generated matrix outright. The same key omission already collides for minimaxm3-fp4-mi355x-atom-agentic-mtp (tp:4/tp:2 rows at amd-master.yaml:648-649), showing the bug is real and pre-existing in…

Verification: normal. The agentic single-node eval-selection key at infx/matrix/generate.py:648-653 is (model, runner, framework, precision) with no tp, so the two rows this PR adds for dsv41flash-fp4-mi355x-vllm-agentic-dspark (amd-master.yaml:1480-1481: - { tp: 4, ... conc-list: [1..128] } and - { tp: 2, ... conc-list: [1..128] }) fall into ONE ag_sn_groups bucket. generate.py:655 selects a single…

[functionstackx]

I opened the matching upstream recipe for this: vllm-project/recipes#1006

It carries the three things this PR establishes for MI355X:

• single_node_tp.tp.mi355x: 2, making TP2 the documented default on this SKU.
• An exact-GPU mi355x override with --engram-config '{"cpu_offload":true}', with the 47.2 GiB per rank at TP4 / 94.4 GiB at TP2 arithmetic and the vllm-project/vllm#57491 gate written into the comment. The guide notes that TP4 users should pass {"cpu_offload":false} to keep the tables resident, so your TP4 row stays comparable with the validated concurrency 1-32 run.
• --no-swa-bounded-replay in the shared amd block, with the vllm-project/vllm#56227 window-clamp explanation and the HSA_STATUS_ERROR_MEMORY_FAULT symptom. I put it in the AMD block rather than an MI355X-only override because the gap is in the ROCm sparse SWA path generally; shout if you would rather scope it to gfx950.

I cited this PR and run 35574128850 as the validation evidence, and stated honestly that 14 of 16 agentic points plus the eval point were green with 2 still running at the time of writing. I will update the upstream PR once they land.

That should satisfy the recipe-link requirement on this PR's sign-off once it merges.