[Jasen2201]

Summary

Adds glm5.2-fp4-mi355x-atom-disagg-agentic-mtp: nine GLM-5.2 MXFP4 AgentX points on MI355X with ATOM prefill/decode disaggregation behind AToMesh, Mooncake RDMA from prefill to decode, MTP, and the prefill's prefix cache in a Mooncake Store. The current ATOM entry stops at concurrency 10 on one node; these points cover concurrency 12-128.

Every point has one CPP4 prefill worker (PP4 x TP1 on GPUs 0-3, layers split 20/20/20/18):

Points	Nodes	Decode	MTP	Mooncake Store 0.3.13 on the prefill node
C56, C72, C96, C112, C128	1 (1P1D)	TP4 + DCP4 on GPUs 4-7 of the prefill node	3	one shared pool, 2 x 768 GiB
C56	3 (1P2D)	2 x TP4 + DCP4, a node each	4	a pool per NIC, 4 x 349 GiB
C12, C16, C24	4 (1P3D)	3 x TP4, a node each	5	a pool per NIC, 4 x 349 GiB

• Mooncake Store. The masters and owners run as generic services on the prefill node. 1P1D, like ATOM's CI, has one master and two 768 GiB owners, one per NUMA node, shared by all four PP stages; each owner registers on rdma0-3, and the prefill sets MC_ENABLE_DEST_DEVICE_AFFINITY so each stage reaches an owner through the NIC of its own name on this rail-isolated fabric. 1P2D and 1P3D start a master and a 349 GiB owner for each prefill-stage NIC (rdma0-3), so each stage reads its own pool on its own rail. An owner registers its segment in 64 GiB regions, which the ionic driver maps only when they are all 2 MiB pages. host_setup therefore reserves HugeTLB pages through the cluster's hugepage_sanity.sh, as the DSv4 UMBP recipe does: the owners' pages plus 2% on top of the pages host services already hold (5656 on the mi355x-amds nodes seen so far). It reserves them only on the head node, where srtctl starts the prefill (it matches the node's addresses against SLURM_LAUNCH_NODE_IPADDR and fails if srun did not set it), checks each NUMA node's share, and teardown releases the pages. Owners set MC_STORE_USE_HUGEPAGE=1, which fails instead of falling back to 4 KiB pages, and run ulimit -l unlimited first: the remapped container root lacks host CAP_IPC_LOCK, so registering the segment counts against RLIMIT_MEMLOCK.
• DRAM budget. With [AgentX] budget decode servers' DRAM from decode GPUs / [AgentX] 按解码 GPU 为解码服务器分配 DRAM 预算 #4211, dram-utilization: 1.0 gives each 4-GPU server 1499 GB. The 1P1D decode shares the prefill's node, so its owners hold 1649 GB, 825 GB per NUMA node, within the prefill's and the decode's shares. The 1P2D and 1P3D decodes run on their own nodes, so those owners hold the prefill's 1499 GB.
• Why no LMCache. An LMCache prefill sometimes hung for good in the startup barrier after registering its 1 TiB pinned pool (3 of 8 starts on mi355x-amds); a Store prefill never has. On the same points the Store matched or beat it (see below).
• Model and memory. amd/GLM-5.2-MXFP4 has no staged copy on mi355x-amds; as the single-node GLM-5.2 points do, workers read it from each node's local hub cache (/var/lib/hf-hub-cache mounted at $HF_HOME/hub, the only path srtctl's pre-download sees), and srtctl fills the first node's cache before they start. Each point's host_setup re-runs the cluster's setup.sh, which drops the page cache, before reserving hugepages. The five 1P1D points share one recipe variant, so the sweep runs one GSM8K eval for them.
• srt-slurm patch. srtctl's ATOM backend passes the worker's GPU count as -tp, so the PP4 prefill would launch as -tp 4 --pipeline-parallel-size 4 (16 ranks on 4 GPUs). runners/srt-slurm/patches/604-atom-tp-from-pp.patch carries fix(atom): size -tp from the worker's GPUs over its PP and PCP stages NVIDIA/srt-slurm#604, which divides by the pipeline and prefill-context stages; it still applies on the v2.59.1 pin. Drop the patch once the pin includes that PR.
• Cache accounting. This PR does not change infx. The ATOM P/D cache rates come from the counters already in server_metrics_export.json, which fix(atom): read P/D cache counters from prefill and ingest CPU offload InferenceX-app#1533 reads at ingest: prefill endpoints only, with atom:prefix_cache_offload_tokens as the CPU tier.
• The image is rocm/atom-dev:nightly_202610131249, pinned by digest (ROCm/ATOM 6fd1bb1be).

Runs on mi355x-amds. Sweep 40578080205 ran every point and eval green on 45743f9f4; the recipe, its master config entry and the srt-slurm patch at the head are unchanged from that commit. Throughput per GPU, TTFT p50/p90:

Point	tok/s/GPU	TTFT p50 / p90	GSM8K (strict)
1P1D C56	31,450	1.18 / 2.38 s
1P1D C72	38,639	1.56 / 3.68 s
1P1D C96	44,039	2.64 / 7.45 s
1P1D C112	45,196	6.11 / 13.47 s
1P1D C128	40,032	15.31 / 36.43 s	0.966
1P2D C56	24,323	1.05 / 2.18 s	0.968
1P3D C12	3,958	0.79 / 1.57 s
1P3D C16	6,535	0.76 / 1.75 s	0.968
1P3D C24	10,213	0.80 / 1.81 s	0.967

Compared with the earlier layout (sweep 38674332505: 1P1D on per-NIC 4 x 349 GiB pools, LMCache for 1P2D and 1P3D), 1P1D C112 rose from 36,779 to 45,196 tok/s/GPU and its TTFT p50 fell from 19.3 to 6.1 s; 1P2D C56 rose from 22,379 to 24,323. This recipe's earlier sweep 39028510364 (on 1a153fd1) measured C112 at 39,245 and C96 at 42,884; every other point agrees with it within 2%. C112 is now within 2% of the 46.1K that ATOM's CI reaches with the same 1536 GiB pool on another cluster.

AI model disclosure

• Model/version: claude-opus-5-5 (Claude Opus 5.5), running in Claude Code.
• Role: traced the InferenceX launcher and srt-slurm's ATOM backend, and wrote the recipe, the master config entry, the perf-changelog entry and the srt-slurm patch (fix(atom): size -tp from the worker's GPUs over its PP and PCP stages NVIDIA/srt-slurm#604). It also rendered the bound recipes and srtctl mock launches, revised the recipe after a sweep-readiness review (shared hub cache, Store hugepage sizing, one Store variant), analysed the sweeps' LMCache startup hangs, ran the test runs above, moved every point to the Store with the [AgentX] budget decode servers' DRAM from decode GPUs / [AgentX] 按解码 GPU 为解码服务器分配 DRAM 预算 #4211 DRAM budget, and gave 1P1D ATOM CI's shared Store pool, and removed the Fix ATOMesh KV-cache counters / 修复 ATOMesh KV-cache 计数 #4397 cache-counter tooling after Fix ATOMesh KV-cache counters / 修复 ATOMesh KV-cache 计数 #4397 was closed in favor of fix(atom): read P/D cache counters from prefill and ingest CPU offload InferenceX-app#1533.
• Delegated agent: a Claude Code Explore subagent surveyed srt-slurm's ATOM backend and service placement. The runtime did not expose its model identifier, so it could not be verified.

PR checklist and change type

Type of Change

• Bug fix
• New feature
• Configuration change
• Documentation update
• Other (please describe)

Checklist

• I have completed the AI model disclosure and kept it current
• I have tested my changes locally
• I have updated documentation if necessary
• For every change that can affect benchmark performance and every recipe addition or modification, I have appended a new entry to the physical end of inferencex-e2e/perf-changelog.yaml and have not edited historical entries
• If this PR can affect benchmark performance or adds or modifies a recipe, it carries exactly one primary sweep label (a maintainer applies it on fork PRs): full-sweep-fail-fast (recommended), full-sweep-enabled, or non-canary-full-sweep-enabled. Optional modifiers all-evals, evals-only, and agentx-fast require a primary label; the last two block reuse while applied.
• Before merging via reuse, an authorized maintainer (OWNER/MEMBER/COLLABORATOR) has commented /use <run_id> (or the legacy /reuse-sweep-run) on this PR. Do this only once there is a final full sweep that is all green with evals passing, since after this comment the primary sweep label will no longer automatically kick off new sweeps. Remove and re-add the primary sweep label to force a new sweep.

中文

概要

新增 glm5.2-fp4-mi355x-atom-disagg-agentic-mtp：在 MI355X 上运行 GLM-5.2 MXFP4 的九个 AgentX 测试点，采用 ATOM 预填充/解码分离式推理，由 AToMesh 路由，通过 Mooncake RDMA 把 KV 从预填充传到解码，开启 MTP，并把预填充的前缀缓存放在 Mooncake Store 中。现有的 ATOM 配置在单节点上最高只到并发 10；这些测试点覆盖并发 12-128。

每个测试点都使用一个 CPP4 预填充 worker（GPU 0-3 上的 PP4 x TP1，层划分为 20/20/20/18）：

测试点	节点数	解码	MTP	预填充节点上的 Mooncake Store 0.3.13
C56、C72、C96、C112、C128	1（1P1D）	TP4 + DCP4，使用预填充节点的 GPU 4-7	3	一个共享池，2 x 768 GiB
C56	3（1P2D）	2 x TP4 + DCP4，各占一个节点	4	每块网卡一个池，4 x 349 GiB
C12、C16、C24	4（1P3D）	3 x TP4，各占一个节点	5	每块网卡一个池，4 x 349 GiB

• Mooncake Store。 master 和 owner 均作为 generic service 运行在预填充节点上。1P1D 与 ATOM CI 一致，使用一个 master 和两个 768 GiB owner（每个 NUMA 节点一个），由四个 PP stage 共享；每个 owner 注册在 rdma0-3 上，预填充设置 MC_ENABLE_DEST_DEVICE_AFFINITY，使每个 stage 在这套按 rail 隔离的网络中通过同名网卡访问 owner。1P2D 和 1P3D 为每个预填充 stage 的网卡（rdma0-3）各启动一个 master 和一个 349 GiB owner，使每个 stage 在自己的 rail 上读取自己的池。owner 以 64 GiB 为单位注册其 segment，而 ionic 驱动只有在这些区域全部由 2 MiB 页组成时才能映射。因此 host_setup 参照 DSv4 UMBP 配方，通过集群的 hugepage_sanity.sh 预留 HugeTLB 大页：在主机服务已占用的大页（目前所见的 mi355x-amds 节点上为 5656 页）之上，预留 owner 所需的页数并多留 2%。只在 srtctl 启动预填充的 head 节点上预留（用 SLURM_LAUNCH_NODE_IPADDR 匹配本机地址，srun 未设置该变量时直接失败），随后检查每个 NUMA 节点上的份额，teardown 时再释放。owner 设置了 MC_STORE_USE_HUGEPAGE=1，大页不足时直接失败，不会退回 4 KiB 页；并且先执行 ulimit -l unlimited：重映射的容器 root 没有主机的 CAP_IPC_LOCK，注册该段会计入 RLIMIT_MEMLOCK。
• 内存预算。 有了 [AgentX] budget decode servers' DRAM from decode GPUs / [AgentX] 按解码 GPU 为解码服务器分配 DRAM 预算 #4211，dram-utilization: 1.0 为每个 4 卡服务器提供 1499 GB。1P1D 的解码与预填充同节点，因此其 owner 共占 1649 GB，每个 NUMA 节点 825 GB，落在预填充和解码各自的份额之内。1P2D 和 1P3D 的解码在各自的节点上，因此这些 owner 只占预填充的 1499 GB。
• 为什么不再用 LMCache。 LMCache 预填充注册 1 TiB pinned 池后，有时会在启动 barrier 永久卡住（mi355x-amds 上 8 次启动卡了 3 次）；Store 预填充从未出现这种情况。在相同测试点上 Store 不逊于 LMCache（见下文）。
• 模型与内存。 mi355x-amds 上没有预置的 amd/GLM-5.2-MXFP4；与单机 GLM-5.2 测试点一样，worker 从各节点本地的 hub 缓存读取模型（/var/lib/hf-hub-cache 挂载到 $HF_HOME/hub，这是 srtctl 预下载唯一能看到的路径），srtctl 会在 worker 启动前先补齐第一个节点的缓存。每个测试点的 host_setup 都会先重新运行集群的 setup.sh 清理页缓存，再预留大页。五个 1P1D 测试点共用一个配方变体，因此扫描只为它们运行一个 GSM8K eval。
• srt-slurm 补丁。 srtctl 的 ATOM 后端把 worker 的 GPU 数直接作为 -tp 传入，PP4 预填充因此会以 -tp 4 --pipeline-parallel-size 4 启动（4 张 GPU 上 16 个 rank）。runners/srt-slurm/patches/604-atom-tp-from-pp.patch 携带 fix(atom): size -tp from the worker's GPUs over its PP and PCP stages NVIDIA/srt-slurm#604，按 pipeline 和 prefill-context stage 数相除；在 v2.59.1 的固定版本上仍能打上。子模块的固定版本包含该 PR 后即可删除此补丁。
• 缓存命中率统计。 本 PR 不修改 infx。ATOM P/D 的缓存命中率取自 server_metrics_export.json 中已有的计数器，由 fix(atom): read P/D cache counters from prefill and ingest CPU offload InferenceX-app#1533 在 ingest 时读取：只统计 prefill 端点，CPU 层取 atom:prefix_cache_offload_tokens。
• 镜像为 rocm/atom-dev:nightly_202610131249，按 digest 固定（ROCm/ATOM 6fd1bb1be）。

在 mi355x-amds 上的运行。 扫描 40578080205 在 45743f9f4 上全部测试点和 eval 均通过；当前 head 的配方、master 配置条目和 srt-slurm 补丁与该 commit 相同。每 GPU 吞吐与 TTFT p50/p90：

测试点	tok/s/GPU	TTFT p50 / p90	GSM8K（strict）
1P1D C56	31,450	1.18 / 2.38 s
1P1D C72	38,639	1.56 / 3.68 s
1P1D C96	44,039	2.64 / 7.45 s
1P1D C112	45,196	6.11 / 13.47 s
1P1D C128	40,032	15.31 / 36.43 s	0.966
1P2D C56	24,323	1.05 / 2.18 s	0.968
1P3D C12	3,958	0.79 / 1.57 s
1P3D C16	6,535	0.76 / 1.75 s	0.968
1P3D C24	10,213	0.80 / 1.81 s	0.967

与之前的布局（扫描 38674332505：1P1D 用按网卡划分的 4 x 349 GiB 池，1P2D 和 1P3D 用 LMCache）相比，1P1D C112 从 36,779 升到 45,196 tok/s/GPU，TTFT p50 从 19.3 s 降到 6.1 s；1P2D C56 从 22,379 升到 24,323。同一配方之前的扫描 39028510364（1a153fd1）测得 C112 为 39,245、C96 为 42,884，其余测试点与之相差在 2% 以内。C112 现已与 ATOM CI 在另一个集群上用同样 1536 GiB 池跑出的 46.1K 相差不到 2%。

AI 模型使用说明

• 模型/版本：claude-opus-5-5（Claude Opus 5.5），在 Claude Code 中运行。
• 工作内容：梳理 InferenceX 启动器和 srt-slurm 的 ATOM 后端，编写配方、master 配置条目、perf-changelog 条目以及 srt-slurm 补丁（fix(atom): size -tp from the worker's GPUs over its PP and PCP stages NVIDIA/srt-slurm#604），并生成绑定后的配方和 srtctl 模拟启动结果；根据扫描前的检查修订配方（共享 hub 缓存、Store 大页预留量、Store 共用一个变体）；分析扫描中 LMCache 的启动卡死，运行上述单点测试，按 [AgentX] budget decode servers' DRAM from decode GPUs / [AgentX] 按解码 GPU 为解码服务器分配 DRAM 预算 #4211 的内存预算把所有测试点改为 Store，并为 1P1D 采用 ATOM CI 的共享 Store 池；在 Fix ATOMesh KV-cache counters / 修复 ATOMesh KV-cache 计数 #4397 因 fix(atom): read P/D cache counters from prefill and ingest CPU offload InferenceX-app#1533 被关闭后，移除了其缓存计数器代码。
• 委派的 agent：一个 Claude Code Explore 子 agent 调研了 srt-slurm 的 ATOM 后端和 service 放置方式。运行环境没有提供它的模型标识，因此无法确认。

改动类型

新功能；配置变更。

🤖 Generated with Claude Code

[github-actions]

Thanks for the contribution!

• Review: If this PR changes files owned by someone other than a repository admin or @SemiAnalysisAI/core, ask one eligible CODEOWNER to complete the latest PR_REVIEW_CHECKLIST.md before contacting a core maintainer on Slack. Follow the template exactly, including As a PR reviewer and CODEOWNER, I have reviewed this and have, so sign-off verification triggers.
• PR verification: Sweeps run only when the PR appends an inferencex-e2e/perf-changelog.yaml entry and carries exactly one primary label: full-sweep-fail-fast (strongly recommended; canary plus per-matrix fail-fast), full-sweep-enabled (canary; matrix jobs continue after a failure), or non-canary-full-sweep-enabled (no canary or fail-fast). The modifiers all-evals, evals-only, and agentx-fast require a primary label. On fork PRs, a maintainer applies the label. See sweep labels and reuse.
• After merging: PR authors must ensure all GitHub Actions jobs pass. Transient failures often pass on rerun; see how to rerun failed jobs.

中文

感谢你的贡献！

• **审阅：**如果 PR 修改的文件归属于仓库管理员及 @SemiAnalysisAI/core 之外的 CODEOWNER，请先联系一位有资格的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，再通过 Slack 联系核心维护者。必须严格遵循模板，并保留 As a PR reviewer and CODEOWNER, I have reviewed this and have，才能触发签核验证。
• **PR 验证：**只有当 PR 在 inferencex-e2e/perf-changelog.yaml 末尾追加了条目，并且恰好带有一个主标签时，才会运行扫描：full-sweep-fail-fast（强烈推荐；canary 加逐矩阵 fail-fast）、full-sweep-enabled（有 canary；矩阵任务在失败后继续运行）或 non-canary-full-sweep-enabled（无 canary，也无 fail-fast）。修饰标签 all-evals、evals-only 和 agentx-fast 必须与主标签一起使用。fork PR 的标签由维护者添加。参见扫描标签与复用。
• **合并后：**PR 作者必须确保所有 GitHub Actions 任务通过。临时性失败通常可以通过重新运行恢复；参见重新运行失败任务的说明。

[functionstackx]

u might need to edit inferencex-e2e/infx/results/agentic/server_metrics.py & inferencex-e2e/infx/results/agentic/backends/atom.py for atom disagg too for it to collect mooncake prom metrics https://github.com/SemiAnalysisAI/InferenceX/pull/3964/changes

[claude]

I reviewed this PR and didn't find any bugs. Because it's a large, complex multi-node benchmark change — new ATOM prefill/decode-disaggregation recipe variants, Mooncake/LMCache service orchestration, hugepage reservation logic, and a backend patch to ATOM's tensor-parallel sizing — a human look would still be worthwhile.

What was reviewed:

• The new atom-tp-from-pp.patch: it only changes -tp sizing when pipeline-parallel-size/prefill-context-parallel-size > 1, and no existing ATOM recipe in the repo sets those, so prior ATOM points are unaffected.
• perf-changelog.yaml: confirmed the diff is a pure append (12 new lines at the file's tail, no earlier bytes touched) and the entry is English-only.
• Cross-checked tp/pp/dcp-size settings between amd-master.yaml's search space and disagg-variants.yaml's role definitions — they match.
• The ruled-out candidate issue (possible eval-result overwrite risk, as the first pp>1 multinode agentic entry) is worth a maintainer's confirmation given it wasn't run before this PR.

Extended reasoning...

The diff adds a new multi-node GLM-5.2 ATOM PD-disaggregation benchmark recipe (335 lines of YAML with Mooncake/LMCache service orchestration, hugepage reservation host_setup/teardown, NUMA-pinned RDMA services), wires it into amd-master.yaml, appends one perf-changelog.yaml block, and includes a vendored srt-slurm patch that changes how ATOM's backend computes -tp from GPU count/pipeline-parallel-size. No auth/crypto/permissions surface is touched; the sensitive bits are correctness of resource math (GPU/hugepage counts, port assignments) and whether the new -tp derivation could regress other ATOM configs — I confirmed no other ATOM recipe currently sets pipeline-parallel-size, so the patch is a no-op elsewhere. Size and multi-service complexity, plus the PR's own note that this has not yet been run on the target cluster, are why a human should still look even though no bugs were found.

This review covers commit 8df9856, which is no longer the latest commit on this pull request; later commits are not covered by it.

[Jasen2201]

u might need to edit inferencex-e2e/infx/results/agentic/server_metrics.py & inferencex-e2e/infx/results/agentic/backends/atom.py for atom disagg too for it to collect mooncake prom metrics https://github.com/SemiAnalysisAI/InferenceX/pull/3964/changes

Thanks! Cherry-picked #3964 into this PR so the sweep uses the fixed ATOM P/D cache accounting. The Mooncake Store needs no separate scrape: ATOM reports its reuse in the same atom:prefix_cache_offload_tokens counter as LMCache. I'll drop the cherry-pick once #3964 merges.

[github-actions]

View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=40578080205

View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=40578080205

[chunfangamd]

As a PR reviewer and CODEOWNER, I have reviewed this and have:

• Verified that as of the moment of typing this, this is the latest version of PR_REVIEW_CHECKLIST.md
• Verified that the general code quality meets the InferenceX standard and does not make the code quality any worse.
• Verified that this PR has passed PR validation. Please link to GitHub Action workflow that shows this. https://github.com/SemiAnalysisAI/InferenceX/actions/runs/38674332505
• Verified that this PR passes evals. Please link to GitHub Action workflow that shows this. https://github.com/SemiAnalysisAI/InferenceX/actions/runs/38674332505
• Verified that speculative decoding PRs uses chat templates to align the AL distribution to real world
• Verified that every draft model and draft head is served as it ships: the draft that ships with the served checkpoint, at its stored precision, through the pinned upstream image's default handling, with the shipped and effective draft precision recorded in the additional detail section. No submission-side quantization, dtype override, checkpoint substitution, or patch may lower draft precision below that default, regardless of eval results or AL. Explicitly verified that SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE is not enabled in the effective recipe, including inherited settings; enabling it is prohibited going forward, and historical runs do not grant an exception. See Draft-model precision for what counts as the default and the MLPerf comparison.
• For agentic workloads: verified that speculative-decoding configs (EAGLE / MTP / draft models) run with simulated synthetic acceptance, with the acceptance-length value taken from the committed golden AL curve in infx/golden_al_distribution/ for that model, thinking mode, and draft length. A submission may choose any supported draft length, but it may not substitute a different acceptance target.
• Verified against the current MODELS.md that this PR does not submit a deprecated model, scenario, or model-scenario combination.
• Verified that the model architecture isn't changed with benchmark hacks like using --hf-overrides to skipping indexer for every x layers on models that don't natively support this. As a general rule, we won't accept optimizations that reduces the number of model architecture FLOPs. Anything that makes that same computation run faster is fair game; target/verifier FLOPs at lower precisions is fine, given that the config passes private evals, but this does not permit lowering draft-model or draft-head precision below what ships. As an general north star princple, we should only use optimizations which is used in production by customers that care about accuracy
• For affected Dynamo TRT-LLM benchmarks, verified that iteration/KV-cache metrics are collected and published under the pinned Dynamo implementation and effective engine settings. Resolve the installed Dynamo version separately from the container tag; use supported metrics controls without requiring KV events. Record the version and configuration evidence in the additional detail section. See verifier Check 17. N/A for unaffected serving paths or frameworks without these Dynamo settings.
• If an company claims that they support vLLM/SGLang as first class LLM inference engines on their hardware, I have verified that the respective vLLM submission made using upstream https://hub.docker.com/u/vllm docker repo, upstream SGLang https://hub.docker.com/u/lmsysorg docker repo. The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet as supported by vLLM/SGLang community maintainers
• If an company claims that they support vLLM/SGLang as first class upstream in-tree LLM inference engines on their hardware, I have have verified that the respective vLLM/SGLang submission has been made before additional frameworks (TRT-LLM, ATOM, etc.). The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet.
• Verified that every affected vLLM/SGLang production serving configuration in this PR, including Dynamo deployments, is documented in the official vLLM recipes and/or the SGLang cookbook, for both single-node and multi-node aggregated and disaggregated serving. The documentation covers production-useful flags, environment variables, and launch commands (including prefill, decode, and router/frontend commands for disaggregation); InferenceX-only benchmark/harness settings are excluded with reasoning in the additional detail section:
  • I linked the corresponding upstream PR in the vLLM recipe repo or SGLang cookbook source repo and verified that it is MERGED before this InferenceX PR merges. An opened, draft, or closed-without-merge upstream PR does not satisfy this requirement. If the matching recipe was already published, I linked the published recipe/cookbook page in the additional detail section below.
• Verified that this PR does not patch the inference engine or serving stack — the pinned image must run as shipped. This covers .patch files / git apply / patch, inline patches embedded in benchmark scripts (e.g. a python3/sed heredoc that rewrites installed engine sources before serving), in-place edits of site-packages, monkey-patching, overwriting container files, and installing forked/rebuilt engine wheels on top of the pinned image. The only exception is a patch covered by a filled-out waiver at docs/waiver/<PR_NUMBER>.md — named after the PR that introduces the patch and filed in that same PR, stating what is patched, why the unmodified upstream image cannot run this benchmark, the upstream PR/issue link, and the removal plan — which I have linked below in the additional detail section.
• If this PR uses append-only: true, verified that it only adds generated points or recipe variants inside a selected existing config/scenario and existing same-image visual curve: every previously generated point remains present with the same recipe, no prior point is removed or rerun, and every benchmark-affecting change in the complete diff can affect only the corresponding newly appended points (never an existing point), regardless of which file contains it.
• If any of the above criteria cannot reasonably be satisfied, I have provided additional reasoning below.
• Reported measured throughput/E2EL Pareto counts and evidence per affected curve (≥5 points strongly recommended). Below 5 or unverifiable: tag a core maintainer for review; recorded admin bypass required before merge. N/A if no curves are affected. Details.

Additional detail section:

• insert any additional info here

Signed: @chunfangamd

[github-actions]

❌❌❌ REJECTED ❌❌❌

@chunfangamd this sign-off cites a sweep from a superseded recipe and the PR has no green sweep on its current tree. The new changelog entry also omits the target quantization and KV dtype.

❌ Check 1 (Passing sweep on in-PR commit): FAIL — The passing sweep predates a material change in e3f27cd4 (every point moved from LMCache to the Mooncake Store) and 1a153fd1 (1P1D shared 2 x 768 GiB pool). Rerun the sweep on the current recipe. Both the sign-off and the only /reuse-sweep-run cite 38674332505 on ca89ac21, which used the old layout. The later green sweep 39028510364 on 1a153fd1 is also not current. Since then, merges 9f774fe3, 999ccb5c and f59e286c brought in base-branch changes to the launch, benchmark and eval path (infx/launch/drivers/srt/*, infx/bench/agentic/*, infx/bench/eval/*, runtime_settings.sh), and 2c794e33 added #4397's ATOM result tooling. The only sweep since then, 40282641074 on 2c794e33, failed the 1P3D c16 benchmark and eval because host_setup could not reserve hugepages on mia1-p01-g14.

❌ Check 18 (Recipe/image/sweep consistency): FAIL — (f2) The new perf-changelog.yaml entry for glm5.2-fp4-mi355x-atom-disagg-agentic-mtp omits that the recipe adds online ptpc_fp8 target quantization (online_quant_config, MTP layer 78 excluded) and kv_cache_dtype: fp8 (disagg-variants.yaml base args). Other parts pass: (a) all keys generate; (b) master and recipe topology agree (PP4 prefill, TP4/DCP4 or TP4 decodes, 2 or 3 decode workers); (c) image digest resolves; (d) srt-slurm pin unchanged, plus an in-repo patch; (g) no removed points.

⚠️ Check 13 (Draft precision): WARN — glm5.2-fp4-mi355x-atom-disagg-agentic-mtp (MTP3/4/5, amd/GLM-5.2-MXFP4): the additional detail section is empty, so no draft checkpoint, stored precision, pinned-image handling or effective draft precision is recorded. The recipe excludes model.layers.78.* from ptpc_fp8, and the KV cache is FP8 for target and draft alike. I could not inspect the checkpoint metadata to confirm that layer 78 is the BF16 MTP head; human review needed.

⚠️ Check 15 (PR description): WARN — The body says "The branch carries #3964 (cherry-picked as 4f75d941) ... this PR's sweep computes the ATOM P/D cache rates with it", but at f59e286c merge 999ccb5c dropped the #3964 tooling and 2c794e33 merged #4397 instead. The body's "Sweep 39028510364 ran this layout" describes 1a153fd1, not the current tree. Please correct the body.

Passed and not applicable checks

✅ Check 0 (CODEOWNER): PASS — @chunfangamd is a named owner of inferencex-e2e/configs/amd-master.yaml and the **/mi355x-*/ recipe tree, so is a recognized CODEOWNER. runners/srt-slurm/patches/ is owned by @cquil11 and @SemiAnalysisAI/core; team membership could not be read.

✅ Check 2 (Evals): PASS — Run 39028510364 ran four GSM8K lanes on the PR's digest-pinned image: 1P1D c128 0.962, 1P2D c56 0.965, 1P3D c16 0.972, 1P3D c24 0.966. Every gap is within 2 sigma of the best sibling and of the ingested ATOM cohort (39540045682). No conditional kernel path is involved.

➖ Check 3 (Upstream recipe): N/A — Only an ATOM (atom-disagg) serving configuration is affected; there is no vLLM or SGLang deployment.

✅ Check 4 (Reuse command): PASS — COLLABORATOR @seungrokj posted /reuse-sweep-run 38674332505. It names the superseded run, which Check 1 covers.

✅ Check 5 (Checklist template): PASS — Every item in the current PR_REVIEW_CHECKLIST.md template is present and checked.

✅ Check 6 (Upstream images / engine-first): PASS — atom-disagg is a vendor framework, so the upstream image rule does not apply. The open-source entry glm5.2-fp4-mi355x-sglang-agentic-mtp already exists for glm5.2 on MI355X.

✅ Check 7 (Deprecated models): PASS — glm5.2 agentic-coding with MTP is active in MODELS.md.

✅ Check 8 (No architecture hacks): PASS — There are no --hf-overrides or layer edits. AITER_QUICK_REDUCE_QUANTIZATION=INT4 and target-only ptpc_fp8 are covered by passing evals on each topology (1P1D, 1P2D and 1P3D).

✅ Check 9 (Chat template): PASS — The recipe sets AIPERF_APPLY_CHAT_TEMPLATE: "true" for every point.

✅ Check 10 (No engine patches): PASS — 604-atom-tp-from-pp.patch changes only how srtctl computes ATOM's -tp launch argument (NVIDIA/srt-slurm#604). No engine source, wheel or container file is modified, so rocm/atom-dev runs as shipped.

✅ Check 11 (Agentic golden AL): PASS — The launcher sets spec-decode-acceptance-length from glm5.2_mtp.yaml thinking_on: 3.61 for MTP5 (job 117384384139) and 2.99 for MTP3 (job 117384384189). Both match the golden curve.

➖ Check 12 (Append-only): N/A — The new changelog entry has no append-only: true.

✅ Check 14 (Pareto coverage): PASS — One curve: glm5.2 agentic-coding, MI355X ATOM, fp4, p90 E2EL, run 39028510364. Seven measured points are on the frontier (1P3D c16 and c24, 1P2D c56, 1P1D c56, c72 and c96), taken from the bmk_agentic_* artifacts.

➖ Check 16 (NVIDIA overview): N/A — The only affected hardware is AMD MI355X.

➖ Check 17 (Metrics publication): N/A — No Dynamo TRT-LLM serving path is affected.

Assessed commit: f59e286c562c027072bfb852139f3e00e981cbc5.

[Jasen2201]

Status of the sign-off verification above (assessed on f59e286c):

• Check 1: sweep 40578080205 on 45743f9f4 ran all 9 points and the 4 GSM8K evals green. The head cdbec8e22 differs from it only by a main merge and dropping Fix ATOMesh KV-cache counters / 修复 ATOMesh KV-cache 计数 #4397's infx tooling (Fix ATOMesh KV-cache counters / 修复 ATOMesh KV-cache 计数 #4397 was closed in favor of fix(atom): read P/D cache counters from prefill and ingest CPU offload InferenceX-app#1533); the recipe, its master config entry and the srt-slurm patch are unchanged. A maintainer /use 40578080205 is still needed.
• Check 18: 45743f9f4 records the ptpc_fp8 target quantization (excluding model.layers.78.*) and kv_cache_dtype fp8 in the changelog entry.
• Check 15: the PR body now describes the current tree and sweep 40578080205.
• Check 13, draft precision: the MTP draft is the head shipped in amd/GLM-5.2-MXFP4 (config.json: num_hidden_layers 78, num_nextn_predict_layers 1, so model.layers.78). The checkpoint stores it unquantized: Quark's exclude list covers model.layers.78.*, and its 791 safetensors entries are 790 BF16 + 1 F32 with no scale tensors. The recipe's online_quant_config (ptpc_fp8 for the target's BF16 linears) excludes model.layers.78.*, so the draft is served at its stored BF16. Like the target, it uses the FP8 KV cache (kv_cache_dtype fp8) and the FP4 indexer cache, which is ATOM's handling on the pinned image; no setting lowers the draft's weight precision.

@chunfangamd, could you re-sign off citing 40578080205, with the draft-precision note above in the additional detail section?

[chunfangamd]

/reuse-sweep-run 40578080205