Skip to content

fix(agentx): bump AIPerf to v1.0.5 and propagate MI355X failures - #2572

Merged
cquil11 merged 1 commit into
mainfrom
agent/bump-aiperf-v1.0.5
Aug 12, 2026
Merged

cquil11 merged 1 commit into
mainfrom
agent/bump-aiperf-v1.0.5

Conversation

@cquil11

@cquil11 cquil11 commented Aug 12, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • bump utils/aiperf from agentx-v1.0.4 to agentx-v1.0.5
  • propagate the single-node MI355X benchmark exit code after Slurm cleanup

Behavior

AIPerf now allows a three-minute sparse tail in one-hour AgentX profiles while still failing when neither TTFT nor inter-token latency reaches 95% of the profile. This fixes healthy c1 profiles rejected at 97.2% coverage.

The MI355X launcher previously followed a failing container srun with a successful scancel, masking the benchmark failure from GitHub Actions. It now preserves the srun result, performs cleanup, and exits with that result.

Evidence: TP2 c1, TP4 c1, AIPerf #41.

Validation

  • AIPerf: 103 focused tests, Ruff/format, and all pre-commit hooks passed
  • InferenceX: bash -n runners/launch_mi355x-amds.sh and git diff --check passed
  • no inference-server, workload, routing, or performance configuration changes

Signed-off-by: Cam Quilici <cjquilici@gmail.com>
@cquil11
cquil11 requested a review from a team August 12, 2026 11:41

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — straightforward fix that preserves the srun exit code across cleanup.

Extended reasoning...

Overview

This PR touches only runners/launch_mi355x-amds.sh (CI runner script for single-node MI355X benchmarks) and bumps the utils/aiperf submodule pointer to agentx-v1.0.5. The shell change captures $? immediately after the benchmark srun invocation into benchmark_rc, then exits with that value after the existing scancel and gpucore cleanup steps run. Previously the script's final exit status was implicitly that of scancel, which silently masked real benchmark failures from GitHub Actions (as shown by the linked TP2/TP4 c1 job evidence).

Security risks

None. This is CI/cluster orchestration shell code with no user input, auth, or credential handling involved. The submodule bump only changes a benchmark client's latency-tail tolerance logic in a separate repo, not anything in this repo's trust boundary.

Level of scrutiny

Low. This is a narrowly scoped, mechanical bug fix in a non-production CI launcher script — exactly the kind of self-contained change (capture exit code, propagate it) that doesn't need design-level input. The multi-node branch of the same script already uses a similar pattern, so this brings the single-node branch in line.

Other factors

bash -n and git diff --check were reported as passing, and the change is easy to read end-to-end in the diff. The bug-hunting pass found no issues, and the one candidate quality nit raised (reimplementing a trap-based pattern from a sibling launcher) was ruled out as not a real bug. No outstanding review comments exist on this PR.

@cquil11
cquil11 merged commit fe433b1 into main Aug 12, 2026
7 checks passed
@cquil11
cquil11 deleted the agent/bump-aiperf-v1.0.5 branch August 12, 2026 12:53
seungrokj added a commit that referenced this pull request Oct 14, 2026
…014 nightly

- Update the image from rocm/atom-dev:nightly_202610071700 to rocm/atom-dev:nightly_202610141644 (digest sha256:ef65fed0...): ATOM a526f0d55 to 6e9b3e1c5, Triton 3.7.0 to 3.8.0; ROCm 7.2.4 and LMCache v0.4.5 are unchanged.
- Points, concurrencies, parallelism and admission are unchanged.
- Turn on ATOM's DeepSeek-V4 bounded state replay at every point (ATOM_DSV4_STATE_REPLAY=1, ROCm/ATOM#2552); the replay span is 128 + 61 x 127 = 7,875 tokens with the bundled DSpark layers.
- Keep state checkpoints at prompt ends only: state-checkpoint-interval-tokens -1 (was 8192) and ATOM_STATE_CHECKPOINT_DEMAND=0.
- Run C1 to C16 with FP8 KV (was BF16); KV is FP8 at every point.
- Drop lmcache.mp.max_pinned_state_bytes at C256/C384: a no-op, since ROCm/ATOM#2434 was already in the previous image.
- List the ATOM changes in the new image that matter here (#2571, #2579, #2551, #2543, #2572, #2556, #2518, #2566).
- No data-type change to the DSpark draft: the bundled mtp.* layers load as shipped with no online quantization; the draft window KV follows the target's kv-cache-dtype (BF16 to FP8 at C1 to C16), which CONTRIBUTING.md allows.

- 镜像由 rocm/atom-dev:nightly_202610071700 更新为 nightly_202610141644:ATOM a526f0d55 → 6e9b3e1c5,Triton 3.7.0 → 3.8.0;ROCm 7.2.4 与 LMCache v0.4.5 不变。
- 测试点、并发、并行方式与准入设置不变。
- 所有点启用 ATOM DeepSeek-V4 有界状态回放(ATOM_DSV4_STATE_REPLAY=1,ROCm/ATOM#2552);含 DSpark 层时回放跨度为 128 + 61 x 127 = 7,875 个 token。
- 仅在 prompt 结尾保存状态检查点:state-checkpoint-interval-tokens 设为 -1(原 8192),ATOM_STATE_CHECKPOINT_DEMAND=0。
- C1 至 C16 改用 FP8 KV(原 BF16),所有点的 KV 均为 FP8。
- C256/C384 去掉 lmcache.mp.max_pinned_state_bytes:上一镜像已包含 ROCm/ATOM#2434,故无实际影响。
- 列出新镜像中与本模型相关的 ATOM 变更(#2571、#2579、#2551、#2543、#2572、#2556、#2518、#2566)。
- DSpark draft 的数据类型不变:内置 mtp.* 层按原始精度加载,无在线量化;draft 窗口 KV 跟随目标模型的 kv-cache-dtype(C1 至 C16 由 BF16 变为 FP8),符合 CONTRIBUTING.md 的规定。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
adibarra added a commit that referenced this pull request Oct 15, 2026
…TOM 镜像与环境变量 (#4065)

* perf(amd): DSv4-Pro ATOM AgentX with bounded state replay and FP8 KV on the 1014 nightly / DSv4-Pro ATOM AgentX 在 1014 nightly 上启用有界状态回放并统一 FP8 KV

Move dsv4-fp4-mi355x-atom-agentic-mtp to rocm/atom-dev:nightly_202610141644, turn on ATOM's bounded
state replay with prompt-end state checkpoints at every point, run C1-C16 with FP8 KV, and drop the
lmcache.mp.max_pinned_state_bytes key ATOM no longer reads.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Link the changelog entry to #4065

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* perf(amd): rewrite the DSv4-Pro ATOM AgentX changelog entry for the 1014 nightly

- Update the image from rocm/atom-dev:nightly_202610071700 to rocm/atom-dev:nightly_202610141644 (digest sha256:ef65fed0...): ATOM a526f0d55 to 6e9b3e1c5, Triton 3.7.0 to 3.8.0; ROCm 7.2.4 and LMCache v0.4.5 are unchanged.
- Points, concurrencies, parallelism and admission are unchanged.
- Turn on ATOM's DeepSeek-V4 bounded state replay at every point (ATOM_DSV4_STATE_REPLAY=1, ROCm/ATOM#2552); the replay span is 128 + 61 x 127 = 7,875 tokens with the bundled DSpark layers.
- Keep state checkpoints at prompt ends only: state-checkpoint-interval-tokens -1 (was 8192) and ATOM_STATE_CHECKPOINT_DEMAND=0.
- Run C1 to C16 with FP8 KV (was BF16); KV is FP8 at every point.
- Drop lmcache.mp.max_pinned_state_bytes at C256/C384: a no-op, since ROCm/ATOM#2434 was already in the previous image.
- List the ATOM changes in the new image that matter here (#2571, #2579, #2551, #2543, #2572, #2556, #2518, #2566).
- No data-type change to the DSpark draft: the bundled mtp.* layers load as shipped with no online quantization; the draft window KV follows the target's kv-cache-dtype (BF16 to FP8 at C1 to C16), which CONTRIBUTING.md allows.

- 镜像由 rocm/atom-dev:nightly_202610071700 更新为 nightly_202610141644:ATOM a526f0d55 → 6e9b3e1c5,Triton 3.7.0 → 3.8.0;ROCm 7.2.4 与 LMCache v0.4.5 不变。
- 测试点、并发、并行方式与准入设置不变。
- 所有点启用 ATOM DeepSeek-V4 有界状态回放(ATOM_DSV4_STATE_REPLAY=1,ROCm/ATOM#2552);含 DSpark 层时回放跨度为 128 + 61 x 127 = 7,875 个 token。
- 仅在 prompt 结尾保存状态检查点:state-checkpoint-interval-tokens 设为 -1(原 8192),ATOM_STATE_CHECKPOINT_DEMAND=0。
- C1 至 C16 改用 FP8 KV(原 BF16),所有点的 KV 均为 FP8。
- C256/C384 去掉 lmcache.mp.max_pinned_state_bytes:上一镜像已包含 ROCm/ATOM#2434,故无实际影响。
- 列出新镜像中与本模型相关的 ATOM 变更(#2571、#2579、#2551、#2543、#2572、#2556、#2518、#2566)。
- DSpark draft 的数据类型不变:内置 mtp.* 层按原始精度加载,无在线量化;draft 窗口 KV 跟随目标模型的 kv-cache-dtype(C1 至 C16 由 BF16 变为 FP8),符合 CONTRIBUTING.md 的规定。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: seungrokj <seungrok.jung@amd.com>
Co-authored-by: seungrokj <144636725+seungrokj@users.noreply.github.com>
Co-authored-by: adibarra <93070681+adibarra@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

1 participant