diff --git a/MODELS.md b/MODELS.md index f845453d5..d461f1657 100644 --- a/MODELS.md +++ b/MODELS.md @@ -159,7 +159,6 @@ Other offloading tiers, including NVMe KV cache offloading, are outside the init | Kimi-K3 | `kimik3` | 2026-07-27 ([#2391](https://github.com/SemiAnalysisAI/InferenceX/pull/2391)) | Agentic coding (DSpark may be disabled for better Pareto points) | Standalone non-DSpark A/B baseline (not required from day 0) | | GLM-5.2 | `glm5.2` | 2026-07-18 ([#2268](https://github.com/SemiAnalysisAI/InferenceX/pull/2268)) | Agentic coding (non-MTP points remain eligible under the Pareto policy; see Deprecation Notice) | | | MiniMax-M3 | `minimaxm3` | 2026-06-12 ([#1724](https://github.com/SemiAnalysisAI/InferenceX/pull/1724)) | Agentic coding | Single-turn 1k1k, Single-turn 8k1k (removed 2026-08-04, [#2493](https://github.com/SemiAnalysisAI/InferenceX/pull/2493)) | -| DeepSeek-V4.1-Flash | `dsv41flash` | Pending | Agentic coding on MI355X (draft; GPU validation pending) | — | | DeepSeek-V4-Pro | `dsv4` | 2026-04-24 ([#1130](https://github.com/SemiAnalysisAI/InferenceX/pull/1130)) | Agentic coding (non-spec-decode points remain eligible under the Pareto policy) | Single-turn 1k1k, Single-turn 8k1k (removed 2026-09-09, [#2921](https://github.com/SemiAnalysisAI/InferenceX/pull/2921)) | | GLM-5 / GLM-5.1 | `glm5`, `glm5.1` | 2026-03-06 ([#762](https://github.com/SemiAnalysisAI/InferenceX/pull/762)), with GLM-5.1 added 2026-04-21 ([#1098](https://github.com/SemiAnalysisAI/InferenceX/pull/1098)) | GLM-5.1 B200 TileRT only: 1k1k and 8k1k added 2026-08-09 ([#2533](https://github.com/SemiAnalysisAI/InferenceX/pull/2533)); Agentic coding added in [#2650](https://github.com/SemiAnalysisAI/InferenceX/pull/2650) | The earlier GLM-5 / GLM-5.1 recipes were retired 2026-07-18 ([#2276](https://github.com/SemiAnalysisAI/InferenceX/pull/2276)) | | MiniMax-M2.5/2.7 | `minimaxm2.5` | 2026-02-18 ([#755](https://github.com/SemiAnalysisAI/InferenceX/pull/755)) | None (retired 2026-06-20, [#1874](https://github.com/SemiAnalysisAI/InferenceX/pull/1874)) | Single-turn 1k1k, Single-turn 1k8k, Single-turn 8k1k | diff --git a/MODELS_zh.md b/MODELS_zh.md index 8215a8de9..dd52db9a2 100644 --- a/MODELS_zh.md +++ b/MODELS_zh.md @@ -159,7 +159,6 @@ InferenceX 支持 SGLang 和 vLLM 双方的维护者,并响应 AI 实验室和 | Kimi-K3 | `kimik3` | 2026-07-27 ([#2391](https://github.com/SemiAnalysisAI/InferenceX/pull/2391)) | 智能体编码(可关闭 DSpark 以获得更优帕累托点) | 独立非 DSpark A/B 基线(自第 0 天起即不要求) | | GLM-5.2 | `glm5.2` | 2026-07-18([#2268](https://github.com/SemiAnalysisAI/InferenceX/pull/2268)) | 智能体编码(非 MTP 数据点仍可按帕累托策略参与发布;见弃用公告) | | | MiniMax-M3 | `minimaxm3` | 2026-06-12([#1724](https://github.com/SemiAnalysisAI/InferenceX/pull/1724)) | 智能体编码 | 单轮 1k1k、单轮 8k1k(2026-08-04 移除,[#2493](https://github.com/SemiAnalysisAI/InferenceX/pull/2493)) | -| DeepSeek-V4.1-Flash | `dsv41flash` | 待验证 | MI355X 上的 Agentic coding(草案;等待 GPU 验证) | — | | DeepSeek-V4-Pro | `dsv4` | 2026-04-24([#1130](https://github.com/SemiAnalysisAI/InferenceX/pull/1130)) | 智能体编码(非投机解码数据点仍可按帕累托策略参与发布) | 单轮 1k1k、单轮 8k1k(已于 2026-09-09 移除,[#2921](https://github.com/SemiAnalysisAI/InferenceX/pull/2921)) | | GLM-5 / GLM-5.1 | `glm5`、`glm5.1` | 2026-03-06([#762](https://github.com/SemiAnalysisAI/InferenceX/pull/762)),GLM-5.1 于 2026-04-21 加入([#1098](https://github.com/SemiAnalysisAI/InferenceX/pull/1098)) | 仅 GLM-5.1 B200 TileRT:1k1k 和 8k1k 于 2026-08-09 加入([#2533](https://github.com/SemiAnalysisAI/InferenceX/pull/2533));智能体编码由 [#2650](https://github.com/SemiAnalysisAI/InferenceX/pull/2650) 加入 | 此前的 GLM-5 / GLM-5.1 配方于 2026-07-18 退役([#2276](https://github.com/SemiAnalysisAI/InferenceX/pull/2276)) | | MiniMax-M2.5/2.7 | `minimaxm2.5` | 2026-02-18([#755](https://github.com/SemiAnalysisAI/InferenceX/pull/755)) | 无(2026-06-20 退役,[#1874](https://github.com/SemiAnalysisAI/InferenceX/pull/1874)) | 单轮 1k1k、单轮 1k8k、单轮 8k1k | diff --git a/benchmarks/single_node/agentic/dsv41flash_fp4_mi355x_vllm_mtp.sh b/benchmarks/single_node/agentic/dsv41flash_fp4_mi355x_vllm_mtp.sh index fd6da6ab9..e53578124 100755 --- a/benchmarks/single_node/agentic/dsv41flash_fp4_mi355x_vllm_mtp.sh +++ b/benchmarks/single_node/agentic/dsv41flash_fp4_mi355x_vllm_mtp.sh @@ -41,15 +41,67 @@ export VLLM_ENGINE_READY_TIMEOUT_S=3600 export VLLM_USE_RUST_FRONTEND=1 export PYTHONUNBUFFERED=1 -# Upstream picks 1024 on GPUs with >= 160 GiB, and 2*CONC starves AgentX -# subagent fan-out at low CONC. 128 also keeps CAPTURE_SIZE deterministic. -MAX_NUM_SEQS=128 +# vllm-project/vllm#57491 widened the two is_cuda() gates to is_cuda_alike(), so +# on gfx950 this image resolves an Engram config and an explicit value is needed +# rather than the VLLM_PLE_CPU_OFFLOAD default. +# +# The tables cost 47.2 GiB per rank at TP=4, so 94.4 GiB at TP=2, which does not +# fit beside half of the 511 GB checkpoint on a 288 GiB card: TP=2 always +# offloads. TP=4 keeps them resident, as the validated concurrency 1-32 run +# measured, because offloaded lookups go to pinned host memory over UVA and +# nothing below c128 is short of KV. Resident leaves 14.83M KV tokens, which is +# 232K per request at c64 and healthy, but only 116K at c128, under the 122K at +# which TP=2 c64 collapsed. Offloading lifts it to 33.06M, so 258K at c128. +if (( TP == 2 || CONC >= 128 )); then + ENGRAM_CONFIG='{"cpu_offload":true}' +else + ENGRAM_CONFIG='{"cpu_offload":false}' +fi + +# Graph capture covers twice the outer concurrency, floored at the #3058 size of +# 128 sequences, across the 1+5 DSpark token shape. Twice leaves headroom for +# AgentX subagent fan-out above the outer concurrency. NUM_SPEC_TOKENS=5 +GRAPH_NUM_SEQS=$((2 * CONC)) +if (( GRAPH_NUM_SEQS < 128 )); then + GRAPH_NUM_SEQS=128 +fi CAPTURE_SIZE=1 -while (( CAPTURE_SIZE < MAX_NUM_SEQS * (1 + NUM_SPEC_TOKENS) && CAPTURE_SIZE < 2048 )); do +while (( CAPTURE_SIZE < GRAPH_NUM_SEQS * (1 + NUM_SPEC_TOKENS) && CAPTURE_SIZE < 2048 )); do CAPTURE_SIZE=$((CAPTURE_SIZE * 2)) done +# The sparse-attention indexer and its companion per-rank buffers scale with +# --max-num-batched-tokens at roughly 4.4 MiB per token, measured on gfx950, so +# a smaller prefill chunk buys KV room. TP=2 starts from half the per-rank space +# and is the arm that runs short: at the upstream 16384 it holds 7.84M KV +# tokens, 122K per request at c64, where run 35574132719 fell to a 17.6% prefix +# cache hit rate, 187 s TTFT and 150 tok/s against 94.8%, 1.3 s and 957 tok/s at +# c32. Every point that held had 232K per request or more, so keep the upstream +# chunk through c32 (245K at TP=2) and trade it away only above that. B300 runs +# 8192 at TP=4 and the Blackwell TP=2 arms run 4096 (#3320, #3321). +if (( CONC >= 128 )); then + (( TP == 2 )) && BATCHED_TOKENS=4096 || BATCHED_TOKENS=8192 +elif (( TP == 2 && CONC >= 64 )); then + BATCHED_TOKENS=8192 +else + BATCHED_TOKENS=16384 +fi + +# DSpark verifies 1+5 tokens per sequence, so a decode batch of max_num_seqs +# needs six times that many token slots. The MI355X API-server default of 1024 +# sequences therefore wants 6144, and below that the engram projection faults +# during profiling: TP=2 c128 at 4096 leaves four slots per sequence and dies +# with HSA_STATUS_ERROR_EXCEPTION at M=1024, N=129280, K=256, reproduced on two +# separate GPU pairs. Where the chunk is that small, cap in-flight sequences at +# the shape graph capture already covers, which also keeps the largest decode +# batch on a captured graph. Leave the default alone everywhere else. +DEFAULT_MAX_NUM_SEQS=1024 +MAX_NUM_SEQS="" +if (( BATCHED_TOKENS < DEFAULT_MAX_NUM_SEQS * (1 + NUM_SPEC_TOKENS) )); then + MAX_NUM_SEQS="$GRAPH_NUM_SEQS" +fi + # Use the runner-specific port assigned by launch_mi355x-amds.sh. export AIPERF_SERVER_URL="http://localhost:${PORT}" export AIPERF_SERVER_METRICS_URLS="${AIPERF_SERVER_URL}/metrics" @@ -73,6 +125,7 @@ VLLM_CMD=( --tokenizer-mode deepseek_v41 --tool-call-parser deepseek_v41 --enable-auto-tool-choice --reasoning-parser deepseek_v41 + --engram-config "$ENGRAM_CONFIG" # aiter, not aiter_triton_mxfp4_bf16: the plain name opens vLLM's full # priority list and the CK kernel at its head wins. CK quantizes # activations to FP8 internally and dispatches the a8w4 experts @@ -82,11 +135,21 @@ VLLM_CMD=( --gpu-memory-utilization 0.9 --speculative-config "$SPEC_CONFIG" --max-model-len 1048576 - --max-num-seqs "$MAX_NUM_SEQS" --max-cudagraph-capture-size "$CAPTURE_SIZE" - --max-num-batched-tokens 16384 + --max-num-batched-tokens "$BATCHED_TOKENS" + # vllm-project/vllm#56227 added SWA bounded replay (default on) after the + # eed1f3d0 pin and before this one. It pads the replayed tokens' slots in the + # prefix-cacheable groups, but the window clamp it relies on landed in the + # FlashInfer and FlashMLA kernels; the ROCm sparse SWA path only gained the + # replay_start kwarg. On gfx950 every TP=2 and TP=4 point of run 35567570539 + # died with HSA_STATUS_ERROR_MEMORY_FAULT at the first prefix hit carrying a + # replay start. Drop this once ROCm clamps too; prefix caching stays on. + --no-swa-bounded-replay --disable-uvicorn-access-log ) +if [[ -n "$MAX_NUM_SEQS" ]]; then + VLLM_CMD+=(--max-num-seqs "$MAX_NUM_SEQS") +fi printf '%q ' "${VLLM_CMD[@]}" | tee "$RESULT_DIR/vllm_command.txt" printf '\n' | tee -a "$RESULT_DIR/vllm_command.txt" SERVER_PID="" diff --git a/configs/amd-master.yaml b/configs/amd-master.yaml index bae2efc10..bcf8c3787 100644 --- a/configs/amd-master.yaml +++ b/configs/amd-master.yaml @@ -1466,8 +1466,14 @@ dsv4-fp4-mi355x-sglang-agentic-mtp: # tag predates vllm-project/vllm#56503, which moves the mHC delayed pre block off # the eager Torch reference and onto AITER. MI355X run 34710937012 passed the # full concurrency 1-32 sweep and eval-only concurrency 32. +# TP=2 joins TP=4, and both extend to concurrency 128. TP=2 became feasible on +# this SKU once the Engram tables moved to host memory: they cost 47.2 GiB of +# device memory per rank, which is what previously forced four GPUs per server. +# Two GPUs per server doubles the servers per node and is the layout that +# decides whether DSv4.1-Flash is throughput- or interactivity-bound here. dsv41flash-fp4-mi355x-vllm-agentic-dspark: - image: vllm/vllm-openai-rocm:nightly-eed1f3d0c6043bd494424a22443ee198dd56f657 + # ROCm 10.0 nightly channel, shared with kimik3-fp4-mi355x-vllm-agentic-mtp. + image: vllm/vllm-openai-rocm:nightly-rocm100-3df4ae153eb385e27b52f26c81f8edb9e20b9984 model: deepseek-ai/DeepSeek-V4.1-Flash model-prefix: dsv41flash runner: cluster:mi355x-amds @@ -1478,5 +1484,10 @@ dsv41flash-fp4-mi355x-vllm-agentic-dspark: agentic-coding: - dram-utilization: 0.60 search-space: - # Follow upstream AMD Engram defaults; omit the CUDA-only config flag. - - { tp: 4, kv-offloading: none, spec-decoding: mtp, conc-list: [1, 2, 4, 8, 16, 32] } + # The recipe only changes the memory split where a point ran short of KV: + # TP=4 offloads the engram tables at c128, and the prefill chunk shrinks at + # TP=2 c64, TP=4 c128 and TP=2 c128. Every point is measured here rather + # than combined from the earlier run: the image move leaves concurrency + # 1-32 measured only on the superseded nightly-eed1f3d0 pin. + - { tp: 4, kv-offloading: none, spec-decoding: mtp, conc-list: [1, 2, 4, 8, 16, 32, 64, 128] } + - { tp: 2, kv-offloading: none, spec-decoding: mtp, conc-list: [1, 2, 4, 8, 16, 32, 64, 128] } diff --git a/docs/configuration-procedures.md b/docs/configuration-procedures.md index de9f1cda5..725e427a0 100644 --- a/docs/configuration-procedures.md +++ b/docs/configuration-procedures.md @@ -555,16 +555,17 @@ A configuration is ready for sweep only when the executable files agree, the exa ## DeepSeek-V4.1-Flash on MI355X -The draft `dsv41flash-fp4-mi355x-vllm-agentic-dspark` recipe extends [#2958](https://github.com/SemiAnalysisAI/InferenceX/pull/2958) to MI355X AgentX: TP4, concurrency 1–32, native five-token DSpark. Throughput uses the [committed golden AL](../golden_al_distribution/dsv41flash_dspark.yaml) of 3.51 for thinking on and five draft tokens, with synthetic rejection sampling and adaptive verification disabled. Accuracy evals retain real block rejection but, unlike the CUDA arms, also keep adaptive verification disabled: it trims verification requests on device, which the ROCm `DeepseekV4IndexerBackend` does not support, and the engine refused to start with it enabled ([run 34651830283](https://github.com/SemiAnalysisAI/InferenceX/actions/runs/34651830283)). FP4 describes the MXFP4 experts; the checkpoint also contains MXFP8 weights. +The `dsv41flash-fp4-mi355x-vllm-agentic-dspark` recipe extends [#2958](https://github.com/SemiAnalysisAI/InferenceX/pull/2958) to MI355X AgentX: TP4 and TP2, concurrency 1–128, native five-token DSpark. Throughput uses the [committed golden AL](../golden_al_distribution/dsv41flash_dspark.yaml) of 3.51 for thinking on and five draft tokens, with synthetic rejection sampling and adaptive verification disabled. Accuracy evals retain real block rejection but, unlike the CUDA arms, also keep adaptive verification disabled: it trims verification requests on device, which the ROCm `DeepseekV4IndexerBackend` does not support, and the engine refused to start with it enabled ([run 34651830283](https://github.com/SemiAnalysisAI/InferenceX/actions/runs/34651830283)). FP4 describes the MXFP4 experts; the checkpoint also contains MXFP8 weights. -Follow the AMD overrides in the merged [upstream recipe #968](https://github.com/vllm-project/recipes/pull/968): `VLLM_ROCM_USE_AITER=1`, `VLLM_ROCM_USE_AITER_MOE=1`, and `--moe-backend aiter`. The generic AITER selector lets vLLM pick the CK a8w4 experts, matching the DSV4-Pro MI355X recipe. The recipe pins `semianalysis_cc_traces_weka_062126` (the unfiltered corpus) via `WEKA_LOADER_OVERRIDE`. KV stays GPU-resident; Engram follows upstream AMD defaults. Do not copy the NVIDIA `--engram-config` option: upstream currently rejects it on ROCm. The MI355X launcher uses the shared HF cache and mounts this model's repository at `/ix`, and exports `INFMAX_CONTAINER_WORKSPACE=/ix` so AgentX dependencies and outputs resolve inside that mount. +Follow the AMD overrides in the merged [upstream recipe #968](https://github.com/vllm-project/recipes/pull/968): `VLLM_ROCM_USE_AITER=1`, `VLLM_ROCM_USE_AITER_MOE=1`, and `--moe-backend aiter`. The generic AITER selector lets vLLM pick the CK a8w4 experts, matching the DSV4-Pro MI355X recipe. The recipe pins `semianalysis_cc_traces_weka_062126` (the unfiltered corpus) via `WEKA_LOADER_OVERRIDE`. KV stays GPU-resident. Engram stayed on GPU under the upstream AMD defaults until [vllm-project/vllm#57491](https://github.com/vllm-project/vllm/pull/57491) widened the two `is_cuda()` gates to `is_cuda_alike()`. From that commit on, ROCm resolves an `EngramConfig` and `cpu_offload` defaults to on through `VLLM_PLE_CPU_OFFLOAD`, so the recipe sets `--engram-config` explicitly rather than leaning on that default. TP=2 always offloads, since the tables need 94.4 GiB per rank there; TP=4 keeps them resident through concurrency 64, where the KV pool is not the constraint, and offloads only at 128. The recipe likewise trims `--max-num-batched-tokens` only above concurrency 32, to 8192 at TP=2 c64 and TP=4 c128 and to 4096 at TP=2 c128, because the sparse-attention indexer and its companion per-rank buffers grow at roughly 4.4 MiB per batched token. Where that chunk falls below six times the API-server default of 1024 sequences, `--max-num-seqs` is capped at the graph-capture shape: DSpark verifies 1+5 tokens per sequence, and at 4096 against 1024 sequences the engram projection faults during profiling. The rule in every case is to spend device memory on KV only at the concurrencies that ran short of it, leaving the validated low-concurrency settings alone. Images built before that merge still reject the option on ROCm. The MI355X launcher uses the shared HF cache and mounts this model's repository at `/ix`, and exports `INFMAX_CONTAINER_WORKSPACE=/ix` so AgentX dependencies and outputs resolve inside that mount. -**GPU validation:** [Run 34710937012](https://github.com/SemiAnalysisAI/InferenceX/actions/runs/34710937012) passed the exact pinned image for throughput at concurrency 1, 2, 4, 8, 16, and 32, plus eval-only concurrency 32. The recipe uses `vllm/vllm-openai-rocm:nightly-eed1f3d0c6043bd494424a22443ee198dd56f657` (digest `sha256:960228cf…`, published 2026-09-12). The earlier `deepseekv41-flash-0909` tag predates [vllm-project/vllm#56503](https://github.com/vllm-project/vllm/pull/56503), which moves the mHC delayed pre block off the eager Torch reference and onto AITER; the merged [upstream recipe #968](https://github.com/vllm-project/recipes/pull/968) pins the same nightly and records the complete InferenceX command. Follow the [AgentX procedure](./eval-agentx-procedures.md#7-run-agentx-fast-feedback-versus-canonical-evidence) for future runtime evidence; local generation and registry metadata alone are not GPU proof. +**GPU validation:** The recipe uses `vllm/vllm-openai-rocm:nightly-rocm100-3df4ae153eb385e27b52f26c81f8edb9e20b9984` (digest `sha256:eccb72b7…`, published 2026-09-21) on the ROCm 10.0 nightly channel that `kimik3-fp4-mi355x-vllm-agentic-mtp` already runs on this cluster. The sweep in [#3326](https://github.com/SemiAnalysisAI/InferenceX/pull/3326) qualifies that pin across TP4 and TP2 at concurrency 1–128, and is the only evidence for it: [run 34710937012](https://github.com/SemiAnalysisAI/InferenceX/actions/runs/34710937012) covered TP4 concurrency 1–32 plus eval-only concurrency 32 on the superseded `nightly-eed1f3d0c6043bd494424a22443ee198dd56f657`, so its points do not carry onto this image. The merged [upstream recipe #1006](https://github.com/vllm-project/recipes/pull/1006) documents the MI355X TP2 Engram offload and `--no-swa-bounded-replay`, and the merged [#968](https://github.com/vllm-project/recipes/pull/968) records the original AMD overrides and the complete InferenceX command. Follow the [AgentX procedure](./eval-agentx-procedures.md#7-run-agentx-fast-feedback-versus-canonical-evidence) for future runtime evidence; local generation and registry metadata alone are not GPU proof. ## DeepSeek-V4.1-Flash on MI300X and MI325X `dsv41flash-fp4-mi300x-vllm-agentic-dspark` and `dsv41flash-fp4-mi325x-vllm-agentic-dspark` -copy the validated MI355X vLLM arm onto gfx942, on the same ROCm nightly and with the same +copy the validated MI355X vLLM arm onto gfx942, on the `nightly-eed1f3d0` ROCm nightly the +MI355X arm used before it moved to the `nightly-rocm100` channel, and with the same AMD overrides (`VLLM_ROCM_USE_AITER=1`, `VLLM_ROCM_USE_AITER_MOE=1`, `VLLM_USE_BREAKABLE_CUDAGRAPH=1`, `--moe-backend aiter`, adaptive verification off). gfx942 is not in the upstream hardware table, and it has no FP4 MFMA: the plain `aiter` MoE diff --git a/docs/configuration-procedures_zh.md b/docs/configuration-procedures_zh.md index b2e0ce2d6..612a997b6 100644 --- a/docs/configuration-procedures_zh.md +++ b/docs/configuration-procedures_zh.md @@ -496,16 +496,17 @@ python -m pytest utils/matrix_logic/ -v ## MI355X 上的 DeepSeek-V4.1-Flash -草案配方 `dsv41flash-fp4-mi355x-vllm-agentic-dspark` 将 [#2958](https://github.com/SemiAnalysisAI/InferenceX/pull/2958) 扩展至 MI355X AgentX:TP4、并发 1–32、原生五 token DSpark。吞吐测试使用[已提交的黄金 AL](../golden_al_distribution/dsv41flash_dspark.yaml):thinking 开启、五个草稿 token 对应 3.51,采用合成拒绝采样并关闭自适应验证。准确率 eval 保留真实块拒绝采样,但与 CUDA 分支不同,同样关闭自适应验证:它会在设备端裁剪验证请求,而 ROCm 的 `DeepseekV4IndexerBackend` 不支持该操作,启用后引擎拒绝启动([运行 34651830283](https://github.com/SemiAnalysisAI/InferenceX/actions/runs/34651830283))。FP4 表示 MXFP4 专家权重;检查点还包含 MXFP8 权重。 +配方 `dsv41flash-fp4-mi355x-vllm-agentic-dspark` 将 [#2958](https://github.com/SemiAnalysisAI/InferenceX/pull/2958) 扩展至 MI355X AgentX:TP4 与 TP2、并发 1–128、原生五 token DSpark。吞吐测试使用[已提交的黄金 AL](../golden_al_distribution/dsv41flash_dspark.yaml):thinking 开启、五个草稿 token 对应 3.51,采用合成拒绝采样并关闭自适应验证。准确率 eval 保留真实块拒绝采样,但与 CUDA 分支不同,同样关闭自适应验证:它会在设备端裁剪验证请求,而 ROCm 的 `DeepseekV4IndexerBackend` 不支持该操作,启用后引擎拒绝启动([运行 34651830283](https://github.com/SemiAnalysisAI/InferenceX/actions/runs/34651830283))。FP4 表示 MXFP4 专家权重;检查点还包含 MXFP8 权重。 -遵循已合并的[上游配方 #968](https://github.com/vllm-project/recipes/pull/968) 中的 AMD 设置:`VLLM_ROCM_USE_AITER=1`、`VLLM_ROCM_USE_AITER_MOE=1` 和 `--moe-backend aiter`。通用 AITER 选择器允许 vLLM 选择 CK a8w4 专家内核,与 DSV4-Pro MI355X 配方一致。配方通过 `WEKA_LOADER_OVERRIDE` 固定使用完整语料 `semianalysis_cc_traces_weka_062126`。KV 驻留 GPU;Engram 沿用上游 AMD 默认设置。不要复制 NVIDIA 的 `--engram-config` 选项:上游目前在 ROCm 上拒绝该选项。MI355X launcher 使用共享 HF 缓存,并将此模型的仓库挂载至 `/ix`,同时导出 `INFMAX_CONTAINER_WORKSPACE=/ix`,确保 AgentX 依赖与输出路径位于该挂载中。 +遵循已合并的[上游配方 #968](https://github.com/vllm-project/recipes/pull/968) 中的 AMD 设置:`VLLM_ROCM_USE_AITER=1`、`VLLM_ROCM_USE_AITER_MOE=1` 和 `--moe-backend aiter`。通用 AITER 选择器允许 vLLM 选择 CK a8w4 专家内核,与 DSV4-Pro MI355X 配方一致。配方通过 `WEKA_LOADER_OVERRIDE` 固定使用完整语料 `semianalysis_cc_traces_weka_062126`。KV 驻留 GPU。在 [vllm-project/vllm#57491](https://github.com/vllm-project/vllm/pull/57491) 将两处 `is_cuda()` 判断放宽为 `is_cuda_alike()` 之前,Engram 按上游 AMD 默认设置常驻 GPU。自该提交起,ROCm 会解析 `EngramConfig`,且 `cpu_offload` 经由 `VLLM_PLE_CPU_OFFLOAD` 默认开启,因此配方显式设置 `--engram-config`,而不依赖该默认值。TP=2 始终下放,因为此时表每 rank 需 94.4 GiB;TP=4 在并发 64 及以下保持常驻(此时 KV 池并非瓶颈),仅在并发 128 时下放。同样地,配方仅在并发高于 32 时调低 `--max-num-batched-tokens`:TP=2 c64 与 TP=4 c128 为 8192,TP=2 c128 为 4096,因为稀疏注意力 indexer 及其配套的每 rank 缓冲区按每个批量 token 约 4.4 MiB 增长。当该分块低于 API server 默认 1024 序列所需的六倍时,`--max-num-seqs` 会被限制为 CUDA graph 捕获的规模:DSpark 每序列验证 1+5 个 token,4096 对 1024 序列会使 engram 投影在 profiling 阶段崩溃。所有情况下的原则一致:只在确实出现 KV 不足的并发点上把设备内存让给 KV,保持低并发处已验证的设置不变。早于该合并的镜像在 ROCm 上仍会拒绝该选项。MI355X launcher 使用共享 HF 缓存,并将此模型的仓库挂载至 `/ix`,同时导出 `INFMAX_CONTAINER_WORKSPACE=/ix`,确保 AgentX 依赖与输出路径位于该挂载中。 -**GPU 验证:** [运行 34710937012](https://github.com/SemiAnalysisAI/InferenceX/actions/runs/34710937012) 使用精确固定的镜像,通过了并发 1、2、4、8、16、32 的吞吐测试以及仅评测并发 32。配方使用 `vllm/vllm-openai-rocm:nightly-eed1f3d0c6043bd494424a22443ee198dd56f657`(摘要 `sha256:960228cf…`,发布于 2026-09-12)。较早的 `deepseekv41-flash-0909` 标签早于 [vllm-project/vllm#56503](https://github.com/vllm-project/vllm/pull/56503),该 PR 将 mHC delayed pre 块从 eager Torch 参考实现切换到 AITER;已合并的[上游配方 #968](https://github.com/vllm-project/recipes/pull/968) 固定使用同一 nightly,并记录了完整的 InferenceX 命令。后续运行时证据请遵循 [AgentX 流程](./eval-agentx-procedures_zh.md);仅有本地矩阵生成和镜像元数据不能证明 GPU 验证完成。 +**GPU 验证:** 配方使用 `vllm/vllm-openai-rocm:nightly-rocm100-3df4ae153eb385e27b52f26c81f8edb9e20b9984`(摘要 `sha256:eccb72b7…`,发布于 2026-09-21),位于 ROCm 10.0 nightly 渠道,本集群上的 `kimik3-fp4-mi355x-vllm-agentic-mtp` 已在使用该渠道。[#3326](https://github.com/SemiAnalysisAI/InferenceX/pull/3326) 的 sweep 在 TP4 与 TP2、并发 1–128 下验证该镜像,且是其唯一证据:[运行 34710937012](https://github.com/SemiAnalysisAI/InferenceX/actions/runs/34710937012) 覆盖的是已被取代的 `nightly-eed1f3d0c6043bd494424a22443ee198dd56f657` 上 TP4 并发 1–32 的吞吐测试与仅评测并发 32,其数据点不能沿用到本镜像。已合并的[上游配方 #1006](https://github.com/vllm-project/recipes/pull/1006) 记录了 MI355X 的 TP2 Engram 卸载与 `--no-swa-bounded-replay`,已合并的 [#968](https://github.com/vllm-project/recipes/pull/968) 则记录了最初的 AMD 设置与完整的 InferenceX 命令。后续运行时证据请遵循 [AgentX 流程](./eval-agentx-procedures_zh.md);仅有本地矩阵生成和镜像元数据不能证明 GPU 验证完成。 ## MI300X 与 MI325X 上的 DeepSeek-V4.1-Flash `dsv41flash-fp4-mi300x-vllm-agentic-dspark` 与 `dsv41flash-fp4-mi325x-vllm-agentic-dspark` -将已验证的 MI355X vLLM 配方复制到 gfx942,使用同一 ROCm nightly 与相同的 AMD 设置 +将已验证的 MI355X vLLM 配方复制到 gfx942,使用 MI355X 迁移到 `nightly-rocm100` 通道之前所用的 +`nightly-eed1f3d0` ROCm nightly,以及相同的 AMD 设置 (`VLLM_ROCM_USE_AITER=1`、`VLLM_ROCM_USE_AITER_MOE=1`、`VLLM_USE_BREAKABLE_CUDAGRAPH=1`、 `--moe-backend aiter`、关闭自适应验证)。gfx942 不在上游硬件表中,且没有 FP4 MFMA:通用的 `aiter` MoE 后端允许 vLLM 选择器跳过仅 gfx950 可用的 CK a8w4 专家内核;若启动时所有候选均被 diff --git a/perf-changelog.yaml b/perf-changelog.yaml index 86eb8dbb6..077f2838d 100644 --- a/perf-changelog.yaml +++ b/perf-changelog.yaml @@ -8484,6 +8484,29 @@ - "Update B200 vLLM AgentX to DSpark6 and a new image with TP8 and DEP8 configurations." pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/3274 +- config-keys: + - dsv41flash-fp4-mi355x-vllm-agentic-dspark + scenario-type: + - agentic-coding + description: + - "Add a TP=2 search space alongside TP=4 for MI355X DSv4.1-Flash AgentX, and extend both to concurrency 128. TP=2 is feasible now that the Engram tables can live in host memory on ROCm rather than costing 47.2 GiB of device memory per rank, which previously forced four GPUs per server." + - "Offload the Engram tables only where the KV pool is short, not everywhere. vllm-project/vllm#57491 widened the two ROCm is_cuda() gates to is_cuda_alike(), so this image resolves an Engram config on gfx950 and the recipe sets the value explicitly instead of relying on the default. TP=2 always offloads: the tables cost 94.4 GiB per rank there against a 288 GiB card. TP=4 keeps them resident through c64, exactly as the validated concurrency 1-32 run measured, and offloads only at c128, where resident would leave 14.83M KV tokens, 116K per concurrent request, below the 122K at which TP=2 c64 collapsed. Offloaded lookups go to pinned host memory over UVA, so that cost is paid only where it buys KV room." + - "Trade prefill chunk size for KV room only above c32: --max-num-batched-tokens drops from the upstream 16384 to 8192 at TP=2 c64 and TP=4 c128, and to 4096 at TP=2 c128. The sparse-attention indexer and its companion per-rank buffers scale at roughly 4.4 MiB per batched token. KV tokens per concurrent request, which is what decides whether the prefix cache holds, rises from 116K to 388K at TP=4 c128, from 122K to 386K at TP=2 c64 and from 23.6K to 260K at TP=2 c128. Concurrency 1-32 keeps the upstream chunk untouched, where TP=2 measured 245K per request and TP=4 463K. B300 runs 8192 at TP=4 and the Blackwell TP=2 arms run 4096 (#3320, #3321)." + - "Cap --max-num-seqs at the graph-capture shape wherever the chunk falls below six times the MI355X API-server default of 1024 sequences, which is TP=2 c128 alone. DSpark verifies 1+5 tokens per sequence, so a decode batch of max_num_seqs needs six times that many token slots; at 4096 against 1024 sequences only four remain and the engram projection faults during profiling with HSA_STATUS_ERROR_EXCEPTION at M=1024, N=129280, K=256, reproduced on two separate GPU pairs. The same probes show why c128 never completed before: at the upstream 16384 with capture at 2048, TP=2 c128 held just 7.72 GiB, 3.02M KV tokens, 23.6K per request." + - "Sweep the full concurrency range 1 through 128 on both TP arms, sixteen points. The changes above leave concurrency 1-32 on the upstream settings, so the recipe there is unchanged, but its only measured points come from run 34710937012 on the superseded nightly-eed1f3d0 pin. Re-measuring them puts the whole MI355X AgentX curve on one image, and TP=2 has no measured point below c64 at all." + - "Move the image from vllm/vllm-openai-rocm:nightly-eed1f3d0c6043bd494424a22443ee198dd56f657 on the plain ROCm nightly channel to vllm/vllm-openai-rocm:nightly-rocm100-3df4ae153eb385e27b52f26c81f8edb9e20b9984 on the ROCm 10.0 nightly channel, published 2026-09-21T05:50:03Z with digest sha256:eccb72b74b8c04ce7406d9212e200b129f58e24be642795a857f755a94fdb3a1. kimik3-fp4-mi355x-vllm-agentic-mtp already runs that channel on this cluster. This sweep is the qualification for the new image; the previous green run is not evidence for it." + - "Drop the fixed max_num_seqs of 128 and use vLLM's MI355X API-server default of 1024, since a fixed 128 caps in-flight sequences at the outer concurrency once the sweep reaches c128. The CUDA graph ceiling stays at 1024 through c64 and rises to 2048 at c128, covering twice the outer concurrency. Supersedes PR #3111." + - "Pass --no-swa-bounded-replay. vllm-project/vllm#56227 added SWA bounded replay, default on, between the eed1f3d0 pin and this one. The window clamp it relies on landed in the FlashInfer and FlashMLA kernels, while the ROCm sparse SWA path only gained the replay_start kwarg, so on gfx950 the replayed tokens' slots are padded but the prefill kernels never stop at replay_start. Every TP=2 and TP=4 point of run 35567570539 died with HSA_STATUS_ERROR_MEMORY_FAULT at the first prefix hit carrying a replay start. Prefix caching itself stays on; drop this once ROCm clamps too." + - "为 MI355X DSv4.1-Flash AgentX 在 TP=4 之外新增 TP=2 搜索空间,并将两者的并发扩展到 128。ROCm 上 Engram 表现在可以驻留主机内存,不再占用每 rank 47.2 GiB 设备内存(此前这迫使每个服务器使用四张 GPU),因此 TP=2 可行。" + - "仅在 KV 池吃紧处下放 Engram 表,而非全面下放。vllm-project/vllm#57491 将 ROCm 上的两处 is_cuda() 判断放宽为 is_cuda_alike(),因此该镜像在 gfx950 上会解析 Engram 配置,recipe 显式设置该值而不再依赖默认行为。TP=2 始终下放:此时表每 rank 需 94.4 GiB,而单卡仅 288 GiB。TP=4 在 c64 及以下保持常驻,与已验证的并发 1-32 运行完全一致,仅在 c128 下放:此时若保持常驻只有 14.83M KV token,即每并发请求 116K,低于 TP=2 c64 崩溃时的 122K。下放后的查表经 UVA 访问锁页主机内存,因此只在能换来 KV 空间处才付出该代价。" + - "仅在并发高于 c32 时用 prefill 分块大小换取 KV 空间:--max-num-batched-tokens 从上游的 16384 降至 TP=2 c64 与 TP=4 c128 的 8192,以及 TP=2 c128 的 4096。稀疏注意力 indexer 及其配套的每 rank 缓冲区按每个批量 token 约 4.4 MiB 增长。决定前缀缓存能否维持的每并发请求 KV token 数随之提升:TP=4 c128 从 116K 升至 388K,TP=2 c64 从 122K 升至 386K,TP=2 c128 从 23.6K 升至 260K。并发 1-32 完全保留上游分块大小,实测 TP=2 每请求 245K、TP=4 463K。B300 在 TP=4 使用 8192,Blackwell 的 TP=2 搜索空间使用 4096(#3320、#3321)。" + - "当分块大小低于 MI355X API server 默认 1024 序列所需的六倍时,将 --max-num-seqs 限制为 CUDA graph 捕获的规模,实际只影响 TP=2 c128。DSpark 每个序列验证 1+5 个 token,因此 max_num_seqs 规模的 decode 批次需要六倍的 token 槽位;4096 对 1024 序列仅剩四个槽位,profiling 阶段 engram 投影会在 M=1024、N=129280、K=256 处以 HSA_STATUS_ERROR_EXCEPTION 崩溃,并在两组不同 GPU 上复现。同一批探测也解释了此前 c128 始终无法完成的原因:在上游的 16384 且捕获上限为 2048 时,TP=2 c128 仅有 7.72 GiB、3.02M KV token,即每请求 23.6K。" + - "在两个 TP 分支上完整扫描并发 1 至 128,共十六个点。上述改动在并发 1-32 区间保留上游设置,recipe 本身没有变化,但该区间仅有的实测点来自运行 34710937012,使用的是已被取代的 nightly-eed1f3d0 镜像。重新测量这些点可使整条 MI355X AgentX 曲线落在同一镜像上;此外 TP=2 在 c64 以下完全没有实测点。" + - "镜像从 plain ROCm nightly 渠道的 vllm/vllm-openai-rocm:nightly-eed1f3d0c6043bd494424a22443ee198dd56f657 切换到 ROCm 10.0 nightly 渠道的 vllm/vllm-openai-rocm:nightly-rocm100-3df4ae153eb385e27b52f26c81f8edb9e20b9984,该镜像于 2026-09-21T05:50:03Z 发布,digest 为 sha256:eccb72b74b8c04ce7406d9212e200b129f58e24be642795a857f755a94fdb3a1。本集群上的 kimik3-fp4-mi355x-vllm-agentic-mtp 已在使用该渠道。本次 sweep 即为新镜像的验证;此前的绿色运行不能作为其证据。" + - "移除固定的 max_num_seqs 128,改用 vLLM 在 MI355X API server 上的默认值 1024:扫描到 c128 时,固定 128 会把在途序列数限制在外层并发上。CUDA graph 上限在 c64 及以下保持 1024,c128 提升至 2048,以覆盖两倍外层并发。本 PR 取代 PR #3111。" + - "传入 --no-swa-bounded-replay。vllm-project/vllm#56227 在 eed1f3d0 与本次固定的镜像之间引入了默认开启的 SWA bounded replay。它所依赖的窗口钳制只落在 FlashInfer 与 FlashMLA 内核中,而 ROCm 稀疏 SWA 路径仅新增了 replay_start 参数,因此在 gfx950 上被重放 token 的 slot 会被填充,但 prefill 内核并不会在 replay_start 处停下。运行 35567570539 的所有 TP=2 与 TP=4 数据点都在首次带 replay start 的前缀命中时以 HSA_STATUS_ERROR_MEMORY_FAULT 崩溃。前缀缓存本身保持开启;待 ROCm 同样实现钳制后即可移除。" + pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/3326 + - config-keys: - dsv41flash-fp4-h200-vllm-agentic-dspark scenario-type: