[Klaud Cold] Add MI325X Qwen3.8-27B-FP8 vLLM MTP 1k1k + 8k1k / 新增 MI325X Qwen3.8-27B-FP8 vLLM MTP 1k1k + 8k1k 配方 - #3295
functionstackx wants to merge 5 commits into
Conversation
…TP 1k1k recipe on MI325X Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
functionstackx
added
full-sweep-fail-fast
qwen3.8-27b
a554869 to
4700faa
Compare
|
Thanks for the contribution!
中文感谢你的贡献！
|
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Beyond the inline pr-link: TBD finding, I checked the MAX_MODEL_LEN/EVAL_ONLY/RUN_EVAL fallback patterns AGENTS.md flags — these are optional, non-required internal values (not in check_env_vars), matching sibling scripts' derived-context conventions, so not a caller-config-fallback violation. Also confirmed the script writes directly into /workspace/ with no mkdir calls, so no new-directory concern.
Extended reasoning...
A CONFIRMED, merge-blocking finding (perf-changelog.yaml pr-link left as TBD despite the PR already existing) is already queued as an inline comment, so this review should not approve. I independently verified the finding is real by inspecting the tail of perf-changelog.yaml, and checked a couple of other AGENTS.md-flagged risk areas from the task description that turned out fine: the MAX_MODEL_LEN/EVAL_ONLY/RUN_EVAL optional-default patterns are consistent with how sibling *_mtp.sh scripts handle optional, non-required inputs (none of these are declared in check_env_vars, so they aren't caller-supplied config being silently defaulted — they're internal derived/optional values, which AGENTS.md explicitly permits), and the script has no mkdir calls and writes only to existing /workspace paths, matching sibling scripts, so the "no new directories under /workspace" convention is not violated. I did not find anything new or actionable beyond the already-reported finding, so per the relationship-management guidance this narrow ruled-out note is the appropriate level of comment rather than a full defer/approve verdict.
| description: | ||
| - "Add the MI325X Qwen3.8-27B-FP8 recipe: vLLM nightly-a8d1aa9c99b8698a2a78b611b7a10c30e6b3995b on one GPU (TP1), 1k1k only, with the checkpoint's native MTP head (mtp_num_hidden_layers=1, --speculative-config method mtp) drafting three tokens as the vLLM recipe page prescribes, Triton attention with a bf16 KV cache (gfx942 has no calibrated fp8 attention scales), --max-num-seqs sized to the sweep point (floor 16; 256 for accuracy evals), prefix caching disabled, the matrix context as --max-model-len, --language-model-only, and chat-templated prompts; concurrency 1-128" | ||
| - "新增 MI325X Qwen3.8-27B-FP8 配方：vLLM nightly-a8d1aa9c99b8698a2a78b611b7a10c30e6b3995b 单卡（TP1），仅 1k1k，使用 checkpoint 自带的原生 MTP 头（mtp_num_hidden_layers=1，--speculative-config method mtp）按 vLLM recipe 页面建议预测 3 个 token，Triton attention 与 bf16 KV cache（gfx942 无校准的 fp8 attention scale），--max-num-seqs 按并发点设定（下限 16；精度评测为 256），关闭 prefix caching，--max-model-len 使用矩阵上下文，--language-model-only，提示词经 chat template 处理；并发 1-128" | ||
| pr-link: TBD |
There was a problem hiding this comment.
🔴 Merging this PR will fail: perf-changelog.yaml:8456 keeps pr-link: TBD though the PR already exists, and merge tooling never accepts TBD once a PR number is known. PR_LINK_PLACEHOLDERS in infx/workflows/validate_perf_changelog.py only allows 'XXX' or the XXX pull URL; validate_added_pr_link (via compare_entries, used by prepare_perf_changelog_merge.py's canonicalize step behind merge_with_reuse.sh) raises ChangelogValidationError for 'TBD'. docs/configuration-procedures.md:495,513 require replacing TBD right after PR creation. Fix: set pr-link to the real PR URL (or 'XXX') before merge/sweep.
Extended reasoning...
PR is already open (per PR metadata) but perf-changelog.yaml:8456 still reads pr-link: TBD. At merge time merge_with_reuse.sh runs prepare_perf_changelog_merge.py canonicalize --pr-number N. canonicalize_appended_links() calls compare_entries(base_entries, head_entries, pr_number) with a real int pr_number. compare_entries() calls validate_added_pr_link(link='TBD', pr_number); since pr_number is not None it requires link in PR_LINK_PLACEHOLDERS={'XXX','.../pull/XXX'} or link==expected URL. 'TBD' matches neither, so ChangelogValidationError is raised before any replacement logic runs. The PR-open check-changelog gate (validate_matrix_compatible_change, same file) never calls compare_entries, so it passes and gives false confidence that TBD is fine. Only the merge-preparation step exercises compare_entries with a real pr_number, so the failure only surfaces at merge time, blocking the automated squash-merge described in docs/ci-procedures.md.
Verification: normal. The new appended entry ends at perf-changelog.yaml:8456 with pr-link: TBD (last line of the file). At merge, utils/merge_with_reuse.sh:169-173 runs prepare_perf_changelog_merge canonicalize --pr-number "$PR" with a real int. canonicalize_appended_links() calls compare_entries(base, head, pr_number), which for each addition calls validate_added_pr_link(str(entry.get("pr-link")),…
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=35501528612 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=35501528612 |
|
/reuse-sweep-run 35465817969 |
Sync with origin/main after the green sweep run 35465817969; the reuse gate authorizes that run on this head. 在绿色 sweep 运行 35465817969 之后与 origin/main 同步；reuse gate 在此 head 上授权该运行。 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
应用 thinking-on 黄金 AL 2.52，保持原始 BF16 原生 MTP 头及草稿 KV cache，并在精度评测路径使用真实验证。
同步 main，逐字节保留 changelog 历史，并在末尾保留本 PR 的原始条目和黄金 AL 更新。
在 MI325X 的 Qwen3.8-27B-FP8 原生 MTP 配方中新增 8192/1024、TP1、并发 1-128；保留 1k1k、thinking-on AL 2.52 和真实精度验证。
Oseltamivir
changed the title
This branch has not been deployed
2 participants
Goal: Add
qwen3.827b-fp8-mi325x-vllm-mtpon MI325X, using native MTP3 with the measured thinking-on synthetic AL 2.52 for throughput.Baseline: N/A — new model/SKU recipe; the acceptance reference is #3304 and SPEED-Bench run 35492787451.
vllm/vllm-openai-rocm:nightly-a8d1aa9c99b8698a2a78b611b7a10c30e6b3995b· TP1 · 1k/1k + 8k/1k · concurrency 1–128. The recipe readsthinking_on[3]fromgolden_al_distribution/qwen3.827b_fp8_mtp.yamland sendsrejection_sample_method=synthetic,synthetic_acceptance_length=2.52to vLLM. The pinned Qwen chat template enables thinking; the script requiresTHINKING_MODE=thinking_on.The FP8 target is pinned to
017b9c7af6b5689d5dd426a76e0bc077eb5ca20a. Its native MTP head comes from the original BF16Qwen/Qwen3.8-27Bat1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0, with BF16 draft KV cache and explicit MTP quantization exclusions. Eval-only and combined throughput+eval paths use real standard verification. The shared helper rejects unmeasured draft lengths outside 1–4 and missing or invalid AL values.Validation: CI Tests and Lint, exact-key matrix generation, Bash syntax, and changelog byte-prefix preservation passed. The fresh full sweep passed all 18 GPU jobs: eight 1k/1k points, eight 8k/1k points, and two separate accuracy evals. Both curves cover concurrency 1, 2, 4, 8, 16, 32, 64 and 128. All eight 8k/1k artifacts were checked for the workload identity, positive throughput, synthetic AL 2.52, and unquantized native MTP linear modules. Real-verification GSM8K scored 97.50% at concurrency 64 and 97.42% at concurrency 128, with all 1,319 questions in each eval. Validated full sweep 35501528612.
The existing reuse authorization predates the AL and 8k/1k changes; select the new validated full sweep before merging.
AI model disclosure
AL integration, 8k/1k coverage and validation: GPT-6; the exact runtime model/version identifier could not be verified. The original PR's AI model/version was not recorded and could not be verified. No delegated agents contributed to this update.
中文
**目标：**在 MI325X 上新增
qwen3.827b-fp8-mi325x-vllm-mtp，使用原生 MTP3，并在吞吐测试中应用实测的 thinking-on 合成 AL 2.52。**基线：**N/A——新增模型/SKU 配方；接受长度依据 #3304 和 SPEED-Bench 运行 35492787451。
vllm/vllm-openai-rocm:nightly-a8d1aa9c99b8698a2a78b611b7a10c30e6b3995b· TP1 · 1k/1k + 8k/1k · 并发 1–128。配方从golden_al_distribution/qwen3.827b_fp8_mtp.yaml读取thinking_on[3]，向 vLLM 传入rejection_sample_method=synthetic、synthetic_acceptance_length=2.52。固定版本的 Qwen chat template 默认开启 thinking，脚本要求THINKING_MODE=thinking_on。FP8 目标固定为 revision
017b9c7af6b5689d5dd426a76e0bc077eb5ca20a。原生 MTP 头来自原始 BF16Qwen/Qwen3.8-27B，revision 为1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0；草稿 KV cache 保持 BF16，并显式排除 MTP 量化。仅精度评测以及吞吐后紧接精度评测的路径均使用真实的标准验证。共享辅助程序拒绝超出 1–4 范围的草稿长度，以及缺失或无效的 AL。验证：CI Tests 和 Lint、精确 key 矩阵生成、Bash 语法和 changelog 历史字节保留检查均通过。新的完整 sweep 的 18 个 GPU 任务全部通过：八个 1k/1k 点、八个 8k/1k 点，以及两个独立精度评测。 两条曲线均覆盖并发 1、2、4、8、16、32、64、128。八个 8k/1k artifact 均已核实工作负载标识、有效吞吐、合成 AL 2.52 和未量化的原生 MTP 线性层。真实验证 GSM8K 在并发 64 时准确率为 97.50%，并发 128 时为 97.42%，每次评测均完成全部 1,319 道题。验证通过的完整 sweep 35501528612。
现有复用授权早于 AL 和 8k/1k 更新；合并前应选择新验证通过的完整 sweep。
AI 模型披露：AL 集成、8k/1k 覆盖和验证由 GPT-6 完成；无法核实运行时精确模型及版本标识。原始 PR 未记录 AI 模型及版本，无法核实。本次更新未使用委派代理。