[Klaud Cold] Add H100 Qwen3.8-27B-FP8 vLLM MTP 1k1k + 8k1k / 新增 H100 Qwen3.8-27B-FP8 vLLM MTP 1k1k + 8k1k 配方 - #3292
functionstackx wants to merge 8 commits into
Conversation
… 1k1k recipe on H100 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
functionstackx
added
full-sweep-fail-fast
qwen3.8-27b
20165dc to
80a6bf1
Compare
|
Thanks for the contribution!
中文感谢你的贡献！
|
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
I reviewed this PR and didn't find any confirmed bugs. Because it introduces a brand-new fp8 + native-checkpoint-MTP + hybrid-attention (GDN/Mamba) recipe with several precision- and config-sensitive knobs, a human look would still be worthwhile.
What was reviewed: the new script's env-var handling (EVAL_ONLY/RUN_EVAL/MAX_MODEL_LEN/SLURM_JOB_ID all use ${VAR:-...} fallbacks, checked against the repo's "no fallback defaults" rule and how sibling *_mtp.sh scripts handle the same inputs); the --kv-cache-dtype fp8 flag, checked against sibling GDN/Mamba hybrid-attention recipes which always pair it with --mamba-ssm-dtype bfloat16; the configs/nvidia-master.yaml entry and append-only perf-changelog.yaml addition (both structurally fine, pr-link: TBD is the usual pre-merge placeholder); and whether the checkpoint's native mtp.* head tensors could inherit the target model's fp8 quantization, which CONTRIBUTING.md's draft-precision section would forbid — I could not independently confirm from the diff alone that they stay unquantized.
Extended reasoning...
Overview
The diff adds a single new benchmark recipe: a shell script (benchmarks/single_node/fixed_seq_len/qwen3.827b_fp8_h100_vllm_mtp.sh) serving Qwen3.8-27B-FP8 on one H100 via vLLM TP1 using the checkpoint's native MTP head, a matching configs/nvidia-master.yaml entry, and an appended perf-changelog.yaml record. No source/library code, tests, or CI is touched.
Security risks
None identified — this is a benchmark shell script and two config/data files, with no auth, crypto, or untrusted-input handling. check_env_vars, set -eo pipefail, and hf download usage all follow existing patterns.
Level of scrutiny
I independently cross-checked the diff against AGENTS.md's explicit "no fallback defaults" rule (with its literal ${IS_MULTINODE:-true} counter-example) and against every sibling fixed_seq_len/*_mtp.sh script. Two things stood out as plausible real issues even though the automated hunt listed them as "investigated and ruled out": (1) sibling scripts reference ${EVAL_ONLY}/${RUN_EVAL} directly (these are always exported by the harness per infx/matrix/validation.py and the workflow templates), while this script adds ${EVAL_ONLY:-false}/${RUN_EVAL:-false}/${MAX_MODEL_LEN:-...} fallbacks that match the exact forbidden pattern; (2) every other hybrid GDN/Mamba recipe in the repo (qwen3.5_fp8_h100_mtp.sh, qwen3.5_fp4_b200_mtp.sh, etc.) pairs --kv-cache-dtype fp8_e4m3 with --mamba-ssm-dtype bfloat16, but this new recipe sets only --kv-cache-dtype fp8 with no Mamba/GDN override. Given the model's own PR description states 48 linear-attention layers, this divergence from an otherwise-universal sibling pattern is worth a maintainer's confirmation rather than my own unilateral call, especially since I could not verify from the diff whether vLLM's fp8 kv-cache setting reaches the GDN state for this model or whether it's a non-issue there.
Other factors
This is also the kind of PR CONTRIBUTING.md flags for elevated scrutiny (draft/MTP precision must remain unquantized even when the target model is quantized) — the recipe relies on the checkpoint's own mtp.* tensors, and I could not confirm from the diff alone whether the FP8 quantization applied to the target weights also touches those tensors. Combined with the label full-sweep-fail-fast (GPU validation pending) and the fact that this is a genuinely new recipe shape (native MTP + fp8 + hybrid attention on H100), a human maintainer's sign-off is the right last step regardless of the automated hunt's conclusion.
This review covers commit 20165dc, which is no longer the latest commit on this pull request; later commits are not covered by it.
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=35501514559 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=35501514559 |
|
/reuse-sweep-run 35465793793 |
Sync with origin/main after the green sweep run 35465793793; the reuse gate authorizes that run on this head. 在绿色 sweep 运行 35465793793 之后与 origin/main 同步；reuse gate 在此 head 上授权该运行。 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
应用 thinking-on 黄金 AL 2.52，保持原始 BF16 原生 MTP 头及草稿 KV cache，并在精度评测路径使用真实验证。
同步 main，逐字节保留 changelog 历史，并在末尾保留本 PR 的原始条目和黄金 AL 更新。
使用官方 --no-enable-flashinfer-autotune 选项规避原生 BF16 MTP 的 FA3 scheduler metadata 初始化错误。
记录已验证的 Qwen 原生 MTP autotune 初始化问题，并链接中英文处理说明及运行证据。
保留 main 新增的诊断记录与 Qwen MTP 的已验证处理说明。
在 H100 的 Qwen3.8-27B-FP8 原生 MTP 配方中新增 8192/1024、TP1、并发 1-128；保留 1k1k、thinking-on AL 2.52 和真实精度验证。
Oseltamivir
changed the title
This branch has not been deployed
2 participants
Goal: Add
qwen3.827b-fp8-h100-vllm-mtpon H100, using native MTP3 with the measured thinking-on synthetic AL 2.52 for throughput.Baseline: N/A — new model/SKU recipe; the acceptance reference is #3304 and SPEED-Bench run 35492787451.
vllm/vllm-openai:nightly-cd10ed6f9f6b37a8ace9cf380007e66fe12ec0c3· TP1 · 1k/1k + 8k/1k · concurrency 1–128. The recipe readsthinking_on[3]fromgolden_al_distribution/qwen3.827b_fp8_mtp.yamland sendsrejection_sample_method=synthetic,synthetic_acceptance_length=2.52to vLLM. The pinned Qwen chat template enables thinking; the script requiresTHINKING_MODE=thinking_on.The FP8 target is pinned to
017b9c7af6b5689d5dd426a76e0bc077eb5ca20a. Its native MTP head comes from the original BF16Qwen/Qwen3.8-27Bat1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0, with BF16 draft KV cache and explicit MTP quantization exclusions. Eval-only and combined throughput+eval paths use real standard verification. The shared helper rejects unmeasured draft lengths outside 1–4 and missing or invalid AL values.Validation: CI Tests and Lint, exact-key matrix generation, Bash syntax, and changelog byte-prefix preservation passed. The fresh full sweep passed all 18 GPU jobs: eight 1k/1k points, eight 8k/1k points, and two separate accuracy evals. Both curves cover concurrency 1, 2, 4, 8, 16, 32, 64 and 128. All eight 8k/1k artifacts were checked for the workload identity, positive throughput, synthetic AL 2.52, and unquantized native MTP linear modules. Both real-verification GSM8K evals scored 97.04% on all 1,319 questions (concurrency 64 and 128). Validated full sweep 35501514559.
The CUDA recipe uses
--no-enable-flashinfer-autotuneto avoid the pinned image's MTP dummy-prefill scheduler-metadata error. The existing reuse authorization predates the AL and 8k/1k changes; select the new validated full sweep before merging.AI model disclosure
AL integration, 8k/1k coverage and validation: GPT-6; the exact runtime model/version identifier could not be verified. The original PR's AI model/version was not recorded and could not be verified. No delegated agents contributed to this update.
中文
**目标：**在 H100 上新增
qwen3.827b-fp8-h100-vllm-mtp，使用原生 MTP3，并在吞吐测试中应用实测的 thinking-on 合成 AL 2.52。**基线：**N/A——新增模型/SKU 配方；接受长度依据 #3304 和 SPEED-Bench 运行 35492787451。
vllm/vllm-openai:nightly-cd10ed6f9f6b37a8ace9cf380007e66fe12ec0c3· TP1 · 1k/1k + 8k/1k · 并发 1–128。配方从golden_al_distribution/qwen3.827b_fp8_mtp.yaml读取thinking_on[3]，向 vLLM 传入rejection_sample_method=synthetic、synthetic_acceptance_length=2.52。固定版本的 Qwen chat template 默认开启 thinking，脚本要求THINKING_MODE=thinking_on。FP8 目标固定为 revision
017b9c7af6b5689d5dd426a76e0bc077eb5ca20a。原生 MTP 头来自原始 BF16Qwen/Qwen3.8-27B，revision 为1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0；草稿 KV cache 保持 BF16，并显式排除 MTP 量化。仅精度评测以及吞吐后紧接精度评测的路径均使用真实的标准验证。共享辅助程序拒绝超出 1–4 范围的草稿长度，以及缺失或无效的 AL。验证：CI Tests 和 Lint、精确 key 矩阵生成、Bash 语法和 changelog 历史字节保留检查均通过。新的完整 sweep 的 18 个 GPU 任务全部通过：八个 1k/1k 点、八个 8k/1k 点，以及两个独立精度评测。 两条曲线均覆盖并发 1、2、4、8、16、32、64、128。八个 8k/1k artifact 均已核实工作负载标识、有效吞吐、合成 AL 2.52 和未量化的原生 MTP 线性层。并发 64 和 128 的真实验证 GSM8K 评测均完成全部 1,319 道题，准确率均为 97.04%。验证通过的完整 sweep 35501514559。
CUDA 配方使用
--no-enable-flashinfer-autotune，避免固定镜像中 MTP dummy prefill 的 scheduler metadata 错误。现有复用授权早于 AL 和 8k/1k 更新；合并前应选择新验证通过的完整 sweep。AI 模型披露：AL 集成、8k/1k 覆盖和验证由 GPT-6 完成；无法核实运行时精确模型及版本标识。原始 PR 未记录 AI 模型及版本，无法核实。本次更新未使用委派代理。