Add H100 TensorRT-LLM ModelScope coverage / 添加 H100 TensorRT-LLM ModelScope 覆盖 - #3323
functionstackx wants to merge 4 commits into
Conversation
为 H100 添加 TensorRT-LLM ModelScope 覆盖，使用 Qwen3-0.6B BF16 和与 1.3.0rc27 镜像源码匹配的回移补丁。
记录 PR #3323 的 TensorRT-LLM ModelScope 运行时补丁豁免和移除计划。
|
Thanks for the contribution!
中文感谢你的贡献！
|
|
Claude finished @functionstackx's task in 6m 45s —— View job PR Review: H100 TensorRT-LLM ModelScope coverage
SummaryOne 🟡 warning posted inline on Everything else checked out: the image is public NGC, the entry is single-node on Two non-blocking notes:
Not verifiedThis sandbox blocked Python execution and all outbound fetches ( 中文已在 其余检查均通过：镜像为公开 NGC、配置为单节点 由于沙箱禁止 Python 执行与外网访问，未能运行矩阵生成器、changelog 验证器，也无法比对上游 |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| if ignore_patterns: | ||
| kwargs["ignore_file_pattern"] = ignore_patterns | ||
| if allow_patterns: | ||
| kwargs["allow_file_pattern"] = allow_patterns |
There was a problem hiding this comment.
🟡 WARNING: Hugging Face glob patterns are forwarded to ModelScope's regex-based filters.
Why it matters: ignore_file_pattern / allow_file_pattern in modelscope.hub.snapshot_download are matched with re.search and patterns that fail re.compile are silently skipped (this repo's own infx/bench_serving/backend_request_func.py:412 uses regex syntax for that kwarg for this reason). original/**/* raises multiple repeat, so the ignore filter becomes a no-op. Worse, any glob such as *.json passed through download_hf_partial raises nothing to repeat, leaving the allow list with zero valid regexes, so ModelScope downloads no files and the caller gets an empty snapshot. Harmless for Qwen/Qwen3-0.6B's full download today, but wrong for the partial path this boundary claims to support.
Fix: use ModelScope's fnmatch-compatible kwargs, which share Hugging Face semantics.
| if ignore_patterns: | |
| kwargs["ignore_file_pattern"] = ignore_patterns | |
| if allow_patterns: | |
| kwargs["allow_file_pattern"] = allow_patterns | |
| if ignore_patterns: | |
| kwargs["ignore_patterns"] = ignore_patterns | |
| if allow_patterns: | |
| kwargs["allow_patterns"] = allow_patterns |
There was a problem hiding this comment.
Fixed in 1e31a68. The backport now uses ModelScope native ignore_patterns / allow_patterns, matching SGLang and ModelScope 1.40.1 glob semantics. I made the same correction in SemiAnalysisAI/TensorRT-LLM#2 at 677d37c443, updated its tests and minimum documented ModelScope version, and passed the local source-matched idempotency/compile checks again. The benchmark client now also receives --tokenizer "$MODEL_PATH", so it reuses the ModelScope snapshot instead of independently resolving the tokenizer through Hugging Face.
中文
已在 1e31a68 中修复。回移补丁现在使用 ModelScope 原生的 ignore_patterns / allow_patterns，与 SGLang 和 ModelScope 1.40.1 的 glob 语义一致。同一修复已提交到 SemiAnalysisAI/TensorRT-LLM#2（677d37c443），并更新测试和最低 ModelScope 版本文档，再次通过与镜像源码匹配的幂等与编译检查。基准客户端现在也传入 --tokenizer "$MODEL_PATH"，复用 ModelScope 快照，不再单独通过 Hugging Face 解析 tokenizer。
改用 ModelScope 原生 glob 过滤参数，并让基准客户端复用 ModelScope 本地 tokenizer。
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=35538521127 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=35538521127 |
There was a problem hiding this comment.
Beyond the inline findings, I also checked whether main() patching utils.py and then failing on llm.py (patch_trtllm_modelscope.py:224-227) could leave a silently-corrupted install across runs: patch_utils's own "partial ModelScope patch found" guard (line 158) trips on any retry, and the pinned image is rebuilt fresh per run, so this doesn't compound unnoticed — ruling it out as a separate concern from the reported findings.
Extended reasoning...
Two CONFIRMED findings are already queued as inline comments (the glob-vs-regex mismatch when forwarding ignore_patterns/allow_patterns into ModelScope's snapshot_download, and the missing set -e/set -eo pipefail in the benchmark script that lets a failed version check or patch step continue silently). Given those findings, per the review protocol I should not approve, and should not restate them in the body. I independently verified the one candidate issue already ruled out by the earlier bug-hunt pass — main() applying patch_utils and patch_llm sequentially with no rollback (runners/patch_trtllm_modelscope.py:224-233) — by reading patch_utils's partial-patch detection (source.count check at line 156-159) and confirming it fails loudly on any subsequent invocation against a half-patched tree, and that the benchmark's runtime is a freshly-provisioned pinned container per run rather than a long-lived mutable install, so a mixed-patch state cannot persist across benchmark runs. This is worth surfacing briefly per the narrow "ruled out beyond the findings" exception, since it's a concrete check beyond what's already flagged inline.
Findings marked 🟡 are optional suggestions and need no follow-up push.
| def _snapshot_download(model: str, | ||
| revision: Optional[str] = None, | ||
| ignore_patterns: Optional[List[str]] = None, | ||
| allow_patterns: Optional[List[str]] = None) -> str: | ||
| """Download a snapshot from ModelScope or Hugging Face. | ||
|
|
||
| ModelScope uses different names for its file filters. Keep the optional | ||
| import in this boundary so standard TensorRT-LLM installations do not need | ||
| the ``modelscope`` package. | ||
| """ | ||
| local_files_only = huggingface_hub.constants.HF_HUB_OFFLINE | ||
| if use_modelscope(): | ||
| try: | ||
| from modelscope.hub.snapshot_download import snapshot_download | ||
| except ImportError as error: | ||
| raise ImportError( | ||
| "TRTLLM_USE_MODELSCOPE is enabled, but ModelScope is not " | ||
| "installed. Install it with `pip install modelscope`.") from error | ||
|
|
||
| kwargs = { | ||
| "model_id": model, | ||
| "local_files_only": local_files_only, | ||
| "revision": revision, | ||
| } | ||
| if ignore_patterns: | ||
| kwargs["ignore_file_pattern"] = ignore_patterns | ||
| if allow_patterns: | ||
| kwargs["allow_file_pattern"] = allow_patterns |
There was a problem hiding this comment.
🔴 Enabling TRTLLM_USE_MODELSCOPE can crash model downloads instead of completing. _snapshot_download forwards HF's glob-style ignore_patterns/allow_patterns straight into ModelScope's ignore_file_pattern/allow_file_pattern (lines 117, 119), but ModelScope treats these as regex, not glob (repo already does this at infx/bench_serving/backend_request_func.py:412 with ..pt). download_hf_model always sends ignore_patterns=[original/**/] (line 55); as regex that raises re.error: multiple repeat, and allow_patterns starting with * fail the same way. Fix: convert HF glob patterns to ModelScope regex before setting these kwargs, at both call sites.
Extended reasoning...
download_hf_model in NEW_DOWNLOAD_BLOCK (patch_trtllm_modelscope.py:55) hardcodes ignore_patterns=[original//], an HF fnmatch glob. When TRTLLM_USE_MODELSCOPE=true, _snapshot_download (line 92) copies it verbatim into kwargs[ignore_file_pattern] at line 117 and calls modelscope's snapshot_download(**kwargs). ModelScope compiles ignore_file_pattern/allow_file_pattern as regex, not glob, confirmed by repo code already passing regex forms like ..pt at infx/bench_serving/backend_request_func.py:412. original//* has two consecutive * with nothing between them, which Python re rejects as multiple repeat at compile time, so the download call raises instead of returning a path. This hits every ModelScope download: the script's preflight download and every trtllm-serve call once TRTLLM_USE_MODELSCOPE is exported. Any TensorRT-LLM allow_patterns starting with * (e.g. *.json) hits nothing to repeat the same way.
Verification: normal. The patch forwards HF fnmatch globs into ModelScope's regex-based file filters, which crashes the ModelScope download. Decisive citations: (1) The repo's own ModelScope call at infx/bench_serving/backend_request_func.py:412 passes regex-style values ignore_file_pattern=[".*.pt", ".*.safetensors", ".*.bin"] (regex .*, not glob *) — confirming this repo treats ModelScope's…
| python3 - <<'PY' | ||
| from importlib.metadata import version | ||
|
|
||
| expected = "1.3.0rc27" | ||
| actual = version("tensorrt_llm") | ||
| if actual != expected: | ||
| raise SystemExit( | ||
| f"Expected TensorRT-LLM {expected} for the pinned NGC image, got {actual}" | ||
| ) | ||
| PY | ||
|
|
||
| python3 -m pip install --quiet --disable-pip-version-check \ | ||
| "modelscope==1.40.1" "modelscope-hub==0.4.3" | ||
| python3 "$(dirname "$0")/../../../runners/patch_trtllm_modelscope.py" |
There was a problem hiding this comment.
🟡 (optional) Operators get a run that silently continues past a bad engine version or a failed ModelScope patch instead of stopping, because the script never sets set -e/set -eo pipefail (sibling scripts like minimaxm3_fp8_h200_mtp.sh:2 do). The pinned-version check at lines 22-31 raises SystemExit on mismatch and the patcher at line 35 exits 1 on an unsupported source tree, but both are plain commands whose non-zero exit is ignored by bash without set -e, so trtllm-serve later starts against an unvalidated or unpatched TensorRT-LLM. Fix: add set -eo pipefail near the top (or explicit || exit 1 after each) so the version check and the patch's fail-closed logic actually stop the run.
Extended reasoning...
No set -e or set -eo pipefail appears anywhere before line 92's set -x, unlike minimaxm3_fp8_h200_mtp.sh which sets it at line 2 right after the shebang. Step 1: the heredoc at lines 22-31 checks the installed tensorrt_llm version and calls raise SystemExit(msg) when it does not equal 1.3.0rc27; python3 exits 1 and prints to stderr, but bash just moves to the next line. Step 2: pip install of modelscope at line 33-34 runs regardless. Step 3: the patcher at line 35 (runners/patch_trtllm_modelscope.py) can itself exit 1 via its fail-closed RuntimeError path (e.g. unsupported source tree) — again ignored. Step 4: TRTLLM_USE_MODELSCOPE is exported and trtllm-serve is launched later in the script against an engine that was never validated and may not actually be patched, instead of the run aborting immediately with the clear diagnostic message the check was designed to produce.
Verification: normal. The new script benchmarks/single_node/fixed_seq_len/qwen3-0.6b_bf16_h100_trt.sh has no set -e/set -eo pipefail; the only set is set -x at line 92. The sourced benchmark_lib.sh applies set -e only inside certain functions (3464-3541), and check_env_vars (the sole lib call before line 22) does not, so errexit is inactive at lines 22-35. Two fail-closed gates therefore have their…
Document the full-split sample audit, external scale context, and conservative 0.60 model-specific floor while preserving deterministic evaluation settings. 为 Qwen3-0.6B 记录完整测试集分析、外部规模参考和保守的 0.60 GSM8K 下限，保持确定性评测设置不变。 Signed-off-by: functionstackx <47992694+functionstackx@users.noreply.github.com>
This branch has not been deployed
1 participant
Summary
Add BF16 H100 coverage for
Qwen/Qwen3-0.6Busing ModelScope andnvcr.io/nvidia/tensorrt-llm/release:1.3.0rc27. The source-matched runtime backport of TensorRT-LLM #2 uses scoped snapshot and path resolution. Both the server and benchmark tokenizer use the ModelScope snapshot.The image matches
v1.3.0rc27, commit6e1cc953c071b8a9055b03ef2ae4ee0bc4c645c4. Runtime dependencies are pinned tomodelscope==1.40.1andmodelscope-hub==0.4.3. The engine-patch waiver records provenance and removal when an NGC image includes the integration.Evaluation policy
The initial H100 sweep passed all five throughput points. GSM8K completed all 1,319 questions at concurrency 64 with 0.6717 strict / 0.6785 flexible accuracy, then failed the global 0.90 threshold; fail-fast cancelled concurrency 32.
Set
models.qwen3-0.6b.gsm8kto a conservative 0.60 integration regression floor. Preserve five-shot chat, deterministic sampling, the 5,376-token budget, and both scored metrics. The policy and sample audit document formatting/repetition errors, the 7.17-point strict-score margin, and Qwen's greedy-decoding caveat. Qwen's published 59.59% result is for the base model with four-shot CoT and is scale context only; no directly comparable published chat baseline was established. This floor is not a claim of optimal model quality.Validation
git diff --checkpassed.v1.3.0rc27source: first application patched, second was idempotent, and patched modules compiled.916278d18: all five throughput points and both evals. All 1,170 benchmark requests completed with zero failures and valid power data. Tests, lint, and Python CodeQL passed.Both evals completed all 1,319 questions with no empty responses and the same unchanged deterministic settings. Raw samples, result metadata, score validation, and all seven server logs were inspected. Each server used TensorRT-LLM
1.3.0rc27and loaded weights from the ModelScope snapshot; the canary job log confirms the benchmark tokenizer used that same snapshot. No server errors or Hugging Face model fallback appeared in the logs.AI model disclosure
Earlier work was disclosed as GPT-5; that earlier exact identifier could not be independently verified in this session. This continuation used Codex to inspect artifacts, document the design and evaluation policy, and validate the PRs; its exact runtime model/version could not be verified. No delegated agents were used.
中文
摘要
使用 ModelScope 和
nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc27，为Qwen/Qwen3-0.6B添加 BF16 H100 覆盖。TensorRT-LLM #2 的源码匹配回移补丁仅在指定的快照下载和路径解析边界工作。服务端与 benchmark tokenizer 均使用 ModelScope 快照。镜像对应
v1.3.0rc27，提交6e1cc953c071b8a9055b03ef2ae4ee0bc4c645c4。运行时依赖固定为modelscope==1.40.1和modelscope-hub==0.4.3。引擎补丁豁免记录 说明了来源，以及 NGC 镜像包含该集成后的移除方案。评测策略
首次 H100 sweep 的五个吞吐点全部通过。并发 64 的 GSM8K 完成全部 1,319 题，strict / flexible 准确率为 0.6717 / 0.6785，随后未达到全局 0.90 下限；fail-fast 取消了并发 32。
将
models.qwen3-0.6b.gsm8k设为保守的 0.60 集成回归下限。保留五样本聊天、确定性采样、5,376 token 预算和两种评分指标。策略与样本分析 记录格式与重复问题、7.17 个百分点的 strict 分数余量，以及 Qwen 对贪心解码的提醒。Qwen 发表的 59.59% 是 base 模型的四样本 CoT 结果，仅作为规模背景；未找到直接可比的已发表聊天基线。该下限不代表模型最佳质量。验证
git diff --check通过。v1.3.0rc27源码上执行两次：首次打补丁，第二次保持幂等，补丁后模块编译通过。916278d18的 canary 门控完整 sweep 35538521127 通过：五个吞吐点和两项评测全部成功。1,170 个 benchmark 请求全部完成，零失败，功耗数据有效。Tests、lint 和 Python CodeQL 均通过。两项评测均完成全部 1,319 题，无空响应，使用相同且未修改的确定性设置。已检查原始样本、结果元数据、分数验证和全部七份服务端日志。每个服务端均使用 TensorRT-LLM
1.3.0rc27并从 ModelScope 快照加载权重；canary 作业日志确认 benchmark tokenizer 使用同一快照。日志中未发现服务端错误或 Hugging Face 模型回退。AI 模型披露
先前工作披露使用 GPT-5；本次会话无法独立核实该早期精确标识。本次继续工作使用 Codex 检查产物、记录设计与评测策略并验证 PR；无法核实其精确运行时模型/版本。未使用委派代理。