[https://nvbugs/6843484][fix] Keep multi-node aggregated perf-sanity jobs in one NVLink domain by chenfeiz0326 · Pull Request #20418 · NVIDIA/TensorRT-LLM

chenfeiz0326 · 2026-10-15T12:30:50Z

Description

Fixes the multi-node gen_only_no_context perf-sanity failures from NVBug 6843484 on main. This is the main counterpart of the first commit of #20292 (release/1.3).

Affected stages (every aggregated multi-node perf-sanity stage):

GB200-16_GPUs-4_Nodes-PyTorch-PerfSanity-GEN1-NODE4-GPU16-Post-Merge-1
GB300-16_GPUs-4_Nodes-PyTorch-PerfSanity-GEN1-NODE4-GPU16-Post-Merge-{1,2}
GB300-32_GPUs-8_Nodes-PyTorch-PerfSanity-GEN1-NODE8-GPU32-Post-Merge-{1,2}
The 2-node PerfSanity-GEN1-NODE2-GPU8 and PerfSanity-Node2-GPU8 stages go through the same path. They have not failed because of it so far.

Symptoms in the bug:

ncclGinGdakiCreateContext ... failed: 2
NCCL-EP NVFP4 dispatch requires uint8 tokens
no new log output for 7201s
CANCELLED by 146504 (the idle-GPU reaper)

Root cause.

runLLMTestlistWithSbatch requests --segment=<nodeCount> only for Disagg-PerfSanity and SingleNvlinkDomain stages. Aggregated multi-node perf-sanity stages (aggMultiNodeMode) get no segment, so Slurm can spread one EP16/EP32 job over several NVL72 racks.
On oci-hsg, oci-jhb and oci-aga, topology/block uses one block per rack.
Across racks, NVLinkOneSided and NVLinkTwoSided fail with CUDA_ERROR_INVALID_HANDLE, because MNNVL memory cannot be mapped across NVLink domains. The MoE communication factory then falls back to NcclEP.
With NcclEP the case either crashes or stalls with idle GPUs until the reaper or the 2 h no-output watchdog cancels the job. It crashes in NCCL GIN or with NCCL-EP NVFP4 dispatch requires uint8 tokens.

Evidence: Slurm nodelist vs. outcome for these 5 stages. Nodelists are read from the Jenkins stage logs (Slurm job N nodelist:). Excluded: builds whose logs expired, runs hit by the main #3009 DNS outage, and one release/1.3 #31 run that was SKIP-waived. Every counted pass uploaded results.

Placement	`main` #3009–#3019	`release/1.3` #20–#31
1 rack	6 / 6 passed	7 / 7 passed
2+ racks	0 / 29 passed	0 / 31 passed

On main, the 4- and 8-node stages have failed in every one of the last 7 builds unless Slurm happened to place them in one rack. GB300 GEN1-NODE4-GPU16-Post-Merge-2 (DeepSeek-R1 con4096) shows this directly: it passed in feat: add conditional disaggregation test #3012, feat : reduce trt engine build time in testing #3014 and chore: Improve trtllm-serve import pattern #3016 on one rack, and failed in chore: Refactor imports inside tensorrt_llm._torch. #3015, fix: creating output of dataset generator in current directory #3018 and chore: bump version to "0.19.0.dev2025032500" #3019 on two.
main chore: bump version to "0.19.0.dev2025032500" #3019, GB200 DeepSeek-R1 con4096 dep16, on nvl72108-T[04-05],nvl72005-T[08,14] (2 racks): the archived gen_server_0.log shows all 944 MoE layer selections as NVLinkOneSided not available: CUDA_ERROR_INVALID_HANDLE and then NcclEP. Then comes gin_plugin.cc:916 ... proxyRegMrSymDmaBuf ... failed: 2. Every rank ends in ValueError: NCCL-EP NVFP4 dispatch requires uint8 tokens with shape (2, 3584), got dtype=torch.bfloat16.
Control: the Disagg-PerfSanity stages already pass --segment. They landed in one rack in 152/152 runs, up to 14 nodes.

Fix. Also request --segment=<nodeCount> in aggMultiNodeMode, the same as for Disagg-PerfSanity and SingleNvlinkDomain.

It is one line in jenkins/L0_Test.groovy.
It affects only multi-node stages whose name contains PerfSanity. Every such stage has at most 8 nodes, well under the 18-node block size. Slurm accepts --segment=4 and --segment=8 for this shape on oci-jhb and oci-aga (sbatch --test-only).
The local reproduction harness (jenkins/scripts/perf/local/submit.py) already emits --segment=<total_nodes> for every job.

Test Coverage

GPU validation on GB200 (oci-hsg), described below.
pre-commit passes on the changed file.
Stages to run in PR CI, with the ci: full pre-merge approved label (multi-GPU stages are blocked without it):
- GB200-16_GPUs-4_Nodes-PyTorch-PerfSanity-GEN1-NODE4-GPU16-Post-Merge-1
- GB300-16_GPUs-4_Nodes-PyTorch-PerfSanity-GEN1-NODE4-GPU16-Post-Merge-1
- GB300-16_GPUs-4_Nodes-PyTorch-PerfSanity-GEN1-NODE4-GPU16-Post-Merge-2
- GB300-32_GPUs-8_Nodes-PyTorch-PerfSanity-GEN1-NODE8-GPU32-Post-Merge-1
- GB300-32_GPUs-8_Nodes-PyTorch-PerfSanity-GEN1-NODE8-GPU32-Post-Merge-2

GPU validation (GB200, oci-hsg, job 7971062): passed. Setup: the exact main CI wheel (GenPostMergeBuilds #2977, c9b18eefa7, includes #20256), CI's pre-imported 26.09 container, this PR's tree, and --nodes=4 --segment=4. Case: GB200 DeepSeek-R1 con4096 dep16 gen_only_no_context.

	This PR	CI `main` #3016 (1 rack, passed)	CI `main` #3019 (2 racks, failed)
Racks	1 (`nvl72174-T[01-02,06,15]`)	1	2
MoE comm strategy	`NVLinkOneSided` ×944	`NVLinkOneSided` ×944	`NcclEP` ×944
Requests	4096/4096, 0 failed	4096/4096, 0 failed	none (crashed in warmup)
Output tok/s	94,715	94,757	–
pytest	`1 passed`	passed	failed (`CANCELLED by 146504`)

Details are in the PR comment.

PR Checklist

Please check this after reviewing the above items as appropriate for this PR.

🤖 Generated with Claude Code

…jobs in one NVLink domain The aggregated multi-node perf-sanity stages (PerfSanity-Node2-GPU8 and the gen_only_no_context PerfSanity-GEN1-NODE{2,4,8} stages) were submitted without --segment, so Slurm could spread one EP16/EP32 job across several NVL72 racks. Across racks the wide-EP MoE all-to-all cannot map MNNVL memory, so the communication factory falls back to NcclEP, which then crashes in NCCL GIN context creation or stalls until the idle-GPU reaper cancels the job. Request --segment=<nodeCount> for these stages, the same way the Disagg-PerfSanity and SingleNvlinkDomain stages already do. Signed-off-by: chenfeiz0326 <203214996+chenfeiz0326@users.noreply.github.com>

chenfeiz0326 · 2026-10-15T12:31:01Z

/bot run --disable-fail-fast --stage-list "GB200-16_GPUs-4_Nodes-PyTorch-PerfSanity-GEN1-NODE4-GPU16-Post-Merge-1,GB300-16_GPUs-4_Nodes-PyTorch-PerfSanity-GEN1-NODE4-GPU16-Post-Merge-1,GB300-16_GPUs-4_Nodes-PyTorch-PerfSanity-GEN1-NODE4-GPU16-Post-Merge-2,GB300-32_GPUs-8_Nodes-PyTorch-PerfSanity-GEN1-NODE8-GPU32-Post-Merge-1,GB300-32_GPUs-8_Nodes-PyTorch-PerfSanity-GEN1-NODE8-GPU32-Post-Merge-2"

coderabbitai · 2026-10-15T12:34:57Z

Important

Review skipped

Review was skipped as selected files did not have any reviewable changes.

⚙️ Run configuration

Configuration used: Repository: NVIDIA/TensorRT-LLM/.coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 39d83c76-d15a-4b14-afc2-f4d7ed8f5b1d

📥 Commits

Reviewing files that changed from the base of the PR and between 294615d and cbf1be3.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository: NVIDIA/TensorRT-LLM/.coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 104a9708-5d81-4d18-9ecc-1cd578cfb674

📥 Commits

Reviewing files that changed from the base of the PR and between ed87ab0 and 294615d.

📒 Files selected for processing: 1

jenkins/L0_Test.groovy

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

Walkthrough

The SLURM task argument condition now enables --segment for aggregated multi-node PerfSanity stages when gpuCount > 1.

Changes

SLURM segment arguments

Layer / File(s)	Summary
Include aggregated stages `jenkins/L0_Test.groovy`	The `getNodeArgs` condition now includes `aggMultiNodeMode` alongside disaggregated multi-node `PerfSanity` and `SingleNvlinkDomain` stages.

Priority: ➖ Normal

Estimated code review effort: 2 (Simple) | ~5 minutes

Change: Bug fix

Suggested reviewers: bowenfu

Merge Risk: ⚪ Minimal · up to 29461

The segmentation change matches the configured multi-node PerfSanity stages and is ready to merge.

Pre-merge checks: 5

Passed checks: 5 passed

Check name	Status	Explanation
Docstring Coverage		No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0…
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.
Title check		The title clearly describes the fix, identifies the NVBug, and follows the required [ticket][type] format.
Description check		The description is complete and on topic. It explains the issue, root cause, fix, affected stages, evidence, test coverage, and checklist status.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create a new PR

Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}

tensorrt-cicd · 2026-10-15T12:37:51Z

PR_Github #78135 [ run ] triggered by Bot. Commit: 294615d Link to invocation

chenfeiz0326 · 2026-10-15T13:00:56Z

Validated on GB200 (oci-hsg). PR CI can't run the 5 multi-GPU stages without the ci: full pre-merge approved label, so I ran one on a cluster myself.

Case: perf/test_perf_sanity.py::test_e2e[aggr_upload-gen_only_no_context-gb200_deepseek-r1-fp4_8k1k_con4096_ctx1_dep4_gen1_dep16_eplb0_mtp1_ccb-NIXL]. This is the case GB200-16_GPUs-4_Nodes-PyTorch-PerfSanity-GEN1-NODE4-GPU16-Post-Merge-1 runs.

Setup:

The exact main CI wheel: tensorrt_llm-1.4.0rc1-cp312-cp312-linux_aarch64.whl from GenPostMergeBuilds fix: The constructor checks useDynamicTree but doesn’t validate dynamicTreeMaxTopK if set #2977 (c9b18eefa7, includes [https://nvbugs/6921137][fix] Fix CuTe MLA page-table batch bounds #20256).
CI's pre-imported 26.09 container.
This PR's tree (294615dfa9).
#SBATCH --nodes=4 --segment=4, the header the fixed groovy now emits.

	This PR (job 7971062)	CI `main` #3016 (passed)	CI `main` #3019 (failed)
sbatch	`--nodes=4 --segment=4`	`--nodes=4`	`--nodes=4`
Nodes	`nvl72174-T[01-02,06,15]` (1 rack)	`nvl72167-T[05-08]` (1 rack)	`nvl72108-T[04-05],nvl72005-T[08,14]` (2 racks)
MoE comm strategy	`NVLinkOneSided` ×944, `NcclEP` ×0	`NVLinkOneSided` ×944	`NcclEP` ×944
GIN errors / crash	0 / none	0 / none	141 `gin_plugin` warnings, `NCCL-EP NVFP4 dispatch` ValueError
Requests	4096/4096, 0 failed	4096/4096, 0 failed	none (warmup crash, then reaped)
Output tok/s	94,715	94,757	–
Mean TTFT / TPOT	10,473 ms / 29.15 ms	10,690 ms / 28.49 ms	–
pytest / Slurm	`1 passed` / `COMPLETED 0:0`	passed	failed / `CANCELLED by 146504`

With the segment, Slurm kept all 4 nodes in one rack and the case passed. Throughput is within 0.05% of CI's last single-rack pass.

An earlier attempt (job 7970844, same setup and nodes) also reached NVLinkOneSided ×944 and a ready server. The harness then aborted it on a transient install line: pip ... OSError: IncompleteRead, which succeeded on retry, matched the harness's OSError log scan. That was a network blip on my side, unrelated to this change.

🤖 Generated with Claude Code

tensorrt-cicd · 2026-10-15T13:42:54Z

PR_Github #78135 [ run ] completed with state FAILURE. Commit: 294615d
/LLM/main/L0_MergeRequest_PR pipeline #64401 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

chenfeiz0326 · 2026-10-15T14:33:25Z

/bot run --disable-fail-fast --stage-list "GB200-16_GPUs-4_Nodes-PyTorch-PerfSanity-GEN1-NODE4-GPU16-Post-Merge-1,GB300-16_GPUs-4_Nodes-PyTorch-PerfSanity-GEN1-NODE4-GPU16-Post-Merge-1,GB300-16_GPUs-4_Nodes-PyTorch-PerfSanity-GEN1-NODE4-GPU16-Post-Merge-2,GB300-32_GPUs-8_Nodes-PyTorch-PerfSanity-GEN1-NODE8-GPU32-Post-Merge-1,GB300-32_GPUs-8_Nodes-PyTorch-PerfSanity-GEN1-NODE8-GPU32-Post-Merge-2"

chenfeiz0326 · 2026-10-15T14:34:30Z

@ZhanruiSunCh @yiqingy0: requesting your review (trt-llm-infra-devs owns jenkins/). This is a one-line change in jenkins/L0_Test.groovy for NVBug 6843484 (P0).

Aggregated multi-node perf-sanity stages (aggMultiNodeMode) never request --segment, so Slurm can split one EP16/EP32 job across NVL72 racks. Across racks MoE falls back to NcclEP, which crashes or stalls. This is the main cause of the GEN1-NODE4/GEN1-NODE8 post-merge failures: placement decides the outcome in 13/13 single-rack runs (all passed) and 60/60 multi-rack runs (all failed).
The fix passes aggMultiNodeMode to getNodeArgs() as well, the same way Disagg-PerfSanity and SingleNvlinkDomain stages already get --segment.
Validated on GB200 with main's CI wheel: 1 rack, NVLinkOneSided ×944, 4096/4096 requests, 1 passed. Details are in the comment above.

The first PR pipeline (#64401) did not reach any test stage. The Build-SBSA job was aborted by Kubernetes: the LLVM build pod failed to start and its image pull failed. That is infra, so I re-triggered it. The five affected stages are multi-GPU, so they also need the ci: full pre-merge approved label, which is added automatically once the PR is approved.

tensorrt-cicd · 2026-10-15T14:39:55Z

PR_Github #78160 [ run ] triggered by Bot. Commit: 294615d Link to invocation

tensorrt-cicd · 2026-10-15T15:40:29Z

PR_Github #78160 [ run ] completed with state FAILURE. Commit: 294615d
/LLM/main/L0_MergeRequest_PR pipeline #64421 (Partly Tested) completed with status: 'UNSTABLE'

CI Report

⚠️ Multi-GPU Label Required:
Multi-GPU tests require the ci: full pre-merge approved label on this PR. Either:

Wait for the PR to be fully approved — the label is added automatically once approval is complete. Having unresolved open comments is fine, or
If needed, ask a member of NVIDIA/trt-llm-ci-approvers to add the label manually.
Then re-trigger CI with the same bot command (no rebase needed).

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

chenfeiz0326 · 2026-10-15T17:10:56Z

@tburt-nv: adding you as a trt-llm-infra-devs reviewer, since the reviewers I requested earlier are offline until tomorrow. This is a one-line jenkins/L0_Test.groovy change for P0 NVBug 6843484: aggregated multi-node perf-sanity stages now also request --segment, the same as Disagg-PerfSanity. The summary, the evidence (13/13 single-rack runs passed, 60/60 multi-rack runs failed) and the GB200 validation are in the description and the comments above. CI #64421 is green up to the multi-GPU stages, which wait for the approval label.

github-actions · 2026-10-16T02:34:35Z

Automatically added "ci: full pre-merge approved" because this PR has satisfied the required GitHub review approvals. Unresolved review conversations and other required checks remain independent merge requirements.

chenfeiz0326 · 2026-10-16T02:35:56Z

/bot run --disable-fail-fast --stage-list "GB200-16_GPUs-4_Nodes-PyTorch-PerfSanity-GEN1-NODE4-GPU16-Post-Merge-1,GB300-16_GPUs-4_Nodes-PyTorch-PerfSanity-GEN1-NODE4-GPU16-Post-Merge-1,GB300-16_GPUs-4_Nodes-PyTorch-PerfSanity-GEN1-NODE4-GPU16-Post-Merge-2,GB300-32_GPUs-8_Nodes-PyTorch-PerfSanity-GEN1-NODE8-GPU32-Post-Merge-1,GB300-32_GPUs-8_Nodes-PyTorch-PerfSanity-GEN1-NODE8-GPU32-Post-Merge-2"

chenfeiz0326 · 2026-10-16T02:36:08Z

Thanks @hanjingtian! The label is now applied, so I've re-run the five affected multi-GPU stages (GEN1-NODE4 / GEN1-NODE8) with the same stage list. I'll merge once they're green.

tensorrt-cicd · 2026-10-16T02:43:02Z

PR_Github #78310 [ run ] triggered by Bot. Commit: 294615d Link to invocation

tensorrt-cicd · 2026-10-16T03:59:30Z

PR_Github #78310 [ run ] completed with state FAILURE. Commit: 294615d
/LLM/main/L0_MergeRequest_PR pipeline #64547 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

chenfeiz0326 · 2026-10-16T04:05:15Z

CI #78310 (pipeline #64547) result. This is the first PR run that reached the five affected stages, and every job was submitted with #SBATCH --segment=<nodes>. Each one landed in a single NVL72 rack:

Stage	Nodes	Result
GB200 `GEN1-NODE4-GPU16-Post-Merge-1` (DeepSeek-R1 con4096)	`nvl72132-T[07,10-11,13]`, 1 rack	✅ passed. `NVLinkOneSided` ×944, 4096/4096 requests
GB300 `GEN1-NODE4-GPU16-Post-Merge-1` (DeepSeek-V4-Pro con666)	1 rack	✅ passed
GB300 `GEN1-NODE4-GPU16-Post-Merge-2` (DeepSeek-R1 con4096)	1 rack	✅ passed
GB300 `GEN1-NODE8-GPU32-Post-Merge-1` (DeepSeek-V4-Pro con180)	`nvl72d299-T[01-08]`, 1 rack	❌ infra, see below
GB300 `GEN1-NODE8-GPU32-Post-Merge-2` (GLM-5 con512)	`nvl72d084-T[07-09,11,15-18]`, 1 rack	❌ infra, see below

Neither 8-node failure is a model failure. In both, the server came up and printed Generation servers are ready. The harness then failed the test on a pip line logged during the install step: WARNING: Retrying (...) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='files.pythonhosted.org', ...): Read timed out.")'. pip retried and the install completed. tests/test_common/error_utils.py scans the whole worker log for TimeoutError, so that warning matched. It's a transient PyPI timeout on oci-jhb, not something this change affects. Re-running the two stages.

chenfeiz0326 · 2026-10-16T04:05:38Z

/bot run --disable-fail-fast --stage-list "GB300-32_GPUs-8_Nodes-PyTorch-PerfSanity-GEN1-NODE8-GPU32-Post-Merge-1,GB300-32_GPUs-8_Nodes-PyTorch-PerfSanity-GEN1-NODE8-GPU32-Post-Merge-2"

tensorrt-cicd · 2026-10-16T04:11:51Z

PR_Github #78345 [ run ] triggered by Bot. Commit: 294615d Link to invocation

tensorrt-cicd · 2026-10-16T05:03:25Z

PR_Github #78345 [ run ] completed with state FAILURE. Commit: 294615d
/LLM/main/L0_MergeRequest_PR pipeline #64575 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

chenfeiz0326 · 2026-10-16T05:08:35Z

Re-run of the two 8-node stages (pipeline #64575):

GB300 GEN1-NODE8-GPU32-Post-Merge-1 (DeepSeek-V4-Pro con180): ✅ passed, on nvl72d004-T[03,05-11] (1 rack).
GB300 GEN1-NODE8-GPU32-Post-Merge-2 (GLM-5 con512): ❌ the same harness false positive as before. It landed on nvl72d320-T[01-02,06,10,13-16] (1 rack) and the server reached Generation servers are ready. The log scan then matched three pip WARNING: Retrying ... ReadTimeoutError lines from the install step (download.pytorch.org, pypi.org, files.pythonhosted.org). pip had recovered from all three.

So far 4 of the 5 affected stages have passed with --segment. The 5th was placed in one rack and reached a ready server both times, then was failed by a transient PyPI timeout logged during install.

--stage-list runs are reported as "Partly Tested" and don't update blossom-ci. Starting the full pre-merge pipeline now.

chenfeiz0326 · 2026-10-16T05:08:36Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-10-16T05:16:41Z

PR_Github #78362 [ run ] triggered by Bot. Commit: 294615d Link to invocation

Signed-off-by: chenfeiz0326 <203214996+chenfeiz0326@users.noreply.github.com>

chenfeiz0326 · 2026-10-16T08:54:50Z

Full pre-merge pipeline #64591: none of the failures come from this PR's change, which is one line in jenkins/L0_Test.groovy.

B300-PyTorch-1: about 650 unittest/_torch/attention/sparse/dsa/test_dsa_indexer.py tests fail with kv_cache_manager_v2 ... CorruptedError: KV cache manager is unusable. This is a regression on main itself, tracked as nvbugs/6939212. It starts at post-merge feat: Support cos_sin_cache in all cases. #3020; chore: bump version to "0.19.0.dev2025032500" #3019 was clean. The same failure hit at least 5 other PRs' pre-merge runs today ([None][feat] Introduce the GDN MTP replay V2 design #19586, [None][feat] Introduce the KDA MTP replay design #19848, [TRTLLM-16480][perf] Speed up GLM-5.3-Flash sparse attention #20454, [#20367][fix] Accept prompt_cache_key in Chat Completions requests #20368, [None][feat] Kimi K3 optimizations on Rubin #19960). main has no fix or waive for it yet.
A10-PyTorch-1: unittest/_torch/sampler/test_torch_sampler.py "terminated unexpectedly", a hang. The same hang hit [TRTLLM-16303][refactor] modularize Mamba cache managers #18843, [None][feat] Introduce the KDA MTP replay design #19848 and [https://nvbugs/6913451][fix] Avoid heavy imports in MPI shutdown #20434 today.

I merged the current main (3ac4e61255) into the branch with a fast-forward push. git diff main HEAD is still just the one line, approvals carried over, and I'm re-running. While nvbugs/6939212 is open on main, B300-PyTorch-1 will probably fail again. That failure is outside this PR's scope.

chenfeiz0326 · 2026-10-16T08:54:53Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-10-16T09:02:55Z

PR_Github #78441 [ run ] triggered by Bot. Commit: cbf1be3 Link to invocation

tensorrt-cicd · 2026-10-16T09:05:36Z

PR_Github #78362 [ run ] completed with state ABORTED. Commit: 294615d

Link to invocation

tensorrt-cicd · 2026-10-16T16:06:02Z

PR_Github #78441 [ run ] completed with state SUCCESS. Commit: cbf1be3
/LLM/main/L0_MergeRequest_PR pipeline #64659 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

chenfeiz0326 · 2026-10-16T18:18:01Z

Full pre-merge pipeline #64659 on cbf1be3 (this PR with main merged in) passed 17,467 tests. It failed in 3 stages, and none of the failures come from this PR's one-line change to jenkins/L0_Test.groovy:

CPU-Generic-x86-1 and CPU-Generic-arm-1. unittest/llmapi/test_trtllm_serve_config_schemas.py::test_checked_in_schemas_are_current fails with ValueError: Missing or stale configuration schemas. The schema JSON checked in under tensorrt_llm/schemas/ on main no longer matches main's own config classes. Pre-merge pipelines of other main PRs fail the same way, for example #64640, #64647 and #64667. This is tracked in nvbugs/6942724, and the waive is [None][infra] Waive 1 failed cases for main in pre-merge 64667 #20478.
H100_PCIe-PyTorch-Perf-1. perf/test_perf.py::test_perf[llama_v3.1_8b_instruct-bench-pytorch-float16-input_output_len:128,128-reqs:8192] failed because trtllm-bench was killed with SIGKILL. The same step fails intermittently elsewhere:
- main post-merge build 3022: also killed with SIGKILL.
- Another PR's pipeline #64674: exited with status 1. It passed when pipeline #64694 re-ran that same commit.

Because single-GPU tests failed, the pre-merge multi-GPU stages were blocked and didn't run. The five affected perf-sanity stages are post-merge stages, so a full pre-merge pipeline doesn't include them. They ran through --stage-list in pipelines #64547 and #64575 (results above).

Pre-merge reads the waive list from the tip of main, so no new commit is needed here. I'll re-run /bot run --disable-fail-fast on cbf1be3 once the waive lands.

chenfeiz0326 · 2026-10-17T00:22:17Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-10-17T00:30:52Z

PR_Github #78579 [ run ] triggered by Bot. Commit: cbf1be3 Link to invocation

tensorrt-cicd · 2026-10-17T05:25:51Z

PR_Github #78579 [ run ] completed with state SUCCESS. Commit: cbf1be3
/LLM/main/L0_MergeRequest_PR pipeline #64739 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

chenfeiz0326 · 2026-10-17T05:29:48Z

Full pre-merge pipeline #64739 on cbf1be3 ran after main waived the stale-schema test (#20478), and the schema failure is gone. The run reported 2,670 passed and 8 failed. Two stages ended FAILED, and neither comes from this PR:

DGX_B200-8_GPUs-PyTorch-4: accuracy/test_llm_api_pytorch.py::TestDeepSeekV4Pro::test_gsm8k_full_accuracy hit RuntimeError: Executor creation failed due to insufficient GPU memory (torch.OutOfMemoryError: ... Tried to allocate 6.56 GiB ... of which 4.34 GiB is free). It fails the same way on main itself: in post-merge builds 3020, 3021, 3022 and 3023, and in at least 7 other PR pipelines over the last two days. It also passes in about half of recent runs.
GB300-4_GPUs-PyTorch-1: accuracy/test_llm_api_pytorch.py::TestDeepSeekV41::test_gsm8k_dspark[nvfp4-dp4] died during executor startup with CUDA error: an illegal memory access was encountered. It passed in the 36 most recent runs where it ran, including main post-merge 3023. The [nvfp4-tp4] variant passed in this same stage.

DGX_H100-2_GPUs-PyTorch-Others-1 also failed on its first attempt: test_llm_rpc_tp2 was still in progress when its Slurm job was cut off. CI's infra retry re-ran the stage, and it passed on attempt 2.

This PR changes one condition in jenkins/L0_Test.groovy. It only affects stages where nodeCount > 1 && stageName.contains("PerfSanity"), and both failed stages are single-node, non-PerfSanity stages.

Re-running on the same commit. Stages that already passed on this commit are reused.

chenfeiz0326 · 2026-10-17T05:29:49Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-10-17T05:37:36Z

PR_Github #78607 [ run ] triggered by Bot. Commit: cbf1be3 Link to invocation

tensorrt-cicd · 2026-10-17T07:15:21Z

PR_Github #78607 [ run ] completed with state SUCCESS. Commit: cbf1be3
/LLM/main/L0_MergeRequest_PR pipeline #64760 completed with status: 'SUCCESS'

CI Report

Link to invocation

chenfeiz0326 · 2026-10-17T07:24:23Z

Merged after full pre-merge pipeline #64760 passed on cbf1be3. The re-run reused the stages that had already passed in #64739, and the 3 stages that failed there passed. NVBug 6843484 is closed as fixed. The release/1.3 side is #20292.

chenfeiz0326 requested a review from a team as a code owner October 15, 2026 12:30

chenfeiz0326 requested review from hanjingtian and mzweilz October 15, 2026 12:30

github-actions Bot assigned chenfeiz0326 Oct 15, 2026

chenfeiz0326 enabled auto-merge (squash) October 15, 2026 14:02

chenfeiz0326 requested review from ZhanruiSunCh and yiqingy0 October 15, 2026 14:34

chenfeiz0326 mentioned this pull request Oct 15, 2026

[https://nvbugs/6843484][fix] Keep multi-node aggregated perf-sanity jobs in one NVLink domain; cherry-pick #20000 to release/1.3 #20292

Merged

1 task done

chenfeiz0326 requested a review from tburt-nv October 15, 2026 17:10

hanjingtian approved these changes Oct 16, 2026

View reviewed changes

trtllm-agent added the ci: full pre-merge approved label Oct 16, 2026

chzblych approved these changes Oct 16, 2026

View reviewed changes

Merge branch 'main' into chenfeiz/perf-sanity-agg-multinode-segment-main

cbf1be3

Signed-off-by: chenfeiz0326 <203214996+chenfeiz0326@users.noreply.github.com>

chenfeiz0326 merged commit 20d5469 into NVIDIA:main Oct 17, 2026
7 checks passed

Conversation

chenfeiz0326 commented Oct 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Test Coverage

PR Checklist

Uh oh!

chenfeiz0326 commented Oct 15, 2026

Uh oh!

coderabbitai Bot commented Oct 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Walkthrough

Changes

Uh oh!

tensorrt-cicd commented Oct 15, 2026

Uh oh!

chenfeiz0326 commented Oct 15, 2026

Uh oh!

tensorrt-cicd commented Oct 15, 2026

Uh oh!

chenfeiz0326 commented Oct 15, 2026

Uh oh!

chenfeiz0326 commented Oct 15, 2026

Uh oh!

tensorrt-cicd commented Oct 15, 2026

Uh oh!

tensorrt-cicd commented Oct 15, 2026

Uh oh!

chenfeiz0326 commented Oct 15, 2026

Uh oh!

github-actions Bot commented Oct 16, 2026

Uh oh!

chenfeiz0326 commented Oct 16, 2026

Uh oh!

chenfeiz0326 commented Oct 16, 2026

Uh oh!

tensorrt-cicd commented Oct 16, 2026

Uh oh!

tensorrt-cicd commented Oct 16, 2026

Uh oh!

chenfeiz0326 commented Oct 16, 2026

Uh oh!

chenfeiz0326 commented Oct 16, 2026

Uh oh!

tensorrt-cicd commented Oct 16, 2026

Uh oh!

tensorrt-cicd commented Oct 16, 2026

Uh oh!

chenfeiz0326 commented Oct 16, 2026

Uh oh!

chenfeiz0326 commented Oct 16, 2026

Uh oh!

tensorrt-cicd commented Oct 16, 2026

Uh oh!

chenfeiz0326 commented Oct 16, 2026

Uh oh!

chenfeiz0326 commented Oct 16, 2026

Uh oh!

tensorrt-cicd commented Oct 16, 2026

Uh oh!

tensorrt-cicd commented Oct 16, 2026

Uh oh!

tensorrt-cicd commented Oct 16, 2026

Uh oh!

chenfeiz0326 commented Oct 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chenfeiz0326 commented Oct 17, 2026

Uh oh!

tensorrt-cicd commented Oct 17, 2026

Uh oh!

tensorrt-cicd commented Oct 17, 2026

Uh oh!

chenfeiz0326 commented Oct 17, 2026

Uh oh!

chenfeiz0326 commented Oct 17, 2026

Uh oh!

tensorrt-cicd commented Oct 17, 2026

Uh oh!

chenfeiz0326 commented Oct 15, 2026 •

edited

Loading

coderabbitai Bot commented Oct 15, 2026 •

edited

Loading

chenfeiz0326 commented Oct 16, 2026 •

edited

Loading