From 294615dfa923ba518f414327236af275068b9a2a Mon Sep 17 00:00:00 2001 From: chenfeiz0326 <203214996+chenfeiz0326@users.noreply.github.com> Date: Thu, 15 Oct 2026 03:05:43 -0700 Subject: [PATCH] [https://nvbugs/6843484][fix] Keep multi-node aggregated perf-sanity jobs in one NVLink domain The aggregated multi-node perf-sanity stages (PerfSanity-Node2-GPU8 and the gen_only_no_context PerfSanity-GEN1-NODE{2,4,8} stages) were submitted without --segment, so Slurm could spread one EP16/EP32 job across several NVL72 racks. Across racks the wide-EP MoE all-to-all cannot map MNNVL memory, so the communication factory falls back to NcclEP, which then crashes in NCCL GIN context creation or stalls until the idle-GPU reaper cancels the job. Request --segment= for these stages, the same way the Disagg-PerfSanity and SingleNvlinkDomain stages already do. Signed-off-by: chenfeiz0326 <203214996+chenfeiz0326@users.noreply.github.com> --- jenkins/L0_Test.groovy | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/jenkins/L0_Test.groovy b/jenkins/L0_Test.groovy index 030e3854db0b..77a88d0cb974 100644 --- a/jenkins/L0_Test.groovy +++ b/jenkins/L0_Test.groovy @@ -2038,7 +2038,9 @@ def runLLMTestlistWithSbatch(pipeline, platform, testList, config=VANILLA_CONFIG .replace("${ARTIFACTORY_DOCKER_HOST}/", "${ARTIFACTORY_DOCKER_HOST}#") } def mounts = getMountListForSlurmTest(cluster, true).join(",") - String[] taskArgs = getNodeArgs(nodeCount, gpuCount, disaggMultiNodeMode || singleNvlinkDomainMode) + // Multi-node perf-sanity and SingleNvlinkDomain stages must land in one NVLink + // domain: across racks the wide-EP MoE all-to-all cannot use MNNVL. + String[] taskArgs = getNodeArgs(nodeCount, gpuCount, disaggMultiNodeMode || aggMultiNodeMode || singleNvlinkDomainMode) if (taskArgs == null) { error "Invalid Slurm test stage name is set" }