docker: add arm64 (sbsa) nccl2.30 build variant and wire into release CI - #4
Merged
Merged
Conversation
…m source Add a dedicated Dockerfile (Dockerfile.cuda13.x.ubuntu22.04.nccl2.30) that builds NCCL 2.30.7 from source instead of installing the pinned apt binary. During the build it patches the NET/IB device enumeration to guard the inner port / data-direct loops with `ncclNIbDevs < MAX_IB_DEVS`, and raises MAX_IB_DEVS to 128. This fixes the out-of-bounds write / crash that happens when a host exposes more active IB/RoCE ports than MAX_IB_DEVS (observed on a 54-active-port node). The existing Dockerfiles and their pinned NCCL versions are left untouched; this is an additive, separate variant. New build args: NCCL_SRC_REF, NCCL_MAX_IB_DEVS, NVCC_GENCODE.
- Add Dockerfile.cuda13.x.ubuntu22.04.nccl2.30.arm64: same patched NCCL 2.30.7 source build as the x86 variant, with the CUDA apt repo switched to sbsa (ARG CUDA_REPO_ARCH=sbsa) and the .run package name tagged with $(uname -m) (aarch64) so it does not collide with amd64 artifacts. - release.yml: give each matrix entry runs-on/platform; add the arm64 variant on the native ubuntu-22.04-arm runner (avoids amd64+QEMU). - Harden the x86 nccl2.30 Dockerfile: raise NCCL_MAX_IB_DEVS 128->400, unset empty NVCC_GENCODE so make uses NCCL defaults, and copy NCCL headers recursively (cp -a build/include/.) to include subdirs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an arm64 (SBSA) build of the patched NCCL 2.30.7
.runpackage and wires it into the release CI, alongside a few hardening fixes to the existing x86 nccl2.30 image.Changes
docker/Dockerfile.cuda13.x.ubuntu22.04.nccl2.30.arm64: same patched NCCL 2.30.7 source build as the x86 variant, with:sbsaviaARG CUDA_REPO_ARCH=sbsa(datacenter ARM: Grace / GH200 / Ampere Altra)..runpackage name tagged with$(uname -m)(aarch64) so it does not collide with amd64 artifacts..github/workflows/release.yml: each matrix entry now carriesruns-on/platform; added the arm64 variant on the nativeubuntu-22.04-armrunner (avoids slow amd64 + QEMU emulation).Dockerfile.cuda13.x.ubuntu22.04.nccl2.30: raiseNCCL_MAX_IB_DEVS128 → 400, unset emptyNVCC_GENCODEsomakeuses NCCL defaults, and copy NCCL headers recursively (cp -a build/include/.).Validation
docker buildx --call checkpasses on the arm64 Dockerfile (lint clean).nvidia/cuda:13.1.0-devel-ubuntu22.04has anarm64manifest;sbsaCUDA keyring URL returns 200.release.ymlparses as valid YAML (4 matrix entries).