Running the cuopt-server container on an NVIDIA DGX Spark (GB10 Grace Blackwell Superchip, compute capability 12.1), the FastAPI layer starts fine and /cuopt/health responds {"status":"RUNNING"}, but any submitted POST /cuopt/request never completes. The background solver worker pool (process_handler.create_process) crash-loops continuously — each forked worker dies immediately and is respawned every ~0.4-0.5s, forever, with no application-level error ever logged.
Isolated the crash to plain GPU memory allocation via RAPIDS, independent of any cuOpt-specific code:
docker run --rm --gpus all nvidia/cuopt:latest-cu13
python -c "import cupy as cp; print(cp.arange(10).sum())"
exit code 135 (SIGBUS), no Python traceback
Also reproduces with numba.cuda (no RMM/cuDF involved at all), and crashes even earlier — on device capability query, before any kernel is compiled or launched:
docker run --rm --gpus all nvidia/cuopt:latest-cu13
python -c "import numba.cuda as cuda; print(cuda.is_available())"
exit code 135 (SIGBUS)
Confirmed this is not a stale-package issue: reproduces identically on today's newest available nightly (nvidia/cuopt:26.10.0a-cuda13.3-py3.14-arm64, bundling cupy-cuda13x 14.2.0 and rmm-cu13 26.10.0a44 — the latest published anywhere).
Also confirmed it's not about RMM's allocator strategy — CudaMemoryResource, CudaAsyncMemoryResource, and ManagedMemoryResource all crash identically.
Expected behavior
The solve request should complete normally, as it does on GPUs with more established compute capabilities.
Environment details
- Hardware: NVIDIA DGX Spark (GB10 Grace Blackwell Superchip), compute_cap=12.1
- Driver: 580.173.02, CUDA 13.0 (per nvidia-smi)
- OS: Ubuntu 24.04.4 LTS, aarch64
- Docker 29.2.1, NVIDIA Container Toolkit 1.19.1
- Images tested: nvidia/cuopt:latest-cu13 (26.8.0 stable) and nvidia/cuopt:26.10.0a-cuda13.3-py3.14-arm64 (nightly)
Additional context
PyTorch works correctly on this same GPU (confirmed via a container already running vllm-openai on this box): it prints a compute-capability-mismatch warning (Minimum and Maximum cuda capability supported by this version of PyTorch is (8.0) - (12.0)) but still runs GPU computations correctly. This suggests the underlying driver/hardware is fine, and the issue is specific to how RAPIDS' CUDA-driver-API bindings (used internally by libcuopt/librmm) handle an unrecognized/newer compute-capability value — the crash occurs at device-query time, before any kernel-image compatibility issue would normally surface, which points to a client-side bug (e.g. an unbounded architecture lookup) rather than a missing compiled kernel.
No kernel-level fault (NVRM: Xid) is logged at the time of the crash (journalctl -k), confirming this is a userspace crash, not a GPU-reported hardware fault.
Happy to provide the full SSH-reproducible command sequence if useful for triage.
Running the cuopt-server container on an NVIDIA DGX Spark (GB10 Grace Blackwell Superchip, compute capability 12.1), the FastAPI layer starts fine and /cuopt/health responds {"status":"RUNNING"}, but any submitted POST /cuopt/request never completes. The background solver worker pool (process_handler.create_process) crash-loops continuously — each forked worker dies immediately and is respawned every ~0.4-0.5s, forever, with no application-level error ever logged.
Isolated the crash to plain GPU memory allocation via RAPIDS, independent of any cuOpt-specific code:
docker run --rm --gpus all nvidia/cuopt:latest-cu13
python -c "import cupy as cp; print(cp.arange(10).sum())"
exit code 135 (SIGBUS), no Python traceback
Also reproduces with numba.cuda (no RMM/cuDF involved at all), and crashes even earlier — on device capability query, before any kernel is compiled or launched:
docker run --rm --gpus all nvidia/cuopt:latest-cu13
python -c "import numba.cuda as cuda; print(cuda.is_available())"
exit code 135 (SIGBUS)
Confirmed this is not a stale-package issue: reproduces identically on today's newest available nightly (nvidia/cuopt:26.10.0a-cuda13.3-py3.14-arm64, bundling cupy-cuda13x 14.2.0 and rmm-cu13 26.10.0a44 — the latest published anywhere).
Also confirmed it's not about RMM's allocator strategy — CudaMemoryResource, CudaAsyncMemoryResource, and ManagedMemoryResource all crash identically.
Expected behavior
The solve request should complete normally, as it does on GPUs with more established compute capabilities.
Environment details
Additional context
PyTorch works correctly on this same GPU (confirmed via a container already running vllm-openai on this box): it prints a compute-capability-mismatch warning (Minimum and Maximum cuda capability supported by this version of PyTorch is (8.0) - (12.0)) but still runs GPU computations correctly. This suggests the underlying driver/hardware is fine, and the issue is specific to how RAPIDS' CUDA-driver-API bindings (used internally by libcuopt/librmm) handle an unrecognized/newer compute-capability value — the crash occurs at device-query time, before any kernel-image compatibility issue would normally surface, which points to a client-side bug (e.g. an unbounded architecture lookup) rather than a missing compiled kernel.
No kernel-level fault (NVRM: Xid) is logged at the time of the crash (journalctl -k), confirming this is a userspace crash, not a GPU-reported hardware fault.
Happy to provide the full SSH-reproducible command sequence if useful for triage.