Repository navigation
TV/PICCS minimizers destroy the caller's CUDA context on success; IRN_TV_CGLS broken under NumPy 2 - #770
Conversation
|
Sorry this now has merge conflicts, can you check? |
minimizeTV (GD_TV.cu) and PICCS (PICCS.cu) ended their SUCCESSFUL host functions with cudaDeviceReset(). A device reset destroys the primary CUDA context of the entire host process, not just the function's own resources - every allocation, module and stream the caller holds through CuPy, PyTorch (the d25 bindings), or any other CUDA library dies with it. TIGRE's own subsequent kernels survive because the runtime silently re-creates the context on their next call, so the failure surfaces in the NEXT library to touch the GPU (e.g. cudaErrorInvalidValue inside CuPy deallocations, then CUDA_ERROR_INVALID_HANDLE) and looks like a bug in whatever ran after ASD_POCS/OS_ASD_POCS/SART_TV rather than in TIGRE. Observed in practice: a method-comparison harness holding CuPy state ran OS_ASD_POCS successfully, and the following algorithm's GPU bookkeeping crashed with CUDA_ERROR_INVALID_HANDLE - reproducible with nothing more than a live cupy array across an asd_pocs call. All device buffers are already freed explicitly before this point, so the reset frees nothing that matters. GD_AwTV.cu has had the same reset commented out for a long time; this brings GD_TV.cu and PICCS.cu in line, with a comment explaining why it must not come back.
|
No problem — checked, and it's a benign one: #774 fixed the same thing independently. The only overlapping file is Python/tigre/algorithms/krylov_subspace_algorithms.py. Both PRs wrap the norm-derived scalars in np.float32(...) so NEP 50 doesn't promote self.res/p to float64 and get them rejected by Ax/Atb; #774 just routes through the new l2norm helper rather than np.linalg.norm(..., 2). All four lines my commit touched are already on master: 575: gamma = np.float32(l2norm(p.ravel())**2) TV/PICCS minimizers: do not cudaDeviceReset() on the normal path I'll force-push the rebase shortly and retitle to drop the now-stale "IRN_TV_CGLS broken under NumPy 2" clause, since #774 covers it. |
02780be to
2c8b87c
Compare
Two independent fixes, found together while running the POCS family from a host process that also uses CuPy.
minimizeTV (GD_TV.cu) and PICCS (PICCS.cu) end their successful path with cudaDeviceReset(). A device reset destroys the primary CUDA context of the whole host process — every allocation, module and stream the caller holds through CuPy, PyTorch (the d25 bindings), or any other CUDA library dies with it. TIGRE's own subsequent kernels survive because the runtime silently re-creates the context, so the failure surfaces in the next library to touch the GPU (cudaErrorInvalidValue inside CuPy deallocations, then CUDA_ERROR_INVALID_HANDLE) and looks like a bug in whatever ran after asd_pocs/os_asd_pocs/sart_tv rather than in TIGRE. Minimal repro: hold a live cupy array, run asd_pocs, then read the array. All device buffers are freed explicitly before the reset, so it protects nothing; GD_AwTV.cu has had the same reset commented out for a long time.
IRN_TV_CGLS cannot complete one iteration under NumPy 2 — completes a86e30d: the sqrt(lmbda) sites were fixed there, but gamma/alpha/beta come from np.linalg.norm(...)**2, a float64 scalar. With NEP 50, self.res + alpha*p promotes the volume to float64 and the next Ax raises Input data should be float32, not float64. Cast the step scalars to np.float32.
Verified on RTX 6000 Ada / CUDA 13 / NumPy 2.5: after the fix a CuPy array survives an asd_pocs call and irn_tv_cgls runs to completion in float32.