Skip to content

TV/PICCS minimizers destroy the caller's CUDA context on success; IRN_TV_CGLS broken under NumPy 2 - #770

Merged
AnderBiguri merged 1 commit into
CERN:masterfrom
liu005:fix/tv-device-reset-and-krylov-dtype
Sep 7, 2026
Merged

AnderBiguri merged 1 commit into
CERN:masterfrom
liu005:fix/tv-device-reset-and-krylov-dtype

Conversation

@yliu88au

Copy link
Copy Markdown
Contributor

Two independent fixes, found together while running the POCS family from a host process that also uses CuPy.

  1. minimizeTV (GD_TV.cu) and PICCS (PICCS.cu) end their successful path with cudaDeviceReset(). A device reset destroys the primary CUDA context of the whole host process — every allocation, module and stream the caller holds through CuPy, PyTorch (the d25 bindings), or any other CUDA library dies with it. TIGRE's own subsequent kernels survive because the runtime silently re-creates the context, so the failure surfaces in the next library to touch the GPU (cudaErrorInvalidValue inside CuPy deallocations, then CUDA_ERROR_INVALID_HANDLE) and looks like a bug in whatever ran after asd_pocs/os_asd_pocs/sart_tv rather than in TIGRE. Minimal repro: hold a live cupy array, run asd_pocs, then read the array. All device buffers are freed explicitly before the reset, so it protects nothing; GD_AwTV.cu has had the same reset commented out for a long time.

  2. IRN_TV_CGLS cannot complete one iteration under NumPy 2 — completes a86e30d: the sqrt(lmbda) sites were fixed there, but gamma/alpha/beta come from np.linalg.norm(...)**2, a float64 scalar. With NEP 50, self.res + alpha*p promotes the volume to float64 and the next Ax raises Input data should be float32, not float64. Cast the step scalars to np.float32.

Verified on RTX 6000 Ada / CUDA 13 / NumPy 2.5: after the fix a CuPy array survives an asd_pocs call and irn_tv_cgls runs to completion in float32.

@AnderBiguri

Copy link
Copy Markdown
Member

Sorry this now has merge conflicts, can you check?

minimizeTV (GD_TV.cu) and PICCS (PICCS.cu) ended their SUCCESSFUL host
functions with cudaDeviceReset(). A device reset destroys the primary
CUDA context of the entire host process, not just the function's own
resources - every allocation, module and stream the caller holds through
CuPy, PyTorch (the d25 bindings), or any other CUDA library dies with
it. TIGRE's own subsequent kernels survive because the runtime silently
re-creates the context on their next call, so the failure surfaces in
the NEXT library to touch the GPU (e.g. cudaErrorInvalidValue inside
CuPy deallocations, then CUDA_ERROR_INVALID_HANDLE) and looks like a bug
in whatever ran after ASD_POCS/OS_ASD_POCS/SART_TV rather than in TIGRE.

Observed in practice: a method-comparison harness holding CuPy state ran
OS_ASD_POCS successfully, and the following algorithm's GPU bookkeeping
crashed with CUDA_ERROR_INVALID_HANDLE - reproducible with nothing more
than a live cupy array across an asd_pocs call.

All device buffers are already freed explicitly before this point, so
the reset frees nothing that matters. GD_AwTV.cu has had the same reset
commented out for a long time; this brings GD_TV.cu and PICCS.cu in
line, with a comment explaining why it must not come back.
@yliu88au

yliu88au commented Sep 7, 2026

Copy link
Copy Markdown
Contributor Author

No problem — checked, and it's a benign one: #774 fixed the same thing independently.

The only overlapping file is Python/tigre/algorithms/krylov_subspace_algorithms.py. Both PRs wrap the norm-derived scalars in np.float32(...) so NEP 50 doesn't promote self.res/p to float64 and get them rejected by Ax/Atb; #774 just routes through the new l2norm helper rather than np.linalg.norm(..., 2). All four lines my commit touched are already on master:

575: gamma = np.float32(l2norm(p.ravel())**2)
584: alpha = np.float32(gamma/(l2norm(q_aux_1.ravel())**2 + ...))
621: gamma1 = np.float32(l2norm(s.ravel())**2)
622: beta = np.float32(gamma1/gamma)
So 02780be is redundant now and I'll drop it rather than resolve it — nothing is lost. Rebasing onto master with that commit skipped leaves just the CUDA half, which never conflicted:

TV/PICCS minimizers: do not cudaDeviceReset() on the normal path
Common/CUDA/GD_TV.cu | 13 ++++++++++++-
Common/CUDA/PICCS.cu | 6 +++++-
That part is unchanged from your earlier review: minTV/AwminTV/PICCS call cudaDeviceReset() on the success path, which destroys the host process's primary CUDA context — every allocation, module and stream the caller holds through CuPy, PyTorch or anything else dies with it. TIGRE's own later kernels survive because the runtime silently recreates the context, so the failure surfaces in the next library to touch the GPU (typically cudaErrorInvalidValue inside CuPy deallocation) rather than in TIGRE. All buffers are already freed explicitly above it, and GD_AwTV.cu has the same call commented out for exactly this reason.

I'll force-push the rebase shortly and retitle to drop the now-stale "IRN_TV_CGLS broken under NumPy 2" clause, since #774 covers it.

@yliu88au
yliu88au force-pushed the fix/tv-device-reset-and-krylov-dtype branch from 02780be to 2c8b87c Compare September 7, 2026 11:48
@AnderBiguri
AnderBiguri merged commit b04c383 into CERN:master Sep 7, 2026
1 of 2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants