Repository navigation
GPU stack 1/8: update feectools (device ghost exchange, single-reduction PCG); pcg_sr solver option - #782
Merged
Conversation
… add the pcg_sr solver option feectools: struphy-hub/feectools#104 (fix-cuda-aware-mpi-issues into devel-tiny). pcg_sr (PConjugateGradientSingleReduction) is accepted wherever pcg is, including the preconditioned solvers of the Poisson, diffusion, curl-curl and variational propagators. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 16, 2026
Merged
Merged
feectools b70cb6c (struphy-hub/feectools#104): the emulated GPU tests (fake CuPy) failed at the first stencil product. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
feectools 1d8d47c (struphy-hub/feectools#104). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
max-models
marked this pull request as draft
October 22, 2026 06:39
max-models
marked this pull request as ready for review
October 23, 2026 07:15
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
spossann
previously approved these changes
Oct 24, 2026
spossann
left a comment
spossann
left a comment
Member
There was a problem hiding this comment.
Bump feectools and good to go 👍
spossann
approved these changes
Oct 24, 2026
max-models
added a commit
that referenced
this pull request
Oct 24, 2026
…in Species (#783) Part 2 of 8 of the split of #772 (`cuda-aware-mpi-fixes`) into stacked PRs. **Requires struphy-hub/feectools#104 to be merged into feectools `devel-tiny` first** (PR 1 bumps the submodule). - `post_processing/output.py`: on the CuPy backend the fields are evaluated on the device and written from the host. The evaluation grids, point data and coordinates are converted where they cross, and HDF5 data is moved to the device before it goes into the spline coefficients. - `Species`: the plasma frequency of the unit conversion is a host scalar (`math.sqrt` instead of `xp.sqrt`). ## Tests Local, macOS without a GPU, in a fresh environment with feectools `1d8d47c` and the pinned cunumpy, after merging devel (with #774 and #775). The PR passes `test_cuda_parity`, `test_cuda_emulation`, `test_kernel_backends`, `test_sorting`, `test_accum_matrix_cupy`, `test_preconditioner_cupy`, `test_poisson` and `test_output` on its own. The top of the stack also passes the mass-matrix, preconditioner, Derham-GPU, fused-gradient, Kronecker-block, Poisson, shear Alfvén, post-processing, polar fake-CuPy and all `pic` tests (1477 passed), the examples test for the two profiling cases, the sorting, Poisson and fused-gradient tests under `mpirun -n 2`, and the device MPI sort on the fake CuPy on 2 and 4 ranks. The Kronecker products match the stencil mass matrices to 5e-16 for M0, M1, M2, M3, Mv, M1n and M2n on a Cuboid, periodic and Dirichlet, on 1 and 2 ranks. **Not tested on a GPU here.** The CUDA kernels and the device paths need a run on the cluster. ## Stack (merge in order) 0. struphy-hub/feectools#104 (feectools, into `devel-tiny`) | # | PR | base | |---|---|---| | 1 | #782 update feectools (device ghost exchange, single-reduction PCG); pcg_sr solver option | `devel` | | 2 | #783 host/device conversions in post-processing; math.sqrt in Species **(this PR)** | #782 | | 3 | #784 CUDA mass-matrix assembly kernel kernel_3d_mat | #783 | | 4 | #785 apply separable mass matrices as Kronecker products of 1D banded matrices | #784 | | 5 | #786 Poisson/diffusion solve with fewer passes over the vectors | #785 | | 6 | #787 marker rows prepared inside the push kernels | #786 | | 7 | #788 chunked marker loading, row-based weight update, leaner MPI sorting; static field in PushVxB | #787 | | 8 | #789 profiling cases Poisson cube and Vlasov slab (CPU node vs GPU, GPU scaling) | #788 | 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
max-models
added a commit
that referenced
this pull request
Oct 24, 2026
Part 3 of 8 of the split of #772 (`cuda-aware-mpi-fixes`) into stacked PRs. **Requires struphy-hub/feectools#104 to be merged into feectools `devel-tiny` first** (PR 1 bumps the submodule). - `kernel_3d_mat_cuda.cu`: one thread per (element, codomain basis function, domain basis function). It replaces the host fallback on the CuPy backend (`missing_cuda="fallback"` removed, launch size from `n_threads_from`). - `WeightedMassOperator.assemble` reads the quadrature grids (spans, weights, basis values) as backend arrays, copied to the device once and cached. - `Simulation.allocate` compiles the assembly kernel before the first assembly, in its own profiling region (`setup: compile cuda kernels (assembly)`), so the ~1 s compilation is not counted in the assembly. Added while splitting: parity cases for `kernel_3d_mat` (random quadrature data on 3×2×2 elements with mixed degrees, with and without a ghost row). `test_cuda_kernels_have_parity_cases` requires one for every kernel with a CUDA version. The emulation test confirms that the CUDA kernel matches the pyccel kernel without a GPU. ## Tests Local, macOS without a GPU, in a fresh environment with feectools `1d8d47c` and the pinned cunumpy, after merging devel (with #774 and #775). The PR passes `test_cuda_parity`, `test_cuda_emulation`, `test_kernel_backends`, `test_sorting`, `test_accum_matrix_cupy`, `test_preconditioner_cupy`, `test_poisson` and `test_output` on its own. The top of the stack also passes the mass-matrix, preconditioner, Derham-GPU, fused-gradient, Kronecker-block, Poisson, shear Alfvén, post-processing, polar fake-CuPy and all `pic` tests (1477 passed), the examples test for the two profiling cases, the sorting, Poisson and fused-gradient tests under `mpirun -n 2`, and the device MPI sort on the fake CuPy on 2 and 4 ranks. The Kronecker products match the stencil mass matrices to 5e-16 for M0, M1, M2, M3, Mv, M1n and M2n on a Cuboid, periodic and Dirichlet, on 1 and 2 ranks. **Not tested on a GPU here.** The CUDA kernels and the device paths need a run on the cluster. ## Stack (merge in order) 0. struphy-hub/feectools#104 (feectools, into `devel-tiny`) | # | PR | base | |---|---|---| | 1 | #782 update feectools (device ghost exchange, single-reduction PCG); pcg_sr solver option | `devel` | | 2 | #783 host/device conversions in post-processing; math.sqrt in Species | #782 | | 3 | #784 CUDA mass-matrix assembly kernel kernel_3d_mat **(this PR)** | #783 | | 4 | #785 apply separable mass matrices as Kronecker products of 1D banded matrices | #784 | | 5 | #786 Poisson/diffusion solve with fewer passes over the vectors | #785 | | 6 | #787 marker rows prepared inside the push kernels | #786 | | 7 | #788 chunked marker loading, row-based weight update, leaner MPI sorting; static field in PushVxB | #787 | | 8 | #789 profiling cases Poisson cube and Vlasov slab (CPU node vs GPU, GPU scaling) | #788 | 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
max-models
added a commit
that referenced
this pull request
Oct 24, 2026
… 1D banded matrices (#785) Part 4 of 8 of the split of #772 (`cuda-aware-mpi-fixes`) into stacked PRs. **Requires struphy-hub/feectools#104 to be merged into feectools `devel-tiny` first** (PR 1 bumps the submodule). - `feec/kronecker_mass.py`: `WeightedMassOperator.dot` applies the blocks of its matrix that are Kronecker products of 1D banded matrices (constant weights, e.g. on a Cuboid) as three banded passes instead of the 3D stencil product. On the GPU this uses the new kernel `kron_band_pass_cuda.cu`. `STRUPHY_KRONECKER_MASS=0` switches it off. - The replacement is built on first use and invalidated by assembly, copies and in-place arithmetic. A matrix changed by arithmetic that is not a Kronecker product (e.g. a Schur complement rebuilt every step) is not checked again. - The boundary and extraction operators around the matrix are skipped when they are identities, instead of being applied as copies. - The band-pass kernel is launched through feectools' `CachedLaunch` (feectools#104): the banded passes run with the same buffers in every solver iteration, so the launch preparation is done once per set of buffers. ## Tests Local, macOS without a GPU, in a fresh environment with feectools `1d8d47c` and the pinned cunumpy, after merging devel (with #774 and #775). The PR passes `test_cuda_parity`, `test_cuda_emulation`, `test_kernel_backends`, `test_sorting`, `test_accum_matrix_cupy`, `test_preconditioner_cupy`, `test_poisson` and `test_output` on its own. The top of the stack also passes the mass-matrix, preconditioner, Derham-GPU, fused-gradient, Kronecker-block, Poisson, shear Alfvén, post-processing, polar fake-CuPy and all `pic` tests (1477 passed), the examples test for the two profiling cases, the sorting, Poisson and fused-gradient tests under `mpirun -n 2`, and the device MPI sort on the fake CuPy on 2 and 4 ranks. The Kronecker products match the stencil mass matrices to 5e-16 for M0, M1, M2, M3, Mv, M1n and M2n on a Cuboid, periodic and Dirichlet, on 1 and 2 ranks. **Not tested on a GPU here.** The CUDA kernels and the device paths need a run on the cluster. ## Stack (merge in order) 0. struphy-hub/feectools#104 (feectools, into `devel-tiny`) | # | PR | base | |---|---|---| | 1 | #782 update feectools (device ghost exchange, single-reduction PCG); pcg_sr solver option | `devel` | | 2 | #783 host/device conversions in post-processing; math.sqrt in Species | #782 | | 3 | #784 CUDA mass-matrix assembly kernel kernel_3d_mat | #783 | | 4 | #785 apply separable mass matrices as Kronecker products of 1D banded matrices **(this PR)** | #784 | | 5 | #786 Poisson/diffusion solve with fewer passes over the vectors | #785 | | 6 | #787 marker rows prepared inside the push kernels | #786 | | 7 | #788 chunked marker loading, row-based weight update, leaner MPI sorting; static field in PushVxB | #787 | | 8 | #789 profiling cases Poisson cube and Vlasov slab (CPU node vs GPU, GPU scaling) | #788 | 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
max-models
added a commit
that referenced
this pull request
Oct 24, 2026
…tors (#786) Part 5 of 8 of the split of #772 (`cuda-aware-mpi-fixes`) into stacked PRs. **Requires struphy-hub/feectools#104 to be merged into feectools `devel-tiny` first** (PR 1 bumps the submodule). - `BoundaryOperator.dot` on device Stencil/Block vectors computes `out = v * mask`, one kernel per block (one for all blocks when they are contiguous float64). The masks are found once from `apply_essential_bc_to_array`, so they zero exactly the same entries. This replaces a copy and one fill per boundary and component. - `WeightedMassOperator.without_boundary_ops()` / `MassWithoutBoundaryOps`: the mass operator without its boundary operators. In `grad.T @ M1 @ grad`, `grad` already applies the same 0/1 masks, so this is the same operator with two fewer passes per product. It is not used with the multigrid preconditioner. - `ImplicitDiffusion`: the automatic `1e-14` stabilization stays out of the solver operator when a Dirichlet boundary already makes `grad.T M1 grad` positive definite. There is no `stab_mat` product for the right-hand side when `sigma_2 = 0` (Poisson). - `struphy.feec.fused_derivatives`: `fuse_boundary_gradient(derham.grad)` and `fuse_boundary_gradient(derham.grad.T)` apply `B_out G B_in` with one CUDA kernel each on the GPU (`fused_derivatives_cuda.cu`), instead of a masked copy, three derivatives and a masked copy. The kernel does the same operations in the same order, so the products are bit for bit those of the composed operators. It reads the input with its ghost regions exchanged, times the input mask with exchanged ghost regions, which equals the exchanged masked input in serial and in parallel. Host data and the fake CuPy use the composed operators. `ImplicitDiffusion` uses it for `grad` and `grad.T` (not with the multigrid preconditioner). The original branch imported this module, but it was never committed, so it is written new here. `test_fused_derivatives` compares the kernels with the composed operators on the fake CuPy with emulated launches, and guards that the kernels actually run. Also added: the fused mask kernel is skipped on cunumpy's fake CuPy, which cannot launch raw kernels (`test_mass_preconditioners_fake_cupy` failed). ## Tests Local, macOS without a GPU, in a fresh environment with feectools `1d8d47c` and the pinned cunumpy, after merging devel (with #774 and #775). The PR passes `test_cuda_parity`, `test_cuda_emulation`, `test_kernel_backends`, `test_sorting`, `test_accum_matrix_cupy`, `test_preconditioner_cupy`, `test_poisson` and `test_output` on its own. The top of the stack also passes the mass-matrix, preconditioner, Derham-GPU, fused-gradient, Kronecker-block, Poisson, shear Alfvén, post-processing, polar fake-CuPy and all `pic` tests (1477 passed), the examples test for the two profiling cases, the sorting, Poisson and fused-gradient tests under `mpirun -n 2`, and the device MPI sort on the fake CuPy on 2 and 4 ranks. The Kronecker products match the stencil mass matrices to 5e-16 for M0, M1, M2, M3, Mv, M1n and M2n on a Cuboid, periodic and Dirichlet, on 1 and 2 ranks. **Not tested on a GPU here.** The CUDA kernels and the device paths need a run on the cluster. ## Stack (merge in order) 0. struphy-hub/feectools#104 (feectools, into `devel-tiny`) | # | PR | base | |---|---|---| | 1 | #782 update feectools (device ghost exchange, single-reduction PCG); pcg_sr solver option | `devel` | | 2 | #783 host/device conversions in post-processing; math.sqrt in Species | #782 | | 3 | #784 CUDA mass-matrix assembly kernel kernel_3d_mat | #783 | | 4 | #785 apply separable mass matrices as Kronecker products of 1D banded matrices | #784 | | 5 | #786 Poisson/diffusion solve with fewer passes over the vectors **(this PR)** | #785 | | 6 | #787 marker rows prepared inside the push kernels | #786 | | 7 | #788 chunked marker loading, row-based weight update, leaner MPI sorting; static field in PushVxB | #787 | | 8 | #789 profiling cases Poisson cube and Vlasov slab (CPU node vs GPU, GPU scaling) | #788 | 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
max-models
added a commit
that referenced
this pull request
Oct 24, 2026
Part 6 of 8 of the split of #772 (`cuda-aware-mpi-fixes`) into stacked PRs. **Requires struphy-hub/feectools#104 to be merged into feectools `devel-tiny` first** (PR 1 bumps the submodule). - `push_eta_stage` and `push_vxb_analytic` prepare each marker themselves at stage 0 (`MarkerArgs.prepare_rows`): they copy the phase-space coordinates to the rows from `first_pusher_idx` and zero the rows after them, as `prepare_push` does. `Pusher._push` then skips the separate `prepare_push` launch when no init or evaluation kernel runs before them (those may read the prepared rows). The kernels cover the same markers, one thread per used marker (`n_markers_used`). - `prepare_rows` is a CUDA-only field of `MarkerArgs`. The CUDA emulation of the tests reads it with default 0. Updated after #774 and #775 were merged into devel: on the original branch this PR also stored the marker buffer column-major on the GPU (`order="F"`). devel now stores it as `(n_cols, n_markers)` on both backends, so that part is gone. The row preparation is ported to `markers(j, ip)` indexing. ## Tests Local, macOS without a GPU, in a fresh environment with feectools `1d8d47c` and the pinned cunumpy, after merging devel (with #774 and #775). The PR passes `test_cuda_parity`, `test_cuda_emulation`, `test_kernel_backends`, `test_sorting`, `test_accum_matrix_cupy`, `test_preconditioner_cupy`, `test_poisson` and `test_output` on its own. The top of the stack also passes the mass-matrix, preconditioner, Derham-GPU, fused-gradient, Kronecker-block, Poisson, shear Alfvén, post-processing, polar fake-CuPy and all `pic` tests (1477 passed), the examples test for the two profiling cases, the sorting, Poisson and fused-gradient tests under `mpirun -n 2`, and the device MPI sort on the fake CuPy on 2 and 4 ranks. The Kronecker products match the stencil mass matrices to 5e-16 for M0, M1, M2, M3, Mv, M1n and M2n on a Cuboid, periodic and Dirichlet, on 1 and 2 ranks. **Not tested on a GPU here.** The CUDA kernels and the device paths need a run on the cluster. ## Stack (merge in order) 0. struphy-hub/feectools#104 (feectools, into `devel-tiny`) | # | PR | base | |---|---|---| | 1 | #782 update feectools (device ghost exchange, single-reduction PCG); pcg_sr solver option | `devel` | | 2 | #783 host/device conversions in post-processing; math.sqrt in Species | #782 | | 3 | #784 CUDA mass-matrix assembly kernel kernel_3d_mat | #783 | | 4 | #785 apply separable mass matrices as Kronecker products of 1D banded matrices | #784 | | 5 | #786 Poisson/diffusion solve with fewer passes over the vectors | #785 | | 6 | #787 marker rows prepared inside the push kernels **(this PR)** | #786 | | 7 | #788 chunked marker loading, row-based weight update, leaner MPI sorting; static field in PushVxB | #787 | | 8 | #789 profiling cases Poisson cube and Vlasov slab (CPU node vs GPU, GPU scaling) | #788 | 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
max-models
added a commit
that referenced
this pull request
Oct 27, 2026
…r MPI sorting; static field in PushVxB (#788) Part 7 of 8 of the split of #772 (`cuda-aware-mpi-fixes`) into stacked PRs. **Requires struphy-hub/feectools#104 to be merged into feectools `devel-tiny` first** (PR 1 bumps the submodule). - Pseudo-random markers are drawn in chunks of `2**20` (was 10000): 16 instead of 1600 iterations for 16M markers, each a few small kernels and a device sync on the GPU. The NumPy draws do not depend on the chunk size. - `update_weights` finds the indices of the valid markers once and reads only the quantities it needs (`xp.take` of one row of the buffer each), instead of a boolean selection per property and a count of the valid markers. - `mpi_sort_markers`: the ghost bookkeeping is skipped without sorting boxes, where no ghosts can exist. There are no barriers around the device exchange, which waits for every send and receive (`cunumpy.mpi.exchange`). The host path keeps them. The sorting positions for `alpha = 1` are one add instead of five kernels. - `PushVxB` without a magnetic field variable computes the total field once, so there is no ghost exchange (an MPI synchronization point) before every push. - Device MPI sort (`Particles._mpi_sort_device`, `struphy.pic.mpi_sort_cuda`): on the GPU one kernel (`sort_destinations`) finds the destination of every marker and counts the markers per destination, instead of a pass over the buffer per direction and per rank. The leaving markers are listed once, stably sorted by destination and gathered once per destination rank. The result is that of the host path (the same markers in the same columns; lost markers become holes), at the cost of one 8-byte read per rank for the counts. `STRUPHY_DEVICE_SORT=1` or `0` forces it on (also on the fake CuPy) or off. The original branch imported `mpi_sort_cuda`, but it was never committed (and called an undefined `_to_numpy_for_kernel`), so it is written new here for the `(n_cols, n_markers)` buffer of devel. `test_mpi_sort_markers_device_kernel_fake_cupy` compares it with the NumPy sort, CUDA-aware and staged, periodic and removing boundaries, on 2 and 4 ranks. ## Tests Local, macOS without a GPU, in a fresh environment with feectools `1d8d47c` and the pinned cunumpy, after merging devel (with #774 and #775). The PR passes `test_cuda_parity`, `test_cuda_emulation`, `test_kernel_backends`, `test_sorting`, `test_accum_matrix_cupy`, `test_preconditioner_cupy`, `test_poisson` and `test_output` on its own. The top of the stack also passes the mass-matrix, preconditioner, Derham-GPU, fused-gradient, Kronecker-block, Poisson, shear Alfvén, post-processing, polar fake-CuPy and all `pic` tests (1477 passed), the examples test for the two profiling cases, the sorting, Poisson and fused-gradient tests under `mpirun -n 2`, and the device MPI sort on the fake CuPy on 2 and 4 ranks. The Kronecker products match the stencil mass matrices to 5e-16 for M0, M1, M2, M3, Mv, M1n and M2n on a Cuboid, periodic and Dirichlet, on 1 and 2 ranks. **Not tested on a GPU here.** The CUDA kernels and the device paths need a run on the cluster. ## Stack (merge in order) 0. struphy-hub/feectools#104 (feectools, into `devel-tiny`) | # | PR | base | |---|---|---| | 1 | #782 update feectools (device ghost exchange, single-reduction PCG); pcg_sr solver option | `devel` | | 2 | #783 host/device conversions in post-processing; math.sqrt in Species | #782 | | 3 | #784 CUDA mass-matrix assembly kernel kernel_3d_mat | #783 | | 4 | #785 apply separable mass matrices as Kronecker products of 1D banded matrices | #784 | | 5 | #786 Poisson/diffusion solve with fewer passes over the vectors | #785 | | 6 | #787 marker rows prepared inside the push kernels | #786 | | 7 | #788 chunked marker loading, row-based weight update, leaner MPI sorting; static field in PushVxB **(this PR)** | #787 | | 8 | #789 profiling cases Poisson cube and Vlasov slab (CPU node vs GPU, GPU scaling) | #788 | 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
max-models
added a commit
that referenced
this pull request
Oct 27, 2026
… vs GPU, GPU scaling) (#789) Part 8 of 8 of the split of #772 (`cuda-aware-mpi-fixes`) into stacked PRs. **Requires struphy-hub/feectools#104 to be merged into feectools `devel-tiny` first** (PR 1 bumps the submodule). - Poisson `gaussian_cube`: `--steps` (repeated solves with the matrices assembled once), `--solver pcg|pcg_sr`, `--gpu-timing` (one thread per rank, CUDA-event device times per region), `--no-output`. Submit scripts for 28 CPU ranks vs 1 GPU and for GPU scaling. `examples/Poisson/sweep_poisson.py` runs several cases. - Vlasov `homogeneous_slab`: particle push profiling case (16M markers by default, `--test-run` for the examples test), with post-processing and the same two submit scripts. - `profiling/profiling_job.py`: launcher detection and job setup for these cases. - `profiling/sweep.py` (`run_sweep`, `variant_entries`): runs several submit scripts as one sweep in one process (`profiling_job.begin_sweep` / `finalize_all`), so they share one packaging folder; a failing script is reported and the sweep goes on. `sweep_poisson.py` imported it, but it was never committed, so it is written new here. Checked with a dry run of `sweep_poisson.py` (30 submit commands) and a run on test scripts (variants, a failing exit code, an exception, registered cases). ## Tests Local, macOS without a GPU, in a fresh environment with feectools `1d8d47c` and the pinned cunumpy, after merging devel (with #774 and #775). The PR passes `test_cuda_parity`, `test_cuda_emulation`, `test_kernel_backends`, `test_sorting`, `test_accum_matrix_cupy`, `test_preconditioner_cupy`, `test_poisson` and `test_output` on its own. The top of the stack also passes the mass-matrix, preconditioner, Derham-GPU, fused-gradient, Kronecker-block, Poisson, shear Alfvén, post-processing, polar fake-CuPy and all `pic` tests (1477 passed), the examples test for the two profiling cases, the sorting, Poisson and fused-gradient tests under `mpirun -n 2`, and the device MPI sort on the fake CuPy on 2 and 4 ranks. The Kronecker products match the stencil mass matrices to 5e-16 for M0, M1, M2, M3, Mv, M1n and M2n on a Cuboid, periodic and Dirichlet, on 1 and 2 ranks. **Not tested on a GPU here.** The CUDA kernels and the device paths need a run on the cluster. ## Stack (merge in order) 0. struphy-hub/feectools#104 (feectools, into `devel-tiny`) | # | PR | base | |---|---|---| | 1 | #782 update feectools (device ghost exchange, single-reduction PCG); pcg_sr solver option | `devel` | | 2 | #783 host/device conversions in post-processing; math.sqrt in Species | #782 | | 3 | #784 CUDA mass-matrix assembly kernel kernel_3d_mat | #783 | | 4 | #785 apply separable mass matrices as Kronecker products of 1D banded matrices | #784 | | 5 | #786 Poisson/diffusion solve with fewer passes over the vectors | #785 | | 6 | #787 marker rows prepared inside the push kernels | #786 | | 7 | #788 chunked marker loading, row-based weight update, leaner MPI sorting; static field in PushVxB | #787 | | 8 | #789 profiling cases Poisson cube and Vlasov slab (CPU node vs GPU, GPU scaling) **(this PR)** | #788 | 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part 1 of 8 of the split of #772 (
cuda-aware-mpi-fixes) into stacked PRs. Requires struphy-hub/feectools#104 to be merged into feectoolsdevel-tinyfirst (PR 1 bumps the submodule).Merge struphy-hub/feectools#104 (feectools
fix-cuda-aware-mpi-issuesintodevel-tiny) first. This PR points thefeectoolssubmodule at that branch (commit1d8d47c).PConjugateGradientSingleReduction, fasterstencil_dot_3don the GPU (details in the feectools PR).pcg_sris accepted whereverpcgis (LiteralOptions.OptsSymmSolver), also with a preconditioner: Poisson, ImplicitDiffusion, GyrokineticPoissonSolve, CurlCurlSolve, VariationalDensityEvolve, VariationalMomentumAdvection. Same iterates aspcg, one global reduction per iteration instead of two.Tests
Local, macOS without a GPU, in a fresh environment with feectools
1d8d47cand the pinned cunumpy, after merging devel (with #774 and #775). The PR passestest_cuda_parity,test_cuda_emulation,test_kernel_backends,test_sorting,test_accum_matrix_cupy,test_preconditioner_cupy,test_poissonandtest_outputon its own. The top of the stack also passes the mass-matrix, preconditioner, Derham-GPU, fused-gradient, Kronecker-block, Poisson, shear Alfvén, post-processing, polar fake-CuPy and allpictests (1477 passed), the examples test for the two profiling cases, the sorting, Poisson and fused-gradient tests undermpirun -n 2, and the device MPI sort on the fake CuPy on 2 and 4 ranks. The Kronecker products match the stencil mass matrices to 5e-16 for M0, M1, M2, M3, Mv, M1n and M2n on a Cuboid, periodic and Dirichlet, on 1 and 2 ranks.Not tested on a GPU here. The CUDA kernels and the device paths need a run on the cluster.
Stack (merge in order)
devel-tiny)devel🤖 Generated with Claude Code