Repository navigation
DGSEM benchmarks: a local-memory layout per backend - #875
Merged
Merged
Conversation
vchuravy
added this pull request to stack #873
October 11, 2026 12:20
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #875 +/- ##
==========================================
+ Coverage 1.96% 80.49% +78.52%
==========================================
Files 24 26 +2
Lines 1980 2210 +230
==========================================
+ Hits 39 1779 +1740
+ Misses 1941 431 -1510 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Contributor
Benchmark ResultsShow table
Benchmark PlotsA plot of the benchmark results have been uploaded as an artifact to the workflow run for this PR. |
vchuravy
force-pushed
the
vc/dgsem-benchmarks-layout-switch
branch
from
October 11, 2026 15:43
8c22ee8 to
fcaf647
Compare
vchuravy
marked this pull request as ready for review
October 11, 2026 15:53
vchuravy
force-pushed
the
vc/dgsem-benchmarks-layout-switch
branch
from
October 11, 2026 15:58
fcaf647 to
863a628
Compare
Turn the local-memory layout into a switch, as the VIJFH and VIJHF data layouts of ClimaCore.jl and a prototype for Trixi.jl do: `DGSEMVariableFirst()` and `DGSEMVariableLast()` are arguments of all kernels with local memory, now also the slab kernels, and `dgsem_default_layout(backend)` picks the variable index first on the CPU and last on GPUs. The variants without a layout in their name use the backend's default, and those named "v-first" or "v-last" the other layout, so that each backend measures what its default saves. Assisted-by: Claude Code (Opus 5.5)
vchuravy
force-pushed
the
vc/dgsem-benchmarks-layout-switch
branch
from
October 11, 2026 15:58
863a628 to
ade3afa
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #871 (← #870 ← #866). #874 (tiles) is stacked on this.
This turns the local-memory layout of the DGSEM benchmarks into a per-backend switch, following a prototype for Trixi.jl's 3D GPU kernels and the
VIJFH/VIJHFdata layouts of ClimaCore.jl. There the layout is a type parameter of the space, and the docstring gives the reason for the variable-index-last variant: it permits linear indexing and is faster for operators that touch one field at a time.DGSEMVariableFirst()(F[v, i, j, (k,) e, ...]) andDGSEMVariableLast()(F[i, j, (k,) e, ..., v]) replace DGSEM benchmarks: local-memory layout with the variable index last #871'sVal{VLAST}. They are now an argument of every kernel with local memory:weak/node localmem,fluxdiff/node halfsweep, and, new here, the four slab kernels (andweak/node localmem tile_barrierin DGSEM benchmarks: variants with tiles (#864) #874).dgsem_default_layout(backend)puts the variable index first on the CPU and last otherwise, as in the Trixi prototype.This adds 4 benchmarks per precision in 3D (the slab kernels in the other layout).
Correctness
I compared every variant with
weak/elementorfluxdiff/elementin 2D and 3D (including a partially filled last workgroup), in Float32 and Float64. All 98 checks are within 100 eps on PoCL and on CUDA, with KAmain; on CUDA with #864's tiles, all 108. The two layouts give exactly the same results for every kernel.Timings
Minimum (and median) in µs, at the suite's mesh sizes. The default layout is listed first.
PoCL, 1 thread, Ryzen 9 5950X (load average about 8), default variable-first:
CUDA, Quadro RTX 4000, CUDA.jl
vc/ki-subgroup-opswith #864's KA. Nothing else was on the GPU at the start; two other processes had started by the end, which raised some 3D Float64 medians but not the minimums. Default variable-last:Observations:
N = 4with 4 or 5 variables. The bank conflicts that cost Trixi's 2D half sweep 2× (2D GPU volume kernels: local memory with the variable index last trixi-framework/Trixi.jl#3337) don't appear here (see DGSEM benchmarks: local-memory layout with the variable index last #871). In the 2D Float32 half sweep, earlier runs had variable-last 3–10% faster, so this row is within run-to-run noise.🤖 Generated with Claude Code