Load-balance, active-box windowing, block-structured AMR#1628
Open
sbryngelson wants to merge 383 commits into
Open
Load-balance, active-box windowing, block-structured AMR#1628sbryngelson wants to merge 383 commits into
sbryngelson wants to merge 383 commits into
Conversation
…port stays coarse (tags suppressed, candidate boxes clipped clear, internal assert); golden 2FC423D3
…pb/mv side-state (piecewise-constant prolong, own rhs scratch, restrict with moments), rank-span guard; golden BCBA6E74
Contributor
There was a problem hiding this comment.
Pull request overview
This PR introduces an opt-in (“default-off”) family of performance/diagnostic features (load-weight and SFC partition diagnostics, weighted init-time decomposition, rank timing), plus major simulation capabilities (active-box RHS windowing and block-structured AMR) and corresponding post-processing support and documentation/validation updates.
Changes:
- Adds new runtime parameters and toolchain metadata/validation hooks for the experimental performance/AMR feature family.
- Extends the simulation code with new modules for active-box restriction, load-weight diagnostics, SFC partition reporting, rank timing, and AMR integration points (including restart/output plumbing).
- Updates post_process to read/write AMR fine-block overlays and adds/updates golden metadata plus documentation/indexing.
Reviewed changes
Copilot reviewed 82 out of 94 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| toolchain/mfc/params/descriptions.py | Adds user-facing descriptions for new experimental/performance parameters. |
| toolchain/mfc/params/definitions.py | Registers new parameters (AMR, hybrid sensors, load-balance diagnostics) and target applicability. |
| toolchain/mfc/lint_docs.py | Treats new validator checks as non-physics doc checks. |
| tests/F980C769/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/ECABA006/golden-metadata.txt | Adds golden metadata for active-box test coverage. |
| tests/DD4CD8F3/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/CC4213FD/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/BD21A5C0/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/BCBA6E74/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/ACE05393/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/987D9025/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/852CCB81/golden-metadata.txt | Adds golden metadata for AMR-related golden tests. |
| tests/65C375B4/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/4DADE04B/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/454C565F/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/3A474BEE/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/2FC423D3/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/13945217/golden-metadata.txt | Adds golden metadata for moving-IB under AMR test coverage. |
| src/simulation/m_viscous.fpp | Clamps FD coefficient indexing to avoid ghost-region coefficient OOB in IB drag gradient evaluation. |
| src/simulation/m_time_steppers.fpp | Integrates active-box bounds into RK update loops and interleaves AMR fine-stage/subcycle operations. |
| src/simulation/m_start_up.fpp | Wires up new modules (rank timing, active-box, load-weight, SFC partition, AMR) into init/timestep/finalize and restart I/O. |
| src/simulation/m_sfc_partition.fpp | Adds analysis-only SFC tiling + weighted partition prediction and reporting. |
| src/simulation/m_rank_timing.fpp | Adds per-rank wall-time imbalance measurement helpers and reporting. |
| src/simulation/m_load_weight.fpp | Adds per-cell load-weight field construction and rank-level imbalance reporting. |
| src/simulation/m_hypoelastic.fpp | Refactors FD coefficient setup into a callable update routine (supporting AMR grid swaps). |
| src/simulation/m_global_parameters.fpp | Adds AMR working-state mirrors and slot selection helper plus defaults for new parameters. |
| src/simulation/m_data_output.fpp | Adds output/report hooks for load-weight, SFC partition, and rank-time diagnostics. |
| src/simulation/m_checker.fpp | Adds input validation/prohibits for active-box, hybrid sensors, load-balance, and AMR configurations. |
| src/simulation/m_active_box.fpp | Adds active-box initialization/growth and debug envelope checking. |
| src/simulation/m_acoustic_src.fpp | Adds AMR-aware handling of acoustic source support (bounding boxes and overlap abort). |
| src/post_process/m_start_up.fpp | Calls AMR fine-data reader and AMR overlay writer when amr is enabled. |
| src/post_process/m_global_parameters.fpp | Adds default-off amr flag for post_process overlay behavior. |
| src/post_process/m_data_output.fpp | Implements AMR fine-block overlay mesh/variables output (Silo/binary) and multimesh registration. |
| src/common/m_phase_change.fpp | Exposes per-cell Newton iteration count and threads it through relaxation to support load-weighting. |
| src/common/m_global_parameters_common.fpp | Adjusts start_idx lifecycle/allocation and makes load_weight_wrt visible to GPU macros. |
| src/common/m_derived_types.fpp | Introduces a simple t_box type used by new partitioning infrastructure. |
| src/common/m_box.fpp | Adds box/partition arithmetic helpers (equal/weighted splits, box-from-splits). |
| src/common/m_boundary_common.fpp | Skips BC buffer population during AMR fine advance to rely on coarse-driven ghost fill. |
| docs/module_categories.json | Registers new modules under documentation categories. |
| docs/documentation/readme.md | Adds AMR section link to the documentation index. |
| .typos.toml | Adds project-specific abbreviations to the spelling allowlist. |
| D = ((gs_min(lp) - 1.0_wp)*cvs(lp))/((gs_min(vp) - 1.0_wp)*cvs(vp)) | ||
|
|
||
| #ifdef MFC_SIMULATION | ||
| if (relax .and. load_weight_wrt) then |
Comment on lines
+68
to
+71
| sfc_start = 0 | ||
| sfc_start(1) = start_idx(1) | ||
| if (num_dims >= 2) sfc_start(2) = start_idx(2) | ||
| if (num_dims >= 3) sfc_start(3) = start_idx(3) |
Comment on lines
+181
to
+188
| ! selection by min code (n_tiles is modest; O(n_tiles^2) acceptable, or replace with a sort) | ||
| do i = 1, n_tiles | ||
| cmin = huge(0_8); jmin = -1 | ||
| do t = 0, n_tiles - 1 | ||
| if (.not. used(t) .and. code(t) < cmin) then; cmin = code(t); jmin = t; end if | ||
| end do | ||
| order(i) = jmin; used(jmin) = .true. | ||
| end do |
Comment on lines
+48
to
+50
| if (.not. rank_time_wrt) return | ||
| tic_depth = tic_depth - 1 | ||
| if (tic_depth > 0) return |
…essure relaxation on fine blocks (lockstep + subcycle), internal energies on the generic conservative path; fix fine IB correct-state to carry pb/mv for non-polytropic QBMM; golden ECFE38B1
…erial amr/sfc read it), widen pc_iter_count guards to the sfc_partition path, O(n log n) SFC tile sort, clamp unmatched rank-timing toc
…m full-PR review - per-rank extents in the parallel AMR file with fail-closed decomposition validation (sim restart + post overlay), amr_max_blocks bound on restart, host-evaluated pc_iter_count kernel guard, active_box/hybrid gate, bounds-safe acoustic overlap check, conforming block-slot loops, pb/mv and IB-gps GPU-mapping deallocation
… (-206 lines) - single canonical weight block per site behind a bounds-safe use_central guard, arms verified identical before merging; add the hybrid sensors' first golden coverage (71E57E55 WENO, C6EA340F Riemann)
…/ghost slot fields, MPI-IO payload integrity (ierr + exact-size check) on the AMR restart, restart box-record and block-count validation, post overlay rank-agreement + header checks, moving-IB ghost-point capacity guard, acoustic overlap check covers restored blocks, paired-swap asserts, sfc dead API trim, comment/doc truth pass (restart format spec, stale physics matrices, self-contradicting PHYSICS_DOCS)
…the MPI-IO AMR restart write/read + 2-rank dynamic regrid + regridded-layout restart (5EFB3277), restart_check on the 1-rank dynamic case, hybrid central smooth-flux enum golden (21272AFB)
…y capacity (size(ghost_points)) - the coarse-derived bound false-aborted the AMR moving-IBM case on every lane; skip the AMR acoustic and nonpolytropic goldens under --single (drift/override_tol below single epsilon)
…ms clobber silently downgraded 5EFB3277 to the serial restart path); regenerate 5EFB3277 as a true MPI-IO restart, add load_balance weighted-split golden 0253D658 (verified split moves: offsets 0/25/64 vs equal 32) and two-body prescribed-motion IB golden 43AF9F25
… run (post converts over the ghost shell; the diffusion-only condition left rank-seam Newton guesses uninitialized -> NaN T/pres/c, first exposed by the 2-rank chemistry AMR case); seed q_T over the interior in the simulation's order
…coefficient recompute on swap/restore (the axis half-width cell makes coarse y-coefficients per-cell; flag-off on uniform grids, 10-golden bit-identity battery), axis-singularity viscous treatment coarse-only, runtime grid-uniformity abort for stretched/external grids, Riemann-extrapolation BC gate, 3D-cyl stays gated (azimuthal Fourier filter); golden 9640CE7F (r-weighted mass drift 1.09e-6 vs reference 1.07e-6, diffs match the Cartesian control)
…mv ghost shell between coarse t^n/t^n+1 sources (ghost pb feeds the mixture pressure through the widened conversion), dynamic regrid bounces the side-state through pb/mv_stor with re-prolong + overlap copy; both v1 gates dropped; golden B0A5D230 (AMR-vs-reference 1.3e-7, 4-golden regression battery clean)
…es expand to fully contain each body plus margin (partial coverage is untested regime), overlapping expansions merge with size-cap and acoustic-conflict aborts, fine IB state rebuilds from geometry after every regrid; moving bodies stay gated; golden 7FC2F9F8 (verified: regrid fires, box contains body bbox, post-regrid rebuild runs; 3-golden IB battery clean)
…nd by exact parent-cell bisection (unified formula: boundary k belongs to parent isect_lo+floor(k/2), even=parent midpoint, odd=parent right edge) replacing the locally-uniform continuation; grid-uniformity aborts relaxed to arm the per-swap WENO coefficient recompute; golden F0DDE1B4 (stretched + dynamic regrid + restart, block parents span 2.08x spacing). Validated: 15-golden uniform battery bit-identical, stretched AMR-vs-reference errors match the uniform control (rho 7.3e-3 vs 7.1e-3 rel-L2; static, dynamic-regrid, and 2-rank), restriction integral error exactly 0; docs matrices updated incl. stale IB dynamic-regrid rows from f3814e5
…d-combination gates; the sensor arrays are coarse-idwbuff sized (fine extent guard keeps fine bounds inside) and the sensor recomputes from the live swapped idwbuff every RHS call, so each level evaluates its own sensor. Validated: AMR-vs-reference error scale matches the plain control to 3 digits on an acoustic case where the sensor is live (mixed verdicts, 56/72 central). Also found the existing hybrid goldens cannot detect a dead sensor (bitwise-identical to plain WENO by design at physical eps: only constant/eps-dominated cells go central and any convex weights reconstruct those identically - verified by forcing all-central, 2e-2 shift); added consequential-eps liveness goldens (eps 0.5 > shock phi ~0.3, dead sensor moves the answer ~5e-4): BA4340EA 60739A3E and AMR combos 053C5DDA DDC4BA8A
…e containment expansion reads the body's LIVE centroid (bbox helper factored from the expansion), the initial block gets the same body expansion when regridding is active, and a per-substage guard aborts if a moving body reaches its block boundary between regrids (consecutive contained positions keep every sub-time interpolate contained). Validated: 200-step tracking run (29 regrids, body rises 25 cells past its initial box, completion is the guard's own proof of following), guard-trip negative test aborts with the named message, error at/below the validated static-block control (rho L2 5.6e-3 vs 8.5e-3, identical body-surface Linf); Mach ~0.85 wakes legitimately exceed the per-rank box cap (named abort). Golden E72953D1 (Mach 0.25, 4 regrids)
…s - regrid suppresses tags and clips candidate boxes around the cloud's padded bbox (positions + mapCells smearing + stencil + drift margin, recomputed collectively per regrid), the fine advance skips the EL hooks (a bubble would map to wrong indices on the swapped grid), and a per-stage guard aborts if the cloud reaches an active block. FIX: EL volume fractions sum to the local liquid fraction (beta), not 1 - the multi-fluid sum-to-one prolongation closure corrupted the EL state (found by the free-stream battery: O(1) momentum injection at block edges in quiescent flow; exactly 0 after the exemption); EL is correspondingly exempt from the mpp_lim gate. Validated: free-stream through block exactly 0, guard-trip named abort, 800-step wave + two-way bubble at 1e-9 field consistency with the bubble trajectory identical; goldens 4B08E9B7 (static) BCE1BBAE (dynamic regrid); multi-fluid AMR battery still green (closure fires for non-EL)
…ing - the fine block runs its own fixed-iteration sigma solve seeded and Dirichlet-bounded by the converged coarse sigma (piecewise-constant parent injection over the full buffered range; per-iteration BC/halo populate skipped under amr_in_fine_advance; coarse jac/jac_old warm-start state bounced across the fine advance), fine RK update uses the IGR coefficient form (dt embedded in the rhs), flux-register capture/apply return early under igr (no face fluxes exposed by the fused IGR kernels - seam conservation is truncation-order, documented), amr_subcycle gated. Validated: free-stream through block EXACTLY preserved, AMR-vs-reference at resolution scale (rho 1.3e-4 rel-L2, static and dynamic regrid), transverse seam artifact 3e-4 absolute from the coarse/fine sigma jump (truncation-level, documented), conservation drift at the reference's own outflow scale. Goldens 6C20B752 (static) 660FFBFE (dynamic regrid)
…mr_cur)%pb_f/mv_f as assumed-shape actuals made nvfortran emit a component-section data clause that dies partially-present on device (CUDA illegal address; caught by the full-suite GPU runs on both acc and omp - CPU passes only because the fine/coarse lbounds coincide); the ghost fill now selects its slot target by integer and references the slot arrays directly inside the kernels (the proven pattern of backup/update/lerp/restrict), and the pb/mv prolongation is now a device kernel reading the device-current coarse side-state (the old host-side copy would read a stale host mirror at regrid time on GPU). CPU nonpoly battery bit-identical (4/4 goldens)
…P_SFs pattern - the previous fix removed the ghost-fill component-section clause but the fine s_compute_rhs call still passed raw amr_slots(amr_cur)%pb_f/mv_f/rhs_* 5D members as actuals (second CUDA illegal address on device); persistent slot state (pb/mv f, stor, ghost_a/b) is now type(pres_field) with %sf allocated + ACC_SETUP_SFs per slot (the exact pb_ts pattern the coarse solver passes through the same RHS interface on GPU), and the fine rhs scratch moves to shared module arrays amr_rhs_pb_f/amr_rhs_mv_f (slots advance sequentially; mirrors the coarse rhs_pb pattern). CPU nonpoly battery bit-identical (4/4)
…on stays host-side - the trace (NV_ACC_NOTIFY) pinned the remaining illegal address to the device prolong kernel referencing pb_ts(1)%sf across modules; ALL pbmv kernels (fill/backup/rk-update/lerp/restrict) now receive the slot and coarse arrays as assumed-shape dummies with %sf pointer-member actuals - the one device-proven pattern (the coarse pb chain and the fine q_cons machinery both pass everything as arguments) - and the prolongation reverts to the original host loops + device push (both call paths make the coarse host mirrors current first: init writes them on the host, regrid refreshes them from the device); also fixes mv_f missing %sf in the two regrid GPU_UPDATE lists after the pres_field conversion. CPU nonpoly battery bit-identical (4/4)
…t-in (honor_io_keys), not content-based - honoring any case's explicit parallel_io/prim_vars_wrt broke the 18 Example-derived goldens whose imported case.py sets those keys (goldens were generated under the clobber): every full CI test lane failed with line-count mismatches (first seen on intel-reldebug only because it finishes first; reproduced locally under nvfortran, exit 0 with 6 extra prim.*.dat pack entries). The two tests the exemption exists for (5EFB3277 MPI-IO AMR restart, 0253D658 load_balance) opt in via define_case_d(honor_io_keys=True); UUIDs and goldens unchanged. Validated: ibm_ellipse/ibm_stl/phasechange_bubble pass again and both opt-ins keep their coverage (5/5 with -a)
…-clip verification after body expansion/merge ran only under acoustic_source, so an expanded box could silently regrow over the cloud in ib+lagrangian configs (fail-closed check now covers both exclusions); (2) the per-stage cloud guard ran only on block-owner ranks, missing bubbles whose smearing support reaches a block across a rank seam (now every rank checks its local bubbles against the global block box); (3) the moving-body containment guard was skipped entirely on static blocks (now always on for moving bodies, margin 0 there since validated static configs sit tighter than the regrid margin); (4) the position-to-cell-index conversions (int((x-beg)/dx(0))) in the IB expansion/guard and cloud exclusion assume uniform spacing and dx(0) is rank-local on stretched grids (rank-inconsistent regrid boxes) - stretched+lagrangian and stretched+ib-dynamic-regrid now fail closed at init (checker+validator). Comment/docs truth pass: fill_fine_ghosts_pbmv doc described the reverted selector design, nonpoly QBMM rows still said regrid/subcycle gated (both supported), amr.md mandatory settings said WENO-only (IGR allowed), recompute-flag comments still axisym-only. New goldens close the stretched coverage gaps: 79B334C7 (2-rank stretched: first exercise of the start_idx offset terms where a wrong offset changes coordinates), B7704247 (2D stretch_y: first exercise of the y-direction bisection formula; 2 regrids, block spans 65% y-spacing variation). 20-golden battery green incl. all touched features
Flip restart_check on the existing 3D static-block golden (476AA3A4): the restart reader validates the fine extent per axis, so a z-extent slip would pass every non-restart 3D golden. Reuses the straight-run golden (no new golden); the midpoint roundtrip passes, confirming 3D multi-axis AMR restart round-trips correctly.
Terse pass over the comments in the PR's new AMR/load-balance modules (m_amr, m_amr_regrid, m_amr_restart, m_amr_registers, m_active_box, m_load_balance, m_load_weight, m_sfc_partition, m_rank_timing, m_box): cut filler, redundant restatement, and obvious-code narration, and re-flow the mangled nest_children comments. Comment-only (~37% fewer words, net -124 lines) - every invariant, index/frame convention, compiler-quirk note, TWIN lockstep marker, and trap warning is preserved in substance. Build + 12 AMR/IGR/MHD/load-balance/active-box goldens bit-identical.
Terse pass over the PR-ADDED comments in m_ibm, m_checker, m_time_steppers, and m_rhs (pre-existing files - only the PR's own AMR / fine-IB / swap / gate comments were edited; verified no pre-existing comment or PROHIBIT message string was modified). Cut filler and run-ons while preserving every gate rationale, compiler-quirk note (CCE lib-4425 module separation, declare-target/present-table, ab_int device-copy, AMD defaultmap/ROCm), TWIN marker, and trap warning. Comment-only; build + 8 AMR/IB/IGR/subcycle/QBMM goldens bit-identical.
The Example suite caps grids at 25x25 (modify_example_case), but an AMR block must sit >= buff_size cells inside the domain and span at most half of it - there is no room for a valid block at 25x25, and the example's block indices (sized to its own 128-cell grid) fall outside the capped grid, so the CI Example run failed case validation ('amr_block_end must be <= global cell max per axis'). Reproduced locally. The example stays valid/runnable at native resolution and AMR is covered directly by amr_golden_tests; this only removes it from the downsized Example smoke suite (same reason as the existing 2D_reacting_mixing_layer skip).
Member
Author
|
…forced cross-rank remap (l0_migrate_step)
… bit-identical (l0_rebalance_interval)
… (gather at output), all paths bit-identical
…ter, bit-identical on nvfortran OpenACC (2xA100)
…ming, allreduced), reproducible on CPU+GPU
…-identical 2D+3D np=1/2
…le wrap-seam), bit-identical 2D+3D
…_bc -> consistent wrap-seam list), fixes cross-rank deadlock, bit-identical
…windows, halves noise-driven migration churn (bit-identical)
…, lint fix), bit-identity re-verified
…), spike bit-identical 2D+3D [Option 2 foundation]
…p) + l0_slot_off (tile prefix), behavior-neutral [spike-only + AMR np1==np2 byte-identical]
…runs the device pack/unpack P2P path that CI must exercise on Cray/AMD-flang (no other test hits it)
…vf intent(in)->intent(inout) (passed to bidirectional s_l0_copy_block inout dummy; nvfortran allowed it, GNU/Intel reject)
…teger div-by-zero SIGFPE on test 33866935 Fortran does not mandate .and. short-circuit: amdflang evaluates mod(t_step, l0_rebalance_interval) before the interval>0 guard, so at the default interval=0 it divides by zero -> SIGFPE (exit 136). gfortran/nvfortran order the divide after the guard and never trap. Nesting the mod inside if(l0_rebalance_interval>0 .and. t_step>0) makes the divide unreachable at interval=0; output is unchanged (the rebalance branch was never taken there). Verified np=2 on AFAR amdflang gfx90a GPU build: 33866935 passes, golden matches, divide now guard-dominated in the Release binary.
…entity today, neutral)
…init populates tile slots only when amr on (no double-allocate)
…f+1 (neutral at l0_slot_off=0)
…R fine loops (neutral scaffolding for coexist)
…nt-3 tiles->L0 coupling); note deferred double-alloc/free
…t double-free (Inc3)
…gated) + dev oracle harness
… + s_l0_tiles_finalize free-slot loop under .not.amr (audit CRIT-1 geometry corruption + CRIT-2 use-after-free; both guard-hidden today)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
An opt-in, default-off family of performance features and the measurement infrastructure they rest on. With all flags at their defaults the only touched production path is
s_mpi_decompose_computational_domain, refactored through the newm_boxmodule (byte-identical; covered by the existing suite).m_box(partition arithmetic),m_load_weight/load_weight_wrt(per-cell load-weight field + imbalance metric),m_sfc_partition/sfc_partition_wrt(Morton-SFC predicted-imbalance diagnostic),m_load_balance/load_balance(weighted static decomposition at init; AMR-fine-work-aware),m_rank_timing/rank_time_wrt(per-rank compute-time diagnostic).m_active_box/active_box: restricts reconstruction/Riemann/RK windows to a light-cone-grown box around non-ambient flow; strict-subset golden-tested.hybrid_wenoandhybrid_riemann(+hybrid_weno_eps,hybrid_smooth_flux): linear-optimal weights / central-or-Rusanov flux in smooth cells, full WENO/HLLC at flagged discontinuities (Jameson sensor, stencil-dilated, per-level under AMR).m_amr+m_amr_registers: two-level 2:1 refined block hierarchy; conservative restriction and conservative-linear prolongation with physics-specific closures; per-stage flux registers with Berger–Colella refluxing; Berger–Rigoutsos multi-block dynamic regrid; optional dt/2 subcycling; multi-rank (single-owner blocks assigned by Morton-SFC work balancing at each regrid, with migration; blocks may span rank seams via P2P coarse↔fine gather/scatter; same-level seam halo; distributed registers); restart (both IO modes, regridded-layout persistence); AMR-aware post-processing (fine blocks visualizable as Silo overlay domains); GPU-resident fine level on both OpenACC and OpenMP offload.Full algorithm and user documentation:
docs/documentation/amr.md(support matrix enforced at runtime by the checker — unsupported combinations abort with named messages, never silently).AMR physics support matrix (abridged; authoritative table in amr.md)
Supported and golden-tested: single- and multi-fluid (5-eq,
mpp_lim) · 6-eq with per-block pressure relaxation · viscous (refluxed) · phase change (relax) · chemistry incl. species diffusion · Euler–Euler bubbles (polytropic/non-polytropic, mono/polydisperse, QBMM incl. non-polytropic with per-blockpb/mvside-state; dynamic regrid + subcycle) · acoustic sources (coarse-grid support with regrid exclusion) · immersed boundaries (multi-body, static or prescribed-motion, incl. dynamic regrid with body-containment expansion and per-substage guards) · 2D axisymmetric (per-block WENO-coefficient recompute) · stretched grids (exact parent-bisection ghost coordinates + per-swap coefficient recompute) · hybrid WENO/Riemann sensors (per-level) · Lagrangian bubbles (cloud excluded from blocks; two-way coupling on the coarse grid; regrid clips around the moving cloud) ·active_box(blocks contained in the growing window; agrees with plain AMR to ~1e-14) · IGR (restriction-only coupling: fine sigma solve seeded/Dirichlet-bounded by the coarse solve; documented truncation-order seam, exact free-stream) · 1D MHD/RMHD (div(B)=0 by construction in 1D; HLL and HLLD, incl. relativistic).Gated with named aborts (documented rationale): surface tension (seam force imbalance is structural — three fixes attempted and diagnosed in amr.md) · 2D/3D MHD (attempted and measured: the c/f seam is a continuous O(1) div(B) source GLM cleaning cannot remove — needs constrained-transport-class B prolongation/reflux) · hyperelasticity · 3D cylindrical (global azimuthal filter) · force-driven IB (
moving_ibm=2) · STL bodies · Riemann-extrapolation BCs (bc=-4) ·amr_subcycleunder IGR · stretched grids with Lagrangian/IB-regrid (uniform-spacing index conversions).Validation evidence
Known issues (all non-gating or in progress)
continue-on-error): an intermittent post-detected NaN on the two Lagrangian+AMR goldens. Exhaustively unreproducible off GitHub's runners — the exact failing stack (NVHPC 24.3 SDK,-tp=px -Kieee, HPC-X MPI, and the CI docker image itself under apptainer) passes elsewhere, as do native/zen2 builds; 24.5+ green. Documented at the golden definitions.Review guide
The commit history is arc-ordered (active-box → load-weight → SFC → weighted decomposition → rank timing → hybrid → m_box → AMR rungs → physics envelope → CI/GPU hardening); reviewing by arc is much easier than by file. The AMR arc builds stepwise: static hierarchy → restriction/prolongation → fine advance → refluxing → regrid → subcycling → multi-rank → GPU → each physics rung with its own validation. Commit messages carry the validation evidence for their change (measured defects, golden UUIDs, repro details for CI fixes).
All parameters ship default-off with
case_validatorentries, runtime checker gates, andcase.md/amr.mddocumentation.