Load-balance, active-box windowing, block-structured AMR - #1628
Open
sbryngelson wants to merge 519 commits into
Open
Load-balance, active-box windowing, block-structured AMR#1628sbryngelson wants to merge 519 commits into
sbryngelson wants to merge 519 commits into
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR introduces an opt-in (“default-off”) family of performance/diagnostic features (load-weight and SFC partition diagnostics, weighted init-time decomposition, rank timing), plus major simulation capabilities (active-box RHS windowing and block-structured AMR) and corresponding post-processing support and documentation/validation updates.
Changes:
- Adds new runtime parameters and toolchain metadata/validation hooks for the experimental performance/AMR feature family.
- Extends the simulation code with new modules for active-box restriction, load-weight diagnostics, SFC partition reporting, rank timing, and AMR integration points (including restart/output plumbing).
- Updates post_process to read/write AMR fine-block overlays and adds/updates golden metadata plus documentation/indexing.
Reviewed changes
Copilot reviewed 82 out of 94 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| toolchain/mfc/params/descriptions.py | Adds user-facing descriptions for new experimental/performance parameters. |
| toolchain/mfc/params/definitions.py | Registers new parameters (AMR, hybrid sensors, load-balance diagnostics) and target applicability. |
| toolchain/mfc/lint_docs.py | Treats new validator checks as non-physics doc checks. |
| tests/F980C769/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/ECABA006/golden-metadata.txt | Adds golden metadata for active-box test coverage. |
| tests/DD4CD8F3/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/CC4213FD/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/BD21A5C0/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/BCBA6E74/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/ACE05393/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/987D9025/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/852CCB81/golden-metadata.txt | Adds golden metadata for AMR-related golden tests. |
| tests/65C375B4/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/4DADE04B/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/454C565F/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/3A474BEE/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/2FC423D3/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/13945217/golden-metadata.txt | Adds golden metadata for moving-IB under AMR test coverage. |
| src/simulation/m_viscous.fpp | Clamps FD coefficient indexing to avoid ghost-region coefficient OOB in IB drag gradient evaluation. |
| src/simulation/m_time_steppers.fpp | Integrates active-box bounds into RK update loops and interleaves AMR fine-stage/subcycle operations. |
| src/simulation/m_start_up.fpp | Wires up new modules (rank timing, active-box, load-weight, SFC partition, AMR) into init/timestep/finalize and restart I/O. |
| src/simulation/m_sfc_partition.fpp | Adds analysis-only SFC tiling + weighted partition prediction and reporting. |
| src/simulation/m_rank_timing.fpp | Adds per-rank wall-time imbalance measurement helpers and reporting. |
| src/simulation/m_load_weight.fpp | Adds per-cell load-weight field construction and rank-level imbalance reporting. |
| src/simulation/m_hypoelastic.fpp | Refactors FD coefficient setup into a callable update routine (supporting AMR grid swaps). |
| src/simulation/m_global_parameters.fpp | Adds AMR working-state mirrors and slot selection helper plus defaults for new parameters. |
| src/simulation/m_data_output.fpp | Adds output/report hooks for load-weight, SFC partition, and rank-time diagnostics. |
| src/simulation/m_checker.fpp | Adds input validation/prohibits for active-box, hybrid sensors, load-balance, and AMR configurations. |
| src/simulation/m_active_box.fpp | Adds active-box initialization/growth and debug envelope checking. |
| src/simulation/m_acoustic_src.fpp | Adds AMR-aware handling of acoustic source support (bounding boxes and overlap abort). |
| src/post_process/m_start_up.fpp | Calls AMR fine-data reader and AMR overlay writer when amr is enabled. |
| src/post_process/m_global_parameters.fpp | Adds default-off amr flag for post_process overlay behavior. |
| src/post_process/m_data_output.fpp | Implements AMR fine-block overlay mesh/variables output (Silo/binary) and multimesh registration. |
| src/common/m_phase_change.fpp | Exposes per-cell Newton iteration count and threads it through relaxation to support load-weighting. |
| src/common/m_global_parameters_common.fpp | Adjusts start_idx lifecycle/allocation and makes load_weight_wrt visible to GPU macros. |
| src/common/m_derived_types.fpp | Introduces a simple t_box type used by new partitioning infrastructure. |
| src/common/m_box.fpp | Adds box/partition arithmetic helpers (equal/weighted splits, box-from-splits). |
| src/common/m_boundary_common.fpp | Skips BC buffer population during AMR fine advance to rely on coarse-driven ghost fill. |
| docs/module_categories.json | Registers new modules under documentation categories. |
| docs/documentation/readme.md | Adds AMR section link to the documentation index. |
| .typos.toml | Adds project-specific abbreviations to the spelling allowlist. |
| D = ((gs_min(lp) - 1.0_wp)*cvs(lp))/((gs_min(vp) - 1.0_wp)*cvs(vp)) | ||
|
|
||
| #ifdef MFC_SIMULATION | ||
| if (relax .and. load_weight_wrt) then |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #1628 +/- ##
==========================================
+ Coverage 60.77% 61.98% +1.21%
==========================================
Files 83 93 +10
Lines 20872 25637 +4765
Branches 3101 4206 +1105
==========================================
+ Hits 12685 15892 +3207
- Misses 6121 6993 +872
- Partials 2066 2752 +686 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Member
Author
|
Upstream latent gap found during the MHD+AMR investigation (independent of this PR): Also for the record: MHD+AMR was attempted and re-gated on measured evidence rather than assumption — the coarse/fine seam is a continuous O(1) div(B) source that cleaning spreads but cannot remove (details in the amr.md support matrix row and commit ac203b1). |
…ng (rr=2 byte-identical)
…t-read/regrid-shift/IB-marker ratio (rr=2 byte-identical)
…r + @ref) — fixes Documentation linkcheck
…nfig-robust uniform golden); dynamic flow-past-IB config-sensitivity tracked separately
…2-core hosted CI runners; np=2 DDD79C8B already covers the distributed pb/mv path)
…p on flang GPU_PARALLEL_LOOP + Cray-only ACC_SETUP/TEARDOWN; fix direction + top-10 + Frontier confirm experiment
…n post_process (rr=2 byte-identical); use file's authoritative per-block extent + rr-way fine-cb
…t, ref_ratio**max_level tighter CFL) — prevents silent unstable fine advance
…atch (remove per-seam malloc in s_amr_fine_fine_halo; byte-identical)
…/ALLREDUCE/ALLGATHER over all blocks, not per-block; file layout byte-identical)
…re kernels (fix offload-runtime hang)
…+gctag) Replace the two O(global-grid) MPI_ALLREDUCE tag-frame reductions in s_amr_cluster (gtag, MPI_MAX) and s_amr_regrid child-nesting (gctag, MPI_LOR) with all-gathers of each rank's SPARSE tagged-cell linear indices (new helpers s_amr_union_gtag/gctag). Tags are 0/1 so the reconstructed field is byte-identical to the dense reduction and clustering stays rank-invariant; comm now scales with the number of tagged cells, not the whole grid. Validated byte-identical on 5 multi-rank AMR goldens (dynamic-regrid, multi-level-static, subcycle, restart, stretched-grid; np=2) + GPU np=1 on H200. NOTE: touching s_amr_cluster/s_amr_regrid made ffmt reindent their pre-existing +4 over-indentation (whitespace + comment re-wrap only, zero logic change) - this is the bulk of the diff; git diff -w isolates the ~90-line logical change.
…blocks) launches -> O(1) per category; byte-identical)
… (fm host array referenced in GPU_PARALLEL_LOOP bounds) s_amr_fine_slice (the c8b1006 device-packed seam halo) used the host local array fm(TB)/fm(TA) as GPU_PARALLEL_LOOP bounds, so nvfortran/Cray demanded fm PRESENT on device -> runtime 'FATAL ERROR: data in PRESENT clause was not found on device: name=fm(2:)'. AMD flang tolerated it; NVHPC (Phoenix) and CCE (Frontier) gpu-acc lanes aborted every np>=2 AMR test. Use the scalars na-1/nb-1 (na=fm(TA)+1 already computed on host) as the bounds - byte-identical, no host-array reference in the device region. Validated: 6 np=2 AMR goldens (244B1E42 F57C3A5B ADA042A2 EF58E377 4644A339 B7704247) pass on 2x H200 gpu-acc.
… (fm host array referenced in GPU_PARALLEL_LOOP bounds) s_amr_fine_slice (the c8b1006 device-packed seam halo) used the host local array fm(TB)/fm(TA) as GPU_PARALLEL_LOOP bounds, so nvfortran/Cray demanded fm PRESENT on device -> runtime 'FATAL ERROR: data in PRESENT clause was not found on device: name=fm(2:)'. AMD flang tolerated it; NVHPC (Phoenix) and CCE (Frontier) gpu-acc lanes aborted every np>=2 AMR test. Use the scalars na-1/nb-1 (na=fm(TA)+1 already computed on host) as the bounds - byte-identical, no host-array reference in the device region. Validated: 6 np=2 AMR goldens (244B1E42 F57C3A5B ADA042A2 EF58E377 4644A339 B7704247) pass on 2x H200 gpu-acc.
…d) dense tag field (per-rank memory now O(#tagged), byte-identical boxes)
…~1290^3 (2048^3+ target) The sparse-clustering encode gi+(mg+1)*(gj+(ng+1)*gk) in default integer overflows once (m_glb+1)*(n_glb+1)*(p_glb+1) >= 2^31 (~1290^3, and the 2048^3 this scaling work targets), silently corrupting decoded box coordinates. Widen locidx/allidx to integer(8) with int8 encode/decode and gather via MPI_INTEGER8; counts/displacements stay int32 (they count tagged cells, < 2^31). Byte-identical at tested grids (7 regrid/multilevel goldens pass).
…ustering compute O(nboxes*ntag) -> O(ntag*depth); byte-identical boxes)
Bumps the ffmt formatter pin in toolchain/pyproject.toml from 0.4.3 to 0.4.4 (fixes single-line-construct indent leak, sbryngelson/ffmt#5). Verified as a 0-change no-op on master formatting; Formatting/Lint Toolchain/Python 3.9-3.14 CI all green.
ffmt 0.4.4 (pinned by the prior commit, fixes sbryngelson/ffmt#5) corrects the pre-existing +4 over-indentation of s_amr_cluster/s_amr_regrid (a 0.3.x single-line 'do ...; end do' indent-leak that 0.4.3 tolerated as a buggy fixed point). Whitespace + doc-comment re-wrap only, zero logic change (git diff -w is comment-rewraps); m_amr.fpp and m_sfc_partition.fpp are the only two files affected. Unblocks clean formatting of subsequent edits.
…5, timing not measurable today Stress matrix in amr-bench/attach/promote/: promotion through a 3-deep cross-TU call chain over a sub-range, poisoned halo verified untouched, 25 advances with host/device deliberately diverged plus present-table pressure, 48 blocks of varying extents through one shared working set, and mid-run reallocation. All five pass. Mechanism detail for implementation: a module-scope deep type needs BOTH the parent descriptor and its members mapped - map(to:) on the parent alone leaves %sf null and the first kernel faults at nil. With promotion that setup runs once at init and the working set is never re-pointed, which is why this design sidesteps the attach problem that killed the view-based one. No speedup is claimed: the node showed a ~134 us per-region launch floor uniform across all 8 GCDs with the GPUs idle, versus ~20 us for the same construct earlier the same day, so a ~53 us mapping delta is inside the floor. MFC's own profile implies ~2-3x per region; the earlier 4.4x is an upper bound pending an idle node.
… on this compiler The premise was an attribution artifact. 'Dummy kernels pay 33-54 maps, module-state kernels pay 0' came from temporal attribution (correlation ids do not match across the two traces). Copies cluster: 11.5% of inter-dispatch intervals hold zero copies and one holds 2893, so a kernel dispatched in a burst reads as 0 regardless of need. The median of 20 per interval matches the deep-member count, so the ~20 maps/dispatch average stands - the per-kernel split does not. Every mechanism then measured directly by copy count, one variant per process under rocprofv3, needing no attribution and immune to the launch-time floor: deep-type dummies 9.4, flat-array dummies 9.4, pointer dummies 9.1, defaultmap(present:aggregate) 9.1, map(present,alloc:) 9.1, declare mapper 39.1 (4x worse), module state in every form including the exact freg pattern 32.8 (3.5x worse). Per-region map traffic is irreducible by interface or clause on this compiler, and module scope makes it worse. That retires promotion, interface flattening, declare mapper, and the view/attach design together. What survives: the tax is real (2.0 s of copies against ~1.5 s of kernels), but it can only be attacked by fewer block advances (amr_max_grid_size, ~20x, already shipped) or fewer regions per advance (kernel fusion, untried). Docs only; src unchanged. precheck clean.
…t, not per-operation cost Gap analysis of a real AMR run: 13.5 s span, 28.3% busy, 71.7% GPU IDLE. The idle is ~354k operations each separated by a ~15 us median gap, not a few big stalls. 95% of the operations are copies, so map traffic costs ~2.0 s of transfer plus ~5 s of serialization - about half the span rather than the 15% a busy-time reading suggests. A faithful reproducer matches the signature (81.6% idle, 18.7 us median gap vs 71.7% / 15.4 us). A clean sweep holding work and arrays fixed while varying only the number of regions gives 202.4 ms at 6555 operations down to 37.6 ms at 345: net of fixed startup, 20x less time for 19x fewer operations, with the median gap flat at ~18 us throughout. span = fixed + operations x ~18 us. nowait is refuted: the same dependent regions with nowait + depend and one taskwait per advance measured 213 ms against 202 ms, median gap unchanged. Priority order follows: fewer block advances (amr_max_grid_size, already shipped), then kernel fusion in s_compute_rhs which pays on both factors at once, then the 143 host-side stalls over 1 ms that account for 16.7% of the span. Docs only; src unchanged. precheck clean.
…, not just operation count Holding operation count fixed (721 kernels, 5834 copies) and varying only kernel duration: 14 us -> 20.5% busy, 37 us -> 25.6%, 201 us -> 51.2% (crossover), 763 us -> 77.4%. Crossover is ~200 us of kernel work per ~8 copies, so ~500 us at MFC's ~20 copies per kernel. MFC's mean kernel is 109 us, i.e. 4-5x below the point where work dominates latency. Conversely, raising work per region 256x at small sizes changed the span by 5% - in that regime work is free and only operation count matters. efficiency = kernel work / (kernel work + operations x ~21 us) This explains the ~20x from amr_max_grid_size mechanistically rather than empirically: a larger cap puts more cells under each kernel and moves the solver out of the latency-bound regime. It also upgrades kernel fusion, which raises work per kernel AND lowers operation count simultaneously. Corrects the previous entry, which treated operation count as the only variable. Docs only; src unchanged. precheck clean.
…not a toy-size artifact Every earlier measurement ran at ~15% of one GCD, so short kernels could have been an artifact of a small problem. Re-profiled at 4x the cells (8.4M -> 33.6M, ~60% of a 64 GB GCD): busy 28.3% -> 28.9%, GPU idle 71.7% -> 71.1%, median gap 15.4 -> 15.5 us, copies/kernel 20.2 -> 21.8. Busy fraction and per-operation gap are invariant to problem size. The mechanism explains why: under AMR the work per kernel is set by amr_max_grid_size, not by the global grid, so a larger domain produces more blocks of the SAME size. Growing the problem moves along the operation-count axis and leaves work-per-kernel untouched - scaling up does not escape the latency-bound regime, only raising the cap does. This strengthens rather than weakens the cap recommendation. Caveat recorded: the 60% run exited 134 on the same getTargetPointer-null mapping abort seen at 15%, after producing 28 s of trace; ratios match the smaller run but it is a partial run and that abort deserves separate investigation. Docs only; src unchanged. precheck clean.
… the plan to match Cross-block kernel fusion is the field-standard answer. AMReX launches one kernel over all boxes in a MultiFab (512 patches of 32^3 performing like a single 256^3 patch) and quantifies the problem on our exact hardware: a simple kernel on an MI250X reaches only ~10% of peak bandwidth on 32^3 boxes. They fuse halo pack/unpack down to one launch per rank. Parthenon does the same via MeshBlockPack with a runtime-tunable pack size. Castro's GPU gridding guidance recommends max_grid_size=128, independently corroborating the cap finding. OpenMP offload also carries higher per-region overhead than CUDA/HIP, which raises the value of cutting operation count. Measured here with the same arithmetic over 64 blocks: one launch per block 433.5 ms / 15370 operations, one launch over all blocks 35.0 ms / 250 operations - 12.4x, byte-identical results. Second finding: both rows show 0 copies per kernel because the store is a plain contiguous array. The per-region map traffic is not module-scope vs dummy argument, it is plain array vs derived type with a pointer component - module-scope scalar_field measured 32.8 copies/kernel, a plain module array measures 0. That is what a MultiFab is. Plan recorded in dependency order: flat contiguous backing store, batched block kernels, fused halo pack/unpack. Target ~3.5x from removing serialization on the 60%-of-GCD case before counting bandwidth gains. Retracts my earlier dismissal of batching: the profile shows dispatch count is the governing quantity and both reference codes converge on this mechanism. The separate slot-based packing disproof is a different mechanism and stands. Docs only; src unchanged. precheck clean.
amr_slots is indexed by GLOBAL block index and allocated lazily, so live slots are sparse across 1:amr_max_blocks (1024 by default). A contiguous per-block field store - the layout AMReX's MultiFab uses, and the prerequisite for launching one kernel over all blocks instead of one per block - must be indexed densely by a local index, or it would have to be sized for the whole global pool. Adds amr_loc_of (global slot -> dense local index, 0 if not live), amr_loc_n (high-water mark), and a recycle stack so the dense range stays tight under regrid churn. Handed out in s_amr_alloc_slot, returned in s_amr_free_slot. Nothing reads it yet: this is pure bookkeeping, so all 76 AMR/L0 goldens are byte-identical. The initialisation deliberately lives in s_amr_loc_index_init, called from BOTH pool-allocation sites - s_initialize_amr_module and s_l0_tiles_init. amr_slots is allocated in two places and the first sits behind 'if (.not. amr) return', while pure-L0 mode (amr = F) still reaches s_amr_alloc_slot; allocating at the first site only failed exactly the three L0-tiles goldens. precheck clean.
A field family cannot be migrated to the flat store independently if it shares a consumer routine with a family that is not migrating. q_ghost_a/b look like the smallest possible first conversion - 9 references, one file, two consumers - but s_amr_fill_fine_ghosts is a general prolong-coarse-into-fine routine whose third caller targets q_cons (m_amr.fpp:4597). Branching inside the routine does not rescue it: a dummy referenced in any branch of a target region is still mapped, so the migrated path would keep paying the tax and the conversion would buy nothing. The natural unit is {q_cons, q_ghost_a, q_ghost_b} together.
Method note: counting references to a family NAME understates blast radius; count the consumers, then the consumers' other targets.
Also records that q_ghost_a/b are allocated unconditionally while their only consumers are on subcycle-only paths, and their pb/mv twins already carry an if (amr_subcycle) guard - adding the matching guard frees ~2 x sys_size x mbuf x live_slots of device memory per non-subcycle run, memory that competes with amr_max_grid_size.
Docs only; src unchanged (step 1a stands at 905e9d7). precheck clean.
Concrete sub-step order for 1b (migrate q_cons + both ghost families together, since s_amr_fill_fine_ghosts targets all three) and step 2 (collapse the per-block loop into one region over 1:amr_loc_n, staging per-block extents into GPU_DECLARE'd arrays indexed by the local index). Records the verification protocol: every sub-step is a pure layout or scheduling change, so all 76 AMR/L0 goldens must stay byte-identical, and after step 2 the gap analysis must confirm the operation count actually fell - passing goldens with an unchanged operation count would mean the batching never engaged. Records four traps already paid for this session: the two pool-allocation sites, printing replaced text before substituting (a bub_pos_frac clamp was nearly rewritten to zero), never passing the store as a dummy, and the build-timeout and pgrep-self-match harness hazards. Docs only; src unchanged. precheck clean.
… q_cons is blocked by 8 shared-solver interfaces
…phic callees it forced
… linear in region count
… the art, AMR captures ~11% of its potential
…d 6.6x per-region overhead
… loss for register-heavy kernels
…ies per launch vs 0.00 is
…ers directly s_finalize_riemann_solver copied flux_rsx_vf into flux_n and flux_gsrc_rsx_vf into flux_gsrc_n. Both copies were vestigial: the rsx buffers are already flat, natural-order (x, y, z, var) module arrays allocated ONCE for all three directions, and both sides of the copy carried an identical index expression in every norm_dir branch. Lifetimes matched too - flux_n(2) and flux_n(3) aliased flux_n(1), so it was already one shared buffer per sweep. Consumers now read the rsx buffers directly. Deletes 4 x sys_size full 3D field arrays (flux_gsrc_n allocated per direction, unaliased), the copy kernel, three deep scalar_field dummies from four solver signatures plus s_cbc and s_compute_qbmm_rhs, and two GPU_ENTER_DATA(attach=) users. Bounds are safe by construction: finalize only ever wrote the is1/is2/is3 window, which is inside the rsx buffers because the solvers write rsx over that same window - so no correct consumer could read outside it. flux_gsrc_n's zero-init moves to s_initialize_riemann_solvers_module; collapsing its per-direction storage is equivalent because finalize already copied the whole shared rsx buffer into each direction's array. s_amr_capture_creg_dense_batch gains a 'flat' selector because it serves both the advective capture (now flux_rsx_vf) and the viscous one (still flux_src_n). It must be a branch, not a merge: merge evaluates both arms, and flux_src_n allocates only mom..E, so touching it across the full advective band would dereference null %sf pointers. Its flux reads stay INSIDE the bclo/bchi guards - a slot activates when either face is owned, and the unowned face's index lies a block width outside this rank's subdomain, which flux_n's wide ghost bounds tolerated and flux_rsx_vf's (-1:m_alloc) does not. scalar_field%sf is real(stp) and flux_rsx_vf is real(wp), so this also removes a lossy wp->stp roundtrip under --mixed. Verified: 706 passed / 0 failed (including AMR 2D axisymmetric, the only flux_gsrc coverage), plus sc3dx_amr 400^3 at np=8 clean - the goldens are all 1D/2D and cannot see rank-boundary memory safety.
… is written Completes the flux-family removal begun in 724ef4f. flux_src_rsx_vf already held the advection-source band in the same flat (x, y, z, var) layout, so s_finalize_riemann_solver had nothing left to copy and is deleted outright; its viscous mom..E band moves to the same array, which retires flux_src_n. m_cbc, all four Riemann solvers, s_compute_advection_source_term and s_amr_capture_boundary_flux now take no scalar_field flux dummy at all - the AMR capture, the second-most-launched kernel in the code, reaches zero field dummies. The adv band ALIASES adv%beg when riemann_solver /= HLL, and %vf(j_adv) is read in the non-HLL branch too (m_rhs 1254/1330/1403), so every non-HLL band read is collapsed to adv%beg explicitly. Getting that wrong is a silent wrong answer, not a crash. m_chemistry keeps a dummy because it lives in src/common and so cannot use m_riemann_state; it takes a flat real(wp) array instead of a scalar_field, which is measurably cheaper per launch. Sizing: these are FULL-DOMAIN arrays, so an unused component is pure waste. flux_gsrc_rsx_vf is not allocated at all in Cartesian runs (every write in the solvers and both reads in m_rhs sit under cyl_coord, and grid_geometry == 3 implies cyl_coord); flux_src_rsx_vf starts at the band actually written; qL_prim/qR_prim allocate their momentum payload only when viscous, since s_get_viscous and s_compute_viscous_source_flux are the only readers. creg/freg now grow geometrically instead of being dimensioned to amr_max_blocks - that is a safety CAP, not a block count, and at 400^3 it was a 34x over-provision costing 1.41 MB of device memory per unused slot. Measured at 400^3 np=8: peak device memory 49.83 -> 36.48 GiB per GCD (-26.8%). Verified 706 passed / 0 failed plus sc3dx_amr 400^3 np=8 clean - goldens are all 1D/2D and cannot see rank-boundary memory safety.
s_amr_gather_coarse_patch is called once per BOX - 794 of them per rebuild at 400^3 - and the contributing side used a BLOCKING MPI_SEND, so every rank rendezvoused with each box's owner in lockstep. Profiling put this routine at 45% of regrid and ~25% of total runtime across its two call sites, which is also why the AMReX head-to-head measured our MPI at 7.6x the calls. Sends are now non-blocking, completed by one WAITALL after the rebuild loop instead of a rendezvous per box. The pool owns the buffers because MPI_ISEND requires them live until completion - the old code freed sbuf immediately after the blocking send, which is exactly what must not happen here - and drains every 64 pending sends to bound both pool memory and outstanding requests. Tags are the slot index, so they were already unique per box. Only the rebuild_slots call site changes semantics; the other six flush immediately and behave exactly as before, so the blast radius is one loop. The owner still posts IRECV and waits per box, which is what preserves patch-assembly order. Measured over 3 reps at 400^3 np=8: gather 10.43 -> 8.43 s (-19%), regrid total 23.14 -> 21.23 s (-8%). The remaining cost is owner-side (per-box IRECV/WAITALL, sys_size GPU_UPDATEs per box), which needs the fully batched exchange. Verified 706 passed / 0 failed, 101/101 AMR including 8 np=2 cases, and sc3dx_amr 400^3 np=8 clean - a mis-ordered non-blocking send corrupts the receiver's patch silently, so the multi-rank cases are the real gate.
Reverses the 2026-08-02 conclusion that batching is ruled out because the tax is per-dispatch argument mapping. That rested on temporal copy attribution, which credits a kernel with copies emitted by intervening data regions - it made capture_boundary_flux look like 0.0 copies/dispatch when it does 8.73, and capture_creg look like 33 when it does 2.67. Controlled 2-factor microbenchmark (amr-bench/scaling) varying launch count and entity count independently, R^2 1.0000 on every HSA-call fit: private/local fixed-size ARRAYS and assumed-shape dummies cost exactly 2.00 copies, ~27 HSA calls and ~31 us per entity per launch, independent of array size. Private scalars, module arrays used directly, explicit-shape dummies and BLOCK-scoped arrays cost zero. The per-launch floor is only 6.2 HSA calls. Also records the measured phase budget, the runtime-knob null (12 settings, two of them 1.8-3.2x slower), and that copy COUNT converts to wall while copy BYTES does not.
…n the AMR arm) A fixed-size private ARRAY in a target region costs 2 copies and ~27 HSA calls per launch on amdflang, because the compiler materialises a per-thread private copy; the entity is never read on entry, so the transfer is pure waste. map(alloc:) keeps the privatisation and drops the transfer. Measured in a controlled microbenchmark at 2.00 copies per entity, R^2 1.0000, independent of array size. hllc:1013 names its 23 private arrays, weno:1117 its 6. Only ARRAYS - naming a scalar would demote it from pass-by-value to a real device allocation and cost more than it saves. Measured 3D 256^3 np=8 restart at steady state, interleaved triplicates, settled tail: 13.357 -> 9.827 s/step, -26.4% (1.36x), ranges non-overlapping. Copy count fell exactly as predicted (hllc 52.7 -> 6.67 per launch, weno 25.00 -> 13.00). Checkpoints after the change are BYTE-IDENTICAL to the baseline binary, both files. NOT VERIFIED on the other CI compilers: a list item appearing in both private() and map() on a combined construct is accepted by amdflang, where private binds to the loop and map to the target, but nvfortran, Cray ftn and ifx are untested here. CI must gate this. A third site (m_variables_conversion) removed its predicted copies exactly and bought no measurable wall, so it was reverted - removed copies only pay when they are on the critical path, and per-copy value ranges from ~0 to 26.6 us across sites.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
An opt-in, default-off family of performance features and the measurement infrastructure they rest on. With all flags at their defaults the only touched production path is
s_mpi_decompose_computational_domain, refactored through the newm_boxmodule (byte-identical; covered by the existing suite).m_box(partition arithmetic),m_load_weight/load_weight_wrt(per-cell load-weight field + imbalance metric),m_sfc_partition/sfc_partition_wrt(Morton-SFC predicted-imbalance diagnostic),m_load_balance/load_balance(weighted static decomposition at init; AMR-fine-work-aware),m_rank_timing/rank_time_wrt(per-rank compute-time diagnostic).m_active_box/active_box: restricts reconstruction/Riemann/RK windows to a light-cone-grown box around non-ambient flow; strict-subset golden-tested.hybrid_wenoandhybrid_riemann(+hybrid_weno_eps,hybrid_smooth_flux): linear-optimal weights / central-or-Rusanov flux in smooth cells, full WENO/HLLC at flagged discontinuities (Jameson sensor, stencil-dilated, per-level under AMR).m_amr+m_amr_registers: two-level 2:1 refined block hierarchy; conservative restriction and conservative-linear prolongation with physics-specific closures; per-stage flux registers with Berger–Colella refluxing; Berger–Rigoutsos multi-block dynamic regrid; optional dt/2 subcycling; multi-rank (single-owner blocks assigned by Morton-SFC work balancing at each regrid, with migration; blocks may span rank seams via P2P coarse↔fine gather/scatter; same-level seam halo; distributed registers); restart (both IO modes, regridded-layout persistence); AMR-aware post-processing (fine blocks visualizable as Silo overlay domains); GPU-resident fine level on both OpenACC and OpenMP offload.Full algorithm and user documentation:
docs/documentation/amr.md(support matrix enforced at runtime by the checker — unsupported combinations abort with named messages, never silently).AMR physics support matrix (abridged; authoritative table in amr.md)
Supported and golden-tested: single- and multi-fluid (5-eq,
mpp_lim) · 6-eq with per-block pressure relaxation · viscous (refluxed) · phase change (relax) · chemistry incl. species diffusion · Euler–Euler bubbles (polytropic/non-polytropic, mono/polydisperse, QBMM incl. non-polytropic with per-blockpb/mvside-state; dynamic regrid + subcycle) · acoustic sources (coarse-grid support with regrid exclusion) · immersed boundaries (multi-body, static or prescribed-motion, incl. dynamic regrid with body-containment expansion and per-substage guards) · 2D axisymmetric (per-block WENO-coefficient recompute) · stretched grids (exact parent-bisection ghost coordinates + per-swap coefficient recompute) · hybrid WENO/Riemann sensors (per-level) · Lagrangian bubbles (cloud excluded from blocks; two-way coupling on the coarse grid; regrid clips around the moving cloud) ·active_box(blocks contained in the growing window; agrees with plain AMR to ~1e-14) · IGR (restriction-only coupling: fine sigma solve seeded/Dirichlet-bounded by the coarse solve; documented truncation-order seam, exact free-stream) · 1D MHD/RMHD (div(B)=0 by construction in 1D; HLL and HLLD, incl. relativistic).Gated with named aborts (documented rationale): surface tension (seam force imbalance is structural — three fixes attempted and diagnosed in amr.md) · 2D/3D MHD (attempted and measured: the c/f seam is a continuous O(1) div(B) source GLM cleaning cannot remove — needs constrained-transport-class B prolongation/reflux) · hyperelasticity · 3D cylindrical (global azimuthal filter) · force-driven IB (
moving_ibm=2) · STL bodies · Riemann-extrapolation BCs (bc=-4) ·amr_subcycleunder IGR · stretched grids with Lagrangian/IB-regrid (uniform-spacing index conversions).Validation evidence
Known issues (all non-gating or in progress)
continue-on-error): an intermittent post-detected NaN on the two Lagrangian+AMR goldens. Exhaustively unreproducible off GitHub's runners — the exact failing stack (NVHPC 24.3 SDK,-tp=px -Kieee, HPC-X MPI, and the CI docker image itself under apptainer) passes elsewhere, as do native/zen2 builds; 24.5+ green. Documented at the golden definitions.Review guide
The commit history is arc-ordered (active-box → load-weight → SFC → weighted decomposition → rank timing → hybrid → m_box → AMR rungs → physics envelope → CI/GPU hardening); reviewing by arc is much easier than by file. The AMR arc builds stepwise: static hierarchy → restriction/prolongation → fine advance → refluxing → regrid → subcycling → multi-rank → GPU → each physics rung with its own validation. Commit messages carry the validation evidence for their change (measured defects, golden UUIDs, repro details for CI fixes).
All parameters ship default-off with
case_validatorentries, runtime checker gates, andcase.md/amr.mddocumentation.