amr: block-structured multi-level AMR (Body 2) — recursive nesting, subcycling, np≥2 - #6
Merged
Conversation
…ated) - Foundation for multi-level AMR nesting (arbitrary refinement depth L0..L_amr_max_level). Add the amr_max_level namelist param (default 1 = current single-level behavior); gate amr_max_level>1 fail-closed in m_checker (not yet supported), documented in case.md + descriptions.py. docs/documentation/amr_multilevel.md captures the design: the core generalization is 'coarse = L0' -> 'coarse for level l+1 = level l' throughout the coupling; FLAT block pool + per-block level/parent tag (so the SFC distribution / lazy owned-only alloc / repartition-restart / P2P migration machinery all carry over unchanged - level-agnostic); RECURSIVE subcycling with time-interpolated C/F BCs (Berger-Colella). Increments: (1) param+gate [this], (2) recursive coupling, (3) recursive subcycle, (4) per-level regrid+nesting, (5) restart/distribution/GPU per level. VALIDATED CPU reldebug: amr_max_level=2 aborts with the gate; 21C71558 + 6C20B752 (np=1) bit-identical (default 1 neutral).
…block_level(:) (refinement level of each active block, 1..amr_max_level) + amr_num_levels to the replicated block metadata in m_global_parameters, allocated + initialized to 1 alongside amr_block_owner and freed at finalize. Both are 1 today (single fine level); the regrid will tag each block's level once multi-level nesting lands. This is the foundation the level-aware coupling (increment 2b) keys off: a level-l block's coarse side is level l-1 (the L0 base grid when l==1), the covering coarser block found by region overlap on the replicated boxes. Bit-identical at amr_max_level=1 (the field is written but not yet read; the level>1 paths stay gated in m_checker). VALIDATED CPU reldebug -b mpirun: 21C71558 + 6C20B752 (np=1 IGR) + 5EFB3277 (np=2) all pass.
… - Add f_amr_parent_block(k): the covering level-(level(k)-1) block that block k refines - its coarse parent - or 0 when block k is level 1 (parent = L0 base grid), found by region overlap on the replicated boxes (proper nesting => exactly one). Pure, bit-identical (unused until the level-aware coupling wires it in). Document the ONE hard detail of the coupling in amr_multilevel.md: the prolong reuses its amr_cg frame unchanged IF amr_cg + (amr_isect_lo, amr_cpat_off, ref_ratio) are in PARENT-LEVEL cell indices. Level-2 block with L0 region R2, parent L1 block p (L0 region R1): parent-fine index of L0 cell c = 2*(c-R1.lo); coarse footprint isect = 2*(R2-R1.lo); fine extent m = ref_ratio^2*|R2|-1; amr_cg gathered from amr_slots(p)%q_cons; coarse coords = the parent's fine coords (needed only for advance/stretched, so a uniform-grid np=1 prolong->restrict operator self-check is the first testable milestone). s_set_amr_fine_geometry + s_amr_gather_coarse_patch each gain a level==1?L0:parent-fine branch. VALIDATED CPU reldebug: 21C71558 (np=1) bit-identical.
…lestone) - Implement the level-aware coarse frame so a level>=2 block's coarse side is its PARENT block's fine cells. s_set_amr_fine_geometry: for level>=2, express the footprint in parent-fine indices (isect = parent_ref_ratio*(R2 - parent R1.lo), m = ref_ratio*footprint) and build coords by bisecting the parent's fine coords. s_amr_gather_from_parent: fill amr_cg from the parent block's fine array in that frame (np=1 local copy; np>=2 P2P is future work). The prolong / restrict / conservation-check are REUSED UNCHANGED - they read (amr_isect_lo, amr_cpat_off, ref_ratio), which is all the coarse frame needs. s_amr_test_multilevel: a np=1 static self-test that nests a level-2 block inside block 1, populates it (level-aware gather + prolong), checks that restrict(fine) recovers the parent-fine coarse, then removes the block (non-intrusive). Relax the m_checker gate to allow np=1 amr_max_level=2 (development self-test; the recursive advance is increment 3). VALIDATED CPU reldebug np=1: MULTILEVEL self-test restrict-prolong conservation err = 0.0000E+00 (EXACT); full run completes and conserves (mass 1.2e-16 / energy 5.7e-16).
…- After the 2b L1->L2 coupling milestone (conservation err = 0.0 on CPU AND GPU/V100), record the increment-3 plan: build the coupling OUT of a level-2 block (restrict-to-parent + Berger-Colella reflux-to-parent, both required for conservation) + a persistent static L2 block + a level-loop driver in m_time_steppers. Lock-step FIRST (all levels at L0 dt, interleaved per RK stage - extends today's non-subcycle mode) to a np=1 static 2-level runs-and-conserves milestone, THEN recursive subcycling on top (generalizing s_advance_amr_fine_substeps, already Berger-Colella for L0<->L1).
…rement 3 step 1) - s_restrict_fine_to_coarse gains a level>=2 branch: instead of folding into the L0 coarse_tgt, s_amr_restrict_to_parent restricts the block's fine averages into its PARENT block's fine array over the covered cells, using the SAME s_amr_restrict_overwrite_device child-sum kernel targeted at the parent in the parent-fine frame (amr_isect already parent-fine; offset 0 = the parent's local fine indexing). Mirror of the 2b gather-from-parent; np=1 local, np>=2 P2P scatter is future work. The self-test now also folds L2 back into the parent and checks the covered cells are unchanged (restrict(prolong) = identity). VALIDATED CPU reldebug np=1: restrict-prolong conservation err = 0.0 AND restrict-to-parent identity err = 0.0 (both EXACT); run conserves (mass 1.2e-16). Next (increment 3): persistent L2 block + level-loop driver (advance L2), then Berger-Colella reflux-to-parent for exact conservation of the running case.
…arent (increment 3 step 2) Mirror of restrict-to-parent: a level>=2 block's fine flux registers correct its PARENT block's cells (the coarse side of level l is level l-1), not the L0 grid. s_amr_reflux_to_parent + s_amr_reflux_apply_parent apply the same Berger-Colella correction as the L0 s_amr_apply_reflux but target amr_slots(pblk)%q_cons in the parent-fine frame: the block footprint is amr_isect (already parent-fine, offset 0), so the outside cells are amr_isect_lo-1 / amr_isect_hi+1, and q_parent(outside) += dt*(F_coarse - Fbar_fine)/dxf (low) / +(Fbar_fine - F_coarse)/dxf (high), Fbar_fine = child-averaged fine register. Three face blocks; collapsed dims pin to 0. State form (the lock-step advance applies it after restrict-to-parent); np=1 local, np>=2 P2P freg delivery is future work; uniform parent-fine dx (stretched-grid coords TODO). creg is now public from m_amr_registers (freg already was). The np=1 self-test (s_amr_test_multilevel) gains a reflux check: inject a known x-face C/F flux mismatch into the L2 block's registers, confirm the correction deposits exactly the mismatch/dxf into the parent's outside cells (reflux is conservative - the flux crossing the boundary is redeposited, not lost), then RESTORE the perturbation so the self-test stays non-intrusive. VALIDATED CPU np=1: reflux-to-parent conservation err = 3.5527E-15 (machine zero) alongside restrict-prolong = 0 and restrict-to-parent identity = 0; full run conserves. Single-level AMR goldens 21C71558 + 852CCB81 bit-identical (all new code is level>=2-gated; creg export neutral). Next (increment 3): persistent L2 block + level-loop driver, then the lock-step advance conserves.
The np=1 self-test (s_amr_test_multilevel) no longer removes the level-2 block it nests: amr_num_blocks stays = L2 and amr_block_level(L2) = 2, so the block persists across timesteps for the advance driver to step. The self-test's operator checks (restrict-prolong, restrict-to-parent identity, reflux-to-parent) still run at init. GPU-safe non-intrusiveness: those checks fold L2 into block 1 and perturb its outside cells on the DEVICE, so the whole intrusive region is now bracketed by a FULL block-1 save (device->host->buffer) and restore (buffer->host->device). The previous partial per-cell restores left block 1's device copy inconsistent - a GPU-only NaN by step 3 that CPU (host==device) never showed. The persistent parent now enters the advance exactly as s_populate_amr_fine left it. The advance is not yet level-aware (increment 3 step 4), so the three m_time_steppers block loops (stage fill, stage advance, post-stage restrict/reflux) skip level>=2: the persistent L2 is INERT (populated at init, not filled/advanced/restricted), so it cannot mis-couple to L0 or fold stale fine data into the evolving parent. s_amr_conservation_check returns early at amr_num_blocks>1 and the mass/energy drift sums the L0 grid, so the inert nested L2 is invisible to conservation. VALIDATED np=1 static (amr_regrid_int=0) 2-level, 6-7 steps, CONSERVES on CPU AND GPU (nvhpc acc, V100): mass drift 6.99E-13 / energy 9.89E-13 (identical CPU/GPU); self-tests 0 / 0 / 3.55E-15. Single-level AMR goldens 21C71558 + 852CCB81 + 5EFB3277 bit-identical. Next (increment 3 step 4): the level-loop driver makes the L2 live (fill/advance/restrict/reflux to parent) - removes the guards.
…N (increment 3 step 4) Lock-step multi-level advance in m_time_steppers: FILL every block top-down (level-aware s_amr_fine_stage_fill gathers a level>=2 block from its PARENT), ADVANCE all blocks, RESTRICT bottom-up (reverse slot order so the child folds into its parent before the parent folds into L0). level>=2 blocks skip the L0 reflux (parent-flux capture is step 4b). The intractable GPU-only NaN was a one-line omission: s_amr_test_multilevel prolongs the persistent L2 block via s_interpolate_coarse_to_fine (s_prolong_one_var is a HOST loop) but never pushed the result to the device, so the freshly ACC_SETUP'd L2 device q_cons stayed NaN and folded into the parent via restrict-to-parent. Added the GPU_UPDATE(device) after the prolong, mirroring s_populate_amr_fine. Invisible on CPU (host==device); localised with count(x/=x) after nvfortran maxval hid the NaN, and compute-sanitizer memcheck came back clean (ruling out OOB, pointing at the missing host-to-device sync). Also: level-aware s_amr_swap_to_fine ghost coords (a level>=2 block sources its parent's fine coords, not the L0 amr_g?cb which its parent-fine amr_isect indexes out of bounds); gather-from-parent copies via a device kernel (s_amr_copy_parent_patch, parent q_cons passed as an argument - present-table safe like s_amr_restrict_overwrite_device). VALIDATED np=1 static 2-level, CPU and GPU (V100): self-test errs 0 and 3.55e-15; full run completes with NO NaN and conserves (mass 5.0e-13, energy 7.1e-13, identical CPU/GPU).
…o (np=1, 2 levels) Make s_amr_regrid hierarchical so level-2 blocks are created/destroyed dynamically nested inside level-1 blocks (replacing the static self-test block), and add the L2->L1 Berger-Colella reflux that the dynamic case needs for exact conservation. Regrid (m_amr.fpp): after the L0->L1 boxes are clustered, append a nested level-2 box per level-1 box (A1 uses a fixed inset; the fine-solution sensor is a follow-up), tag amr_block_level per box, and order the box list parents-first so the build loop fills a parent before its child's gather-from-parent reads it. THE conservation-critical fix: the new block's overlap-copy (which restores fine detail from the old blocks' stash) must only copy from SAME-LEVEL old blocks - an old level-2 stash is in the 4x parent-fine frame, so copying it into a new level-1 block with the L0-frame shift 2*(isect-old_ilo) corrupted the parent (the L2 came out uniform high-density, injecting +6% mass on the first advance). Track old_level per old block and skip cross-level overlap sources; level-2 destinations re-prolong from their fresh parent (detail-preserving same-level L2 migration is future work). Reflux (m_amr_registers.fpp + m_time_steppers.fpp): capture creg for each level-2 child from its PARENT block's fine flux during the parent's advance (child footprint = amr_isect in the parent-fine frame), rk3_w-weighted for the once-per-step STATE correction; give freg the same rk3_w weighting for level-2 blocks; skip level-2 blocks in the L0 coarse-creg loop; call s_amr_reflux_to_parent(dt) per level-2 block in the lock-step restrict loop. Advective flux only - viscous/chemistry-diffusion multi-level reflux is now gated fail-closed in m_checker. VALIDATED CPU np=1 (scratchpad hyb_dyn.py, central shock, reflective bc): dynamic level-2 regrid tracks the moving shock and conserves mass 1.97e-16 / energy 0.0 (was 0.12 before the overlap-copy fix, 3.9e-5 with the fix but no reflux). No regression: single-level dynamic regrid 3.9e-16, static multi-level self-test 5e-13 with errs 0/0/3.55e-15.
…(np=1) The prior commit gated viscous/chemistry-diffusion multi-level fail-closed because the L2->L1 reflux captured only the advective flux. It turned out the NaN that motivated the gate was NOT the reflux but the viscous CFL: the level-2 cells are ref_ratio^2 finer, so their parabolic (diffusion) stability limit is 16x tighter, and lock-step advances every level at the L0 dt - so the L2 viscous advance was simply unstable. With a dt that respects the finest level's viscous stability, viscous multi-level runs; the efficient fix (only the fine levels take small substeps) is subcycling (still TODO). For conservation, extend the L2 child-creg capture to add the viscous (flux_src on mom..E) and chemistry-diffusion (flux_src on species, plus energy when not viscous) face fluxes, so the L2 reflux matches the TOTAL advective+viscous+diffusive flux exactly as the single-level coarse creg already does. Drop the m_checker gate. VALIDATED CPU np=1: viscous multi-level (scratchpad hyb_dyn_visc, Re=100, dt=5e-6 viscous-stable, reflective bc) conserves mass 0.0 / energy 0.0 (machine zero); at too-large dt it NaNs on the L2 viscous CFL as expected. No regression: inviscid multi-level 1.97e-16, viscous single-level 3.9e-16. GPU validation to follow.
Replace the single level-2 append pass in s_amr_regrid with a hierarchical loop that appends a box at level l nested inside each level-(l-1) box for l = 2..amr_max_level, tracking each box's level in box_level(:) and ordering parents-before-children so the build loop fills a parent before its child's gather-from-parent reads it. The assembly now tags amr_block_level from box_level and sets amr_num_levels = max level present. Drop the amr_max_level > 2 checker gate. Everything downstream was already recursive: the level-aware geometry/gather/prolong/restrict (a level-l block's coarse side is its level-(l-1) parent), the child-creg reflux capture (captured during ANY fine block's advance for its level+1 children), and the reverse-slot-order restrict/reflux (finest first) all key off amr_block_level, so no per-level special-casing was needed. VALIDATED CPU np=1: amr_max_level=3 (scratchpad hyb_l3.py, m=255, reflective bc) nests L1[121:135] > L2[124:132] > L3[127:129] and conserves mass 1.97e-16 / energy 0.0 (machine zero). No regression: amr_max_level=2 still 1.97e-16. Fixed-inset placement shrinks ~2*inset per level, so deep nesting needs a broad feature (sensor-on-fine tagging is a follow-up). GPU validation to follow.
…havior-preserving refactor Split s_advance_amr_fine_substeps into (a) L0-sourced ghost setup and (b) a new recursive subroutine s_amr_advance_subtree that subcycles the currently-selected block (amr_cur) over its two SSP-RK3 substeps and zeroes/accumulates its flux registers. Pure code motion: the substep loop body is byte-identical, so depth-1 (single-level L0<->L1) behavior is unchanged. The recursive keyword + a future child clause are the seam for multi-level subcycling (inc.2). VALIDATED CPU: 10/10 AMR subcycle/two-fluid/viscous/QBMM/multi-level goldens byte- identical. VALIDATED GPU (V100, OpenACC): --gpu acc build clean; subcycle (852CCB81) + multi-level (75AD6885) golden-match; self-tests exact, mass/energy drift ~1e-13.
…l at its own dt Build the coupling into a recursion out of inc.1's s_amr_advance_subtree: - s_amr_advance_subtree gains a dt_sub arg (top call passes amr_dt_fine=dt/2; the recursion halves), so per-level dt (L1 dt/2, L2 dt/4, ...) falls out of the recursion. - s_amr_gather_from_parent_field(pblk, qp): gather a level>=2 block's coarse patch from a SPECIFIC parent snapshot field (reuses the existing s_amr_copy_parent_patch GPU kernel); the lock-step s_amr_gather_from_parent now delegates to it. - s_amr_advance_children: after each parent substep (parent @ t_b in q_cons, t_a in q_cons_stor - both live for free, no new storage), for each level+1 child gather its two ghost-lerp sources from those snapshots, recurse at dt_sub/2, then fold back with s_amr_restrict_to_parent + per-substep Berger-Colella s_amr_reflux_to_parent(dt_sub). - Driver (m_time_steppers) skips level>=2 blocks in the top-level subcycle loop; they are advanced/restricted/refluxed inside the parent's recursion. m_amr_registers needs ZERO changes: the per-substep weights (freg 1/r*rk3_w, creg rk3_w from the MFlowCode#31 lock-step child-capture) already close conservation, and the capture/consume timing is self-consistent across the recursion. inc.2 adds NO new GPU device kernels. VALIDATED np=1: static 2-level + amr_subcycle=T (case D95E1B08). L2 genuinely active (2-level vs 1-level solution differs 1.66e-2 where the wave crosses the block); global drift 9.7345E-10 mass / 1.3777E-09 energy = IDENTICAL to the single-level subcycle baseline (L2<->L1 reflux is internal, adds zero global leak). Self-test exact (reflux 3.55e-15). All 10 existing AMR goldens byte-identical. CPU and GPU(V100/OpenACC) results byte-for-byte identical - no present-table/sync divergence. New golden D95E1B08 protects the recursion.
…l subcycle conservation s_amr_advance_children is called as s_amr_advance_children(amr_cur, ...), so its intent(in) dummy pslot is argument-associated with the module variable amr_cur. The in-loop s_amr_select_slot(kc) reassigns amr_cur and silently corrupted pslot, so the final restore left amr_cur on the child slot: the driver's L1->L0 restrict/reflux then operated on the child, the L1 fine solution was discarded, and apply_reflux_state used the child's registers -> conservation leak (up to ~1e-3 with a moving L2; masked in the static golden where the child sits in smooth flow). Copy pslot to a local read before any slot switch. Regenerate D95E1B08 (captured the buggy state) and add 00E15144 (dynamic regrid + subcycle + multi-level). CPU+GPU(V100) both conserve to machine zero (~4e-15); full AMR golden group 47/47.
… refinement towers) Multi-level (amr_max_level>1) was hard-gated to num_procs=1. Enable it for a STATIC hierarchy (amr_regrid_int=0) at np>=2 via co-located refinement towers: the L2 child inherits its parent's rank (s_amr_test_multilevel already sets amr_block_owner(L2)=amr_block_owner(1)), so the L1<->L2 fold stays LOCAL (bit-identical to np=1) and only L0<->L1 crosses ranks via the existing Body-1 P2P. Changes: (1) checker gate now only prohibits DYNAMIC (amr_regrid_int>0) multi-level at np>1 - the cross-rank sensor-on-fine nesting is still np=1; (2) un-gate s_amr_test_multilevel for np>=2; (3) owner-guard s_amr_restrict_to_parent + s_amr_reflux_to_parent - without them the lock-step L2->parent fold dereferences a non-owner's unallocated parent slot (SIGSEGV on rank 1); the subcycle path already guarded. Validated: static 2-level conserves machine-zero at np=1/2/4, both lock-step and subcycle; the 4 np=1 multi-level goldens are unchanged (guards are no-ops at np=1). New golden 4644A339 (multi-level static np=2). GPU np=1 unchanged (no-op guards); np>=2 GPU validation is the multi-GPU scaling study (sbatch).
…sting + co-located owners)
… fine-block migration segfaults; use n>0 not num_dims which is unset at checker time)
…r arg + fine-fine halo buffer sizing on non-owner ranks)
…(missing per-substep fine-fine seam halo, not reflux/regrid ordering)
… subcycle driver for per-substep fine-fine seam halo)
…el-aware fine-fine halo extent + reflux seam-face exclusion + enable L2 tiling)
…zero + static under-refine) 1) s_amr_tile_box: clamp the per-dim tile size to >= 1. The lock-step multi-level nesting path passes amr_maxc_fit/2, which is 0 when a rank's fine half-extent is 1 (small subdomain at high np) -> integer div-by-zero (SIGFPE) instead of refining. 2) static multi-level (amr_max_level>1, amr_regrid_int=0) built its only L2 in s_amr_test_multilevel, which SILENTLY returned (no L2, under-resolved 'success') when amr_max_blocks was too small. Abort at the point of failure (the L1 tile count is only known there) + a checker gate for the trivially-detectable <2 case.
- s_amr_fine_slice now moves only the buff_size-deep near-seam slab it packs/ unpacks device<->host (interior transverse only), replacing the two full-block q_cons GPU_UPDATE loops in s_amr_fine_fine_halo. Byte-identical (same cells, same order) at a fraction of the PCIe volume - the halo runs per stage, 6x per fine step after the MFlowCode#28 transpose. - s_amr_reflux_to_parent: swap the sibling-seam detection nest (block outer, dim inner) so f_amr_parent_block - an O(nblocks) scan - is evaluated once per sibling instead of once per (sibling, dim). Behavior-identical. Both from the high-effort branch review. GPU (V100): all L1/L2-tiling np=2 goldens byte-identical (EF58E377 tiled-L2, F57C3A5B, 244B1E42, ADA042A2, 5EFB3277, ...).
…construction s_amr_test_multilevel was the ONLY builder of the static (amr_regrid_int=0) level-2 hierarchy, but it also ran ~90 lines of intrusive development checks on every populate: block-1 full save/restore, restrict-prolong / restrict-to-parent / reflux-to-parent identity probes (perturbing creg/freg), and three ES12.4 prints PER STEP. Those identities are now permanently protected by the static multi-level goldens (75AD6885, 4644A339, ADA042A2, D95E1B08) and the runtime conservation-defect probe, so the scaffolding is dead weight (and self-test spam) in production. Remove the checks, keep only the construction (inset -> L2 metadata -> reconcile -> fine geometry -> prolong -> device push -> persist), and rename the routine to s_amr_build_static_multilevel to match its now-sole purpose. The checks restored block 1 exactly and registers are re-zeroed each step, so output is byte-identical: GPU (V100) static ML goldens all pass unchanged (prints, which no golden captures, are simply gone).
s_amr_apply_reflux_state (L0/L1, coarse/sidx frame, unit weights from ownership, ref_ratio=2) and s_amr_reflux_apply_parent (L2->L1, parent-fine frame, sibling-seam weights, ref_ratio=slot) were near-verbatim copies of the same three-face flux correction, so a future physics/stencil fix could silently drift the two apart and break conservation on one path only (the review's concern). Extract the shared math into s_amr_reflux_apply_faces(q, islot, rr, dtl, olo, ohi, glo, ghi, woff, w_lo, w_hi, mlo, mhi): the caller passes all framing (outside index, creg-local loop range, transverse write origin, per-face weights, cell widths). Both callers become thin wrappers; the L2->L1 copy is deleted (net -60 lines). A zero weight now SKIPS the write (not multiply-by-0) so an unowned coarse face's out-of-bounds outside index is never touched. Output uses the wp-add store (matches the coarse path; byte-identical to both in the default double build, and the more accurate choice in the untested --mixed config). GPU (V100): all 12 subcycle-single-level + multi-level goldens (1D/2D, static/dynamic/ tiled) byte-identical. (A GPU-only present-table trap from array-element loop bounds was caught by this validation and fixed by hoisting the bounds to scalars.)
The coarse-side creg capture (islot loop, region/sidx frame, per-face ownership gating) and the child creg capture (kc loop, parent-fine frame, both faces) in s_amr_capture_boundary_flux were near-duplicate: same advective + total-flux-viscous + chemistry-species/energy math over two frames. A future physics term added to one side but not the other would silently break conservation on one path (the review's concern). Extract two shared kernels: s_amr_capture_creg_dense (dense eq range, per-face gated, accumulate-or-overwrite via merge(creg,0,acc)+cf*flux) covers advective and viscous; s_amr_capture_creg_chem covers the species-seq + conditional-energy chemistry capture. The coarse and child branches now just compute their frame and call the same kernels (net ~-190 lines). The merge idiom is byte-identical to the old if(accum)/else and never reads uninitialized creg (merge selects, no arithmetic). GPU (V100): 20 goldens across inviscid / viscous / chemistry / bubbles / QBMM / multi-level / subcycle / 3D all byte-identical.
sbryngelson
marked this pull request as ready for review
July 13, 2026 01:22
|
Claude Code Review Head SHA: 99ff235 Files changed:
Findings:
|
|
Claude Code Review Head SHA: 99ff235 Files changed:
Findings:
|
…ware extents, nesting guard Review of PR #6 (amr-multilevel). Correctness/safety fixes: - static builder (s_amr_build_static_multilevel): fail-closed guards for an inverted L2 box (small parent) and for an L2 whose L0-extent > amr_maxc_fit/2 (would overrun the creg register 0:amr_maxc_fit-1 in the L2->L1 reflux capture) — the dynamic regrid path already caps/clamps. - checker: gate multi-level+IB at ALL rank counts (was np>1 only); gate static AMR (amr_regrid_int=0) to amr_max_level<=2. - proper-nesting guard in the regrid: abort fail-closed if a level>=2 block overlaps != 1 parent-level block (gather/reflux take the FIRST overlap; a tile straddling a parent-tile seam would silently couple to one parent — wrong BC + leak). - stretched-grid reflux-to-parent: per-face parent-fine dx mirroring s_amr_apply_reflux_state (byte-identical on uniform grids). - level-aware extents: tower load-weight and the regrid stash old_ext use 2**level (level-2 is 4x its L0 footprint); level-1 byte-identical. Fast-path also compares box_level; guard tower roll-up against amr_block_level(0). - fix stale comment: the child creg captures total flux (advective+viscous+chem), not advective only, so viscous/chem multi-level conserves. Validated: 8/8 multi-level AMR goldens byte-identical on GPU (V100); guards are fail-closed.
The amr_max_level description in case.md, descriptions.py, and amr_multilevel.md still said only level 1 was supported / multi-level was 'planned', contradicting the shipped, golden-tested amr_max_level=2 (static up to 2, dynamic regrid deeper). Correct all three.
…path (MFlowCode#691) s_amr_copy_parent_patch synced the gathered coarse patch amr_cg device->host after every parent gather, but the runtime C/F ghost-fill (s_amr_fine_stage_advance lock-step, s_amr_fill_fine_ghosts subcycle) reads amr_cg on the DEVICE (filled by the copy kernel); only the init/regrid host prolong and the restrict-prolong self-test consume it on the host. Thread a to_host flag through s_amr_gather_from_parent_field -> s_amr_copy_parent_patch (= .not. pull_host: init/regrid true, runtime false; subcycle call sites explicit false) so the D2H runs only when a host consumer follows. Removes a per-substep, per-level>=2-block whole-patch device->host transfer. Validated: 8/8 multi-level AMR goldens byte-identical on GPU (V100).
…t round-trip) s_amr_gather_coarse_patch (the L0->L1 coarse-BC gather) pulled q_coarse device->host, unpacked it into amr_cg on the host, then pushed amr_cg host->device - a full round-trip every RK stage per fine block. At num_procs==1 the sole owner holds every covered coarse cell, so copy q_coarse (device) -> amr_cg (device) with a DEVICE kernel over the in-domain patch (same index map as s_amr_unpack_patch), and skip the q_coarse pull. Gated on pull_host (runtime): at init/regrid q_coarse's device copy may be stale, so those keep the host path. np>1 keeps the host pack for the MPI P2P gather. amr_cpat_off hoisted to scalars for present-table safety (mirrors s_amr_copy_parent_patch). With this + the MFlowCode#691 amr_cg fix, the np=1 AMR advance's coarse-BC gather (L0->L1 and L1->L2) stays entirely on-device per stage - no host round-trips except the seam halo. Validated: full 56-test AMR golden suite byte-identical on GPU (V100).
…k/active copies were non-conforming at amr_max_level>=2)
…evel nesting (IB path left it at the cluster count, overrunning on child append)
…clears the body (refine the surface, not the interior)
…>=2, owned-only IB sizing, ref_ratio=4)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Body 2 of the block-structured AMR family: generalizes the single-level AMR on
up/mega(MFlowCode#1628, Body 1) to arbitrary refinement depth (L0…L_amr_max_level) with recursive Berger–Colella coupling. Stacked onup/mega— merge after Body 1. Draft.30 commits, built incrementally; each validated (CPU reldebug + GPU/V100) as it landed.
Increments
amr_max_levelnamelist param (default 1 = current behavior);amr_max_level>1fail-closed inm_checker; design roadmap indocs/documentation/amr_multilevel.md. Core idea: replace "coarse = L0" with "coarse for level l+1 = level l" throughout the coupling; flat block pool + per-block level/parent tag so SFC distribution / lazy alloc / repartition-restart / P2P migration all carry over level-agnostically.amr_block_level; parent-block finder; a level-l block's coarse side is its parent block's fine cells (parent-fine index frame). gather-from-parent / prolong / restrict / conservation-check reused unchanged. np=1 restrict∘prolong = identity, exact.m_time_steppers(lock-step). Conserves to machine zero, CPU + GPU. Dynamic per-level regrid + L2→L1 reflux; viscous/chemistry via total-flux reflux; generalized nesting to >2 levels.np 2parallel test cases don't work with--no-mpibuild option MFlowCode/MFC#26 — recursive subcycling: each level advances at its own dt; extracts_amr_advance_subtree; fix pslot/amr_cur arg-aliasing conservation bug.m_rhs? MFlowCode/MFC#27 — np≥2 multi-level: co-located refinement towers + distributed nesting (global tag frame +MPI_ALLREDUCE(LOR), owner-guarded sensor); static and dynamic; 1D → 2D/3D.2**level, not2×) + reflux seam-face exclusion; enables tiled/wide L2 at np≥3.Validation
Known scope / follow-ups
s_amr_advance_children) — low priority; scaling uses lock-step.