amr: block-structured multi-level AMR (Body 2) — recursive nesting, subcycling, np≥2

docs/documentation/amr_multilevel.md

@@ -0,0 +1,90 @@
+# Multi-level AMR nesting — design and implementation plan
+
+Status: **multi-level nesting implemented.** The block-structured AMR core supports arbitrary
+refinement depth (L0, L1, …, L`amr_max_level`, 2:1 per level): static AMR (`amr_regrid_int = 0`)
+nests one level-2 block, dynamic regrid (`amr_regrid_int > 0`) nests deeper and per-level. This
+document is the design record. It was implemented in behavior-preserving increments: at
+`amr_max_level = 1` the code is bit-identical to the single-level core.
+
+## The one assumption to generalize
+
+Everywhere in `m_amr`, "coarse" means the **L0 base grid**:
+
+- `t_level%%region` is a box in **L0 cell indices**.
+- Every coupling routine reads/writes the L0 fields (`q_cons_base` / `q_cons_ts(1)`):
+  gather (`s_amr_gather_coarse_patch`), prolong (`s_interpolate_coarse_to_fine`),
+  restriction (`s_restrict_fine_to_coarse`), reflux (fine flux corrects L0).
+- The advance driver (`m_time_steppers`) loops the single fine-block pool once per L0 step.
+
+Multi-level replaces "coarse = L0" with "**the coarse side of level `l+1` is level `l`**".
+
+## Target architecture
+
+- Levels `0 .. amr_max_level`. A level-`l` block (`l ≥ 1`) refines a covering level-`(l-1)`
+  region by `ref_ratio` (2 today). Level `l+1` must be **properly nested** inside level `l`
+  (surrounded by level-`l` cells; never adjacent to level `l-1`).
+- **Flat pool + per-block level tag** (chosen over a per-level pool array): `amr_slots` stays
+  one pool; each block gains `level` and `parent` (the covering coarser block, or 0 for an
+  L1 block whose parent is L0). Rationale: the distribution machinery (SFC owners, P2P
+  gather/restriction/reflux/migration, lazy owned-only slot allocation, repartition-on-restart)
+  all operate on the flat pool and are level-agnostic, so they carry over with near-zero change.
+  A per-level pool array would force rewriting every `amr_slots(k)` reference (high churn,
+  silent-bug-prone — the same reason the #4 lazy-alloc avoided a global→local pool remap).
+- **Recursive subcycling** (chosen over lock-step): `advance(l)` advances level `l` by `dt_l`,
+  then for `s = 1..ref_ratio` recursively advances level `l+1` by `dt_{l+1} = dt_l/ref_ratio`
+  with time-interpolated C/F boundary data from level `l` (extends today's 2-level subcycle,
+  `q_ghost_a/b`, recursively), then refluxes `l+1 → l`. Most accurate and efficient; the
+  standard Berger–Colella time integration.
+
+## Increments (each validated np=1 bit-identical + np≥2 conservation + GPU)
+
+1. **Foundation (bit-identical).** `amr_max_level` namelist param (default 1), gated
+   `amr_max_level > 1` fail-closed in `m_checker` until the recursion lands. *(this increment)*
+2. **Recursive coupling.** Add the per-block `%%level`/`%%parent` tags (done: `amr_block_level`
+   in 2a; `f_amr_parent_block(k)` finds the covering coarser block by region overlap), then
+   parameterize gather/prolong/restriction/reflux by a coarse-level source: the level-`l` block
+   data instead of only `q_cons_base`. The L1↔L0 path is the `l = 0` case and stays byte-identical.
+
+   **The one hard detail — the coarse frame.** The prolong reads `amr_cg` via
+   (`amr_isect_lo`, `amr_cpat_off`, `ref_ratio`) and is reused unchanged as long as `amr_cg` and
+   that frame are in **parent-level cell indices**. Level-1 block: parent level is L0, frame is
+   L0 indices (today). Level-2 block with L0 region `R2`, parent L1 block `p` with L0 region `R1`:
+   - parent-fine index of L0 cell `c` is `2*(c - R1.lo)` (ref_ratio per level);
+   - coarse footprint in the parent's fine frame: `isect = 2*(R2 - R1.lo)`;
+   - fine extent `m = ref_ratio*(parent-fine footprint) - 1 = ref_ratio^2 * |R2| - 1`;
+   - `amr_cg` holds the parent's fine cells `[isect.lo - nmar : isect.hi + nmar]`, gathered from
+     `amr_slots(p)%%q_cons` (np=1: local copy; np≥2: P2P via `amr_block_owner`);
+   - "coarse coords" are the parent's fine coords `amr_slots(p)%%x_cb`, not `amr_gxcb` — needed
+     only for the advance/stretched grids, so a **uniform-grid np=1 operator self-check**
+     (`prolong → restrict` conserves) is the first testable milestone and can skip coords.
+   `s_set_amr_fine_geometry` and `s_amr_gather_coarse_patch` choose this frame; each gains a
+   `level == 1 ? L0 : parent-fine` branch.
+3. **Advance (make level-2 evolve).** 2b built the coupling *into* a level-2 block
+   (gather-from-parent, prolong); this builds the coupling *out* of it plus the driver:
+   - **restrict-to-parent** — level-aware `s_restrict_fine_to_coarse` folds a level-2 block's
+     fine averages back into its parent L1 block's fine array (mirror of gather-from-parent,
+     same parent-fine frame);
+   - **reflux-to-parent** — the Berger-Colella C/F flux correction from L2 into L1's cells
+     (the reflux registers key off "the coarse", which becomes level l-1);
+   - a **persistent static L2 block** (replacing the non-intrusive self-test), and a
+     **level-loop driver** in `m_time_steppers`: for level 1..maxlevel, fill+advance from the
+     parent then restrict+reflux to it.
+   Both restrict-to-parent and reflux-to-parent are required for conservation.
+   **Lock-step first** (all levels advance at L0's `dt`, interleaved per RK stage — extends
+   today's non-subcycle mode; no time-interpolation/recursion): first milestone is a np=1
+   static 2-level case that runs several steps and conserves (~1e-13). **Then** add recursive
+   subcycling (level l+1 takes `ref_ratio` substeps with time-interpolated ghosts) on top,
+   generalizing `s_advance_amr_fine_substeps` (already Berger-Colella for L0↔L1).
+4. **Per-level regrid + proper nesting.** Tag each level for the next-finer one; build level
+   `l+1` boxes clustered + tiled + nested inside level `l`; distribute (reusing the SFC map).
+5. **Restart / distribution / GPU per level.** Extend the fine-restart record with a per-block
+   level; validate repartition-on-restart and the device present-table across levels.
+
+## Design decisions on record
+
+- Flat pool + per-block `level`/`parent` tag (not a per-level pool array).
+- Recursive subcycling with time-interpolated C/F BCs (not lock-step).
+- `ref_ratio` stays 2 for now (ratio-4 is separate, banked work); nesting/coupling are written
+  ratio-generic so ratio-4 drops in later.
+- Load balance: the existing SFC map distributes blocks across **all** levels from the flat
+  pool; the `amr_max_blocks < num_procs` warning applies per the total block count.

docs/documentation/case.md

@@ -684,6 +684,7 @@ To restart the simulation from $k$-th time step, see @ref running "Restarting Ca
 | `amr_buf`               | Integer | Coarse-cell padding around tagged cells when regridding (default 3) |
 | `amr_subcycle`          | Logical | Advance the coarse level at the case dt and the fine level at dt/2 (two substeps; Berger-Colella refluxing). Requires `amr`; incompatible with `cfl_dt`. |
 | `amr_max_blocks`       | Integer | Number of fixed refined-block slots preallocated (each max-block sized; ~N x device memory); must be >= 1 (default 4) |
+| `amr_max_level`        | Integer | Maximum AMR refinement depth (number of refined levels above L0); must be >= 1 (default 1). Multi-level nesting (>= 2) is supported: static AMR (`amr_regrid_int = 0`) nests up to level 2, dynamic regrid (`amr_regrid_int > 0`) nests deeper (see `docs/documentation/amr_multilevel.md`) |
 | `amr_cluster_eff`       | Real    | Berger-Rigoutsos min tag efficiency a clustered block box reaches before splitting stops; must satisfy 0 < eff <= 1 (default 0.7) |
 | `hybrid_weno`           | Logical | Use linear-optimal reconstruction in smooth cells, full WENO only at flagged discontinuities (requires WENO reconstruction) |
 | `hybrid_weno_eps`       | Real    | Smoothness threshold for hybrid WENO shock flagging; must be > 0 (default 1e-2) |
@@ -954,6 +955,7 @@ visualization output is future work.
 | `amr_buf`               | Integer | Coarse-cell padding around tagged cells; must be >= 1 when `amr_regrid_int > 0` (default 3) |
 | `amr_subcycle`          | Logical | Advance fine level at dt/2 (two substeps per coarse step) with Berger–Colella refluxing |
 | `amr_max_blocks`       | Integer | Number of fixed refined-block slots preallocated (each max-block sized; ~N x device memory); must be >= 1 (default 4) |
+| `amr_max_level`        | Integer | Maximum AMR refinement depth (number of refined levels above L0); must be >= 1 (default 1). Multi-level nesting (>= 2) is supported: static AMR (`amr_regrid_int = 0`) nests up to level 2, dynamic regrid (`amr_regrid_int > 0`) nests deeper (see `docs/documentation/amr_multilevel.md`) |
 | `amr_cluster_eff`       | Real    | Berger-Rigoutsos min tag efficiency a clustered block box reaches before splitting stops; must satisfy 0 < eff <= 1 (default 0.7) |
 
 ### 8. Acoustic Source {#sec-acoustic-source}

docs/superpowers/plans/2026-07-13-amr-banked-increments.md

@@ -0,0 +1,129 @@
+# Banked AMR increments — design + plan (for fresh-session execution)
+
+Three independent, conservation-critical AMR follow-ups, deferred from the multi-level-IB
+session (2026-07-13) to be executed **fresh, one at a time** (silent-wrong-answer risk +
+context fatigue). Each is its own brainstorm-refine → SDD cycle. **Execution order: #29 →
+#30a → #30b** (self-contained first; ref_ratio=4 broadest last). Branch `amr-multilevel`
+(PR #6). Each currently **fail-closed** in `m_checker.fpp`.
+
+Ground rules (all three): golden-file validation on CPU **and** GPU (V100, targeted
+`--only <UUIDs> -- -b mpirun`, NOT the full-suite sbatch which flakes on SIGILL); existing
+goldens byte-identical where the feature is inactive; `./mfc.sh format` → build → precheck-on-commit.
+
+---
+
+## #29 — Distributed pb/mv coupling for non-polytropic QBMM at np≥2
+
+**Problem.** Non-polytropic QBMM carries a per-cell quadrature side-state `pb` (bubble
+pressure) and `mv` (vapor mass), `nnode × nb` per cell. It **evolves cell-locally** (no
+face flux), but its AMR coarse↔fine coupling is done LOCALLY, exact only at np=1. At np≥2
+the SFC block owner needn't hold the block's coarse cells, so pb/mv couple to the wrong
+rank's coarse side-state = silent wrong answer. Gated: `m_checker.fpp:106`
+`@:PROHIBIT(qbmm .and. .not. polytropic .and. num_procs > 1, ...)`.
+
+**Current local sites (`m_amr.fpp`):**
+- Prolong (coarse→fine): `s_amr_prolong_pbmv` (~1884), called ~1347. Reads coarse pb/mv locally.
+- Restrict (fine→coarse fold-back): `s_restrict_pbmv` (~1884 def; called 1509/1590). Local; comment 1589 "np>=2 QBMM fold-back is not yet distributed."
+
+**Approach: mirror the `q_cons` P2P distribution for pb/mv (no reflux — pb/mv aren't fluxed).**
+The pattern to copy: `s_amr_gather_coarse_patch` (the coarse-patch P2P gather:
+`f_amr_rank_coarse_range` + `MPI_IRECV`/`MPI_ISEND`, owner assembles `amr_cg`) and the
+restrict scatter in `s_restrict_fine_to_coarse` (owner restricts, scatters covered coarse
+slices to coarse-cell owners). Two differences from q_cons: (a) per-cell payload is
+`nnode*nb` (both pb and mv), so buffer sizes scale by that; (b) NO Berger-Colella reflux
+(pb/mv have no C/F flux correction — prolong sets the fine ghost/interior, restrict folds
+back; that's the whole coupling).
+
+**Task plan (SDD):**
+1. **Distributed pb/mv gather** — assemble the block's coarse pb/mv patch on the owner via
+   P2P (mirror `s_amr_gather_coarse_patch`; a parallel `amr_cg`-like pb/mv buffer, or extend
+   the gather to carry pb/mv alongside q_cons). Prolong reads that instead of local coarse.
+   Gate: np=1 byte-identical (single owner → local copy path unchanged); build.
+2. **Distributed pb/mv restrict scatter** — owner restricts fine pb/mv → coarse averages,
+   scatters covered coarse slices to coarse-cell owners (mirror the q_cons restrict P2P).
+   Gate: np=1 byte-identical.
+3. **Lift the gate + golden** — drop the `num_procs > 1` term from `m_checker.fpp:106`; add a
+   np=2 non-polytropic-QBMM + AMR golden. Validate: per-node pb/mv moments conserved
+   (machine-zero for the conservative moments; pb/mv are a side-state so define the gate as
+   "np=2 trajectory == np=1 trajectory" on a bit-uniform grid, mirroring the q_cons np-cross
+   check). CPU + GPU.
+
+**Open questions for the fresh brainstorm:** whether to fold pb/mv into the existing q_cons
+gather/scatter buffers (one MPI round) or a separate exchange (simpler, more messages);
+device-buffer handling for the larger payload (pb/mv are `pres_field`, GPU_DECLARE'd).
+**Validation oracle:** np=2 == np=1 trajectory on a bit-uniform grid (the WENO-table-ulp
+finding means non-bit-uniform grids diverge at ulp).
+
+---
+
+## #30a — Lazy owned-only IB marker sizing
+
+**Problem.** The multi-level-IB increment sized the declare-target `ib_markers` (and the park
+slots) to `2**amr_max_level * base_block_extent` — the **global** deepest extent, right at
+np=1. At np≥2 (once multi-level IB is un-gated there — a separate future item) that
+over-allocates the never-realloc'd device field on every rank to the global deepest, even
+ranks owning only shallow/no fine blocks. Task #30's "lazy owned-only sizing" = size the
+marker field to the deepest fine block **a given rank actually owns**, allocated lazily.
+
+**Current state:** `s_ibm_marker_bounds` (m_ibm.fpp, added 2026-07-13) computes the deepest
+bound from `amr_max_level` + `amr_block_beg/end` (global). Multi-level IB is currently np=1
+only, so this is not yet a live cost — **#30a is a memory optimization that pairs with
+un-gating multi-level IB at np>1** (itself deferred). Low priority until np>1 IB lands.
+
+**Approach.** Replace the global `2**amr_max_level` bound with the per-rank owned deepest
+level. Because `ib_markers` is a device declare-target that must NOT be reallocated after
+mapping, "lazy" means: at init, size to the deepest level the rank's INITIAL decomposition
+could own; if a later regrid would need deeper, that's the same never-realloc constraint —
+so either (a) size to the rank's static owned-region deepest possible, or (b) accept the
+global bound at np=1 (status quo) and only optimize once np>1 IB + repartition-on-restart
+lands. **Recommend deferring #30a until np>1 multi-level IB exists** — optimizing an
+allocation for a configuration the checker doesn't yet admit is speculative.
+
+**Task plan:** small — parameterize `s_ibm_marker_bounds` by a per-rank owned-level input;
+validate memory footprint drops at np>1 with no golden change. Blocked on np>1 IB.
+
+---
+
+## #30b — ref_ratio = 4
+
+**Problem.** Refinement ratio is hard-coded 2:1 per level. One 4:1 level reaches the
+resolution of two 2:1 levels with one fewer coupling layer. The `2*`/`2**level` extent
+factors are threaded through every coupling kernel.
+
+**Approach.** Introduce a runtime `ref_ratio ∈ {2,4}` (param, default 2 = byte-identical).
+De-hardcode the fine-extent, prolong, restrict, halo, and reflux stencils from `2` to
+`ref_ratio` (and `2**level` to `ref_ratio**level`). The prolong stencil widens (4:1 injection
++ the multi-fluid/species closure over a 4-wide child block); restrict averages `ref_ratio^d`
+children; the fine-fine seam halo and reflux child-face counts scale by `ref_ratio`.
+
+**Highest-risk feature of the three** — it touches every coupling kernel's stencil, and a
+wrong factor conserves-to-machine-zero while being physically wrong.
+
+**Task plan (SDD):**
+1. **Param + gate** — add `ref_ratio` (definitions.py + descriptions.py + checker default 2);
+   gate `ref_ratio ∉ {2,4}` and any unsupported combos fail-closed. Byte-identical at
+   ref_ratio=2. `amr_slots(:)%ref_ratio` already exists (per-block) — audit that it's
+   populated from the param, not a `2` literal.
+2. **De-hardcode extents** — sweep `2*(...)`/`2**level` → `ref_ratio*(...)`/`ref_ratio**level`
+   in fine-geometry sizing, marker sizing, `old_ext`, tower weight, `s_amr_fine_fine_halo`
+   (the level-aware `2**level` from #35), reflux child-face counts. Gate: ref_ratio=2
+   byte-identical (every site reduces to `2`).
+3. **Prolong/restrict stencils** — generalize `s_prolong_one_var` (injection/interp for a
+   `ref_ratio`-wide child), `s_restrict_one_var` (`ref_ratio^d` child sum), and the alpha/
+   species closures. Gate: ref_ratio=2 byte-identical.
+4. **ref_ratio=4 golden** — a single-level ref_ratio=4 case; validate conservation
+   machine-zero (inviscid) + a resolution check vs two 2:1 levels. CPU + GPU.
+
+**Open questions:** buff_size / stencil reach for a 4-wide prolong (does the coarse patch
+margin `amr_cpat_mar` suffice?); interaction with multi-level (ref_ratio=4 AND amr_max_level>1
+= 16:1 two levels — likely gate to one at a time first); whether `ref_ratio` is global or
+per-level.
+
+---
+
+## Cross-cutting notes
+- All three keep existing goldens byte-identical when inactive (param defaults / gated).
+- GPU validate on V100 via targeted `-- -b mpirun` (the full-suite sbatch SIGILL-flakes on
+  arch-mismatched nodes; see the `phoenix-srun-mpirun` memory + the hpcx-bin-on-PATH note).
+- Conservation gates: q_cons machine-zero; pb/mv and any non-conservative side-state use the
+  "np=2 == np=1 trajectory on a bit-uniform grid" oracle.