mocs_to_orders: ragged batch moc_to_order — plus an API sweep for bulk-by-default operators

[espg]

🤖 from Claude (filed at espg's direction, 2026-08-07)

Part 1 — mocs_to_orders: ragged batch moc_to_order

zagg's batch shardmap rewire (englacial/zagg#400, consuming #153's polygons_to_morton_mocs) eliminated the per-ring coverage calls, and profiling the survivor shows moc_to_order is now the residual per-ring Python call — roughly a third of the remaining wall. Same shape as #153, one level up the pipeline:

values, out_offsets = mortie.mocs_to_orders(
    moc_values, moc_offsets,   # ragged MOCs, arrow list layout (e.g. polygons_to_morton_mocs output, verbatim)
    order,                      # shared scalar target order (mirrors #153's shared-setting ruling)
    max_cells=...,              # per-MOC budget, same semantics as the scalar
)
# values: flattened order-cells, all MOCs concatenated; out_offsets: arrow list layout

• Input composes with Batch polygon coverage: polygons_to_morton_moc over ragged (offsets) arrays — one call, rayon across polygons #153's output verbatim — the strict-contract ragged pair goes straight back in, so zagg's ring pipeline becomes two Rust calls total.
• Naming per the Batch polygon coverage: polygons_to_morton_moc over ragged (offsets) arrays — one call, rayon across polygons #153 plural ruling: plural marks many→many (one cell array per input MOC); the target order stays a shared scalar like Batch polygon coverage: polygons_to_morton_moc over ragged (offsets) arrays — one call, rayon across polygons #153's tolerance/budget.
• Per-item budget refusal, fail-fast naming the offending MOC index — same error posture as Batch polygon coverage: polygons_to_morton_moc over ragged (offsets) arrays — one call, rayon across polygons #153. This matters for the consumer: zagg's per-ring try/except around the scalar call preserves a cell-budget refusal semantic (batch mortie coverage in ShardMap.build (issue #396) englacial/zagg#400 question (2) explicitly declined a zagg-side vectorization because it would drop that refusal); the batch keeping it per-item is what lets zagg adopt without a behavior change.
• rayon across MOCs, GIL released, chunked assembly with the Batch polygon coverage: polygons_to_morton_moc over ragged (offsets) arrays — one call, rayon across polygons #153 memory posture (peak ≈ result + one item, not 2×).
• Acceptance: per-MOC byte parity with scalar moc_to_order (randomized + the Batch polygon coverage: polygons_to_morton_moc over ragged (offsets) arrays — one call, rayon across polygons #153 fixture rings run through both stages), the budget-refusal index test, bench vs the scalar loop at ≥100k MOCs.

Part 2 — sweep mortie for scalar-only operations; bulk operators by default

This is the second consumer-driven batch request in two days (#153, now this). Rather than filing these one at a time, espg's direction: audit the public API for scalar-only / per-object-loop operations and add ragged/array batch variants as the default posture — new operations should take array/ragged inputs natively, with the scalar form as the convenience wrapper, not the other way around.

Known candidates to seed the audit (from zagg's call sites):

• mort2polygon — zagg loops it per shard key in Python (shards_bbox, healpix.py; also its _leak_check) — a natural ragged batch (words in → ragged vertex arrays out).
• moc_to_order — part 1 above.
• The MOC set ops (moc_and/moc_or/…) are pairwise array-based already, but an N-way variadic fold (many MOCs → one union/intersection in Rust) would serve the ShardMap.build at bulk-catalog scale: process-pool parallel build; persistent footprint index englacial/zagg#396 phase-2 column fast path (AOI ∩ per-granule cells over a whole catalog).
• Scalar helpers backing the Scalar MortonWord type: packed word that reprs as the decimal label #152 MortonWord properties are fine as-is (display-bound) — the audit should distinguish hot-path candidates from convenience scalars rather than batching everything.

Deliverable for part 2: an audit table on this issue (operation → scalar-only? → consumer/hot-path evidence → batch variant proposed y/n), then individual implementation as follow-ups sized like #153/#154.

Refs: #153 / PR #154 (the pattern: strict ragged contract, plural naming, shared scalar params, fail-fast index naming, chunked assembly), englacial/zagg#396 + englacial/zagg#400 (the consumer), #152 (the convenience-scalar counterexample).

[espg]

🤖 from Claude — part 2 deliverable: the API audit, plus an implementation plan for both parts (espg's direction, 2026-08-07)

Trees read: mortie main (PR #154 merged, unreleased — latest tag is 0.9.3), zagg main + the claude/396-batch-shardmap worktree (the post-#154 state), moczarr main. Every row below is backed by a call site; ops with no consumer are called out as such rather than proposed.

Correction to this issue's own text

The part-1 text names an N-way variadic fold over moc_and/moc_or/moc_minus/moc_xor as a candidate. The sweep does not support it: moc_or, moc_minus and moc_xor have zero call sites in zagg or moczarr combined. The evidenced shape is a 1×N broadcast of moc_and — one AOI/left MOC against N right MOCs — at zagg/src/zagg/grids/aoi.py:250 (per shard), moczarr/src/moczarr/open.py:148 and store.py:385 (per leaf), zagg/src/zagg/coverage.py:136 (per leaf), and it is exactly the englacial/zagg#396 phase-2 join (one AOI MOC ∩ 555,867 per-granule cell sets). Proposing mocs_and instead; the generic fold is parked until a use case exists.

Audit — proposed batch variants

proposed	replaces	consumer evidence	size
mocs_to_orders(values, offsets, order, max_cells=) → ragged	per-MOC moc_to_order	zagg .../shardmap.py:448, inside the per-ring loop that survived #154 — 555,867 granules × rings; also grids/aoi.py:268 per shard per worker	small
mocs_and(a, values, offsets) + predicate mocs_intersect(...) → bool[N]	1×N moc_and	the four sites above; the zagg#396 phase-2 join primitive. Structural win beyond FFI: build_bmoc(a) (moc.rs:56) is hoisted once instead of rebuilt per pair, and the predicate form allocates nothing	medium
common_ancestors(values, offsets) → dense uint64[N]	per-group common_ancestor	zagg/src/zagg/stats/tdigest.py:198, for j in np.flatnonzero(~single) — per multi-member centroid (δ=512), per cell, per build and per fold, on every Lambda worker; 65,536 cells/shard at the shipped ATL03 config. tdigest.py:177-179 already names this as "the same O(n) Python-loop shape issue #279 removed"	small
children_of(words, order) → dense uint64[N, 4**d]	scalar-parent generate_morton_children (tools.py:1167 coerces to a scalar)	moczarr wrote the gap down in its own source: dggs.py:310 np.stack([generate_morton_children(int(w), level) for w in words]), comment at dggs.py:302-305 — "there is still no vectorized many-parent children kernel". zagg: grids/healpix.py:199 per sub-chunk on every worker, catalog/shardmap.py:708, column.py:365	small
words_to_decimals(words) and hive_paths(words, root=, suffix=), numpy-only	the pandas-gated MortonIndexArray.decimal_repr() / .hive_path()	the Rust kernel exists (rust_mi_decimal_repr) but the only public route requires pandas (mortie/pandas.py:40), so zagg/src/zagg/hive.py:247 builds a 1-element MortonIndexArray per word — and shard_leaf_path is called from 11 sites including per-shard sweep loops, dragging pandas onto the Lambda worker. moczarr says the same at convention.py:160-162. Asymmetric twin of the already-shipped pandas-free decimals_to_words	small
moc_contains(cells, moc) → bool[N]	— (new)	moczarr reimplements it in Python: coverage.py:214-221, a nested loop over members × distinct orders using clip2order equality, run on every read's whole morton coordinate array (open.py:452, pyramid.py:474). zagg has the same shape at grids/aoi.py:268. A range-linearized traversal needs no densification, unlike today's moc_to_order + isin	medium
mort2polygons(words, step=) → dense (N, 4·step+1, 2)	per-cell mort2polygon	9/9 zagg sites pass a scalar: grids/healpix.py:257 (shards_bbox loop), healpix.py:439 (shard_footprint, called per shard from three sites), data/build_aoi_shardmap.py:126, data/conus/*.py. moczarr already uses the array form correctly (dggs.py:250). Note the wall is not the kernel — rust_boundaries/rust_vec2ang are already batched and rayon'd; it is the per-cell Python loop at tools.py:1088-1100 plus _normalize_antimeridian_polygon (tools.py:939), which is what would move to Rust	medium
morton_coverages (flat twin of #154)	per-polygon morton_coverage	zagg .../shardmap.py:347/365, non-HEALPix branch only	small

Counter-classification — deliberately NOT batched

The audit's value depends on this half being honest, so:

• Display/convenience-bound: Scalar MortonWord type: packed word that reprs as the decimal label #152's MortonWord property helpers (that issue's own scope note says pure-Python convenience, no Rust); res2display (a 30-row ladder).
• Bounded loops, not per-object: split_base_cells (12 base cells); the group-by-order dispatch inside mort2geo/mort2bbox/mort2polygon/uniq2geo/geo2uniq (≤30 orders).
• Structurally batchable but ZERO consumers — batching these would be speculative: mort2bbox, ring_validity, ring_is_simple, linestring_coverage, the whole prefix_trie public surface, to_geometry/to_wkb/to_wkt, and moc_or/moc_minus/moc_xor (above). Leave them until a consumer appears.
• Already array-native and fine: geo2mort (rayon + GIL released), clip2order, mort2geo/mort2norm/norm2mort/orders_of/validate_morton/morton_buffer, decimals_to_words, the arrow import/export pair.

Not mortie gaps — consumer-side fixes (recorded so the sweep does not inflate)

Four sites look like mortie gaps and are not; they call a scalar form where the array form already exists. Filing these against the consumers, not here: rank_to_xy/xy_to_rank called per cell at zagg .../tdigest_tensor.py:867/973/1053 (array form exists and is already used correctly at :596); mort2healpix at zagg .../grids/healpix.py:197 (1-element array inside a loop over an array it already holds); the two _leak_check sites (data/build_aoi_shardmap.py:126, data/conus/build_conus_shardmap.py:115) want cell centres, so vectorized mort2geo already serves them; moczarr convention.py:166 still imports the deprecated private _decimal_to_word per label instead of the shipped decimals_to_words.

One blocking question, and it is cheap

#108 Group B still has the moc_to_order budget-guard fork undecided (from #97): (i) ValueError vs MemoryError, and (ii) whether the hard raise stays pinned to the 1<<20 warn line or gets a separate higher ceiling. mocs_to_orders inherits that semantic per item, so building it first would freeze an undecided contract into the batch API — and the batch's per-item refusal is precisely what lets englacial/zagg#400 adopt it without the behavior change it explicitly declined (question (2) there). Settle #108's fork, then phase 1. It is the only hard ordering constraint in the sweep.

Module placement — needs a ruling before phase 1

mortie/coverage.py is at 977 lines and mortie/tools.py at 1354, against this repo's ~1,000-line aim (mortie's CLAUDE.md §4 has no 1,200 raise). Adding any of the above to either file trips it. Proposal: a new mortie/batch.py front plus src_rust/src/moc/batch.rs, alongside the existing src_rust/src/coverage/batch.rs — one obvious home for the bulk surface rather than growing two already-full modules.

Plan

Phase 0 (blocking, no code) — settle #108's budget-guard fork; record the ruling here.

Phase 1 — mocs_to_orders (part 1 of this issue). Mirrors coverage/batch.rs around moc::to_order; per-item budget refusal carrying #108's ruling; strict ragged contract, lowest-index fail-fast, rayon under allow_threads, chunked assembly (CHUNK = 2048) so peak ≈ result + one chunk — the #154 posture, which took that PR's 556k-polygon peak from 2.50× to 1.34× of result size. Acceptance: per-MOC byte parity with scalar moc_to_order over randomized inputs and the #154 fixture rings run through both stages; budget-refusal index test; bench vs the scalar loop at ≥100k MOCs.

Phase 2 — mocs_and + mocs_intersect. Hoist build_bmoc out of the per-pair path. The predicate-vs-materialize split is an API fork worth deciding in the PR: the predicate is what both evidenced consumers actually want.

Phase 3 — common_ancestors + children_of. Both dense-output and small; the two highest-value fleet-worker wins, independent of the shardmap path.

Phase 4 — words_to_decimals + hive_paths. Wrapper + export only; takes pandas off the Lambda worker's import path.

Phase 5 — moc_contains, then mort2polygons. The latter folds #151 item 4 (the unused on_antimeridian in the very function that would move to Rust). morton_coverages rides whenever the non-HEALPix branch justifies it.

Each phase is its own PR sized like #153/#154, with the adversarial-review pass between them.

Interaction with #48

#48 is the 1.x API-freeze gate, and "bulk by default, scalar as the convenience wrapper" is a posture decision, not a per-function one — it should be ratified before the freeze, not discovered after. Recommend recording the posture on #48 once phase 1 lands and the shape is proven.

Note on release ordering

#154 is merged but unreleased (latest tag 0.9.3) while englacial/zagg#400 already pins mortie>=0.9.4 — a 0.9.4 tag is a prerequisite for that PR landing, independent of everything in this sweep.

Answering the pipeline question this issue was filed from

After mocs_to_orders, moc_to_order at zagg .../shardmap.py:448 is the last mortie call in that loop, and the loop collapses to vectorized numpy — to_order already returns sorted-unique (moc.rs:152-154), so even the np.unique beside it is redundant. The residual is then not a mortie call: it is _flatten_rings (shardmap.py:139-150), a Python loop over all 555,867 CMR records marshalling dicts into arrays — which is exactly what englacial/zagg#396's own phases exist to remove. So after phase 1, the sweep's priority moves off the shardmap path entirely and onto the phase-2/3 consumers above.

[espg]

The part-1 text names an N-way variadic fold over moc_and/moc_or/moc_minus/moc_xor as a candidate. The sweep does not support it: moc_or, moc_minus and moc_xor have zero call sites in zagg or moczarr combined.

these libraries (moczarr, zagg) are evolving, and might call these in the future. I don't see any reason to not include batch versions of the moc_{and,or,minus,xor} within this refactor.

Structurally batchable but ZERO consumers — batching these would be speculative: mort2bbox, ring_validity, ring_is_simple, linestring_coverage, the whole prefix_trie public surface, to_geometry/to_wkb/to_wkt, and moc_or/moc_minus/moc_xor (above). Leave them until a consumer appears.

maybe not all of these... but moc_{and,or,minus,xor} , mort2bbox , and to_geometry/to_wkb/to_wkt should be included as well.

mortie/coverage.py is at 977 lines and mortie/tools.py at 1354, against this repo's ~1,000-line aim (mortie's CLAUDE.md §4 has no 1,200 raise). Adding any of the above to either file trips it. Proposal: a new mortie/batch.py front plus src_rust/src/moc/batch.rs, alongside the existing src_rust/src/coverage/batch.rs — one obvious home for the bulk surface rather than growing two already-full modules.

approved -- mortie/batch makes sense

#154 is merged but unreleased (latest tag 0.9.3) while englacial/zagg#400 already pins mortie>=0.9.4 — a 0.9.4 tag is a prerequisite for that PR landing, independent of everything in this sweep.

it is released-- we're on 0.9.4 now

Sounds like we need to also open issues in zagg and moczarr for where they should be using batch implementations (either already, or newly)

Make sure that the batch version of these functions have plural grammer, so they're easy to understand and identify

[espg]

🤖 from Claude — plan revision, folding in espg's rulings (2026-08-07)

Five rulings landed since the audit; the plan changes as follows.

1. Module placement — domain split, superseding the batch.py approval

espg initially approved mortie/batch.py, then approved the domain split instead — that is the plan of record now. Reason: splitting by arity puts moc_to_order in coverage.py and mocs_to_orders in batch.py, separating twins that should be read together and creating a namespace organised by a property nobody searches by.

Extract the MOC algebra into mortie/moc.py: compress_moc, moc_to_order, moc_and/moc_or/moc_minus/moc_xor, moc_not, moc_min, common_ancestor — roughly coverage.py:597–977 — each landing beside its own plural twin. That leaves coverage.py at ~600 lines with room for the coverage-side batch ops and gives moc.py ~380 with room for its own, both under the ~1,000 aim, with every scalar next to its plural.

Rust side follows the same axis: src_rust/src/moc/batch.rs alongside the existing src_rust/src/coverage/batch.rs.

Related, deferred: espg notes tools.py (1,354 lines, already over the aim, and a name that describes nothing) should be renamed/split at some point. Recorded on #48 — it is an API-surface change and therefore must land before the 1.x freeze, not after.

2. Roster expansion — batch the full set-op family, not just the evidenced one

The audit recommended only mocs_and, on the grounds that moc_or/moc_minus/moc_xor have zero call sites in zagg or moczarr. espg overrules: those consumers are evolving and may call them, and there is no reason to ship a partial family. All four set ops get plural twins, plus mort2bbox and to_geometry/to_wkb/to_wkt.

The counter-classification stands for the rest (ring_validity, ring_is_simple, linestring_coverage, the prefix_trie surface) — those remain out until a consumer appears.

3. Plural grammar throughout

Every batch entry point takes the plural form, per the #153/#154 ruling: the name marks many→many and distinguishes it from any many→one op. mocs_to_orders, mocs_and/mocs_or/mocs_minus/mocs_xor, common_ancestors, children_of, mort2bboxes, mort2polygons, to_geometries/to_wkbs/to_wkts, words_to_decimals, hive_paths.

4. #108's blocking fork is RULED — phase 1 is unblocked

espg ruled both halves: keep ValueError (caller-controlled argument condition, not a failed allocation), and give the hard raise a separate, higher ceiling than the 1<<20 warn line (pinning them makes the warn branch unreachable). One number is still open — recommended 1<<24 for the raise. mocs_to_orders inherits the semantic per item.

5. Correction: 0.9.4 IS released

My release-ordering note was wrong — 0.9.4 is out, so englacial/zagg#400's mortie>=0.9.4 floor resolves today and nothing in this sweep gates that PR.

New: a question the WKB batching raises

espg asked whether the WKB converters should be batched (now in scope per (2) above). One thing to settle in that phase's design: mortie does not parse WKB itself — geometry_from_wkb delegates to a backend, and _require_backend() (geometry.py:39-78) tries shapely first, accepting spherely only if shapely is absent. So a batched from_wkbs would import shapely on any machine that has it, which cuts against zagg's stated direction of moving off shapely onto the spherely + mortie pair.

Worth deciding before that phase: (a) leave backend preference as-is and let consumers control it by what they install; (b) make the batch WKB entry points backend-explicit (backend="spherely"|"shapely"); or (c) flip the preference to spherely-when-present. Note spherely.from_wkb is already array-capable (verified: object array in, ndarray of Geographies out), so (b)/(c) are live options rather than aspirational. This matters to englacial/zagg#400, whose phase-2 redesign wants to feed a WKB column straight through to a backend instead of extracting ring coordinates via shapely.

Revised phase order

Phase 1 mocs_to_orders (now unblocked) → phase 2 the set-op family + mocs_intersect predicate → phase 3 common_ancestors + children_of → phase 4 words_to_decimals/hive_paths → phase 5 moc_contains, mort2polygons, mort2bboxes, the WKB/geometry converters (carrying the backend decision above). The mortie/moc.py extraction lands with phase 1, since that is the first thing that needs the home.

[espg]

🤖 from Claude

Follow-up on the WKB/backend question raised in the plan revision — espg's read is "we should be falling back to shapely, not preferring it". Investigating that turned up something more consequential than a preference: a one-line flip would break the path, because spherely is a decode-only backend.

What is actually there

_require_backend() (geometry.py:39-78) prefers shapely, and its docstring justifies that by the WKB/WKT codec being "mature and all we lean on". But the binding constraint is in _require_shapely() (geometry.py:80-100):

ring decomposition and SRID-tagged emit lean on shapely's geometry-introspection API (get_exterior_ring / get_parts / set_srid), which spherely's published surface does not yet expose

The full path is from_wkb → geometry_from_wkb (either backend) → from_geometry → decompose → _require_shapely. So with spherely as the active backend, WKB decodes fine and then raises NotImplementedError one call later. Flipping the preference alone converts a working shapely path into a broken spherely one. decompose's own docstring already records the open question: "Whether to invest in a spherely introspection shim is an open question for the issue thread."

Three ways out — and a recommendation

1. Spherely gains ring introspection (get_exterior_ring/get_parts equivalents). Upstream or fork work, on someone else's release cadence, and it leaves mortie depending on a Python geometry library either way.
2. A spherely introspection shim inside mortie — mortie's own suggestion. Same dependency shape, plus a shim to maintain across two libraries' versions.
3. Parse WKB in Rust, inside mortie. WKB is a small, stable, well-specified binary format, and mortie already has the Rust core to do it. This removes the backend from the ingest path entirely — no shapely, no spherely, no Python geometry objects — and turns the batched converter into a pure-Rust from_wkbs: bytes in, ragged MOCs out, rayon across items, rather than N Python decodes feeding a Rust kernel.

Recommendation: (3). Beyond being the fastest, it is the only option that makes mortie's stated contract true — _require_backend's own error message says "mortie's runtime is numpy-only, so the backend is an optional extra", which today holds only for callers who never touch WKB/WKT. It also decouples mortie from both geometry libraries' release cadences at exactly the moment the ecosystem is in flux.

Scope note for whoever takes it: (3) covers the ingest direction (WKB → rings → coverage). The emit direction (to_geometry/to_wkb/to_wkt, also in this issue's expanded roster) still needs a geometry object to hand back, so a backend stays relevant there — but emit is the direction where the caller already chose a library to receive the result, so preferring whichever is installed is defensible there in a way it is not for ingest.

Consumer context

This gates englacial/zagg#400's phase-2 redesign. zagg wants to feed its STAC-geoparquet WKB column straight through to a coverage call instead of extracting exterior-ring coordinates via shapely — and its stated direction is spherely + mortie, with shapely on the way out. Under (3) that becomes: WKB column in, ragged MOCs out, no geometry library in the loop, and zagg's _flatten_rings marshalling loop (the residual identified in that PR's review after mocs_to_orders) disappears with it.

One measured datum from that investigation, relevant to the ingest contract: across all 555,867 granules in the full ATL03 v007 catalog, every footprint is a single Polygon with no interior rings — 0 MultiPolygons, 0 holes. So a Rust ingest path can treat multipart/holes as correctness-completeness (mortie's even-odd descent already handles both) rather than as a hot-path concern, and zagg's current largest-part-only selection is provably dead code on real data.