Unindexed builds: cover from WKB and intersect before materializing records (#439 for the live path)

[espg]

🤖 from Claude

Surfaced by espg's post-#440 demo rerun (2026-08-16): the unindexed polygon-direct build is still record-decode-first — issue #439's inversion was applied only to the stored-index path. Measured on the 555,867-granule ATL03 clone, California AOI, o9:

• indexed query (post-Stored-index query path: intersect before materializing granule records #440): 2.7 s
• one-time index_footprints(9): 32.5 s
• no-index build: 86.9 s (build_wall_s 60.9 s + ~26 s granule_records() outside it)

Structurally the no-index build computes the same thing as index-then-query, yet costs ~2.5× their sum. Two inherited inefficiencies:

1. _intersect_mortie consumes decoded ring coordinates from granule records, so granule_records() runs over every row before any intersection — the exact pattern Stored-index query path: intersect before materializing granule records #439 removed from the indexed path.
2. The live cover runs from the decoded rings rather than the raw WKB column; index_footprints's from_wkbs covers the same 555k footprints in ~32 s vs the ~60 s in-build cover+intersect.

Proposed fix (the #439 ruling extended): the unindexed mortie-backend build becomes build-an-ephemeral-index-and-query-it — cover from the WKB column (from_wkbs, no records), intersect positionally (the same _intersect_footprint_cells machinery, values/offsets in memory), then materialize records for hit rows only via the existing _hit_records. Expected: no-index ≈ the cover cost (~33-35 s at clone scale), and the code path largely unifies with the stored-index plan (one intersection engine; "indexed" = the cover was persisted). Care points carry over from #440: screen-predicate parity (granule_row_mask), total_granules stability, the MultiPolygon superset disclosure (from_wkbs covers all parts — the live path would inherit the index's documented superset semantics, a small behavioral delta vs today's largest-ring cover that must be disclosed like #441's schema note), and the paired-asset exclusion (PR #432's resolution: paired builds stay eager).

Refs #439 (the indexed-path inversion, PR #440), #396, #400, PR #432 (paired-build exclusivity).

[espg]

🤖 from Claude

Plan (espg directed plan+implement, in-session 2026-08-16). One design point the issue body glossed needs stating first, because it shapes the whole implementation.

The order problem, and the two-stage answer

The stored-index path intersects at the column's order (o9, the shard order). The live path intersects at the resolved mortie order — _resolve_mortie_order, default the chunk order (o13 for the ATL03 grids, the #92 rule). A naive "cover everything from WKB at the resolved order" is infeasible at clone scale: MOC words per granule grow ~2× per order, and index_footprints' own docstring prices the full clone at ~270–420 MB at o9 but ~9–13 GB at o13. Full-catalog o13 covers cannot be materialized.

So the inversion is two-stage:

1. Coarse candidate prefilter — one from_wkbs cover of the whole geometry column at parent_order (the o9-scale cover that costs ~32 s at clone scale), intersected positionally against the AOI's shard MOCs. Sound as a prefilter by ancestor containment: a footprint whose fine-order cover intersects a shard necessarily has its coarse cover intersecting that shard's cell — the coarse test is a strict superset of the fine one, so no true hit is ever dropped.
2. Fine cover on candidates only — from_wkbs again at the resolved order, over just the surviving rows (thousands, not 555k), then the exact intersection at that order — the same result today's live path computes.
3. Records for hits only — the existing _hit_records (issue Stored-index query path: intersect before materializing granule records #439/PR Stored-index query path: intersect before materializing granule records #440 machinery), unchanged.

Expected: the unindexed clone-scale California build drops from ~87 s to ≈ the coarse cover cost (~33–35 s), and the intersection engine unifies — "indexed" simply means stage 1's cover was persisted.

Semantics deltas (deliberate, disclosed)

• MultiPolygon superset: from_wkbs covers the union of parts; today's live path covers only the largest part's exterior ring. The new path inherits the stored index's documented superset semantics — multi-part footprints may assign to more shards. Every CMR ATL03/06 granule is single-part, so no production delta; pinned by test and documented in build's Notes.
• Null geometry refuses loudly (inherits granule_row_mask's post-Stored-index query path: intersect before materializing granule records #440 posture) where the old records path also raised — parity preserved.
• Explicit mortie_order= stays honored on the unindexed path (the ephemeral cover runs at whatever order resolves) — unlike the stored plan, which refuses the pin because a persisted column cannot restate its order.

Path exclusions (unchanged behavior)

Paired builds (sibling_catalog — stay eager per the PR #432 resolution), footprint="beams" (per-beam corridors are records-derived), the spherely backend, and rectilinear grids all keep today's path. _intersect_mortie remains for beams.

Structure

The ephemeral-cover helper lives beside its siblings in sources.py (it is index_footprints minus persistence, sharing granule_row_mask and the from_wkbs call) — deliberately NOT in shardmap.py, which sits past the 1,200-line raise trigger (flagged on PR #440; a split proposal is standing).

Tests

Parity: unindexed build ≡ the pre-#445 records path on single-part fixtures at the same resolved order (oracle via the old path, byte-compared to_json); the coarse prefilter drops no true hit (a fixture where coarse admits a candidate that fine rejects); MultiPolygon superset divergence pinned as intended; explicit mortie_order= honored and matching the oracle; paired/beams/spherely/rect untouched (each pinned); total_granules parity (screen popcount ≡ old len(records)); null-geometry refusal on both paths; build_wall_s spans cover+intersect (today's convention — records stay outside).

Refs #439/PR #440 (the pattern), #396/#400 (the machinery), PR #432 (paired exclusivity), #429 (the WKB screen peak — stage 1 inherits index_footprints' documented memory posture).

[espg]

🤖 from Claude

Correction to the plan above — the two-stage design is withdrawn (espg, in-session): the shardmap encodes only parent_order shard membership, and _resolve_mortie_order's own docstring cites the real-catalog order sweep (bench/neon_order_sweep.py): granules/shard is flat for every order ≥ parent_order — the chunk-order default buys precision the shard cells cannot see. So the o13 concern the two-stage design answered does not exist for shard assignment.

Revised design: single-stage — one ephemeral from_wkbs cover at parent_order over the screened rows, positional intersect, _hit_records. The ephemeral path defaults to parent_order (documented with the order-sweep citation); an explicit mortie_order= pin is honored literally. The parity suite gains a test asserting new-at-parent ≡ old-at-chunk-default on fixtures with chunk_inner set — turning the order-sweep invariant into an in-suite pin. All other plan points (exclusions, superset disclosure, screen parity, timer semantics, sources.py placement) stand.