Adopt zarr-python sparse-read primitives (zarr-developers/zarr-python#4028) when released

[espg]

🤖 from Claude

Context

Reading zagg's sparse HEALPix outputs through stock arr[:] pays one store round-trip
per empty chunk — the dominant cost at scale (bench/REPORT.md: ~10× overhead on
LocalStore, ~64× on S3 for the cycle-22 case, 49,152 chunks / ~1,300 populated). Two
upstream tracks now exist:

• Enhancement proposal: empty-chunk-aware read path (array.prefetch_populated_keys) zarr-developers/zarr-python#3929 (filed from our benchmarks): an opt-in
  prefetch_populated_keys flag making Array.__getitem__ itself empty-chunk-aware.
  Status: open proposal, no implementation commitment yet.
• Add sparse-read primitives: shards_initialized and read_regions zarr-developers/zarr-python#4028 (our PR, in review): caller-side primitives —
  shards_initialized, initialized_regions, initialized_chunk_regions,
  read_regions — that deliver the same skip-the-empties read today as explicit API:
  read_regions(arr, initialized_regions(arr)).

These are complementary (transparent flag vs. explicit primitives), but #4028 is the one
we control and the one likely to land first. This issue tracks the zagg-side flips for
when a zarr-python release ships it.

Flip inventory

1. Reader utilities / examples: anywhere we document or script "read a zagg store
   back" (including the End-to-end sponsor example: catalog → shardmap → dual aggregation → cached reads #168 sponsor example's cached-reads leg), switch the sparse-array
   read from arr[:] to read_regions(arr, initialized_regions(arr)) and scatter into a
   fill-value-initialized output. Keeps the 10×/64× win without any zagg-side listing
   code.
2. Post-Make sharded: true the default for HEALPix output #233 sharded outputs: with sharded: true now the HEALPix default, shard
   granularity under-describes occupancy (a shard is one stored object over K inner
   chunks). initialized_chunk_regions reads only the shard indexes and reports written
   inner chunks — use it where zagg needs true occupancy (verification tooling,
   shardmap-style diagnostics) instead of any bespoke index-walking.
3. Benchmark refresh: re-run bench/REPORT.md against the released primitives so the
   upstream numbers cite shipped API rather than a patched read path; feed the result
   back to Enhancement proposal: empty-chunk-aware read path (array.prefetch_populated_keys) zarr-developers/zarr-python#3929 as adoption evidence.
4. Docs note re: negative caching (prototype for negative caching in StoreCache zarr-developers/zarr-python#4042, also in
   review): consumers reading zagg stores repeatedly through
   zarr.experimental.cache_store.CacheStore get absent-chunk reads remembered
   (finite TTL). Worth one paragraph in the read-side docs once released — it helps
   exactly the users not routing reads through the primitives above.

Gating / acceptance

• Blocked on the first zarr-python release carrying #4028; bump the reader-path zarr
  floor to it when adopting.
• Acceptance: sponsor-example read leg uses the primitives; benchmark table refreshed;
  no zagg-internal listing/probing workaround remains on the read path.

Cross-refs: #189 (separate upstream ask: batch array creation on the write path),
#233 (sharded default), zarr-developers/zarr-python#3929, #4028, #4042.