Adopt zagg ragged decoding: generic leaf-decode layer + HHDC tensor profile (not the only contract)

[espg]

🤖 from Claude

Refs englacial/zagg#265 (parent design), englacial/zagg#336 / PR englacial/zagg#339 (the reader contract), and the flag on #1's read-side inventory ("leaf ragged-field decoding is squarely reader territory"). Blocked by englacial/zagg#340 (the zagg-ragged/1 + digest + composition spec doc — the migration gate).

Framing caveat (espg ruling, 2026-07-29): moczarr reads anything MOC-bearing that zagg produces — HHDC is the funder-primary product profile today, not moczarr's only contract. So this lands as two layers:

1. Generic ragged-leaf decode layer: decode any zagg-ragged/1 field by its attrs (element dtype/shape, located sibling) into per-cell arrays — product-agnostic, works for every current and future zagg ragged field. This is the durable moczarr surface.
2. HHDC product profile on top: the tensor-emission contract PR Spatially faithful tensor emission: rank_to_xy deinterleave, block assembly, mask channel englacial/zagg#339 defines — (tensor, mask, (offset, gain), morton_id) blocks via mortie.rank_to_xy deinterleave (mortie ≥0.9.3, spec §8), block assembly, coverage.moc mask channel with 2-/3-state semantics keyed to store attrs. Digest→bin rasterization may import zagg for the digest algebra (espg ruling: reader may depend on zagg; the algebra stays zagg-owned) — likely as an optional extra so moczarr core stays light.

Also in scope under the profile: strata stores (two ragged fields + stratum attrs) and the 3-state mask. Out of scope: percentile materialization (separate issue), waveform/PhotonFlux algebra (zagg-side).

Acceptance: conformance against #340's golden fixtures; parity with zagg's readers/tdigest_tensor outputs on a shared fixture; the generic layer usable with zero HHDC knowledge.

[espg]

🤖 from Claude

Implementation plan — generic zagg-ragged/1 decode + HHDC tensor profile

Gate: implementation starts when englacial/zagg#340 lands (the normative zagg-ragged/1 + digest + composition spec doc). Nothing below changes shape at that point — the interim contract is already live in zagg's array attrs (attrs["ragged"] = {"spec": "zagg-ragged/1", "element": {dtype, shape}, "locations": ...}, grids/base.py RAGGED_SPEC/RAGGED_ELEMENT_ATTR) and the profile contract is PR englacial/zagg#339's read_tensors tuple. Reference implementation studied: readers/tdigest_tensor.py and readers/_layout.py on zagg's claude/336-reader-deinterleave branch.

Module placement

moczarr is a flat module tree (convention.py, coverage.py, open.py, … — one file per concern, one docs/api/<module>.md page each, mkdocs.yml nav entry, __init__ re-exports). Two new flat modules, no profiles/ subpackage:

• src/moczarr/ragged.py — the generic, product-agnostic decode layer. Zero zagg import, ever.
• src/moczarr/hhdc.py — the HHDC profile on top. Imports zagg lazily for the digest algebra only (the dggs.py lazy-extra posture: pointed ImportError hint naming moczarr[zagg]).

A subpackage buys nothing until a second profile exists; if one lands later, hhdc.py moves then (per the framing caveat in this issue: generic layer first-class, HHDC = one profile).

Two PRs

PR-A — ragged.py (generic decode, zero zagg)

• Phase A1 — attrs binding + element decode: parse_ragged_attrs (strict spec gate on "zagg-ragged/1"; missing/foreign/newer spec raises — see open question (2)), decode_cell(raw, element) = np.frombuffer little-endian + reshape to (n, *inner_shape).
• Phase A2 — store access: read_ragged(store, field) generator yielding (morton_word, array[, location_words]) per populated cell; sweep restricted to stored objects (one list_prefix of the array's c/ keys; sharded and flat layouts share the path — both self-describing in array metadata, mirroring zagg's _stored_chunk_spans); whole-span slicing (one GET per stored object, never per-inner-chunk re-fetch of the shard index). read_cell(store, field, cell) random access — 2 ranged GETs on a sharded store. Located-sibling binding via attrs["ragged"]["locations"] (metadata, never naming convention); morton sibling for cell identity, raise when absent. Store-scoped like zagg's readers: never traverses the hive digit tree (leaf discovery stays with open_hive/coverage).
• Phase A3 — fixture, goldens, docs/api/ragged.md, nav, exports.

PR-B — hhdc.py (profile) — blocked by PR-A:

• Phase B1 — layout kernel: chunk-local rank ↔ (row, col) via mortie.rank_to_xy/xy_to_rank (mortie spec §8, frozen for 1.x), orientation pinned identically to zagg's readers/_layout.py (row = y, col = x, [0, 0] at the subtree's south corner — the gridlook texture convention), never a row-major reshape.
• Phase B2 — read_tensors: the Spatially faithful tensor emission: rank_to_xy deinterleave, block assembly, mask channel englacial/zagg#339 contract — yields (tensor, mask, (offset, gain), morton_id) per block; block_order assembly of whole read chunks; block-wide z-window from per-cell bottom/top quantile trims with the three-way fit policy (raise/degrade_resolution/collapse_bins); rasterization via zagg's cdf_from_tdigest (lazy import, moczarr[zagg]).
• Phase B3 — mask channel: decode the leaf coverage.moc occupancy through moczarr's own coverage.decode_bitmap (we already own the frozen bitmap convention — no zagg call here, unlike zagg's reader which routes through zagg.hive); 2-state base (occupancy=1, digest-bearing→2), upgraded to 3-state on strata stores per the count-keyed rule in zagg's docs/signal_strata.md (unobserved / observed-zero-signal / observed-with-signal).
• Phase B4 — strata stores (two ragged fields + stratum/signal_threshold attrs), parity goldens, docs/api/hhdc.md.

Dependency / extras changes (flagged for sign-off — moczarr has no CLAUDE.md dependency clause; adopting zagg's discussion-first posture)

1. mortie floor >=0.9.1 → >=0.9.3 (core): rank_to_xy/xy_to_rank ship in 0.9.3 (rank_to_xy / xy_to_rank: rank-space (x, y) deinterleave (issue #149) mortie#150). Lean: core bump — mortie is already a light core dep and the kernels are spec-frozen.
2. New optional extra zagg (moczarr[zagg]): the digest algebra (cdf_from_tdigest, and Percentile-surface materialization: dense (quantile, cells) views for the gridlook feeder #21's quantile_from_tdigest/merge_tdigests) is imported, never vendored — vendoring is parity drift by construction. Core import graph unchanged.

Fixture strategy

• Interim (until Promote zagg-ragged/1 + digest + composition byte layouts to a normative spec (reader-migration gate) englacial/zagg#340's conformance vectors): a committed micro-fixture written by zagg's real writer, per the ratified posture (tools/generate_serc_fixture.py — run from a zagg checkout, every byte from zagg's own write path so writer↔reader drift fails the suite). New tools/generate_strata_fixture.py: a tiny SERC-area hive leaf with located h_tdigest_signal/h_tdigest_noise, the composition word, and count — one fixture serving Adopt zagg ragged decoding: generic leaf-decode layer + HHDC tensor profile (not the only contract) #19, zagg-composition/1 decoding helpers (unpack / counts / presence, attrs-bound) #20, and Percentile-surface materialization: dense (quantile, cells) views for the gridlook feeder #21.
• Parity goldens as committed .npy (the serc_cell_ids_golden.npy precedent): expected tensors/masks/windows computed by zagg's readers/tdigest_tensor.read_tensors at generation time, so moczarr-only CI pins parity without zagg installed; live-parity tests additionally run when the zagg extra is present (skip-marker otherwise).
• When Promote zagg-ragged/1 + digest + composition byte layouts to a normative spec (reader-migration gate) englacial/zagg#340 lands: re-point normative doc references and swap/add its golden conformance vectors; API unchanged.

Acceptance criteria

• Generic layer decodes a synthetic ragged field with a non-HHDC element shape, with zero HHDC knowledge; strict spec gate raises on missing/foreign/newer spec; sharded and flat layouts through one code path; read_cell GET-count posture pinned by test.
• HHDC: bit-identical (tensor, mask, (offset, gain), morton_id) vs zagg's reader on the shared fixture; mask 2-state and 3-state cases; block assembly parity (multi-chunk block = one tensor, one shared window); all three fit-policy behaviors.
• ruff + mypy + pytest green; docs page + nav + __init__ exports per module.

Open questions (with leans)

1. Where does the tensor profile live — moczarr ships it (this plan) vs moczarr exposes decode-only and zagg keeps rasterize? Lean: profile in moczarr — the gridlook feeder should be pip install moczarr[zagg], not a zagg checkout; the espg ruling already allows the import, and the algebra stays zagg-owned. zagg's in-repo reader then serves as the parity reference; whether it eventually deprecates is a zagg-side call, out of scope here.
2. Attrs posture: strict-raise vs coverage-style tolerant-None. Lean: strict — coverage's D9 tolerance is for caches (degrade to the walk, never wrong answers); ragged attrs are truth, and guessing an element layout misinterprets every payload (zagg's _open_ragged posture).
3. mortie bump: core vs gated behind the profile. Lean: core (see above).
4. open_hive integration — should ragged fields surface as variables in the opened Dataset? Lean: out of scope for v1; readers stay store-scoped generators. One check-item for PR-A: verify open_hive on a strata leaf doesn't choke on the vlen arrays it isn't decoding.

Ordering

#20 is independent (lands anytime); this issue gates #21 (its decode layer is #21's input); implementation here gates on englacial/zagg#340.

[espg]

🤖 from Claude

The englacial/zagg#340 gate this issue is blocked by now has its PR up: englacial/zagg#346 (draft, all phases landed, pending review). What it provides for this issue's PR-A:

• docs/specification.md — normative §1 zagg-ragged/1 (layout, attrs block, codec chain, golden wire framing, sharded/unsharded geometries, succession), §2 digest payload semantics (centroid array + located channel words; algebra deliberately unspecified), §6 zagg-ragged/2 (full normative section: vlen-ndarray, byte-identity, revision signaling — so the optional /2 decode can be written against spec + package, ahead of any zagg writer).
• Conformance fixtures to vendor: tests/data/spec/ — minimal (unlocated digest) and kitchen_sink (located strata + composition + count) hive stores, each with a *.expected.json golden (decoded values + O11 hashes). Edge cases pinned: empty inner chunk (index sentinel), b"" fill cells, empty (0, 2) stratum payload, merged-centroid common-ancestor locations.
• zagg's tests/test_spec_conformance.py decodes the same fixtures with spec-text-only decoders (struct + zstd, no zagg read path), so the acceptance criterion here — decode from spec + fixtures alone — is already exercised on the zagg side.

The gate formally opens when englacial/zagg#346 merges; fixture paths are stable in the PR branch (claude/340-store-spec) if PR-A wants to start against it.

[espg]

🤖 from Claude

Dependency sign-off recorded (espg, 2026-07-30, in-session, verbatim "deps signed off"): both flagged changes are approved —

1. mortie core floor bump >=0.9.1 → >=0.9.3 (rank_to_xy/xy_to_rank, rank_to_xy / xy_to_rank: rank-space (x, y) deinterleave (issue #149) mortie#150);
2. new optional extra moczarr[zagg] importing zagg's digest algebra (cdf_from_tdigest, quantile_from_tdigest, merge_tdigests) — imported, never vendored, core import graph unchanged.

Also directed: implementation starts now, against the claude/340-store-spec branch (englacial/zagg#346) rather than waiting for its merge — spec references and conformance-vector paths point at the branch and get confirmed/re-pointed when #346 merges. Implementation of PR-A proceeding under the standing per-PR review/fold cycle.