Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
afa9660
phase 1 of issue #410
espg Aug 17, 2026
7a0fb03
phase 2 of issue #410
espg Aug 17, 2026
19ab557
phase 3 of issue #410
espg Aug 17, 2026
0e8b3b6
phase 4 of issue #410
espg Aug 17, 2026
7df13b1
fold review: state the reserved 0 word as cost-free (issue #410)
espg Aug 17, 2026
4a82bd5
fold review: license the per-centroid to per-cell reduction in 8.4 (i…
espg Aug 17, 2026
684102d
fold review: scope 8.4 to the companion shapes (issue #410)
espg Aug 17, 2026
45bd869
fold review: require the reserved per-cell fill explicitly (issue #410)
espg Aug 17, 2026
26427ca
fold review: gate the unproduced temporal key at config validation (i…
espg Aug 17, 2026
4d68570
key the companion word claims on the word, not the weight (issue #410)
espg Aug 17, 2026
10c63d8
fold review: state the point word's no-area claim, not an exact posit…
espg Aug 17, 2026
47225ce
fold review: key the ancestor claim on words, drop the finest-over-ob…
espg Aug 17, 2026
f2b1c36
fold review: carry the per-word decode duty into 2.2 (issue #410)
espg Aug 17, 2026
9b05883
fold review: key the per-cell word on the join, not the member count …
espg Aug 17, 2026
879549f
fold review: restore the companions' MUST on encoding discipline (iss…
espg Aug 17, 2026
11def4d
fold review: rekey the located pin onto recorded member runs (issue #…
espg Aug 17, 2026
4827134
fold review: describe the fixture centroids as merged, not weighted (…
espg Aug 17, 2026
e2a0105
fold review: scope the uncertainty note to the reader (issue #410)
espg Aug 17, 2026
9dabc7d
fold review: scope coarse ingest to the declared companion (issue #410)
espg Aug 17, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion docs/hive_layout.md
Original file line number Diff line number Diff line change
Expand Up @@ -906,7 +906,8 @@ Raster (pull-NN) pipelines write the same tree with **windowed `(time, cells)`
leaves**: one vanilla zarr v3 leaf per **(shard, window)** unit at
`shard_leaf_path(root, shard, window=label)`, each carrying leaf-local `time`
(int64 microseconds with CF attrs by default, or `uint64` mortie toc words
carrying the spec §8 `temporal` declaration and no CF attrs under
carrying the spec §8.1 `temporal` declaration at `shape: "coordinate"`
and no CF attrs under
`output.time_encoding: toc` — which the shipped Sentinel-2 config sets, issue
#443) and `morton` (packed u64 words) as the sole cell
coordinate — `cell_ids` (NESTED) rides only the `emit_cell_ids` transition hatch
Expand Down
41 changes: 38 additions & 3 deletions docs/ragged_layout.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ migration is metadata-only.

## Why the attrs are the contract

The element interpretation (dtype, inner shape, the located sibling's name)
The element interpretation (dtype, inner shape, the companion siblings' names)
is self-describing in the array's attrs — the
[spec §1.2 block](specification.md#12-the-ragged-attrs-block) — so a reader
decodes exactly what the writer declared rather than hardcoding a dtype or
Expand All @@ -37,6 +37,38 @@ layout. A vlen array without a well-formed `element` declaration is **not** a
zagg ragged array — pre-issue-209 CSR stores are a hard break, and the readers
raise a pointed error rather than decode under a guessed layout.

## Why the companion words are declared, not implied

A companion sibling holds **packed words**, not self-describing scalars — a
morton cell in `{field}_locations`, a toc time in `{field}_times` — so
nothing in a uint64 array says which grammar produced it. The spec's
word-typed declaration
([§8](specification.md#8-zagg-toc1), instantiated for space in
[§9](specification.md#9-zagg-located1)) closes that: a `{spec, shape,
grammar}` block, stamped by the writer on the array that *holds* the words,
with the payload array carrying only the *binding* to it. One rule follows
from that split and is worth stating plainly here: **read the declaration off
the sibling you bound, never off the payload.**

Two properties the family is designed around:

- **Absence is never a refusal.** Every located store written before §9
carries no `located` key, and reads exactly as
[§2.2](specification.md#22-the-location-channel) always said it did. The
declaration adds self-description and the shape vocabulary; it never
reinterprets a byte.
- **Composition gates on `{shape, grammar}`** — the same discipline the
[§2.0 `weights`](specification.md#20-the-weights-declaration) declaration
established, and for the same reason: merging two arrays whose words mean
different things yields an array that means neither.

The binding for the temporal sibling is a **sibling attrs key** (`times`),
not a key inside the versioned `ragged` block, matching `weights` and for the
same reason — the block is retired wholesale under
[`zagg-ragged/2`](specification.md#6-zagg-ragged2), and a key outside it
survives that metadata-only migration untouched. `locations` predates that
ruling and stays where it is; the asymmetry is history, not design.

## Why the wire framing is golden-pinned

Round-trip tests pass under *any* self-consistent encoding, so only a fixed
Expand Down Expand Up @@ -85,7 +117,9 @@ store). The sharded 2-GET count is pinned by
A [hive](hive_layout.md) leaf zarr is exactly this layout scoped to one shard.
Under the leaf's `{group}` path a reader finds the ragged vlen array with its
versioned `ragged` attrs, the sibling `morton` coordinate array (chunk
identity), and, for a located field, the `{field}_locations` sibling — all
identity), and the field's companion siblings — `{field}_locations` for a
located field, `{field}_times` for one carrying the per-centroid temporal
companion ([spec §8.3](specification.md#83-shape-per-centroid)) — all
sharded as one whole-leaf `ShardingCodec` object (one stored span). So a hive
product is **read one leaf at a time**: open the leaf store
(`hive.shard_leaf_path`) and pass the same `field` path to the readers. The
Expand Down Expand Up @@ -114,7 +148,8 @@ and hive-leaf writers are pinned to store byte-identical per-cell payloads
then the one inner chunk holding the cell), never the whole shard object. An
out-of-range index raises `IndexError` naming the valid range (no negative-index
wrap); an absent cell returns the zero-length `(0, *inner_shape)` array. Works
on the `{field}_locations` sibling too.
on the `{field}_locations` and `{field}_times` siblings too — both are
themselves `zagg-ragged/1` uint64 arrays.
- **Sub-leaf subtree read** (`subtree=` on the sweep readers —
[issue #351](https://github.com/englacial/zagg/issues/351)) — the nested
cells axis makes the subtree below any ancestor one **contiguous cell span**
Expand Down
7 changes: 5 additions & 2 deletions docs/signal_strata.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,8 +31,11 @@ is **committed at ingest** and carried through the store in three pieces:

The shipped template is `zagg/configs/atl03_tdigest_strata_healpix.yaml` —
**located strata is the default**: both digest fields carry `location:
leaf_id`, so each centroid stores its order-29 morton word (an exact photon
position at weight 1) in the `{field}_locations` sibling arrays. The
leaf_id`, so each centroid stores a morton word in the `{field}_locations`
sibling arrays — an order-29 **point** word (the photon's reported position,
no area claim) for an unmerged centroid, the deepest common ancestor of its
members' words once centroids merge; a reader decodes kind and order from
each word, never from the centroid's weight (spec §2.2/§9.1). The
five confidence columns are read from the single 2-D `signal_conf_ph` dataset
via the per-variable `column` selector (`{path: ..., column: k}` — the
variable analogue of the structured-filter `column`); the shared path is
Expand Down
Loading
Loading