GEDI waveforms 1/3: §2 counts/flux weights declaration + δ=8,192 raise

docs/specification.md

@@ -144,7 +144,9 @@ A vlen array without a well-formed `element` declaration is **not** a
 `zagg-ragged/1` array; a reader MUST refuse it with a pointed error rather
 than decode under a guessed layout (pre-issue-209 CSR stores are a hard
 break). The `ragged` attrs key is reserved: config-declared field attrs MUST
-NOT shadow it (enforced at config validation). A located field's provenance
+NOT shadow it (enforced at config validation). The §2.0 `weights` key is
+likewise spec-owned on a ragged payload array — writer-stamped from the
+field's declaration, never author-transcribed. A located field's provenance
 attrs (e.g. `stratum`, `signal_threshold` — §3.3) land on the **payload array
 only**; the `{field}_locations` sibling carries no user attrs.
 
@@ -239,15 +241,48 @@ A t-digest field is a `zagg-ragged/1` (or `/2`) array whose element
 declaration is `{"dtype": "float32", "shape": [-1, 2]}`. Source of truth in
 code: `zagg.stats.tdigest`.
 
+### 2.0 The `weights` declaration
+
+**Contract** ([issue #422](https://github.com/englacial/zagg/issues/422)).
+A digest payload array declares the semantics of its weight column under the
+**`weights`** attrs key — a **sibling** of the §1.2 `ragged` block on the
+payload array, never a key inside it (the `ragged` block is retired wholesale
+under `/2` — §1.6/§6.3 — so a sibling key survives that metadata-only
+migration untouched). Two values are defined:
+
+- **`"counts"`** — weights are observation counts: integers ≥ 1 whose sum is
+  the cell's exact observation count, per §2.1. **An absent `weights` key
+  MUST be read as `"counts"`** — every store written before this revision is
+  conformant verbatim, no byte rewritten.
+- **`"flux"`** — weights are calibrated flux: positive finite float32 reals
+  (a zero-weight observation carries no flux and MUST NOT produce a row);
+  `sum(weights)` estimates the cell's detected **photoelectrons**, not an
+  observation count. A flux-declared array MUST record its calibration
+  provenance in the same attrs: a `gain` key carrying at minimum the gain
+  constant's `name` and `version` (the operating point of any write-time
+  clip rides alongside, writer-defined).
+
+A reader MUST strict-check the value: an unknown declaration is a future
+revision of this section and MUST be refused, never read as either defined
+value. **Merges are legal only between payloads carrying the same
+declaration** (counts with counts, flux with flux — an absent key is
+`"counts"` for this rule too): a mixed merge would produce a weight column
+whose sum means neither thing, so a merging reader or writer MUST refuse it.
+The declaration rides the payload array only; a located field's
+`{field}_locations` sibling carries no `weights` key (§1.2's no-user-attrs
+rule for siblings is unchanged).
+
 ### 2.1 Centroid array
 
 **Contract.** A populated cell's decoded payload is a `(k, 2)` **float32**
 array of weighted centroids:
 
-- column 0 is the centroid **mean**; column 1 is the centroid **weight**
-  (the number of observations merged into it, ≥ 1);
+- column 0 is the centroid **mean**; column 1 is the centroid **weight** —
+  under the `"counts"` declaration (§2.0, the default) the number of
+  observations merged into it, an integer ≥ 1; under `"flux"` a positive
+  real per §2.0;
 - rows MUST be sorted **ascending by mean**;
-- `sum(weights)` MUST equal the cell's **exact** observation count — the
+- under `"counts"`, `sum(weights)` MUST equal the cell's **exact** observation count — the
   number of finite `source` values the digest was built over (non-finite
   source rows are dropped before building) — **while that count is
   representable in float32, i.e. `<= 2^24` (16,777,216)**; above that bound
@@ -258,6 +293,9 @@ array of weighted centroids:
   watch at coarse overview orders (§4.4). For a stratified product (§3) each
   stratum digest's total weight is the exact stratum count, under the same
   bound;
+- under `"flux"` (§2.0) `sum(weights)` is a float32 photoelectron estimate,
+  not a count: the exact-count recovery above (and §3.3's) is undefined for
+  a flux payload, and no integrality holds;
 - an absent cell decodes as the zero-length `(0, 2)` array (the `b""` fill).
 
 ### 2.2 The location channel
@@ -823,7 +861,13 @@ staged sweep's finisher.
   method on an excluded field would declare a t-digest array that does not
   exist. `exact`/`approximate` entries carry the fold `method`, any further
   fold provenance (an `exact` fold's `nan_policy`), and enough dtype/shape
-  metadata to know the overview array's form up front. This map is the
+  metadata to know the overview array's form up front. An `approximate`
+  entry MAY additionally carry `overview_delta` — the compression budget
+  overview folds run at when it is split from the leaf `delta`
+  ([issue #424](https://github.com/englacial/zagg/issues/424); both budgets
+  are fold algebra, informative per §2.3) — and, for a non-default §2.0
+  declaration, `weights`; a reader MUST tolerate entry keys it does not
+  bind. This map is the
   **all-fields** view; the per-overview `zagg_overview.fields` attrs map
   (§4.3) is the materialized subset.
 - **`all_time`** — whether the `all.zarr` all-time fold is materialized at
@@ -1322,13 +1366,20 @@ drift fails zagg's own suite (`tests/test_spec_conformance.py`) on
 whichever side moved. moczarr vendors the same fixtures for its parity
 gates (espg/moczarr#19/#20).
 
-Three tiny single-shard hive stores plus one manifest-only declaration, all
+Four tiny single-shard hive stores plus one manifest-only declaration, all
 on the same deliberately small geometry — shard order 4, inner-chunk order
 5, cell order 6 (16 cells, K = 4 inner chunks of 4 cells), sharded (the
 hive default):
 
 - **`minimal/`** — one *unlocated* digest field (`h_tdigest`) plus `count`.
   The smallest thing that is a conforming store.
+- **`flux/`** — the §2.0 `weights` declaration surface: one flux-declared
+  digest field (`rx_flux`, `weights: "flux"` stamped beside the `ragged`
+  block, `gain` provenance attrs) plus `count`. Its payloads carry
+  fractional positive weights whose per-cell sums are **not** integers —
+  the pin that a flux reader must not round-trip weights through counts —
+  while `minimal/` (committed before this revision, unregenerated) pins the
+  absent-key ⇒ `"counts"` default.
 - **`kitchen_sink/`** — the full stratified-product surface: located
   signal/noise digest strata (payload + `{field}_locations` siblings,
   `stratum`/`signal_threshold` provenance attrs), the `composition` word

src/zagg/column.py

@@ -167,7 +167,7 @@ def fold_column(slabs: dict, fields: dict, *, cell_order: int, resolutions: list
     a fractional fold factor, which no guard downstream can read as a divisor
     (both classes would surface it as an opaque numpy failure instead).
     """
-    from zagg.sweep_overview import decode_digest, fold_dense, fold_digests
+    from zagg.sweep_overview import decode_digest, fold_dense, fold_digests, overview_fold_delta
 
     cell_order = int(cell_order)
     fields = composable_fields(fields)
@@ -192,7 +192,7 @@ def fold_column(slabs: dict, fields: dict, *, cell_order: int, resolutions: list
             else:
                 dtype = meta.get("dtype") or "float32"
                 inner = tuple(meta.get("inner_shape") or (2,))
-                delta = int(meta.get("delta") or 512)
+                delta = overview_fold_delta(meta)
                 if slab.shape[0] % factor:
                     raise ValueError(
                         f"cannot fold {slab.shape[0]} cells {factor}-to-one for {name!r}"
@@ -223,11 +223,14 @@ def _column_provenance(meta: dict) -> dict:
     shared helper — the overview's identical gap is a spec call for the
     issue #383 phase 4 section, not a reason to leave this artifact short.
     """
-    from zagg.sweep_overview import _field_provenance
+    from zagg.sweep_overview import _field_provenance, overview_fold_delta
 
     entry = dict(_field_provenance(meta))
     if meta.get("class") == "approximate":
         entry["delta"] = int(meta.get("delta") or 512)
+        # The budget the column fold actually compressed at (issue #424):
+        # the split overview_delta, not the leaf δ.
+        entry["overview_delta"] = overview_fold_delta(meta)
         entry["dtype"] = meta.get("dtype") or "float32"
         entry["inner_shape"] = list(meta.get("inner_shape") or (2,))
     return entry

src/zagg/config.py

@@ -565,13 +565,22 @@ def validate_config(config: PipelineConfig) -> None:
         if attrs is not None:
             if not isinstance(attrs, dict) or not all(isinstance(k, str) for k in attrs):
                 raise ValueError(f"Variable '{name}': attrs must be a mapping with string keys")
-            from zagg.grids.base import RAGGED_ELEMENT_ATTR
+            from zagg.grids.base import RAGGED_ELEMENT_ATTR, WEIGHTS_ATTR
 
             if RAGGED_ELEMENT_ATTR in attrs:
                 raise ValueError(
                     f"Variable '{name}': attrs key {RAGGED_ELEMENT_ATTR!r} is reserved "
                     f"for the ragged layout block (issue #209)"
                 )
+            # The §2.0 weights key is spec-owned on ragged payload arrays:
+            # the template stamps it from the field-level ``weights:``
+            # declaration, so an attrs transcription could silently disagree.
+            if meta.get("kind") == "ragged" and WEIGHTS_ATTR in attrs:
+                raise ValueError(
+                    f"Variable '{name}': attrs key {WEIGHTS_ATTR!r} is spec-owned on a "
+                    f"ragged payload array — declare the field-level 'weights:' key "
+                    f"instead (spec §2.0, issue #424)"
+                )
             import json
 
             try:
@@ -1473,7 +1482,11 @@ def _validate_output_kind(name: str, meta: dict) -> None:
     ``ragged``). ``scalar`` fields need neither and stay the default path.
     ``vector`` and ``ragged`` fields may be driven by either ``function`` or
     ``expression``; ``len``/``count`` are rejected for both (they short-circuit
-    to a scalar count). See issue #29 (vector) and issue #48 (ragged).
+    to a scalar count). See issue #29 (vector) and issue #48 (ragged). A ragged
+    field may additionally declare ``weights`` (the spec §2.0 counts/flux
+    payload declaration; flux requires ``gain`` provenance attrs) and
+    ``overview_delta`` (the split pyramid-fold budget) — both issue #424,
+    both rejected on other kinds.
 
     A field may also declare ``resolution`` (``cell`` default, or ``chunk``).
     A ``resolution: chunk`` field (issue #30 item 2) is written ONCE per chunk
@@ -1514,6 +1527,16 @@ def _validate_output_kind(name: str, meta: dict) -> None:
             f"Variable '{name}': 'location' is only valid for kind 'ragged', not '{kind}'"
         )
 
+    # ``weights`` (spec §2.0, issue #424) declares a digest payload's
+    # weight-column semantics; ``overview_delta`` (issue #424) is the split
+    # pyramid-fold compression budget. Both describe a ragged digest payload —
+    # nothing else has a weight column or an overview digest fold.
+    for key in ("weights", "overview_delta"):
+        if key in meta and kind != "ragged":
+            raise ValueError(
+                f"Variable '{name}': '{key}' is only valid for kind 'ragged', not '{kind}'"
+            )
+
     # resolution (cell default, or chunk). A chunk-resolution field stores one
     # value per chunk in a companion array (issue #30 item 2). ``scalar`` and
     # ``vector`` chunk companions are wired (issue #82): a scalar companion is a
@@ -1576,6 +1599,44 @@ def _validate_output_kind(name: str, meta: dict) -> None:
         raise ValueError(f"Variable '{name}': kind 'ragged' requires 'inner_shape'")
     _validate_trailing_shape(name, meta["inner_shape"], key_name="inner_shape")
 
+    # The §2.0 weights declaration (issue #424): "counts" (integer weights,
+    # sum exact — the absent-key default) or "flux" (positive reals, sum an
+    # estimate of detected photoelectrons). A flux field MUST carry its
+    # calibration provenance — the spec requires a ``gain`` attrs mapping
+    # naming at minimum the gain constant's name and version — so a store
+    # never holds calibrated weights whose calibration is unrecoverable.
+    weights = meta.get("weights")
+    if weights is not None:
+        from zagg.grids.base import WEIGHTS_KINDS
+
+        if weights not in WEIGHTS_KINDS:
+            raise ValueError(
+                f"Variable '{name}': weights {weights!r} is not one of {WEIGHTS_KINDS} (spec §2.0)"
+            )
+        if weights == "flux":
+            gain = (meta.get("attrs") or {}).get("gain")
+            if not isinstance(gain, dict) or not {"name", "version"} <= set(gain):
+                raise ValueError(
+                    f"Variable '{name}': weights 'flux' requires calibration provenance "
+                    f"in attrs — a 'gain' mapping with at least 'name' and 'version' "
+                    f"(spec §2.0, issue #424)"
+                )
+
+    # ``overview_delta`` (issue #424): the pyramid/overview fold budget, split
+    # from the leaf ``params.delta`` (leaf δ is the loss-free bound, overview δ
+    # the ~1/δ accuracy bound). A top-level key, NOT a params entry — params
+    # values are forwarded to the reducer as kwargs, which never take it.
+    overview_delta = meta.get("overview_delta")
+    if overview_delta is not None:
+        if not isinstance(overview_delta, int) or isinstance(overview_delta, bool):
+            raise ValueError(
+                f"Variable '{name}': overview_delta must be a positive int (got {overview_delta!r})"
+            )
+        if overview_delta < 1:
+            raise ValueError(
+                f"Variable '{name}': overview_delta must be a positive int (got {overview_delta!r})"
+            )
+
     # Same restriction as vector: ``len``/``count`` produce a scalar count.
     if meta.get("function") in ("len", "count"):
         raise ValueError(
@@ -2166,6 +2227,10 @@ def get_output_signature(meta: dict) -> dict:
         # Ragged location channel (issue #87): the per-observation morton column
         # the reducer folds per centroid; ``None`` for unlocated fields.
         "location": meta.get("location"),
+        # Ragged weights declaration (spec §2.0, issue #424): "counts"/"flux",
+        # or ``None`` when undeclared (the spec reads absence as counts, so
+        # ``None`` keeps pre-#424 templates and signatures byte-identical).
+        "weights": meta.get("weights"),
     }
 
 
@@ -2204,6 +2269,10 @@ def output_field_signature(config: PipelineConfig) -> list[dict]:
         # so existing shard-map signatures are byte-identical.
         if sig["location"] is not None:
             entry["location"] = sig["location"]
+        # A weights declaration changes the payload semantics (spec §2.0,
+        # issue #424) — same keyed-only-when-set discipline as location.
+        if sig["weights"] is not None:
+            entry["weights"] = sig["weights"]
         fields.append(entry)
     return sorted(fields, key=lambda f: f["name"])
 

src/zagg/configs/atl03_tdigest_healpix.yaml

@@ -68,7 +68,7 @@ aggregation:
     h_tdigest:
       # Per-cell t-digest of photon heights: a ragged (vlen-bytes) field whose payload
       # is an (n_centroids, 2) array of (mean, weight) centroids. ``build_tdigest``
-      # is called as build_tdigest(h_ph_values, delta=256); ``inner_shape: [2]``
+      # is called as build_tdigest(h_ph_values, delta=8192); ``inner_shape: [2]``
       # declares the per-element centroid width. To store ONE digest per chunk
       # instead of per cell, add ``resolution: chunk`` (one payload per inner
       # chunk under the chunk-uniform contract).
@@ -77,7 +77,15 @@ aggregation:
       source: h_ph
       inner_shape: [2]
       params:
-        delta: 256
+        # Leaf centroid budget (issues #414/#424): loss-free while a cell's
+        # count stays <= delta. 8192 covers every real surface cell in the
+        # statewide CA scan (only atmospheric storm artifacts exceed it);
+        # digest size scales with actual counts, so typical cells are unmoved.
+        delta: 8192
+      # Pyramid/overview folds compress at this split budget instead (~1/512
+      # quantile accuracy) — overviews are summaries, not loss-free carriers,
+      # and the cap bounds the sweep's k-way fold buffers (issue #424).
+      overview_delta: 512
       dtype: float32
       fill_value: 0
 

src/zagg/configs/atl03_tdigest_healpix_hive.yaml

@@ -66,7 +66,8 @@ aggregation:
       source: h_ph
       inner_shape: [2]
       params:
-        delta: 256               # centroid budget (accuracy knob)
+        delta: 8192              # leaf centroid budget: loss-free bound (issues #414/#424)
+      overview_delta: 512        # pyramid-fold budget: ~1/512 accuracy (issue #424)
       dtype: float32
       fill_value: 0
 

src/zagg/configs/atl03_tdigest_located_healpix.yaml

@@ -68,7 +68,8 @@ aggregation:
       location: leaf_id
       inner_shape: [2]
       params:
-        delta: 256
+        delta: 8192              # leaf centroid budget: loss-free bound (issues #414/#424)
+      overview_delta: 512        # pyramid-fold budget: ~1/512 accuracy (issue #424)
       dtype: float32
       fill_value: 0
 

src/zagg/configs/atl03_tdigest_strata_healpix.yaml

@@ -97,11 +97,12 @@ aggregation:
       location: leaf_id
       inner_shape: [2]
       params:
-        delta: 256
+        delta: 8192              # leaf centroid budget: loss-free bound (issues #414/#424)
         where: >-
           (signal_conf_land >= 2) | (signal_conf_ocean >= 2) |
           (signal_conf_sea_ice >= 2) | (signal_conf_land_ice >= 2) |
           (signal_conf_inland_water >= 2)
+      overview_delta: 512        # pyramid-fold budget: ~1/512 accuracy (issue #424)
       dtype: float32
       fill_value: 0
       attrs:
@@ -114,11 +115,12 @@ aggregation:
       location: leaf_id
       inner_shape: [2]
       params:
-        delta: 256
+        delta: 8192              # leaf centroid budget: loss-free bound (issues #414/#424)
         where: >-
           ~((signal_conf_land >= 2) | (signal_conf_ocean >= 2) |
           (signal_conf_sea_ice >= 2) | (signal_conf_land_ice >= 2) |
           (signal_conf_inland_water >= 2))
+      overview_delta: 512        # pyramid-fold budget: ~1/512 accuracy (issue #424)
       dtype: float32
       fill_value: 0
       attrs: