Skip to content

D24: admit build_tdigest_where to the digest family; give composition an honest composability declarationΒ #515

Description

@espg

πŸ€– from Claude

Filed at espg's direction (in-session 2026-08-24, ruling: admit). Split from PR #510's standing question (1) β€” the gate drift its phase-1 characterization surfaced and deliberately did not unify.

The drift, and the ruling

build_tdigest_where (the strata builder) is admitted by the spill gate (_TDIGEST_SPILL_FUNCTIONS, issue #370: stratum membership is per-photon row selection before the build, so per-block strata digests merge like any digest) but not by D24's composability gate β€” so strata fields classify {class: none}. This is not hypothetical: the published CA store's manifest declares h_tdigest_signal/h_tdigest_noise/composition all class: none (verified live on atl03_tdigest_o9.zarr/morton_hive.json), while every one of its 2,726 shards was built by k-way merging per-block strata partials under spill β€” the store's bytes already rest on the law D24 declines to state. Espg ruled: admit.

Scope (two admissions, one honest declaration)

  1. build_tdigest_where β†’ the digest-family registry (PR D24 composability: admit build_waveform_digest via a shared digest-family registry (issue #508)Β #510's _DIGEST_FAMILY_FUNCTIONS): same k-way law, same (2,) centroid element, delegates to build_tdigest β€” pure registry admission plus parity tests (strata payload + located/temporal companions through the fold; the WHERE clause has no effect on merge semantics).
  2. composition: merges by a different, exact law β€” merge_composition_kway over packed (word, n_signal) pairs (issues Carry ATL03 signal confidence through aggregation: signal/noise split + t-digest encoding for signal-weighted photon fluxΒ #321/spill: cross-block merge for located / strata / composition fields (lift the single-block restriction)Β #370; also already spill-admitted). D24's approximate arm is digest-shaped, so composition needs its own honest class/method declaration rather than shoehorning β€” determine the right shape (likely an exact-law entry with its packed-ragged encoding declared); if that needs a new D24 method/class token, the spec page and conformance fixtures move in the same PR (Promote zagg-ragged/1 + digest + composition byte layouts to a normative spec (reader-migration gate)Β #340 rule). Without this, a strata pyramid loses its signal-fraction companion at overview levels β€” arguably the most useful browse quantity.

Why now

The pyramid retrofit sweep planned for the 0.50 fleet declares fields from the classifier β€” admitting strata (+composition) is what makes the CA ATL03 retrofit produce digest pyramids instead of count-only. Retrofit mechanics note: leaf columns were never written for these fields (class none at build time), so the retrofit uses the leaf-reading path (declare_pyramid + cascade-from-leaves) β€” more reads, zero re-aggregation.

Refs: #508 / PR #510 (the registry + the raised drift, issue comment 5398201934), #370, #321, #334/#337 (strata design), #340 (spec+fixtures rule).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions