Skip to content

spill: carry the temporal companion through the cross-block fold (block_bytes unusable on temporal stores) #477

Description

@espg

🤖 from Claude

Espg-directed, from the 2026-08-17 spill-path profiling (scratch_agg_profile.py fold-regime arm).

The gap

A temporal-carrying config refuses the spill fold regime with a named error at block close:

field 'h_tdigest_signal': temporal companions (temporal: 'per-centroid') have no cross-block fold state yet — the spill block close carries the located channel only; run the pooled path (no aggregation.streaming block) for a temporal store (spec §8.2/§8.3, issue #410)

The fold law exists and is shipped — merge_tdigests_kway(temporal=) (PR #463) folds the channel at overviews with order-independence pinned — but SpillAggregator's cross-block close (_close_block → the mergeable fold, issue #370) carries only the located channel through its per-block reduce, so _mergeable classifies temporal fields as unfoldable and block_bytes refuses.

Why it matters (the chain this blocks)

Every temporal store — all of California — is confined to the single-block spill regime, where aggregate-phase memory is bounded by partition skew (measured floor ~4,020 MB on the fat five, invariant to δ/buffer/workers; stats run ca173e7a). #474 exposes block_bytes precisely to bound that via the fold regime; without temporal fold state the knob cannot apply to the stores that need it. Chain: this issue → #474/#475 (+ its D19 ruling) → the fat five heal as a config edit → obs-dense-shard robustness fleet-wide.

The fix

Carry the temporal channel through the spill block close exactly as the located channel is carried (the working precedent, one seam over): persist the per-centroid toc words beside the located words in the block's per-partition ragged state, and fold cross-block through the existing kway path (merge_tdigests_kway(temporal=) — the same call the overview folds use, so no new algebra). Lift the _mergeable/_fold_problems classification for temporal: fields once the carry exists, and update the refusal text. Spec-clean: §8.4 composition + the amended ruling 3 (option (c), per-centroid at every level) already define the semantics; the committed temporal/ fixture pins leaf/overview bytes (the spill fold regime is documented as NOT byte-identical to single-block — the same t-digest/kway law bounds as located, §2.3/issue #370).

Tests: extend tests/test_spill_crossblock.py's located machinery to the temporal channel (envelope conservation across a forced block close: every observation instant contained by its folded centroid's envelope; order-independence vs the pooled result within the kway law); the refusal removal pinned; a temporal+located both-channels cross-block case.

Refs #410/#463 (channel + law), #370 (fold regime), #474/#475 (the knob this unblocks), #279 (kway lineage).

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions