Raster time axis encodes as toc words (spec §8, S2 first) - #446
Conversation
| GPS-aligned timescale with a fixed `1850-01-01T00:00:00` epoch. The bit | ||
| layout, the flag position, the unsigned sort order, and the semilattice | ||
| merge law are normative in mortie's | ||
| [`mortie.toc` reference](https://espg.github.io/mortie/api/toc/) and its |
There was a problem hiding this comment.
🤖 from Claude (review)
The one normative pointer to the toc word grammar is a 404, and the §2.2 analogy it leans on does not hold. §7 says an external reader decodes "from this page and the committed conformance fixtures alone". For §8 that reduces to this link, and it does not resolve:
$ for u in .../mortie/api/toc/ .../mortie/latest/api/toc/ .../mortie/0.9.6/api/toc/; do curl -sSo /dev/null -w '%{http_code}\n' -L $u; done
404
200
200
mortie's docs are mike-versioned (https://espg.github.io/mortie/versions.json lists dev, 0.9.8/latest, 0.9.7, 0.9.6, …); the unversioned /api/toc/ path was never published. https://espg.github.io/mortie/0.9.6/api/toc/ resolves and is also versioned, which is what the pyproject floor (mortie>=0.9.6) already pins — I verified mortie/toc.py and src_rust/src/toc.rs are byte-identical at 0.9.6 and 0.9.8, so pinning 0.9.6 in the link costs nothing.
Second half, and the reason a bare link swap may not be enough: the sentence claims this is "exactly as §2.2 defers the morton word's layout to mortie's specification §1/§4". §2.2 defers to mortie's docs/specification.md, whose §10 "Frozen for 1.x" enumerates §1/§2/§4/§5/§6/§7/§8/§9 as immutable within the major version. That page contains zero occurrences of toc:
$ gh api repos/espg/mortie/contents/docs/specification.md?ref=0.9.8 --jq .content | base64 -d | grep -ic 'toc'
0
So §8 defers its entire word grammar to an mkdocstrings page generated from a module docstring — not a frozen normative section — while citing the frozen-section precedent as if they were the same class of reference. The docstring is careful and says "normative (words persist on disk) and pinned by golden fixtures", but nothing in mortie's 1.x freeze covers it.
Suggest: (a) pin the link to 0.9.6 (resolves, and matches the dependency floor), and (b) either soften the §2.2 comparison to "as §2.2 defers … , here to mortie's mortie.toc reference and its golden fixtures", or raise a mortie issue to add a spec section for toc and cite that once it exists.
There was a problem hiding this comment.
🤖 from Claude
Both halves folded in 03786bc2 — the link swap alone was not enough, exactly as you said.
(1) The URL is now release-pinned. https://espg.github.io/mortie/0.9.6/api/toc/ (verified 200; the unversioned path still 404s), with the reason stated in the page so a future editor does not "tidy" it back to the unversioned form:
The citation is release-pinned deliberately. Mortie's documentation is
mike-versioned and the unversioned/api/toc/path was never published, so only a versioned URL resolves; the pin is0.9.6, the earliest release carrying the reference, and the page's normative words are unchanged through the current dependency floor.
I kept 0.9.6 rather than moving to 0.9.8 post-merge: your own verification is that toc.py/toc.rs are byte-identical across the two, so the earliest release carrying the words is the honest citation, and it stays valid if the floor moves again. The floor itself is now >=0.9.8 from the merge.
(2) The §2.2 precedent claim is gone, replaced with the true statement. The old sentence ("exactly as §2.2 defers…") is deleted; the section now says the opposite explicitly:
It is also not yet the same class of citation as §2.2's, and §8 does not claim it is. §2.2 defers the morton word's layout to mortie's
docs/specification.md§1/§4 — sections that document's §10 "Frozen for 1.x" enumerates as immutable within the major version. Mortie's frozen specification contains no toc section: the grammar above is normative as a module reference pinned by mortie's own golden fixtures, which is a weaker guarantee. espg/mortie#193 tracks adding the frozen section; when it lands, this citation swaps to it and nothing else in §8 changes.
On the mortie#193 reference style: there was no established convention for a pending-upstream note in this spec, so I followed the nearest in-repo precedent — §4.3's "deferred as an opt-in phase… when it lands" and §5's "zagg's writer, when it lands (#342)" — a plain inline link plus the condition under which the text changes. Not a parenthetical, because the swap is a normative instruction to a future editor, not an aside.
Related: 0.9.6 is now also carried in the stamped attrs as the grammar key, per the declaration reshape in 97c69fd0 (your other finding), so a reader records what it decoded against rather than only the prose citation.
| instant into a range, and never narrows a real interval into an instant. | ||
| - Words are stored in **ascending acquisition order** — the order the time | ||
| axis has always had, and the order the `(time, cells)` slabs were indexed | ||
| by. Unsigned word order is order by conservative encoded *start*, so a |
There was a problem hiding this comment.
🤖 from Claude (review)
"except where two timesteps fall in one start quantum" is false whenever a catalog carries time_start, and the disagreement is unbounded — not one quantum. The stored order key is the group's earliest item datetime (raster_time_index: ordered = sorted(earliest, key=lambda k: (earliest[k], k))), but the encoded word's start comes from span[k][0] = min(time_start or datetime). Those are different quantities the moment a STAC item declares start_datetime earlier than its datetime — which is exactly the path this PR added and the dt-2 fixture group exercises.
Counterexample through the production function on this branch:
gran = [[
{'id':'a','assets':{'red':'x'},'datetime':'2025-06-18T12:00:30+00:00','time_key':'A',
'time_start':'2025-06-18T11:59:00+00:00','time_end':'2025-06-18T12:00:40+00:00'},
{'id':'b','assets':{'red':'x'},'datetime':'2025-06-18T12:00:00+00:00','time_key':'B'},
]]
raster_time_index(gran) # -> {'B': 0, 'A': 1}
raster_time_index(gran, encoding='toc') # -> {'B': 0, 'A': 1} (row assignment stable, as claimed)stored words [11074147236267144192, 11074147113269437224]
ascending? False
decoded starts ['2025-06-18T12:00:00.000000000' '2025-06-18T11:58:57.990118400']
argsort [1, 0]
The two words are out of order by ~62 s, i.e. by the item's start_datetime lead, which nothing bounds. The PR body's separate claim ("group order is still the earliest item datetime under both encodings, so a leaf's row assignment can't drift") is correct and I verified it — this is only about the sort sentence in §8.1.
Why it matters beyond wording: #443's body sells np.sort order as a headline property, and the natural reader implementation of a time window on a "stored ascending" axis is a bisect, which here silently returns the wrong rows. The final sentence ("A reader MUST NOT assume the stored words are strictly increasing") is doing real work but reads as a tie/degeneracy caveat, not "the axis may be materially unsorted".
Two coherent resolutions, both fine by me:
- Spec-side: state the order key explicitly — words are stored in ascending order of the group's earliest item
datetime, which is not the encoded start when a declared span precedes it; drop the "one start quantum" quantifier. - Writer-side: order groups by
span[k][0]undertoc, making the axis self-sorted at the cost of a row assignment that differs from the legacy encoding (which the PR deliberately avoids).
tests/test_time_axis.py::TestRoundTrip::test_mixed_variants_sort_chronologically and test_spec_conformance.py::TestTemporalDeclaration::test_stored_words_match_the_golden both assert np.sort(words) == words on data with no spans, so neither catches this.
There was a problem hiding this comment.
🤖 from Claude
Fixed the claim, not the ordering — resolution (1) — in 5805922c. Your counterexample is reproduced verbatim by the new test; the disagreement is unbounded and the old sentence's "one start quantum" was simply wrong.
Taking the writer-side option (2) would have bought a self-sorted axis at the cost of the cross-encoding row-assignment stability this PR built deliberately, and that property is the more load-bearing one (a leaf's slab rows must not move when the encoding changes). So the axis stays as-is and §8.1 now says what it actually does. The bullet is split in two, because the two facts were being conflated:
- Row order is the acquisition-group order: timesteps are ordered by the group's earliest member observation time — the order the time axis has always had, the order the
(time, cells)slabs were indexed by, and identical under both encodings, so a row assignment never drifts with the encoding.- Stored word order is not that key, and a reader MUST NOT assume the stored words ascend. Unsigned word order is order by the encoded start, which is the conservative envelope start — and an envelope may begin before its group's earliest member observation time, by an amount nothing in this section bounds. Where it does, the word leads the row key and the stored axis is materially out of order:
np.sort(words)yields ascending envelope-start order, which is not guaranteed to equal row order. In particular a reader MUST NOT bisect the stored axis to resolve a time window; use the overlap predicate below, which is correct regardless of stored order.
The explicit bisect prohibition is there because you're right that it's the natural reader implementation and the failure is silent.
Tests. Both existing assertions are kept, and the divergence is now pinned in two places:
tests/test_raster_pipeline.py::TestTocTimeIndex::test_a_leading_span_puts_the_stored_words_out_of_row_order— your counterexample through the production function:dt-2'sstart_datetime(16:00:59) precedesdt-1's item datetime (16:02:20), an ~81 s lead. It asserts both halves at once —index == raster_time_index(granules)[0] == {"dt-1": 0, "dt-2": 1}(row assignment stable) andwords[1] < words[0]/not np.array_equal(np.sort(words), words)(stored axis unsorted).tests/test_time_axis.py::test_a_leading_envelope_start_breaks_stored_word_order— the same divergence at the encode layer.
test_mixed_variants_sort_chronologically is renamed test_mixed_variants_sort_by_envelope_start — its data ascends by envelope start, which is what unsigned word order actually is, so the test is fine once it stops claiming to be about acquisition order.
The conformance assertion at test_stored_words_match_the_golden stays, with the reason it holds spelled out rather than implied:
# These particular words ascend, but that is INCIDENTAL to this
# fixture: its three groups are days apart, so no envelope start can
# lead the row before it. §8.1 does NOT promise ascending stored
# words -- the span-lead counterexample is pinned by
# test_raster_pipeline.py::TestTocTimeIndex::
# test_a_leading_span_puts_the_stored_words_out_of_row_order.(Worth noting the fixture does carry a span lead — dt-2's envelope starts at 15:06:38 vs dt-1's 15:06:40 — it just cannot reorder anything, because the groups are three days apart. Which is exactly why it never caught this.)
Also corrected the same overclaim at its source: raster_time_index's docstring said "the axis is ascending acquisition order under both encodings" and returned "both in ascending time order". It now says row order is the earliest item datetime, that this is not ascending stored words under toc, and points at the overlap predicate.
| than a parallel one; a reader MUST refuse a `shape` it does not implement. | ||
| - **`epoch`** / **`timescale`** — the word grammar's fixed time origin and | ||
| time scale. They are properties of the grammar, not writer choices; a | ||
| reader MUST refuse a declaration whose values differ from the grammar it |
There was a problem hiding this comment.
🤖 from Claude (review)
The block ships the four per-store guards the #410 plan explicitly ruled out, and omits the one field it ruled in — and it is committed fixture bytes. The correction comment closing the #410 plan pass (#410 comment, 2026-08-17T00:35Z) says:
The declaration design also simplifies per espg's question: toc pins its epoch and quantization in the word grammar itself, so the zagg declaration needs only {shape, versioned grammar citation} — no per-store epoch/quantization guards.
What landed is {spec, shape, epoch, timescale, quantum_start_ns, quantum_end_ns}: epoch/timescale are a per-store guard (this section MUST-refuses a mismatch, and temporal_declaration raises), and the two quanta are per-store echoes of grammar constants. There is no versioned grammar citation in the block — spec: "zagg-toc/1" versions zagg's section, not the mortie grammar it defers to, and the mortie version appears nowhere in the stamped attrs (see my other comment: the prose citation is currently unversioned and 404).
The PR body presents this as settled ("echoed for reader self-description … and MUST-refused on mismatch, not re-specified") but does not surface it as a divergence from the plan, so it is not covered by "Questions for review" (5) which is only about the shape token.
Concretely, the divergence has teeth in both directions:
- The guards are self-fulfilling. A writer stamps these from
mortie.toc's own constants, and a reader refuses anything else. So they can only ever fail if a store was written by a different grammar — which thespecmarker already gates. The cost is not zero: they are in the fixture bytes and in the O11 hash (6/timeattrs are part of the leaf), so a future mortie that changed a quantum would force a §8 revision and a fixture regeneration that thespecmarker alone would have handled. - The omission is the load-bearing half. With the grammar deferred to a non-frozen API page, a
mortie_version(or agrammar: "mortie/toc@0.9.6") in the block is what would let moczarr pin what it decoded against. That is what "versioned grammar citation" appears to mean, and it is the field that is missing.
Not asking for a redesign on my say-so — but this is a spec-owned block going into committed fixture bytes right before #410's other shapes reuse it, so the divergence should be either reconciled or recorded as a deliberate overrule under "Questions for review".
There was a problem hiding this comment.
🤖 from Claude
Reconciled to the ruled shape, not recorded as an overrule — 97c69fd0. You read the correction comment correctly and the block had diverged from it in both directions.
The declaration is now exactly {spec, shape, versioned grammar citation}:
"temporal": {
"spec": "zagg-toc/1",
"shape": "axis",
"grammar": "mortie/toc@0.9.6"
}The four guard keys (epoch, timescale, quantum_start_ns, quantum_end_ns) are gone from the normative grammar, from the stamped attrs (time_axis_attrs), and from the read-side check (temporal_declaration no longer compares epoch/timescale — it strict-checks spec, shape, grammar). TOC_TIMESCALE is deleted; TOC_EPOCH survives only as the internal guard that refuses a time the words cannot represent, and is documented as such:
#: The cited grammar's time origin. A grammar property, not a declared key
#: (#410 ruled out per-store epoch/quantization guards); kept here only to
#: refuse a time the words cannot represent.On the missing half. grammar is a fixed token of this spec revision, deliberately not mortie.__version__ at write time — a dynamic stamp would move every store's bytes (and the §5 hash) whenever a floor moved, reintroducing the exact cost you flagged. It cites the release whose reference §8's prose pins, and mortie#193's frozen section becomes the referent when it lands. A reader MUST refuse a grammar it does not implement.
The removal is justified in the spec rather than left as a silent deletion:
The epoch, the timescale, and the range variant's rounding quanta are properties of the cited grammar and are deliberately NOT echoed here. A store cannot re-base them — a differing origin would be a different grammar, which is precisely what
grammar(andspecabove it) already discriminates — so a per-store copy could only ever restate a constant, while entering the committed conformance bytes and the §5 content hash and forcing a fixture regeneration for any upstream constant that changed.
I also added the informative-keys escape hatch the ruling implies, since dropping the guards otherwise leaves no room for source-time lineage later: a writer MAY add non-normative keys, and a reader MUST ignore unrecognized ones rather than refuse the store. Pinned by test_informative_keys_are_ignored_not_refused.
Fixture regeneration. --only raster_toc regenerated; the diff is attrs-only, three keys removed and one added, in two files:
tests/data/spec/raster_toc/1/1/2/1/3/11213.zarr/6/time/zarr.json— thetemporalblock;tests/data/spec/raster_toc.expected.json— itstime_attrsmirror.
Plus the two informative written_at/generated_at stamps §7 already declares non-asserted.
No content hash moved — FROZEN_COMBINED["raster_toc"] and FROZEN_ARRAYS[("raster_toc", …)] are untouched, and TestTemporalDeclaration::test_frozen_digests_pin_the_recipe passes on the regenerated bytes. So the §5 recipe digests array data, not array attrs: the "they are in the O11 hash" half of your cost argument turns out not to hold. The "they are in the committed fixture bytes" half does, and stands on its own.
Test side: test_declaration_grammar_and_dtype now asserts the whole block by equality (== {"spec", "shape", "grammar"}) rather than key-by-key, so a re-added guard key fails immediately instead of passing unnoticed; test_rebased_epoch_refused becomes test_uncited_grammar_refused; test_toc_stamps_the_declaration_and_no_cf_attrs likewise asserts the exact block, with a new test_the_cited_grammar_is_the_one_this_reader_decodes_with keeping the fixed citation honest against the constants the decode actually runs on.
Disclosed in the PR body under a new "Fixture diff" heading, and the divergence-from-plan item is dropped from "Questions for review" — it is no longer a divergence.
There was a problem hiding this comment.
🤖 from Claude
Follow-up on this thread, since the block you reshaped is the block that just moved again: espg ruled both token spellings in session on 2026-08-17, folded in b932fa11.
shape: "axis"→"coordinate"— the raster time array is the CF/xarray coordinate variable of the time dimension, and the downstream consumers (xdggs, moczarr's DataTree model) live in that vocabulary.grammar: "mortie/toc@0.9.6"→"mortie-toc/1"— a grammar revision token in the ecosystem's{name}/{major}style (zagg-ragged/1,morton-hive/2), so store bytes never move when documentation moves. This is the durable answer to your original finding here: the mike-versioned API URL is now explicitly §8's prose documentation pointer (it stays release-pinned, and swaps to mortie's frozen spec section when Spec: normative section for the toc word grammar (frozen-for-1.x) espg/mortie#193 lands), while the stored token stays put across both.
The same ruling generalized §8's preamble to the domain-neutral form: {spec, shape, grammar} is defined once as a word-typed coordinate declaration with shape vocabulary {"coordinate", "per-cell", "per-centroid"}, then instantiated here for the temporal domain, with #410's temporal companions and the located mortie-morton/1 family named as non-normative forward instantiations (and the store's primary morton axis explicitly out of scope — it stays on the morton-hive/{1,2} manifest grammar + dggs attrs).
Fixture regenerated --only raster_toc: the entire diff is those two attrs lines in 6/time/zarr.json and their mirror in raster_toc.expected.json. No content hash and no frozen literal moved (§5 digests array data, not attrs). Disclosed in the PR body under "Regenerated for the ruled tokens", and body question (5) is marked resolved.
| ], | ||
| # The REAL acquisition envelopes the stored words must contain — the | ||
| # §8 conservative-containment claim, pinned on committed bytes. | ||
| "acquisitions": [ |
There was a problem hiding this comment.
🤖 from Claude (review)
The acquisitions block — the thing that makes §8.1's containment claim assertable — is hand-transcribed and not derived from RASTER_GRANULES, so it cannot detect the drift it exists to prevent. Everything else in expected is read back from the written store (words, attrs, lo/hi, morton, bands, hashes). These three literals are typed in beside it.
They are correct today — I checked each against RASTER_GRANULES above: dt-1 = min/max of its two member datetimes (15:06:40 / 15:06:47), dt-2 = its time_start/time_end (15:06:38 / 15:06:49), dt-3 = its single instant. The failure mode is later: edit a granule's datetime or add a time_start, regenerate, and time_words/time_bounds_ns move while acquisitions does not. TestTemporalDeclaration::test_bounds_conservatively_contain_the_real_acquisitions then asserts containment of an interval that is no longer the fixture's real acquisition — and because outward rounding gives it seconds of slack on each side, it will very likely still pass. The one assertion in §7 that pins honesty rather than bytes is the one with no tie back to its inputs.
Cheap fix, same shape as the rest of the builder — derive it from the same grouping the encoder used:
groups: dict = {}
for g in RASTER_GRANULES:
lo = g.get("time_start", g["datetime"])
hi = g.get("time_end", g["datetime"])
k = g["time_key"]
groups[k] = (min(groups[k][0], lo), max(groups[k][1], hi)) if k in groups else (lo, hi)
"acquisitions": [
{"key": k, "start": lo[:-6], "end": hi[:-6]}
for k, (lo, hi) in sorted(groups.items(), key=lambda kv: kv[1][0])
],I confirmed this regenerates byte-identically otherwise: --only raster_toc into a temp dir reproduces every chunk object and raster_toc.expected.json exactly, with only the two informative written_at/generated_at stamps differing (which §7 already declares non-conformance-asserted).
There was a problem hiding this comment.
🤖 from Claude
Taken as written — 293ea6cb. Your diagnosis is exact: it was the one assertion in §7 pinning honesty rather than bytes, and it had no tie back to its inputs.
Lifted to a named helper next to RASTER_GRANULES rather than inlined, since it needs the row-order key too (the group's earliest item datetime, which is not the envelope start — see the span-lead thread above, so sorted(..., key=envelope_start) would have produced the wrong order for a fixture with a leading span):
def _raster_acquisitions() -> list[dict]:
span: dict[str, tuple[str, str, str]] = {}
for g in RASTER_GRANULES:
key = g["time_key"]
lo, hi = g.get("time_start", g["datetime"]), g.get("time_end", g["datetime"])
if key in span:
was_lo, was_hi, was_dt = span[key]
span[key] = (min(was_lo, lo), max(was_hi, hi), min(was_dt, g["datetime"]))
else:
span[key] = (lo, hi, g["datetime"])
return [
{"key": key, "start": lo[:-6], "end": hi[:-6]}
for key, (lo, hi, _dt) in sorted(span.items(), key=lambda kv: (kv[1][2], kv[0]))
]with the reason at the callsite so it does not get re-transcribed later:
DERIVED from RASTER_GRANULES, never transcribed: a hand-typed copy would keep passing after an input edit moved the words, which is the one drift this block exists to catch.
Byte-identity confirmed independently of your run, and after the declaration reshape in 97c69fd0: --only raster_toc --out <tmp> against the committed tree gives an empty diff -r except the two informative written_at/generated_at stamps, and raster_toc.expected.json compares equal key-for-key — acquisitions included, so the derived values reproduce the three literals exactly.
| # apart, so the honest coordinate value is a conservative RANGE -- which | ||
| # int64 microseconds cannot state and a toc word can. Output-defining: a | ||
| # store born on toc is a different product from a legacy-axis one. | ||
| time_encoding: toc |
There was a problem hiding this comment.
🤖 from Claude (review)
This flip makes docs/hive_layout.md state the wrong thing about the default raster leaf. That page's "Raster hive stores (issue #247)" section, line 908, describes every raster hive leaf unconditionally:
each carrying leaf-local
time(int64 microseconds, CF attrs) andmorton(packed u64 words) as the sole cell coordinate
After this line, the shipped S2 config — the only raster config zagg packages, and the production HEALPix raster path — writes that array as uint64 toc words with no CF attrs (time_axis_attrs("toc") returns {"temporal": {...}} and nothing else; TestTocTimeIndex::test_template_declares_and_round_trips asserts "units" not in tarr.attrs). So the narrative doc now contradicts the default for the one product it is describing.
One clause fixes it — e.g. "leaf-local time (int64 microseconds with CF attrs, or uint64 toc words carrying the §8 temporal declaration under output.time_encoding: toc)". §4 puts narrative markdown on the same footing as docstrings and the spec, and docs/specification.md was updated in this PR while this page was not.
Same class, lower stakes, in the wire-payload docstrings: deployment/aws/lambda_handler.py:83 ("times_us": [int, ...] (raster only, issue #264) -- the catalog-derived…) and src/zagg/runner.py:5060 ("times_us is the catalog-derived global time coordinate (int64 μs since the epoch), JSON-safe as a plain int list") both still declare the payload as int64 µs. The values are still JSON-safe plain ints so nothing breaks — [int(t) for t in times_us] on a uint64 array yields Python ints of arbitrary precision — but the sizing note two lines below (int64 μs stamps serialize at ~17 B each → ~350K-timestep headroom) is computed off the wrong width; a toc word is 20 decimal digits.
There was a problem hiding this comment.
🤖 from Claude
All three fixed in 2aad8790.
docs/hive_layout.md:908 — the unconditional claim now covers both encodings, and names the config that flips it, since that's the part that made the page wrong:
each carrying leaf-local
time(int64 microseconds with CF attrs by default, oruint64mortie toc words carrying the spec §8temporaldeclaration and no CF attrs underoutput.time_encoding: toc— which the shipped Sentinel-2 config sets, issue #443) andmorton(packed u64 words) as the sole cell coordinate
deployment/aws/lambda_handler.py:83 — the payload docstring. Added the point that the key name is not a width claim, because that is what will mislead the next reader:
"times_us": [int, ...] (raster only, issue #264) -- the catalog-derived time coordinate, in whatever encoding "config" declares: int64 microseconds since the Unix epoch by default, or uint64 mortie toc words under output.time_encoding: toc (spec §8, issue #443). Plain ints either way; the worker re-derives the dtype from the config, so the key name is historical and NOT a width claim.
Verified that last clause rather than asserting it: emit_raster_template does times_us = np.asarray(times_us, dtype=time_axis_dtype(time_encoding(config))), so the wire really does carry untyped ints and the dtype really is re-derived worker-side.
src/zagg/runner.py:5060 — same correction on _invoke_raster_setup's docstring, plus the sizing note two lines below, which you're right was computed off the wrong width:
the widest stamp is a 20-digit uint64 toc word at ~21 B → ~285K-timestep headroom (int64 μs stamps are ~17 B / ~350K), both orders above any real catalog (the pinned NEON benchmark catalog is 85 items).
The conclusion survives the recomputation — 285K is still orders above the 85-item pinned catalog — but it is now the number for the encoding this PR makes default.
| # The §8 temporal fixture (issue #443) — asserted by | ||
| # TestTemporalDeclaration (raster_toc is not in FIXTURES: it carries no | ||
| # ragged array at all, so nothing the leaf-shaped suite asserts applies). | ||
| "raster_toc": "42263e046ecf4d71de8460063b38e6f15522e245cdc2182ce3a0acaf35db7e4e", |
There was a problem hiding this comment.
🤖 from Claude (review)
The branch is behind main, and two of the three failures the PR body calls "pre-existing and unrelated" are already fixed there — so the tested-state claim is stale and this file will conflict on rebase.
$ git merge-base HEAD origin/main
ed8a31ac... # 20+ commits behind
main has since landed #438: 30866c7d re-pin the kitchen_sink spec fixture + content hashes under authalic and 8855bc32 bump the mortie floor to 0.9.8 for the authalic convention. Running the two named tests on each side:
# this branch
FAILED tests/test_content_hash.py::TestWorkerWiring::test_matches_pinned_fixture_hashes
FAILED tests/test_content_hash.py::TestStreamingLeafStaging::test_unsharded_leaf_reproduces_pinned_hashes_from_staged_values
# origin/main (9538d244), same interpreter, same mortie 0.9.8
2 passed, 36 deselected
So they are not #441's open pinned-hash pair any more — post-rebase the PR should be green on both, and the "How it was tested" section should say so rather than carry them as known-failing.
Two concrete consequences beyond the bookkeeping:
pyproject.tomldiverges: this branch still saysmortie>=0.9.6,mainsaysmortie>=0.9.8. That is the floor the §8 grammar citation should track (see my comment ondocs/specification.md, where I suggested pinning the doc URL to0.9.6— after rebase the right pin is0.9.8; both resolve,0.9.8matches the post-rebase floor).- This file conflicts: Re-pin benchmark shardmap fixtures under the authalic convention (mortie 0.9.8) #438 re-pinned
kitchen_sink's hashes inFROZEN_COMBINED/FROZEN_ARRAYS, the same literals this commit adds an entry to. A textual conflict only — resolve by keeping main'skitchen_sinkvalues and this PR'sraster_tocentry.
I checked the raster_toc fixture itself is not affected by the authalic change and needs no regeneration: #438's fixture diff touches only *_locations chunk bytes (lat/lon → point words) and their derived hashes, and raster_toc carries no locations array — its morton comes from grid.children(shard_key), pure word arithmetic with no latitude in it. I re-ran generate_spec_fixtures.py --only raster_toc under mortie 0.9.8 and got byte-identical chunks and expected.json.
There was a problem hiding this comment.
🤖 from Claude
Merged before folding anything else — 847fe59b, origin/main (9538d244) into the branch. Your read was right on every point.
The conflict did not materialize. Git auto-merged tests/test_spec_conformance.py take-both: #438's re-pinned kitchen_sink literals and this branch's raster_toc entry sit in the same FROZEN_COMBINED/FROZEN_ARRAYS dicts without overlapping lines. I checked the merged file rather than trusting the clean exit — kitchen_sink carries main's 7ee8ad92… under the authalic comment, raster_toc carries 42263e04…, and flux is untouched.
Post-merge evidence, on the same interpreter (mortie 0.9.8, zarr 3.3.0):
$ pytest tests/test_content_hash.py -q
36 passed, 2 skipped
$ pytest tests/test_spec_conformance.py -q
114 passed
So the two tests the PR body carried as "pre-existing and unrelated" —
TestWorkerWiring::test_matches_pinned_fixture_hashes and
TestStreamingLeafStaging::test_unsharded_leaf_reproduces_pinned_hashes_from_staged_values — now pass, exactly as you predicted. The "How it was tested" section is rewritten accordingly; they are no longer listed as known-failing.
No fixture regeneration was needed for the merge, confirming your analysis: raster_toc carries no *_locations array, its morton comes from grid.children(shard_key) (pure word arithmetic, no latitude), and the full conformance suite passed on the committed bytes under 0.9.8 before I touched anything. The regeneration that did land later (97c69fd0) is from the declaration reshape, not from authalic.
pyproject.toml now carries main's mortie>=0.9.8 floor. On the doc URL: I kept the citation at 0.9.6 rather than moving it to 0.9.8 — since toc.py/toc.rs are byte-identical across the two, the earliest release carrying the words is the more durable citation, and it stays valid the next time the floor moves. Reasoning is written into §8 itself; full detail on the citation thread.
Full suite after the merge and all five folds: 4008 passed, 38 skipped, 2 failed — both failures environmental in this worktree, neither touched by this PR:
TestLambdaImports::test_h5coro_hidefix_available— the local venv still has h5coro-hidefix 0.3.1, below the>=0.3.2floor main raised ine6057012; the test assertsio_statsinSidecarIndex.read_group's signature, which 0.3.1 lacks. A local-env staleness the merge introduced, not a code failure.TestFunctionBuild::test_function_build_succeeds— needs Docker.
ruff check src tests and ruff format --check src tests are clean apart from two pre-existing findings on files this PR does not touch (src/zagg/registry.py N818, tests/data/benchmark/README.md formatting) — both reproduce on origin/main, so flagged rather than fixed.
1 participant
Closes #443. Refs #410 (the temporal series this is the first implementation slot of), #218 (S2 ingest), espg/mortie#175 (the toc word grammar).
What this does
The raster family's
(time, cells)time coordinate can now be mortie toc words — oneuint64per timestep that is either an exact nanosecond timestamp or an outward-rounded conservative range — instead ofint64microseconds. Sentinel-2 is flipped to it (src/zagg/configs/sentinel2_l2a.yaml), because a datatake is honestly a ~seconds-long acquisition whose adjacent MGRS tiles are sensed seconds apart, whichdatetime64/int64-µs cannot state and a toc range can.The encoding is declared, not inferred: a spec-owned
temporalattrs block on the time array, with absent ⇒ legacy so every store written before this PR reads exactly as before.Approach
Declaration (spec §8, new).
docs/specification.mdgains## 8. zagg-toc/1. The grammar mirrors §2.0'sweightspattern (PR #431): a strict-checkedspecmarker, an absent-key default that keeps old stores conformant verbatim, a reader MUST-refuse on unknown values, and a composition-legality rule. The block is:Three keys and no more, which is exactly the shape #410 ruled (correction comment:
{shape, grammar}, no per-store epoch/quantization guards). An earlier revision of this branch also stampedepoch/timescale/quantum_start_ns/quantum_end_ns; the adversarial review caught the divergence and they are gone — they are properties of the cited grammar, so a per-store copy could only restate a constant while entering committed fixture bytes.Both token spellings are espg's rulings (in-session, 2026-08-17), replacing the placeholders this branch carried — see the resolved question (5) below and the fixture-diff disclosure.
shape: "coordinate"because the raster time array is the CF/xarray coordinate variable of the time dimension, and downstream consumers (xdggs, moczarr's DataTree model) live in that vocabulary.grammar: "mortie-toc/1"because a grammar revision in the ecosystem's{name}/{major}style (zagg-ragged/1,morton-hive/2) keeps store bytes still when documentation moves — the earlier"mortie/toc@0.9.6"welded a docs URL's release pin into every store's committed bytes.§8's preamble is now the domain-neutral form (espg-ruled design). The section opens by defining a word-typed coordinate declaration — the
{spec, shape, grammar}pattern for any array whose elements are packed words rather than self-describing scalars — with theshapevocabulary{"coordinate", "per-cell", "per-centroid"}defined once, domain-neutrally, and then instantiates it for the temporal domain (attrs keytemporal,spec: "zagg-toc/1",grammar: "mortie-toc/1"). Two forward instantiations are stated as non-normative notes: #410's per-cell/per-centroid temporal companions (samespec, same grammar, furthershapevalues), and the located (spatial) companion family declaring under the same pattern withgrammar: "mortie-morton/1"in the #410 kernel PR. That note also draws the boundary explicitly: the store's primary morton axis stays declared by themorton-hive/{1,2}manifest grammar and the DGGS-conventiondggsattrs — this pattern covers word-typed coordinate/companion arrays beyond that primary surface and never a second, competing declaration of it.The word grammar itself is not restated: §8 cites
mortie.tocat 0.9.6 + espg/mortie#175. That URL is now explicitly the documentation pointer in prose, not the stored token: it stays release-pinned because mortie's docs aremike-versioned and the unversioned path 404s, but re-pointing it moves no store byte. §8 still states plainly that it is not yet the same class of reference as §2.2's — §2.2 defers to mortie's frozen spec §1/§4, whereas mortie's frozen spec carries no toc section at all. espg/mortie#193 tracks adding one; when it lands the pointer swaps to it and the storedgrammartoken does not move.One module owns both halves. New
src/zagg/time_axis.py(~230 lines): declaration constants +temporal_declaration()strict-check,time_axis_dtype/time_axis_attrsfor the writer,encode_time_axis,decode_time_axis,time_axis_overlaps,read_time_axis. It is a new module rather than more ofsrc/zagg/grids/base.py(where #431 putWEIGHTS_ATTR) for two reasons:grids/base.pyowns the aggregation path's array specs and the raster path deliberately bypasses it (processing/raster.py:827-833), andprocessing/raster.pyis already 1,495 lines — the §4 cap says don't grow it. Flagged under "Questions for review" in case you'd rather the constants live besideweights.Write path.
raster_time_index(granules, *, encoding=...)(src/zagg/processing/raster.py) now also accumulates each acquisition group's envelope — earliesttime_start(or itemdatetime) to latesttime_end(or itemdatetime) across its members. Undertoca degenerate envelope encodes withtime2toc(exact instant, never widened) and a real interval withspan2toc(conservative range, never narrowed). Row order is still the group's earliest item datetime under both encodings, so a leaf's row assignment cannot drift with the encoding.That is not the same as an ascending stored axis, and the review caught the PR overclaiming it. A word encodes the envelope start, which a declared
start_datetimecan push before an earlier row's key — by an amount nothing bounds (the review demonstrated ~62 s). Sonp.sort(words)gives ascending envelope-start order, which is not guaranteed to equal row order; §8.1 now says so and explicitly forbids bisecting the stored axis. Window selection was already correct — it runs the overlap predicate, not a bisect. Pinned byTestTocTimeIndex::test_a_leading_span_puts_the_stored_words_out_of_row_order._raster_memberstakes dtype + attrs from the declaration, so both the flat template and the hive leaf template are covered by one change; the legacy branch is byte-identical to today (_TIME_ATTRSmoved verbatim intoLEGACY_TIME_ATTRS).Read path. There was no reader for the raster time coordinate at all —
readers/tdigest_tensor.py:297-301refuses any 2-D array andclient.py:493refuses raster outright, so users reach forxr.open_zarrthemselves (D13,docs/design/generalized_grids.md:425). So this adds the minimum:read_time_axis/decode_time_axis/time_axis_overlaps, re-exported fromzagg.readers.Decode returns
(start, end), not a midpoint — justified two ways. (1) The existing reader surface returns numpy tuples (read_tensorsyields(tensor, mask, (offset, gain), morton_index)), so a pair is the idiomatic shape and needs no new container. (2) The midpoint of a conservative envelope is not an observation — presenting it as the acquisition time would invent precision the encoder deliberately refused to claim. §8.1 says a reader that must show one instant SHOULD showstart. Window selection runstoc_overlapson the rawuint64words with no decode, which is conservative in the honest direction (over-reports ≤ one quantum at an edge, never under-reports).Semantic hash.
output.time_encoding: tocis folded intosemantic_core, keyed only when non-default — the identical treatmentweights: "flux"gets, and for the identical reason ("the stored axis MEANS something else"). Because it is keyed only when non-default, every config written before this PR hashes byte-identically to today;tests/test_semantics.py::TestTimeEncodingHashing::test_default_encoding_hashes_as_absentpins that. This was the one place the change could have cascaded into a semantic-hash question, and the absent-key-default normalization dissolves it rather than deferring it — but it is called out under "Questions for review" since it is your call whether the raster axis encoding belongs in D19's core at all.Phases
93a25b5a)b98dc94c)zagg.readers(c66fb020)raster_toc/conformance fixture + generator + conformance tests (f408907b)origin/mainmerged in (847fe59b), then one commit per finding:03786bc2— version-pin the toc grammar citation and drop the false §2.2 precedent5805922c— state the real time-axis order key and pin the span-lead divergence97c69fd0— reshape the temporal declaration to the ruled{shape, grammar}(fixture regenerated)293ea6cb— derive the fixtureacquisitionsfromRASTER_GRANULES2aad8790— make the time-axis docs conditional on the declared encodingb932fa11—shape: "axis"→"coordinate",grammar: "mortie/toc@0.9.6"→"mortie-toc/1", and §8's preamble generalized to the domain-neutral word-typed coordinate declaration (fixture regenerated)Merge note (
847fe59b)The branch was 38 commits behind
main;origin/main(9538d244) is merged in rather than rebased.tests/test_spec_conformance.pyauto-merged take-both — #438's re-pinnedkitchen_sinkliterals and this branch'sraster_tocentry live in the sameFROZEN_COMBINED/FROZEN_ARRAYSdicts without overlapping lines — and no fixture needed regenerating for the merge:raster_toccarries no*_locationsarray, so #438's authalic change cannot reach it, and the full conformance suite passed on the committed bytes under mortie 0.9.8 before any fold landed.pyproject.tomlnow carries main'smortie>=0.9.8floor.The two
test_content_hash.pyfailures this body previously listed as "pre-existing and unrelated" were fixed by #438 onmainand now pass; "How it was tested" below is the post-merge, post-fold state.Fixture diff — the blast radius, precisely
Per the #340 rule this is a wire-format change, so fixtures move in the same PR. The blast radius is exactly one new fixture; nothing committed before this PR changed by a single byte.
git show --stat f408907bis:tests/data/spec/raster_toc/— 38 files: one raster hive leaf (1/1/2/1/3/11213.zarr) withtime(3 toc words, 1 chunk),morton(4 chunks),red/scl(3, 16)bands (4 chunks each × 3 timesteps),coverage.moc,granules.json, plusmorton_hive.json.tests/data/spec/raster_toc.expected.jsontools/generate_spec_fixtures.py(theraster_tocbuilder + docstring) andtests/test_spec_conformance.py(TestTemporalDeclaration+ two frozen literals).minimal/,kitchen_sink/,flux/,column/,pyramid/and all four of their*.expected.json— regenerated with--only raster_toc, so the "stale by design" manifests the generator docstring warns about were not refreshed.FROZEN_COMBINED/FROZEN_ARRAYSgained two entries and no existing literal moved.git statusafter generation listed only the two new paths.Regenerated in the review fold (
97c69fd0)The declaration reshape changed stamped attrs, so
--only raster_tocwas re-run. The diff is attrs-only — three keys removed, one added — in two files:tests/data/spec/raster_toc/1/1/2/1/3/11213.zarr/6/time/zarr.json— thetemporalblock:epoch,timescale,quantum_start_ns,quantum_end_nsremoved;grammar: "mortie/toc@0.9.6"added.tests/data/spec/raster_toc.expected.json— itstime_attrsmirror, same four-out/one-in.Plus the two informative
written_at/generated_atstamps that §7 already declares non-conformance-asserted.No chunk object changed and no content hash moved.
time_words,time_bounds_ns,acquisitions,morton,bandsand everycontent_hashesentry are byte-identical;FROZEN_COMBINED["raster_toc"](42263e04…) and theFROZEN_ARRAYSentry are untouched literals, andTestTemporalDeclaration::test_frozen_digests_pin_the_recipepasses on the regenerated bytes. The §5 recipe digests array data, not array attrs, so the declaration lives in the committed bytes but not in the O11 hash. No other fixture was regenerated —minimal/,kitchen_sink/,flux/,column/,pyramid/are still untouched by this PR.Regenerated for the ruled tokens (
b932fa11)The two ruled token spellings are stamped attrs, so
--only raster_tocwas re-run into a temp dir and diffed against the branch's committed state. The whole diff is four lines in two files (diff -rqover the fixture tree plus ajson.tooldiff ofraster_toc.expected.json):tests/data/spec/raster_toc/1/1/2/1/3/11213.zarr/6/time/zarr.json—"shape": "axis"→"coordinate","grammar": "mortie/toc@0.9.6"→"mortie-toc/1".tests/data/spec/raster_toc.expected.json— itstime_attrsmirror, the same two lines.The only other files
diff -rqflagged are the two informative stamps §7 already declares non-conformance-asserted (.../11213.zarr/zarr.json'swritten_at,morton_hive.json'sgenerated_at), so those two were left at their committed values and the token change was applied in place — the committed fixture is now byte-identical to the regenerated one apart from those stamps (verified:json.tooldiff of both changed files against the regen output is empty).No hashed literal moved.
time_words,time_bounds_ns,acquisitions,morton,bands, everycontent_hashesentry,FROZEN_COMBINED["raster_toc"](42263e04…) andFROZEN_ARRAYS[("raster_toc", "6/time")](551f7be5…) are all untouched — the §5 recipe digests array data, not array attrs, so a declaration reword lives in the committed bytes but not in the O11 hash.TestTemporalDeclaration::test_frozen_digests_pin_the_recipepasses on the reworded bytes. No other fixture was regenerated;minimal/,kitchen_sink/,flux/,column/,pyramid/remain untouched by this PR.Separately,
293ea6cbreplaced the hand-transcribedacquisitionsliterals with a derivation fromRASTER_GRANULES; verified byte-identical by regenerating into a temp dir and diffing (empty except the two informative stamps), so the derived values reproduce the three literals exactly.What the fixture pins
The new fixture is deliberate about what it pins: its three timesteps commit both word variants — two multi-member groups become RANGE words (one derived from member instants 7 s apart, one from a STAC
start_datetime/end_datetimepair) and one single-member group stays an exact TIMESTAMP — so a reader that implements only one variant fails a §7 fixture.raster_toc.expected.jsonrecords the words as decimal strings, the(start, end)ns a conforming decode yields, and the real acquisition spans they must contain, which makes §8.1's conservative-containment claim assertable on committed bytes. The other four fixtures carry notemporalkey anywhere, which is the committed absent-key ⇒ legacy pin.It is written through the production raster hive seam (
process_and_write_raster_hive— leaf template, per-timestep slab streaming, coverage sidecar, commit stamp, O11 hashing) with only the COG sampling faked, mirroring howbuild()fakesprocess_shardfor the digest fixtures. A committed fixture must regenerate with no network and no GDAL, and the pull-NN arithmetic istests/test_raster.py's to pin, not §7's.How it was tested
Post-merge (
847fe59b) and post-fold (throughb932fa11), mortie 0.9.8 / zarr 3.3.0:pytest -q— 4009 passed, 38 skipped, 1 failed. The single failure is environmental in the dev worktree and untouched by this PR:test_lambda_build.py::TestFunctionBuild::test_function_build_succeedsneeds Docker. (TestLambdaImports::test_h5coro_hidefix_available, previously listed here, now passes — the local venv picked up the>=0.3.2floormainraised ine6057012.)pytest tests/test_time_axis.py tests/test_spec_conformance.py tests/test_raster_pipeline.py -q— 232 passed, the targeted gate for the token rulings.test_content_hash.pyfailures previously listed here now pass (36 passed, 2 skipped), fixed by Re-pin benchmark shardmap fixtures under the authalic convention (mortie 0.9.8) #438 onmain.pytest tests/test_spec_conformance.py -q— 114 passed.ruff check src testsandruff format --check src tests— clean apart from two pre-existing findings on files this PR does not touch (src/zagg/registry.py:64N818, a markdown block intests/data/benchmark/README.md), both of which reproduce onorigin/main. Flagged, not fixed, per §4.pre-commit run --all-files: mypy reports 146 pre-existing errors across 29 files, none in any file this PR adds and none at any line it edits; codespell and check-yaml failures are all pre-existing (CloudFormation!Ref-style tags, "GES"/"statics" in untouched modules). Repo-config ruff also reports the pre-existingN818onregistry.py:64, which the CI flag set excludes.New coverage:
tests/test_time_axis.py(declaration grammar and every strict-check including the uncited-grammarrefusal and the informative-keys pass-through, span→word→range containment bounded by one quantum each end, degenerate-envelope-stays-exact, sort-by-envelope-start and the span-lead case that breaks stored word order, empty axis, pre-epoch and inverted-envelope refusals, window predicate, legacy parity);TestTocTimeIndexintest_raster_pipeline.py(group envelopes, the STAC start/end path, the row-order/word-order divergence on the production function, legacy values unmoved when a catalog gains span columns, flat + leaf templates, legacy store still reads);TestTimeAxisReaderintest_readers.py;TestTimeEncodingintest_config.py;TestTimeEncodingHashingintest_semantics.py;TestTemporalDeclarationintest_spec_conformance.py(11 tests over committed bytes).Questions for review
output.time_encodingis now insemantic_core, keyed only when non-default so no existing hash moves (test pins this). The alternative reading is that it belongs under D19's "packaging" — it changes stored bytes but not values in any physical sense. I took theweights: "flux"precedent as decisive; say the word if you'd rather it stay out.src/zagg/time_axis.pyrather thansrc/zagg/grids/base.pybesideWEIGHTS_ATTR, for the module-cap and path-ownership reasons above. Happy to move them if you want one home for spec-owned attrs keys.deployment/aws/lambda_handler.py. The flat-pathsetupbranch cast the wire payload withnp.asarray(event["times_us"], dtype=np.int64), which silently overflows on auint64toc word. It now casts withtime_axis_dtype(time_encoding(config)). That file is worker application code (routinely touched by Unified second-pass rollup sweep: stats, shardmap sub-maps, MOC, overviews #300/Run-record write violates D8 orchestrator-no-write: route via worker invoke like the root MOC #313/zagg.client v2 transport: Event invoke + status-object future resolver #327/Sweep telemetry: per-family duration_s in run_sweep summary, handler response, and a persisted record #353), not an infra/deploy script, but it is underdeployment/aws/so I am flagging it explicitly. The hive path — the production default for HEALPix raster — needed no handler change at all: hive events carry notimes_usand the worker derives its leaf-local axis itself.tests/data/benchmark/configs/s2_neon_o9.yamlstill says it is "identical to the shippedsentinel2_l2a.yamlexceptparent_order: 9", which is now one knob short of true. I did not flip it because doing so moves the raster release leg's product identity mid-series. Tell me which you want and I will make the config and its comment agree.shape: "axis"naming, and thegrammartoken's spelling.b932fa11:shape: "coordinate"(the raster time array is the CF/xarray coordinate variable of the time dimension, which is the vocabulary xdggs and moczarr's DataTree model already speak) andgrammar: "mortie-toc/1"(a grammar revision token in the{name}/{major}style ofzagg-ragged/1/morton-hive/2, so store bytes never move when documentation moves; the mike-versioned API URL lives in §8's prose and swaps to mortie's frozen spec section when Spec: normative section for the toc word grammar (frozen-for-1.x) espg/mortie#193 lands). The same ruling generalized §8's preamble to the domain-neutral word-typed coordinate declaration described above. Fixture blast radius disclosed under "Regenerated for the ruled tokens" — four lines, no hashed literal moved.zagg.clientsurface.client.pyrefuses raster outright, so the time-axis reader lives onzagg.readersonly. Not obviously wrong, but worth a sanity check that this is the surface you want raster consumers reaching for.Not done here (deliberate)
shapevocabulary and grammar-revision slot are planted for them and their forward instantiations are named as non-normative notes, nothing more. Themortie-morton/1located family lands with the Per-centroid temporal companion for t-digests: a 64-bit hierarchical time cell, mirroring the spatial location companion #410 kernel PR.docs/specification.md:1131-1133), and raster hive under a /2 pyramid declaration: leaf columns are never written (rule PR #391 options (a)/(b)) #399 owns their overview regime.