Conversation
`ArrayV3Metadata` validated every codec against the array-level shape and chunk grid, and threaded the *array* spec (not a chunk spec) through `resolve_metadata` during evolution. Both wrongly reject chains in which an earlier array->array codec changes a chunk's shape or rank, e.g. the zarr-extensions `reshape` codec followed by `transpose` with an order of the reshaped rank -- a combination the reshape spec explicitly endorses and that the encode path already handles correctly. Codecs are now evolved and validated in a single threaded pass (`evolve_and_validate_codecs`): each codec sees the chunk spec produced by the previous codec's `resolve_metadata`, exactly as at encode time. The array-level shape/chunk grid are passed to `Codec.validate` unchanged until a codec changes the chunk shape, after which the resolved chunk shape (and a regular grid of it) stands in for them. `ShardingCodec.validate` now validates its inner chain the same way against the inner chunk shape. Assisted-by: ClaudeCode:claude-fable-5
…rectilinear chunk shape Review feedback on the threaded-chunk-spec validation: after a codec changes the chunk shape, validating the rest of the chain against a single representative (max-edge) chunk shape is unsound for rectilinear grids -- an inner shard size that divides the largest chunk need not divide the others. Concretely, transpose over a rectilinear grid followed by sharding falsely accepted an inner chunk shape that only divided the largest transposed chunk. The representative was also used to *detect* shape changes, which could miss changes affecting only non-representative chunks. `evolve_and_validate_codecs` now threads every distinct chunk shape of the grid (the cross product of per-dimension distinct edges, capped at 4096 with a ZarrUserWarning on truncation) through `resolve_metadata`, and validates each one individually once any codec has changed a chunk shape. The representative spec remains the single spec used for codec evolution and dtype tracking. Assisted-by: ClaudeCode:claude-fable-5
…idation Two hypothesis oracles over the threaded-chunk-spec validation: - acceptance implies round-trip: any reshape of a chunk into a valid factorization followed by a transpose of the reshaped rank (with and without sharding) is accepted, encodes/decodes losslessly, and its metadata survives JSON serialization; a transpose order of any other rank is rejected. - transpose-then-shard over a rectilinear grid is accepted exactly when every chunk shape in the grid, transposed, is divisible by the inner shard shape (verified against a brute-force cross-product oracle; this test fails on the max-edge-representative implementation). Assisted-by: ClaudeCode:claude-fable-5
Validate inner codecs during evolution, where the actual fill value is available, and cover fill-changing chains with a round-trip property. Assisted-by: Codex:GPT-6
Review fixes for the threaded codec-chain validation: - Attach a note to any exception raised by a codec's evolve_from_array_spec, validate or resolve_metadata naming the codec's position in the chain, its class and the shape it was checked against. Codec messages talk about "the array", which is misleading once an earlier codec has changed the chunk shape; the exception type and message are unchanged. - Drop the unused `evolve` parameter of evolve_and_validate_codecs left over from the removed ShardingCodec.validate inner-chain check. - Document in ShardingCodec.evolve_from_array_spec why the inner chain is validated there (validate has no fill value) and against which grid. - Docstrings use single backticks; the towncrier fragment is 303.bugfix.md. - Tests: one parametrized happy-path test for rectilinear grids plus one test per error case; the sharding inner-chain test asserts the note. Assisted-by: ClaudeCode:claude-fable-5-1 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…s the transposed chunk Add the `transposed_sharding_chains` strategy: a codec chain with a TransposeCodec (random permutation) ahead of a ShardingCodec in one of three layouts (transpose then shard; nested shard with the transpose between the levels; transpose inside a shard as the always-valid control), together with an oracle for its validity (every edge of the sharding codec's chunk shape divides the transposed edge it applies to). Half the drawn chains are invalid, breaking exactly one axis. The property asserts that create_array accepts a chain exactly when the oracle says it is valid, and that an accepted chain round-trips its data and its persisted metadata. On main before the fix it fails two ways: a nested sharding codec's inner chain is never validated, so an invalid inner chunk shape is accepted and reads back wrong data, and a valid transpose-then-shard chain is rejected because the sharding codec was validated against the untransposed chunk grid. Assisted-by: ClaudeCode:claude-fable-5-1 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…rom chain validation Apply the second-lens review cuts: - Enumerate every distinct rectilinear chunk shape without a cap. The warn-and-continue path validated only a subset of shapes and then accepted the metadata, which is worse than either failing or checking everything. - Drop the exception notes and the try/except blocks around evolve, validate and resolve_metadata; they were not part of the fix. - Move `transposed_sharding_chains` out of the public `zarr.testing.strategies` module into `tests/test_properties.py`, its only user. - Drop the metadata-only rectilinear oracle property; the create_array plus round-trip property in `test_properties.py` covers the same acceptance oracle for regular and nested sharding. Assisted-by: ClaudeCode:claude-fable-5-1 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Assisted-by: ClaudeCode:claude-fable-5-1 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Documentation build overview
24 files changed ·
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #4352 +/- ##
==========================================
+ Coverage 94.34% 94.35% +0.01%
==========================================
Files 92 92
Lines 12935 12981 +46
==========================================
+ Hits 12203 12248 +45
- Misses 732 733 +1
🚀 New features to boost your workflow:
|
Preserve grid geometry for identity, cast, scale, and transpose codecs. Stream arbitrary resolver chains without retaining the cross product, and test bounded work for common paths. Assisted-by: Codex:GPT-6
d-v-b
marked this pull request as ready for review
September 14, 2026 14:43
Contributor
Author
|
this PR has a performance impact for codec creation with rectilinear chunk grids, where we loop over every chunk shape defined in a grid, checking each one for compatibility with the entire codec chain. I'm working on a mitigation. |
Merging this PR will not alter performance
Comparing 
Footnotes
|
…umerating chunks Validating codec chains on rectilinear grids streamed every combination of per-axis chunk edges through each codec whose `resolve_metadata` was overridden, e.g. numcodecs `delta`. A 3-d grid with 100 distinct edges per axis took ~3.8 s per metadata construction, growing as n**ndim. Follow zarrs: codecs map a whole chunk grid via the new `BaseCodec.resolve_chunk_grid`. Dtype/fill-value codecs declare the identity and transpose declares a permutation, so those chains stay exact. An undeclared codec is exact on regular grids; on rectilinear grids it makes the rest of the chain chunk-local, validated against one representative chunk, with other chunk shapes checked at encode/decode time. ShardingCodec now enforces divisibility at run time, which would otherwise floor-divide and silently corrupt data. Document the trade-off in the codec docstrings, the extending guide, the rectilinear sharding docs and the changelog. Assisted-by: ClaudeCode:claude-opus-5
…chunk shape `create_codec_pipeline` evolved V3 pipelines against an all-ones placeholder chunk spec whenever the grid was not regular, so any codec whose `resolve_metadata` depends on the chunk shape (e.g. a reshape filter) failed at array creation for rectilinear grids even though metadata validation had accepted the chain, and shape-sensitive evolution output (sharding's inner chain, BytesCodec endian) was computed against a meaningless shape. Use `representative_chunk_shape`, the same shape `ArrayV3Metadata.__init__` threads through evolution, so the pipeline carries exactly the codecs the metadata produced. The ChunkTransform itself remains shape-agnostic: it only holds the evolved codecs and resolves specs per call. Assisted-by: ClaudeCode:claude-fable-5-1
Contributor
Author
|
requesting review because this adds to codec API |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This AI-authored PR ensures that codec chains validate against the chunk shape emitted by the preceding codec, not the original chunk grid.
🤖 AI text below 🤖
Codec validation needs the geometry and dtype produced by preceding codecs. In particular, transpose followed by sharding must check divisibility against the transposed chunk sizes. Validating against the original grid can accept chains that read back incorrect data or reject valid chains. The same validation is needed inside sharding.
evolve_and_validate_codecsthreads chunk specs through evolution, validation, and metadata resolution. Inner sharding chains use the actual inner spec, including its fill value.Geometry is threaded as a whole chunk grid
Following zarrs (
encoded_chunk_grid/ chunk-local grids), codecs describe how they map the chunk grid through a newBaseCodec.resolve_chunk_grid(shape=..., chunk_grid=...)method, and validation carries the grid through the chain rather than enumerating chunk shapes. Cost is proportional to the number of codecs and dimensions, independent of how many distinct chunk shapes a rectilinear grid has.resolve_metadatadeclare the identity by default.CastValue,ScaleOffsetand the numcodecsDelta,FixedScaleOffsetandAsTypedeclare the identity explicitly (they change dtype or fill value, never shape).TransposeCodecpermutes the shape and the per-axis edge lists.resolve_metadatawithout the hook, or if it disagrees withresolve_metadataon the representative chunk.Trade-off: chunk-local validation
A codec that overrides
resolve_metadataand returnsNonefromresolve_chunk_grid(the default) is still validated exactly on a regular grid, since every chunk has one shape. On a rectilinear grid it makes the rest of the chain chunk-local: later codecs are validated against one representative chunk (largest edge per axis). Rejections are always correct, but acceptance is incomplete: a chain that is invalid only for some other chunk shape is accepted at creation and fails when such a chunk is first encoded or decoded.To make that failure an error rather than corruption,
ShardingCodec._get_chunks_per_shardnow checks divisibility at run time (previously it floor-divided; with the check removed, a non-dividing shard writes and reads back wrong rows). Codecs whose correctness depends on chunk shape must do the same.The trade-off is documented in the docstrings of
evolve_and_validate_codecs,resolve_chunk_gridand the sharding check, in a new "Chunk geometry and validation" section of the extending guide, in the rectilinear sharding docs, and in the changelog entry.Tests
Accepted and rejected transpose/sharding chains (also behind
Delta), reshape followed by transpose, nested sharding, rectilinear divisibility, actual fill values, data and metadata round trips. Chunk-local behaviour is pinned from both sides: an undeclared filter on a rectilinear grid still rejects a shard shape the representative chunk fails, accepts one that only a smaller chunk fails and then raises on the first write to that shard, and does not enumerate chunks to find a shape-changing codec's error. A table test covers when a declared grid is trusted or ignored. The bounded-call-count regression now includesDeltaand an undeclared filter chains at ranks 4 and 12.Validation: full suite 8806 passed, 1291 skipped, 4 xfailed; pre-commit hooks incl. mypy pass. A 3-d rectilinear grid with 100 distinct edges per axis and
(Delta, Bytes, Zstd)took ~3.8 s per metadata construction with the previous streaming fallback; now ~0 s.Known, separate issue (not fixed here):
create_codec_pipelineevolves the pipeline against a placeholder all-ones chunk shape on rectilinear grids, which breaks any codec whoseresolve_metadatadepends on chunk shape, onmaintoo.Supersedes d-v-b#303.
🤖 Generated with Claude Code