feat(zarr-indexing): factor chunk plans into a columnar GridPartition - #4310
Open
d-v-b wants to merge 10 commits into
Open
feat(zarr-indexing): factor chunk plans into a columnar GridPartition#4310d-v-b wants to merge 10 commits into
d-v-b wants to merge 10 commits into
Conversation
Restricting a transform to a chunk box distributes over output dimensions whenever each output map reads its own input axis, which is every basic and orthogonal selection. Chunk resolution therefore no longer intersects the whole transform with every candidate chunk; it resolves each axis once against its grid into a table (StridedSet / IndexedSet), sorts correlated (vindex) index arrays into chunks once into a JointSet, and derives each ChunkProjection as one row of each table. ChunkPlan.partition() and partition_transform() expose the factored form, so a consumer can read the tables directly instead of materializing an object graph per chunk. The projections a plan yields are unchanged; the general whole-transform walk remains for hand-built diagonals, which have no factored form. Along the way: _intersect_general reuses a precomputed _CorrelatedBlock and accepts survivor positions; checked_affine has identity and dtype-bounded fast paths; ArrayMap._with_affine shares frozen index arrays on translate; IndexDomain._unchecked / IndexTransform._unchecked skip validation for objects derived from an already-valid transform. Assisted-by: ClaudeCode:claude-fable-5-1
Assisted-by: ClaudeCode:claude-fable-5-1
The package and its tests import nothing from zarr; the old comments claimed the chunk-resolution tests needed zarr's ChunkGrid, which stopped being true once the package grew its own grids. The real reason is the shared pinned test toolchain. Assisted-by: ClaudeCode:claude-fable-5-1
…hunk narrative The module docstring described intersecting the whole transform with every candidate chunk as "the algorithm"; that walk is now the fallback for hand-built diagonals only. It now explains the factored form and its three tables, and why they cost the sum of the touched chunks per axis. The visual guide gains a final integrator section, "A plan is a product of per-axis tables", with an executable snippet that reads the StridedSet, IndexedSet and JointSet tables off real plans and checks the plan's projections against the partition's rows. Integration boundaries gains "Reading the tables directly", a consumer that assembles a strided box from the tables with no projection materialized. The API index, landing page and design notes (TensorStore lineage, the performance caveat, and the box/query split) point at the new section. Assisted-by: ClaudeCode:claude-fable-5-1
…w fixes Adversarial review (roborev, a correctness reviewer, a complexity reviewer, and ~24k differential examples against main) of the grid partition. Cuts. The whole-transform walk that remained for hand-built diagonals is gone: it was unreachable for every index-array shape, its key builder was duplicated verbatim in _chunk_keys, and for the one shape it served it produced wrong projections (a three-point diagonal yielded four projections covering six cells, on main too). A DimensionMap diagonal is now rejected with ValueError. With it go the sorted-1-D fast path, the three cell-transform helpers, the block/positions parameters of _intersect_general, the correlated-residual check that admitted a diagonal and then crashed, GridPartition.__getitem__, partition_transform as public API, the object-dtype column fallback (StridedSet.origin is now a position along the request axis, so every column is intp), checked_affine's dtype-bound shortcut (measured at noise; the identity shortcut stays and now accepts bool via np.can_cast, as main did), and StridedSet.chunk_map/cell_map. Fixes. GridPartition.n_rows is an exact integer and len raises OverflowError instead of wrapping to zero; table columns are read-only, so a memoized partition cannot drift under a consumer; the documented table consumer now handles reversed axes, inserted axes and transposed transforms, and the snippet checks all three. Docs. Corrected the diagonal statement everywhere it appeared, the memoized "fresh walk" wording, the "vectorized per axis" claim, and the TensorStore correspondence (its strided sets are per input dimension; it keeps one index array set per connected component). The guide no longer restates the class docstrings. Assisted-by: ClaudeCode:claude-fable-5-1
A zero-stride DimensionMap over a domain wider than np.intp is valid and touches one storage cell; coercing every StridedSet column to intp made it raise OverflowError where main returned one projection. `extent` and `origin` are the two columns measured along the request axis, whose bounds are arbitrary Python ints, so they now fall back to exact-int (object) columns when a value does not fit. Chunk-local columns stay intp. Also corrects the design note that said both affine-diagonal cases raise NotImplementedError: two slice maps sharing an axis now raise ValueError. Assisted-by: ClaudeCode:claude-fable-5-1
github-actions
Bot
added
the
needs release notes
Documentation build overview
10 files changed ·
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #4310 +/- ##
=======================================
Coverage 94.21% 94.21%
=======================================
Files 92 92
Lines 12863 12863
=======================================
Hits 12119 12119
Misses 744 744 🚀 New features to boost your workflow:
|
…d the minimal grid protocol Every varying grid in the partition cases summed exactly to its extent, so the boundary where a chunk's data extent is shorter than its declared size, the rectilinear-specific case, was unpinned; so was a grid without data_size. Both now run through the evaluation oracle for strided, orthogonal and correlated selections. Assisted-by: ClaudeCode:claude-fable-5-1
d-v-b
marked this pull request as ready for review
September 3, 2026 18:22
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This is an update to the data structures used in
zarr-indexingfor relating a selection on a chunked array to a per-chunk plan. We currently generate objects per chunk eagerly, which is inefficient due to object creation overhead. FWIW I investigated pushing this down to rust but crossing the py03 boundary for a ton of tiny objects ends up pretty expensive.a better solution is to model the chunk plan as the result of a product of per-axis plans, which is what this PR adds. the end result restores a lot of performance that had been lost with the eager object creation.
based on a claude-authored PR here: d-v-b#316
Author attestation
TODO
docs/user-guide/*.mdchanges/