feat(zarr-indexing): factor chunk plans into a columnar GridPartition

.github/workflows/codspeed.yml

@@ -32,7 +32,7 @@ jobs:
       with:
         version: '1.16.5'
     - name: Run the benchmarks
-      uses: CodSpeedHQ/action@4296e51e7041e24dadb86d1d6e8b9320d223dbe8 # v5.0.3
+      uses: CodSpeedHQ/action@373d6868929f444bc08d901fd0eb0ad52a8875ea # v5.2.1
       env:
         ZARR_BENCHMARK_CLEAR_CACHE: '1'
       with:

.github/workflows/zarr-indexing.yml

@@ -43,11 +43,10 @@ jobs:
         uses: extractions/setup-just@53165ef7e734c5c07cb06b3c8e7b647c5aa16db3 # v4
       - name: Set up Python ${{ matrix.python-version }}
         run: uv python install ${{ matrix.python-version }}
-      # The transform tests exercise chunk resolution against zarr's ChunkGrid,
-      # so they run against the repo-root environment (which provides `zarr`)
-      # with this package as an editable overlay rather than in package
-      # isolation. The recipes carry that invocation; this step only fixes the
-      # interpreter the matrix asked for.
+      # The suite imports nothing from `zarr`; it runs against the repo-root
+      # environment, with this package as an editable overlay, to share the
+      # parent project's pinned test toolchain. The recipes carry that
+      # invocation; this step only fixes the interpreter the matrix asked for.
       - name: Sync test dependency group
         run: uv sync --project ../.. --group test --python ${{ matrix.python-version }}
       - name: Run pytest

packages/zarr-http-server/changes/4311.doc.md

@@ -0,0 +1,2 @@
+The docs now render the release notes as a page instead of linking to
+`CHANGELOG.md` on GitHub, so each docs version shows its own changelog.

packages/zarr-http-server/docs/index.md

@@ -67,5 +67,5 @@ Reads are all that is enabled by default: `GET` and `HEAD` are served, and
 - [User guide](guide.md) — building apps, running them, byte ranges, CORS,
   writes, notebooks, and Uvicorn configuration
 - [API reference](api/index.md)
-- [Changelog](https://github.com/zarr-developers/zarr-python/blob/main/packages/zarr-http-server/CHANGELOG.md)
+- [Release notes](release-notes.md)
 - [License (MIT)](https://github.com/zarr-developers/zarr-python/blob/main/packages/zarr-http-server/LICENSE.txt)

packages/zarr-http-server/docs/release-notes.md

@@ -0,0 +1,8 @@
+<!-- CHANGELOG.md at the package root is the single source of truth: towncrier
+     writes releases into it, and PyPI and GitHub render it directly. This page
+     includes it so each docs build shows the release notes for its own version.
+     The page is not named changelog.md on purpose: on a case-insensitive
+     filesystem that name shadows CHANGELOG.md and the include resolves to the
+     page itself. -->
+
+--8<-- "CHANGELOG.md"

packages/zarr-http-server/mkdocs.yml

@@ -16,7 +16,7 @@ nav:
   - guide.md
   - API Reference:
       - api/index.md
-  - Changelog: https://github.com/zarr-developers/zarr-python/blob/main/packages/zarr-http-server/CHANGELOG.md
+  - Release notes: release-notes.md
 
 watch:
   - src
@@ -98,3 +98,11 @@ markdown_extensions:
       line_spans: __span
       pygments_lang_class: true
   - pymdownx.inlinehilite
+  - pymdownx.snippets:
+      # The package root lets docs/release-notes.md include the
+      # towncrier-managed CHANGELOG.md without moving it out of the package
+      # root, where PyPI and GitHub readers expect it.
+      base_path: [docs, .]
+      # Fail the build on an unresolvable include instead of silently
+      # rendering nothing.
+      check_paths: true

packages/zarr-indexing/README.md

@@ -25,6 +25,9 @@ Key types:
 - `ChunkPlan` and `ChunkProjection` — lazily partition a selection over a
   caller-selected grid and pair each chunk-local transform with its placement in
   the request, without binding a storage backend or scheduler
+- `GridPartition` — the plan's factored, columnar form: one table per axis
+  (`StridedSet`, `IndexedSet`) plus a `JointSet` for correlated index arrays,
+  from which projections are derived on demand
 - `ConstantMap`, `DimensionMap`, `ArrayMap` — the three ways a single output
   dimension can depend on the input
 - `compose` — chain two transforms into one

packages/zarr-indexing/changes/4310.feature.md

@@ -0,0 +1,11 @@
+Chunk plans now have a factored, columnar form. `ChunkPlan.partition()`
+returns a `GridPartition`: one `StridedSet` or
+`IndexedSet` table per output dimension the transform reads independently,
+plus a `JointSet` for correlated (`vindex`) index arrays. A `ChunkProjection`
+is one row of each table and is derived on demand, so planning costs the sum
+of the touched chunks per axis rather than their product, and a vectorized
+consumer can read the tables without materializing a projection per chunk.
+The projections a plan yields are unchanged.
+A hand-built transform in which two output maps read one input axis (a
+diagonal, which no selection produces) is now rejected with `ValueError`
+instead of being walked by whole-transform intersection.

packages/zarr-indexing/changes/4311.doc.md

@@ -0,0 +1,2 @@
+The docs now render the release notes as a page instead of linking to
+`CHANGELOG.md` on GitHub, so each docs version shows its own changelog.

packages/zarr-indexing/docs/api/index.md

@@ -11,9 +11,9 @@ links:
   [Lazy views compose](../guide/index.md#lazy-views-compose),
   then open [`zarr_indexing.lazy_array`](lazy_array.md) for `LazyArray`.
 - **Integrate a chunked source:** finish
-  [One cell domain, two projections](../guide/index.md#one-cell-domain-two-projections),
+  [A plan is a product of per-axis tables](../guide/index.md#a-plan-is-a-product-of-per-axis-tables),
   then open [`zarr_indexing.chunk_resolution`](chunk_resolution.md) for
-  `plan_chunks`. Start with
+  `plan_chunks` and `GridPartition`. Start with
   [Coordinates are addresses](../guide/index.md#coordinates-are-addresses) if
   literal coordinates are unfamiliar.
 
@@ -36,7 +36,10 @@ and the wire format built on top of it.
 
 - [`zarr_indexing.chunk_resolution`](chunk_resolution.md) —
   `plan_chunks`, which lazily projects a request through a caller-selected grid,
-  plus the reusable `ChunkPlan` and paired-transform `ChunkProjection` values
+  the reusable `ChunkPlan` and paired-transform `ChunkProjection` values, and
+  the plan's factored form: `GridPartition` (from `ChunkPlan.partition`),
+  holding one `StridedSet` or `IndexedSet` table per axis and a `JointSet`
+  for correlated index arrays
 - [`zarr_indexing.grid`](grid.md) — `DimensionGridLike`, the Protocol
   describing the narrow chunk-grid surface chunk resolution consumes, so that
   nothing here imports `zarr`, plus `EdgeDimensionGrid` and

packages/zarr-indexing/docs/design-notes.md

@@ -35,27 +35,18 @@ about the deliberately matching semantics:
   discriminator, and `tests/test_ndsel_tensorstore.py` loads our bodies into
   `tensorstore.IndexTransform(json=...)` and round-trips them back through our
   engine layer.
-
-The representations differ in one place: index arrays. Both models want an index
-array at the transform's full input rank, with singleton axes for the dimensions
-a map does not vary over. TensorStore enforces it — its JSON parser rejects a
-rank-1 array over a rank-2 domain outright, with `Index array for output
-dimension 0 has rank 1 but must have rank 2` (checked against tensorstore
-0.1.84) — while our loader is the more permissive of the two and also accepts a
-lower-rank array that broadcasts against the input domain. That is a
-compatibility affordance, not a difference in the model: ndsel leaves index-array
-rank to [the engine layer](ndsel.md#lowering-to-a-transform), and everything the
-algebra builds itself is at full rank.
-
-The reason full rank matters here is that we *derive* meaning from those
-singletons rather than merely tolerating them: an array full-sized on one axis
-and singleton elsewhere is orthogonal, and one varying over several shared axes
-is vectorized, so the distinction is readable off the shape — and the shape is
-the *only* place it lives. An earlier `ArrayMap.input_dimension` field pinned
-the orthogonal axis redundantly and was retired: the one shape it disambiguated
-(a single-coordinate array, all axes singleton) is now normalized away at
-construction, collapsed to the `ConstantMap` it equals, exactly as
-[the serializer](api/json.md) has always collapsed it on the wire.
+- **Chunk partitioning.** Both factor a transform over a grid before visiting
+  any cell, rather than intersecting the whole transform with each chunk.
+  TensorStore's `IndexTransformGridPartition` holds strided sets and index
+  array sets and derives a per-cell transform on iteration;
+  [`GridPartition`](api/chunk_resolution.md) has the same shape with two
+  simplifications. Its `StridedSet` is per storage axis, where TensorStore's
+  is per input dimension and spans every grid axis reading it (which is how
+  TensorStore serves the diagonal this library rejects), and it keeps one
+  `JointSet` for all correlated arrays where TensorStore keeps one index
+  array set per connected component. Both derive the per-chunk transforms
+  from the partition ([the guide](guide/index.md#a-plan-is-a-product-of-per-axis-tables)
+  shows the tables).
 
 Four deliberate differences:
 
@@ -85,11 +76,14 @@ safe, `partial` proves it is not, and `unknown` conservatively covers fancy
 selections whose duplicates would require additional work to classify.
 
 The comparison also runs the other way. TensorStore is a mature, heavily
-optimized C++ system whose performance this library cannot approach: resolution
-here is Python-level bookkeeping over NumPy, and the per-part overhead is
-significant. This library is small and depends on nothing beyond NumPy, so the
-algebra can be adopted by a Python project that wants the model without the C++
-runtime.
+optimized C++ system whose performance this library cannot approach. Planning
+here is per axis, never per chunk, but each materialized `ChunkProjection` is
+Python-level bookkeeping over NumPy — two domains, two transforms and the
+projection itself — so the per-part overhead of the object view is
+significant; a consumer that reads the partition's tables directly pays no
+per-chunk object construction. This library is small and depends on nothing beyond
+NumPy, so the algebra can be adopted by a Python project that wants the model
+without the C++ runtime.
 
 ## Bounding-box selections vs query selections
 
@@ -282,10 +276,13 @@ transform) — takes the pointwise path that collapses the joint block.
 Three limits remain, all intentional and all expected to be lifted:
 
 - **Affine diagonals.** A hand-built transform in which an *index array* and a
-  *slice map* bind the same input dimension, or two slice maps share one, is
-  rejected at resolution with `NotImplementedError`. No selection dialect
-  produces one; supporting them means lowering the slice maps into the joint
-  block too. *Planned.*
+  *slice map* bind the same input dimension is rejected at resolution with
+  `NotImplementedError`; one in which two slice maps share an input dimension
+  is rejected with `ValueError`, since it has no factored form. No selection
+  dialect produces either. Supporting the first means lowering the slice map
+  into the joint block; supporting the second means a strided table per
+  *input* dimension spanning every storage axis that reads it, as
+  TensorStore's does. *Planned.*
 - **Finite explicit bounds only.** `IndexDomain` has no implicit or unbounded
   dimensions; the message layer will normalize a body with `"-inf"`/`"+inf"`
   bounds, but the engine layer refuses to lower one into a transform.

packages/zarr-indexing/docs/guide/index.md

@@ -8,8 +8,9 @@ through those stages.
 The first four sections are for anyone indexing arrays: coordinates,
 transforms, composition, and result axes. **If you are using lazy indexing
 rather than building a storage backend, you can stop after section four.**
-The last two sections are for integrators: they turn a request into a chunk
-plan and pair each chunk read with its place in the result.
+The last three sections are for integrators: they turn a request into a chunk
+plan, pair each chunk read with its place in the result, and show the per-axis
+tables the plan is built from.
 
 Throughout, one division of labor holds: the transform answers **which
 values?** and is independent of the backend; the reader answers **how do I
@@ -435,6 +436,72 @@ The paired representation preserves information that a bounding box or local
 selector discards: exact request order, duplicate destinations, and the
 correspondence between every request position and its chunk-local source cell.
 
+## A plan is a product of per-axis tables {#a-plan-is-a-product-of-per-axis-tables}
+
+Look again at the two projections of `image[1, :]`. Each chunk transform has
+one map per source axis, and every one of those maps came from restricting
+the request's map for *that axis alone* to *that axis's* chunk: the fixed row
+`ConstantMap(1)` lands in row-chunk 0 whatever the column chunk is, and the
+column slice meets column-chunk 0 as local columns `0:2` and column-chunk 1 as
+local columns `0:2` whatever the row chunk is. Restricting a transform to a
+chunk distributes over axes whenever each output map reads its own request
+axis — which every basic and orthogonal selection satisfies. So the plan does
+not intersect the whole transform with every chunk. It resolves each axis
+once, into a table with one row per chunk that axis touches, and a projection
+is one row of each table combined.
+
+```text
+image[1, :] over 2-by-2 chunks
+
+axis 0 (rows): ConstantMap(1)          axis 1 (columns): DimensionMap
+row | chunk  start  local  extent      row | chunk  start  local_start  extent  origin  full
+ 0  |   0      0      1      1          0  |   0      0        0          2       0     yes
+                                        1  |   1      2        0          2       2     yes
+
+row_shape (1, 2): 1 x 2 = 2 projections
+projection (0, 0) = axis-0 row 0 x axis-1 row 0 -> chunk (0, 0), local (1, 0:2), request columns 0:2
+projection (0, 1) = axis-0 row 0 x axis-1 row 1 -> chunk (0, 1), local (1, 0:2), request columns 2:4
+```
+
+`ChunkPlan.partition()` returns this factored form, a `GridPartition`. Its
+`sets` hold one table per source axis, in axis order; `row_shape` is the
+number of rows in each; and the plan walks the rows in row-major order over
+it. The executable example reads the two tables above off the plan, checks
+that the plan's projections are exactly the partition's rows, and evaluates
+the second row on both of its transforms:
+
+```python
+--8<-- "snippets/grid_partition.py:strided-tables"
+```
+
+There are three kinds of table, matching the three map kinds and the one
+arrangement that does not factor. A `StridedSet` holds a `ConstantMap` or
+`DimensionMap` axis; an `IndexedSet` holds an orthogonal `ArrayMap` axis
+(`.oindex`) with its coordinates grouped by chunk; and a `JointSet` holds the
+correlated arrays of a `.vindex` selection, which read the same request axes
+and so do not distribute — a chunk constrains all of them at once, so their
+points are sorted into chunks together, once. The
+[API reference](../api/chunk_resolution.md) documents every column.
+
+The gather from the previous section, `oindex[[4, 1, 1], 2:6]` over 3-by-4
+chunks, groups rows `1, 1` into chunk 0 and row `4` into chunk 1 while
+remembering that row `4` fills request position 0. A `vindex` selection keeps
+its points paired in the joint table:
+
+```python
+--8<-- "snippets/grid_partition.py:indexed-and-joint"
+```
+
+Two properties follow from the factoring. Building the tables costs the *sum*
+of the chunks touched per axis, never their product, and correlated points
+cost one sort. And projections are derived from rows only when asked for:
+`chunk_coords()` lists every chunk the plan touches without materializing a
+row, and a consumer can read the columns directly, as
+[Integration boundaries](integrations.md#reading-the-tables-directly) shows.
+The one shape with no factored form is a hand-built diagonal — two
+`DimensionMap`s reading one request axis, which no selection produces — and
+the plan rejects it with `ValueError`.
+
 ---
 
 <nav aria-label="Guide navigation" markdown>

packages/zarr-indexing/docs/guide/integrations.md

@@ -27,6 +27,33 @@ example intentionally begins with already decoded in-memory chunks: storage
 keys, codecs, scheduling, caching, and asynchronous orchestration remain the
 consumer's policy rather than responsibilities of the plan.
 
+## Reading the tables directly {#reading-the-tables-directly}
+
+A consumer that fetches many chunks per request — a codec pipeline, a
+prefetcher — does not need a `ChunkProjection` object per chunk. The plan's
+[factored form](index.md#a-plan-is-a-product-of-per-axis-tables) is a few
+NumPy arrays per axis, and everything a chunk copy needs is a row of each:
+the chunk index, the chunk-local start and extent, and where the cells land
+in the request. `chunk_coords()` alone answers "which chunks?" for a prefetch,
+without materializing anything.
+
+The example assembles a strided box from its `StridedSet` tables, one slice
+per chunk. Three things a real consumer also has to get right are checked at
+the end: a descending stride needs the same care with a negative stop that any
+NumPy slice does, a chunk slab comes out in storage-axis order while the
+result is in request-axis order, and a request axis no map reads (`None` in
+the selection) is a length-one axis of the result.
+
+```python
+--8<-- "snippets/grid_partition.py:table-consumer"
+```
+
+The four reads are the four chunks the two axis tables multiply out to, and no
+intermediate transform, domain, or projection was built. A gather reads its
+`IndexedSet` rows the same way — `index[pointer[i]:pointer[i + 1]]` and the
+matching `positions` — and a `vindex` selection its `JointSet` rows, whose
+`local` coordinates already have the chunk origin subtracted.
+
 ## One slab read or many part reads
 
 A backend with its own native subset read — a Rust or C zarr implementation,

packages/zarr-indexing/docs/index.md

@@ -38,7 +38,9 @@ the chain stays one description, and where the materialization boundary is.
   chunk plan. Using lazy indexing, start at
   [An index selects coordinates](guide/index.md#an-index-selects-coordinates);
   integrating a chunked backend, start at
-  [A request becomes a chunk plan](guide/index.md#a-request-becomes-a-chunk-plan).
+  [A request becomes a chunk plan](guide/index.md#a-request-becomes-a-chunk-plan)
+  and finish with the per-axis tables a plan is
+  [built from](guide/index.md#a-plan-is-a-product-of-per-axis-tables).
 - [Indexing pattern reference](guide/patterns.md) — every selection form with
   its NumPy-verified result.
 - [Integration boundaries](guide/integrations.md) — what a reader, writer, or
@@ -49,5 +51,5 @@ the chain stays one description, and where the materialization boundary is.
 - [Design notes](design-notes.md) — TensorStore lineage, box vs query, and
   deliberate limits.
 - [API reference](api/index.md)
-- [Changelog](https://github.com/zarr-developers/zarr-python/blob/main/packages/zarr-indexing/CHANGELOG.md)
+- [Release notes](release-notes.md)
   · [License (MIT)](https://github.com/zarr-developers/zarr-python/blob/main/packages/zarr-indexing/LICENSE.txt)

packages/zarr-indexing/docs/release-notes.md

@@ -0,0 +1,8 @@
+<!-- CHANGELOG.md at the package root is the single source of truth: towncrier
+     writes releases into it, and PyPI and GitHub render it directly. This page
+     includes it so each docs build shows the release notes for its own version.
+     The page is not named changelog.md on purpose: on a case-insensitive
+     filesystem that name shadows CHANGELOG.md and the include resolves to the
+     page itself. -->
+
+--8<-- "CHANGELOG.md"