diff --git a/.github/workflows/codspeed.yml b/.github/workflows/codspeed.yml index 872f6d3945..1441950c3f 100644 --- a/.github/workflows/codspeed.yml +++ b/.github/workflows/codspeed.yml @@ -32,7 +32,7 @@ jobs: with: version: '1.16.5' - name: Run the benchmarks - uses: CodSpeedHQ/action@4296e51e7041e24dadb86d1d6e8b9320d223dbe8 # v5.0.3 + uses: CodSpeedHQ/action@373d6868929f444bc08d901fd0eb0ad52a8875ea # v5.2.1 env: ZARR_BENCHMARK_CLEAR_CACHE: '1' with: diff --git a/.github/workflows/zarr-indexing.yml b/.github/workflows/zarr-indexing.yml index f0b0cec65d..3b106e16aa 100644 --- a/.github/workflows/zarr-indexing.yml +++ b/.github/workflows/zarr-indexing.yml @@ -43,11 +43,10 @@ jobs: uses: extractions/setup-just@53165ef7e734c5c07cb06b3c8e7b647c5aa16db3 # v4 - name: Set up Python ${{ matrix.python-version }} run: uv python install ${{ matrix.python-version }} - # The transform tests exercise chunk resolution against zarr's ChunkGrid, - # so they run against the repo-root environment (which provides `zarr`) - # with this package as an editable overlay rather than in package - # isolation. The recipes carry that invocation; this step only fixes the - # interpreter the matrix asked for. + # The suite imports nothing from `zarr`; it runs against the repo-root + # environment, with this package as an editable overlay, to share the + # parent project's pinned test toolchain. The recipes carry that + # invocation; this step only fixes the interpreter the matrix asked for. - name: Sync test dependency group run: uv sync --project ../.. --group test --python ${{ matrix.python-version }} - name: Run pytest diff --git a/packages/zarr-http-server/changes/4311.doc.md b/packages/zarr-http-server/changes/4311.doc.md new file mode 100644 index 0000000000..674389780c --- /dev/null +++ b/packages/zarr-http-server/changes/4311.doc.md @@ -0,0 +1,2 @@ +The docs now render the release notes as a page instead of linking to +`CHANGELOG.md` on GitHub, so each docs version shows its own changelog. diff --git a/packages/zarr-http-server/docs/index.md b/packages/zarr-http-server/docs/index.md index 271de6445e..f44c087e77 100644 --- a/packages/zarr-http-server/docs/index.md +++ b/packages/zarr-http-server/docs/index.md @@ -67,5 +67,5 @@ Reads are all that is enabled by default: `GET` and `HEAD` are served, and - [User guide](guide.md) — building apps, running them, byte ranges, CORS, writes, notebooks, and Uvicorn configuration - [API reference](api/index.md) -- [Changelog](https://github.com/zarr-developers/zarr-python/blob/main/packages/zarr-http-server/CHANGELOG.md) +- [Release notes](release-notes.md) - [License (MIT)](https://github.com/zarr-developers/zarr-python/blob/main/packages/zarr-http-server/LICENSE.txt) diff --git a/packages/zarr-http-server/docs/release-notes.md b/packages/zarr-http-server/docs/release-notes.md new file mode 100644 index 0000000000..767cd56a84 --- /dev/null +++ b/packages/zarr-http-server/docs/release-notes.md @@ -0,0 +1,8 @@ + + +--8<-- "CHANGELOG.md" diff --git a/packages/zarr-http-server/mkdocs.yml b/packages/zarr-http-server/mkdocs.yml index 7ebfe8c017..44978f0223 100644 --- a/packages/zarr-http-server/mkdocs.yml +++ b/packages/zarr-http-server/mkdocs.yml @@ -16,7 +16,7 @@ nav: - guide.md - API Reference: - api/index.md - - Changelog: https://github.com/zarr-developers/zarr-python/blob/main/packages/zarr-http-server/CHANGELOG.md + - Release notes: release-notes.md watch: - src @@ -98,3 +98,11 @@ markdown_extensions: line_spans: __span pygments_lang_class: true - pymdownx.inlinehilite + - pymdownx.snippets: + # The package root lets docs/release-notes.md include the + # towncrier-managed CHANGELOG.md without moving it out of the package + # root, where PyPI and GitHub readers expect it. + base_path: [docs, .] + # Fail the build on an unresolvable include instead of silently + # rendering nothing. + check_paths: true diff --git a/packages/zarr-indexing/README.md b/packages/zarr-indexing/README.md index 7e2cec10dd..110ebcb722 100644 --- a/packages/zarr-indexing/README.md +++ b/packages/zarr-indexing/README.md @@ -25,6 +25,9 @@ Key types: - `ChunkPlan` and `ChunkProjection` — lazily partition a selection over a caller-selected grid and pair each chunk-local transform with its placement in the request, without binding a storage backend or scheduler +- `GridPartition` — the plan's factored, columnar form: one table per axis + (`StridedSet`, `IndexedSet`) plus a `JointSet` for correlated index arrays, + from which projections are derived on demand - `ConstantMap`, `DimensionMap`, `ArrayMap` — the three ways a single output dimension can depend on the input - `compose` — chain two transforms into one diff --git a/packages/zarr-indexing/changes/4310.feature.md b/packages/zarr-indexing/changes/4310.feature.md new file mode 100644 index 0000000000..e9c2d41664 --- /dev/null +++ b/packages/zarr-indexing/changes/4310.feature.md @@ -0,0 +1,11 @@ +Chunk plans now have a factored, columnar form. `ChunkPlan.partition()` +returns a `GridPartition`: one `StridedSet` or +`IndexedSet` table per output dimension the transform reads independently, +plus a `JointSet` for correlated (`vindex`) index arrays. A `ChunkProjection` +is one row of each table and is derived on demand, so planning costs the sum +of the touched chunks per axis rather than their product, and a vectorized +consumer can read the tables without materializing a projection per chunk. +The projections a plan yields are unchanged. +A hand-built transform in which two output maps read one input axis (a +diagonal, which no selection produces) is now rejected with `ValueError` +instead of being walked by whole-transform intersection. diff --git a/packages/zarr-indexing/changes/4311.doc.md b/packages/zarr-indexing/changes/4311.doc.md new file mode 100644 index 0000000000..674389780c --- /dev/null +++ b/packages/zarr-indexing/changes/4311.doc.md @@ -0,0 +1,2 @@ +The docs now render the release notes as a page instead of linking to +`CHANGELOG.md` on GitHub, so each docs version shows its own changelog. diff --git a/packages/zarr-indexing/docs/api/index.md b/packages/zarr-indexing/docs/api/index.md index 674c701a0f..c9d05d00bc 100644 --- a/packages/zarr-indexing/docs/api/index.md +++ b/packages/zarr-indexing/docs/api/index.md @@ -11,9 +11,9 @@ links: [Lazy views compose](../guide/index.md#lazy-views-compose), then open [`zarr_indexing.lazy_array`](lazy_array.md) for `LazyArray`. - **Integrate a chunked source:** finish - [One cell domain, two projections](../guide/index.md#one-cell-domain-two-projections), + [A plan is a product of per-axis tables](../guide/index.md#a-plan-is-a-product-of-per-axis-tables), then open [`zarr_indexing.chunk_resolution`](chunk_resolution.md) for - `plan_chunks`. Start with + `plan_chunks` and `GridPartition`. Start with [Coordinates are addresses](../guide/index.md#coordinates-are-addresses) if literal coordinates are unfamiliar. @@ -36,7 +36,10 @@ and the wire format built on top of it. - [`zarr_indexing.chunk_resolution`](chunk_resolution.md) — `plan_chunks`, which lazily projects a request through a caller-selected grid, - plus the reusable `ChunkPlan` and paired-transform `ChunkProjection` values + the reusable `ChunkPlan` and paired-transform `ChunkProjection` values, and + the plan's factored form: `GridPartition` (from `ChunkPlan.partition`), + holding one `StridedSet` or `IndexedSet` table per axis and a `JointSet` + for correlated index arrays - [`zarr_indexing.grid`](grid.md) — `DimensionGridLike`, the Protocol describing the narrow chunk-grid surface chunk resolution consumes, so that nothing here imports `zarr`, plus `EdgeDimensionGrid` and diff --git a/packages/zarr-indexing/docs/design-notes.md b/packages/zarr-indexing/docs/design-notes.md index 78b352d376..de786150ca 100644 --- a/packages/zarr-indexing/docs/design-notes.md +++ b/packages/zarr-indexing/docs/design-notes.md @@ -35,27 +35,18 @@ about the deliberately matching semantics: discriminator, and `tests/test_ndsel_tensorstore.py` loads our bodies into `tensorstore.IndexTransform(json=...)` and round-trips them back through our engine layer. - -The representations differ in one place: index arrays. Both models want an index -array at the transform's full input rank, with singleton axes for the dimensions -a map does not vary over. TensorStore enforces it — its JSON parser rejects a -rank-1 array over a rank-2 domain outright, with `Index array for output -dimension 0 has rank 1 but must have rank 2` (checked against tensorstore -0.1.84) — while our loader is the more permissive of the two and also accepts a -lower-rank array that broadcasts against the input domain. That is a -compatibility affordance, not a difference in the model: ndsel leaves index-array -rank to [the engine layer](ndsel.md#lowering-to-a-transform), and everything the -algebra builds itself is at full rank. - -The reason full rank matters here is that we *derive* meaning from those -singletons rather than merely tolerating them: an array full-sized on one axis -and singleton elsewhere is orthogonal, and one varying over several shared axes -is vectorized, so the distinction is readable off the shape — and the shape is -the *only* place it lives. An earlier `ArrayMap.input_dimension` field pinned -the orthogonal axis redundantly and was retired: the one shape it disambiguated -(a single-coordinate array, all axes singleton) is now normalized away at -construction, collapsed to the `ConstantMap` it equals, exactly as -[the serializer](api/json.md) has always collapsed it on the wire. +- **Chunk partitioning.** Both factor a transform over a grid before visiting + any cell, rather than intersecting the whole transform with each chunk. + TensorStore's `IndexTransformGridPartition` holds strided sets and index + array sets and derives a per-cell transform on iteration; + [`GridPartition`](api/chunk_resolution.md) has the same shape with two + simplifications. Its `StridedSet` is per storage axis, where TensorStore's + is per input dimension and spans every grid axis reading it (which is how + TensorStore serves the diagonal this library rejects), and it keeps one + `JointSet` for all correlated arrays where TensorStore keeps one index + array set per connected component. Both derive the per-chunk transforms + from the partition ([the guide](guide/index.md#a-plan-is-a-product-of-per-axis-tables) + shows the tables). Four deliberate differences: @@ -85,11 +76,14 @@ safe, `partial` proves it is not, and `unknown` conservatively covers fancy selections whose duplicates would require additional work to classify. The comparison also runs the other way. TensorStore is a mature, heavily -optimized C++ system whose performance this library cannot approach: resolution -here is Python-level bookkeeping over NumPy, and the per-part overhead is -significant. This library is small and depends on nothing beyond NumPy, so the -algebra can be adopted by a Python project that wants the model without the C++ -runtime. +optimized C++ system whose performance this library cannot approach. Planning +here is per axis, never per chunk, but each materialized `ChunkProjection` is +Python-level bookkeeping over NumPy — two domains, two transforms and the +projection itself — so the per-part overhead of the object view is +significant; a consumer that reads the partition's tables directly pays no +per-chunk object construction. This library is small and depends on nothing beyond +NumPy, so the algebra can be adopted by a Python project that wants the model +without the C++ runtime. ## Bounding-box selections vs query selections @@ -282,10 +276,13 @@ transform) — takes the pointwise path that collapses the joint block. Three limits remain, all intentional and all expected to be lifted: - **Affine diagonals.** A hand-built transform in which an *index array* and a - *slice map* bind the same input dimension, or two slice maps share one, is - rejected at resolution with `NotImplementedError`. No selection dialect - produces one; supporting them means lowering the slice maps into the joint - block too. *Planned.* + *slice map* bind the same input dimension is rejected at resolution with + `NotImplementedError`; one in which two slice maps share an input dimension + is rejected with `ValueError`, since it has no factored form. No selection + dialect produces either. Supporting the first means lowering the slice map + into the joint block; supporting the second means a strided table per + *input* dimension spanning every storage axis that reads it, as + TensorStore's does. *Planned.* - **Finite explicit bounds only.** `IndexDomain` has no implicit or unbounded dimensions; the message layer will normalize a body with `"-inf"`/`"+inf"` bounds, but the engine layer refuses to lower one into a transform. diff --git a/packages/zarr-indexing/docs/guide/index.md b/packages/zarr-indexing/docs/guide/index.md index 42e47530fb..57befaa1d8 100644 --- a/packages/zarr-indexing/docs/guide/index.md +++ b/packages/zarr-indexing/docs/guide/index.md @@ -8,8 +8,9 @@ through those stages. The first four sections are for anyone indexing arrays: coordinates, transforms, composition, and result axes. **If you are using lazy indexing rather than building a storage backend, you can stop after section four.** -The last two sections are for integrators: they turn a request into a chunk -plan and pair each chunk read with its place in the result. +The last three sections are for integrators: they turn a request into a chunk +plan, pair each chunk read with its place in the result, and show the per-axis +tables the plan is built from. Throughout, one division of labor holds: the transform answers **which values?** and is independent of the backend; the reader answers **how do I @@ -435,6 +436,72 @@ The paired representation preserves information that a bounding box or local selector discards: exact request order, duplicate destinations, and the correspondence between every request position and its chunk-local source cell. +## A plan is a product of per-axis tables {#a-plan-is-a-product-of-per-axis-tables} + +Look again at the two projections of `image[1, :]`. Each chunk transform has +one map per source axis, and every one of those maps came from restricting +the request's map for *that axis alone* to *that axis's* chunk: the fixed row +`ConstantMap(1)` lands in row-chunk 0 whatever the column chunk is, and the +column slice meets column-chunk 0 as local columns `0:2` and column-chunk 1 as +local columns `0:2` whatever the row chunk is. Restricting a transform to a +chunk distributes over axes whenever each output map reads its own request +axis — which every basic and orthogonal selection satisfies. So the plan does +not intersect the whole transform with every chunk. It resolves each axis +once, into a table with one row per chunk that axis touches, and a projection +is one row of each table combined. + +```text +image[1, :] over 2-by-2 chunks + +axis 0 (rows): ConstantMap(1) axis 1 (columns): DimensionMap +row | chunk start local extent row | chunk start local_start extent origin full + 0 | 0 0 1 1 0 | 0 0 0 2 0 yes + 1 | 1 2 0 2 2 yes + +row_shape (1, 2): 1 x 2 = 2 projections +projection (0, 0) = axis-0 row 0 x axis-1 row 0 -> chunk (0, 0), local (1, 0:2), request columns 0:2 +projection (0, 1) = axis-0 row 0 x axis-1 row 1 -> chunk (0, 1), local (1, 0:2), request columns 2:4 +``` + +`ChunkPlan.partition()` returns this factored form, a `GridPartition`. Its +`sets` hold one table per source axis, in axis order; `row_shape` is the +number of rows in each; and the plan walks the rows in row-major order over +it. The executable example reads the two tables above off the plan, checks +that the plan's projections are exactly the partition's rows, and evaluates +the second row on both of its transforms: + +```python +--8<-- "snippets/grid_partition.py:strided-tables" +``` + +There are three kinds of table, matching the three map kinds and the one +arrangement that does not factor. A `StridedSet` holds a `ConstantMap` or +`DimensionMap` axis; an `IndexedSet` holds an orthogonal `ArrayMap` axis +(`.oindex`) with its coordinates grouped by chunk; and a `JointSet` holds the +correlated arrays of a `.vindex` selection, which read the same request axes +and so do not distribute — a chunk constrains all of them at once, so their +points are sorted into chunks together, once. The +[API reference](../api/chunk_resolution.md) documents every column. + +The gather from the previous section, `oindex[[4, 1, 1], 2:6]` over 3-by-4 +chunks, groups rows `1, 1` into chunk 0 and row `4` into chunk 1 while +remembering that row `4` fills request position 0. A `vindex` selection keeps +its points paired in the joint table: + +```python +--8<-- "snippets/grid_partition.py:indexed-and-joint" +``` + +Two properties follow from the factoring. Building the tables costs the *sum* +of the chunks touched per axis, never their product, and correlated points +cost one sort. And projections are derived from rows only when asked for: +`chunk_coords()` lists every chunk the plan touches without materializing a +row, and a consumer can read the columns directly, as +[Integration boundaries](integrations.md#reading-the-tables-directly) shows. +The one shape with no factored form is a hand-built diagonal — two +`DimensionMap`s reading one request axis, which no selection produces — and +the plan rejects it with `ValueError`. + ---