docs(blog): zarr 3.4.0 release post

docs/blog/posts/3.3.0-release.md

@@ -167,3 +167,30 @@ We hope these new features are helpful, and we would appreciate any feedback tha
 The updates in this release are just the first step of a larger performance-oriented direction for Zarr-Python. Landing these two enhancements taught us a *lot* about the performance-sensitive areas of the library. We can and will invest more time in performance tuning, e.g. by adding or changing abstractions, writing code for special cases, etc.
 
 We plan to consider including compiled code that should enable significant performance improvements. The [`zarrs`](https://zarrs.dev/) project is an ecosystem of Zarr tools written in Rust, with [extremely high performance](https://book.zarrs.dev/#-zarrs-is-fast-). Is there a `zarrs` binding in Zarr-Python's future? I hope so! We are keenly observing development of [`zarrista`](https://developmentseed.org/zarrista/latest/) as a proof-of-concept for what a Python-`zarrs` binding layer might look like. Stay tuned!
+
+## Contributors
+
+Thanks to everyone who contributed to this release (`*` marks a first-time contributor to Zarr-Python):
+
+- [@aldenks](https://github.com/aldenks)
+- [@AMBRA7592](https://github.com/AMBRA7592) *
+- [@binggao1230](https://github.com/binggao1230) *
+- [@chuckwondo](https://github.com/chuckwondo) *
+- [@d-v-b](https://github.com/d-v-b)
+- [@DimitriPapadopoulos](https://github.com/DimitriPapadopoulos)
+- [@goutamadwant](https://github.com/goutamadwant) *
+- [@ilan-gold](https://github.com/ilan-gold)
+- [@jhamman](https://github.com/jhamman)
+- [@josh-ag2](https://github.com/josh-ag2) *
+- [@kabilar](https://github.com/kabilar) *
+- [@keewis](https://github.com/keewis)
+- [@lhoupert](https://github.com/lhoupert) *
+- [@maxrjones](https://github.com/maxrjones)
+- [@NIK-TIGER-BILL](https://github.com/NIK-TIGER-BILL) *
+- [@oldrobotdev](https://github.com/oldrobotdev) *
+- [@SAY-5](https://github.com/SAY-5) *
+- [@sehoffmann](https://github.com/sehoffmann) *
+- [@selmanozleyen](https://github.com/selmanozleyen) *
+- [@stibrew](https://github.com/stibrew) *
+- [@TomAugspurger](https://github.com/TomAugspurger)
+- [@zkoppert](https://github.com/zkoppert) *

docs/blog/posts/3.4.0-release.md

@@ -0,0 +1,124 @@
+---
+date: 2026-09-10
+authors:
+  - d-v-b
+categories:
+  - Release
+draft: true
+---
+
+# `zarr` 3.4.0
+
+`zarr` 3.4.0 is out! This post covers the highlights of the latest version of `zarr`, and a few other packages you might not have heard of. See the full [release notes](../../release-notes.md) for the per-PR breakdown, or keep reading to catch the digested version.
+
+## Stackification
+
+This first item is not a new `zarr` feature. Rather, it's a **Zarr-Python** feature: Zarr-Python (the project) is becoming a *stack* of related Python packages, foremost among them `zarr`, the Python library. In the last few months Zarr-Python gained the following stand-alone Python packages:
+
+- [`zarr-metadata`](https://zarr-metadata.readthedocs.io/), a library narrowly scoped to modelling Zarr metadata documents (e.g., `zarr.json`), with precise type definitions and validation routines. Designed for anyone who wants type safety for Zarr V2 or V3 metadata without re-implementing the Zarr specs. `zarr-metadata` is very lightweight -- it does not depend on `zarr` -- so it should be a very cheap dependency for projects to add, either as a runtime or test dependency. We hope to replace most of the metadata logic in `zarr` with the contents of `zarr-metadata`.
+- [`zarr-indexing`](https://zarr-indexing.readthedocs.io/), a library concerned with data structures and algorithms for lazy, chunked array indexing routines based on Google's trailblazing [TensorStore](https://google.github.io/tensorstore/) library. We hope to replace the array indexing logic in `zarr` with routines defined in `zarr-indexing`, and we also want to use `zarr-indexing` to support new lazy indexing functionality for arrays in `zarr`.
+- [`zarr-http-server`](https://zarr-http-server.readthedocs.io/), a library that serves a `zarr` `Store`, `Array` or `Group` over HTTP via an ASGI app. This allows any array-like data to be dynamically exposed as Zarr. For example, you can take TIFF stacks, access them with `tifffile`, and use `tifffile`'s Zarr layer with `zarr-http-server` to expose the TIFF data as Zarr over HTTP for consumption by clients that don't know how to read TIFF. There are no plans for `zarr` to depend on this package.
+
+These three packages are just a start. We have in-flight PRs for defining packages for Zarr chunk key encodings ([PR 4298](https://github.com/zarr-developers/zarr-python/pull/4298)), stores ([PR 4318](https://github.com/zarr-developers/zarr-python/pull/4318)), and codecs ([PR 4319](https://github.com/zarr-developers/zarr-python/pull/4319)).
+
+All of these packages are pre-1.0 and changing rapidly; `zarr` does not yet depend on them. When the relevant APIs stabilize, we plan to gradually replace functionality defined in `zarr` today for the equivalent functionality in a subpackage. For example, the [store PR](https://github.com/zarr-developers/zarr-python/pull/4318) includes a `legacy` module that contains a verbatim copy of `zarr`'s current `Store` API, so that when `zarr` depends on `zarr-storage`, we can swap out functionality defined in `zarr` for identical functionality in `zarr-storage`, and users won't notice any difference beyond an additional installed package.
+
+The strategy here is to identify the logically separate parts of a Zarr implementation, and build a separate package around each part. This is not an original idea: the Rust [`zarrs`](https://zarrs.dev/) project is composed of many separate packages and this architecture has worked well for that project.
+
+Why are we doing this? Factoring `zarr` into separate packages may seem like a lot of churn with little upside beyond more maintenance. It's true that there will be new kinds of maintenance, and publishing gets a bit more complicated. But we see the following advantages to organizing Zarr-Python into subpackages:
+
+- Modularity
+  Installing `zarr` brings in `numcodecs` and other dependencies required for the full Zarr format. If a Python user only needs to parse Zarr metadata documents, e.g. because they are using [`zarrista`](https://developmentseed.org/zarrista/latest/) or [`tensorstore`](https://google.github.io/tensorstore/) to do IO, then the full set of `zarr` dependencies is wasteful. We want to make it as easy as possible for people to use Zarr correctly. That means it should be easy to install a package that defines the syntax and semantics of the Zarr metadata documents, without committing consumers of that package to a full runtime.
+- Agility
+  We believe that separate packages will enable more effective development of the Zarr stack. For example, we want to develop a new version of our storage APIs that formally separate sync and async stores. We could carry out this development inside `zarr`, but `zarr` releases would mix storage API developments with ordinary `zarr` development, and so changes to the store API itself would not be transparent to consumers, e.g. implementers of third-party storage backends. If the storage API is defined entirely in a separate package, then we can version that package exactly when the storage API changes, which makes the design history much more visible to consumers.
+- Focus
+  Functionality in `zarr` has a tendency to be as good as it needs to be for `zarr`'s needs. This is reasonable, but it can complicate the decision about when to make internal APIs public. For example, the core metadata APIs in `zarr` are not clearly marked public, which is unfortunate because the metadata is a really important part of Zarr! We hope that devoting an entire package to, e.g., Zarr metadata, makes it easier to define a public API surface that `zarr` and any other consumer can depend on. We want these subpackages to treat `zarr` as the main, but not sole, consumer, which means we can implement useful features even if `zarr` doesn't need them today.
+- Packaging
+  Defining core APIs in separate packages solves a packaging problem for `zarr`. If `zarr` defines core APIs like storage and codec interfaces internally, third-party packages must depend on `zarr` to access those APIs. This in turn means `zarr` cannot offer those third-party packages as optional dependencies without incurring a circular dependency. Introducing a common dependency for `zarr` and `third-party-zarr-package` removes the circular dependency issue entirely.
+
+For the full list of subpackages, see the "Related Projects" [page in our docs](../../subprojects.md).
+
+## Our Roadmap
+
+With the Zarr V2 -> V3 format transition largely settled, it's worth reflecting on how we want `zarr` to evolve. The previous section outlined one facet of that evolution -- defining `zarr` as the top of a stack of interlocking Zarr tools. There are other big changes ahead, foremost among them our goal of unlocking a vast performance improvement by wrapping the `zarrs` Rust crate via [PyO3](https://pyo3.rs/) bindings defined in [`zarrista`](https://developmentseed.org/zarrista/latest/). You can read more about this direction and other future plans in our new [roadmap document](../../roadmap.md), which is published as part of the 3.4.0 release.
+
+## Rectilinear Chunk Grid Improvements
+
+The experimental rectilinear (irregular) chunk grid support (opt in with `zarr.config.set({"array.rectilinear_chunks": True})`) got several API improvements and bug fixes, mostly in [#4218](https://github.com/zarr-developers/zarr-python/pull/4218) and [#4272](https://github.com/zarr-developers/zarr-python/pull/4272):
+
+- Chunk grid flavor (regular vs rectilinear) now consistently follows input syntax: a nested per-dimension list always requests a rectilinear grid, even when the edges happen to be uniform. This fixes a regression in 3.3.0, where uniform-plus-short-tail lists requested a regular grid, which changed resize semantics.
+- `zarr.from_array(..., chunks="keep")` reproduces a rectilinear source grid
+  exactly (in O(ndim)); sharding under a rectilinear shard grid is preserved
+  instead of dropped.
+- `Array.chunks` is defined for any sharded array; `Array.info` and
+  `Array.nchunks_initialized` work for rectilinear shard grids instead of
+  raising (`info` shows the shard shape as `<variable>`).
+- Zero-length axes: `chunks=-1` and `chunks=False` resolve to chunk size 1 ([#4307](https://github.com/zarr-developers/zarr-python/pull/4307), [#4328](https://github.com/zarr-developers/zarr-python/pull/4328)) instead of `ZeroDivisionError` / invalid metadata.
+
+## Useful Codec Errors
+
+`zarr` does not support every possible codec out of the box. Instead, we allow external packages to register their codec implementations with `zarr` at runtime, which broadens the palette of codecs we can support. When `zarr` reads an array with a codec it doesn't recognize, there's a good chance that the fix is to install the package that ships that codec implementation. But how should a user know which package they need?
+
+Previously, `zarr` handled an unknown codec with an opaque error, conveying only that `zarr` doesn't recognize the codec. Now, thanks to PRs [#4277](https://github.com/zarr-developers/zarr-python/pull/4277) and [#4351](https://github.com/zarr-developers/zarr-python/pull/4351), we maintain a table that relates the names of well-known codecs `zarr` doesn't ship with to packages that provide implementations. We use this table to provide a useful error message when an unknown codec is encountered. For example, opening an array whose metadata names the `imagecodecs_jpeg` codec, without any package that provides it installed:
+
+```python exec="true" session="codec-errors" source="above" result="ansi"
+import json
+
+import zarr
+from zarr.buffer.cpu import buffer_prototype
+
+metadata = {
+    "zarr_format": 3,
+    "node_type": "array",
+    "shape": [4, 4],
+    "data_type": "uint8",
+    "chunk_grid": {"name": "regular", "configuration": {"chunk_shape": [4, 4]}},
+    "chunk_key_encoding": {"name": "default"},
+    "fill_value": 0,
+    "codecs": [{"name": "bytes"}, {"name": "imagecodecs_jpeg"}],
+    "attributes": {},
+}
+store = {"zarr.json": buffer_prototype.buffer.from_bytes(json.dumps(metadata).encode())}
+
+try:
+    zarr.open_array(store, mode="r")
+except zarr.errors.UnknownCodecError as e:
+    print(e)
+```
+
+The table lives in `zarr.registry`; if you maintain a package that provides a codec, please add it there.
+
+## New Dependency: `msgspec`
+
+Zarr metadata is composed of JSON documents that must comply with various specification documents, e.g. the Zarr V2 and V3 specs. `zarr` should model these JSON documents in a way that's correct, so that we generate interoperable data, and useful for developers, so they can also generate interoperable data. To do that we need to define two things: precise types, and routines for coercing or narrowing input values with unknown shapes into our precise types.
+
+Writing types for JSON data structures has gotten a lot easier in Python with the development of `TypedDict` utility types, but writing a validation function that checks an unknown input against an arbitrary `TypedDict` is arbitrary work. This isn't a hard programming task; rather, it's one that's very easy to mess up, e.g. by omitting an important check, or failing to model a field as optional.
+
+Runtime validation libraries like `pydantic`, `beartype`, and `msgspec` use type annotations to generate runtime type checking routines. Using one of these libraries could spare us a lot of hand-written type checking code. We have long discussed adding a runtime type checker to `zarr`, and as of 3.4.0 we are now using [`msgspec`](https://github.com/msgspec/msgspec) in this capacity ([#4063](https://github.com/zarr-developers/zarr-python/pull/4063), resolving [#3285](https://github.com/zarr-developers/zarr-python/issues/3285)). We chose `msgspec` because of its performance and minimal dependency count, and we expect it to deliver big value whenever we introduce new codecs, chunk grids, or data types to `zarr`, as we can just define the type for the metadata and get a validation routine for free.
+
+## Tell us what you think
+
+As always, we would love any and all feedback about the work highlighted in this release! We would especially appreciate feedback from Python users who work with Zarr data and **don't** use the `zarr` package, since we are hoping to target their needs with our new family of subpackages.
+
+## Contributors
+
+Thanks to everyone who contributed to this release (`*` marks a first-time contributor to Zarr-Python):
+
+- [@arcusbuilds](https://github.com/arcusbuilds) *
+- [@cucuwang](https://github.com/cucuwang) *
+- [@d-v-b](https://github.com/d-v-b)
+- [@dylanpulver](https://github.com/dylanpulver) *
+- [@glaziermag](https://github.com/glaziermag) *
+- [@jhamman](https://github.com/jhamman)
+- [@JOhnsonKC201](https://github.com/JOhnsonKC201) *
+- [@paraseba](https://github.com/paraseba)
+- [@selmanozleyen](https://github.com/selmanozleyen)
+- [@sheikhayaan](https://github.com/sheikhayaan) *
+- [@Tomatokeftes](https://github.com/Tomatokeftes) *
+- [@vup903](https://github.com/vup903) *
+
+__Our [previous blog post](3.3.0-release.md) omitted the contributor list, so we've updated that post to include one.__
+
+!!! info "AI Usage Disclaimer"
+
+    The outline for this post was generated by Claude Fable after prompting with the set of unreleased changes since 3.3.0. That outline was then heavily expanded by @d-v-b, introducing a large number of typographical errors. Those errors were removed, code snippets added, and hyperlinks resolved, by Claude.