feat(zarr-metadata): add optional msgspec integration module

packages/zarr-metadata/changes/312.feature.md

@@ -0,0 +1,8 @@
+Added `zarr_metadata.msgspec`, an optional msgspec integration module: field
+types over the core metadata models plus a `dec_hook` (and a `make_dec_hook`
+composer for applications with hooks of their own) that route raw documents
+through the models' strict `from_json` parser, so the models can be used as
+field types in `msgspec.Struct` classes and with `msgspec.json.decode` /
+`msgspec.convert`. Invalid documents surface as `msgspec.ValidationError`
+with the loc-annotated problem messages. msgspec stays an optional
+dependency of `zarr-metadata`; the module requires msgspec 0.19 or newer.

packages/zarr-metadata/docs/api/index.md

@@ -10,6 +10,8 @@ The package is organized to mirror the structure of the Zarr specifications:
   structural validators, loc-aware parsers, and the `UNSET` sentinel
 - [`zarr_metadata.pydantic`](pydantic.md) — optional Pydantic field types
   over the models
+- [`zarr_metadata.msgspec`](msgspec.md) — optional msgspec field types and
+  decode hook over the models
 - [`zarr_metadata.v2`](v2.md) — `TypedDict` shapes for Zarr v2 documents
   (`.zarray`, `.zgroup`, `.zattrs`, `.zmetadata`)
 - [`zarr_metadata.v3`](v3/index.md) — `TypedDict` shapes for Zarr v3

packages/zarr-metadata/docs/api/msgspec.md

@@ -0,0 +1,5 @@
+---
+title: msgspec
+---
+
+::: zarr_metadata.msgspec

packages/zarr-metadata/docs/index.md

@@ -35,6 +35,9 @@ closely model the content of the Zarr specifications, such as:
 - **Optional Pydantic integration** ([`zarr_metadata.pydantic`](api/pydantic.md),
   requires Pydantic 2.13 or newer): each model as a Pydantic field type that
   validates raw documents through the same strict parser.
+- **Optional msgspec integration** ([`zarr_metadata.msgspec`](api/msgspec.md),
+  requires msgspec 0.19 or newer): field types and a decode hook that route
+  raw documents through the same strict parser.
 
 ## What this is for
 
@@ -67,6 +70,20 @@ A bare `TypeAdapter` over a public document `TypedDict` is a coercive shape
 adapter, not a Zarr conformance validator; it may coerce values or discard
 members that the strict model parser rejects.
 
+The optional msgspec integration does the same through msgspec's decode hook:
+
+```python
+import msgspec
+import zarr_metadata.msgspec as zmm
+
+metadata = msgspec.convert(raw, zmm.ZarrV3ArrayMetadata, dec_hook=zmm.dec_hook)
+encoded = metadata.to_key_value()["zarr.json"]
+```
+
+Serialization stays explicit (`to_json` / `to_key_value`): msgspec encodes
+dataclasses natively, so no hook can make `msgspec.json.encode` emit the
+canonical document — see [`zarr_metadata.msgspec`](api/msgspec.md).
+
 ## Validation boundary
 
 The model validators enforce the declared document structure and a small set

packages/zarr-metadata/mkdocs.yml

@@ -17,6 +17,7 @@ nav:
       - api/index.md
       - '<code class="doc-symbol doc-symbol-toc doc-symbol-module"></code> <code>zarr_metadata.model</code>': api/model.md
       - '<code class="doc-symbol doc-symbol-toc doc-symbol-module"></code> <code>zarr_metadata.pydantic</code>': api/pydantic.md
+      - '<code class="doc-symbol doc-symbol-toc doc-symbol-module"></code> <code>zarr_metadata.msgspec</code>': api/msgspec.md
       - '<code class="doc-symbol doc-symbol-toc doc-symbol-module"></code> <code>zarr_metadata.v2</code>': api/v2.md
       - '<code class="doc-symbol doc-symbol-toc doc-symbol-module"></code> <code>zarr_metadata.v3</code>':
         - api/v3/index.md

packages/zarr-metadata/pyproject.toml

@@ -47,7 +47,9 @@ Changelog = "https://github.com/zarr-developers/zarr-python/blob/main/packages/z
 Documentation = "https://zarr-metadata.readthedocs.io/"
 
 [dependency-groups]
-test = ["pytest", "pydantic>=2.13", "jsonschema"]
+# The msgspec floor matches the requirement of the zarr package, the
+# integration's motivating consumer.
+test = ["pytest", "pydantic>=2.13", "jsonschema", "msgspec>=0.19"]
 docs = [
     # Pins match the zarr-python docs environment in the repo-root
     # pyproject.toml so the two sites render with the same toolchain.

packages/zarr-metadata/src/zarr_metadata/msgspec.py

@@ -0,0 +1,220 @@
+"""Optional msgspec integration: field types and a decode hook over the core models.
+
+Importing this module requires msgspec; the core package deliberately does
+not depend on it, so this module is never imported by `zarr_metadata` itself.
+
+msgspec's extension point is a decode hook consulted only for annotated
+types msgspec does not support natively — and the core models are
+dataclasses, a kind msgspec supports natively, so annotating a field with a
+core model class directly would engage msgspec's own field-by-field
+dataclass coercion and bypass `from_json` (the single source of truth for
+structural validation and normalization). Each name this module exports is
+therefore a runtime marker class that msgspec treats as a custom type,
+forcing every value for the field through this module's `dec_hook`. Each
+marker registers its core model class as a virtual subclass, so the values
+`dec_hook` produces satisfy msgspec's type check while the instances ARE
+the core classes — they interoperate freely with non-msgspec code
+(equality, isinstance, nesting). Static type checkers see each field type
+as its core model class, so `manifest.metadata` below is a
+`zarr_metadata.model.ZarrV3ArrayMetadata`. That alias holds for
+annotations only: at runtime the exported names are empty markers with
+none of the core classes' methods, so construct, parse, and serialize
+through `zarr_metadata.model` — `zmm.ZarrV3ArrayMetadata.from_json(...)`
+type-checks but fails.
+
+`dec_hook` routes a raw document through `from_json` and passes an existing
+model instance through unchanged. `MetadataValidationError` subclasses
+`ValueError`, so a failed parse surfaces as a `msgspec.ValidationError`
+carrying the loc-annotated problem messages, with msgspec appending the
+path of the failing field (``- at `$.metadata` ``).
+
+Usage:
+
+    import msgspec
+
+    import zarr_metadata.msgspec as zmm
+
+    class ArrayManifest(msgspec.Struct):
+        path: str
+        metadata: zmm.ZarrV3ArrayMetadata
+
+    manifest = msgspec.json.decode(data, type=ArrayManifest, dec_hook=zmm.dec_hook)
+
+The same hook serves `msgspec.convert` — e.g.
+`msgspec.convert(doc, zmm.ZarrV3ArrayMetadata, dec_hook=zmm.dec_hook)` —
+and a prebuilt `msgspec.json.Decoder`. An application that already has a
+decode hook of its own composes via `make_dec_hook(wrapped=...)`.
+
+Three msgspec limits shape what this module can offer:
+
+- Serialization cannot be delegated: msgspec's encoders are value-driven
+  and consult their `enc_hook` only for objects msgspec cannot encode
+  natively, which a dataclass never is, so no hook can route a model
+  instance through `to_json`. Encoding a model directly either raises (the
+  `UNSET` sentinel is unencodable) or silently emits a raw field dump that
+  is NOT the canonical document (no shorthand collapse, no omit-empty
+  conventions). Serialize explicitly: put `model.to_json()` — the
+  canonical document — wherever msgspec-encodable output is needed.
+- Unions may contain at most one hook-handled type, so a field cannot be
+  typed as, say, array-or-group metadata. Decode such a field as an
+  untyped mapping and dispatch on its content.
+- JSON Schema generation (`msgspec.json.schema`) rejects custom types
+  unless the caller supplies a `schema_hook`, so a Struct using these
+  field types cannot produce a schema out of the box. When a JSON Schema
+  for the document forms is what you need, `zarr_metadata.pydantic` is
+  the schema-capable integration.
+"""
+
+from __future__ import annotations
+
+import abc
+from typing import TYPE_CHECKING, Any, Final
+
+# The import is unused by name: it makes the module fail fast where msgspec
+# is absent (everything exported here is inert without it), mirroring how
+# `zarr_metadata.pydantic` fails at import when pydantic is absent.
+import msgspec  # noqa: F401 # pyright: ignore[reportUnusedImport]
+
+from zarr_metadata import model as _model
+
+if TYPE_CHECKING:
+    from collections.abc import Callable
+
+    # For static type checkers the field types ARE the core model classes.
+    ZarrV3ArrayMetadata = _model.ZarrV3ArrayMetadata
+    ZarrV2ArrayMetadata = _model.ZarrV2ArrayMetadata
+    ZarrV3GroupMetadata = _model.ZarrV3GroupMetadata
+    ZarrV2GroupMetadata = _model.ZarrV2GroupMetadata
+    ZarrV3ConsolidatedMetadata = _model.ZarrV3ConsolidatedMetadata
+    ZarrV2ConsolidatedMetadata = _model.ZarrV2ConsolidatedMetadata
+    ZarrV3MetadataField = _model.ZarrV3NamedConfig
+else:
+    # At runtime each field type is a marker: an empty ABC msgspec treats as
+    # a custom type (so `dec_hook` is consulted) with the core model class
+    # registered as a virtual subclass (so the core instances `dec_hook`
+    # returns satisfy msgspec's isinstance check on hook results). The
+    # registrations are derived from `_DECODERS` below, keeping one table
+    # that pairs each marker with its core class.
+
+    class _FieldType(abc.ABC):  # noqa: B024
+        """Base of the runtime field-type markers; never instantiated."""
+
+        __slots__ = ()
+
+        def __new__(cls) -> None:
+            raise TypeError(
+                f"{cls.__name__} is a field-type marker for msgspec annotations only; "
+                "construct instances via the corresponding zarr_metadata.model class"
+            )
+
+    class ZarrV3ArrayMetadata(_FieldType):
+        """Field type for a v3 array metadata document (`zarr.json` content)."""
+
+    class ZarrV2ArrayMetadata(_FieldType):
+        """Field type for a v2 array metadata document (merged `.zarray` + `.zattrs` form)."""
+
+    class ZarrV3GroupMetadata(_FieldType):
+        """Field type for a v3 group metadata document (`zarr.json` content)."""
+
+    class ZarrV2GroupMetadata(_FieldType):
+        """Field type for a v2 group metadata document (merged `.zgroup` + `.zattrs` form)."""
+
+    class ZarrV3ConsolidatedMetadata(_FieldType):
+        """Field type for v3 inline consolidated metadata."""
+
+    class ZarrV2ConsolidatedMetadata(_FieldType):
+        """Field type for a v2 `.zmetadata` document."""
+
+    class ZarrV3MetadataField(_FieldType):
+        """Field type for one normalized v3 metadata extension envelope."""
+
+
+_DECODERS: Final[dict[type, tuple[type, Callable[[object], object]]]] = {
+    ZarrV3ArrayMetadata: (_model.ZarrV3ArrayMetadata, _model.ZarrV3ArrayMetadata.from_json),
+    ZarrV2ArrayMetadata: (_model.ZarrV2ArrayMetadata, _model.ZarrV2ArrayMetadata.from_json),
+    ZarrV3GroupMetadata: (_model.ZarrV3GroupMetadata, _model.ZarrV3GroupMetadata.from_json),
+    ZarrV2GroupMetadata: (_model.ZarrV2GroupMetadata, _model.ZarrV2GroupMetadata.from_json),
+    ZarrV3ConsolidatedMetadata: (
+        _model.ZarrV3ConsolidatedMetadata,
+        _model.ZarrV3ConsolidatedMetadata.from_json,
+    ),
+    ZarrV2ConsolidatedMetadata: (
+        _model.ZarrV2ConsolidatedMetadata,
+        _model.ZarrV2ConsolidatedMetadata.from_json,
+    ),
+    ZarrV3MetadataField: (_model.ZarrV3NamedConfig, _model.ZarrV3NamedConfig.from_json),
+}
+"""Marker class -> (pass-through core class, document parser)."""
+
+if not TYPE_CHECKING:
+    for _marker, (_core_cls, _) in _DECODERS.items():
+        _marker.register(_core_cls)
+
+
+def _lookup(type: type) -> tuple[type, Callable[[object], object]] | None:
+    """Return the decode entry for `type`, or None for types not covered here.
+
+    msgspec can hand a hook parametrized annotation objects, which may be
+    unhashable; those are never this module's markers, so a failed hash is
+    an ordinary miss rather than an error.
+    """
+    try:
+        return _DECODERS.get(type)
+    except TypeError:
+        return None
+
+
+def _decode(entry: tuple[type, Callable[[object], object]], obj: Any) -> Any:
+    core_cls, parse = entry
+    if isinstance(obj, core_cls):
+        return obj
+    return parse(obj)
+
+
+def dec_hook(type: type, obj: Any) -> Any:
+    """Decode `obj` for a field annotated with one of this module's field types.
+
+    Pass as `dec_hook=` to `msgspec.json.decode`, `msgspec.convert`, or a
+    `msgspec.json.Decoder`. An existing core model instance passes through
+    unchanged; anything else is parsed by the core model's `from_json`. A
+    type this module does not cover raises `NotImplementedError`, msgspec's
+    convention for "still unsupported"; to keep decoding custom types of
+    your own alongside these, chain your hook with `make_dec_hook`.
+    """
+    entry = _lookup(type)
+    if entry is None:
+        raise NotImplementedError(f"Objects of type {type} are not supported")
+    return _decode(entry, obj)
+
+
+def make_dec_hook(wrapped: Callable[[type, Any], Any] | None = None) -> Callable[[type, Any], Any]:
+    """Return a decode hook that also delegates unknown types to `wrapped`.
+
+    The returned hook handles this module's field types exactly like
+    `dec_hook` and hands every other type to `wrapped`, so an application's
+    existing custom-type decoding keeps working alongside the model field
+    types. With no `wrapped` hook this returns `dec_hook` itself.
+    """
+    if wrapped is None:
+        return dec_hook
+
+    def hook(type: type, obj: Any) -> Any:
+        entry = _lookup(type)
+        if entry is None:
+            return wrapped(type, obj)
+        return _decode(entry, obj)
+
+    return hook
+
+
+__all__ = [
+    "ZarrV2ArrayMetadata",
+    "ZarrV2ConsolidatedMetadata",
+    "ZarrV2GroupMetadata",
+    "ZarrV3ArrayMetadata",
+    "ZarrV3ConsolidatedMetadata",
+    "ZarrV3GroupMetadata",
+    "ZarrV3MetadataField",
+    "dec_hook",
+    "make_dec_hook",
+]

packages/zarr-metadata/tests/model/_cases.py

@@ -6,9 +6,28 @@
 
 import pytest
 
+from zarr_metadata.model import ZarrV2ArrayMetadata, ZarrV3ArrayMetadata
+
 if TYPE_CHECKING:
     from contextlib import AbstractContextManager
 
+# Canonical documents shared by the integration test modules
+# (test_pydantic_module.py, test_msgspec_module.py), so a model change that
+# alters a canonical document is corrected in one place.
+V3_ARRAY_DOC = dict(ZarrV3ArrayMetadata.create_default(shape=(4,)).to_json())
+V2_ARRAY_DOC = dict(ZarrV2ArrayMetadata.create_default(shape=(4,), chunks=(2,)).to_json())
+V3_GROUP_DOC = {"zarr_format": 3, "node_type": "group", "attributes": {"a": 1}}
+V2_GROUP_DOC = {"zarr_format": 2, "attributes": {"a": 1}}
+V3_CONSOLIDATED_DOC = {
+    "kind": "inline",
+    "must_understand": False,
+    "metadata": {"a": dict(V3_ARRAY_DOC)},
+}
+V2_CONSOLIDATED_DOC = {
+    "zarr_consolidated_format": 1,
+    "metadata": {".zgroup": {"zarr_format": 2}},
+}
+
 TIn = TypeVar("TIn")
 TOut = TypeVar("TOut")