feat(zarr-metadata): composition rules layer, shape-exact codec guards, typed builders

packages/zarr-metadata/changes/+array-v3-builder.feature.md

@@ -0,0 +1,33 @@
+Added `zarr_metadata.builder`: incremental, validated construction of v3
+array metadata documents over the plain JSON TypedDict shapes.
+`ZarrV3ArrayMetadataBuilder` accumulates a
+`ZarrV3ArrayMetadataJSONPartial` and hands out evolved copies:
+`evolve(**kwargs)` is the single fully-typed setter (unknown keys and
+wrong value types are static errors), `evolve_extension` sets extension
+fields, `without` unsets keys (absence is UNSET; a stored `None` is JSON
+`null`, and the two never convert), per-field properties answer
+`T | UNSET`, `build()` returns the validated complete document, and
+`to_partial_json()` returns the fragment under an honest partial type —
+there is no `build_unchecked`. Input arrays are materialized as tuples at
+ingestion (so documents straight from `json.loads` normalize, builders
+never share mutable state with callers, and equality is
+list/tuple-spelling-insensitive).
+
+Semantic rules live as data (`Rule`, keyed by the document keys they
+depend on) and fire eagerly after every change, whenever their
+dependencies are all present — so field order is unconstrained, coupled
+fields are checked as soon as they coexist, and a conflict names both
+fields plus the batch-`evolve` escape hatch. The initial rule set checks
+fill_value against the data type (with range checks for ints and the
+spec's special float spellings), codec pipeline kind ordering
+(classified by name across every spelling, so two spellings of the same
+pipeline always get the same verdict), known-name codec and chunk-grid
+spellings (bare short-hand only where the concrete type permits it,
+required configuration keys present — derived from the TypedDicts'
+`__required_keys__`, never restated by hand), `dimension_names` length,
+and `regular` chunk grid dimensionality. Rules never reject what they
+cannot interpret: unknown data types, codecs, and configurations pass
+through (extension openness) — but openness is for genuinely unknown
+names only; a name this package defines is held to its canonical
+spellings rather than passing as an unknowable extension. All problems
+from a check are reported together in one `MetadataValidationError`.

packages/zarr-metadata/changes/+codec-kind.feature.md

@@ -0,0 +1,19 @@
+Added `zarr_metadata.v3.codec.kind`: codec kind classification. One
+branded union per spec pipeline kind over the concrete codec types
+(`ArrayArrayCodecMetadata`, `ArrayBytesCodecMetadata`,
+`BytesBytesCodecMetadata`, plus `KnownCodecMetadata` and the paired
+`*_CODEC_NAMES` constants), and `TypeIs` guards
+(`is_array_array_codec`, `is_array_bytes_codec`, `is_bytes_bytes_codec`,
+`is_known_codec`) that narrow a codec entry to the matching union. The
+guards are **shape-exact**: `TypeIs` narrowing is two-sided, so a guard
+answers `True` exactly when the value is an instance of a canonical
+codec type — bare spellings only where the codec's spec permits them,
+object forms deep-checked against their TypedDicts (key sets and
+configuration value types, judged at the canonical data level with JSON
+arrays as tuples and `int` meaning JSON integer, not boolean). Unknown
+codec names answer `False` to every guard, so extension codecs classify
+as "unknown kind" rather than being misassigned. The companion
+`codec_kind_of_name` classifies by name alone, returning a `CodecKind`
+literal or `None`, for ordering semantics where a known codec in an
+invalid spelling must still rank as its kind. All names are re-exported
+from `zarr_metadata.v3.codec`.

packages/zarr-metadata/changes/+document-factories.feature.md

@@ -0,0 +1,31 @@
+Added `create_*` factories in `zarr_metadata.builder` — one per public
+document TypedDict (`create_zarr_v3_array_metadata_json`,
+`create_zarr_v3_group_metadata_json`, `create_zarr_v3_consolidated_metadata_json`,
+`create_zarr_v2_array_metadata_json`, `create_zarr_v2_group_metadata_json`,
+`create_zarr_v2_z_array_json`, `create_zarr_v2_z_group_json`,
+`create_zarr_v2_consolidated_metadata_json`), each taking
+`**kwargs: Unpack[<TypedDict>]`. At literal-keyword call sites, unpacking
+the total TypedDict makes a missing required key and a wrong value type
+static errors (a `**`-splatted mapping bypasses that coverage — the
+runtime pass exists for exactly those callers): each factory deep-copies
+its inputs, materializes JSON arrays as tuples, runs the structural
+validator and the composition rules, and raises one
+`MetadataValidationError` carrying every problem. The strict on-disk
+`.zarray`/`.zgroup` factories reject `attributes` at runtime, and the v2
+consolidated factory checks the `.zmetadata` envelope. The open v3
+array/group factories take an `extensions=` mapping for extension fields
+(the hatch for type checkers without PEP 728 support); extension names
+that shadow standard keys are rejected.
+
+This encodes a package rule — every public *document* TypedDict gets a
+factory — enforced by a drift test against the `DOCUMENT_FACTORIES`
+registry, so a new document type cannot ship without one. The rule is
+deliberately scoped to documents: entity TypedDicts (codec objects,
+configurations, ...) are constructed with TypedDict constructor syntax,
+which already enforces their shape statically, and no semantic rules
+apply to an entity in isolation.
+
+Prefer the factories for one-shot construction (the common case);
+`ZarrV3ArrayMetadataBuilder` remains for staged assembly across program
+points, where its eager rule firing and cross-call conflict attribution
+apply.

packages/zarr-metadata/changes/+entity-assignability.bugfix.md

@@ -0,0 +1,25 @@
+Concrete v3 entity types are now assignable to the fields they describe.
+Previously, none of the package's canonical codec / chunk-grid /
+chunk-key-encoding / data-type types (e.g. `BloscCodecMetadata`,
+`RegularChunkGridMetadata`) satisfied `ZarrV3MetadataFieldJSON`, so a
+type checker rejected putting them into the very fields they document
+(`codecs`, `chunk_grid`, `data_type`, ...). Three changes fix this:
+
+- `ZarrV3NamedConfigJSON.name` and `.configuration` are now `ReadOnly`
+  (PEP 705), making them covariant so concrete `name: Literal[...]` and
+  required-`configuration` shapes are accepted.
+- `ZarrV3NamedConfigJSON` is now `closed` (PEP 728): the spec's
+  named-configuration envelope has exactly `name` / `configuration` /
+  `must_understand`, and closing the type also makes it usable as a
+  `JSONValue` (needed for e.g. the `sharding_indexed` inner `codecs`).
+- Every concrete `*Object` / `*Configuration` TypedDict is now `closed`,
+  and object forms declare `must_understand: NotRequired[bool]` (any v3
+  metadata field may carry the extension member).
+
+**Soft-breaking** for type-checking consumers: dicts with keys beyond the
+declared shape no longer satisfy the closed types, and `name` /
+`configuration` can no longer be mutated through `ZarrV3NamedConfigJSON`.
+Both were previously accepted by type checkers but produced documents
+outside the spec's shapes. `zarr_metadata.pydantic` serializers now
+declare their return schema via the pydantic-facing shadow types, so
+pydantic schema generation stays warning-free.

packages/zarr-metadata/changes/+rules-layer.feature.md

@@ -0,0 +1,40 @@
+Added `zarr_metadata.rules`: composition rules for full metadata
+documents, promoting the rule engine out of the builder into a layer of
+its own. The package now models metadata in three layers with one
+contract each — `model` checks structure element by element,
+`rules` judges composition across the document, and `builder` constructs
+while applying both.
+
+- **Rule sets**: `ZARR_V3_ARRAY_RULES` grows to twelve rules — fill
+  value vs. data type, codec pipeline kind ordering, known-name shapes,
+  dimension-name counts, chunk-grid values (positive extents) and
+  geometry (regular rank; rectilinear rank and per-dimension chunk-size
+  sums, RLE pairs included), transpose orders (self-permutation at any
+  depth, rank agreement with `shape`), and sharding (inner `codecs` and
+  `index_codecs` judged as pipelines recursively at every nesting depth;
+  inner chunk shapes positive, rank-matched, and evenly dividing the
+  enclosing chunk, recursively). New `ZARR_V2_ARRAY_RULES`
+  (chunks/shape rank agreement) and `ZARR_V3_GROUP_RULES` (inline
+  consolidated metadata recurses, judging each embedded child document
+  by its own rules at its path).
+- **Read-side trios**: `validate_*` / `is_*` / `parse_*` for array and
+  group documents in both format versions mirror the model layer's
+  grammar with a stronger judgment — structure *and* composition, every
+  problem reported together, JSON arrays normalized to tuples before
+  judgment. The `is_*` functions deliberately return `bool` rather than
+  `TypeIs`: a composition-invalid document is still an instance of the
+  TypedDict, so only the structural layer can narrow honestly.
+- **Boundary change**: two composition checks that lived in the
+  structural validator moved here — v3 `dimension_names` vs `shape` and
+  v2 `chunks` vs `shape` rank agreement. `zarr_metadata.model`'s
+  validators, parsers, and dataclasses now accept those documents (they
+  are lossless, structurally well-formed representations of what a store
+  may contain); use the `rules` trios to judge them. This also removes
+  the double report the overlap used to produce.
+- **Strictness stance**, now documented on the package: `zarr_metadata`
+  models canonical documents and is deliberately stricter than any given
+  implementation; implementations coerce ambiguous input as they see fit
+  and then validate the canonical result.
+
+Known follow-up: v2 fill-value/dtype consistency (NumPy dtype grammar)
+has no rule yet.

packages/zarr-metadata/docs/api/builder.md

@@ -0,0 +1,5 @@
+---
+title: builder
+---
+
+::: zarr_metadata.builder

packages/zarr-metadata/docs/api/index.md

@@ -8,6 +8,13 @@ The package is organized to mirror the structure of the Zarr specifications:
 
 - [`zarr_metadata.model`](model.md) — frozen-dataclass document models,
   structural validators, loc-aware parsers, and the `UNSET` sentinel
+- [`zarr_metadata.rules`](rules.md) — composition rules: cross-field
+  judgments over full documents (fill value vs. data type, codec pipeline
+  ordering, chunk geometry), plus whole-document `validate`/`is`/`parse`
+  trios combining structure and composition
+- [`zarr_metadata.builder`](builder.md) — validated construction:
+  one-shot `create_*` factories per document type and an incremental
+  builder for v3 arrays
 - [`zarr_metadata.pydantic`](pydantic.md) — optional Pydantic field types
   over the models
 - [`zarr_metadata.v2`](v2.md) — `TypedDict` shapes for Zarr v2 documents

packages/zarr-metadata/docs/api/rules.md

@@ -0,0 +1,5 @@
+---
+title: rules
+---
+
+::: zarr_metadata.rules

packages/zarr-metadata/mkdocs.yml

@@ -16,6 +16,8 @@ nav:
   - API Reference:
       - api/index.md
       - '<code class="doc-symbol doc-symbol-toc doc-symbol-module"></code> <code>zarr_metadata.model</code>': api/model.md
+      - '<code class="doc-symbol doc-symbol-toc doc-symbol-module"></code> <code>zarr_metadata.rules</code>': api/rules.md
+      - '<code class="doc-symbol doc-symbol-toc doc-symbol-module"></code> <code>zarr_metadata.builder</code>': api/builder.md
       - '<code class="doc-symbol doc-symbol-toc doc-symbol-module"></code> <code>zarr_metadata.pydantic</code>': api/pydantic.md
       - '<code class="doc-symbol doc-symbol-toc doc-symbol-module"></code> <code>zarr_metadata.v2</code>': api/v2.md
       - '<code class="doc-symbol doc-symbol-toc doc-symbol-module"></code> <code>zarr_metadata.v3</code>':

packages/zarr-metadata/src/zarr_metadata/_common.py

@@ -9,7 +9,7 @@
 from collections.abc import Mapping
 from typing import NotRequired
 
-from typing_extensions import TypeAliasType, TypedDict
+from typing_extensions import ReadOnly, TypeAliasType, TypedDict
 
 JSONValue = TypeAliasType(
     "JSONValue",
@@ -31,14 +31,34 @@
 """
 
 
-class ZarrV3NamedConfigJSON(TypedDict):
+class ZarrV3NamedConfigJSON(TypedDict, closed=True):
     """
     Externally-tagged union member for a metadata field.
 
     The optional `configuration` mapping holds arbitrary JSON-encodable
     values. `must_understand` is implicitly true when absent.
+
+    `name` and `configuration` are `ReadOnly` (PEP 705) so that concrete
+    entity types — `BloscCodecObject`, `RegularChunkGridObject`, and the
+    rest — are assignable to this type, and therefore to
+    `ZarrV3MetadataFieldJSON`. Without `ReadOnly` both items are invariant,
+    so a concrete `name: Literal["blosc"]` does not satisfy `name: str`, and
+    a required `configuration` does not satisfy a `NotRequired` one. That
+    made the package's own codec types unusable in the very fields they
+    describe (`codecs`, `data_type`, `chunk_grid`, ...), and made
+    `TypeIs`-based codec classification impossible to declare, since `TypeIs`
+    requires the narrowed type to be assignable to the input type.
+
+    `must_understand` stays writable: nothing needs to narrow it, and
+    keeping it mutable lets writers set it on an already-constructed field.
+
+    The type is `closed` (PEP 728): the spec's named-configuration envelope
+    has exactly these three members, and closing it is also what makes this
+    type — and every concrete entity type embedding it, e.g. the
+    `sharding_indexed` configuration's inner `codecs` list — assignable to
+    `Mapping[str, JSONValue]` (i.e. usable as a `JSONValue`).
     """
 
-    name: str
-    configuration: NotRequired[Mapping[str, JSONValue]]
+    name: ReadOnly[str]
+    configuration: NotRequired[ReadOnly[Mapping[str, JSONValue]]]
     must_understand: NotRequired[bool]

packages/zarr-metadata/src/zarr_metadata/builder/__init__.py

@@ -0,0 +1,54 @@
+"""Validated construction of Zarr metadata documents.
+
+The construction layer is a consumer of the package's other two layers:
+the model layer checks structure, `zarr_metadata.rules` judges
+composition, and the surfaces here apply both while a document is being
+put together.
+
+Two construction surfaces over the plain JSON shapes defined in
+`zarr_metadata.v2` / `zarr_metadata.v3`:
+
+- **`create_*` factories** — one per public document TypedDict, taking
+  `**kwargs: Unpack[<TypedDict>]`. One-shot construction: at
+  literal-keyword call sites, required keys and value types are enforced
+  statically; the runtime pass normalizes, checks structure, and runs
+  the composition rules, raising one `MetadataValidationError` with
+  every problem. Prefer these when all fields are known at a single call
+  site — which is the common case. For validating documents you *read*
+  rather than construct, use the trios in `zarr_metadata.rules`.
+- **`ZarrV3ArrayMetadataBuilder`** — incremental accumulation for staged
+  assembly across program points. Composition rules fire eagerly as
+  fields land, whenever their dependencies are all present, so field
+  order is unconstrained, problems surface at the `evolve` call that
+  completes them, and a cross-call conflict names both fields involved.
+  Completeness can only be checked at `build()` time, at runtime — the
+  price of accumulating through a partial type. Incremental building is
+  currently implemented for v3 arrays only, the document type with the
+  richest cross-field coupling.
+"""
+
+from zarr_metadata.builder._array_v3 import ZarrV3ArrayMetadataBuilder
+from zarr_metadata.builder._create import (
+    DOCUMENT_FACTORIES,
+    create_zarr_v2_array_metadata_json,
+    create_zarr_v2_consolidated_metadata_json,
+    create_zarr_v2_group_metadata_json,
+    create_zarr_v2_z_array_json,
+    create_zarr_v2_z_group_json,
+    create_zarr_v3_array_metadata_json,
+    create_zarr_v3_consolidated_metadata_json,
+    create_zarr_v3_group_metadata_json,
+)
+
+__all__ = [
+    "DOCUMENT_FACTORIES",
+    "ZarrV3ArrayMetadataBuilder",
+    "create_zarr_v2_array_metadata_json",
+    "create_zarr_v2_consolidated_metadata_json",
+    "create_zarr_v2_group_metadata_json",
+    "create_zarr_v2_z_array_json",
+    "create_zarr_v2_z_group_json",
+    "create_zarr_v3_array_metadata_json",
+    "create_zarr_v3_consolidated_metadata_json",
+    "create_zarr_v3_group_metadata_json",
+]