Merge branch 'main' into claude/structured-dtype-bugfixes

Author: d-v-b

packages/zarr-indexing/changes/4347.bugfix.md

@@ -0,0 +1 @@
+Validate inclusive `index_array_bounds` against all raw index values when loading transforms and output maps from JSON. Accept valid finite and one-sided constraints, and reject out-of-bounds values eagerly before offset, stride, or map simplification. Validated immutable maps need not retain the constraints; message normalization preserves the original bounds.

packages/zarr-indexing/changes/4348.bugfix.md

@@ -0,0 +1 @@
+Delegate LazyArray source tokenization to Dask, honoring its registered normalizers, source hooks, and deterministic-token requirements. Remove local content-hashing and UUID fallbacks. Dask remains optional for indexing and reading.

packages/zarr-indexing/docs/guide/integrations.md

@@ -243,3 +243,14 @@ that implementation or reproduce its full worker/GPU lifecycle.
 ·
 **API:** [API reference](../api/index.md)
 </nav>
+
+## Dask tokenization
+
+`LazyArray.__dask_tokenize__()` combines Dask's token for the wrapped source
+with the serialized view transform. Dask owns source hashing, registered
+normalizers, custom source hooks, and deterministic-token requirements.
+Tokenization may read or hash source values. Dask is optional for indexing
+and reading, but required when requesting a Dask token.
+
+The reader and partitioning are omitted because they must preserve values.
+Changing a source after graph construction does not update existing Dask keys.

packages/zarr-indexing/docs/ndsel.md

@@ -99,8 +99,22 @@ assert t.to_json() == canonical
 map kind has a `to_json`; `output_index_map_from_json` dispatches the wire's
 structurally discriminated union back to the right kind. Exact JSON equality
 in this example is not a general round-trip guarantee: implicit flags are
-removed, finite `index_array_bounds` are not retained or enforced by the
-engine, and degenerate array maps are collapsed.
+removed and degenerate array maps are collapsed.
+
+`index_array_bounds` constrains raw index-array values before the map's offset
+and stride are applied. Both `IndexTransform.from_json` and
+`output_index_map_from_json` validate every supplied value against the inclusive
+bounds when loading. Finite and one-sided bounds are supported; omitted bounds
+and `["-inf", "+inf"]` impose no additional constraint. Values outside the
+bounds raise `NdselError("invalid_json", ...)`, including in singleton arrays
+and zero-stride maps. Empty arrays satisfy any well-formed, ordered bounds.
+
+Validation is eager: an invalid entry rejects the entire map even if a later
+selection would avoid that entry. After validation the engine owns immutable
+index coordinates, so it need not retain the bounds; serialization emits
+unbounded constraints for non-degenerate maps. Message normalization preserves
+the original bounds without checking array contents. This implementation does
+not defer bounds errors until individual positions are accessed.
 
 A canonical body carrying a
 `"-inf"` or `"+inf"` bound cannot be lowered — an `IndexDomain` addresses a

packages/zarr-indexing/pyproject.toml

@@ -99,9 +99,8 @@ extend = "../../pyproject.toml"
 target-version = "py312"
 
 [tool.ruff.lint.per-file-ignores]
-# Chunk discovery and __dask_tokenize__ deliberately catch Exception: a
-# foreign source's attributes or token hooks may fail, so these paths provide
-# fallback metadata or tokens when ordinary exceptions occur. Configured here (not as
+# Chunk discovery deliberately catches Exception: a foreign source's
+# attributes may fail, so this path provides fallback metadata. Configured here (not as
 # noqa comments) because different ruff versions
 # have differed on whether these rules fire; RUF100 can remove unused noqa comments.
 "src/zarr_indexing/lazy_array.py" = ["BLE001", "S110"]

packages/zarr-indexing/src/zarr_indexing/_wire.py

@@ -2,9 +2,9 @@
 
 Package-private: the types that serialize themselves (`IndexDomain`,
 `IndexTransform`, the output map kinds) all need these, so they cannot live in
-any one of them, and they are not API. The three engine constraints named in
-[`zarr_indexing.json`][zarr_indexing.json] — finite bounds, implicit bounds
-lowering by value, integer `index_array` content — are enforced here.
+any one of them, and they are not API. Domain bounds, index-array bounds,
+implicit bounds lowering by value, and integer `index_array` content are
+handled here, as described in [`zarr_indexing.json`][zarr_indexing.json].
 """
 
 from __future__ import annotations
@@ -13,13 +13,37 @@
 
 import numpy as np
 
-from zarr_indexing.messages import NdselError
+from zarr_indexing.messages import NdselError, validate_index_array_bounds
 
 if TYPE_CHECKING:
     from zarr_indexing.domain import IndexDomain
     from zarr_indexing.json import BoundJSON
 
 
+def check_index_array_bounds(array: np.ndarray[Any, Any], bounds: Any, where: str) -> None:
+    """Validate every raw index value against an inclusive interval.
+
+    Validate interval syntax even for empty arrays. Once checked, immutable
+    index coordinates need no retained constraint. Use Python integer extrema
+    to avoid overflow or floating-point rounding at integer limits.
+    """
+    lo, hi = validate_index_array_bounds(bounds, where)
+    if array.size == 0 or (lo == "-inf" and hi == "+inf"):
+        return
+    minimum, maximum = int(array.min()), int(array.max())
+    if (
+        lo == "+inf"
+        or hi == "-inf"
+        or (isinstance(lo, int) and minimum < lo)
+        or (isinstance(hi, int) and maximum > hi)
+    ):
+        raise NdselError(
+            "invalid_json",
+            f"{where}.index_array values [{minimum}, {maximum}] are outside "
+            f"index_array_bounds {bounds!r}",
+        )
+
+
 def lower_bound(bound: BoundJSON, where: str) -> int:
     """Lower a canonical bound to a finite integer, rejecting infinities.
 

packages/zarr-indexing/src/zarr_indexing/json.py

@@ -14,6 +14,9 @@
 
 The engine lowering rules include:
 
+- **Validated index arrays.** Raw index values must satisfy the inclusive
+  `index_array_bounds` before offset and stride are applied. Lowering checks
+  all supplied values eagerly; validated immutable maps need not retain bounds.
 - **Finite bounds.** An `IndexDomain` addresses a finite array, so a canonical
   body carrying a `"-inf"`/`"+inf"` bound cannot be lowered; `from_json` raises.
 - **Implicit bounds lower by value.** The `[n]`-bracket implicit/explicit flag
@@ -132,8 +135,9 @@ class OutputIndexMapJSON(TypedDict, total=False):
     index_array_bounds: list[IndexValueJSON]
     """Wire bounds on index-array values; `["-inf", "+inf"]` if unconstrained.
 
-    The engine currently discards this field on load and does not enforce
-    finite bounds against the array values. Serialization emits unconstrained bounds.
+    The message layer preserves these inclusive constraints on raw index values.
+    Engine lowering validates all values eagerly before offset and stride.
+    Serialization emits unconstrained bounds for validated non-degenerate maps.
     """
 
 

packages/zarr-indexing/src/zarr_indexing/lazy_array.py

@@ -12,8 +12,8 @@
 ```
 
 Selection construction does not read source values: `result()`, `__array__`,
-and eager `__getitem__` perform reads. Tokenization can also read or hash source
-data, depending on the source and tokenization path. `.lazy` operations inspect
+and eager `__getitem__` perform reads. Source tokenization is delegated to
+Dask and may inspect source values. `.lazy` operations inspect
 selection metadata and may copy or process supplied index arrays. Composition
 does not accumulate wrapper layers: a view of a view is still a single transform
 and retains its reader.
@@ -138,7 +138,6 @@
 import json
 import math
 import operator
-import uuid
 from collections.abc import Sequence
 from dataclasses import dataclass, field
 from typing import TYPE_CHECKING, Any, Protocol, cast
@@ -173,10 +172,6 @@
 
 __all__ = ["LazyArray", "Partition"]
 
-# Above this declared byte count, the no-Dask token fallback adds a fresh UUID
-# instead of digesting contents. See `_wrapped_token`.
-_TOKEN_DIGEST_LIMIT = 1 << 20
-
 
 def _invoke_reader(
     reader: Reader,
@@ -525,66 +520,6 @@ def _validate_prepared_parts(parts: Sequence[Partition], out_shape: tuple[int, .
         raise ValueError("prepared parts do not tile the view exactly")
 
 
-# --------------------------------------------------------------------------- #
-# Tokenization
-# --------------------------------------------------------------------------- #
-
-
-def _wrapped_token(array: Any) -> Any:
-    """A token for the wrapped array.
-
-    In order of preference: the array's own `__dask_tokenize__`;
-    `dask.base.tokenize` when dask is importable (imported lazily — this package
-    never requires it); otherwise a local fallback that digests the contents of
-    a small array.
-
-    Tokens can differ depending on whether Dask is available and on the source
-    hook. This fallback does not provide a portable content identifier.
-
-    Above `_TOKEN_DIGEST_LIMIT`, or when conversion is unavailable, the local
-    fallback adds a fresh UUID on each call, so repeated calls normally differ.
-    The returned tuple is still equal to itself. Below the limit, conversion
-    can read the source, and the hash is of its NumPy buffer bytes. Object-array
-    buffer bytes contain object references, not a recursive content snapshot.
-    """
-    hook = getattr(array, "__dask_tokenize__", None)
-    if hook is not None:
-        try:
-            return hook()
-        # A failing source hook falls through to the remaining tokenization paths.
-        except Exception:  # pragma: no cover - a hook that refuses to run
-            pass
-    try:
-        # dask is an optional peer, never a dependency of this package, so it is
-        # imported here and its absence is ordinary.
-        from dask.base import tokenize  # pyright: ignore[reportMissingImports]
-    except ImportError:
-        pass
-    else:
-        return tokenize(array)
-
-    shape = tuple(int(s) for s in getattr(array, "shape", ()))
-    dtype = getattr(array, "dtype", None)
-    structural = (type(array).__qualname__, shape, str(dtype))
-    # A fresh identifier per call when contents cannot be identified. It
-    # is the shape and dtype that would otherwise be mistaken for an identity,
-    # so they are kept alongside it for a reader looking at a graph.
-    unidentified = (*structural, "unidentified", uuid.uuid4().hex)
-
-    # Decide whether to digest the contents from the *declared* size. Measuring
-    # it by converting first would read the whole array — a multi-gigabyte store
-    # pulled into memory by a token call, which is the opposite of the point.
-    itemsize = getattr(dtype, "itemsize", None)
-    if not isinstance(itemsize, int) or itemsize * math.prod(shape) > _TOKEN_DIGEST_LIMIT:
-        return unidentified
-    try:
-        contents = np.ascontiguousarray(array)
-    # Failed NumPy conversion leaves this source unidentified.
-    except Exception:
-        return unidentified
-    return (*structural, hashlib.sha256(contents.tobytes()).hexdigest())
-
-
 # --------------------------------------------------------------------------- #
 # The wrapper
 # --------------------------------------------------------------------------- #
@@ -1249,15 +1184,18 @@ def __dask_tokenize__(self) -> Any:
         tokens produce equal tokens; arbitrary semantically equivalent mappings
         are not guaranteed to serialize identically.
 
-        Source tokenization can read or hash data and need not be deterministic
-        on every fallback path; see `_wrapped_token`. The reader and partitioning
-        are omitted under the contract that they preserve values. Cache users
-        must also account for source mutation and the source's token semantics.
+        Dask tokenizes the wrapped source using its normal dispatch and
+        determinism policy. This may read or hash source values. Dask is
+        imported only when this method is called and is otherwise optional.
+        The reader and partitioning are omitted because they must preserve
+        values. Mutating a source does not update keys in existing Dask graphs.
         """
+        from dask.base import tokenize  # pyright: ignore[reportMissingImports]
+
         canonical = json.dumps(self._transform.to_json(), sort_keys=True)
         return (
             type(self).__qualname__,
-            _wrapped_token(self._array),
+            tokenize(self._array),
             hashlib.sha256(canonical.encode()).hexdigest(),
         )
 

packages/zarr-indexing/src/zarr_indexing/messages.py

@@ -556,7 +556,7 @@ def _normalize_output_map(raw: Any, where: str) -> dict[str, Any]:
     if has_index_array:
         stride = _check_int(raw["stride"], f"{where}.stride") if "stride" in raw else 1
         bounds = (
-            _check_index_array_bounds(raw["index_array_bounds"], where)
+            validate_index_array_bounds(raw["index_array_bounds"], where)
             if "index_array_bounds" in raw
             else ["-inf", "+inf"]
         )
@@ -585,7 +585,8 @@ def _normalize_output_map(raw: Any, where: str) -> dict[str, Any]:
     return {"offset": offset}
 
 
-def _check_index_array_bounds(value: Any, where: str) -> list[int | str]:
+def validate_index_array_bounds(value: Any, where: str) -> list[int | str]:
+    """Validate the syntax and ordering of an inclusive index-array interval."""
     if not isinstance(value, list) or len(value) != 2:
         raise NdselError(
             "invalid_json",

packages/zarr-indexing/src/zarr_indexing/output_map.py

@@ -357,18 +357,24 @@ def output_index_map_from_json(data: OutputIndexMapJSON) -> OutputIndexMap:
     selects an array map, `input_dimension` selects a dimension map, and
     neither selects a constant map.
 
+    Validate every raw index value against the inclusive `index_array_bounds`
+    before applying offset and stride. Out-of-bounds values raise `NdselError`
+    at load time. Validated immutable maps do not retain the bounds.
+
     Examples
     --------
     >>> output_index_map_from_json({"offset": 5})
     ConstantMap(offset=5)
     >>> output_index_map_from_json({"offset": 0, "stride": 2, "input_dimension": 1})
     DimensionMap(input_dimension=1, offset=0, stride=2)
     """
-    from zarr_indexing._wire import lower_index_array
+    from zarr_indexing._wire import check_index_array_bounds, lower_index_array
 
     if "index_array" in data:
+        array = lower_index_array(data["index_array"], "index_array")
+        check_index_array_bounds(array, data.get("index_array_bounds", ["-inf", "+inf"]), "output")
         return ArrayMap(
-            index_array=lower_index_array(data["index_array"], "index_array"),
+            index_array=array,
             offset=data.get("offset", 0),
             stride=data.get("stride", 1),
         )