Add n5_varlen codec for N5 varlength block format

codecs/n5_varlen/README.md

@@ -0,0 +1,215 @@
+# n5_varlen codec
+
+Defines an `array -> bytes` codec which makes Zarr readers/writers compatible with
+[N5 blocks](https://github.com/saalfeldlab/n5) in "varlength" mode (`0x0001`).
+
+This codec is the varlength-mode counterpart to the
+[`n5_default`](../n5_default/README.md) codec, which handles N5 default-mode blocks.
+It is intended for variable-width array data types — most notably the
+[`label_multiset`](../../data-types/label_multiset/README.md) data type — where the
+per-chunk byte size is not determined solely by the chunk shape and element size.
+
+## Codec name
+
+The value of the `name` member in the codec object MUST be `n5_varlen`.
+
+## Configuration parameters
+
+The `configuration` object MUST have a `codecs` key whose value is an array of Zarr v3
+codec objects. These inner codecs form a sub-pipeline that converts between the array and
+the raw payload bytes (after the N5 header, before any compression). The inner pipeline
+MUST begin with exactly one array-to-bytes codec followed by zero or more bytes-to-bytes
+codecs.
+
+No additional fields are permitted in the configuration.
+
+## Compatibility notes
+
+### Storage
+
+Arrays can be made compatible with both N5 and Zarr by having an N5-style
+`attributes.json` and a Zarr-style `zarr.json` under the same directory/prefix, following
+the same pattern described in the [`n5_default`](../n5_default/README.md) codec.
+
+### Chunk key encoding
+
+Zarr arrays reading N5 data MUST use a `/`-separated
+[v2 chunk key encoding](https://zarr-specs.readthedocs.io/en/latest/v3/chunk-key-encodings/v2/).
+
+### Compressors / bytes-to-bytes codecs
+
+Inner bytes-to-bytes codecs correspond to N5 block compressors. The same mapping table
+described in [`n5_default`](../n5_default/README.md) applies.
+
+## Procedure
+
+### Decoding
+
+1. Parse the [N5 block header](#header-format).
+2. Validate that the block mode is varlength (`0x0001`) and that the number of dimensions
+   in the header matches the number of dimensions of the Zarr array.
+3. Read the next `numElements × element_size` bytes as the compressed payload.
+   For `label_multiset`, the element is a byte, so `numElements` equals the byte count.
+4. Apply the inner codec pipeline (in decode order) to produce the decoded array.
+
+### Encoding
+
+1. Apply the inner codec pipeline (in encode order) to the array, producing a byte sequence.
+2. Write the N5 block header with `numElements` set to `byte_count / element_size`.
+   For `label_multiset`, the element is a byte, so `numElements` equals the byte count.
+3. Write the byte sequence.
+
+## Header format
+
+The N5 block header for varlength-mode blocks (copied from the
+[N5 spec](https://github.com/saalfeldlab/n5)):
+
+```
+Offset  Size   Endian  Field
+------  -----  ------  ------------------------------------------
+0       2      BE      mode         uint16  must be 0x0001 (varlength)
+2       2      BE      ndim         uint16  number of dimensions
+4       4·ndim BE      dims[0..ndim-1]  uint32 each  block shape
+4+4·ndim  4    BE      numElements  uint32  number of elements in payload
+```
+
+Total header size: `2 + 2 + 4·ndim + 4` bytes.
+
+The `numElements` field records the number of elements in the payload, following the
+[N5 spec](https://github.com/saalfeldlab/n5/tree/fb50c2c3f1b411abd201a6c701cfd9c61486cd85).
+The byte length of the payload is `numElements × element_size`. For `label_multiset`,
+the serialized payload is a byte stream (element size = 1), so `numElements` equals
+the byte count of the payload and equals `file_size - header_size`.
+
+> **Note:** For default-mode N5 blocks, the header has no 4th field; after `dims[]`,
+> the payload begins immediately. The varlength header adds this 4-byte `numElements`
+> field, since the payload length cannot be derived from the chunk shape alone for
+> variable-width data types.
+
+## Example
+
+### label_multiset data with no compression
+
+A label multiset array using N5 varlength blocks without compression. The inner codec
+[`n5_label_multiset`](../n5_label_multiset/README.md) serializes the array to the N5
+legacy payload format; `n5_varlen` wraps the result with the varlength block header.
+
+```json
+{
+    "zarr_format": 3,
+    "node_type": "array",
+    "shape": [80, 64, 64],
+    "data_type": "label_multiset",
+    "chunk_grid": {
+        "name": "regular",
+        "configuration": {
+            "chunk_shape": [32, 32, 32]
+        }
+    },
+    "chunk_key_encoding": {
+        "name": "v2",
+        "configuration": {"separator": "/"}
+    },
+    "fill_value": "0xFFFFFFFFFFFFFFFE",
+    "codecs": [
+        {
+            "name": "n5_varlen",
+            "configuration": {
+                "codecs": [
+                    {"name": "n5_label_multiset"}
+                ]
+            }
+        }
+    ]
+}
+```
+
+### label_multiset data with gzip compression
+
+```json
+{
+    "zarr_format": 3,
+    "node_type": "array",
+    "shape": [80, 64, 64],
+    "data_type": "label_multiset",
+    "chunk_grid": {
+        "name": "regular",
+        "configuration": {
+            "chunk_shape": [32, 32, 32]
+        }
+    },
+    "chunk_key_encoding": {
+        "name": "v2",
+        "configuration": {"separator": "/"}
+    },
+    "fill_value": "0xFFFFFFFFFFFFFFFE",
+    "codecs": [
+        {
+            "name": "n5_varlen",
+            "configuration": {
+                "codecs": [
+                    {"name": "n5_label_multiset"},
+                    {"name": "gzip", "configuration": {"level": 6}}
+                ]
+            }
+        }
+    ]
+}
+```
+
+## Annotated binary layout
+
+The varlength block header format is specified in the
+[N5 spec](https://github.com/saalfeldlab/n5/tree/fb50c2c3f1b411abd201a6c701cfd9c61486cd85).
+The payload layout below is derived from the
+[imglib2-label-multisets](https://github.com/saalfeldlab/imglib2-label-multisets)
+Java reference implementation.
+
+For a 32×32×32 `label_multiset` chunk using `n5_label_multiset` with no compression,
+the complete on-disk block is:
+
+```
+Bytes 0–1:       00 01              mode = 0x0001 (varlength, uint16 BE)
+Bytes 2–3:       00 03              ndim = 3 (uint16 BE)
+Bytes 4–7:       00 00 00 20        dims[0] = 32 (uint32 BE)
+Bytes 8–11:      00 00 00 20        dims[1] = 32 (uint32 BE)
+Bytes 12–15:     00 00 00 20        dims[2] = 32 (uint32 BE)
+Bytes 16–19:     XX XX XX XX        numElements (uint32 BE) — element count (bytes, for label_multiset)
+─── n5_label_multiset payload begins here ───────────────────────────────────────
+Bytes 20–23:     00 00 00 00        argMaxSize = 0 (int32 BE)
+Bytes 24–131099: …                  listEntryOffsets[32768] (int32 BE each, 4·32768 bytes)
+Bytes 131100–…:  …                  listData (all little-endian)
+```
+
+## Limitations
+
+This codec can only decode N5 data which uses the varlength block mode (`0x0001`).
+
+This codec can only encode Zarr arrays using variable-width data types whose inner
+array-to-bytes codec produces an output whose byte length cannot be inferred from the
+chunk shape alone.
+
+## Background
+
+N5 defines three block modes:
+
+| Mode | Value | Description |
+|------|-------|-------------|
+| default | `0x0000` | Fixed-width elements; shape from header |
+| varlength | `0x0001` | Variable-width payload; element count from header |
+| object | `0x0002` | Arbitrary serialized objects |
+
+The `n5_default` codec handles mode `0x0000`. This codec handles mode `0x0001`.
+
+In N5's Java reference implementation, varlength blocks are used for `LabelMultisetType`
+arrays written by imglib2-label-multisets. For `LabelMultisetType`, the serialized payload
+is a byte stream (element size = 1), so the `numElements` field equals
+`file_size - header_size`.
+
+## Change log
+
+No changes yet.
+
+## Current maintainers
+
+* [Mark Kittisopikul](https://github.com/mkitti)

codecs/n5_varlen/schema.json

@@ -0,0 +1,48 @@
+{
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "type": "object",
+  "properties": {
+    "name": {
+      "type": "string",
+      "const": "n5_varlen"
+    },
+    "configuration": {
+      "type": "object",
+      "properties": {
+        "codecs": {
+          "type": "array",
+          "minItems": 1,
+          "items": {
+            "$ref": "#/$defs/codec"
+          }
+        }
+      },
+      "required": ["codecs"],
+      "additionalProperties": false
+    }
+  },
+  "required": ["name", "configuration"],
+  "additionalProperties": false,
+  "$defs": {
+    "codec": {
+      "oneOf": [
+        {
+          "type": "object",
+          "properties": {
+            "name": {
+              "type": "string"
+            },
+            "configuration": {
+              "type": "object"
+            }
+          },
+          "required": ["name"],
+          "additionalProperties": false
+        },
+        {
+          "type": "string"
+        }
+      ]
+    }
+  }
+}