diff --git a/codecs/n5_varlen/README.md b/codecs/n5_varlen/README.md new file mode 100644 index 0000000..69ae9bf --- /dev/null +++ b/codecs/n5_varlen/README.md @@ -0,0 +1,215 @@ +# n5_varlen codec + +Defines an `array -> bytes` codec which makes Zarr readers/writers compatible with +[N5 blocks](https://github.com/saalfeldlab/n5) in "varlength" mode (`0x0001`). + +This codec is the varlength-mode counterpart to the +[`n5_default`](../n5_default/README.md) codec, which handles N5 default-mode blocks. +It is intended for variable-width array data types — most notably the +[`label_multiset`](../../data-types/label_multiset/README.md) data type — where the +per-chunk byte size is not determined solely by the chunk shape and element size. + +## Codec name + +The value of the `name` member in the codec object MUST be `n5_varlen`. + +## Configuration parameters + +The `configuration` object MUST have a `codecs` key whose value is an array of Zarr v3 +codec objects. These inner codecs form a sub-pipeline that converts between the array and +the raw payload bytes (after the N5 header, before any compression). The inner pipeline +MUST begin with exactly one array-to-bytes codec followed by zero or more bytes-to-bytes +codecs. + +No additional fields are permitted in the configuration. + +## Compatibility notes + +### Storage + +Arrays can be made compatible with both N5 and Zarr by having an N5-style +`attributes.json` and a Zarr-style `zarr.json` under the same directory/prefix, following +the same pattern described in the [`n5_default`](../n5_default/README.md) codec. + +### Chunk key encoding + +Zarr arrays reading N5 data MUST use a `/`-separated +[v2 chunk key encoding](https://zarr-specs.readthedocs.io/en/latest/v3/chunk-key-encodings/v2/). + +### Compressors / bytes-to-bytes codecs + +Inner bytes-to-bytes codecs correspond to N5 block compressors. The same mapping table +described in [`n5_default`](../n5_default/README.md) applies. + +## Procedure + +### Decoding + +1. Parse the [N5 block header](#header-format). +2. Validate that the block mode is varlength (`0x0001`) and that the number of dimensions + in the header matches the number of dimensions of the Zarr array. +3. Read the next `numElements × element_size` bytes as the compressed payload. + For `label_multiset`, the element is a byte, so `numElements` equals the byte count. +4. Apply the inner codec pipeline (in decode order) to produce the decoded array. + +### Encoding + +1. Apply the inner codec pipeline (in encode order) to the array, producing a byte sequence. +2. Write the N5 block header with `numElements` set to `byte_count / element_size`. + For `label_multiset`, the element is a byte, so `numElements` equals the byte count. +3. Write the byte sequence. + +## Header format + +The N5 block header for varlength-mode blocks (copied from the +[N5 spec](https://github.com/saalfeldlab/n5)): + +``` +Offset Size Endian Field +------ ----- ------ ------------------------------------------ +0 2 BE mode uint16 must be 0x0001 (varlength) +2 2 BE ndim uint16 number of dimensions +4 4·ndim BE dims[0..ndim-1] uint32 each block shape +4+4·ndim 4 BE numElements uint32 number of elements in payload +``` + +Total header size: `2 + 2 + 4·ndim + 4` bytes. + +The `numElements` field records the number of elements in the payload, following the +[N5 spec](https://github.com/saalfeldlab/n5/tree/fb50c2c3f1b411abd201a6c701cfd9c61486cd85). +The byte length of the payload is `numElements × element_size`. For `label_multiset`, +the serialized payload is a byte stream (element size = 1), so `numElements` equals +the byte count of the payload and equals `file_size - header_size`. + +> **Note:** For default-mode N5 blocks, the header has no 4th field; after `dims[]`, +> the payload begins immediately. The varlength header adds this 4-byte `numElements` +> field, since the payload length cannot be derived from the chunk shape alone for +> variable-width data types. + +## Example + +### label_multiset data with no compression + +A label multiset array using N5 varlength blocks without compression. The inner codec +[`n5_label_multiset`](../n5_label_multiset/README.md) serializes the array to the N5 +legacy payload format; `n5_varlen` wraps the result with the varlength block header. + +```json +{ + "zarr_format": 3, + "node_type": "array", + "shape": [80, 64, 64], + "data_type": "label_multiset", + "chunk_grid": { + "name": "regular", + "configuration": { + "chunk_shape": [32, 32, 32] + } + }, + "chunk_key_encoding": { + "name": "v2", + "configuration": {"separator": "/"} + }, + "fill_value": "0xFFFFFFFFFFFFFFFE", + "codecs": [ + { + "name": "n5_varlen", + "configuration": { + "codecs": [ + {"name": "n5_label_multiset"} + ] + } + } + ] +} +``` + +### label_multiset data with gzip compression + +```json +{ + "zarr_format": 3, + "node_type": "array", + "shape": [80, 64, 64], + "data_type": "label_multiset", + "chunk_grid": { + "name": "regular", + "configuration": { + "chunk_shape": [32, 32, 32] + } + }, + "chunk_key_encoding": { + "name": "v2", + "configuration": {"separator": "/"} + }, + "fill_value": "0xFFFFFFFFFFFFFFFE", + "codecs": [ + { + "name": "n5_varlen", + "configuration": { + "codecs": [ + {"name": "n5_label_multiset"}, + {"name": "gzip", "configuration": {"level": 6}} + ] + } + } + ] +} +``` + +## Annotated binary layout + +The varlength block header format is specified in the +[N5 spec](https://github.com/saalfeldlab/n5/tree/fb50c2c3f1b411abd201a6c701cfd9c61486cd85). +The payload layout below is derived from the +[imglib2-label-multisets](https://github.com/saalfeldlab/imglib2-label-multisets) +Java reference implementation. + +For a 32×32×32 `label_multiset` chunk using `n5_label_multiset` with no compression, +the complete on-disk block is: + +``` +Bytes 0–1: 00 01 mode = 0x0001 (varlength, uint16 BE) +Bytes 2–3: 00 03 ndim = 3 (uint16 BE) +Bytes 4–7: 00 00 00 20 dims[0] = 32 (uint32 BE) +Bytes 8–11: 00 00 00 20 dims[1] = 32 (uint32 BE) +Bytes 12–15: 00 00 00 20 dims[2] = 32 (uint32 BE) +Bytes 16–19: XX XX XX XX numElements (uint32 BE) — element count (bytes, for label_multiset) +─── n5_label_multiset payload begins here ─────────────────────────────────────── +Bytes 20–23: 00 00 00 00 argMaxSize = 0 (int32 BE) +Bytes 24–131099: … listEntryOffsets[32768] (int32 BE each, 4·32768 bytes) +Bytes 131100–…: … listData (all little-endian) +``` + +## Limitations + +This codec can only decode N5 data which uses the varlength block mode (`0x0001`). + +This codec can only encode Zarr arrays using variable-width data types whose inner +array-to-bytes codec produces an output whose byte length cannot be inferred from the +chunk shape alone. + +## Background + +N5 defines three block modes: + +| Mode | Value | Description | +|------|-------|-------------| +| default | `0x0000` | Fixed-width elements; shape from header | +| varlength | `0x0001` | Variable-width payload; element count from header | +| object | `0x0002` | Arbitrary serialized objects | + +The `n5_default` codec handles mode `0x0000`. This codec handles mode `0x0001`. + +In N5's Java reference implementation, varlength blocks are used for `LabelMultisetType` +arrays written by imglib2-label-multisets. For `LabelMultisetType`, the serialized payload +is a byte stream (element size = 1), so the `numElements` field equals +`file_size - header_size`. + +## Change log + +No changes yet. + +## Current maintainers + +* [Mark Kittisopikul](https://github.com/mkitti) diff --git a/codecs/n5_varlen/schema.json b/codecs/n5_varlen/schema.json new file mode 100644 index 0000000..dab6966 --- /dev/null +++ b/codecs/n5_varlen/schema.json @@ -0,0 +1,48 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "properties": { + "name": { + "type": "string", + "const": "n5_varlen" + }, + "configuration": { + "type": "object", + "properties": { + "codecs": { + "type": "array", + "minItems": 1, + "items": { + "$ref": "#/$defs/codec" + } + } + }, + "required": ["codecs"], + "additionalProperties": false + } + }, + "required": ["name", "configuration"], + "additionalProperties": false, + "$defs": { + "codec": { + "oneOf": [ + { + "type": "object", + "properties": { + "name": { + "type": "string" + }, + "configuration": { + "type": "object" + } + }, + "required": ["name"], + "additionalProperties": false + }, + { + "type": "string" + } + ] + } + } +}