From df4eb10a411ad65cf444ae541ff2f0a556b380aa Mon Sep 17 00:00:00 2001 From: Mark Kittisopikul Date: Mon, 13 Apr 2026 14:52:37 -0400 Subject: [PATCH 1/4] Add n5_varlen codec for N5 varlength block format Defines an array-to-bytes codec that wraps an inner codec pipeline with the N5 varlength block header (mode=0x0001). Intended for variable-width data types such as label_multiset where the per-chunk byte size cannot be derived from the chunk shape alone. Co-Authored-By: Claude Sonnet 4.6 --- codecs/n5_varlen/README.md | 204 +++++++++++++++++++++++++++++++++++ codecs/n5_varlen/schema.json | 48 +++++++++ 2 files changed, 252 insertions(+) create mode 100644 codecs/n5_varlen/README.md create mode 100644 codecs/n5_varlen/schema.json diff --git a/codecs/n5_varlen/README.md b/codecs/n5_varlen/README.md new file mode 100644 index 0000000..bfedd86 --- /dev/null +++ b/codecs/n5_varlen/README.md @@ -0,0 +1,204 @@ +# n5_varlen codec + +Defines an `array -> bytes` codec which makes Zarr readers/writers compatible with +[N5 blocks](https://github.com/saalfeldlab/n5) in "varlength" mode (`0x0001`). + +This codec is the varlength-mode counterpart to the +[`n5_default`](../n5_default/README.md) codec, which handles N5 default-mode blocks. +It is intended for variable-width array data types — most notably the +[`label_multiset`](../../data-types/label_multiset/README.md) data type — where the +per-chunk byte size is not determined solely by the chunk shape and element size. + +## Codec name + +The value of the `name` member in the codec object MUST be `n5_varlen`. + +## Configuration parameters + +The `configuration` object MUST have a `codecs` key whose value is an array of Zarr v3 +codec objects. These inner codecs form a sub-pipeline that converts between the array and +the raw payload bytes (after the N5 header, before any compression). The inner pipeline +MUST begin with exactly one array-to-bytes codec followed by zero or more bytes-to-bytes +codecs. + +No additional fields are permitted in the configuration. + +## Compatibility notes + +### Storage + +Arrays can be made compatible with both N5 and Zarr by having an N5-style +`attributes.json` and a Zarr-style `zarr.json` under the same directory/prefix, following +the same pattern described in the [`n5_default`](../n5_default/README.md) codec. + +### Chunk key encoding + +Zarr arrays reading N5 data MUST use a `/`-separated +[v2 chunk key encoding](https://zarr-specs.readthedocs.io/en/latest/v3/chunk-key-encodings/v2/). + +### Compressors / bytes-to-bytes codecs + +Inner bytes-to-bytes codecs correspond to N5 block compressors. The same mapping table +described in [`n5_default`](../n5_default/README.md) applies. + +## Procedure + +### Decoding + +1. Parse the [N5 block header](#header-format). +2. Validate that the block mode is varlength (`0x0001`) and that the number of dimensions + in the header matches the number of dimensions of the Zarr array. +3. Read the next `num_bytes` bytes as the compressed payload. +4. Apply the inner codec pipeline (in decode order) to produce the decoded array. + +### Encoding + +1. Apply the inner codec pipeline (in encode order) to the array, producing a byte sequence. +2. Write the N5 block header with `num_bytes` set to the length of the byte sequence. +3. Write the byte sequence. + +## Header format + +The N5 block header for varlength-mode blocks (copied from the +[N5 spec](https://github.com/saalfeldlab/n5)): + +``` +Offset Size Endian Field +------ ----- ------ ------------------------------------------ +0 2 BE mode uint16 must be 0x0001 (varlength) +2 2 BE ndim uint16 number of dimensions +4 4·ndim BE dims[0..ndim-1] uint32 each block shape +4+4·ndim 4 BE num_bytes uint32 byte length of payload +``` + +Total header size: `2 + 2 + 4·ndim + 4` bytes. + +The `num_bytes` field records the total number of bytes in the payload that follows the +header (i.e., the encoded output of the inner codec pipeline). For uncompressed data this +equals `file_size - header_size`. + +> **Note:** For default-mode N5 blocks, the header omits `num_bytes` and instead contains +> the number of array elements. The varlength header always includes `num_bytes` in place +> of an element count, since the number of bytes cannot be derived from the chunk shape +> for variable-width data types. + +## Example + +### label_multiset data with no compression + +A label multiset array using N5 varlength blocks without compression. The inner codec +[`n5_label_multiset`](../n5_label_multiset/README.md) serializes the array to the N5 +legacy payload format; `n5_varlen` wraps the result with the varlength block header. + +```json +{ + "zarr_format": 3, + "node_type": "array", + "shape": [80, 64, 64], + "data_type": "label_multiset", + "chunk_grid": { + "name": "regular", + "configuration": { + "chunk_shape": [32, 32, 32] + } + }, + "chunk_key_encoding": { + "name": "v2", + "configuration": {"separator": "/"} + }, + "fill_value": "0xFFFFFFFFFFFFFFFE", + "codecs": [ + { + "name": "n5_varlen", + "configuration": { + "codecs": [ + {"name": "n5_label_multiset"} + ] + } + } + ] +} +``` + +### label_multiset data with gzip compression + +```json +{ + "zarr_format": 3, + "node_type": "array", + "shape": [80, 64, 64], + "data_type": "label_multiset", + "chunk_grid": { + "name": "regular", + "configuration": { + "chunk_shape": [32, 32, 32] + } + }, + "chunk_key_encoding": { + "name": "v2", + "configuration": {"separator": "/"} + }, + "fill_value": "0xFFFFFFFFFFFFFFFE", + "codecs": [ + { + "name": "n5_varlen", + "configuration": { + "codecs": [ + {"name": "n5_label_multiset"}, + {"name": "gzip", "configuration": {"level": 6}} + ] + } + } + ] +} +``` + +## Annotated binary layout + +For a 32×32×32 `label_multiset` chunk using `n5_label_multiset` with no compression, +the complete on-disk block is: + +``` +Bytes 0–1: 00 01 mode = 0x0001 (varlength, uint16 BE) +Bytes 2–3: 00 03 ndim = 3 (uint16 BE) +Bytes 4–7: 00 00 00 20 dims[0] = 32 (uint32 BE) +Bytes 8–11: 00 00 00 20 dims[1] = 32 (uint32 BE) +Bytes 12–15: 00 00 00 20 dims[2] = 32 (uint32 BE) +Bytes 16–19: XX XX XX XX num_bytes (uint32 BE) — byte count of payload +─── n5_label_multiset payload begins here ─────────────────────────────────────── +Bytes 20–23: 00 00 00 00 argMaxSize = 0 (int32 BE) +Bytes 24–131099: … listEntryOffsets[32768] (int32 BE each, 4·32768 bytes) +Bytes 131100–…: … listData (all little-endian) +``` + +## Limitations + +This codec can only decode N5 data which uses the varlength block mode (`0x0001`). + +This codec can only encode Zarr arrays using variable-width data types whose inner +array-to-bytes codec produces an output whose byte length cannot be inferred from the +chunk shape alone. + +## Background + +N5 defines three block modes: + +| Mode | Value | Description | +|------|-------|-------------| +| default | `0x0000` | Fixed-width elements; shape from header | +| varlength | `0x0001` | Variable-width payload; byte count from header | +| object | `0x0002` | Arbitrary serialized objects | + +The `n5_default` codec handles mode `0x0000`. This codec handles mode `0x0001`. + +In N5's Java reference implementation, varlength blocks are used for `LabelMultisetType` +arrays written by imglib2-label-multisets. The `num_bytes` field in the header equals +`file_size - header_size` for uncompressed data. + +## Change log + +No changes yet. + +## Current maintainers + +* [Mark Kittisopikul](https://github.com/mkitti) diff --git a/codecs/n5_varlen/schema.json b/codecs/n5_varlen/schema.json new file mode 100644 index 0000000..dab6966 --- /dev/null +++ b/codecs/n5_varlen/schema.json @@ -0,0 +1,48 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "properties": { + "name": { + "type": "string", + "const": "n5_varlen" + }, + "configuration": { + "type": "object", + "properties": { + "codecs": { + "type": "array", + "minItems": 1, + "items": { + "$ref": "#/$defs/codec" + } + } + }, + "required": ["codecs"], + "additionalProperties": false + } + }, + "required": ["name", "configuration"], + "additionalProperties": false, + "$defs": { + "codec": { + "oneOf": [ + { + "type": "object", + "properties": { + "name": { + "type": "string" + }, + "configuration": { + "type": "object" + } + }, + "required": ["name"], + "additionalProperties": false + }, + { + "type": "string" + } + ] + } + } +} From 17369ab0e0aa7499c6ff803204b6c865a506b354 Mon Sep 17 00:00:00 2001 From: Mark Kittisopikul Date: Wed, 6 May 2026 16:03:00 -0400 Subject: [PATCH 2/4] Fix incorrect note: default-mode N5 header has no 4th field Per the N5 spec, the varlength `num_bytes` field is conditionally present only in varlength-mode blocks. Default-mode headers end after dims[] with no extra field. Co-Authored-By: Claude Sonnet 4.6 --- codecs/n5_varlen/README.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/codecs/n5_varlen/README.md b/codecs/n5_varlen/README.md index bfedd86..84e769c 100644 --- a/codecs/n5_varlen/README.md +++ b/codecs/n5_varlen/README.md @@ -77,10 +77,10 @@ The `num_bytes` field records the total number of bytes in the payload that foll header (i.e., the encoded output of the inner codec pipeline). For uncompressed data this equals `file_size - header_size`. -> **Note:** For default-mode N5 blocks, the header omits `num_bytes` and instead contains -> the number of array elements. The varlength header always includes `num_bytes` in place -> of an element count, since the number of bytes cannot be derived from the chunk shape -> for variable-width data types. +> **Note:** For default-mode N5 blocks, the header has no 4th field; after `dims[]`, +> the payload begins immediately. The varlength header adds this 4-byte `num_bytes` +> field to record the payload length, since it cannot be derived from the chunk shape +> alone for variable-width data types. ## Example From 7b6d31f69489ee9411d462e417847b5137f2dd63 Mon Sep 17 00:00:00 2001 From: Mark Kittisopikul Date: Wed, 6 May 2026 16:03:34 -0400 Subject: [PATCH 3/4] Add canonical source links to annotated binary layout section Links the N5 spec commit (same reference used by n5_default) and the imglib2-label-multisets Java reference implementation. Co-Authored-By: Claude Sonnet 4.6 --- codecs/n5_varlen/README.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/codecs/n5_varlen/README.md b/codecs/n5_varlen/README.md index 84e769c..2f060c5 100644 --- a/codecs/n5_varlen/README.md +++ b/codecs/n5_varlen/README.md @@ -155,6 +155,12 @@ legacy payload format; `n5_varlen` wraps the result with the varlength block hea ## Annotated binary layout +The varlength block header format is specified in the +[N5 spec](https://github.com/saalfeldlab/n5/tree/fb50c2c3f1b411abd201a6c701cfd9c61486cd85). +The payload layout below is derived from the +[imglib2-label-multisets](https://github.com/saalfeldlab/imglib2-label-multisets) +Java reference implementation. + For a 32×32×32 `label_multiset` chunk using `n5_label_multiset` with no compression, the complete on-disk block is: From 3c51863830883244d383849b7b1c3b0462af9332 Mon Sep 17 00:00:00 2001 From: Mark Kittisopikul Date: Wed, 6 May 2026 16:35:51 -0400 Subject: [PATCH 4/4] =?UTF-8?q?Rename=20num=5Fbytes=20=E2=86=92=20numEleme?= =?UTF-8?q?nts=20to=20match=20N5=20spec;=20clarify=20element=20size?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The N5 spec names this field 'number of elements'. For label_multiset, the serialized payload is a byte stream (element size = 1), so numElements equals the byte count. The procedure steps now note the element_size factor explicitly. Co-Authored-By: Claude Sonnet 4.6 --- codecs/n5_varlen/README.md | 31 ++++++++++++++++++------------- 1 file changed, 18 insertions(+), 13 deletions(-) diff --git a/codecs/n5_varlen/README.md b/codecs/n5_varlen/README.md index 2f060c5..69ae9bf 100644 --- a/codecs/n5_varlen/README.md +++ b/codecs/n5_varlen/README.md @@ -48,13 +48,15 @@ described in [`n5_default`](../n5_default/README.md) applies. 1. Parse the [N5 block header](#header-format). 2. Validate that the block mode is varlength (`0x0001`) and that the number of dimensions in the header matches the number of dimensions of the Zarr array. -3. Read the next `num_bytes` bytes as the compressed payload. +3. Read the next `numElements × element_size` bytes as the compressed payload. + For `label_multiset`, the element is a byte, so `numElements` equals the byte count. 4. Apply the inner codec pipeline (in decode order) to produce the decoded array. ### Encoding 1. Apply the inner codec pipeline (in encode order) to the array, producing a byte sequence. -2. Write the N5 block header with `num_bytes` set to the length of the byte sequence. +2. Write the N5 block header with `numElements` set to `byte_count / element_size`. + For `label_multiset`, the element is a byte, so `numElements` equals the byte count. 3. Write the byte sequence. ## Header format @@ -68,19 +70,21 @@ Offset Size Endian Field 0 2 BE mode uint16 must be 0x0001 (varlength) 2 2 BE ndim uint16 number of dimensions 4 4·ndim BE dims[0..ndim-1] uint32 each block shape -4+4·ndim 4 BE num_bytes uint32 byte length of payload +4+4·ndim 4 BE numElements uint32 number of elements in payload ``` Total header size: `2 + 2 + 4·ndim + 4` bytes. -The `num_bytes` field records the total number of bytes in the payload that follows the -header (i.e., the encoded output of the inner codec pipeline). For uncompressed data this -equals `file_size - header_size`. +The `numElements` field records the number of elements in the payload, following the +[N5 spec](https://github.com/saalfeldlab/n5/tree/fb50c2c3f1b411abd201a6c701cfd9c61486cd85). +The byte length of the payload is `numElements × element_size`. For `label_multiset`, +the serialized payload is a byte stream (element size = 1), so `numElements` equals +the byte count of the payload and equals `file_size - header_size`. > **Note:** For default-mode N5 blocks, the header has no 4th field; after `dims[]`, -> the payload begins immediately. The varlength header adds this 4-byte `num_bytes` -> field to record the payload length, since it cannot be derived from the chunk shape -> alone for variable-width data types. +> the payload begins immediately. The varlength header adds this 4-byte `numElements` +> field, since the payload length cannot be derived from the chunk shape alone for +> variable-width data types. ## Example @@ -170,7 +174,7 @@ Bytes 2–3: 00 03 ndim = 3 (uint16 BE) Bytes 4–7: 00 00 00 20 dims[0] = 32 (uint32 BE) Bytes 8–11: 00 00 00 20 dims[1] = 32 (uint32 BE) Bytes 12–15: 00 00 00 20 dims[2] = 32 (uint32 BE) -Bytes 16–19: XX XX XX XX num_bytes (uint32 BE) — byte count of payload +Bytes 16–19: XX XX XX XX numElements (uint32 BE) — element count (bytes, for label_multiset) ─── n5_label_multiset payload begins here ─────────────────────────────────────── Bytes 20–23: 00 00 00 00 argMaxSize = 0 (int32 BE) Bytes 24–131099: … listEntryOffsets[32768] (int32 BE each, 4·32768 bytes) @@ -192,14 +196,15 @@ N5 defines three block modes: | Mode | Value | Description | |------|-------|-------------| | default | `0x0000` | Fixed-width elements; shape from header | -| varlength | `0x0001` | Variable-width payload; byte count from header | +| varlength | `0x0001` | Variable-width payload; element count from header | | object | `0x0002` | Arbitrary serialized objects | The `n5_default` codec handles mode `0x0000`. This codec handles mode `0x0001`. In N5's Java reference implementation, varlength blocks are used for `LabelMultisetType` -arrays written by imglib2-label-multisets. The `num_bytes` field in the header equals -`file_size - header_size` for uncompressed data. +arrays written by imglib2-label-multisets. For `LabelMultisetType`, the serialized payload +is a byte stream (element size = 1), so the `numElements` field equals +`file_size - header_size`. ## Change log