Register vlen-ndarray data type and codec

[espg]

This PR registers vlen-ndarray, a data type for arrays whose elements are variable-length ndarrays — each element has shape (n, *inner_shape) with n varying per element and the trailing inner_shape plus the scalar dtype fixed in the configuration — together with its paired parameter-free array -> bytes codec:

{
  "data_type": {
    "name": "vlen-ndarray",
    "configuration": {"dtype": "float32", "inner_shape": [2]}
  },
  "codecs": [{"name": "vlen-ndarray"}]
}

Wire format. The codec serializes each element as its raw little-endian C-order bytes and frames chunks exactly as the registered vlen-bytes codec (u32le count; per element u32le length + payload). Encoded chunks are therefore byte-identical to bytes + vlen-bytes chunks holding each element's raw bytes: existing stores using that common ragged-data convention upgrade to the typed form with a metadata-only rewrite, and implementations without vlen-ndarray support can read the data as bytes + vlen-bytes after metadata substitution.

Relation to existing discussions. This is a concrete, fixed-inner-shape point in the "generic container data types" design space discussed in #57, scoped to fixed-size numeric/boolean core scalar types so every element has a well-defined item size (element length is implied by encoded byte length — no per-element shape storage). It does not preclude a more general variable_length[<base>] family later.

Implementation. A working implementation (zarr-python >= 3.1, registered via zarr.data_type / zarr.codecs entry points) with round-trip, sharding, and byte-identity tests: https://github.com/espg/zarr-vlen-ndarray

Motivating use. The zagg aggregation pipeline (and its moczarr reader) stores per-cell t-digest centroid sets ((n, 2) float32) and location lists ((n,) uint64) on HEALPix grids; the store spec's /2 revision cites this name (englacial/zagg#340, englacial/zagg#210).

[espg]

Some broader background on this: we're encoding photon fluxes. First consumer is NASA Sliderule and the ICESat-2 mission; raw photons vary, and we encode them as 'digests' on a grid with fixed attribute width, but variable column length per cell. (Hoping to have an example zarr store that can opened publicly for California sometime next week).

We'll be using the same data type / codec for laser waveforms from the GEDI mission as well.

[LDeakin]

Just reading your description (I haven't looked at the spec), but my first thought is why not generalise to arbitrary shaped ndarrays? (n, *inner_shape) is quite restrictive / niche . Was that done just because it exactly fits your data, or was it to shoehorn this into a vlen- style codec?

Also I see potential variants data types that could be encoded differently:

• Arbitrary shaped arrays could be encoded with dimensionality + [shape] + [elements].
• Fixed dimensionality with [shape] + [elements]
• Fixed shape with [elements]. This can just use the bytes codec.

[jbms]

It sounds like the codec is designed to exactly match data that has already been written using vlen-bytes. The representation isn't ideal but defining a codec for that purpose seems reasonable. However I think you could avoid having a separate vlen-ndarray dtype specifically for this codec, and instead just have an ndarray data type. Then the vlen-ndarray codec would just impose extra constraints. You could also have an array-to-array codec that converts the ndarray dtype to bytes dtype, and then use vlen-bytes as the array-to-bytes codec.

[espg]

Thanks — these both seem to push in the same direction and I think you're right: the dtype should be general and the codec should carry the constraints.

@jbms — agreed on the split. I'll restructure this PR to register a ndarray data type (configuration: a scalar dtype plus a shape where null marks a variable-length dimension, e.g. {"dtype": "float32", "shape": [null, 2]}) and keep vlen-ndarray as an array → bytes codec that imposes the extra constraint: exactly one variable dimension, in the leading position. The array→array + vlen-bytes composition is workable... but a single array→bytes codec produces the same bytes with one fewer moving part in the chain. Unless you think the composition is the better precedent?

@LDeakin — fair question. With the trailing shape fixed in configuration, n is derivable from the payload length ÷ (itemsize × trailing width), so elements need no per-element shape header — and the resulting chunk bytes are identical to what vlen-bytes produces. Arbitrary shapes need exactly the dimensionality + shape + elements headers you describe — different bytes, different codec. Under the split above, your three variants become three codecs over the one ndarray dtype: fixed shape → the plain bytes codec (or #62's fixed-size list, which the general dtype is adjacent to); fixed dimensionality → a shape-headered codec; arbitrary → dimensionality + shape + elements. Our data is broadly ragged-array shaped -- per array, elements share a fixed trailing shape (the width k varies across sensors/fields via configuration), varying only in length along the leading dimension. So the codec this PR registers is the header-free variable-leading-dim codec, since it's the one with a consumer and deployed data -- general to any payload that is consistent in width but varies in length along the leading dimension.

Also I see potential variants data types that could be encoded differently:

• Arbitrary shaped arrays could be encoded with dimensionality + [shape] + [elements].
• Fixed dimensionality with [shape] + [elements]
• Fixed shape with [elements]. This can just use the bytes codec.

I'll note that mapping in the data-type text — as compatible-codec notes, not registrations — so the grammar's generality is on record without claiming codecs nobody is building.

I'd like to keep the dtype registration minimal (the shape grammar plus this one codec) rather than grow it into a full ndarray design effort in this PR — but you two are the most likely implementors, so if the general dtype needs more decided up front to be implementable in tensorstore/zarrs — tell me what you need and I'll extend it.

[espg]

ok, I've updated both this PR and https://github.com/espg/zarr-vlen-ndarray — restructured per the dtype/codec split, details in the diffs.

One correction from my earlier comment: any single variable dimension is length-inferable regardless of position — the leading restriction is this codec's design choice, and the text now says so.

One thing that strikes me as a long time numpy user is the collision users have reading a data type that's ndarray. In numpy, that's assumed to be an n-dimensional array, but that's already true for basically any data that's stored in zarr. We're talking about a datatype that's an array of ndarrays; in numpy that concept would be a subarray (subarray dtypes), but that name already conflicts within the zarr namespace (sub-arrays of groups). So... what's the best path here? Keep ndarray, swap the prefix to a suffix as arraynd, or a compound like nested-array? Or just document it and move on?

[jbms]

ok, I've updated both this PR and https://github.com/espg/zarr-vlen-ndarray — restructured per the dtype/codec split, details in the diffs.

One correction from my earlier comment: any single variable dimension is length-inferable regardless of position — the leading restriction is this codec's design choice, and the text now says so.

I'm not sure if the data type should specify shape constraints --- or if that should be a part of the codec.

In any case it seems reasonable to allow any single dimension to be variable-length under this design.

One thing that strikes me as a long time numpy user is the collision users have reading a data type that's ndarray. In numpy, that's assumed to be an n-dimensional array, but that's already true for basically any data that's stored in zarr. We're talking about a datatype that's an array of ndarrays; in numpy that concept would be a subarray (subarray dtypes), but that name already conflicts within the zarr namespace (sub-arrays of groups). So... what's the best path here? Keep ndarray, swap the prefix to a suffix as arraynd, or a compound like nested-array? Or just document it and move on?

The data type specifies the type of an individual element within the array. Therefore, a regular array has a scalar type as its element, while in this case we are talking about an array where the element type is itself an array. Therefore, ndarray as the element type makes sense.

Another thing to consider is to specify the codec chain of array -> array, array -> bytes, and bytes -> bytes codecs to apply to each individual ndarray element, rather than using a fixed little-endian bytes codec. Similar to index_codecs used for sharing_indexed, the codecs specified here would be subject to the additional constraint that the size of the input array can be inferred from the byte length of the output. On the other hand if this is intended to be just a very special purpose codec, and a more general purpose codec perhaps introduced later to handle other use cases, then that may be overengineering.