feat: fixed size list data type - #62
Open
d-v-b wants to merge 18 commits into
Open
Conversation
Spec for a new Zarr v3 data type extension modeling Apache Arrow's FixedSizeList: a fixed-length, positional, homogeneous sequence of a single base data type. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Bite-sized task plan for adding the fixed_size_list data type extension: schema.json first (with positive and negative validation checks), README.md second (with JSON-block and cross-link verification), final verification last. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Schema for the new fixed_size_list data type extension, validating base_data_type (string or object) and list_size (integer >= 1). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Specification for a new Zarr v3 data type extension modeling Apache Arrow's FixedSizeList: a fixed-length, positional, homogeneous sequence of a single base data type. Each scalar contains exactly list_size elements of base_data_type, encoded as a packed concatenation under the bytes codec. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Rename "A recursive example" to "A nested example" — the example shows cross-type nesting (fixed_size_list of struct), not recursion of fixed_size_list itself. Matches the design spec wording. - Add a third endianness bullet covering the case where base_data_type is itself a compound type, removing a normative gap for the recursive-nesting feature. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The cross-type case (fixed_size_list of struct) is nested composition, not recursion. Avoids the imprecise implication that the type self-references. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The object form of base_data_type previously required a configuration
key, which made it impossible to use extension data types whose object
form is name-only (e.g. {"name": "bfloat16"}). Drop "configuration"
from the required list in the schema's $defs/dtype, and update the
README to describe extension base types accordingly.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
These were authoring artifacts for the fixed_size_list extension and shouldn't ship in the registry repository. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The first example uses a fill_value of [0.0, 0.0, 0.0], which is only meaningful once the JSON scalar and fill value encoding rules have been stated. Promote "Example" to a top-level "Examples" section placed after Fill value representation so each example is fully decodable when read. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The bytes-codec section was previously a top-level sibling of "Codec compatibility", but it specifies how the bytes codec satisfies the compatibility requirement. Demote it to a subsection (with Endianness nested one level deeper) so the general statement and the specific bytes-codec rule live together. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Drop the stub opening sentence of the base_data_type subsection; the Configuration table already states what base_data_type is. - Tighten the first Note to lead with the bytes-codec equivalence claim instead of restating the counterpart-of-struct framing already in Background. - Remove the third Note, which restated the variable-length prohibition already in the table and the base_data_type subsection. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The explicit "string for core / object for extension" breakdown is already covered by the upstream Zarr v3 data type representation spec. Defer to it with "the JSON representation of a Zarr v3 data type" rather than restating the form. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The "array of length list_size" rule above already excludes any scalar form. Defining the spec by what it isn't bloats the prose and implies a rejected design alternative. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The previous wording required each scalar to encode to a contiguous fixed-size blob, which excluded perfectly valid array-to-bytes codecs like RLE that compress the tuple before serializing. The actual requirement is that the codec can faithfully round-trip a tuple of list_size base_data_type values; the on-disk byte layout is the codec's business. Also drop the vlen-* exclusion paragraph: it had the same shape-vs-type confusion (those codecs don't fit because they're string codecs, not because they emit variable-size bytes), and the new framing makes the question self-answering. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Endianness is a property of the on-disk byte representation, not of
the bytes codec specifically. Lift "Endianness" out from under
"Bytes codec encoding" to a sibling subsection, and rephrase the
rules in terms of "any array-to-bytes codec that exposes an
endianness configuration" rather than pinning them to the bytes
codec. Drop the inline {"name": "bytes", ...} example since the
rule is no longer codec-specific.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Trim restatements, redundant lead-ins, and explanations of Zarr basics (what data_type / configuration / fill_value are, what "compatible" means for a codec, what "packed concatenation" implies). - Collapse the two intro sentences into one. - Drop the use-cases paragraph from Background. - Drop the "represented in array metadata as the value of the data_type metadata key" lead-in to Data type representation. - Tighten the base_data_type table cell; fold recursion into a single sentence under Configuration. - Delete the base_data_type and list_size H4 subsections — their surviving content was redundant with the Configuration table. - Collapse JSON scalar encoding from two paragraphs to one. - Trim Fill value to one sentence. - Trim example lead-ins. - Tighten Codec compatibility and the bytes-codec body; drop the total-encoded-size restatement (the math speaks for itself). - Tighten Endianness intro. No normative content removed; file is ~50 lines shorter. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
A Zarr-fluent reader does not confuse a data type with an array shape. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Comparing this data type's binary layout to a different data type is implementation trivia, not part of this spec. Removes the now-empty Notes section. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR adds a new data type
fixed_size_listcomprised of ordered, fixed-size lists of some other scalar type. It's similar to thefixed_size_listdata type in apache arrow (is there a better documentation available for arrow types?).I'm not sure how we are going to implement this in zarr-python, but it's a generally useful data type so we'll figure out something.