feat: Add suffix chunk key encoding

[mkitti]

This document outlines the suffix chunk key encoding for Zarr v3, which allows appending user-defined file extensions to chunk keys, enhancing interoperability with traditional file-based tools.

[normanrz]

Suggested change

	* `"base-encoding"`: **(Optional)** A chunk key encoding configuration object. This specifies the "base" encoding to be used *before* the suffix is appended. If omitted, the store's `default` chunk key encoding is used.
	* `"base_encoding"`: **(Optional)** A chunk key encoding configuration object. This specifies the "base" encoding to be used *before* the suffix is appended. If omitted, the store's `default` chunk key encoding is used.

Zarr uses snake_case in the JSON metadata.

I would make this required, because it is also required in the top-level array metadata. It helps implementations to make this explicit.

[jbms]

Agreed, should be required.

[LDeakin]

Cool, I wrote something similar here https://docs.rs/zarrs/latest/zarrs/array/chunk_key_encoding/default_suffix/struct.DefaultSuffixChunkKeyEncoding.html, but this is much better because it can compose over other chunk key encodings.

If omitted, the store's default chunk key encoding is used.

Stores don't have a default chunk key encoding?

[d-v-b]

i love this! do we care at all about suffixes that ascend the directory tree on local file systems, e.g. `"../foo.tif"?

[mkitti]

i love this! do we care at all about suffixes that ascend the directory tree on local file systems, e.g. `"../foo.tif"?

I think the base key would need to be a directory first for this to work on a local file system. I'm not sure how .. works on S3.

I'm thinking that we do not care about this. If something needs to be contained within a directory, it would not be Zarr's responsibilty to maintain that containment. Someone would probably want to use a chroot jail.

[mkitti]

A complimentary chunk key encoding might be something to ascend directories to assist in mapping from a high dimensionality arrays to a squeezed lower dimensionality array.

Imagine an array with dimensions (256, 1024, 1024, 1, 1) in ZYXCT order. We might want a way to map a 3D XYZ array to chunks with a 5D chunk layout or a 5D array mapped to a 3D chunk layout. I suspect this is probably a separate chunk key encoding and pull request.

[LDeakin]

Implemented here: zarrs/zarrs#286

I question the base-encoding being optional given that chunk_key_encoding is itself mandatory. Despite the name, the default chunk key encoding is not default and must be explicitly listed.

[mkitti]

base-encoding is now a required configuration parameter as of 08d9470

[normanrz]

This PR meets the requirements for being merged. However, there still seems to be some discussion.
Let me know when you're ready to.

[mkitti]

Should I put something in place of "Identifier: (A unique URI to be assigned upon formal adoption)" ?

[mkitti]

I will add a JSON schema here and then we can consider merging.

[normanrz]

Should we instead focus on #67?

[d-v-b]

yes IMO #67 looks better than this. closing

[d-v-b]

oops, I confused this with prefix. re-opening!

[mkitti]

The implementation of #67 looks much more complicated than this. Considering that a chunk key encoding needs to be evaluated for every chunk I would like to see some benchmark numbers before saying that is obviated by a full blown string interpolation engine. Naive string interpolation could quickly lead to considerable memory thrashing considering 10³ to 10⁶ chunks.

[normanrz]

That seems like an implementation detail and not a spec concern. I can imagine a number of approaches to get good performance. For example, implementations could detect simple suffix or prefix cases when opening the array and then using optimized implementations. Alternatively, they could JIT a function to generate the chunk keys.

[clbarnes]

I suspect that one string interpolation op per chunk index would be negligible compared to the IO cost, but you're right that if you're not using a language's native string interpolation machinery, it could be awkward to write and optimise. I'll have a fiddle.

Also agreed with Norman that it would be easy to detect a prefix{*} or {*}suffix pattern and push them through a fast path.