add jpeg codec

[konstibob]

add jpeg codec specification for pr that I did
'
zarr-developers/zarr-java#81

[d-v-b]

I think there are more parameters that jpeg could take -- see https://github.com/cgohlke/imagecodecs/blob/8776afc59c94b2531aaecb5dd969308388c9e2d0/imagecodecs/numcodecs.py#L949

[clbarnes]

I think there are more parameters that jpeg could take

Different JPEG implementations may or may not allow you that level of control (I know a bunch don't).

[psobolewskiPhD]

Very excited to see this; crazy timing because I'm playing with jpeg encoded chunks in zarrs.
I know cart horse and all that, but it would be nice if compatibility with the zarr v3 jpeg codec from imagecodecs.zarr (imagecodecs_jpeg) was also kept in mind here:
https://github.com/cgohlke/imagecodecs/blob/f25a75bc0b6e6773b116640ff3c9c9bbcef3d37b/imagecodecs/zarr.py#L2901

For example, it's been useful to have the ability to pass a header in the config -- and colorspace data, i forgot about that one!

[Tom-TBT]

Should you precise that RGB images and microscopy images with 3 channels are different things? Otherwise this distinction may not be clear for everyone that RGB images have colored pixel (they are measured together) while 3 channels fluorescence microscopy images are another dimension of the array (~independant from each other).

[d-v-b]

I don't think there needs to be a strong link between how a channels are acquired and how they are represented when stored in a Zarr array. it's perfectly valid to store multichannel microscopy images with the different bands packed in a struct data type (although it would not work with the proposed jpeg codec without a data type conversion), and not all RGB images have the color channels acquired simultaneously. The choice of stored representation is just a choice here.

[clbarnes]

I think the point is that common JPEG settings are optimised for human eyes looking at natural scenes rather than for equal fidelity across each of 3 interchangeable channels. So we should provide guidance on when it is appropriate to use "natural" JPEG compression (e.g. taking a photo down a bright field microscope) vs when it may be more representative to compress each channel separately as greyscale (e.g. some arbitrary number of different fluorescent labels, even if that arbitrary number is 3).

[jbms]

Note that while YCbCr 420 is the default representation for libjpeg, the JPEG format (and libjpeg) do allow many other possibilities, including storing RGB directly as RGB with no color space conversion or chroma subsampling. It isn't clear whether that would be particularly useful compared to just using sharding and encoding each channel as a separate jpeg, but it is a possibility.

[clbarnes]

A quick rust implementation for zarrs, using the encoder crate's defaults to fill in all the other settings: https://github.com/clbarnes/zarrs_jpeg

Deviations from the spec as written are in the README. The chunks it creates are valid JFIFs (which is a must-have feature IMO). Because it accepts (x, y, 1) arrays as greyscale, you can chunk/shard over the channels if you're not dealing with RGB.

[konstibob]

Thanks everyone, this was super helpful. I am summarizing my specifications of the codec below:

Supported shapes

The codec will now accept only these shapes (inspired by @clbarnes' implementation https://github.com/clbarnes/zarrs_jpeg):

Input chunk shape	Result
(N,) (1D)	rejected — reshape to 2D first
(H, W)	grayscale JPEG
(H, W, 1)	grayscale JPEG
(H, W, 3)	3-component JPEG (color choice required — see below)
(H, W, 2) or (H, W, ≥4)	rejected — reshape/shard so the channel axis is 1 or 3
4D+	rejected — transpose/reshape down to 2D or 3D

(H, W, 1) is accepted as grayscale too so you can shard over channels easily.

Color conversion & subsampling — no default for 3-channel

@jbms mentioned YCbCr + 4:2:0 is designed for natural photos and silently destroys fidelity for scientific data whose channels aren't real colors. So for 3-component data there will be no default, you have to choose explicitly:

• color_transform: ycbcr (RGB→YCbCr, JFIF standard, enables perceptual subsampling) or none (store components as-is; needs the Adobe APP14 marker so decoders skip the inverse transform)
• subsampling: 4:4:4 / 4:2:2 / 4:2:0 (only meaningful with ycbcr; forced to 4:4:4 with none) (mentioned by @Tom-TBT)
• quality (0–100, default 90) stays and applies to all data.

Usecases for parameters:

Data	color_transform	subsampling
grayscale (2D or (H,W,1))	–	– (just quality)
natural color photo	ycbcr	your choice: 4:4:4 / 4:2:2 / 4:2:0
independent channels (fluorescence, multispectral, …)	none	forced 4:4:4 (no chroma loss)

Also max dimensions: JPEG stores width/height as 16-bit, so max 65,535 in any dimension (@clbarnes).

[clbarnes]

Just from digging through rust implementations, it seems like it's pretty common not to support encoding without color transformation. Even the libjpeg-turbo bindings, some expansive vibecoded codec ecosystems, and foundational image crate don't offer it at all. Maybe that's just a rust problem. As that config is hopefully only used on write, it may not be a showstopper (although I don't have a lot of faith that the decoders will support it if the encoders don't); it's just something I'm a bit worried about. It can be worked around by sharding or transposing and reshaping to split the channels out into greyscales; we could propose that instead of allowing the none transform.

[clbarnes]

Just want to check that I understand the algorithm correctly wrt when to ignore and default config values:

def jpeg_encode(rgb_bytes: bytes, quality: int, internal_color_space: str, subsampling: str=None):
    ...

def encode(config, bytes, shape):
    assert len(shape) in (2, 3)

    if len(shape) == 2 or (len(shape) == 3 and shape[2] == 1):
        # ignore color_transform and subsampling
        return jpeg_encode(bytes, config.quality, "grayscale")
    else:
        assert shape[2] == 3
        assert config.color_transform is not None, "color_transform must be set"
        if config.color_transform == "ycbcr":
            # default subsampling if not defined
            subsampling = config.subsampling or "4:2:0"
            return jpeg_encode(bytes, config.quality, "ycbcr", subsampling)
        elif config.color_transform == "none":
            # ignore subsampling, maybe raise a warning
            return jpeg_encode(bytes, config.quality, "rgb", "4:4:4")

[jbms]

I looked a bit more at the color space issue. The plain JFIF format does not indicate color space, and instead 1 channel is assumed to be stored as grayscale while 3 channels are assumed to be stored as YCbCr.

There is, however, an APP14 Adobe marker that is supported by many implementations, including libjpeg, that can indicate the stored color space as one of: unknown, YCbCr, or YCCK (not widely supported 4-channel color space).

Therefore for the color space transform, arguably it should specify both the input and output color spaces, rather than a more ambiguous transform.

For example, there could be separate encoded_color_space and decoded_color_space properties, where we have:

decoded_color_space	encoded_color_space	conversion
grayscale	grayscale	none
RGB	YCbCr	RGB -> YCbCr
RGB	RGB	none
YCbCr	YCbCr	none

This representation allows the YCbCr -> YCbCr noop color space conversion while still allowing the JPEG to correctly indicate that it stores YCbCr data.

We could indicate that not all implementations support combinations other than "grayscale -> grayscale" and "RGB -> YCbCr", and therefore it is not recommended to use the other options.

Subsampling is entirely independent of the color space conversion, at least as far as the file format is concerned. The jpeg format itself allows separate vertical and horizontal subsampling factors to be specified for each component, as a number of 1, 2, 3, or 4, which correspond to subsampling factors of 1/4, 2/4, 3/4, 4/4. Therefore I think it would make sense to specify the horizontal and vertical subsampling factors for each encoded component separately rather than the "4:2:0" syntax which is more limited and also rather confusing.

[jbms]

@clbarnes Regardless of the specific representation of the color space parameters, I'd suggest that invalid color space parameters for the number of channels be treated as an error rather than ignored.

[konstibob]

@jbms regarding color space, I'd keep a single color_transform: grayscale, ycbcr (RGB→YCbCr), and none (RGB→RGB, marked "unknown") cover every case that occurs in practice.
The only thing the two-field model adds is YCbCr→YCbCr, which round-trips identically to none. This happens rare enough that I'd rather keep the config modular and add it later if a real need shows up or what do you think?

On subsampling, I'd also keep the named presets (4:4:4/4:2:2/4:2:0). The per-component [[2,2],[1,1],[1,1]] notation is more faithful to the format, but this codec targets scientists, not JPEG experts 4:2:0 is the industry-standard notation everyone recognizes, while [[1,1],[1,1],[1,1]] requires knowing JPEG's internals to read. The three presets cover every meaningful case, I feel like more would be over engineering.

[clbarnes]

There are other color models, like CMYK (potential for 4-channel storage) and XYB (better visual fidelity for natural RGB data), but neither are in the original spec and don't have the same breadth of support. The benefit of being more explicit is that if the codec was ever extended to include them in future, it would be a backwards-compatible relaxation rather than needing new fields. Allowing an explicit serialisation but then strict guidance on which ones are allowed (or should be used for maximum compatibility) may be the middle ground - APIs can be defined around those usage patterns.

The same sort of goes for subsampling, but the subsampling regimes are more limited and there isn't really a use case for the full flexibility. It looks like turboJPEG just uses an enum covering 4:4:4, 4:2:2, 4:2:0, greyscale, 4:4:0, 4:1:1, 4:4:1, and unknown.

Just from digging through rust implementations, it seems like it's pretty common not to support encoding without color transformation. Even the libjpeg-turbo bindings, some expansive vibecoded codec ecosystems, and foundational image crate don't offer it at all.

FWIW, I raised issues and adding color transforms (or lack thereof) to the rust libjpeg-turbo bindings seems tractable, and the aforementioned vibecoded ecosystem has now vibecoded them in, for better or for worse.

[konstibob]

adjusted specs according to discussion

[d-v-b]

but this codec targets scientists, not JPEG experts 4:2:0 is the industry-standard notation everyone recognizes, while [[1,1],[1,1],[1,1]] requires knowing JPEG's internals to read.

I think the main audience for the JSON form of the codec will be software engineers with some degree of Zarr knowledge. For that audience, this codec will be one of many codecs. IMO for this audience the per-dimension arrays of subsampling factors will be easier to explain than a string like "4:4:0". The spec should ensure that translation from a string like "4:4:0" to the more declarative JSON form is established clearly.

[clbarnes]

I have been staring at 4:2:2-like values for some hours now and still have no idea what they're supposed to mean. A serialisation referring to the size of the luminance blocks represented by a single chrominance pixel makes more sense. Is the explicit inclusion of the chrominance pixel block sizes in case of future colour spaces where there might be some whackier ratio?

Also, I've updated the rust impl for the current spec, using my fork of the libjpeg-turbo bindings which supports color transforms (PR up for review).

[jbms]

I have been staring at 4:2:2-like values for some hours now and still have no idea what they're supposed to mean. A serialisation referring to the size of the luminance blocks represented by a single chrominance pixel makes more sense. Is the explicit inclusion of the chrominance pixel block sizes in case of future colour spaces where there might be some whackier ratio?

Also, I've updated the rust impl for the current spec, using my fork of the libjpeg-turbo bindings which supports color transforms (PR up for review).

Yes, the 420 syntax is commonly used for specifying chroma subsampling for image and video formats but is not flexible enough to represent what the JPEG format actually supports, and is frankly quite confusing compared to explicit values for vertical and horizontal subsampling:
https://en.wikipedia.org/wiki/Chroma_subsampling#Sampling_systems_and_ratios

Even referring to the wikipedia page it is easy to pick the wrong one by accident .

[clbarnes]

Colorspaces with different numbers of channels should use subsampling arrays of different lengths, right?

In my code I've modeled the colorspace/ subsampling config as a tagged union rather than make the user remember which combinations are and aren't valid. It serialises to two separate fields.

use serde::{Deserialize, Serialize};

#[derive(Serialize, Deserialize)]
#[serde(tag = "encoded_color_space", rename_all = "lowercase")]
enum ColorConfig {
    Rgb,
    Grayscale,
    YCbCr { subsampling: [[u8; 2]; 3] },
    // maybe in future
    // CMYK { subsampling: [[u8; 2]; 4] }
}

#[derive(Serialize, Deserialize)]
struct JpegCodecConfiguration {
    quality: u8,
    #[serde(flatten)]
    color_config: ColorConfig,
}

I think the clearest explanation might be

encoded_color_space MUST be given as "ycbcr", "rgb", or "grayscale".
If encoded_color_space is "ycbcr", subsampling MUST be given; otherwise, subsampling MUST be omitted.

Unrelated: I think the spec should include an instruction that decoders should basically ignore the encoding arguments and decode whatever they can, then just check that the shape and dtype matches expectation. This gives us some leeway for converting legacy JPEG-compressed formats which might have more baroque settings and still being able to read them.

[jbms]

Colorspaces with different numbers of channels should use subsampling arrays of different lengths, right?

Logically, the steps to encode a JPEG are:

1. Perform color space conversion
2. Perform subsampling on converted color space channels
3. Encode as JPEG, indicating the converted color space in the metadata

The subsampling applies to the encoded color space and therefore must match the number of channels in the encoded color space.

In my code I've modeled the colorspace/ subsampling config as a tagged union rather than make the user remember which combinations are and aren't valid. It serialises to two separate fields.

use serde::{Deserialize, Serialize};

#[derive(Serialize, Deserialize)]
#[serde(tag = "encoded_color_space", rename_all = "lowercase")]
enum ColorConfig {
    Rgb,
    Grayscale,
    YCbCr { subsampling: [[u8; 2]; 3] },
    // maybe in future
    // CMYK { subsampling: [[u8; 2]; 4] }
}

There is also 4-channel YCCK, which is like YCbCr but for CMYK.

#[derive(Serialize, Deserialize)]
struct JpegCodecConfiguration {
quality: u8,
#[serde(flatten)]
color_config: ColorConfig,
}

I think the clearest explanation might be

> `encoded_color_space` MUST be given as "ycbcr", "rgb", or "grayscale".
> If `encoded_color_space` is "ycbcr", `subsampling` MUST be given; otherwise, `subsampling` MUST be omitted.

Subsampling is independent of color space --- it occurs after color space conversion and even be used with grayscale. In practice, however, it is unlikely to be useful except for YCbCr and YCCK representations.

Unrelated: I think the spec should include an instruction that decoders should basically ignore the encoding arguments and decode whatever they can, then just check that the shape and dtype matches expectation. This gives us some leeway for converting legacy JPEG-compressed formats which might have more baroque settings and still being able to read them.

I would say that the decoders should indeed ignore encoded_color_space and subsampling when decoding, because that is already indicated in the JPEG file itself.

However, I have suggested that there is a decoded_color_space parameter in addition to encoded_color_space. The color space conversion done, if any, during encoding and decoding is based on this pair of parameters. The JPEG file itself does not indicate decoded_color_space in any way, and therefore it is needed to determine the color space conversion to perform, if any, when decoding.

For example, you might already have a color image in YCbCr format. Then you can specify encoded_color_space and decoded_color_space both equal to YCbCr. During encoding no color conversion will be done, but the JPEG will still record that the color space is YCbCr. If the rule is that decoding ignores decoded_color_space, then:

• If the JPEG indicates YCbCr as the color space, presumably it would get decoded to RGB, which would not correctly round trip the original data
• In order to correctly round trip the original data, the JPEG would need to record the color space incorrectly as "unknown / RGB", which means other software would not interpret it correctly.

[konstibob]

@normanrz @clbarnes I completely agree with your view on the parameters:

▎ encoded_color_space MUST be given as "ycbcr", "rgb", or "grayscale".
▎ If encoded_color_space is "ycbcr", subsampling MUST be given; otherwise, subsampling MUST be omitted.

I went looking at how this is handled in practice, and I think imagecodecs is the most relevant reference here.

https://github.com/cgohlke/imagecodecs/blob/master/imagecodecs/_jpeg8.pyx
(look for the jpeg8_encode function, where on line 215 its being checked if the color format is ycbcr)

On decoded_color_space @jbms

The parameter pair already exists in libjpeg, and imagecodecs exposes it directly:

as functions parameters (step: in, out) we would have:
(encode: in_color_space , jpeg_color_space)
(decode: jpeg_color_space, out_color_space)

On encode that's precisely (decoded, encoded); on decode the same two reappear as "what the file is" and "what I want handed back".

I'd want decoded_color_space to be optional, defaulting to rgb for 3-component and grayscale for 1-component data, therefore defaulting to what the spec currently hardcodes.

This isn't inconsistent with encoded_color_space having no default: defaulting that to ycbcr would silently destroy non-colour channels, whereas rgb here is just the single value the spec allows today, so the default can't surprise anyone. imagecodecs already does exactly this — when colorspace (the decoded one) isn't given, it infers JCS_GRAYSCALE for 1 sample and JCS_RGB for 3 line 187-197 same link from above.