Make ObjectStore backend generic over any obspec store

[kylebarron]

In #1661 we merged an Obstore-based backend. It sounds like the Obstore backend has become very popular, and @maxrjones found that, at least in some situations, the Obstore backend can be significantly faster.

One problem with the Obstore backend, however, is that it's strictly tied to Obstore. The type hinting and runtime behavior all require exact instances from the Obstore package.

But very often people might want to insert some middleware.

• Caching: disk cache, memory cache of different flavors
• Timing: how long are different requests taking
• Request inspection, e.g. for debugging, or for evaluating async performance

The goal of Obspec is to define generic protocols to cleanly enable this. As described in the initial release post from last summer, Obspec should allow downstream libraries to depend on a protocol-based API that works with any implementation that provides the given signature.

E.g. if you think of the simplest pseudocode example of a cache:

from __future__ import annotations
from typing_extensions import Buffer
from obspec import GetRange

class SimpleCache(GetRange):
    """A simple cache for synchronous range requests that never evicts data."""

    def __init__(self, client: GetRange):
        self.client = client
        self.cache: dict[tuple[str, int, int | None, int | None], Buffer] = {}

    def get_range(
        self,
        path: str,
        *,
        start: int,
        end: int | None = None,
        length: int | None = None,
    ) -> Buffer:
        cache_key = (path, start, end, length)
        if cache_key in self.cache:
            return self.cache[cache_key]

        response = self.client.get_range(
            path,
            start=start,
            end=end,
            length=length,
        )
        self.cache[cache_key] = response
        return response

Then if a function expects an object implementing GetRange:

def my_function(client: GetRange, path: str, *, start: int, end: int):
    buffer = client.get_range(path, start=start, end=end)
    # Do something with the buffer
    print(len(memoryview(buffer)))

Then now you can pass in either the raw obstore backend or the backend wrapped by the cache:

from obstore.store import S3Store

store = S3Store("bucket")
caching_wrapper = SimpleCache(store)
my_function(caching_wrapper, "path.txt", start=0, end=10)
# second request will be cached by `SimpleCache`
my_function(caching_wrapper, "path.txt", start=0, end=10)

This architecture is much more tractable for end users than if Obstore implemented its own caching natively. Since users have full access to the cache (i.e. it isn't hidden away inside Rust), users can check methods of SimpleCache to track how much memory the cache is using and to manually evict cache items if they wanted.

@maxrjones has been starting to collect utilities around obspec in https://github.com/virtual-zarr/obspec-utils.

This is backwards-compatible (at least if you ignore the obstore version bump from 0.5.1 — released March 2025 — to 0.7.0 — released June 2025).

Implementation notes

• This relies on structural subtyping, i.e. that the shape matters not the name. This works for all protocols, but we have to implement special support for exceptions, since exceptions don't support structural subtyping 🥲. As described in Exceptions in the obspec docs, the workaround I chose is to use well-defined names, and map_exception will convert any external exceptions to exceptions subclassing from obspec.exceptions.

TODO:

• Add unit tests and/or doctests in docstrings
• Add docstrings and API docs for any new/modified user-facing classes and functions
• New/modified features documented in docs/user-guide/*.md
• Changes documented as a new file in changes/
• GitHub Actions have all passed
• Test coverage is 100% (Codecov passes)

[TomNicholas]

+1 Xarray could make use of this. (When reading non-icechunk stores - icechunk already has a caching layer)

[kylebarron]

I'm not sure where the other locations are where I need to define obspec as a dependency

[d-v-b]

looks good, just need a release note in changes

[kylebarron]

As discussed in zarr-developers/VirtualiZarr#1092, this would also enable users to use the ObjectStore adapter class without a direct dependency on Obstore, which could be useful for users in constrained environments.

[d-v-b]

sounds good, do you want me to resolve the conflicts?

[d-v-b]

with the conflicts resolved there are some blockers, which I will summarize here:

• the obspec import is not conditional
• we still require that store classes have a FQN that starts with "obstore", so we are still doing nominal typing :/
• exception handling is not complete

i'm happy to deal with these

[kylebarron]

If you'd like to make updates, I can review. I have a lot going on/catching up on at the moment

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.44%. Comparing base (26820c7) to head (11477ce).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3698      +/-   ##
==========================================
+ Coverage   94.34%   94.44%   +0.10%     
==========================================
  Files          92       92              
  Lines       12935    12941       +6     
==========================================
+ Hits        12203    12222      +19     
+ Misses        732      719      -13     

Files with missing lines	Coverage Δ
src/zarr/__init__.py	100.00% <ø> (ø)
src/zarr/storage/_obstore.py	100.00% <100.00%> (+5.85%)	⬆️

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[d-v-b]

@kylebarron have a look, i fixed the conflicts and the issues I mentioned