Skip to content

Open an array in a remote store #713

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
thewtex merged 5 commits into from
Sep 2, 2026
Merged

Open an array in a remote store #713

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
dfc13e6
feat(py): open an array in a remote store
vboussot Sep 1, 2026
55ac867
test(py): read a real S3 store through open_array
vboussot Sep 1, 2026
ce3a908
docs(py): pass the bucket region in the S3 resample notebook
vboussot Sep 1, 2026
b43ee4c
style(docs): reformat the migration guide under ruff 0.15
vboussot Sep 1, 2026
1c1e615
fix(py): resolve the bucket region an S3 read needs
vboussot Sep 1, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 12 additions & 6 deletions docs/migration_guides/v0-43_to_v0-44.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -485,8 +485,12 @@ error.
```python
# Before -- silently produced a plain `bytes` array on v0.44 and v0.45
to_ome_zarr(
"image.ome.zarr", multiscales, version="0.5",
filters=[...], serializer=..., compressors=[...],
"image.ome.zarr",
multiscales,
version="0.5",
filters=[...],
serializer=...,
compressors=[...],
)

# After
Expand All @@ -500,13 +504,15 @@ for dataset, level in zip(multiscales.metadata.datasets, multiscales.images):
"image.ome.zarr",
name=dataset.path,
shape=skeleton.shape,
chunks=skeleton.chunks, # keep the layout ngff-zarr chose ...
shards=skeleton.shards, # ... including its sharding
chunks=skeleton.chunks, # keep the layout ngff-zarr chose ...
shards=skeleton.shards, # ... including its sharding
dtype=skeleton.dtype,
filters=[...], serializer=..., compressors=[...],
filters=[...],
serializer=...,
compressors=[...],
fill_value=0,
dimension_names=list(level.dims),
overwrite=True, # replace the skeleton, codec chain and all
overwrite=True, # replace the skeleton, codec chain and all
)
# One dask chunk per stored write unit: every chunk (or shard) has a
# single writer and needs no lock, and no level is held in memory whole.
Expand Down
11 changes: 11 additions & 0 deletions docs/python.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -443,6 +443,17 @@ the fill stays safe across processes and cluster ranks. The handle from
writer can fill the arrays as well; `to_ome_zarr` is not involved. Chunks that
are never written read back as the fill value.

`open_array` also opens an array in a remote store (http(s), S3, GCS, Azure),
through the same engine `from_ome_zarr` reads one with, and takes the same
`storage_options`. A remote handle is read-only -- filling a store region by
region needs a local directory -- but it reports the same `shape`, `dtype` and
`chunks`, and serves the same windowed reads:

```python
level0 = nz.open_array("s3://bucket/volume.ome.zarr", "scale0/image")
window = level0[0:1, 0:64, :, :]
```

### Append coarser levels to an existing store

Once the finest level is on disk, `start_level` derives the next levels from
Expand Down
134 changes: 110 additions & 24 deletions py/examples/itk_elastix_transform_resample_s3.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,14 @@
"cell_type": "code",
"execution_count": 1,
"id": "acae54e37e7d407bbb7b55eff062a284",
"metadata": {},
"metadata": {
"execution": {
"iopub.execute_input": "2026-09-01T19:24:57.496022Z",
"iopub.status.busy": "2026-09-01T19:24:57.495950Z",
"iopub.status.idle": "2026-09-01T19:24:57.921963Z",
"shell.execute_reply": "2026-09-01T19:24:57.921613Z"
}
},
"outputs": [],
"source": [
"import logging\n",
Expand Down Expand Up @@ -104,7 +111,14 @@
"cell_type": "code",
"execution_count": 2,
"id": "8dd0d8092fe74a7c96281538738b07e2",
"metadata": {},
"metadata": {
"execution": {
"iopub.execute_input": "2026-09-01T19:24:57.923699Z",
"iopub.status.busy": "2026-09-01T19:24:57.923494Z",
"iopub.status.idle": "2026-09-01T19:25:10.418404Z",
"shell.execute_reply": "2026-09-01T19:25:10.417048Z"
}
},
"outputs": [
{
"name": "stdout",
Expand All @@ -131,7 +145,9 @@
"\n",
"dask.config.set(scheduler=\"threads\", num_workers=16)\n",
"\n",
"storage_options = {\"anon\": True}\n",
"# The AIND bucket is public and lives in us-west-2. The region is explicit\n",
"# because the S3 backend does not discover it from the URI.\n",
"storage_options = {\"anon\": True, \"region\": \"us-west-2\"}\n",
"fixed_multiscales = from_ome_zarr(FIXED_URI, storage_options=storage_options)\n",
"moving_multiscales = from_ome_zarr(MOVING_URI, storage_options=storage_options)\n",
"\n",
Expand Down Expand Up @@ -194,13 +210,20 @@
"cell_type": "code",
"execution_count": 3,
"id": "8edb47106e1a46a883d545849b8ab81b",
"metadata": {},
"metadata": {
"execution": {
"iopub.execute_input": "2026-09-01T19:25:10.420792Z",
"iopub.status.busy": "2026-09-01T19:25:10.420700Z",
"iopub.status.idle": "2026-09-01T19:25:22.843132Z",
"shell.execute_reply": "2026-09-01T19:25:22.842680Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Registration at level 3: 0.9 s\n"
"Registration at level 3: 1.0 s\n"
]
}
],
Expand Down Expand Up @@ -237,7 +260,14 @@
"cell_type": "code",
"execution_count": 4,
"id": "10185d26023b46108eb7d9f57d49d2b3",
"metadata": {},
"metadata": {
"execution": {
"iopub.execute_input": "2026-09-01T19:25:22.844931Z",
"iopub.status.busy": "2026-09-01T19:25:22.844839Z",
"iopub.status.idle": "2026-09-01T19:25:23.221234Z",
"shell.execute_reply": "2026-09-01T19:25:23.220677Z"
}
},
"outputs": [
{
"data": {
Expand Down Expand Up @@ -306,7 +336,14 @@
"cell_type": "code",
"execution_count": 5,
"id": "7623eae2785240b9bd12b16a66d81610",
"metadata": {},
"metadata": {
"execution": {
"iopub.execute_input": "2026-09-01T19:25:23.222863Z",
"iopub.status.busy": "2026-09-01T19:25:23.222764Z",
"iopub.status.idle": "2026-09-01T19:25:30.367860Z",
"shell.execute_reply": "2026-09-01T19:25:30.367397Z"
}
},
"outputs": [
{
"name": "stdout",
Expand All @@ -319,7 +356,7 @@
"name": "stdout",
"output_type": "stream",
"text": [
"Wrote registered_822174_on_773889_level2.ome.zarr in 15 s (1.27 Mvoxel/s end to end)\n"
"Wrote registered_822174_on_773889_level2.ome.zarr in 7 s (2.77 Mvoxel/s end to end)\n"
]
}
],
Expand Down Expand Up @@ -347,7 +384,14 @@
"cell_type": "code",
"execution_count": 6,
"id": "7cdc8c89c7104fffa095e18ddfef8986",
"metadata": {},
"metadata": {
"execution": {
"iopub.execute_input": "2026-09-01T19:25:30.368913Z",
"iopub.status.busy": "2026-09-01T19:25:30.368839Z",
"iopub.status.idle": "2026-09-01T19:25:40.140009Z",
"shell.execute_reply": "2026-09-01T19:25:40.138742Z"
}
},
"outputs": [
{
"data": {
Expand Down Expand Up @@ -408,7 +452,14 @@
"cell_type": "code",
"execution_count": 7,
"id": "938c804e27f84196a10c8828c723f798",
"metadata": {},
"metadata": {
"execution": {
"iopub.execute_input": "2026-09-01T19:25:40.143725Z",
"iopub.status.busy": "2026-09-01T19:25:40.143636Z",
"iopub.status.idle": "2026-09-01T19:25:51.700051Z",
"shell.execute_reply": "2026-09-01T19:25:51.699427Z"
}
},
"outputs": [
{
"name": "stdout",
Expand Down Expand Up @@ -507,7 +558,14 @@
"cell_type": "code",
"execution_count": 8,
"id": "504fb2a444614c0babb325280ed9130a",
"metadata": {},
"metadata": {
"execution": {
"iopub.execute_input": "2026-09-01T19:25:51.701128Z",
"iopub.status.busy": "2026-09-01T19:25:51.701049Z",
"iopub.status.idle": "2026-09-01T19:25:51.877916Z",
"shell.execute_reply": "2026-09-01T19:25:51.877262Z"
}
},
"outputs": [
{
"data": {
Expand Down Expand Up @@ -542,7 +600,14 @@
"cell_type": "code",
"execution_count": 9,
"id": "59bbdb311c014d738909a11f9e486628",
"metadata": {},
"metadata": {
"execution": {
"iopub.execute_input": "2026-09-01T19:25:51.879675Z",
"iopub.status.busy": "2026-09-01T19:25:51.879586Z",
"iopub.status.idle": "2026-09-01T19:25:51.993036Z",
"shell.execute_reply": "2026-09-01T19:25:51.992556Z"
}
},
"outputs": [
{
"data": {
Expand Down Expand Up @@ -576,7 +641,14 @@
"cell_type": "code",
"execution_count": 10,
"id": "b43b363d81ae4b689946ece5c682cd59",
"metadata": {},
"metadata": {
"execution": {
"iopub.execute_input": "2026-09-01T19:25:51.994330Z",
"iopub.status.busy": "2026-09-01T19:25:51.994228Z",
"iopub.status.idle": "2026-09-01T19:25:52.258503Z",
"shell.execute_reply": "2026-09-01T19:25:52.257221Z"
}
},
"outputs": [
{
"data": {
Expand Down Expand Up @@ -641,15 +713,22 @@
"cell_type": "code",
"execution_count": 11,
"id": "c3933fab20d04ec698c2621248eb3be0",
"metadata": {},
"metadata": {
"execution": {
"iopub.execute_input": "2026-09-01T19:25:52.260535Z",
"iopub.status.busy": "2026-09-01T19:25:52.260446Z",
"iopub.status.idle": "2026-09-01T19:26:01.007678Z",
"shell.execute_reply": "2026-09-01T19:26:01.007245Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"level block in flight voxels (M) blocks build (s) seconds Mvoxel/s peak RSS (GB)\n",
" 3 222 8 2.4 1 0.0 15.4 0.2 1.57\n",
" 2 256 8 18.9 4 0.01 24.1 0.8 1.73\n"
" 3 222 8 2.4 1 0.0 3.7 0.6 1.81\n",
" 2 256 8 18.9 4 0.01 5.0 3.8 1.92\n"
]
}
],
Expand Down Expand Up @@ -741,19 +820,26 @@
"cell_type": "code",
"execution_count": 12,
"id": "06873177ff2d10fdd2e9ac23bd3ab778",
"metadata": {},
"metadata": {
"execution": {
"iopub.execute_input": "2026-09-01T19:26:01.008656Z",
"iopub.status.busy": "2026-09-01T19:26:01.008556Z",
"iopub.status.idle": "2026-09-01T19:26:25.656477Z",
"shell.execute_reply": "2026-09-01T19:26:25.656018Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"level block in flight voxels (M) blocks build (s) seconds Mvoxel/s peak RSS (GB)\n",
" 2 128 8 18.9 12 0.02 21.8 0.9 1.74\n",
" 2 128 16 18.9 12 0.02 13.7 1.4 1.72\n",
" 2 192 16 18.9 6 0.01 14.3 1.3 1.86\n",
" 2 192 24 18.9 6 0.01 21.9 0.9 1.86\n",
" 2 256 8 18.9 4 0.01 15.3 1.2 1.85\n",
" 2 256 16 18.9 4 0.01 15.1 1.3 1.84\n"
" 2 128 8 18.9 12 0.02 3.2 5.9 1.96\n",
" 2 128 16 18.9 12 0.02 3.9 4.9 1.96\n",
" 2 192 16 18.9 6 0.01 4.3 4.3 1.99\n",
" 2 192 24 18.9 6 0.01 3.3 5.7 2.05\n",
" 2 256 8 18.9 4 0.01 5.7 3.3 2.05\n",
" 2 256 16 18.9 4 0.01 4.1 4.6 2.02\n"
]
}
],
Expand Down Expand Up @@ -864,7 +950,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.13"
"version": "3.11.16"
}
},
"nbformat": 4,
Expand Down
Loading
Loading