Feature request: add option to skip writing generic empty group metadata

[tlambert03]

I have a use case (writing complex, complete ome-zarr hierarchies using yaozarrs to generate and write all of the group nodes, with any array-writing library (zarr-python, tensorstore) to just write the actual array bytes. i.e. leave the hard performance-critical stuff to C++, but handle finicky metadata on the python side. I'm quite happy with the pattern, and would like to be able to use acquire-zarr with it.

This one line here causes me problems when I try to use the pattern with acquire-zarr (with specified ArraySettings.output_keys only to write the array-nodes)

acquire-zarr/src/streaming/zarr.stream.cpp

Lines 1473 to 1475 in 80a3949

	} else { // generic group
	metadata_str = group_metadata.dump(4);
	}

it results in acquire-zarr always writing an empty zarr.json group (over-writing whatever is there) for all intermediate directories required for ArraySettings.output_key relative to root. Moreover, because it happens later, it's not sufficient to just wait for acquire-zarr to write the file when creating the stream and then overwrite it (instead, you have to queue some function to run only after acquire-zarr has run its finalization logic).

my current workaround

  # Backup complete zarr.json file structure prior to creating zarr stream
  self._zarr_json_backup: dict[Path, bytes] = {} 

  stream = az.ZarrStream()
  stream.close()

  # later...
  # Restore yaozarrs metadata that acquire-zarr overwrote
  for path, content in self._zarr_json_backup.items():
      path.write_bytes(content)

... however, that workaround is causing me problems on Windows, where I have so far been unable to get around Windows file-permissions write-access problems

Question:

@aliddell, would you consider a PR that either adds a public-facing option write_empty_groups

        } else { // generic group
            if (!write_empty_groups) {                                      
              continue;
            }
            metadata_str = group_metadata.dump(4);
        }

... or just turns off the empty-group writing logic without a new public API? What would your preference be? new public API flag? or just continue.?

[aliddell]

Question:

@aliddell, would you consider a PR that either adds a public-facing option write_empty_groups

    } else { // generic group
        if (!write_empty_groups) {                                      
          continue;
        }
        metadata_str = group_metadata.dump(4);
    }

... or just turns off the empty-group writing logic without a new public API? What would your preference be? new public API flag? or just continue.?

What about checking to see if a file is there before writing, and backing out of writing one if it's already there (unless overwrite is set to True of course)?

[tlambert03]

that's an improvement... but is there any need to write the empty group metadata at all? The zarr spec doesn't care right?

and even if overwrite is set to True... I still don't want it to overwrite my meaningful data there with empty metadata.

Point is, if I'm interpreting your intended semantics of output_key correctly to mean "don't put this inside of an ngff group"

az.StreamSettings(
    arrays=[
        az.ArraySettings(
            output_key="my_array",
        )
    ],
    store_path="store",
)

then I don't want acquire-zarr to ever touch store/zarr.json... only stuff inside of store/my_array. That would let me write whatever metadata i want at those higher levels, and use acquire-zarr as a pure array-writing library. This argument extends to plates and more complex stuff. I want to use something else to set up the metadata, and just tell acquire-zarr to append this frame to that array over there...

[aliddell]

I'm not sure what the spec says about it exactly, but the reader gives some unusual behavior. I set up an experiment with the following directory structure:

> tree /f
Folder PATH listing for volume Windows
Volume serial number is ACDE-9564
C:.
│   zarr.json
│
└───path
    └───to
        └───array
            │   zarr.json
            │
            └───c
                ├───0
                │       0
                │       1
                │
                └───1
                        0
                        1

We have an array at path/to/array, with zarr.json under path/to/array and a zarr.json at the root of the dataset, but no intermediate metadata. You can access the array if you know what you're looking for, but zarr-python will throw a warning about a child (path) not being a component of a Zarr hierarchy.

In [1]: import zarr

In [2]: d = zarr.open("nested-no-intermediate.zarr/")

In [3]: for k in d.keys():
   ...:     print(k)
   ...:
C:\repos\acquire-zarr\venv\Lib\site-packages\zarr\core\group.py:3530: ZarrUserWarning: Object at path is not recognized as a component of a Zarr hierarchy.
  warnings.warn(

In [4]: arr = d["path/to/array"]

In [5]: arr.shape
Out[5]: (64, 128)

If I use zarr-python to create a new array nested underneath the base, it does end up creating those intermediate metadata files:

In [13]: import numpy as np

In [14]: d.create_array("a/new/array", shape=(2,48,64), dtype=np.uint32)
Out[14]: <Array file://nested-no-intermediate.zarr/a/new/array shape=(2, 48, 64) dtype=uint32>

In [15]: for k in d.members():
    ...:     print(k)
    ...:
C:\repos\acquire-zarr\venv\Lib\site-packages\zarr\core\group.py:3530: ZarrUserWarning: Object at path is not recognized as a component of a Zarr hierarchy.
  warnings.warn(
('a', <Group file://nested-no-intermediate.zarr/a>)

> tree /f
Folder PATH listing for volume Windows
Volume serial number is ACDE-9564
C:.
│   zarr.json
│
├───a
│   │   zarr.json
│   │
│   └───new
│       │   zarr.json
│       │
│       └───array
│               zarr.json
│
└───path
    └───to
        └───array
            │   zarr.json
            │
            └───c
                ├───0
                │       0
                │       1
                │
                └───1
                        0
                        1

So without looking at the spec (though I will), I'm inclined to say that those intermediate metadata files are necessary after all.

[tlambert03]

I think it all comes down to how you interpret the role of acquire-zarr. If you see it as

• acquire-zarr own everything inside of ZarrStream.store_path
• acquire-zarr always make ZarrStream.store_path a zarr group
• we expect to be able to inspect the entire tree structure using the zarr-python zarr.Group API

Then your example makes sense: since the path you're providing to zarr.open is a group. and you're using group-like inspection API.

But if all the above is always true, that's sort of imposing a high-level API interpretation on what is otherwise (in my opinion) a library whose real strength is in the low-level high performance array-writing capabilities. Just as zarr-python has the capability to write zarr.Array without a group, and tensorstore only has an Array API (can't write groups), I think it would be nice if acquire-zarr had the option of only writing array nodes. With a relatively simple, unmagic mental model of one array node for each ArraySettings:

import acquire_zarr as az
import numpy as np

settings = az.StreamSettings(
    arrays=[
        az.ArraySettings(
            output_key="array0",
            dimensions=[...],
            data_type=np.uint16,
        ),
        az.ArraySettings(
            output_key="array1",
            dimensions=[...],
            data_type=np.uint16,
        ),
        az.ArraySettings(
            output_key="deeply/nested/array2",
            dimensions=[...],
            data_type=np.uint16,
        ),
    ],
    store_path="root_path",
)

stream = az.ZarrStream(settings)
stream.append(..., key="array0")
stream.append(..., key="array1")
stream.append(..., key="deeply/nested/array2")
stream.close()

would do:

$ tree root
root
├── array0
|   ├── c
|   └── zarr.json
├── array1
|   ├── c
|   └── zarr.json
└── deeply
    └── nested
        └── array2
            ├── c
            └── zarr.json

and

import zarr

zarr.open_array("root_path/array0")
zarr.open_array("root_path/array1")
zarr.open_array("root_path/deeply/nested/array2")

would work fine.

It does mean that the user doesn't have the convenience of using zarr.open("root_path") and the zarr-python zarr.Group API to inspect the full hierarchy... and that's why I propose that it's optional. But I think it's a straightforward interpretation, and a reasonable access to the "heart" of the low-level API here?