Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
a417371
remove funny-looking line and refactor to ensure reading consolidated…
TomNicholas Mar 13, 2025
8756919
parametrize over whether or not we write consolidated metadata
TomNicholas Mar 13, 2025
b85d70d
fix consolidated metadata
TomNicholas Mar 13, 2025
f1cc331
ian hcanges
ianhi Mar 13, 2025
296ed03
open_datatree_specific_group consolidated true works
ianhi Mar 13, 2025
4da72ae
test: add consolidated parametrize to zarr datatree test
ianhi Mar 13, 2025
5f7c6b9
fix: group finding behavior consolidated
ianhi Mar 13, 2025
5dc7df7
Merge remote-tracking branch 'ianhi/aladinor/ian/updates' into dtree_…
TomNicholas Mar 17, 2025
9823d64
remove more debugging print statements
TomNicholas Mar 17, 2025
980ebb4
Merge branch 'dtree-zarrv3' into dtree-zarrv3-2
TomNicholas Mar 17, 2025
30f5bba
revert changes to test fixture
TomNicholas Mar 18, 2025
4d1fdb5
formatting
TomNicholas Mar 18, 2025
ecef578
add decorator to parametrize over zarr formats
TomNicholas Mar 18, 2025
c2a1f5f
ensure both versions of zarr-python and both versions of zarr-python …
TomNicholas Mar 18, 2025
cde6b65
change datatree fixture to not produce values that would be fill_valu…
TomNicholas Mar 18, 2025
09fad6e
refactor test to make expected behaviour clearer
TomNicholas Mar 18, 2025
77575b5
fix wrongly expected behaviour - should not expect inherited variable…
TomNicholas Mar 19, 2025
0a9f874
make arrays no longer scalars to dodge https://github.com/pydata/xarr…
TomNicholas Mar 19, 2025
565938b
Merge branch 'dtree-zarrv3-2' of https://github.com/TomNicholas/xarra…
TomNicholas Mar 19, 2025
daf0f42
fix bad merge
TomNicholas Mar 19, 2025
84bde40
parametrize almost every test over zarr_format
TomNicholas Mar 19, 2025
04d937c
parametrize encoding test over zarr_formats
TomNicholas Mar 19, 2025
765c5f0
use xfail in encoding test
TomNicholas Mar 19, 2025
7eee31c
updated expected behaviour of zarr on-disk in light of https://github…
TomNicholas Mar 19, 2025
0969422
fully revert change to simple_datatree test fixture by considered zar…
TomNicholas Mar 19, 2025
cacf419
parametrize unaligned_zarr test fixture over zarr_format
TomNicholas Mar 19, 2025
1a60ebe
move parametrize_over_zarr_format decorator to apply to entire test c…
TomNicholas Mar 19, 2025
d98abe3
for now explicitly consolidate metadata in test fixture
TomNicholas Mar 19, 2025
2dcefe4
correct bug in writing of consolidated metadata
TomNicholas Mar 19, 2025
a88e503
delete commented-out lines
TomNicholas Mar 19, 2025
22ac9b4
merges from main
TomNicholas Mar 19, 2025
69dc976
Revert "merges from main"
TomNicholas Mar 19, 2025
6e3e2aa
fix encodings test for zarr_format=3
TomNicholas Mar 19, 2025
6ce9578
tidy up
TomNicholas Mar 19, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion xarray/backends/zarr.py
Original file line number Diff line number Diff line change
Expand Up @@ -1047,7 +1047,7 @@ def store(
kwargs = {}
if _zarr_v3():
# https://github.com/zarr-developers/zarr-python/pull/2113#issuecomment-2386718323
kwargs["path"] = self.zarr_group.name.lstrip("/")

@TomNicholas TomNicholas Mar 19, 2025

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI @TomAugspurger @jhamman and I are 95% sure that this line was unnecessary and a bug. It was causing consolidated metadata to not be written at the store root each time, and now that we changed it, it fixed various buggy behaviour we were seeing.

relevant to zarr-developers/zarr-python#2113 (comment)

# kwargs["path"] = self.zarr_group.name.lstrip("/")
kwargs["zarr_format"] = self.zarr_group.metadata.zarr_format
zarr.consolidate_metadata(self.zarr_group.store, **kwargs)

Expand Down
12 changes: 3 additions & 9 deletions xarray/tests/test_backends_datatree.py
Original file line number Diff line number Diff line change
Expand Up @@ -157,36 +157,30 @@ def _unaligned_datatree_zarr(zarr_format: Literal["2", "3"]) -> Path:
set1_data = xr.Dataset({"a": 0, "b": 1})
set2_data = xr.Dataset({"a": ("y", [2, 3]), "b": ("x", [0.1, 0.2])})

# consolidated metadata is deliberately written only at the root
root_data.to_zarr(
filepath, mode="w", zarr_format=zarr_format, consolidated=False
filepath,
mode="w",
zarr_format=zarr_format,
)
set1_data.to_zarr(
filepath,
group="/Group1",
mode="a",
zarr_format=zarr_format,
consolidated=False,
)
set2_data.to_zarr(
filepath,
group="/Group2",
mode="a",
zarr_format=zarr_format,
consolidated=False,
)
set1_data.to_zarr(
filepath,
group="/Group1/subgroup1",
mode="a",
zarr_format=zarr_format,
consolidated=False,
)

import zarr

zarr.consolidate_metadata(filepath, zarr_format=zarr_format)

return filepath

yield _unaligned_datatree_zarr
Expand Down