[WIP] Internal threading + in-place codec API for sharded reads - #265

DocumenterVitepress: No favicon.ico file found in `docs/src/assets`. Skipping favicon replacement.

DocumenterVitepress: No logo.png file found in `docs/src/assets`. Skipping logo replacement.

DocumenterVitepress: un-expanded `@jldoctest` block encountered on page src/UserGuide/operations.md. The first few lines of code in this node are: ``` julia> mean(t_w, dims = (1,2))./mean(latweights) 1×1×1980 Array{Float64,3}: [:, :, 1] = 12.492234157689309 [:, :, 2] = ```

DocumenterVitepress: un-expanded `@jldoctest` block encountered on page src/UserGuide/operations.md. The first few lines of code in this node are: ``` julia> latweights = reshape(cosd.(g["lat"])[:],1,192,1); julia> t_celsius = g["tas"].-273.15 Disk Array with size 288 x 192 x 1980 julia> t_w = t_celsius .* latweights ```

DocumenterVitepress: un-expanded `@jldoctest` block encountered on page src/UserGuide/operations.md. The first few lines of code in this node are: ``` julia> using Zarr, Statistics julia> g = zopen("gs://cmip6/CMIP/NCAR/CESM2/historical/r9i1p1f1/Amon/tas/gn/", consolidated=true) ZarrGroup at Consolidated S3 Object Storage Variables: lat time tas lat_bnds lon_bnds lon time_bnds ```

77 docstrings not included in the manual: Zarr.Codecs.V3Codecs._codecs_to_v3pipeline :: Tuple{Vector{<:Zarr.Codecs.V3Codecs.V3Codec}} Zarr.desttype Zarr.Codecs.V3Codecs.encode_shard_index :: Union{Tuple{N}, Tuple{Zarr.Codecs.V3Codecs.ShardIndex{N}, Zarr.Codecs.V3Codecs.ShardingCodec}} where N Zarr.Codecs.V3Codecs.compute_encoded_index_size :: Union{Tuple{N}, Tuple{NTuple{N, Int64}, Zarr.Codecs.V3Codecs.ShardingCodec}} where N Zarr.ConsolidatedStore Zarr.Codecs.V3Codecs.validate_index_pipeline :: Tuple{Zarr.V3Pipeline} Zarr.V2Pipeline Zarr.Codecs.V3Codecs.getCodec :: Union{Tuple{Dict}, Tuple{Dict, Any}} Zarr.Codecs.V3Codecs.getCodec :: Union{Tuple{Vector}, Tuple{Vector, Any}} Zarr.Codecs.V3Codecs.decode_shard_index :: Union{Tuple{N}, Tuple{Vector{UInt8}, NTuple{N, Int64}, Zarr.Codecs.V3Codecs.ShardingCodec}} where N Zarr.Codecs.V3Codecs.ChunkShardInfo Zarr.gcs_credentials :: Tuple{Any, Any, Any} Zarr.gcs_credentials :: Tuple{} Zarr.Codecs.V3Codecs.codec_parsers Zarr.Codecs.V3Codecs.set_chunk_slice! :: Union{Tuple{N}, Tuple{Zarr.Codecs.V3Codecs.ShardIndex, NTuple{N, Int64}, Int64, Int64}} where N Zarr.chunkindices :: Tuple{ZArray} Zarr.Codecs.V3Codecs.read_shard_partial! :: Union{Tuple{N}, Tuple{T}, Tuple{AbstractArray{T, N}, NTuple{N, Int64}, Zarr.Codecs.V3Codecs.ShardingCodec{N}, Union{Nothing, Vector{UInt8}}, NTuple{N, Int64}, NTuple{N, Int64}, NTuple{N, UnitRange{Int64}}, Any}} where {T, N} Zarr.MetadataV2 Zarr.V3Pipeline Zarr.Codecs.V3Codecs.get_chunk_slice :: Union{Tuple{N}, Tuple{Zarr.Codecs.V3Codecs.ShardIndex, NTuple{N, Int64}}} where N Zarr.Codecs.V3Codecs.get_chunk_slice_in_shard :: Union{Tuple{N}, Tuple{NTuple{N, Int64}, NTuple{N, Int64}, NTuple{N, Int64}}} where N Zarr.VLenArrayFilter Zarr.get_order :: Tuple{Zarr.MetadataV3} Zarr.HTTPStore Zarr.sourcetype Zarr.Codecs.V3Codecs.ShardingCodec Zarr.AbstractStore Zarr.Codecs.V3Codecs.read_shard_partial_with_source! :: Union{Tuple{F}, Tuple{N}, Tuple{T}, Tuple{AbstractArray{T, N}, NTuple{N, Int64}, Zarr.Codecs.V3Codecs.ShardingCodec{N}, Vector{UInt8}, F, NTuple{N, Int64}, NTuple{N, Int64}, NTuple{N, UnitRange{Int64}}, Any}} where {T, N, F} Zarr._readblock_sharded_partial! :: Union{Tuple{N}, Tuple{Any, ZArray, CartesianIndices{N, R} where R<:NTuple{N, OrdinalRange{Int64, Int64}}, CartesianIndices{N, R} where R<:NTuple{N, OrdinalRange{Int64, Int64}}, NTuple{N, Int64}, Any, Any}} where N Zarr.trans_ind :: Tuple{AbstractUnitRange, Any} Zarr.missing_chunk_return_code! :: Tuple{Zarr.ConsolidatedStore, Any} Zarr.Codecs.V3Codecs.set_chunk_empty! :: Union{Tuple{N}, Tuple{Zarr.Codecs.V3Codecs.ShardIndex, NTuple{N, Int64}}} where N Zarr.concurrent_io_tasks Zarr.Codecs.V3Codecs.sharding_codec :: Tuple{Zarr.V3Pipeline} Base.resize! :: Union{Tuple{N}, Tuple{T}, Tuple{ZArray{T, N, S, M} where {S<:Zarr.AbstractStore, M<:(Zarr.AbstractMetadata{T, N})}, NTuple{N, T} where T}} where {T, N} Zarr.FixedScaleOffsetFilter Zarr.AbstractMetadata Zarr.lower_chunk_key_encoding :: Tuple{Zarr.ChunkKeyEncoding} Zarr.ChunkKeyEncodingEntry StructUtils.lower :: Tuple{Zarr.Codecs.V3Codecs.ShardingCodec} Zarr.Codecs.V3Codecs.calculate_chunks_per_shard :: Union{Tuple{N}, Tuple{NTuple{N, Int64}, NTuple{N, Int64}}} where N Zarr.MetadataV3 :: Union{Tuple{E}, Tuple{N}, Tuple{T2}, Tuple{Any, Any, NTuple{N, Int64}, NTuple{N, Int64}, String, Any}} where {T2, N, E} Zarr.MetadataV3 Zarr.Metadata :: Union{Tuple{C}, Tuple{N}, Tuple{T}, Tuple{AbstractArray{T, N}, NTuple{N, Int64}}, Tuple{AbstractArray{T, N}, NTuple{N, Int64}, Any}} where {T, N, C} Zarr.Metadata :: Tuple{AbstractDict, Any} Zarr.Codecs.V3Codecs._shard_index_byte_range :: Union{Tuple{N}, Tuple{Zarr.Codecs.V3Codecs.ShardingCodec{N}, NTuple{N, Int64}, Int64}} where N Zarr.Fletcher32Filter Zarr.subkeys Zarr.Codecs.V3Codecs.register_codec :: Union{Tuple{T}, Tuple{Function, String, Type{T}}} where T<:Zarr.Codecs.V3Codecs.V3Codec Zarr.getfilter Zarr.chunk

invalid local link/image: file does not exist in docs/src/get_started.md link = @ast MarkdownAST.Link("./tutorials/tutorial", "") do MarkdownAST.Text("User Guide") end

invalid local link/image: file does not exist in docs/src/get_started.md link = @ast MarkdownAST.Link("./UserGuide/storage", "") do MarkdownAST.Text("Storage Backends") end

Cannot resolve @ref for md"[`readblock!`](@ref)" in docs/src/UserGuide/partial_shard_reads.md. - No docstring found in doc for binding `Zarr.readblock!`. - No docstring found in doc for binding `Main.readblock!`.