perf: collect prune used ids in sharded maps after the walk by BradKollmyer · Pull Request #568 · rustic-rs/rustic_core

BradKollmyer · 2026-09-07T00:32:55Z

Part of #564.

Stacked on #567. Review the unique range:

BradKollmyer/rustic_core@perf/prune-json...perf/prune-used-ids

Hash used-blob ids from the first 8 SHA-256 bytes.
Skip unused prune tree JSON strings with a word-wise scan, then memchr.
Collect used blob ids in per-loader vecs; fill the map in parallel shards after the walk.
Parse short JSON keys without memchr; pre-size used-id vecs at 4M then allocate on demand.
Shard the used-id map on prefix bits 28–31 (identity hasher already uses the low bits).

OpenDAL only sends maxFileCount when ListOptions.limit is set. Without it, B2 defaults to 100 names per page, so prune/check pack listing becomes thousands of round-trips. Request 10000 for scheme b2.

256 KiB MAX_HOLESIZE split prune/restore pack reads into extra HTTP range GETs on high-latency stores. 4 MiB is cheaper than another RTT on a ~100 Mbps link.

Unused gaps in a pack were fetched one after another. Issue those range GETs in parallel; the packer already serializes writes.

stream_list used CPU-count Rayon workers and a zero-capacity channel, so reading index/snapshots was one GET at a time on high-latency backends. Use 16-32 workers with prefetch so GETs overlap decrypt/parse.

TreeStreamerOnce was hardcoded to 4 workers, which left B2 tree-pack GETs idle. Use 2x CPUs (8-32) with a larger out channel, and a HashSet for visited tree ids.

The used-id set was a BTreeMap, so inserts got slower as prune walked more blobs. A HashMap keeps that path O(1).

A single Vec of blob ids doubles on growth. On a large repo that request is hundreds of MiB while the old buffer is still live, and musl aborts (openzwave: memory allocation of 807665664 bytes failed at reading index 906/1551). Store ids and full entries in 1 Mi-entry chunks and binary-search each sorted chunk.

Cache read_partial opened and closed the pack file for every tree blob. Keep up to 2048 FDs and pread from them. Hits skip LockPool and exists(); write/remove drop the cached handle.

Prune only needs file content ids and directory subtree ids. Deserialize a compact UsedBlobsTree instead of a full Vec<Node>, matching restic's streaming tree walk. Check and copy still load complete trees.

TreeStreamer dumped every snapshot root into a FIFO, so loaders jumped across packs. Keep a LIFO backlog and only feed one job per loader so consecutive trees tend to hit the same cached packs, matching restic.

getting packs was a serial B2 prefix list after finding used blobs. Start that list in the background so it finishes during the tree walk.

Overlapping the B2 pack list with index GETs made reading index... take minutes on a cold-ish server cache. List packs only during the tree walk so getting packs still hides without stalling the index.

16–32 parallel index readers help B2 RTTs but on a warm local cache they thrash disk. Use 2–4 workers when at least 75% of listed index/snapshot files already exist in the cache.

32 loaders on a 16-vCPU QEMU host spent most of finding used blobs... in KVM PV spinlocks and musl malloc. Use 4–8 loaders when the cache already has pack files; keep 2×CPUs (8–32) for a cold remote cache.

`i as u8` trips clippy::cast_possible_truncation and cast_sign_loss, which CI runs with -D warnings. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TdgDzcDaQVR4uqEPMDKZsu

rlimit/8 mapped Darwin ulimit 8192 to 1024 open pack files and added about 5s to prune getting-packs. Keep the 2048 cap and a 64-FD reserve.

nix getrlimit returns rlim_t, which is u32 on armv7.

decode_all created a DCtx for every compressed tree blob. During prune finding used blobs that showed up as ZSTD_createDCtx, munmap, and ~60k page faults/s. Keep one bulk Decompressor in thread-local storage.

hex::FromHex<&str> was ~6% of finding used blobs after zstd reuse. Parse the 64-char ids with a lookup table during UsedBlobsTree JSON decode instead.

A single consumer HashMap::insert serialized the used-blob walk. Loaders now insert into a 16-way sharded map while they decode trees.

A derived UsedBlobNode still paid serde skip/parse of every field. Visit only type, content, and subtree and push ids straight into UsedBlobsTree.

Sharded Mutex HashMaps still spent ~35% on insert and lock contention. Each tree loader now owns a HashMap and the maps are merged after the walk.

siphash HashMap insert of BlobId was ~27% of finding used blobs after locks were removed. FxHash is enough for in-memory blob ids.

Unused node fields (names, mtime, xattrs) were still UTF-8-validated and skipped through serde. A dedicated restic-tree scanner only hex-decodes file content and dir subtree ids.

FxHash of the full 32-byte id plus map growth was ~16% of finding used blobs. SHA-256 ids are uniform, so the prefix is the hash; equality stays 32 bytes. Per-loader maps start at 2M slots.

skip_string_body was ~5% of finding used blobs, scanning names/mtime/xattrs a byte at a time. Restic strings almost never contain escapes; scan 16-byte chunks for quote or backslash instead.

HashMap insert of 32-byte ids was ~14% of finding used blobs plus memcmp during the walk. Push ids during the tree load and sort+dedup when the loader finishes.

Serial HashMap extend of ~22M ids after per-loader vecs added ~10s of wall. Partition by id prefix into 16 shards and build each HashMap on Rayon so merge is not single-threaded.

find_unescaped_quote was ~5% of finding used blobs with a scalar 16-byte word scan. memchr2 uses SIMD to find quote or backslash.

parse_key was ~5% of finding used blobs because AVX2 memchr startup dominates on 4–20 byte keys. Scan those scalar; keep memchr for long unused strings.

kernel_init_pages was ~11% of finding used blobs as per-loader vecs grew past 2M. 4M slots is 128 MiB per loader and covers typical unique-blob counts without realloc.

clippy::needless_borrows_for_generic_args, which CI runs with -D warnings. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TdgDzcDaQVR4uqEPMDKZsu

Shards were picked from the low 4 bits of the id prefix, but the identity hasher hands that same prefix to hashbrown, which takes bucket positions from the low bits. Every key in a shard then shared its bucket bits and probed more. Bits 28-31 are used by neither the bucket index nor the top-7-bit tag. A 16M-id micro-benchmark showed 15-45% faster lookups; not measured on a real repository. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TdgDzcDaQVR4uqEPMDKZsu

BradKollmyer added 6 commits September 6, 2026 14:49

fix: request B2's maximum list page size

9210d41

OpenDAL only sends maxFileCount when ListOptions.limit is set. Without it, B2 defaults to 100 names per page, so prune/check pack listing becomes thousands of round-trips. Request 10000 for scheme b2.

fix: fetch 4 MiB unused pack gaps in one read

d468d44

256 KiB MAX_HOLESIZE split prune/restore pack reads into extra HTTP range GETs on high-latency stores. 4 MiB is cheaper than another RTT on a ~100 Mbps link.

fix: range-GET prune repack chunks in parallel

a75f3c8

Unused gaps in a pack were fetched one after another. Issue those range GETs in parallel; the packer already serializes writes.

fix: prefetch index files with extra IO workers

b99e719

stream_list used CPU-count Rayon workers and a zero-capacity channel, so reading index/snapshots was one GET at a time on high-latency backends. Use 16-32 workers with prefetch so GETs overlap decrypt/parse.

fix: scale tree loaders during prune used-blob walk

ec07d6a

TreeStreamerOnce was hardcoded to 4 workers, which left B2 tree-pack GETs idle. Use 2x CPUs (8-32) with a larger out channel, and a HashSet for visited tree ids.

fix: store prune used blob ids in a HashMap

43365be

The used-id set was a BTreeMap, so inserts got slower as prune walked more blobs. A HashMap keeps that path O(1).

This was referenced Sep 7, 2026

perf: reuse per-thread zstd output buffer for prune trees #569

Open

perf: speed up prune used-blob walk on a warm pack cache #564

Open

BradKollmyer force-pushed the perf/prune-used-ids branch from 9f13d68 to ba3d186 Compare September 7, 2026 01:02

BradKollmyer and others added 21 commits September 7, 2026 13:40

fix: keep cached pack files open for tree blob reads

0464477

Cache read_partial opened and closed the pack file for every tree blob. Keep up to 2048 FDs and pread from them. Hits skip LockPool and exists(); write/remove drop the cached handle.

fix: stream-decode trees during prune used-blob walk

4674695

Prune only needs file content ids and directory subtree ids. Deserialize a compact UsedBlobsTree instead of a full Vec<Node>, matching restic's streaming tree walk. Check and copy still load complete trees.

fix: walk unique trees depth-first

721dc03

TreeStreamer dumped every snapshot root into a FIFO, so loaders jumped across packs. Keep a LIFO backlog and only feed one job per loader so consecutive trees tend to hit the same cached packs, matching restic.

fix: list pack files during prune index and tree walk

c65570c

getting packs was a serial B2 prefix list after finding used blobs. Start that list in the background so it finishes during the tree walk.

fix: start prune pack listing after the index is loaded

5e7996f

Overlapping the B2 pack list with index GETs made reading index... take minutes on a cold-ish server cache. List packs only during the tree walk so getting packs still hides without stalling the index.

fix: use fewer stream_list workers on a warm cache

b664070

16–32 parallel index readers help B2 RTTs but on a warm local cache they thrash disk. Use 2–4 workers when at least 75% of listed index/snapshot files already exist in the cache.

fix: use fewer tree loaders when pack files are cached

188e5bc

32 loaders on a 16-vCPU QEMU host spent most of finding used blobs... in KVM PV spinlocks and musl malloc. Use 4–8 loaders when the cache already has pack files; keep 2×CPUs (8–32) for a cold remote cache.

fix: build cache test payloads without lossy int casts

4ef023f

`i as u8` trips clippy::cast_possible_truncation and cast_sign_loss, which CI runs with -D warnings. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TdgDzcDaQVR4uqEPMDKZsu

fix: bound cached handles and synchronize portable range reads

34d586c

fix: size cached pack handles from rlimit minus a reserve

3631eab

rlimit/8 mapped Darwin ulimit 8192 to 1024 open pack files and added about 5s to prune getting-packs. Keep the 2048 cap and a 64-FD reserve.

fix: convert rlimit to u64 for 32-bit prune cache cap

a9716cd

nix getrlimit returns rlim_t, which is u32 on armv7.

fix: reuse a zstd decompressor per thread for blob reads

be68643

decode_all created a DCtx for every compressed tree blob. During prune finding used blobs that showed up as ZSTD_createDCtx, munmap, and ~60k page faults/s. Keep one bulk Decompressor in thread-local storage.

fix: decode prune tree blob ids with a hex nibble table

7961821

hex::FromHex<&str> was ~6% of finding used blobs after zstd reuse. Parse the 64-char ids with a lookup table during UsedBlobsTree JSON decode instead.

fix: record prune used blob ids from loader threads

186a925

A single consumer HashMap::insert serialized the used-blob walk. Loaders now insert into a 16-way sharded map while they decode trees.

fix: stream-decode prune tree nodes without full node structs

19c41bd

A derived UsedBlobNode still paid serde skip/parse of every field. Visit only type, content, and subtree and push ids straight into UsedBlobsTree.

fix: fill prune used blob ids in per-loader maps

9aa8740

Sharded Mutex HashMaps still spent ~35% on insert and lock contention. Each tree loader now owns a HashMap and the maps are merged after the walk.

fix: use FxHash for prune used-blob maps and tree visited set

461977d

siphash HashMap insert of BlobId was ~27% of finding used blobs after locks were removed. FxHash is enough for in-memory blob ids.

fix: scan prune tree JSON without serde node field parse

8d37cf5

Unused node fields (names, mtime, xattrs) were still UTF-8-validated and skipped through serde. A dedicated restic-tree scanner only hex-decodes file content and dir subtree ids.

fix: preserve escaped JSON references during prune

b29a5cd

fix: hash prune used-blob ids from the first 8 SHA-256 bytes

940950e

FxHash of the full 32-byte id plus map growth was ~16% of finding used blobs. SHA-256 ids are uniform, so the prefix is the hash; equality stays 32 bytes. Per-loader maps start at 2M slots.

BradKollmyer and others added 9 commits September 7, 2026 13:41

fix: skip unused prune tree JSON strings with a word-wise scan

45f8e57

skip_string_body was ~5% of finding used blobs, scanning names/mtime/xattrs a byte at a time. Restic strings almost never contain escapes; scan 16-byte chunks for quote or backslash instead.

fix: collect prune used blob ids in per-loader vecs

ff16a97

HashMap insert of 32-byte ids was ~14% of finding used blobs plus memcmp during the walk. Push ids during the tree load and sort+dedup when the loader finishes.

fix: fill prune used-id map in parallel shards after the walk

03132bd

Serial HashMap extend of ~22M ids after per-loader vecs added ~10s of wall. Partition by id prefix into 16 shards and build each HashMap on Rayon so merge is not single-threaded.

fix: skip unused prune tree JSON strings with memchr

00f23d3

find_unescaped_quote was ~5% of finding used blobs with a scalar 16-byte word scan. memchr2 uses SIMD to find quote or backslash.

fix: parse short prune tree JSON keys without memchr

a40b5fc

parse_key was ~5% of finding used blobs because AVX2 memchr startup dominates on 4–20 byte keys. Scan those scalar; keep memchr for long unused strings.

fix: pre-size prune used-id vecs at 4M entries

250f83c

kernel_init_pages was ~11% of finding used blobs as per-loader vecs grew past 2M. 4M slots is 128 MiB per loader and covers typical unique-blob counts without realloc.

fix: pass Copy ids to hash_one by value in used-id tests

d95c108

clippy::needless_borrows_for_generic_args, which CI runs with -D warnings. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TdgDzcDaQVR4uqEPMDKZsu

fix: allocate prune loader IDs on demand

2740ef2

BradKollmyer force-pushed the perf/prune-used-ids branch from ba3d186 to 2740ef2 Compare September 7, 2026 20:41

Navigation Menu

Uh oh!

perf: collect prune used ids in sharded maps after the walk - #568

perf: collect prune used ids in sharded maps after the walk#568
BradKollmyer wants to merge 36 commits into
rustic-rs:mainfrom
BradKollmyer:perf/prune-used-ids

BradKollmyer commented Sep 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

BradKollmyer commented Sep 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant