[core] Introduce local cleanup for unreferenced managed BLOB packs. - #9609
Open
Stephen0421 wants to merge 1 commit into
Open
[core] Introduce local cleanup for unreferenced managed BLOB packs.#9609Stephen0421 wants to merge 1 commit into
Stephen0421 wants to merge 1 commit into
Conversation
PK managed BLOB packs are shared payloads. Compaction reuses pack bytes without copying them, so snapshot expire only deletes the data file and its .blobref; this change reclaims unreachable packs through LocalManagedBlobOrphanFilesClean.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
This is the first PR split from #9207.
PK managed BLOB packs (
.managed.blob) are shared payloads, not first-class data files. Compaction reuses pack bytes without copying them, so snapshot expire only deletes the data file and its.blobrefextra file. Unreferenced packs previously accumulated forever.This change reclaims those packs through
LocalManagedBlobOrphanFilesClean:ADDdata files'.blobrefsidecars.storageRootId + relativePath), not basename, so same-named packs in different directories are not mixed. Qualified and unqualified paths use the same URI path sohdfs:///warehouse/...still matcheshdfs://nn:8020/warehouse/.....managed.blobfiles older thanolder_than(1 day by default).ADDentries whose data files are already gone after expire are ignored, so they do not abort pack GC.remove_orphan_filesnever deletes.managed.blobpacks.This is a conservative, best-effort first version. There is still no commit lease. Keep a non-zero
older_than. Do not treat this as strictly proven concurrent-safe delete semantics.Flink and Spark
remove_orphan_blobsprocedures are intentionally out of this PR and will follow in stacked PRs.Tests
ManagedBlobReachabilityCollectorTest: empty extras / empty sidecar, referenced packs, missing sidecar (live vs already gone), corrupt / unsupported version, unsafe merge.ManagedBlobOrphanFilesCleanTest: delete unreferenced pack, keep referenced pack, join by full pack path, empty sidecar does not block others, missing / corrupt / unsupported sidecar skips all packs, unreferenced pack after update+expire, abort when used set changes between the two collections, compaction interleaving after the final mark, and database executor cancellation / bounded termination.