Skip to content

[core] Introduce local cleanup for unreferenced managed BLOB packs. - #9609

Open
Stephen0421 wants to merge 1 commit into
apache:masterfrom
Stephen0421:support-clean-orphan-blob-core
Open

[core] Introduce local cleanup for unreferenced managed BLOB packs.#9609
Stephen0421 wants to merge 1 commit into
apache:masterfrom
Stephen0421:support-clean-orphan-blob-core

Conversation

@Stephen0421

Copy link
Copy Markdown
Contributor

Purpose

This is the first PR split from #9207.

PK managed BLOB packs (.managed.blob) are shared payloads, not first-class data files. Compaction reuses pack bytes without copying them, so snapshot expire only deletes the data file and its .blobref extra file. Unreferenced packs previously accumulated forever.

This change reclaims those packs through LocalManagedBlobOrphanFilesClean:

  • Collect reachability from live ADD data files' .blobref sidecars.
  • Join candidates by full pack identity (storageRootId + relativePath), not basename, so same-named packs in different directories are not mixed. Qualified and unqualified paths use the same URI path so hdfs:///warehouse/... still matches hdfs://nn:8020/warehouse/....
  • Collect used packs twice and abort the run (delete nothing) if the snapshot topology or used-pack set changed between those collections. That shrinks the compaction-reuse race; it is not a commit lease.
  • Delete unused .managed.blob files older than older_than (1 day by default).
  • If a listed sidecar on a still-existing data file is missing, corrupt, or unsupported, skip all pack deletes for that run. An empty valid sidecar is not missing: it means the file references no pack.
  • Unmerged ADD entries whose data files are already gone after expire are ignored, so they do not abort pack GC.

remove_orphan_files never deletes .managed.blob packs.

This is a conservative, best-effort first version. There is still no commit lease. Keep a non-zero older_than. Do not treat this as strictly proven concurrent-safe delete semantics.

Flink and Spark remove_orphan_blobs procedures are intentionally out of this PR and will follow in stacked PRs.

Tests

  • ManagedBlobReachabilityCollectorTest: empty extras / empty sidecar, referenced packs, missing sidecar (live vs already gone), corrupt / unsupported version, unsafe merge.
  • ManagedBlobOrphanFilesCleanTest: delete unreferenced pack, keep referenced pack, join by full pack path, empty sidecar does not block others, missing / corrupt / unsupported sidecar skips all packs, unreferenced pack after update+expire, abort when used set changes between the two collections, compaction interleaving after the final mark, and database executor cancellation / bounded termination.

PK managed BLOB packs are shared payloads. Compaction reuses pack bytes without copying them, so snapshot expire only deletes the data file and its .blobref; this change reclaims unreachable packs through LocalManagedBlobOrphanFilesClean.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant