Skip to content

feat(iceberg): add v3 deletion vector support - #1162

Open
vikaxsh wants to merge 66 commits into
stagingfrom
feat/iceberg-deletion-vectors
Open

feat(iceberg): add v3 deletion vector support#1162
vikaxsh wants to merge 66 commits into
stagingfrom
feat/iceberg-deletion-vectors

Conversation

@vikaxsh

@vikaxsh vikaxsh commented Aug 26, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Description

Adds dv as a third per-stream delete mode alongside eq and pos, writing Iceberg v3 Puffin deletion vectors instead of Parquet positional deletes.

Introduces PositionalDeleteSink so PositionalDeltaWriter can target either representation, plus DeletionVectorConverter and PreviousDeleteLoader to merge new deletes with existing Parquet positional deletes or deletion vectors without resurrecting previously deleted rows.

Updates EqualityDeleteMigrator.migrate() to accept a targetMode, allowing tables to switch from equality deletes directly to pos or dv encoding in a single atomic RewriteFiles commit.

Fixes #1168

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

How Has This Been Tested?

  • Scenario A
  • Scenario B

Screenshots or Recordings

Documentation

  • Documentation Link: [link to README, olake.io/docs, or olake-docs]
  • N/A (bug fix, refactor, or test changes only)

Related PR's (If Any):

hash-data and others added 30 commits July 15, 2026 15:43
@hash-data
wi[
b8d965b
@hash-data
chore: interoperability with pebble db, in next commit we might chang…
72a5db5 
…e so for reference it is
@hash-data
resolving conflict with master and some changes in flow
2467688
@hash-data
chore: till now pushing all batch level things where we can create in…
f45ed95 
…dex on the fly
@hash-data
chore: tested eq to pos migration and incremental snapshot upgrade + …
d17a330 
…emitting pos deltes
@hash-data
Merge branch 'staging' into feat/interoperability
60f410c
@hash-data
Merge branch 'staging' into feat/interoperability
78a0f4c
@hash-data
chore: rm unnecessary package
5a98fc8
@hash-data
chore: integration test for interoperability
d7f551b
@hash-data
Merge branch 'staging' into feat/interoperability
f37565d
@hash-data
chore: fixing integration test
f781a0e
@hash-data
chore: add refresh cache in integration test
9ee5c55
@hash-data
chore: changing sync mode in integration tests
6247ccd
@hash-data
chore: refactoring in go side remains protobuf and java side
8d59899
@hash-data
fix: integration tests
2b5fa92
@hash-data
chore: no need for diff store and index
81da415
@hash-data
chore: remove deprecated function
b9f4714
@hash-data
chore: fixing case of external commit, and reviewed java side
67096c9
@hash-data
feat: finaly everything is fine
b34e99b
@hash-data
Merge branch 'staging' into feat/interoperability
fdc7a9d
@hash-data
chore: comment resolution and product discussion
fb3b430
@hash-data
chore: merge staging branch
1503cfc
@hash-data
chore: merge staging
8c8b476
@hash-data
chore: fixing lint
b2fdcd8
@hash-data
chore: fixing flow according to last product discussion and only iceb…
7851939 
…erg validation test pending
@hash-data
chore: adding validations
aebe1ca
@hash-data
chore: merge staging
292fdef
@hash-data
chore: delete granularity is partition now
79d1405
@hash-data
chore: fix unit test and workflow
986fc84
@hash-data
fix: for conflict check and first snapshot
92e7348
@vikaxsh
vikaxsh temporarily deployed to integration_tests August 28, 2026 05:32 — with GitHub Actions Inactive
@vikaxsh
vikaxsh temporarily deployed to integration_tests August 28, 2026 05:32 — with GitHub Actions Inactive
@vikaxsh
vikaxsh temporarily deployed to integration_tests August 28, 2026 05:32 — with GitHub Actions Inactive
@vikaxsh
vikaxsh temporarily deployed to integration_tests August 28, 2026 05:32 — with GitHub Actions Inactive
@vikaxsh
vikaxsh temporarily deployed to integration_tests August 28, 2026 05:50 — with GitHub Actions Inactive
@vikaxsh
vikaxsh temporarily deployed to integration_tests August 28, 2026 05:50 — with GitHub Actions Inactive
@vikaxsh
vikaxsh temporarily deployed to integration_tests August 28, 2026 05:50 — with GitHub Actions Inactive
@vikaxsh
vikaxsh temporarily deployed to integration_tests August 28, 2026 05:50 — with GitHub Actions Inactive
@vikaxsh
vikaxsh temporarily deployed to integration_tests August 28, 2026 05:50 — with GitHub Actions Inactive
@vikaxsh
vikaxsh temporarily deployed to integration_tests August 28, 2026 05:50 — with GitHub Actions Inactive
@vikaxsh
vikaxsh temporarily deployed to integration_tests August 28, 2026 05:50 — with GitHub Actions Inactive
@vikaxsh
vikaxsh temporarily deployed to integration_tests August 28, 2026 05:50 — with GitHub Actions Inactive
@vikaxsh
vikaxsh requested a deployment to integration_tests August 28, 2026 06:34 — with GitHub Actions Waiting
@vikaxsh
vikaxsh requested a deployment to integration_tests August 28, 2026 06:34 — with GitHub Actions Waiting
@vikaxsh
vikaxsh requested a deployment to integration_tests August 28, 2026 06:34 — with GitHub Actions Waiting
@vikaxsh
vikaxsh requested a deployment to integration_tests August 28, 2026 06:34 — with GitHub Actions Waiting
@vikaxsh
vikaxsh requested a deployment to integration_tests August 28, 2026 06:34 — with GitHub Actions Waiting
@vikaxsh
vikaxsh requested a deployment to integration_tests August 28, 2026 06:34 — with GitHub Actions Waiting
@vikaxsh
vikaxsh requested a deployment to integration_tests August 28, 2026 06:34 — with GitHub Actions Waiting
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@vikaxsh @hash-data @vaibhav-datazip