feat(iceberg): experimental deletion vector delete mode

[shubham19may]

Stacks on #1118. Reference implementation, not intended to merge as is — see Risks.

Writes Iceberg v3 deletion vectors as a third delete mode, alongside equality and positional deletes. Built now, while the context from the positional work is fresh, so the eventual implementation has a worked example and a test matrix to start from rather than a blank page.

Why it is cheap

Everything needed is already on the classpath — no new dependency, no Puffin plumbing written by hand:

Need	Iceberg 1.10.2 provides
Write vectors	BaseDVFileWriter — delete(path, pos, spec, partition), the same shape the existing call already had
Merge with a file's existing vector	BaseDeleteLoader.loadPositionDeletes(files, path) (iceberg-data, already a dependency)
Read vectors in the row index scanner	the same loader — it handles Puffin and parquet, so it replaces existing code
Identify vectors	ContentFileUtil.isDV
Retired-vector bookkeeping	DeleteWriteResult.rewrittenDeleteFiles()

What changed

Delete representation is named, not boolean. delete_mode ("eq" / "pos" / "dv") travels on the handshake, with use_positional_deletes kept behind it so a server built before this still behaves. MigrateEqualityDeletesRequest carries target_mode the same way, so a table leaving equality deletes lands directly on whichever representation the job now writes.

Generated stubs were regenerated with protoc 23.4 (taken from the Maven repo), not the system protoc 35.1 — 35.1 emits protobuf-java 4.x APIs (RuntimeVersion) that the pinned 3.25.5 runtime does not have. Only a message field changed, so the gRPC service stubs are untouched and no protoc-gen-grpc-java was needed.

PositionalDeltaWriter no longer knows how a delete is encoded. Its delete side is a PositionalDeleteSink with two implementations:

• PositionalFiles — the per-partition sorting writers and the file-scoped granularity that lets Iceberg match deletes to data files by path.
• DeletionVectors — wraps BaseDVFileWriter. Iceberg permits one vector per data file, so a commit deleting further rows from a file must publish the union of old and new positions and retire the vector it replaces. PreviousDeleteLoader supplies the existing positions — reading either representation, so a half-migrated table is handled — and the retired files are reported so the commit can removeDeletes them.

Threading the sink through also removed the per-row PartitionKey copy: a partition's data writer and partition value are now held together, so recording where a row landed allocates one small object rather than a copied key.

Format versions. Tables are created at the version their mode needs (v2 for eq/pos, v3 for dv), and an existing v2 table is upgraded in place when a job switches to vectors.

Row index scanner resolves deleted positions through Iceberg's delete loader instead of reading positional delete parquet directly, so it sees vectors, delete files, or a table part way between them. This is a net simplification — the hand-rolled BitSet decoding is gone.

EqualityDeleteMigrator takes a target mode and writes vectors or delete files from the same plan, still in one atomic RewriteFiles so readers never observe the rows as undeleted. eq → pos and eq → dv are both supported; pos → dv is deliberately out of scope for now.

Testing

Unit — 36 tests

Every positional scenario from #1118 is now a @ParameterizedTest over both POSITION and DELETION_VECTOR, against a real table on a local Hadoop catalog, committed through RowDelta and read back with a normal scan:

• unpartitioned inserts; delete superseding a committed row
• same key updated twice and three times within one batch
• same key updated across batches (chained)
• partitioned fan-out with interleaved records
• partitioned delete carries the target's partition
• partitioned same-key updates in one batch
• 10k-row batch: 1000 keys × 10 rounds → exactly 1000 live rows
• file rolling: 5000 rows at a 4 KB target, then superseding every row across the rolled files
• partition-changing updates
• the delta reports the data files its deletes depend on, across partitions

Vector-specific coverage on top:

• deletionVectorMergesWithTheOneAlreadyOnTheDataFile — a second commit deleting a different row of the same data file must publish the union; asserts the earlier deletion is not resurrected and that the previous vector is reported as retired
• aDataFileEndsUpWithExactlyOneDeletionVector — three successive commits each delete another row of the same seed file; asserts exactly one vector survives, and that every commit after the first retires its predecessor
• equalityDeletesMigrateIntoDeletionVectors — seeds real equality deletes, migrates, then asserts none survive and every remaining delete is a vector

End to end — Postgres → Iceberg, --delete-type dv

Scenario	Result
70k-row backfill, 4 tables	all counts exact
25k-record CDC round	big_table 50000 / 50000 distinct, 0 stale intermediates, 3000 final versions
Chunked + partitioned table	chunk_table 20000 / 20000 distinct, 50 rows moved partitions cleanly
Delete file format	PUFFIN, content=1, referenced_data_file set — real vectors
Vector density	12000 deletes collapsed into 2 vectors (one per referenced data file), where positional mode wrote separate files
Partitioned CDC	part_table 6 / 6, including a partition-changing update and a soft delete
Cursor-based incremental	4 / 4, updated row superseded
eq → dv mid-life switch	job seeded in eq mode, then run with --delete-type dv: table upgraded v2 → v3, 1 equality delete file rewritten into 2 vectors, current snapshot left with only PUFFIN deletes, data correct

Risks — why this should not ship yet

Engine support is the deciding factor, and it is a product question, not an engineering one. This whole line of work exists for Snowflake compatibility. Positional deletes are v2 and universally read. Vectors are v3: Spark 3.5 with Iceberg 1.9+ reads them, but Snowflake's v3 support is far newer and may not cover vectors at all. Shipping this could narrow interop rather than widen it. Worth confirming against Snowflake's current v3 support before investing further.

FORMAT_VERSION is a one-way door. v2 → v3 upgrades in place and cannot be reverted; any reader that does not speak v3 loses the table. Every existing olake table is v2. The upgrade is logged at WARN, but it is silent from the user's point of view.

Vectors depend on the commit validation from #1118. Committing a vector without validateFromSnapshot makes Iceberg scan from the first snapshot and read your own previous vector as "concurrently added" (MergingSnapshotProducer.validateAddedDVs). Production sets it via applyRowIndexValidations; my test harness initially did not, which is how I found this. Vector mode is not viable without that change.

PreviousDeleteLoader plans the table once per writer. Correct, and only files actually deleted from are loaded — but on a table with many delete files this is a real cost at writer creation, and it has not been profiled at scale.

pos → dv migration is not implemented. A table already carrying positional delete files will not be converted. PreviousDeleteLoader reads both forms, so the two can coexist, but that path is untested.

Note on the test tree

The tests live in src/test/java/.../PositionalDeltaWriterTest.java and had to be added with git add -f: .gitignore carries a bare test pattern that matches any directory of that name, including src/test, so the whole Java test tree is ignored repo-wide. That is also why this project had surefire configured but no tests until now. The pattern is left alone here — narrowing it could unignore other paths — but it is worth fixing separately.

pom.xml gains junit-jupiter at test scope; without it the branch does not compile.

Since every positional scenario is parameterised over both modes, this file also covers the writer from #1118. Running mvn test on this branch exercises both.