perf(manifest): project scan columns during reads

[fallintoplace]

Summary

• Projects manifest data-file reads to the fields needed for scan tasks.
• Keeps metric maps only when row pruning or positional-delete matching needs them.
• Drops transient metric maps before local Arrow tasks retain the files.
• Keeps PlanFiles() on the existing full-metadata path for callers that inspect task stats.
• Caches projected Avro schemas by writer schema and projection.

Benchmark

Command: go test . -run=^\$ -bench BenchmarkManifestEntryProjection -benchtime=1x -count=1

Apple M1 Pro, 10,000 entries per case:

Stats fields	Full bytes	Scan bytes	Full allocs	Scan allocs
10	32.6 MB	15.6 MB	443,928	65,797
100	260.2 MB	100.5 MB	2,257,763	77,930
1,000	2.93 GB	1.45 GB	20,512,582	332,865

Validation

• go test . ./table -count=1
• go test ./... -run=^\$ -count=1
• go test -race ./table -run=TestOpenManifestWithProjectionDropsStatsAfterFiltering -count=1
• go vet ./...
• golangci-lint v2.12.2

[zeroshade]

The projection keeps every field its consumers need with one exception that only became a problem while this PR was open. Flagging it before it bites on rebase.

Major — dropped data-file stats defeat #1960's equality-delete pruning (table/scanner.go, projection/stat-drop branch)

Stats are retained for data manifests only when rowFilter != AlwaysTrue. So ToArrowRecords with the default AlwaysTrue filter drops ValueCounts, NullCounts, NaNCounts, and the lower/upper bounds.

Those are exactly the statistics that #1960 (perf(table): prune equality deletes by data-file metrics) reads to decide an equality-delete file cannot match a data file. Without them, equalityDeleteCanContainData falls back to its conservative answer and attaches every equality delete to every candidate data file. The unfiltered full-scan case is precisely where that pruning pays off most, so the optimisation is silently neutralised in the case it was written for.

To be clear about the impact: this is a performance regression, not a correctness bug. Conservative means more deletes get attached than necessary — results stay correct and deletes still apply. Nothing returns wrong rows.

Also to be fair about the timing: #1960 merged today, almost certainly after this PR's base. The projection logic was correct against the tree you wrote it on; the interaction only becomes live once this rebases onto current main. This isn't an oversight on your part, it's two changes meeting.

Suggested fix: include data-file stats in the projection whenever the scan has equality deletes attached, independent of whether a row filter is present. The filter presence turns out to be the wrong signal for "are stats needed" now that a second consumer exists.

Minor — projection test matrix is thin

Coverage is v3 data manifests and v2 data manifests only. There's no v1 case, and no delete-manifest or DV-manifest case. Projection correctness is version-sensitive — field sets genuinely differ across v1/v2/v3 — so a matrix across versions and manifest kinds would be worth having, especially for delete manifests where a missing field silently changes delete application rather than erroring.

What I verified is intact

The reassuring half — every other consumer still receives its required fields:

• Partition values, for partition filtering and residuals.
• Task basics: file_path, file_format, record_count, file_size_in_bytes.
• Delete-file fields including content, equality_ids, and — importantly — referenced_data_file, content_offset, and content_size_in_bytes, all three of which deletion vectors require.
• Row lineage: first_row_id, plus entry-level sequence_number and file_sequence_number that gate whether equality deletes apply at all.
• status and snapshot_id.

CI green. Benchmark evidence is present.

---

This review was drafted by an AI-assisted tool and confirmed by an Apache Iceberg Go maintainer. The findings cite the project's review criteria; if you think one of them is mis-applied, please reply on the PR and a maintainer will weigh in.

More on how to contribute to Apache Iceberg Go: CONTRIBUTING.md

[laskoviymishka]

Second pass, catching up to the three commits since my last review.

The two things I was holding on are resolved. The partition-copy race is fixed: DataFileWithoutColumnStats now calls initPartitionData() before the clone shares fieldIDToPartitionData by reference, with a new concurrency test hammering it. And the delete path is in good shape now, which is exactly where I had it backwards last round. I'd suggested dropping delete-manifest stats under an AlwaysTrue filter as wasted work, but that would have broken equality-delete pruning, which needs the data file's bounds and the equality-delete file's bounds on either side of the range comparison. The new retainDataFileStats / manifestProjectionForManifest / manifestProjectionRetainsDataStats logic keeps data-file stats whenever any delete manifest is present, and the comment spells out why delete manifests always keep stats: the manifest-list metadata doesn't distinguish equality from positional deletes, so you can't safely narrow it. That's the right conservative call, and a correctness fix I'd missed. TestManifestProjectionRetainsDataFileStatsForDeleteScans and TestManifestEntryProjectionSupportsManifestVersionsAndDeletes (v1/v2/v3 data, v2 equality delete, v3 DV) now pin the decision logic and the reader across versions.

DataFileWithoutColumnStats still returns non-*dataFile inputs unchanged, but the doc comment now states the reason, so I'm happy with that being documented rather than changed.

What's left is non-blocking polish, noted inline: the projection cache still calls writerSchema.String() on every lookup and stores it as the key, which quietly gives back some of the allocation win this PR is going for; the projected schema node shares Props / Aliases with the writer schema by reference; and the field whitelist still zeroes block_size_in_bytes on the projected path versus a full read. None of these block merge.

Approve with nits from me. Nice work on the equality-delete pruning fix.

[laskoviymishka]

This one survived the rework, and it's the most worthwhile of what's left. We build the key with writerSchema.String() before the Get, so every lookup pays a full JSON serialization of the writer schema even on a hit. A scan opening a thousand manifests that share one schema does ~999 serializations of something that can run to tens of KB, and the same string is what we store as the key, so 256 entries can pin a few MB on a wide schema. For a PR whose whole point is cutting planning-time allocations, this quietly gives some of that back.

Could we key on a hash of the schema string, or the pointer identity of the *avro.Schema from reader.Schema(), and only touch String() on the miss path? wdyt?

[laskoviymishka]

projectedRoot := *root shallow-copies the node and we clone Fields, but Props and Aliases still alias the writer schema's, which is cached in ocf.Reader and can be shared across goroutines. CI is green so twmb/avro isn't mutating those in Schema() today, but it's an unstated assumption. A one-line comment noting we rely on Schema() being non-mutating would be enough. wdyt?

[laskoviymishka]

The new version/delete-type test is good coverage for the fields that are kept, so this is softer than last round. The remaining edge: default: return false still silently zeroes block_size_in_bytes on the projected path while a full NewManifestReader carries its real value (a required long in v1). It's deprecated and unused for planning so it's harmless today, but the two readers returning different DataFiles for the same manifest is the kind of thing that bites a future field. A test cross-checking this whitelist against the avro-tagged fields on dataFile would make an omission fail loudly instead of vanishing. Non-blocking. wdyt?

[laskoviymishka]

LGTM,

What's left is non-blocking polish, noted inline: the projection cache still calls writerSchema.String() on every lookup and stores it as the key, which quietly gives back some of the allocation win this PR is going for; the projected schema node shares Props / Aliases with the writer schema by reference; and the field whitelist still zeroes block_size_in_bytes on the projected path versus a full read. None of these block merge.

Approve with nits from me. Nice work on the equality-delete pruning fix.

[laskoviymishka]

This one survived the rework, and it's the most worthwhile of what's left. We build the key with writerSchema.String() before the Get, so every lookup pays a full JSON serialization of the writer schema even on a hit. A scan opening a thousand manifests that share one schema does ~999 serializations of something that can run to tens of KB, and the same string is what we store as the key, so 256 entries can pin a few MB on a wide schema. For a PR whose whole point is cutting planning-time allocations, this quietly gives some of that back.

Could we key on a hash of the schema string, or the pointer identity of the *avro.Schema from reader.Schema(), and only touch String() on the miss path? wdyt?

[laskoviymishka]

projectedRoot := *root shallow-copies the node and we clone Fields, but Props and Aliases still alias the writer schema's, which is cached in ocf.Reader and can be shared across goroutines. CI is green so twmb/avro isn't mutating those in Schema() today, but it's an unstated assumption. A one-line comment noting we rely on Schema() being non-mutating would be enough. wdyt?

[laskoviymishka]

The new version/delete-type test is good coverage for the fields that are kept, so this is softer than last round. The remaining edge: default: return false still silently zeroes block_size_in_bytes on the projected path while a full NewManifestReader carries its real value (a required long in v1). It's deprecated and unused for planning so it's harmless today, but the two readers returning different DataFiles for the same manifest is the kind of thing that bites a future field. A test cross-checking this whitelist against the avro-tagged fields on dataFile would make an omission fail loudly instead of vanishing. Non-blocking. wdyt?