Optimize commit performance of RewriteFilesAction

[nagraham]

Is your feature request related to a problem or challenge?

We observe that each attempt to execute the RewriteFileAction commit can take minutes, even hours, especially when the table has a large number of active manifests (+500 manifests for more). An increasing number of active manifest makes the performance worse. If a commit takes too long to run, it increases the likelihood that another write to the table will doom the commit to failure due to a 409 conflict. The RewriteFileAction transaction will retry on 409s, but it throws away all computed work, and starts from the beginning.

From investigating the underlying logic within replace_files.rs, I found that it will read the current snapshot's manifest-list file and all manifest files up to three times (there's a 3rd read if check_file_existence is set to true). To make matters worse, only one of those three batches of manifest reads is executed concurrently. This root cause aligns with our observation of degraded performance as the number of active manifests raises for tables.

Three manifest reads:

• Validation: concurrent via .buffer_unordered(available_parallelism()) in snapshot.rs:264-290.
• Survivor processing: sequential for loop with an awaited load in replace_files.rs:196-248.
• Deleted-entry generation: sequential for loop with an awaited load in replace_files.rs:140-165.

Describe the solution you'd like

I propose two optimizations to speed up commit attempts. If the commit can complete faster (despite a long list of active manifests), it has a better chance to commit before a conflicting write arrives. It also reduces the total time to commit, which is always a win.

1. Optimize Manifest Reads

Instead of scanning the current snapshot's manifest-list and full set of manifest files from file storage 3 times, we optimize to reduce this to as close to only 1 read as possible. And we make this read concurrent. This is a general performance improvement for the ReplaceFilesAction (and thus benefits Rewrite / Overwrite), and should speed up any transaction attempt with multiple active manifests in the snapshot.

The goals here would be to:

• Read the current snapshot's manifest list once per preparation attempt.
• Reduce source-manifest reads from approximately 3M to M + |A union D|, where A is the set of manifests containing requested removals and D is the set of delete manifests requiring cleanup inspection.
• Make both the validation pass and the targeted rewrite pass concurrent with an explicit bound.
• Keep peak memory independent of the total number of manifest entries in the table.
• Preserve file-existence validation and all existing rewrite semantics.
• Preserve the existing append-only retry optimization and prepared-manifest reuse.

2. Re-use Manifest Data on 409 Conflicts

This optimization is inspired by Iceberg's Java library. it's a bit more situational than the first optimization. If the commit encounters a 409 conflict error, then it will check the delta of commits that have occurred in the meantime. If it is safe to do so (e.g. all commits were merely Appends), then it will only read the new manifests and apply the delta to the state that is already staged for commit. It only benefits RewriteFilesAction, not the Overwrite case.

For a normal concurrent writer append, retry work should become:

1. Reload table metadata.
2. Read one refreshed manifest list.
3. Read only newly appended manifests.
4. Reuse previously written survivor, addition, deletion, and filtered-delete manifests.
5. Write one new manifest list and retry the catalog update.

No original manifest bodies should be reread or rewritten.

This only applies if all transactions which occurred since the start of the work up to the commit attempt are safe to rebuild on. This would not apply if any changes include:

• A cached source manifest is removed or replaced.
• Concurrent RewriteFiles, RewriteManifests, delete, or overwrite operations.
• Even logically equivalent manifest compaction triggers a hard refresh.
• A new delete manifest appears.
• Position deletes, equality deletes, or deletion vectors can change delete applicability.
• The table format version changes.
• A new data manifest is backdated or has an unassigned sequence number.
• A new data manifest contains Existing or Deleted entries rather than only Added entries.
• A concurrent append includes a file this action intends to remove.
• A concurrent append reuses the path of a removed data file.
• This matters for dangling delete and deletion-vector handling.
• With check_file_existence=true, a concurrent append includes a file this action intends to add.

Willingness to contribute

I can contribute to this feature independently