Skip to content

EXPORT PARTITION: force-replace and the new Keeper key can duplicate or strand exports #2576

Open

Description

@DimensionWieldr

Related: #2548

Two high-severity defects in the plain and replicated EXPORT PARTITION changes on antalya-26.8. Codex already filed both as P1 review comments on commit aaf00db (2026-10-12). The follow-up commit 5705da65 does not fix them. This issue tracks them so they are not left only as review comments.

Defect 1 — Force-replace commits the old export and the new one

Codex: #2548 (comment)
Anchor: MergeTreePartitionExportScheduler::addTask, MergeTreeData::killExportPart, MergeTreePartitionExportScheduler::tryCommit

KILL refuses to cancel a plain MergeTree export while committing is set. export_merge_tree_partition_force_export does not. It persists a new transaction id, replaces the in-memory task, and calls killExportPart. That only sets a cancel flag and drops the part manifest immediately. It does not wait for the part writer to stop, and it does not stop tryCommit if the destination commit has already started.

The old Iceberg commit then publishes a snapshot under the old transaction id. tryCommit discards that result locally because the registry now holds the new id. The replacement export commits the same files again under a different id. Iceberg idempotency is per transaction id, so both snapshots are appended and the current snapshot lists the rows twice. With overwrite, the replacement can also rewrite data files while the first commit is still publishing.

How to reproduce

  1. Start EXPORT PARTITION from a plain MergeTree table to an Iceberg destination with enough data that the part upload or the Iceberg commit is still running.
  2. While that task is in tryCommit (or while a part write is still running), run the same EXPORT PARTITION with export_merge_tree_partition_force_export = 1.
  3. Read the destination Iceberg table and its snapshots.

Expected: one snapshot for the replacement export. The previous transaction does not publish.
Actual: two snapshots append the same files. Concurrent writers can also share one part after the old manifest is removed.

Defect 2 — The new Keeper key does not match existing export znodes

Codex: #2548 (comment)
Anchor: ExportPartitionUtils::compositeKey, ExportReplicatedMergeTreePartitionTaskEntry::getCompositeKey, ExportPartitionManifestUpdatingTask::removeStaleEntries, StorageReplicatedMergeTree::exportPartitionToTable

Replicated export task keys changed from {partition_id}_{database}.{table} to {escaped partition}.{escaped database}.{escaped table}. poll still discovers the old Keeper child and inserts its manifest, but the in-memory index is the new key. removeStaleEntries then compares that new key with the old child name, treats the task as deleted, calls killExportPart, and erases it before the model is published. The scheduler never resumes it.

A later EXPORT PARTITION of the same partition checks only the new path, so a completed export stored under the old key does not conflict. The new task gets a new transaction id and appends the data again.

How to reproduce

  1. On a build from before this key change, run EXPORT PARTITION on a ReplicatedMergeTree table so Keeper has an exports/{partition_id}_{database}.{table} znode. Leave one PENDING and one COMPLETED.
  2. Upgrade to this PR.
  3. Check system.partition_exports and whether the PENDING task keeps exporting.
  4. Run EXPORT PARTITION again for the COMPLETED partition without export_merge_tree_partition_force_export.

Expected: existing tasks stay visible and keep their old Keeper path. A completed export still blocks a second export unless force is set.
Actual: the PENDING task is dropped from memory on every poll and stalls until timeout. The COMPLETED task is invisible, and a second export appends another snapshot.

Fix direction

  • Reject or defer force-replace while committing is set, and do not schedule the replacement until the previous transaction's part tasks have stopped.
  • Keep the Keeper child name as the task identity, or migrate old nodes, and resolve the legacy key in the exists-check, scheduler, and stale-entry pass.

Activity

  1. changed the title [-]EXPORT PARTITION: force-replace and the new Keeper key can duplicate or strand exports (PR #2548)[/-] [+]EXPORT PARTITION: force-replace and the new Keeper key can duplicate or strand exports[/+] on Oct 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    antalyabugSomething isn't workingexport partitionIssues related to export partition functionality

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions