[xieandrew]

Add basic implementation of DROP PARTITION for Iceberg:

• supports transformations in partition expression e.g icebergBucket and etc
• removes only manifests which were present on the moment of execution of a query
• does not support catalogs
• does not support schema evolution
• does not support partition evolution
• does not support "mixed manifests", when one manifest file has data files from different partitions

Changelog category (leave one):

• Experimental Feature

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Add basic implementation of DROP PARTITION for Iceberg (ClickHouse#111219 by @Diskein)

Documentation entry for user-facing changes

...

CI/CD Options

Exclude tests:

• Fast test
• Integration Tests
• Stateless tests
• Stateful tests
• Unit tests
• Performance tests
• Aarch64 tests
• All with ASAN
• All with TSAN
• All with MSAN
• All with UBSAN
• All with Coverage
• All Regression
• Disable CI Cache

Regression jobs to run:

• Fast suites (mostly <1h)
• [] Aggregate Functions (2h)
• Alter (1.5h)
• Benchmark (30m)
• CAS (content-addressed storage; Antalya only)
• ClickHouse Keeper (1h)
• Iceberg (2h)
• LDAP (1h)
• OAuth (5m)
• Parquet (1.5h)
• RBAC (1.5h)
• SSL Server (1h)
• S3 (2h)
• S3 Export (2h)
• Swarms (30m)
• Tiered Storage (2h)

[github-actions]

Workflow [PR], commit [e5e6582]

[xieandrew]

@blau-ai

[blau-ai]

CI triage — PR #2533 (Iceberg DROP PARTITION backport)

Verdict: 0 PR-caused failures. Every red check is pre-existing infra / environment / test-data, not a product regression from this change. The PR's own code is healthy — all builds and all of its new tests pass.

This PR touches only Iceberg data-lake code (src/Storages/ObjectStorage/DataLakes/Iceberg/*, StorageObjectStorage, a new FailPoint, an avro submodule bump) plus its own tests. None of the failing checks exercise any of that.

✅ Green — including everything this PR adds

• Builds: all amd (debug, asan+ubsan, binary, release, wasm_parser) — Failures: 0.
• Fast test 0 failed · Quick functional 0 failed.
• Stateless (amd_debug parallel/sequential, s3-storage, asan+ubsan): Failed: 0 — this is where the PR's new tests ran and passed (04114_iceberg_drop_partition, 04250…edge_cases, 04262…history…summary, 04613…whole_manifest, 05213…time_travel, 05214…explicit_metadata).
• Integration (amd_asan_ubsan ×8 shards): Failures: 0.
• Unit tests (asan+ubsan): 0 fail / 20696 passed — includes the new gtest_iceberg_snapshot_summary.
• Stress (amd_debug, amd_asan_ubsan) · AST fuzzer · BuzzHouse · SQLLogic · SQLStorm · Compatibility · Install packages · Docker server/keeper images — all OK.

❌ Red — all pre-existing / infra, none PR-related

Check	Root cause (from logs)	Classification
Parser memory check	Errors at setup step Resolve master binary: No master clickhouse-examples artifact was found. The job never ran — it couldn't fetch the master baseline binary it diffs against.	CI infra (missing baseline artifact)
Grype Scan (keeper, server, server-alpine)	Container-image CVE scan flagging vulnerabilities in base image / bundled libs. No dependency, Dockerfile, or lib change in this PR.	Infra / image scan
Regression → settings (1840/1846 failed)	Every scenario fails identically: SnapshotNotFoundError … default_values.py.default values>=26.8_antalya.snapshot. The version-specific baseline snapshot file doesn't exist, so the whole suite errors out.	Missing test-data baseline
Regression → s3/azure (2 failed)	Server never becomes healthy: URL "http://azurite:10000" is not allowed in configuration file, see <remote_url_allow_hosts> + SSL … tlsv1 alert unknown ca. Config/allowlist + cert env problem; never reaches Iceberg code.	Env / config setup
Regression → oauth (9 failed)	Bearer-token auth assertions (assert http_code == status_code) in the external-OAuth flow. Unrelated subsystem.	Pre-existing / flaky
Regression → swarms (4 failed)	Code: 279 … ALL_CONNECTION_TRIES_FAILED on an s3Cluster(...) query against minio — cluster replica connectivity. Not the Iceberg engine.	Infra / connectivity
PR (aggregate)	Red solely because the checks above are red.	Rollup, not independent

Corroboration

Sibling backport PR #2543 (same antalya-26.8 base) shows the same red checks independent of content — Parser memory check: fail, Grype keeper/alpine: fail, Regression swarms: fail — while its Iceberg regression suites (iceberg_1, iceberg_2) pass. That's the signature of branch-level infra flakiness, not a regression introduced here.

Suggested next steps

None of these are fixable in this PR — there's no code change to make:

• Parser memory check / Grype / settings-snapshot: infra/baseline issues owned by CI. Re-running won't help the parser check or the settings snapshot until the master clickhouse-examples artifact and the >=26.8_antalya settings snapshot exist on the branch.
• s3/azure, oauth, swarms regression: environment/connectivity; safe to re-run, but they're not gated on this change.

Recommend treating the red checks as known branch infra and proceeding on the strength of the green functional/unit/integration results (which include all the new DROP-PARTITION coverage). If a maintainer wants, I can re-trigger the environment-sensitive regression suites, but I'd expect no change.

_{Analysis is from CI artifacts (praktika result_pr.json, regression report.html, sibling PR checks). I can't build/run ClickHouse in this job; correctness of the change itself is validated by the green test suites above.}

[DimensionWieldr]

AI audit (Grok 4.7) found a defect, which apparently also exists upstream. I'm unsure if this should be an issue we open upstream, or if we want to make the fix here. Generally, we open issues upstream when we can (when it's a base-CH bug and not Antalya-specific).

Overwrite summary fields are copied onto the delete snapshot and can abort the commit

Impact: DROP PARTITION throws BAD_ARGUMENTS (Duplicate key in SnapshotSummary) and leaves the table unchanged when the parent snapshot is an overwrite that already recorded removed position or equality deletes and the dropped partition still has those delete files. If this drop removes none of those files, the commit succeeds but the new snapshot keeps the parent's removed-position-deletes / removed-equality-deletes / removed-delete-files values, so the delete snapshot's deltas do not describe this commit.

Anchor: SnapshotSummary::fromJSON / SnapshotSummary::toJSON, called from AlterDropPartitionExecutor::writeManifestList

Trigger: Parent snapshot operation is overwrite and its summary contains a positive removed-position-deletes (Spark copy-on-write merge/delete or a rewrite that drops delete files). Then drop a partition that still has a position-delete file.

Why defect: The overwrite parser never marks those keys as consumed, so with_extra_fields treats them as opaque properties. The delete serializer writes the same keys again, and toJSON rejects the duplicate. This happens before the manifest list is written, so the failure is fail-closed, but a valid drop does not commit.

Fix direction (short): Do not copy summary keys that the new operation emits; keep only stable properties.

[DimensionWieldr]

Decided to open an issue upstream (125922).
We can ignore this bug for now, since it applies more broadly to SnapshotSummary than to drop partition itself.

[DimensionWieldr]

Otherwise, CI looks fine.

Because this backport does not support catalogs, schema evolution, partition evolution, or a manifest that holds files from more than one partition, the drop partition tests in regression aren't very helpful in diagnosing anything at the moment. Once we get support for those (maybe in another PR?), then the tests will come in handy.

If no further changes are to be made, LGTM.