[#12453] improvement(core): add OCC for schema writes - #12456
Draft
yuqi1129 wants to merge 3 commits into
Draft
Conversation
yuqi1129
force-pushed
the
feat/12342-occ-schema
branch
from
August 13, 2026 07:41
c4b723d to
9abb06d
Compare
Advance the metalake OCC version on every alter and guard alter and delete with a compare-and-set on the observed version, classifying a failed CAS as either a stale conflict or a missing entity. Keep the metalake root CAS and the non-empty check or the cascade cleanup inside one database transaction. A cascade locks the catalog rows first, then compare-and-set deletes descendant catalogs and schemas with their observed identifier-and-version pairs, so a concurrent child write is reported instead of silently dropped. Also add the shared OptimisticLockException factories used by the follow-up catalog and schema changes.
Advance the catalog OCC version on every alter and guard alter and delete with a compare-and-set on the observed version, classifying a failed CAS as either a stale conflict or a missing entity. Protect catalog creation with a shared lock on the parent metalake row on MySQL and PostgreSQL, without changing the parent version, so a catalog can no longer be created below a metalake that is being dropped. H2 uses an exclusive lock because it has no shared row-lock syntax. Keep the catalog CAS and the non-empty check or the cascade cleanup inside one database transaction, and CAS-delete descendant schemas with their observed identifier-and-version pairs.
Advance the schema OCC version on every alter and guard alter and drop with a compare-and-set on the observed version, classifying a failed CAS as either a stale conflict or a missing entity. Make managed schema creation insert-only so a concurrent same-name create returns SchemaAlreadyExistsException instead of overwriting the winner, and take a shared lock on the parent catalog row so a schema cannot be created below a catalog that is being dropped. Serialize hierarchical ancestor materialization and schema drops through the catalog row so overlapping cascades share one lock order. Lock the parent schema row before writing a table, view, fileset, function, model, or topic, and check views and functions before a non-cascade schema drop. Accepted tradeoff: a hierarchical schema create that materializes implicit ancestors takes an exclusive lock on the catalog row, because two concurrent creates can both find the same ancestor missing and both insert it, and a shared lock does not prevent that under MySQL REPEATABLE READ.
yuqi1129
force-pushed
the
feat/12342-occ-schema
branch
from
August 13, 2026 08:07
9abb06d to
531ca89
Compare
Code Coverage Report
Files
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Add database-backed optimistic concurrency control and transaction boundaries for schema writes.
Accepted tradeoff: a hierarchical schema create that materializes implicit ancestors takes an exclusive lock on the catalog row, so every other schema create under that catalog waits until that transaction ends, even when it touches a different ancestor path. The exclusive lock is needed because two concurrent creates can both find the same ancestor missing and both insert it, and a shared lock does not prevent that under MySQL REPEATABLE READ. Catalogs with heavy concurrent hierarchical schema creation will therefore serialize on this lock. If it becomes a bottleneck, a narrower fence — locking only the ancestor rows being created and relying on the unique constraint plus a retry — can be done in a follow-up.
Rebased on current
main(on top of #12374). Third of three PRs replacing #12350. Stacked on the catalog PR; review the top commit only. This PR also restores the two cross-entityTestMetalakeMetaServicetests that could not pass before schema writes took the catalog row lock.Why are the changes needed?
Managed schema operations previously consisted of multiple independent reads and writes. Concurrent alter, create, and drop requests could overwrite newer metadata, create children below a deleted parent, leave view and function rows orphaned, or run partial cascade cleanup. Overlapping hierarchical schema drops could also acquire descendant row locks in different orders.
Fix: #12453
Does this PR introduce any user-facing change?
Concurrent schema version conflicts are reported as HTTP 409. If the observed entity was deleted or renamed away, alter reports not found and drop preserves its idempotent false result. A managed schema create that loses a concurrent same-name create returns
SchemaAlreadyExistsExceptioninstead of overwriting the winner.How was this patch tested?
./gradlew :core:test :core:javadoc :catalogs:catalog-fileset:test :catalogs:catalog-kafka:test -PskipITs(H2)TestSchemaMetaService,TestMetalakeMetaService,TestFilesetCatalogOperations,TestKafkaCatalogOperations.-PskipDockerTests=false).