Skip to content

[#12297] feat(secret): Add catalog and schema create-time secrets - #12420

Open
lasdf1234 wants to merge 32 commits into
apache:mainfrom
lasdf1234:feat/entity-secrets-create-catalog-schema
Open

[#12297] feat(secret): Add catalog and schema create-time secrets#12420
lasdf1234 wants to merge 32 commits into
apache:mainfrom
lasdf1234:feat/entity-secrets-create-catalog-schema

Conversation

@lasdf1234

@lasdf1234 lasdf1234 commented Aug 11, 2026

Copy link
Copy Markdown
Collaborator

What changes were proposed in this pull request?

Follow-up to #12366 (fileset create-time secrets). Adds create-time
secretBindings / secretReferences for catalog and schema:

  • API / REST / Java & Python clients / OpenAPI
  • Create path: assembleSecretMaterialswriteSecrets; failure rollback via rollbackSecrets; drop cleanup via deleteSecretsFromProperties
  • testConnection rejects raw secret URNs via checkSecretKeys
  • Catalog posthook still drops on owner/future-grant failure (pre-existing); schema hook does not drop on setOwner failure (aligned with fileset / [#12297] feat(secret): Support fileset create-time secretBindings/secretReferences #12366)
  • listCatalogsInfo authorize-before-resolve

Why are the changes needed?

Fix: #12297

Create-time secret bindings/references were added for fileset in #12366;
catalog and schema need the same create-path support.

Does this PR introduce any user-facing change?

  • Yes: catalog/schema create APIs accept optional secretBindings /
    secretReferences (same model as fileset).

How was this patch tested?

# Format
./gradlew spotlessApply

# Core: catalog/schema create + secret rollback
./gradlew :core:test \
  --tests 'org.apache.gravitino.catalog.TestCatalogManager.testCreateWithSecrets' \
  --tests 'org.apache.gravitino.catalog.TestCatalogManager.testCreateSecretRollback' \
  --tests 'org.apache.gravitino.catalog.TestCatalogManager.testCreateSecretSkippedNoMetalake' \
  --tests 'org.apache.gravitino.catalog.TestSchemaOperationDispatcher.testCreateWithSecrets' \
  -PskipITs

# Hooks (catalog posthook rollback retained; schema no posthook drop)
./gradlew :core:test \
  --tests 'org.apache.gravitino.hook.TestCatalogHookDispatcher' \
  --tests 'org.apache.gravitino.hook.TestSchemaHookDispatcher' \
  -PskipITs

# Shared secret helpers
./gradlew :core:test \
  --tests 'org.apache.gravitino.secret.TestSecretManager' \
  --tests 'org.apache.gravitino.secret.TestSecretPropertyUtils' \
  -PskipITs

# REST layer
./gradlew :server:test \
  --tests 'org.apache.gravitino.server.web.rest.TestCatalogOperations' \
  --tests 'org.apache.gravitino.server.web.rest.TestSchemaOperations' \
  -PskipITs

# OpenAPI (if docs changed)
./gradlew :docs:build

# Python client (optional local check)
cd clients/client-python
python -m compileall gravitino/dto/requests/catalog_create_request.py \
  gravitino/dto/requests/schema_create_request.py \
  gravitino/client/gravitino_metalake.py \
  gravitino/client/gravitino_client.py \
  gravitino/client/base_schema_catalog.py

lasdf1234 and others added 12 commits August 11, 2026 18:11
Follow-up to the fileset-only create secrets PR: wire secretBindings and
secretReferences through catalog/schema API, REST, clients, OpenAPI,
dispatchers, drop cleanup, hook rollback, and listCatalogsInfo auth
before resolve.

Co-authored-by: Cursor <cursoragent@cursor.com>
…cret helpers

Rebase onto the fileset create-secrets tip, switch to rollbackBindings /
deleteBindingsFromProperties / checkSecretKeys, and shorten catalog/schema
secret unit tests to match apache#12366 cleanup.

Co-authored-by: Cursor <cursoragent@cursor.com>
…rial APIs

Use assembleSecretMaterials, writeSecrets(List), rollbackSecrets, and
deleteSecretsFromProperties after rebasing onto main post-apache#12366.

Co-authored-by: Cursor <cursoragent@cursor.com>
…che#12366 style

Match fileset create-time secrets conventions: empty secret maps in Java
and Python DTOs/clients, remove schema posthook drop on setOwner failure,
and drop the unrelated GlueCatalogOperations Javadoc-only change.

Co-authored-by: Cursor <cursoragent@cursor.com>
Authorize details=true via listCatalogs then loadCatalog so secrets
resolve only for authorized catalogs, without a new API method.

Co-authored-by: Cursor <cursoragent@cursor.com>
…eset

Match FilesetCatalog defaults, SchemaCreateRequest Lombok/builder,
and Python empty-map defaults / response validate.

Co-authored-by: Cursor <cursoragent@cursor.com>
Fold secret rollback into needClean, with an explicit rollback on
EntityAlreadyExists where the existing catalog must be kept.

Co-authored-by: Cursor <cursoragent@cursor.com>
…unchanged

Remove the extra checkSecretKeys calls; testConnection only probes
connectivity and should not take part in create-time secret wiring.

Co-authored-by: Cursor <cursoragent@cursor.com>
…elds

Match CatalogCreateRequest style: final fields, JsonCreator, and
constructor-based client usage instead of Lombok builder.

Co-authored-by: Cursor <cursoragent@cursor.com>
…lizeDispatcher

createSchema now validates assembled entityProperties (empty map when
request properties is null), so the test must supply required key k1.

Co-authored-by: Cursor <cursoragent@cursor.com>
…no secrets

copyEntityProperties returns null only when properties and both secret
maps are absent, and assembleSecretMaterials skips URN writes in that
case so validatePropertyForCreate keeps historical null-skip behavior.

Co-authored-by: Cursor <cursoragent@cursor.com>
FrontendIT testClickCatalogLink timed out waiting for hive schema
"default" while metastore was still warming; prior FrontendIT on this
PR passed. Empty commit to re-run checks.

Co-authored-by: Cursor <cursoragent@cursor.com>
@github-actions

github-actions Bot commented Aug 12, 2026

Copy link
Copy Markdown

Code Coverage Report

Overall Project 68.69% +0.34% 🟢
Files changed 78.82% 🟢

Module Coverage
aliyun 1.72% 🔴
api 49.23% -0.53% 🟢
authorization-common 85.96% 🟢
aws 42.04% 🟢
azure 2.47% 🔴
catalog-common 9.92% 🔴
catalog-fileset 79.77% 🟢
catalog-glue 68.95% 🟢
catalog-hive 79.4% 🟢
catalog-jdbc-common 45.7% 🟢
catalog-jdbc-doris 81.8% 🟢
catalog-jdbc-mysql 79.33% 🟢
catalog-jdbc-postgresql 83.39% 🟢
catalog-jdbc-starrocks 79.16% 🟢
catalog-kafka 77.01% 🟢
catalog-lakehouse-generic 59.18% 🟢
catalog-lakehouse-hudi 79.1% 🟢
catalog-lakehouse-iceberg 85.86% 🟢
catalog-lakehouse-paimon 84.23% 🟢
catalog-model 77.72% 🟢
cli 44.48% 🟢
client-java 78.65% +4.51% 🟢
common 52.75% -0.05% 🟢
core 83.63% -0.75% 🟢
filesystem-hadoop3 77.28% 🟢
flink 0.0% 🔴
flink-common 48.68% 🟢
flink-runtime 0.0% 🔴
gcp 14.12% 🔴
hadoop-auth 68.0% 🟢
hadoop-common 12.7% 🔴
hive-metastore-common 53.4% 🟢
iceberg-aliyun-bundle 0.0% 🔴
iceberg-common 64.75% 🟢
iceberg-rest-server 75.03% 🟢
idp-basic 86.02% 🟢
integration-test-common 0.0% 🔴
jobs 62.92% 🟢
lance-common 31.75% 🔴
lance-rest-server 63.47% 🟢
lineage 53.02% 🟢
optimizer 83.24% 🟢
optimizer-api 21.95% 🔴
server 87.44% +0.76% 🟢
server-common 79.41% 🟢
spark 28.57% 🔴
spark-common 45.89% 🟢
tencent 69.84% 🟢
trino-connector 40.29% 🟢
Files
Module File Coverage
api SupportsCatalogs.java 0.0% 🔴
SupportsSchemas.java 0.0% 🔴
Fileset.java 0.0% 🔴
FilesetCatalog.java 0.0% 🔴
client-java BaseSchemaCatalog.java 100.0% 🟢
FilesetCatalog.java 100.0% 🟢
GravitinoMetalake.java 92.43% 🟢
GravitinoClient.java 73.26% 🟢
common CatalogCreateRequest.java 76.47% 🟢
SchemaCreateRequest.java 0.0% 🔴
core SchemaNormalizeDispatcher.java 100.0% 🟢
SecretPropertyUtils.java 100.0% 🟢
SchemaEventDispatcher.java 98.21% 🟢
CatalogEventDispatcher.java 97.75% 🟢
SchemaHookDispatcher.java 92.31% 🟢
FilesetOperationDispatcher.java 91.95% 🟢
SecretManager.java 88.68% 🟢
SchemaOperationDispatcher.java 82.7% 🟢
CatalogManager.java 71.57% 🟢
CatalogNormalizeDispatcher.java 42.42% 🔴
CatalogHookDispatcher.java 39.29% 🔴
GravitinoEnv.java 30.81% 🔴
SupportsCatalogs.java 12.5% 🔴
SchemaDispatcher.java 0.0% 🔴
server CatalogOperations.java 100.0% 🟢
SchemaOperations.java 100.0% 🟢

});
});
} catch (RuntimeException e) {
secretManager.rollbackSecrets(secretMaterials);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will this delete a secret that is currently referenced by an existing schema?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Each schema's secret is exclusively owned by that schema and there is no sharing involved, so this issue does not exist.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The concern isn't sharing between schemas but liveness: the catch also covers the code after s.createSchema() has already succeeded (isManagedEntity, getHiddenPropertyNames), so a failure there deletes the secret while the schema still exists in the underlying catalog holding that URN — suggest rolling back only when the schema was not created.

.withCreator(PrincipalUtils.getCurrentPrincipal().getName())
.withCreateTime(Instant.now())
.build())
.build();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Schema properties are not stored in Gravitino's entity store at all; their only source is the underlying catalog. Will they be lost for catalogs that don't support them?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got resolved.

// the cache with stale data between invalidate and delete.
Map<String, String> catalogProperties =
catalogWrapper.catalog().entity().getProperties();
boolean deleted = store.delete(ident, EntityType.CATALOG, true);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cascade deletion does not clean up the secrets of child entities at all.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Before deleting the catalog/schema, first note down the properties (including URNs) of the sub-schemas and filesets. After successfully deleting the entity, call the function "deleteSecretsFromProperties" to remove their write-through secrets.

catalogIdent,
c -> c.doWithSchemaOps(s -> s.loadSchema(ident)),
NoSuchSchemaException.class);
schemaProperties = schema.properties();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If load throws any other exception, the drop operation will fail.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you very much for your review. I have added a "catch(Exception)" block to solve this problem.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

schema.properties() is not read properties from entity store, some catalog‘s properties does not contain URN

lasdf1234 and others added 7 commits August 12, 2026 22:29
…iew comments

- Roll back create-time secrets only when catalog create did not succeed
- Persist schema properties (incl. URNs) on SchemaEntity for unmanaged schemas
- Clean schema/fileset write-through secrets on catalog cascade drop
- Do not fail schema drop when pre-drop catalog load throws unexpected errors

Co-authored-by: Cursor <cursoragent@cursor.com>
Keep schema secret URNs in the entity store when the underlying catalog
does not retain custom properties, so drop-time cleanup can still find them.

Co-authored-by: Cursor <cursoragent@cursor.com>
store.delete(cascade) only soft-deletes meta rows. Snapshot schema and
fileset properties first, drop entities, then delete write-through secrets
so provider material is not left orphaned.

Co-authored-by: Cursor <cursoragent@cursor.com>
…snapshot

Co-authored-by: Cursor <cursoragent@cursor.com>
dropSchema cascade removes filesets without FilesetOperationDispatcher.
Snapshot fileset properties before drop and delete write-through secrets
after the schema drop succeeds.

Co-authored-by: Cursor <cursoragent@cursor.com>
… errors

Secret property capture before drop is best-effort; catch unexpected load
failures so dropSchema can still proceed.

Co-authored-by: Cursor <cursoragent@cursor.com>
Map<String, SecretBinding> secretBindings,
Map<String, SecretReference> secretReferences)
throws NoSuchMetalakeException, CatalogAlreadyExistsException {
throw new UnsupportedOperationException("Not implemented");

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could u use complete error message here? For example. xxxx is not implemented.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got. Use 'Creating xxx with secrets is not supported' instead.(Consistent with the original code of the project)

*/
Schema createSchema(String schemaName, String comment, Map<String, String> properties)
throws NoSuchCatalogException, SchemaAlreadyExistsException;
default Schema createSchema(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got. Use 'Creating xxx with secrets is not supported' instead.(Consistent with the original code of the project)

* @throws SchemaAlreadyExistsException if the schema with specified identifier already exists.
*/
@Override
public Schema createSchema(String schemaName, String comment, Map<String, String> properties)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do u modify the method signature? We should keep backwards compatibility.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for your review. This issue has been fixed.

Map<String, SecretReference> secretReferences)
throws NoSuchMetalakeException, CatalogAlreadyExistsException {
return getMetalake().createCatalog(catalogName, type, provider, comment, properties);
return getMetalake()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for your review. This issue has been fixed.

lasdf1234 and others added 4 commits August 14, 2026 15:22
…ages

Use descriptive UnsupportedOperationException text consistent with other
API defaults, instead of a bare "Not implemented".

Co-authored-by: Cursor <cursoragent@cursor.com>
…n messages

Align FilesetCatalog/Fileset default messages with other API defaults
instead of a bare "Not implemented".

Co-authored-by: Cursor <cursoragent@cursor.com>
…ted message

Co-authored-by: Cursor <cursoragent@cursor.com>
…ventions

Use Fileset/ViewCatalog-style wording for fileset defaults, and keep
create-with-secrets messages consistent across api and core dispatchers.

Co-authored-by: Cursor <cursoragent@cursor.com>
schemaEntity ->
SchemaEntity.builder()
.withId(schemaEntity.id())
.withName(schemaEntity.name())

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

alterSchema will remove the persisted properties.

@lasdf1234 lasdf1234 Aug 14, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you very much for your review. I chage my code and now alter also has properties.

});
});
} catch (RuntimeException e) {
secretManager.rollbackSecrets(secretMaterials);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The concern isn't sharing between schemas but liveness: the catch also covers the code after s.createSchema() has already succeeded (isManagedEntity, getHiddenPropertyNames), so a failure there deletes the secret while the schema still exists in the underlying catalog holding that URN — suggest rolling back only when the schema was not created.

catalogIdent,
c -> c.doWithSchemaOps(s -> s.loadSchema(ident)),
NoSuchSchemaException.class);
schemaProperties = schema.properties();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

schema.properties() is not read properties from entity store, some catalog‘s properties does not contain URN

* Deletes write-through secrets for a schema and its fileset children using properties snapped
* before the entities were dropped. Entity drop must succeed before calling this.
*/
private void deleteSecretsFromPropertySnapshots(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The logic is duplicate with SchemaOperationDispatcher.java snapshotFilesetProperties and deleteSecretsAfterSchemaDrop

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. The concern isn't sharing between schemas but liveness: the catch also covers the code after s.createSchema() has already succeeded (isManagedEntity, getHiddenPropertyNames), so a failure there deletes the secret while the schema still exists in the underlying catalog holding that URN — suggest rolling back only when the schema was not created.

It has been resolved. The solution is similar to that for the catalog.

  1. schema.properties() is not read properties from entity store, some catalog‘s properties does not contain URN
    Read properties from SchemaEntity (entity store) before dropping

  2. The logic is duplicate with SchemaOperationDispatcher.java snapshotFilesetProperties and deleteSecretsAfterSchemaDrop
    This method has been removed.

…ility

Restore the original createSchema/createCatalog/createMultipleLocationFileset
methods on Java clients and add secret-parameter overloads. Old methods
delegate to the new ones with empty secret maps.

Co-authored-by: Cursor <cursoragent@cursor.com>
lasdf1234 and others added 8 commits August 14, 2026 17:10
Keep entity-store properties (including secret URNs) when rebuilding
SchemaEntity after alter, instead of dropping them.

Co-authored-by: Cursor <cursoragent@cursor.com>
…atalog

Update SchemaEntity properties from existing store props plus SchemaChange
only, matching CatalogManager.alterCatalog.

Co-authored-by: Cursor <cursoragent@cursor.com>
Align schema/fileset create with CatalogManager: set needClean=false after
the underlying create succeeds, and roll back write-through secrets only
in finally when create did not succeed.

Co-authored-by: Cursor <cursoragent@cursor.com>
Match CatalogManager: write secrets inside the locked try/finally so
partial create failures share the same needClean rollback path.

Co-authored-by: Cursor <cursoragent@cursor.com>
Align with CatalogManager.dropCatalog: capture write-through URNs from
SchemaEntity in the store instead of catalog loadSchema().properties().

Co-authored-by: Cursor <cursoragent@cursor.com>
…op cleanup

Use getEntity directly in dropSchema instead of a dedicated helper.

Co-authored-by: Cursor <cursoragent@cursor.com>
…ade delete

Let schema/catalog cascade drop list filesets and call FilesetDispatcher
so each fileset cleans its own secrets, instead of snapshotting and
deleting fileset secrets from the parent path.

Co-authored-by: Cursor <cursoragent@cursor.com>
… drop

Wire FilesetDispatcher into SchemaOperationDispatcher via constructor
and CatalogManager via setter, instead of looking it up from GravitinoEnv.

Co-authored-by: Cursor <cursoragent@cursor.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[EPIC] Entity connection secrets via GravitinoSecretProvider

3 participants