[#12337] feat(web): Support ClickHouse data-skipping indexes and engine-aware sort orders validation

[LauraXia123]

What changes were proposed in this pull request?

This PR enhances the CreateTableDialog for ClickHouse table creation with two features:

1. Support ClickHouse data-skipping index types with properties

• Add Granularity column in the Indexes tab, shown for data-skipping index types (data_skipping_minmax, data_skipping_bloom_filter, data_skipping_set), hidden (-) for primary_key
• Add Set Max Values column, shown only for data_skipping_set index type
• Populate granularity and set_max_values when editing an existing table
• Include index properties object in the submit payload for data-skipping indexes
• Clear granularity/setMaxValues fields when index type changes
• Add 4 required properties (cluster-name, cluster-remote-database, cluster-remote-table, cluster-sharding-key) for ClickHouse Distributed engine, conditionally shown only when engine is Distributed with required validation.

2. Engine-aware sort orders validation

• Add clickHouseEngine and isClickHouseMergeTreeEngine computed variables to detect the current engine type in real-time
• Show required indicator (*) on Sort Orders tab only for MergeTree family engines
• Move sort orders validation from Form.Item rules (which can cache stale closure values) to handleSubmit where form.getFieldValue('engine') reads the current engine value
• Add useEffect on isClickHouseMergeTreeEngine to:
  • Clear sort orders data and validation errors when switching to non-MergeTree engines
  • Show required validation error when switching to MergeTree engines with empty sort orders
• Remove validateTrigger='onSubmit' and rules from sortOrders Form.Item to avoid Ant Design Form caching stale validator closures

Why are the changes needed?

1. ClickHouse data-skipping indexes require granularity and set_max_values properties to be configured. Without these fields in the UI, users cannot properly create data-skipping indexes through the Gravitino web interface.

2. Sort orders are only required for MergeTree family engines. For other ClickHouse engines (Distributed, TinyLog, Log, StripeLog), sort orders are optional. The previous implementation always validated sort orders as required regardless of engine type, causing form submission to fail for non-MergeTree engines when sort orders were empty.

Fix: #12337

Does this PR introduce any user-facing change?

Yes:

1. Indexes tab: Two new columns (Granularity, Set Max Values) are added for ClickHouse tables. These columns show - for non-data-skipping index types.
2. Sort Orders tab: The required indicator (*) now dynamically appears/disappears based on the selected ClickHouse engine type. Sort orders are only required for MergeTree family engines.
3. Engine switch: Switching from a MergeTree engine to a non-MergeTree engine now clears sort orders data and validation errors automatically.

How was this patch tested?

1. Create a ClickHouse table with MergeTree engine - verify Sort Orders tab shows * and validation requires at least one sort order
2. Switch engine from MergeTree to TinyLog/Log/StripeLog/Distributed - verify * disappears, sort orders data is cleared, and form can be submitted without sort orders
3. Switch engine back to MergeTree - verify * reappears and validation error shows if sort orders are empty
4. Create a table with data_skipping_minmax index - verify Granularity field is shown, Set Max Values shows -
5. Create a table with data_skipping_set index - verify both Granularity and Set Max Values fields are shown
6. Create a table with primary_key index - verify both Granularity and Set Max Values show -
7. Edit an existing table with data-skipping indexes - verify granularity and set_max_values are populated correctly
8. Submit a table with data-skipping index properties - verify properties are included in the API payload

[Copilot]

Pull request overview

Note

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Enhances the ClickHouse table creation/editing UX by adding support for data-skipping index properties and making Sort Orders validation depend on the selected ClickHouse engine.

Changes:

• Add ClickHouse data-skipping index types and surface granularity / set_max_values in UI and submit payloads.
• Make Sort Orders required only for MergeTree-family engines, with dynamic required indicator and submit-time validation.
• Extend table details views/store mapping to include index properties.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
web-v2/web/src/lib/store/metalakes/index.js	Includes index properties in table details mapping for edit/view.
web-v2/web/src/config/index.js	Adds ClickHouse defaults/immutables and defines MergeTree engine list + index types.
web-v2/web/src/config/catalog.js	Adds ClickHouse engine default property options for table creation.
web-v2/web/src/app/catalogs/rightContent/entitiesContent/TableDetailsPage.js	Renders index granularity and set_max_values columns in table details.
web-v2/web/src/app/catalogs/rightContent/CreateTableDialog.js	Adds index-property fields, payload building, and engine-aware Sort Orders validation/UX.

Suppressed comments (1)

web-v2/web/src/app/catalogs/rightContent/CreateTableDialog.js:19

• Index type comparisons and payload building rely on startsWith('data_skipping_'), but when editing an existing table you set indexType using capitalizeFirstLetter(item.indexType). If that transforms data_skipping_* into a different casing, the UI will fail to show Granularity/Set Max Values and the submit payload will omit properties (and may send an unexpected indexType value). Store/compare indexType in a canonical format (e.g., keep the backend value as-is / lowercase) and only transform for display/labels if needed; alternatively, normalize comparisons with .toLowerCase().

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated no new comments.

Suppressed comments (2)

web-v2/web/src/config/catalog.js:231

• The ClickHouse engine list is duplicated here and in web-v2/web/src/config/index.js (clickHouseMergeTreeEngines). Keeping two separate lists in sync is error-prone and can cause UI/validation drift (e.g., an engine selectable here but not recognized as MergeTree elsewhere). Consider reusing the shared constant (or moving the list to a single module) so both the Select options and validation logic stay aligned.

    }
  ],
  'jdbc-clickhouse': [
    {
      key: 'engine',

web-v2/web/src/app/catalogs/rightContent/CreateTableDialog.js:134

• isClickHouseMergeTreeEngine uses a case-sensitive includes check, but the engine value is read from form values/properties and may not have consistent casing (e.g., existing tables or user-provided properties). This can cause the required indicator and engine-aware validation to be skipped for MergeTree engines. Consider normalizing the engine string (e.g., lowercase) before comparison (and apply the same normalization in handleSubmit's isCurrentMergeTree check).

    provider === 'jdbc-clickhouse'
      ? (values?.engine || values?.properties?.find(item => item?.key === 'engine')?.value)?.toLowerCase()

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.

Suppressed comments (3)

web-v2/web/src/app/catalogs/rightContent/CreateTableDialog.js:1

• For ClickHouse, indexType is being uppercased at submit time. The UI options (indexesInfoMap) and edit path use lowercase values (e.g., data_skipping_set), so uppercasing risks sending an unexpected value to the API and/or breaking round-tripping when editing. Unless the backend explicitly requires uppercase enum values, keep the same casing as the selected value (or normalize consistently both on load and submit).

/*

web-v2/web/src/app/catalogs/rightContent/CreateTableDialog.js:1652

• The PR description says sort orders validation was moved out of Form.Item rules and validateTrigger='onSubmit' was removed to avoid Ant Design caching stale closures. However, this diff still adds validateTrigger='onSubmit' and an inline validator that closes over isSortOrdersRequired/isMergeTreeEngine. To align with the described fix (and prevent stale engine-dependent validation), move this engine-aware validation into submit-time logic (or a validator that reads current values inside the function) and remove validateTrigger/rules here.

                  <Form.Item
                    className={tabKey !== 'sortOrders' ? 'hidden' : ''}
                    label=''
                    name='sortOrders'
                    validateTrigger='onSubmit'
                    rules={[
                      {
                        validator: (_, val) => {
                          if (isSortOrdersRequired) {
                            if (Array.isArray(val) && val.length > 0) {
                              if (isMergeTreeEngine) {
                                const columns = form.getFieldValue('columns') || []

                                const nullableSortFields = val
                                  ?.filter(s => {
                                    const col = columns.find(c => c?.name === s?.fieldName)

                                    return col && !col.required
                                  })
                                  ?.map(s => s?.fieldName)
                                if (nullableSortFields?.length > 0) {
                                  return Promise.reject(
                                    new Error(
                                      `Nullable columns cannot be used in ORDER BY for MergeTree engines: ${nullableSortFields.join(', ')}`
                                    )
                                  )
                                }
                              }

                              return Promise.resolve()
                            }

                            return Promise.reject(new Error('Sort orders are required for MergeTree family engines'))
                          }

                          return Promise.resolve()
                        }
                      }
                    ]}
                  >

web-v2/web/src/app/catalogs/rightContent/CreateTableDialog.js:195

• This hide handling returns false (hides the prop) when the parentField value is unset. That makes properties disappear by default whenever hide is configured but the parent value isn't present yet, which is usually the opposite of what's desired (typically: show until the parent matches a hidden value). Consider returning true when parentValue is missing, and only hiding when parentValue is present and matches a hidden option.

      // If the prop has a "hide" list, hide it when the parentField value matches
      if (prop.hide?.length) {
        const parentValue = values?.[prop.parentField]?.toLowerCase()
        if (!parentValue) {
          return false
        }

        return !prop.hide.map(item => item.toLowerCase()).includes(parentValue)
      }