[SPARK-59014][SQL] Reject a data column that hides a captured metadata column during DSv2 refresh validation

[yyanyy]

What changes were proposed in this pull request?

V2TableUtil.validateCapturedMetadataColumns now rejects the case where a data column has taken the name of a captured metadata column, instead of letting the conflict pass silently.

DSv2 relations capture their metadata-column attributes at analysis time and re-validate them when the table is refreshed / re-resolved. The existing check compared them only against the metadata columns the connector still reports, so a conflict arriving on the data side was invisible.

Connectors that rename conflicting metadata columns (SupportsMetadataColumns.canRenameConflictingMetadataColumns() is true) keep the column reachable and are unaffected. When the conflict is suppressed instead (the default), LogicalPlan.metadataOutputWithOutConflicts drops the metadata column, and on a partially-pruned scan PushDownUtils.toOutputAttrs resolves the read schema by name and collapses the two same-named fields onto one attribute — so a query for the metadata column silently returns the data column's values.

All three callers go through the new check, but only the two that admit new data columns can fire it: refresh (ALLOW_NEW_FIELDS) and the dataframe temp-view path (ALLOW_NEW_TOP_LEVEL_FIELDS). Under PROHIBIT_CHANGES, V2TableReference.validateNoChanges already throws on the added data column first, so the transactional-write path is unchanged. Only metadata columns the table still reports are considered — one the connector dropped is already flagged as removed — and the check is skipped when none remain.

Related to #58298 (SPARK-59015, DSv2 refresh schema rebind), which raises an internal-error assertion for this same case that this PR's user-facing rejection makes unreachable. Either can merge first, but both now touch InMemoryBaseTable.scala and DataSourceV2DataFrameSuite.scala, so the later one should rebase and re-run both suites.

Why are the changes needed?

Returning the wrong column's values is far worse than failing, and the SupportsMetadataColumns contract already recommends that non-renaming sources reject data columns that collide with metadata columns.

Does this PR introduce any user-facing change?

Yes. A relation that referenced a metadata column and is later re-resolved against a table where a data column took that name now fails with "<name> metadata column is hidden by a data column with the same name", instead of returning wrong results or working by luck depending on pruning. The condition is INCOMPATIBLE_TABLE_CHANGE_AFTER_ANALYSIS.METADATA_COLUMNS_MISMATCH on the refresh path and INCOMPATIBLE_COLUMN_CHANGES_AFTER_VIEW_WITH_PLAN_CREATION (colType = metadata) for a dataframe temp view. Connectors that rename are unaffected; this tightens a validation gap on unreleased master.

How was this patch tested?

V2TableUtilSuite covers the suppressed conflict under both validation modes, case sensitivity in both settings, a renamable connector, and a metadata column the connector stopped reporting (removed, not hidden).

DataSourceV2DataFrameSuite adds an end-to-end test where a newly added data column shadows a captured metadata column and the query now fails through the real refresh path. Reaching the suppressed branch needs a connector that does not rename, so InMemoryBaseTable gains a rename-conflicting-metadata-columns property defaulting to true.

catalyst/testOnly *V2TableUtilSuite 54 tests, sql/testOnly *DataSourceV2DataFrameSuite 204 tests, 0 failures.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code 2.1.246