[VL] Iceberg REST-catalog vended credentials (Polaris etc.) never reach the native S3 reader — native scans 403 while vanilla Spark reads fine

[guangyu-yang-rokt]

Backend

VL (Velox)

Bug description

When a Spark job reads an Apache Iceberg table through a REST catalog that uses credential vending (X-Iceberg-Access-Delegation: vended-credentials, e.g. Apache Polaris), Gluten's native scan path cannot read the table's data files: every native TableScan fails with S3 403 while the same query on vanilla Spark succeeds.

VeloxRuntimeError ... S3ReadFile.cpp:163 preadInternal: Failed to get S3 object due to: 'Access denied'

The failure is deterministic (task retries exhaust → job abort) whenever the compute's own IAM identity has no direct grant on the warehouse bucket — which is the point of credential vending: the catalog returns per-table, short-TTL, prefix-scoped STS credentials on loadTable, and only the JVM S3FileIO ever sees them.

Mechanism

• JVM path (works): Iceberg REST loadTable puts s3.access-key-id / s3.secret-access-key / s3.session-token on the table's FileIO properties; S3FileIO reads data files with them.
• Native path (fails): GlutenIcebergSourceUtil.genSplitInfo extracts only file paths/starts/lengths/deletes into the LocalFiles split; Velox's S3 filesystem builds its client from the global hive.s3.* config + the AWS default credential chain. The vended credentials never leave the JVM. There is no per-table (or even session-token) credential channel: velox S3Config has no session-token key at all, so an STS triple is not even expressible statically.

Reproduced on Gluten v1.6.0 (Spark 3.5.2, iceberg-spark-runtime 1.6.0, Apache Polaris with STS vending). Code on main has the same gap: the iceberg module still reads no FileIO properties, and the fs.s3a.* → hive.s3.* static mapping has no session-token entry.

Related but distinct: #10113 / #8170 (session-level / custom credential providers not reaching native) are session-global; vended credentials are per-table, so two tables in one bucket can carry different credentials in one query — a bucket- or session-scoped channel cannot express this.

Prior art: apache/datafusion-comet hit exactly this and fixed it in two stages — static extraction of the vended credentials from the table's FileIO properties into the native scan payload (comet#3523), then a pluggable per-path credential provider for refresh (comet#4309).

How we fixed it (patch set available, happy to upstream)

We run Gluten v1.6.0 with three bundle patches that make native scans of vended-credential tables work end-to-end, validated against Apache Polaris:

1. JVM extract + carry: at split planning, read the scan table's FileIO.properties(); when a vended s3.access-key-id/s3.secret-access-key pair is present, attach it (+ session token/expiry/endpoint/region) with the table location to a new table-scoped ReadRel.LocalFiles.read_properties map (one LocalFiles is single-table by construction). Config-gated with a kill switch whose off branch fails native validation → clean vanilla fallback instead of 403s.
2. Native install: parse the map onto gluten's SplitInfo; union all scans' entries into a filesystems::TokenProvider implementation (longest-prefix match of file path against normalized table locations → per-table resolution, multi-table-same-bucket safe) passed to QueryCtx::create. The velox line Gluten pins (IBM/velox dft-* branches) already threads fsTokenProvider() through FileHandleKey (credential identity keys the handle cache) into openFileForRead for both data and positional-delete files.
3. Velox S3 consumption: S3FileSystem::openFileForRead/Write consult FileOptions::tokenProvider (new S3AccessTokenKey/S3AccessToken types) and serve the open from a per-credential-fingerprint client cache; S3Config gains hive.s3.session-token (per-bucket capable) so static STS credentials are expressible too.

Known limitation (same as comet#3523's explicit non-goal): credentials are snapshotted at planning; refresh-on-open is a follow-up.

If maintainers are interested I can open PRs for the gluten side (and the velox piece to the pinned velox line). Adoption of REST-catalog credential vending is growing quickly (Polaris, Unity, Glue IRC all vend), so I'd expect this class to hit more users as use_gluten-style rollouts meet lakehouse catalogs.

Spark version

Spark-3.5.x

Spark configurations

spark.plugins=org.apache.gluten.GlutenPlugin
spark.sql.catalog.=org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog..type=rest
spark.sql.catalog..header.X-Iceberg-Access-Delegation=vended-credentials

System information

Gluten v1.6.0 (velox backend, IBM/velox dft-2026_02_06), Spark 3.5.2, JDK 17, iceberg-spark-runtime-3.5_2.12:1.6.0, aws-sdk-cpp 1.11.655, arm64 + x86_64

Relevant logs

VeloxRuntimeError: ... S3ReadFile.cpp:163 preadInternal: Failed to get S3 object due to: 'Access denied'
(HTTP 403 GetObject on the REST-catalog warehouse bucket; IAM: the pod role, not the vended credentials)

[zhouyuan]

@guangyu-yang-rokt Thanks for bringing this up! I believe the REST catalog feature hasn't been supported or verified on Gluten yet. Would you be open to working on a patch for this?