[GLUTEN-10134][VL] (STORE_ASSIGNMENT_POLICY defaults to ANSI) Preserve store assignment cast modes by ReemaAlzaid · Pull Request #12051 · apache/gluten

ReemaAlzaid · 2026-05-07T14:23:01Z

What changes are proposed in this pull request?

This PR preserves Spark cast eval mode when Gluten translates casts to Velox, so store assignment casts keep ANSI behavior even when spark.sql.ansi.enabled=false.
#10134

Preserves Spark cast mode in Gluten Substrait.
Adds Velox ANSI/legacy cast handling.
Keeps TRY_CAST as Velox try-cast.
Tests default STORE_ASSIGNMENT_POLICY=ANSI.

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

github-actions · 2026-05-07T14:26:46Z

Run Gluten Clickhouse CI on x86

ReemaAlzaid · 2026-05-12T07:11:35Z

Could you please review this? @rui-mo

rui-mo

Do we have any background to support this policy, and would it make sense to add some tests covering the configurable behaviors under different policy settings?

philo-he · 2026-05-14T06:11:34Z

+      type == BIGINT();
+}
+
+bool isAnsiSupported(const TypePtr& fromType, const TypePtr& toType) {


There's also a check on the Velox side. Will we maintain the ANSI support check only here?

This is intended to mirror the existing Velox ANSI support side check not to replace it, since that check is private today, I added this local helper for the expression level ANSI/legacy cast mode. I’ll add a comment to make sure we keep it aligned with Velox

philo-he · 2026-05-14T06:11:43Z

    }
  }

+  testGluten("storeAssignmentPolicy default ANSI is independent from ANSI mode") {


This doesn't appear to be a variant of a Spark test. Can we move it to a new suite under backends-velox/?

philo-he · 2026-05-14T06:14:07Z

+
+        spark.sql("CREATE TABLE store_assignment_ansi (c INT) USING PARQUET")
+        intercept[Exception] {
+          spark.sql("INSERT INTO store_assignment_ansi SELECT '2147483648'").collect()


Can we add a check for the exception message to confirm the expected exception is thrown?

ReemaAlzaid · 2026-05-14T11:38:12Z

Do we have any background to support this policy, and would it make sense to add some tests covering the configurable behaviors under different policy settings?

My understanding is that Spark applies spark.sql.storeAssignmentPolicy during analysis, and Gluten should just preserve the cast mode Spark selected when translating the resulting casts. Otherwise INSERT/store assignment behavior may differ from Spark. I’ll add tests for the configurable policy cases.

github-actions · 2026-05-14T12:06:37Z

Run Gluten Clickhouse CI on x86

github-actions · 2026-05-14T16:19:28Z

Run Gluten Clickhouse CI on x86

github-actions · 2026-05-27T23:32:27Z

Run Gluten Clickhouse CI on x86

philo-he

Thanks for the PR. Two comments, possibly due to missing context on my side.

philo-he · 2026-05-29T22:14:43Z

+      std::make_shared<SparkCastHooks>(config, allowOverflow));
+}
+
+class SparkAnsiCastCallToSpecialForm : public exec::CastCallToSpecialForm {


I am still wondering if we can move or keep these code on Velox side, and just call the register API in Gluten C++ code. Could you please clarify?

Yes. Existing Velox Spark cast depends on session ansiEnabled, but Gluten needs Substrait’s per expression cast mode. This patch adds thin Gluten registration while reusing Velox cast code. I agree we can move it to Velox and call the registration API from Gluten

philo-he · 2026-05-29T22:17:02Z

+// Keep this in sync with Velox's SparkCastCallToSpecialForm::isAnsiSupported.
+// Velox's helper is private today; this local copy is needed for expression-level
+// ANSI and legacy cast modes.
+bool isAnsiSupported(const TypePtr& fromType, const TypePtr& toType) {


If we really need it, to reduce code maintenance effort, could we make it public in Velox to allow calling it here?

Created the pr here facebookincubator/velox#17687

github-actions · 2026-06-01T12:28:59Z

Run Gluten Clickhouse CI on x86

…om/ReemaAlzaid/incubator-gluten into store-assignment-policy-cast-modes

github-actions · 2026-06-08T11:53:58Z

Run Gluten Clickhouse CI on x86

github-actions · 2026-06-08T12:01:53Z

Run Gluten Clickhouse CI on x86

philo-he · 2026-06-11T05:11:53Z

@ReemaAlzaid, thanks for the update. Could you check if the CI failures are related?

ReemaAlzaid · 2026-07-07T19:43:01Z

The pr got merged facebookincubator/velox#17687 @philo-he

philo-he · 2026-07-08T00:00:36Z

The pr got merged facebookincubator/velox#17687 @philo-he

@ReemaAlzaid, thanks for the update. I noticed the CI failed. Is it because Gluten's referenced Velox branch doesn't yet include that Velox commit? If so, you could reference the PR ID in the following code so Gluten picks up that Velox patch for the CI tests.

gluten/ep/build-velox/src/get-velox.sh

Line 28 in cd99450

UPSTREAM_VELOX_PR_ID=""

github-actions · 2026-07-08T22:20:54Z

Run Gluten Clickhouse CI on x86

…om/ReemaAlzaid/incubator-gluten into store-assignment-policy-cast-modes

github-actions · 2026-07-08T22:33:15Z

Run Gluten Clickhouse CI on x86

ReemaAlzaid · 2026-07-08T22:39:04Z

Hey @philo-he, the velox commit is added now and has been merged I just forgot to update the ansi config key class here Fix Spark ANSI config key in cast tests. So the CI should run fine now

philo-he · 2026-07-08T23:57:42Z

@ReemaAlzaid, thanks for the update. It seems that CI failure is related. Could you check further? Thank you!

- storeAssignmentPolicy preserves configured cast modes *** FAILED ***
  executedPlan.exists(((plan: org.apache.spark.sql.execution.SparkPlan) => tag.runtimeClass.isInstance(plan))) was false Expect ProjectExecTransformer exists in executedPlan:
   CommandResult <empty>
     +- Execute InsertIntoHadoopFsRelationCommand file:/__w/gluten/gluten/backends-velox/spark-warehouse/org.apache.gluten.execution.VeloxInsertSuite/store_assignment_legacy, false, Parquet, [path=file:/__w/gluten/gluten/backends-velox/spark-warehouse/org.apache.gluten.execution.VeloxInsertSuite/store_assignment_legacy], Append, `spark_catalog`.`default`.`store_assignment_legacy`, org.apache.spark.sql.execution.datasources.InMemoryFileIndex(file:/__w/gluten/gluten/backends-velox/spark-warehouse/org.apache.gluten.execution.VeloxInsertSuite/store_assignment_legacy), [c]
        +- WriteFiles
           +- *(1) Project [cast(c#242653 as int) AS c#242655]
              +- *(1) ColumnarToRow
                 +- FileScan parquet spark_catalog.default.store_assignment_legacy_src[c#242653] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/__w/gluten/gluten/backends-velox/spark-warehouse/org.apache.glut..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<c:string> (GlutenQueryTest.scala:427)

github-actions · 2026-07-16T19:27:23Z

Run Gluten Clickhouse CI on x86

github-actions · 2026-08-12T10:43:35Z

Run Gluten Clickhouse CI on x86

github-actions · 2026-08-12T12:31:18Z

Run Gluten Clickhouse CI on x86

github-actions · 2026-08-12T15:48:35Z

Run Gluten Clickhouse CI on x86

github-actions · 2026-08-12T17:01:41Z

Run Gluten Clickhouse CI on x86

github-actions · 2026-08-12T17:34:54Z

Run Gluten Clickhouse CI on x86

github-actions · 2026-08-13T17:09:33Z

Run Gluten Clickhouse CI on x86

philo-he · 2026-08-13T18:46:26Z

@ReemaAlzaid, thanks for the iterating. Could you check if the clickhouse backend CI failure is related? You can use this account to login:

gluten/docs/get-started/ClickHouse.md

Line 651 in 82da039

public read-only account：gluten/hN2xX3uQ4m

If not related, you can re-trigger that CI by posting a comment to this PR page: Run Gluten Clickhouse CI on x86.

philo-he · 2026-08-13T18:39:56Z

+      toType: String)(f: => Unit): Unit = {
+    val exception = intercept[AnalysisException](f)
+    // Older Spark versions report the types in lower case, e.g. "string to int",
+    // while newer ones quote them in upper case, e.g. "STRING" to "INT".


Nit:
Could you leave some details about since which Spark versions upper case is used? Then, we can remove the case conversion when those old Spark versions are deprecated.

You may clarify like this "Since Spark xxx, the types in the exception message are in upper case, e.g. ...."

Nit: Could you leave some details about since which Spark versions upper case is used? Then, we can remove the case conversion when those old Spark versions are deprecated.

You may clarify like this "Since Spark xxx, the types in the exception message are in upper case, e.g. ...."

Sure I have added the comment thanks

github-actions · 2026-08-13T18:50:19Z

Run Gluten Clickhouse CI on x86

github-actions · 2026-08-13T18:54:16Z

Run Gluten Clickhouse CI on x86

ReemaAlzaid · 2026-08-13T19:02:27Z

Hey @philo-he thanks for the review! I have addressed the comments and reran the CI

Preserve store assignment cast modes

10414a2

ReemaAlzaid mentioned this pull request May 7, 2026

[VL] Add ANSI mode support #10134

Open

53 tasks

github-actions Bot added CORE works for Gluten Core VELOX labels May 7, 2026

rui-mo reviewed May 12, 2026

View reviewed changes

philo-he reviewed May 14, 2026

View reviewed changes

Address Velox insert review comments

44d0731

ReemaAlzaid requested a review from philo-he May 14, 2026 12:15

Add store assignment policy insert tests

6e4b609

ReemaAlzaid requested a review from rui-mo May 14, 2026 16:20

ReemaAlzaid and others added 4 commits May 17, 2026 14:51

Preserve store assignment cast modes

0b6c65a

Address Velox insert review comments

222f1f1

Add store assignment policy insert tests

a998015

Merge branch 'main' into store-assignment-policy-cast-modes

eea937a

philo-he reviewed May 29, 2026

View reviewed changes

Merge branch 'apache:main' into store-assignment-policy-cast-modes

b2f41b5

ReemaAlzaid mentioned this pull request Jun 1, 2026

feat: Add Spark cast mode special form registration facebookincubator/velox#17687

Closed

ReemaAlzaid added 2 commits June 8, 2026 14:16

Merge branch 'store-assignment-policy-cast-modes' of https://github.c…

89f3284

…om/ReemaAlzaid/incubator-gluten into store-assignment-policy-cast-modes

Use upstream Velox cast mode special forms

8e1a1cb

ReemaAlzaid requested a review from philo-he June 9, 2026 12:03

Merge branch 'main' into store-assignment-policy-cast-modes

469c319

ReemaAlzaid added 2 commits July 9, 2026 00:32

Fix Spark ANSI config key in cast tests

570e3c2

Merge branch 'store-assignment-policy-cast-modes' of https://github.c…

a21cffc

…om/ReemaAlzaid/incubator-gluten into store-assignment-policy-cast-modes

Merge branch 'main' into store-assignment-policy-cast-modes

06856ba

Merge branch 'main' into store-assignment-policy-cast-modes

7dc2bef

ReemaAlzaid force-pushed the store-assignment-policy-cast-modes branch from 10c7de5 to 7dc2bef Compare August 12, 2026 15:48

Merge branch 'main' into store-assignment-policy-cast-modes

abb474c

[GLUTEN-10134][VL] Fix CI failures for store assignment cast modes

83d88df

Merge branch 'main' into store-assignment-policy-cast-modes

d69e751

philo-he reviewed Aug 13, 2026

View reviewed changes

Merge branch 'main' into store-assignment-policy-cast-modes

71c3610

address comments

dac6f8f

ReemaAlzaid requested a review from philo-he August 13, 2026 19:44

Uh oh!

Conversation

ReemaAlzaid commented May 7, 2026

What changes are proposed in this pull request?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

github-actions Bot commented May 7, 2026

Uh oh!

ReemaAlzaid commented May 12, 2026

Uh oh!

rui-mo left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ReemaAlzaid commented May 14, 2026

Uh oh!

github-actions Bot commented May 14, 2026

Uh oh!

github-actions Bot commented May 14, 2026

Uh oh!

github-actions Bot commented May 27, 2026

Uh oh!

philo-he left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jun 1, 2026

Uh oh!

github-actions Bot commented Jun 8, 2026

Uh oh!

github-actions Bot commented Jun 8, 2026

Uh oh!

philo-he commented Jun 11, 2026

Uh oh!

ReemaAlzaid commented Jul 7, 2026

Uh oh!

philo-he commented Jul 8, 2026

Uh oh!

github-actions Bot commented Jul 8, 2026

Uh oh!

github-actions Bot commented Jul 8, 2026

Uh oh!

ReemaAlzaid commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

philo-he commented Jul 8, 2026

Uh oh!

github-actions Bot commented Jul 16, 2026

Uh oh!

github-actions Bot commented Aug 12, 2026

Uh oh!

github-actions Bot commented Aug 12, 2026

Uh oh!

github-actions Bot commented Aug 12, 2026

Uh oh!

github-actions Bot commented Aug 12, 2026

Uh oh!

github-actions Bot commented Aug 12, 2026

Uh oh!

github-actions Bot commented Aug 13, 2026

Uh oh!

philo-he commented Aug 13, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ReemaAlzaid commented Jul 8, 2026 •

edited

Loading