[VL] Add ANSI mode support

[philo-he]

Description

In this issue, we will list and track all tasks for ANSI mode support.

There are two Spark configurations directly related to ANSI usage.

• spark.sql.ansi.enabled (default is true since Spark 4.0)
• spark.sql.storeAssignmentPolicy (default is ANSI since Spark 3.0)

Please note this task list may not be complete; suggestions for additional tasks are welcome. Checked tasks have been claimed and may already have an associated PR or be finished. You are welcome to take on the unchecked tasks, but before doing so, please verify against the Velox codebase whether a relevant PR has already been created or merged.

Basic

• [VL] Set ANSI mode for Velox according to Spark's configuration #10387

1. Type Casting Functions (ANSI Strict)

• cast string to boolean (FINISHED)
  Cast.scala#L701

• Cast decimal to string (FINISHED): test: Validate ANSI support for Spark CAST(decimal as string) facebookincubator/velox#16124
  Follow-up PR: fix: Use plain string for Spark CAST(decimal as string) when ANSI is enabled facebookincubator/velox#17647
  // In ANSI mode, Spark always use plain string representation on casting Decimal values
  // as strings. Otherwise, the casting is using BigDecimal.toString which may use scientific
  // notation if an exponent is needed.
  Cast.scala#L678

• cast string to timestamp (@infvg) [FINISHED]: feat: Make Spark CAST(string as timestamp) ANSI-compliant facebookincubator/velox#17102
  Cast.scala#L733

• cast String to timestampNTZ
  Cast.scala#L775

• cast float/double to timestamp (@infvg): feat: Support ANSI for Spark CAST(float/double as timestamp) facebookincubator/velox#17219
  Cast.scala#L758
  Cast.scala#L765

• cast string to date (FINISHED): test: Validate Spark string-to-date cast facebookincubator/velox#16092
  Cast.scala#L811

• cast string to time (@malinjawi): feat: Add Spark CAST(string as TIME) with ANSI support facebookincubator/velox#16123
  Cast.scala#L826
  The implementation for codegen, assume equivalent with the above link:
  Cast.scala#L1493

• cast string to long/int/short/byte (FINISHED)
  As one example, here is the related code for long type:
  Cast.scala#L883

• cast NumericType to long/int/short/byte (@minni31): feat(spark): Add ANSI mode support for CAST(NumericType as integral) facebookincubator/velox#16962
  As one example, here is the related code for long type:
  Cast.scala#L896

• cast timestamp to int/short/byte (FINISHED): fix(spark): Align timestamp to integral cast with Spark overflow semantics under ANSI facebookincubator/velox#17977
  As one example, here is the related code for int type:
  Cast.scala#L933

• cast time to short/byte (requires TimeType support)
  As one example, here is the related code for short type:
  Cast.scala#L980

• cast string to double/float: feat(spark): Support ANSI-mode cast from string to REAL and DOUBLE facebookincubator/velox#17285
  ANSI controls the behavior in handling incorrect number format
  As one example, here is the related code for double type:
  Cast.scala#L1159

cast several types to decimal
ANSI controls the overflow behavior in changePrecision
Cast.scala#L1103

• cast VARCHAR to decimal [FINISHED]: fix: Enable ANSI-compliant cast from VARCHAR to DECIMAL facebookincubator/velox#18019
• cast INTEGRAL to decimal [FINISHED]: fix(sparksql): Enable ANSI-compliant cast from INTEGRAL to DECIMAL facebookincubator/velox#18107
• fix: Make Spark CAST(floating-point as decimal)[FINISHED]: ANSI-compliant: fix: Make Spark CAST(floating-point as decimal) ANSI-compliant facebookincubator/velox#18206

2. Math Functions (ANSI Overflow Check)

• A base type: AnsiIntervalType (@malinjawi): feat(function): Add Spark ANSI interval unary minus, add, and subtract facebookincubator/velox#17098
  AbstractDataType.scala#L168
  Unary expressions like Abs, UnaryMinus
  The ANSI config controls failOnError.
  arithmetic.scala#L152C35-L152C46
• Abs
• UnaryMinus [FINISHED]:
  feat: Add ANSI mode support for Spark arithmetic operations facebookincubator/velox#16361
  feat: Add ANSI mode support for unaryminus overflow detection facebookincubator/velox#16672
  feat(sparksql): Make unaryminus ANSI-compliant for integer overflow detection facebookincubator/velox#18096
  Binary arithmetic expressions using BinaryArithmetic as base, such as add, divide, multiply, remainder, pmod, etc., for all supported types (@malinjawi)
  arithmetic.scala#L209
• add/subtract (decimal type): feat: Support decimal type for the Spark checked_add and checked_subtract functions facebookincubator/velox#16302
• divide (decimal type): feat: Add checked_divide for Spark decimal arithmetic facebookincubator/velox#16323
• multiply (decimal type)(FINISHED): feat: Support decimal type for Spark checked_multiply function facebookincubator/velox#16307
• add/subtract/divide/multiply (other types): feat: Add ANSI mode support for Spark arithmetic operations facebookincubator/velox#16361
• remainder (integral types and floating point types): Add checked_remainder for integer and floating-point types facebookincubator/velox#16403
• pmod (integral types and floating point types): Add checked_pmod for integer and floating-point types facebookincubator/velox#16405
• pmod (decimal type): Add decimal pmod and checked_pmod for Spark arithmetic facebookincubator/velox#16406
  round functions
  As one example, see how to round to ByteType with ANSI enabled:
  mathExpressions.scala#L1579
• Round: feat: Add ANSI overflow check to Spark round function facebookincubator/velox#16479
• BRound
• RoundCeil
• RoundFloor

3. Date/Time Functions (ANSI Validation)

• ToUnixTimestamp
• UnixTimestamp
• GetTimestamp
• TryToTimestampExpressionBuilder
• NextDay
• DateAddInterval
• ParseToDate
• TryToDateExpressionBuilder
• ParseToTimestamp
• MakeDate
• TryMakeTimestampLTZExpressionBuilder
• MakeTimestamp: fix(sparksql): Make make_timestamp ANSI-aware facebookincubator/velox#17745

4. Misc

• String expressions: Elt
  stringExpressions.scala#L286
• Collection expression Size
  Its legacySizeOfNull is impacted by ANSI config.
  collectionOperations.scala#L118
• Collection expression ElementAt
  collectionOperations.scala#L2622
• conv mathExpressions.scala#L451
• STORE_ASSIGNMENT_POLICY defaults to ANSI: [GLUTEN-10134][VL] Preserve store assignment cast modes (STORE_ASSIGNMENT_POLICY defaults to ANSI) #12051
  SQLConf.scala#L4487

5. Aggregation Functions (ANSI Overflow)

SUM, AVG, VAR_POP, VAR_SAMP, STDDEV_POP, STDDEV_SAMP

• In ANSI mode: overflow checks during accumulation.
• SUM: feat: Add checked_sum aggregate for Spark ANSI mode overflow detection facebookincubator/velox#16435
• AVG: feat: Add checked_avg aggregate for Spark ANSI mode decimal overflow facebookincubator/velox#16439
• VAR_POP
• VAR_SAMP
• STDDEV_POP
• STDDEV_SAMP

TRY_SUM (Spark 3.4+)

• Returns NULL on overflow instead of error.

6. Window Functions (ANSI Overflow)

Same overflow checks apply in window operations:

• SUM(...) OVER(...)
• AVG(...) OVER(...)

7. ANSI SQL Compliant String Functions

SUBSTRING / SUBSTR

• ANSI SQL standard argument order: SUBSTRING(str FROM start [FOR len])
• Also supports classic form: SUBSTRING(str, start, len)

TRIM

• ANSI syntax: TRIM(LEADING '0' FROM col) Also TRIM(BOTH ...), TRIM(TRAILING ...)

OVERLAY

• ANSI SQL string replacement: OVERLAY(string PLACING replacement FROM start [FOR length])

---

See Spark ANSI compliance: https://github.com/apache/spark/blob/v4.0.0/docs/sql-ref-ansi-compliance.md
Related discussion: #4740.
facebookincubator/velox#3869

[FelixYBW]

@philo-he are you going to work on this?

[philo-he]

@PHILO-HE are you going to work on this?

@FelixYBW, some developers in the community are interested in this work. I will work with them. I assume it requires a lot of small changes distributed in multiple files in Velox.

[FelixYBW]

Yes, several customer required it. I added to project track. Could you update the status weekly or bi-weekly?

We may need to enable the ANSI UT firstly and list the failure cases. Also create the sub-issue for work to do. I think some of work is already done in velox.

[nimesh1601]

@philo-he We at Uber also require this support, so feel free to include me too

[philo-he]

Yes, several customer required it. I added to project track. Could you update the status weekly or bi-weekly?

We may need to enable the ANSI UT firstly and list the failure cases. Also create the sub-issue for work to do. I think some of work is already done in velox.

@FelixYBW, I will update the status bi-weekly and will also create some subtasks. Yes, there are some relevant work already done.

[malinjawi]

Hey @philo-he @FelixYBW I will be picking up cast string to boolean. Looking forward to making my first contribution!