[SPARK-58581][PS][FOLLOWUP] Fix NumPy floor_divide rounding and integer precision

[Spenserrrr]

What changes were proposed in this pull request?

The native floor_divide mapping computes floor(c1 / c2). That gives a wrong answer in two cases, both fixed here.

• Floating operands. 1.0 // 0.1 returned 10, where pandas, NumPy and Python all give 9. The exact quotient is a little under 10, and the division rounds it up before the floor ever runs. NumPy works the quotient out from the remainder instead, which is exact in IEEE arithmetic, so the floating branch now does the same, including NumPy's sign correction and snap-to-nearest.
• Integral operands. They were cast to double before dividing, which drops the low bits above 2^53, so -9007199254740993 // 2 came back as -4503599627370496 instead of -4503599627370497. They now divide in integer space, and only the result is cast.

A second commit handles -2**63 // -1, the one quotient a long cannot hold. Spark's integer division raises ARITHMETIC_OVERFLOW there, so that pair is guarded and returns NumPy's wrapped value, which is what this mapping's original pandas UDF produced.

A third commit covers a quotient that overflows to an infinity, which the old and new formulas both returned as nan. NumPy keeps the infinity, and an infinite quotient is its own floor, so 1e300 // 1e-300 now skips the flooring step.

Why are the changes needed?

Both answers disagree with pandas, which pandas-on-Spark is meant to match.

The 1.0 // 0.1 case is also inconsistent inside Spark itself. 1.0 % 0.1 gives 0.09999999999999995, and a remainder that large only makes sense if the quotient is 9: with a quotient of 10 there would be nothing left over.

Does this PR introduce any user-facing change?

No. np.floor_divide goes through the pandas-on-Spark floordiv dunder operation before this registry is consulted, so the mapping is not reached today, as the comment on the entry notes.

A follow-up PR will cover the reachable path. The same two bugs are live in data_type_ops/num_ops.py (IntegralOps.floordiv, FractionalOps.floordiv and both rfloordiv), which use F.floor(lc / rc): ps.Series([1.0]) // 0.1 gives 10.0 where pandas gives 9.0, and psser // 2 is off by one above 2^53. That one does change what users see on a long-standing path, so it gets its own JIRA and PR rather than riding along here.

How was this patch tested?

Added three cases to test_floor_divide_func: divisors that binary cannot represent exactly, integral operands above 2^53, and -2**63 // -1. They compare exactly rather than with almost=True, which would let an off-by-one through at these magnitudes.

Ran pyspark.pandas.tests.test_numpy_compat and pyspark.pandas.tests.connect.test_parity_numpy_compat; both pass.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Claude Opus 5)

[Spenserrrr]

Hi @zhengruifeng! This is a PR fixing thefloor_divide. 1.0 // 0.1 was returning 10, but pandas and NumPy give 9. This is because the division rounds the quotient up before the floor runs. Right now, the code calculates the quotient out of the remainder the way NumPy does. And integral operands were cast to double first, which loses the low bits above 2^53, so they now divide in integer space. Could you take a look when you have time?

One thing is worth your attention. The mapping I fixed here is the unreachable one, since np.floor_divide goes through the floordiv dunder first. The same two bugs are live in data_type_ops/num_ops.py, where ps.Series([1.0]) // 0.1 gives 10.0 against pandas' 9.0 and psser // 2 is off by one above 2^53. I will fix that as a separate PR with its own JIRA, since it changes what users see on a path that has behaved this way for a long time. Also please tell me if you
would rather leave it alone. Thanks!
cc @Yicong-Huang

[zhengruifeng]

what about dispatching earlier by datatype?

when(is int, _floor_divide_integral)
.when(is float, _floor_divide_floating)
.otherwise

[Spenserrrr]

Thanks for the advice @zhengruifeng! I originally choose to write it nested to keep the diff small. I revise the structure in the new commit. It is now a single chain:

• null/nan and a zero divisor first, since they answer the same way for every operand type;
• when(both integral) -> _floor_divide_integral;
• when(either floating) -> the infinity and signed-zero cases, then _floor_divide_floating;
• otherwise is decimal

All the existing tests still pass, so the refactor is behaviour-preserving.

[zhengruifeng]

Merge Summary:

• merged into master 670de9d
• merged into branch-4.x 966e567

Posted by merge_spark_pr.py