feat: add DataFrame.equals and Series.equals by AdrianSosic · Pull Request #3717 · narwhals-dev/narwhals

AdrianSosic · 2026-06-26T22:12:32Z

Description

Just a draft for now ... Tests and docs missing!

Adds Series.equals and DataFrame.equals.

What type of PR is this? (check all applicable)

Related issues

Closes [Enh]: Add support for {DataFrame, Series}.equals #3715

Checklist

Code follows style guide (ruff)
Tests added
Documented the changes

…ackends

AdrianSosic · 2026-06-28T06:53:33Z

@FBruzzesi: Is this what you had in mind? If yes, I could add DataFrame.equals in the same style.

…s backends

…es=False

camriddell · 2026-07-01T16:30:45Z

Hi @AdrianSosic thanks for the PR to review this feature prototype. I took a look at the code and it looks like you've got everything in the right places and the implementations look solid.

Can you add some tests so we know the cases you're considering? The tests should at least encompass:

Comparison within all datatypes
Comparison across datatypes (e.g. both within and across supertypes, so int -> string; and across supertypes float -> int)
Comparison with Nulls (Null to Null, Null to Value)
Comparison with NaN (NaN to NaN, NaN to Value)

Here is a quick test I wrote against your PR for some local investigation. It's not exhaustive for the above cases, and only considers Series objects, but should at lest get you started:

Quick/incomplete Pytest code to bootstrap testing code

import narwhals as nw

import pytest


@pytest.mark.parametrize(
    ('left', 'right', 'null_equal', 'expected'),
    [
        ( [1,    2,    3],             [1,    2,    3],             True, True ),
        ( [1,    2,    3],             [1,    2,    3],             False, True ),

        ( [1.1,  2.2,  3.3],           [1.1,  2.2,  3.3],           True, True ),
        ( [1.1,  2.2,  3.3],           [1.1,  2.2,  3.3],           False, True ),

        # should we expect either of these cases to be equal?
        ( [1.1,  2,    float('nan')],  [1.1,  2,    float('nan')],  True,  True ),
        ( [1.1,  2,    float('nan')],  [1.1,  2,    float('nan')],  False, True ),

        ([1,     2,    None],          [1,    2,    3],             True, False),
        ([1,     2,    None],          [1,    2,    3],             False, False),

        ( [1,    2,    None],          [1,    2,    None],          True, True ),
        ( [1,    2,    None],          [1,    2,    None],          False, False ),

        ([1,     2,    3],             [1,    2,    0],             True, False),
        ([1,     2,    3],             [1,    2,    0],             False, False),
    ],
)
def test_equals(constructor_eager: ConstructorEager, left, right, null_equal, expected) -> None:
    left_native = constructor_eager({'left': left})['left']
    right_native = constructor_eager({'right': right})['right']
    left_nw = nw.from_native(left_native, series_only=True)
    right_nw = nw.from_native(right_native, series_only=True)

    result = left_nw.equals(right_nw, null_equal=null_equal)
    assert result == expected

I have also brought up another point in the issue about NaN comparisons, so the remaining chunk of effort will be to implement that within this feature depending on what the conclusion is.

camriddell

@AdrianSosic looking good so far! I've addressed most of the questions you left in the PR about expected behaviors. For now, you can leave the within supertype (e.g. int/float) comparison as-is.

For the test cases themselves, these look awesome for the numeric comparisons. Go ahead and add in cases for non-numeric datatypes (strings, dates, Categorical, etc.) as well.

Once the full test-suite is in-place, spruce up the code as much as possible. If you get stuck working on the implementation push back up and we can help iterate. Thanks so much.

camriddell · 2026-07-02T22:16:18Z


+    def equals(
+        self, other: Self, *, check_dtypes: bool, check_names: bool, null_equal: bool
+    ) -> bool:


This should also compare NaNs, which will always be equivalent to one another.

You could do something like

lhs = ... rhs = ... try: both_nan = pc.fill_null(pc.and_(pc.is_nan(lhs), pc.is_nan(rhs)), False) except ArrowNotImplementedError: both_nan = pa.scalar(False) # may need to broadcast this back out to a full array? values_equal = pc.equal(lhs, rhs) values_or_nan_equal = pc.fill_null(pc.or_(values_equal, both_nan), False) # no nulls in this array if null_equal: # make the extra comparison with null values if null_equal is True .... return ... return bool(pc.all(values_or_nan_equal).as_py())

16c7980

@camriddell: Thanks for the suggestion 👌🏼 Note that I slightly rewrote so that the question of broadcasting doesn't even arise in the first place

camriddell · 2026-07-02T22:24:42Z

+TBD = None
+_NAN_HANDLING = pytest.mark.xfail(
+    strict=True, reason="NaN/null disambiguation not yet decided"
+)


NaNs should always be equivalent to one another. The only exception to this rule will be when calling equals(…, null_equal=False) on pandas objects, since it cannot disambiguate Null from NaN.

I understand the problem, but not sure if I'm 100% what is the desired behavior for this case now. For the arrow backend, the problem could in principle be resolved, but not for the numpy backend. So what do we do?

Resolve for pyarrow or

Keep behavior consistent for both backends?

And then, regarding the failing case(s) that remain:

Do we throw an error if the unsupported comparison is requested by the user?

Or do we silently just return True even though we should return False for null-comparison when null_equals=False?

Or do we document this somewhere?

camriddell · 2026-07-02T22:29:36Z

+)
+_CROSS_TYPE = pytest.mark.xfail(
+    strict=True, reason="Inconsistent behavior across backends for incompatible types"
+)


Backends like pyarrow error out if it does not have an available kernel to perform the operation. For these cases, start with a try/except and catch the ArrowNotImplementedError. I don't believe pyarrow exposes a way to interact with the type hierarchy to check what is comparable beforehand.

Polars returns False for these cases

>>> import polars as pl >>> pl.Series([1,2,3]).equals(pl.Series([*'123'])) False

pyarrow raises errors when working with un-related dtypes

>>> import pyarrow.compute as pc >>> import pyarrow as pa >>> import pyarrow.compute as pc >>> pc.equal(pa.array([1,2,3]), pa.array([*'123'])) Traceback (most recent call last): File "<python-input-5>", line 1, in <module> pc.equal(pa.array([1,2,3]), pa.array([*'123'])) ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/cameron/.pyenv/versions/3.14.2/lib/python3.14/site-packages/pyarrow/compute.py", line 252, in wrapper return func.call(args, None, memory_pool) ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^ File "pyarrow/_compute.pyx", line 399, in pyarrow._compute.Function.call File "pyarrow/error.pxi", line 155, in pyarrow.lib.pyarrow_internal_check_status File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status pyarrow.lib.ArrowNotImplementedError: Function 'equal' has no kernel matching input types (int64, string)

camriddell · 2026-07-02T22:31:25Z

+        "Bare ChunkedArray carries no name metadata, resulting in empty names - "
+        "TBD whether check_names=True should raise for pyarrow in case of empty names"
+    ),
+)


The names should exist on the narwhals._arrow.series.ArrowSeries object. The code seems to already have checks for names in them, unless this note was about something else?

Yes, the _name field exists on ArrowSeries, but my question is a different one: pa.ChunkedArray is an anonymous array. So when narwhals wraps one, _name is always set to "" regardless of what column it came from:

import pyarrow as pa import narwhals as nw t = pa.table({"left": [1, 2], "right": [1, 2]}) left = nw.from_native(t["left"], series_only=True) right = nw.from_native(t["right"], series_only=True) print(left.name) # "" print(right.name) # "" print(left.equals(right, check_names=True)) # True — names are both "", so check passes silently

Ah, I see the where the issue arises now. Can you rewrite the test in this fashion so the metadata is preserved appropriately?

left_native = constructor_eager({left_name: data}) right_native = constructor_eager({right_name: data}) left_nw = nw.from_native(left_native)[left_name] right_nw = nw.from_native(right_native)[right_name] result = left_nw.equals(right_nw, check_names=check_names) assert result == expected

camriddell · 2026-07-02T23:02:42Z

+_DTYPE_INFERENCE = pytest.mark.xfail(
+    strict=True,
+    reason="Constructor dtype inference is backend-dependent for whole-number floats",
+)


@FBruzzesi are there any test helpers that can help with this problem? We want the constructors to NOT eagerly downcast values. For example, these should each end up as floats dtype.

>>> pd.Series([1.0]) 0 1.0 dtype: float64 >>> pd.Series([1.0]).convert_dtypes(dtype_backend="numpy_nullable") 0 1 dtype: Int64 >>> pd.Series([1.0]).convert_dtypes(dtype_backend="pyarrow") 0 1 dtype: int64[pyarrow]

There is convert_dtypes(convert_integer=False, dtype_backend="pyarrow"), but that causes ints to fall out of the requested dtype:

>>> pd.Series([1]).convert_dtypes(convert_integer=False, dtype_backend="pyarrow").dtype dtype('int64') >>> pd.Series([1.0]).convert_dtypes(convert_integer=False, dtype_backend="pyarrow").dtype double[pyarrow]

I assume you realized there was none and went ahead with #3749 🙏🏼

FBruzzesi

Thanks @AdrianSosic, this is definitely moving towards the right direction. I think some details and edge cases cross typing are still missing.

In principle you could re-use/re-factor a lot of what's used in assert_{frame,series}_equal by taking out the core functionalities. Here are the helper functions I added in there were for:

numeric: it just uses Series.is_close
nested types comparison: _check_list_like and _check_struct - these are loops over rows, hence quite slow
categorical: _check_categorical

You might also re-use the test cases in tests/testing/assert_{frame,series}_equal_test.py to validate the implementation

FBruzzesi · 2026-07-05T19:12:44Z

+        return all(
+            self.get_column(name).equals(
+                other.get_column(name),
+                check_dtypes=False,


Shouldn't we check also the dtype match? I think it's ok to turn off the check as long as above

- if self.columns != other.columns: + if self.schema != other.schema:

FBruzzesi · 2026-07-05T19:17:49Z

    cum_sum: Method[Self]
    diff: Method[Self]
    drop_nulls: Method[Self]
+    equals: Method[bool]


You might need to actually implement the method for backward compat. From the Series.equals docs:

Changed in version 0.20.31: The strict parameter was renamed check_dtypes.

You can find a lot of examples in the codebase for which we align past behavior to more recent one

FBruzzesi · 2026-07-05T19:22:50Z

+            if self.native.isna().any() or other.native.isna().any():
+                return False
+        other_native = (
+            other.native.astype(self.native.dtype)


Not sure how I feel about this, can't downcasting potentially raise an error? This is where a supertype(left_dtype, right_dtype) would be quite useful

Agreed we should avoid manually casting here.

>>> import pandas as pd >>> self = pd.Series([1, 2, 3]) >>> other = pd.Series([*'123']) >>> self.equals(other) False >>> self.equals(other.astype(self.dtype)) True dtype: int64 >>> other.astype(self.dtype) 0 1 1 2 2 3

Instead of directly using pandas.Series.equals we should probably use the == operator along with a .isna() check on both sides since the == operator will take care of the casting for us:

>>> pd.Series([1,2,3], dtype="int64") == pd.Series([1.0, 2.0, 3.0], dtype="float64") 0 True 1 True 2 True dtype: bool

Good point! But I guess the supertype utility does not yet exist, right? Is this something for a separate PR? Also, for which backends would we need it? All of them, or just pandas, specifically for this PR? For the numpy backend, I think that np.result_type may do the job, but unclear how to best handle the pyarrow backend...

Sorry for the "not enough" context - that's correct get_supertype is still not here. Hopefully it will eventually get merged #3396

Ok, so I guess I'll be waiting then until this happens

I'll be waiting then until this happens

Please don't! There is not enough consensus on it for now - I brought it up as a nice use case for it, not for blocking you

FBruzzesi · 2026-07-05T19:23:35Z

+            return bool(pc.all(result, skip_nulls=False).as_py())
+        if lhs.null_count or rhs.null_count:
+            return False
+        return bool(pc.all(pc.equal(lhs, rhs)).as_py())


Does pc.equal handle upcasting internally?

yep

>>> import pyarrow as pa >>> import pyarrow.compute as pc >>> pc.equal(pa.array([1,2,3], type=pa.int64()), pa.array([1.0, 2.0, 3.0], type=pa.float64())) <pyarrow.lib.BooleanArray object at 0x7f46f53ceda0> [ true, true, true ]

so no TODO here, right?

camriddell · 2026-07-08T18:55:38Z

@AdrianSosic we recently landed #3749 so if you rebase/merge against the main branch you can now preserve the float dtype for tests of whole number floats. This means that in the test fixtures [1.0, 2.0, 3.0] will return a Series with a float dtype for the pandas constructors in the test suite.

…w series equals

…dtypes rename

AdrianSosic · 2026-07-16T09:00:04Z

@camriddell @FBruzzesi: Thanks for you reviews ⭐ I've addressed all comments / replied with further questions, and the currently failing tests are related to the remaining open threads.

So the only remaining task on my desk is adding more comparisons like @camriddell requested. Is there any specific pattern you'd like me to follow, i.e. how "exhaustive" should these type comparisons be?

Also, any further things to be addressed?

camriddell · 2026-07-23T20:47:48Z

@camriddell @FBruzzesi: Thanks for you reviews ⭐ I've addressed all comments / replied with further questions, and the currently failing tests are related to the remaining open threads.

So the only remaining task on my desk is adding more comparisons like @camriddell requested. Is there any specific pattern you'd like me to follow, i.e. how "exhaustive" should these type comparisons be?

Also, any further things to be addressed?

This is shaping up very nicely! I addressed the question about pyarrow array names, this is unavoidable but if we turn the tables into Narwhals DataFrames, then pull the PyarrowSeries out we can preserve the names. The other usage is impossible for the reasons you pointed out previously.

Additionally, I realize that I had a comment pending that was never sent about how to handle the pandas cases for and how to avoid manual casting:

>>> import pandas as pd
>>> self = pd.Series([1, 2, 3])
>>> other = pd.Series([*'123'])
>>> self.equals(other)
False
>>> self.equals(other.astype(self.dtype))
True
dtype: int64
>>> other.astype(self.dtype)
0    1
1    2
2    3

Instead of directly using pandas.Series.equals we should probably use the == operator along with a .isna() check on both sides since the == operator will take care of the casting for us:

>>> pd.Series([1,2,3], dtype="int64") == pd.Series([1.0, 2.0, 3.0], dtype="float64")
0    True
1    True
2    True
dtype: bool

AdrianSosic mentioned this pull request Jun 26, 2026

[Enh]: Add support for {DataFrame, Series}.equals #3715

Open

dangotbanned added the needs discussion label Jun 27, 2026

feat: implement Series.equals across pandas-like, Arrow, and Polars b…

2dfd834

…ackends

AdrianSosic force-pushed the feat/equals branch from dfaa074 to 2dfd834 Compare June 28, 2026 06:52

AdrianSosic added 2 commits June 28, 2026 09:58

feat: implement DataFrame.equals across pandas-like, Arrow, and Polar…

fdec695

…s backends

fix: cast other to self dtype in pandas Series.equals when check_dtyp…

5f5af7e

…es=False

tests: add tests for Series.equals

faed3fc

camriddell requested changes Jul 2, 2026

View reviewed changes

camriddell mentioned this pull request Jul 3, 2026

feat: stabilize dtype conversion across pandas, pyarrow, and modin test constructors #3749

Merged

10 tasks

FBruzzesi reviewed Jul 5, 2026

View reviewed changes

AdrianSosic added 6 commits July 16, 2026 09:39

Merge remote-tracking branch 'main' into feat/equals

6a6fa3a

fix: remove xfail for int vs string equals comparison

deb5259

fix: wrap pc.equal in try-except for ArrowNotImplementedError in Arro…

a89ce90

…w series equals

fix: check schema instead of columns in Arrow DataFrame.equals

8b54202

fix: implement PolarsSeries.equals explicitly to handle strict→check_…

b9c2326

…dtypes rename

fix: handle NaN equality in Arrow series equals

16c7980

Uh oh!

Conversation

AdrianSosic commented Jun 26, 2026

Description

What type of PR is this? (check all applicable)

Related issues

Checklist

Uh oh!

AdrianSosic commented Jun 28, 2026

Uh oh!

camriddell commented Jul 1, 2026

Uh oh!

camriddell left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

FBruzzesi left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

camriddell commented Jul 8, 2026

Uh oh!

AdrianSosic commented Jul 16, 2026

Uh oh!

camriddell commented Jul 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development