Skip to content

feat: Add empty_as_null and keep_nulls options to explode - #3709

Open
FBruzzesi wants to merge 11 commits into
mainfrom
feat/explode-signature
Open

feat: Add empty_as_null and keep_nulls options to explode#3709
FBruzzesi wants to merge 11 commits into
mainfrom
feat/explode-signature

Conversation

@FBruzzesi

@FBruzzesi FBruzzesi commented Jun 25, 2026

Copy link
Copy Markdown
Member

Description

In light of polars deprecation warnings showing up in our CI, this PR expands the explode signature to match polars' one. Adapted to all backends and backported for polars (safe for 1 column explode, let polars do polars internal on multiple columns).


Test cases were Claude assisted, trying to cover all four flag combos for single and multi columns including null-vs-empty same-index case, and shape-mismatch errors.


Merging this PR would make a lot of the jobs in #3711 happy

What type of PR is this? (check all applicable)

  • 💾 Refactor
  • ✨ Feature
  • 🐛 Bug Fix
  • 🔧 Optimization
  • 📝 Documentation
  • ✅ Test
  • 🐳 Other

@FBruzzesi FBruzzesi added the enhancement New feature or request label Jun 25, 2026
@FBruzzesi

Copy link
Copy Markdown
Member Author

CI failures are related to a change in polars error message for get_column

@FBruzzesi

Copy link
Copy Markdown
Member Author

Regarding the pyspark-connect, I guess we can lower coverage to 90%? (it's 94 now, target is 95)

@MarcoGorelli MarcoGorelli left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks @FBruzzesi !

Regarding the pyspark-connect, I guess we can lower coverage to 90%? (it's 94 now, target is 95)

yup sure

@dangotbanned

dangotbanned commented Jun 26, 2026

Copy link
Copy Markdown
Member

@FBruzzesi
FBruzzesi requested a review from dangotbanned June 26, 2026 12:29
@dangotbanned dangotbanned added the high priority Your PR will be reviewed very quickly if you address this label Jun 27, 2026
@dangotbanned

Copy link
Copy Markdown
Member

I've marked this as high priority (#3709 (comment)) because of the CI fix.

I obviously agree with adding the flags 😅 (#3347).
But would have suggested just passing empty_as_null=True, keep_nulls=True now to fix CI - and adding the feature as a follow-up.

I haven't reviewed the changes in each backend, but if @MarcoGorelli has and is happy - then maybe we just merge?

@MarcoGorelli

Copy link
Copy Markdown
Member

that's a good point thanks! sure, let's do that, that'll make this easier to revert if necessary

@dangotbanned dangotbanned removed the high priority Your PR will be reviewed very quickly if you address this label Jun 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants