Preview an exploration result without converting the whole frame - #2058
Merged
Conversation
numpy has no decimal and no date, so both sit in an object column, and the summary reads an object column as categorical: a money column loses its mean and range and comes out as an enum of decimal.Decimal, which yaml.safe_load cannot read back at all. Cast both to the types numpy does have before the summary. Inferred from the values rather than the source dtype, because pandas produces such a column for a DECIMAL result of its own, so this is not only about frames arriving from another dataframe library.
BaseSQLSource gains a dataframe_backend param and a fetch method that get reads through, so the data path can hand back polars or pyarrow while execute stays on pandas for the schema and metadata queries, which read results with .iloc and pandas dtypes. DuckDB and DataFusion build the requested frame themselves rather than converting a pandas one: on a 5M row DuckDB table that takes the fetch from 857ms and 566MB to 229ms and 73MB for arrow, and 628ms and 358MB for polars. Nothing changes by default, since to_backend returns the frame it was given when no backend is named. A geometry result stays pandas whatever is asked for, because geometry only survives as a GeoDataFrame and narwhals has no geometry dtype, so any other library would hold the column as opaque bytes and drop it from the schema. The consumers that were not already behind an as_pandas boundary are guarded: the layout download writers, the AI export and docx table, the DuckDB mirrors, the source preview and result_to_dataframe.
fetch_arrow_table is deprecated in duckdb 1.5 and the suite turns that warning into an error, while to_arrow_table does not exist on a connection before 1.5. arrow() is present throughout, but answers with a Table up to 1.3 and a RecordBatchReader from 1.4, and Pipeline.data rejects the reader, so the result goes through pa.table either way. Verified against 1.2, 1.3, 1.4 and 1.5.
execute_exploration_sql read through execute, which is contracted to return pandas, so a source set to build polars or pyarrow paid for a full copy of the result before the model saw five rows of it. It reads through fetch now, and format_exploration_result takes a frame from any library: the shape and the column names come off the frame itself, and only the cells actually rendered are converted. Dates and decimals are normalised on the way out. Both land in an object column, which the dtype line reports as str, and which of the two arrives varies by library, since pandas converts a DECIMAL to float itself while polars and pyarrow keep it. That normaliser is named for what it does now rather than for its first caller, since the summary path is no longer the only one using it.
sanitize_column_names copied the whole frame to relabel its columns. rename gives the same columns and the same isolation from the caller's frame, and skips the copy outright on a pandas that has copy-on-write. Covers the helper, which had no tests.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #2058 +/- ##
==========================================
+ Coverage 75.24% 75.75% +0.50%
==========================================
Files 212 214 +2
Lines 39182 39965 +783
==========================================
+ Hits 29484 30276 +792
+ Misses 9698 9689 -9 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
ahuang11
approved these changes
Aug 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The exploration SQL tool read its result through
execute, which is contracted to return pandas, so a source set to build polars or pyarrow paid for a full copy of the result before the model saw five rows of it. It reads throughfetchnow and converts only the cells it renders, taking a 5M row preview from 1.09s and 468MB to 0.48s and 70MB.Fixes #2041