Repository navigation
Add .arrows output format, streaming rows as Arrow IPC - #1
Merged
Merged
Conversation
Registers an output renderer that returns one Arrow record batch per page of rows, and every row of a table or view with ?_stream=1. Requires streaming output renderers from Datasette main. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Bvfq9SBhxPegj4hvi5NF4
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Bvfq9SBhxPegj4hvi5NF4
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a
.arrowsoutput format for table, view, row and SQL query pages. It returns an Arrow IPC stream, served asapplication/vnd.apache.arrow.stream, with one record batch per page of rows.Claude Fable 5.2 Medium
It is built on the streaming output renderers in Datasette main (
row_streamandResponse.stream()), so?_stream=1streams every row of a table or view without loading it into memory.Column types
Arrow needs the schema before the first row, so each column's type comes from its declared SQLite type together with the values in the first page of rows:
int64,float64,stringorbinary. Mixed columns widen tostring.A value on a later page is converted if the column's type can represent it. If it cannot - text in a column declared as
INTEGER, for example - the export fails by dropping the connection, which is how Datasette handles any error part of the way through a stream. pyarrow reads a stream truncated that way without raising an error, so clients need to check for the failed HTTP transfer.?_labels=onadds acolumn_labelcolumn after each foreign key column, as CSV does.Dependency on unreleased Datasette
Streaming output renderers are not yet in a Datasette release, so:
pyproject.tomlrequiresdatasette>=1.0a42with a uv source pointing at Datasette main.These should be replaced with a version pin once the next Datasette alpha is out.
simonw/datasette#3070 makes
?_stream=1about five times faster and lists this format with TSV and CSV in the export dialog, so that "Include all rows" applies to it. This plugin works without that change.Testing
14 tests, passing against Datasette main with and without that pull request. Also run against a real server: a 200,000 row table streamed and read back correctly in pyarrow.
🤖 Generated with Claude Code
https://claude.ai/code/session_016Bvfq9SBhxPegj4hvi5NF4