Skip to content

Add .arrows output format, streaming rows as Arrow IPC - #1

Merged
simonw merged 5 commits into
mainfrom
claude/funny-noether-7rc2rv
Nov 18, 2026
Merged

simonw merged 5 commits into
mainfrom
claude/funny-noether-7rc2rv

Conversation

@simonw

@simonw simonw commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Adds a .arrows output format for table, view, row and SQL query pages. It returns an Arrow IPC stream, served as application/vnd.apache.arrow.stream, with one record batch per page of rows.

Claude Fable 5.2 Medium

It is built on the streaming output renderers in Datasette main (row_stream and Response.stream()), so ?_stream=1 streams every row of a table or view without loading it into memory.

Column types

Arrow needs the schema before the first row, so each column's type comes from its declared SQLite type together with the values in the first page of rows: int64, float64, string or binary. Mixed columns widen to string.

A value on a later page is converted if the column's type can represent it. If it cannot - text in a column declared as INTEGER, for example - the export fails by dropping the connection, which is how Datasette handles any error part of the way through a stream. pyarrow reads a stream truncated that way without raising an error, so clients need to check for the failed HTTP transfer.

?_labels=on adds a column_label column after each foreign key column, as CSV does.

Dependency on unreleased Datasette

Streaming output renderers are not yet in a Datasette release, so:

  • pyproject.toml requires datasette>=1.0a42 with a uv source pointing at Datasette main.
  • Both workflows install Datasette from main after installing the plugin.

These should be replaced with a version pin once the next Datasette alpha is out.

simonw/datasette#3070 makes ?_stream=1 about five times faster and lists this format with TSV and CSV in the export dialog, so that "Include all rows" applies to it. This plugin works without that change.

Testing

14 tests, passing against Datasette main with and without that pull request. Also run against a real server: a 200,000 row table streamed and read back correctly in pyarrow.

🤖 Generated with Claude Code

https://claude.ai/code/session_016Bvfq9SBhxPegj4hvi5NF4

claude and others added 3 commits November 15, 2026 16:39
Registers an output renderer that returns one Arrow record batch per
page of rows, and every row of a table or view with ?_stream=1.
Requires streaming output renderers from Datasette main.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Bvfq9SBhxPegj4hvi5NF4
@simonw simonw added the enhancement New feature or request label Nov 18, 2026
@simonw
simonw merged commit 43e7f1f into main Nov 18, 2026
10 checks passed
simonw added a commit that referenced this pull request Nov 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants