Skip to content

Faster ?_stream=1, export dialog groups formats that can stream - #3070

Merged
simonw merged 2 commits into
mainfrom
claude/funny-noether-7rc2rv
Nov 18, 2026
Merged

simonw merged 2 commits into
mainfrom
claude/funny-noether-7rc2rv

Conversation

@simonw

@simonw simonw commented Nov 15, 2026 •

Copy link
Copy Markdown
Owner

Two follow-ups to the streaming output renderers in #3069, found while building datasette-arrows against main.

  • ?_stream=1 fetches pages of max_returned_rows
  • Export dialog groups formats that can stream with TSV and CSV
CleanShot 2026-11-18 at 7 45 15 AM@2x
Claude Code

?_stream=1 fetches pages of max_returned_rows

Streaming every row paginated using the default page size of 100, so a large export ran ten times more queries than it needed. It now uses the largest page size.

  • An explicit ?_size= in the URL is still respected.
  • A size set in table configuration no longer applies to a streaming request.
  • row_stream.pages() therefore yields pages of up to max_returned_rows rows when streaming.

Microbenchmark: /db/table.csv?_stream=1 for a 200,000 row, four column table (7MB of CSV), median of five runs against a local server:

Page size Time
100 rows (before) 5.21s
1,000 rows (after) 1.09s

The "before" figure was measured by adding &_size=100 to the URL on this branch.

Export dialog groups formats that can stream with TSV and CSV

"Include all rows" only applied to TSV and CSV, so a plugin format that accepts row_stream could only be streamed by editing the URL. Those formats now share the TSV/CSV form, so "Include all rows" and "Expand labels" apply to them.

  • Formats that cannot stream stay in the list below, unchanged.
  • is_binary and is_webpage formats in the group keep their reduced set of actions.
  • Formats in the group request ?_size=max, as TSV and CSV do, so without the checkbox they return up to max_returned_rows rows rather than the rows on the page.
  • The "these exports start from this page" note lists every format that can stream.

Testing

  • The full test suite passes.
  • New tests cover the page sizes and the dialog markup, including a binary format that can stream.
  • The Playwright tests were not run: the environment's browser did not match the project's Playwright version. The dialog was exercised with a separate Playwright script with datasette-arrows installed: opening it, ticking "Include all rows" and checking the URL the Arrow Download button requests.

Generated by Claude Code

@codecov

codecov Bot commented Nov 15, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 0% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 0.00%. Comparing base (1770c90) to head (defca32).

Files with missing lines Patch % Lines
datasette/views/table.py 0.00% 4 Missing ⚠️
Additional details and impacted files
@@          Coverage Diff          @@
##            main   #3070   +/-   ##
=====================================
  Coverage   0.00%   0.00%           
=====================================
  Files         80      80           
  Lines      14282   14285    +3     
=====================================
- Misses     14282   14285    +3     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@simonw simonw added this to the Datasette 1.0 stable milestone Nov 16, 2026
Streaming every row previously paginated using the default page size
of 100, so a large export ran ten times more queries than it needed.
It now uses the largest page size unless ?_size= is in the URL.

Microbenchmark: /db/table.csv?_stream=1 for a 200,000 row, four column
table (7MB of CSV), median of five runs against a local server:

    100 row pages (before):   5.21s
    1,000 row pages (after):  1.09s

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Bvfq9SBhxPegj4hvi5NF4
Plugin output formats that accept row_stream now share the form used
by TSV and CSV, so the "Include all rows" and "Expand labels"
options apply to them.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Bvfq9SBhxPegj4hvi5NF4
@simonw
simonw force-pushed the claude/funny-noether-7rc2rv branch from dcc61cf to defca32 Compare November 18, 2026 15:32
@simonw
simonw merged commit c397483 into main Nov 18, 2026
15 checks passed
@simonw
simonw deleted the claude/funny-noether-7rc2rv branch November 18, 2026 15:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants