Retry with backoff + stall timeout for HTTP range requests - #145
Merged
Conversation
Fault injection showed the http backend dies on the first hiccup: one
mid-body connection drop or transient 503 fails the whole proof, and a
stalled response hangs it forever (fetch has no timeout of its own).
Range GETs are idempotent and If-Range already guards consistency, so
transient failures are safely retryable:
- httpReadRangeInto retries up to 3 times with exponential backoff
(300ms doubling). Retries happen inside the connection-cap slot, so a
failing origin is not hammered by other readers meanwhile.
- A stall watchdog (AbortController, bumped on headers and on every body
chunk, default 10s) turns a wedged connection into a bounded, retryable
failure.
- Errors that retrying cannot fix are classified permanent at their throw
site and rethrown immediately: 4xx (except 429) and a changed validator.
The degrade-to-full control-flow signal passes through untouched.
- Tunables exported as httpRetryConfig {retries, backoffMs, stallTimeoutMs}
rather than plumbed through per-call options.
Out of scope, deliberately: retrying the open() probe, resuming a partial
range from its break point, and un-sticking a failed degrade-mode full-body
download.
4 new tests against a fault-injecting server: mid-body drop recovery,
transient-503 recovery vs fail-fast 404, bounded stall recovery, retry
exhaustion. 82 tests pass; lint clean; bundles rebuilt.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01833VaUEJmrFZ7bVprrWwpp
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Fault injection against real browser proving showed the http backend dies on the first hiccup: a single mid-body connection drop or transient 503 fails the whole proof, and a stalled response (headers arrive, bytes stop) hangs it forever —
fetchhas no timeout of its own. On an unstable connection with a large zkey, some hiccup during the multi-minute transfer is near-certain.Change (+102 lines in
src/httpfile.js, no new API surface)Range GETs are idempotent and
If-Rangealready guards consistency, so transient failures are safely retryable:AbortControllerarmed before the fetch and bumped on headers and every body chunk; 10 s of silence aborts the attempt, which the retry loop turns into a bounded delay.err.permanent); the degrade-to-full control signal passes through untouched.httpRetryConfig { retries, backoffMs, stallTimeoutMs }instead of per-call option plumbing.Measured (authV3 28 MiB, fault every 4th zkey request, single
groth16.provecall in Chromium)Deliberately out of scope
Retrying the
open()probe; resuming a partial range from its break point (a retry re-requests the chunk; the block cache caps the loss when enabled); un-sticking a failed degrade-mode full-body download (Range-less origins without cache still fail on drops).Tests
4 new cases against a fault-injecting server asserting exact attempt counts: mid-body drop recovery, transient-503 recovery vs fail-fast 404, bounded stall recovery, retry exhaustion. 85 tests pass; lint clean; bundles rebuilt.
🤖 Generated with Claude Code
https://claude.ai/code/session_01833VaUEJmrFZ7bVprrWwpp