Repository navigation
Consolidate retry layers; fast-fail 5xx for instant failover - #9
Merged
Merged
Conversation
The request path had three retry budgets stacked on top of each other: retryableCall (4× per-node), MultiClient.FetchBlocks (failover across N endpoints), and fetchWithRetry (4× outer). Combined with bisect and the dead-letter sweep, a single bad block could amplify into ~600 HTTP requests in absolute worst case — and ~30+ in realistic flaky-endpoint scenarios. The 5xx in-place retries also delayed failover by ~3.5s against a sick node before MultiClient even got to try a different one. Each layer now has one job: - retryableCall: only retries network errors, once. Covers the keep-alive-closed race; if it doesn't help in one retry it isn't going to. 5xx returns *ErrServerError immediately so MultiClient fails over to a different node — same fast-fail shape that 429 (*ErrRateLimited) already uses, for the same reason. - MultiClient.FetchBlocks: AUTHORITATIVE failover loop. Walks every eligible endpoint headroom-sorted; logs server-error failover at Debug level (matching the existing 429 path). - fetchWithRetry: deleted. processBatch now calls p.client.FetchBlocks directly. The outer 4× wrapper just re-hit the same exhausted endpoint set and amplified every transient blip. - processWithSplit (bisect): unchanged — still isolates one bad block from a healthy batch. - Dead-letter sweep: unchanged — slow-time recovery for whole-infra outages, capped by failure_max_retries. New worst case: 5 bisect levels × 2 endpoints × 1 in-place retry × 3 sweep cycles = 30 requests. ~20× reduction. Healthy blocks stay at one request. Tests added: - TestRPCClient_call_5xxFailsFastNoRetry: server returns 500; *ErrServerError surfaces on the first hit, no in-place retry. - TestRPCClient_call_NetworkErrRetriesOnce: torn connection on the first attempt is recovered by the second; no further retries. README "Retry, failover, and dead-letter" rewritten to document the layer map, per-layer retry budgets, the fast-fail rule for 5xx/429, and the worst-case amplification calculation.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Audit found three retry budgets stacked on top of each other in the request path, plus 5xx in-place retries that delayed endpoint failover. A single bad block could amplify into ~600 HTTP requests in absolute worst case (~30+ in realistic flaky scenarios) before the dead-letter sweep gave up.
This PR collapses them into one retry budget per layer, each with a distinct job, and makes 5xx fail-over immediately instead of burning four in-place retries against a sick node.
Changes
internal/rpc/rpc.go,internal/rpc/rest.gomaxRetries=4withhttpRetryNetwork=1. Network errors only — covers the keep-alive-closed race.*ErrServerErrorfor 5xx, mirroring the existing*ErrRateLimitedfor 429. Both fail-fast so MultiClient can route to a different node on the next iteration.RPCClient.callto lay out the policy explicitly.internal/rpc/multi.goFetchBlocksfailover loop now logsendpoint server-error, failing overat Debug for the new error class, matching the 429 path.internal/pipeline/pipeline.gofetchWithRetry. Replaced its caller with a directp.client.FetchBlocks(ctx, heights)call.internal/rpc/rpc_test.goTestRPCClient_call_5xxFailsFastNoRetry: server returns 500; assert*ErrServerErrorreturned on the FIRST hit, no in-place retry.TestRPCClient_call_NetworkErrRetriesOnce: torn connection on attempt 1 (Hijacker close); assert recovery on attempt 2 with exactly 2 hits.README.mdWorst-case math (after this PR)
At
batch_size=20, N=2, failure_max_retries=3: 30 requests. Down from ~600 in absolute worst case before, and ~30+ in realistic flaky scenarios.Behaviour intentionally unchanged
Test plan
go build ./...clean.go vet ./...clean.go test ./...green (full suite).fetch failed, retryinglog lines drop to ~0 (replaced byendpoint server-error, failing overat Debug, only visible when--log.level=debug).pg_stat_activitydoesn't show worker stalls during transient endpoint outages.Follow-up worth doing later (out of scope)
*ErrServerErrorfrom the same node within window W, markDisabledfor a short cooldown so we stop hitting it altogether. Adds resilience under partial-infra outages without changing the routing or retry shape. Estimated ~50-line addition to MultiClient.