Skip to content

Consolidate retry layers; fast-fail 5xx for instant failover - #9

Merged
nimrod-teich merged 1 commit into
mainfrom
feat/consolidate-retry-layers
Apr 19, 2026
Merged

nimrod-teich merged 1 commit into
mainfrom
feat/consolidate-retry-layers

Conversation

@nimrod-teich

Copy link
Copy Markdown
Contributor

Summary

Audit found three retry budgets stacked on top of each other in the request path, plus 5xx in-place retries that delayed endpoint failover. A single bad block could amplify into ~600 HTTP requests in absolute worst case (~30+ in realistic flaky scenarios) before the dead-letter sweep gave up.

This PR collapses them into one retry budget per layer, each with a distinct job, and makes 5xx fail-over immediately instead of burning four in-place retries against a sick node.

Changes

internal/rpc/rpc.go, internal/rpc/rest.go

  • Replace maxRetries=4 with httpRetryNetwork=1. Network errors only — covers the keep-alive-closed race.
  • New typed error *ErrServerError for 5xx, mirroring the existing *ErrRateLimited for 429. Both fail-fast so MultiClient can route to a different node on the next iteration.
  • Updated doc comment on RPCClient.call to lay out the policy explicitly.

internal/rpc/multi.go

  • FetchBlocks failover loop now logs endpoint server-error, failing over at Debug for the new error class, matching the 429 path.
  • Updated docstring: this is now the authoritative "I tried everyone" loop — no outer wrapper above it.

internal/pipeline/pipeline.go

  • Delete fetchWithRetry. Replaced its caller with a direct p.client.FetchBlocks(ctx, heights) call.
  • The outer 4× retry was just re-hitting the same exhausted set and amplifying every transient blip. Bisect + dead-letter sweep cover the remaining recovery cases at the right scale.

internal/rpc/rpc_test.go

  • TestRPCClient_call_5xxFailsFastNoRetry: server returns 500; assert *ErrServerError returned on the FIRST hit, no in-place retry.
  • TestRPCClient_call_NetworkErrRetriesOnce: torn connection on attempt 1 (Hijacker close); assert recovery on attempt 2 with exactly 2 hits.

README.md

  • "Retry, failover, and dead-letter" section rewritten to document:
    • The full layer map (runWindow → indexGaps → worker → processWithSplit → processBatch → MultiClient → retryableCall → http.Client).
    • Per-layer retry budgets in a table.
    • Why we don't retry the whole MultiClient call.
    • Worst-case amplification math.
    • The fast-fail rule for 5xx/429.

Worst-case math (after this PR)

1 bad block × log₂(batch_size) bisect levels
            × N endpoints
            × 1 same-node retry on net err
            × failure_max_retries dead-letter cycles

At batch_size=20, N=2, failure_max_retries=3: 30 requests. Down from ~600 in absolute worst case before, and ~30+ in realistic flaky scenarios.

Behaviour intentionally unchanged

  • AIMD per-endpoint budget controller.
  • Bisect on multi-block transient failure.
  • Dead-letter sweep cadence + cap.
  • Permanent-error short-circuit (HeightPrunedError, NoEndpointCoversError).
  • Healthy/success paths — same code path, same call count.

Test plan

  • go build ./... clean.
  • go vet ./... clean.
  • go test ./... green (full suite).
  • New tests verify 5xx fast-fail and network-err one-retry behaviour.
  • Deploy to mainnet+testnet, observe fetch failed, retrying log lines drop to ~0 (replaced by endpoint server-error, failing over at Debug, only visible when --log.level=debug).
  • Verify pg_stat_activity doesn't show worker stalls during transient endpoint outages.

Follow-up worth doing later (out of scope)

  • Endpoint cooldown circuit breaker: after K consecutive *ErrServerError from the same node within window W, mark Disabled for a short cooldown so we stop hitting it altogether. Adds resilience under partial-infra outages without changing the routing or retry shape. Estimated ~50-line addition to MultiClient.

The request path had three retry budgets stacked on top of each other:
retryableCall (4× per-node), MultiClient.FetchBlocks (failover across
N endpoints), and fetchWithRetry (4× outer). Combined with bisect and
the dead-letter sweep, a single bad block could amplify into ~600 HTTP
requests in absolute worst case — and ~30+ in realistic flaky-endpoint
scenarios. The 5xx in-place retries also delayed failover by ~3.5s
against a sick node before MultiClient even got to try a different one.

Each layer now has one job:

- retryableCall: only retries network errors, once. Covers the
  keep-alive-closed race; if it doesn't help in one retry it isn't
  going to. 5xx returns *ErrServerError immediately so MultiClient
  fails over to a different node — same fast-fail shape that 429
  (*ErrRateLimited) already uses, for the same reason.
- MultiClient.FetchBlocks: AUTHORITATIVE failover loop. Walks every
  eligible endpoint headroom-sorted; logs server-error failover at
  Debug level (matching the existing 429 path).
- fetchWithRetry: deleted. processBatch now calls
  p.client.FetchBlocks directly. The outer 4× wrapper just re-hit
  the same exhausted endpoint set and amplified every transient blip.
- processWithSplit (bisect): unchanged — still isolates one bad
  block from a healthy batch.
- Dead-letter sweep: unchanged — slow-time recovery for whole-infra
  outages, capped by failure_max_retries.

New worst case: 5 bisect levels × 2 endpoints × 1 in-place retry × 3
sweep cycles = 30 requests. ~20× reduction. Healthy blocks stay at one
request.

Tests added:
- TestRPCClient_call_5xxFailsFastNoRetry: server returns 500;
  *ErrServerError surfaces on the first hit, no in-place retry.
- TestRPCClient_call_NetworkErrRetriesOnce: torn connection on the
  first attempt is recovered by the second; no further retries.

README "Retry, failover, and dead-letter" rewritten to document the
layer map, per-layer retry budgets, the fast-fail rule for 5xx/429,
and the worst-case amplification calculation.
@nimrod-teich
nimrod-teich merged commit cddb918 into main Apr 19, 2026
1 check passed
@nimrod-teich
nimrod-teich deleted the feat/consolidate-retry-layers branch April 19, 2026 08:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant