Skip to content

fix(rpc): decay adaptive backoff on successful RPC calls so it resets after transient failures - #16

Open
mpsc0x wants to merge 1 commit into
masterfrom
fix/mpsc0x/adaptive-backoff-reset
Open

fix(rpc): decay adaptive backoff on successful RPC calls so it resets after transient failures#16
mpsc0x wants to merge 1 commit into
masterfrom
fix/mpsc0x/adaptive-backoff-reset

Conversation

@mpsc0x

@mpsc0x mpsc0x commented Jul 20, 2026

Copy link
Copy Markdown

Summary

Fixes the production issue where a single failed RPC call permanently degraded the indexer: the adaptive backoff got triggered and never reset, so every subsequent RPC request across all networks was delayed until the pod was restarted.

Cause

The RPC transport layer (layer_extensions.rs) is asymmetric:

  • On a throttle/unavailability error (429/503/-32001) it calls record_rate_limit() - growing the global backoff (500ms → 30s, doubling) and halving concurrency/batch size. It sits on every RPC method.
  • On success it recorded nothing. The only record_success() call sites were the historic-sync batch paths (fetch_logs parallel workers, tables.rs), which a caught-up live indexer never hits.

So in steady-state live indexing, one transient blip ratcheted the backoff up with no code path that could ever wind it back down. The pre-request wait in the same layer then delayed every call indefinitely. (Restarting the pod "fixed" it because the controller state is in-memory.)

Changes

  • layer_extensions.rs: call ADAPTIVE_CONCURRENCY.record_success() in the layer's success path, making it symmetric with the failure path. Backoff now decays 25% per successful call down to exactly 0, and concurrency/batch scale back up after sustained success - so the controller self-heals once the provider recovers.
  • Guard against decaying mid-outage: a JSON-RPC error payload delivered over a successful HTTP call (e.g. Alchemy's -32001 over HTTP 200 - the exact shape from the original incident) also lands in the transport's Ok branch. New has_throttle_error_payload() checks the response packet (code + message; data is excluded so hex/block numbers can't false-positive tokens like "429") and skips record_success for those.
  • Non-throttle error payloads (e.g. "block range too large") still count as provider-alive: rindexer's fetch-large-then-shrink range probing produces these routinely and they must not block recovery.

Notes

  • Failure still doubles the backoff while success decays it 25%, so during a real outage (mostly failing calls) the backoff still grows; it only drains on genuine recovery.
  • The controller is global (shared across networks) - unchanged; both growth and decay now operate on the same scope.
  • The layer sits under alloy's RetryBackoffLayer, so each retry attempt is observedindividually - one success records exactly one decay.

@mpsc0x mpsc0x self-assigned this Jul 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant