fix(rpc): decay adaptive backoff on successful RPC calls so it resets after transient failures - #16
Open
mpsc0x wants to merge 1 commit into
Open
fix(rpc): decay adaptive backoff on successful RPC calls so it resets after transient failures#16mpsc0x wants to merge 1 commit into
mpsc0x wants to merge 1 commit into
Conversation
… after transient failures
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes the production issue where a single failed RPC call permanently degraded the indexer: the adaptive backoff got triggered and never reset, so every subsequent RPC request across all networks was delayed until the pod was restarted.
Cause
The RPC transport layer (
layer_extensions.rs) is asymmetric:429/503/-32001) it callsrecord_rate_limit()- growing the global backoff (500ms → 30s, doubling) and halving concurrency/batch size. It sits on every RPC method.record_success()call sites were the historic-sync batch paths (fetch_logsparallel workers,tables.rs), which a caught-up live indexer never hits.So in steady-state live indexing, one transient blip ratcheted the backoff up with no code path that could ever wind it back down. The pre-request wait in the same layer then delayed every call indefinitely. (Restarting the pod "fixed" it because the controller state is in-memory.)
Changes
layer_extensions.rs: callADAPTIVE_CONCURRENCY.record_success()in the layer's success path, making it symmetric with the failure path. Backoff now decays 25% per successful call down to exactly 0, and concurrency/batch scale back up after sustained success - so the controller self-heals once the provider recovers.-32001over HTTP 200 - the exact shape from the original incident) also lands in the transport'sOkbranch. Newhas_throttle_error_payload()checks the response packet (code + message;datais excluded so hex/block numbers can't false-positive tokens like"429") and skipsrecord_successfor those.Notes
RetryBackoffLayer, so each retry attempt is observedindividually - one success records exactly one decay.