Repository navigation
feat(api): retry transient chain-RPC failures + fail loud in provider fan-out - #42
Merged
Merged
Conversation
…der fan-out
Two layered fixes for the "8 providers silently missing from historical
rewards responses" bug: fetchRest now retries transient failures with
exponential backoff, and fetchProvidersWithSpecs / fetchAllProvidersImpl
no longer swallow errors via .catch(() => []).
Before:
• 25-wide concurrent chain-RPC fan-out against the public archive node
occasionally triggered rate-limits / slow paths / connection drops
• Errors were caught with `.catch(() => [])`, dropping an entire spec's
providers without any signal to the caller or in logs
• A historical /provider-estimated-rewards?block=N response could be
missing 8-20% of providers vs jsinfo; the wrong response would then
be cached for a year under CACHE_TTL.IMMUTABLE
After:
• fetchRest wraps its fetch() in fetchWithRetry:
- Retries on 408, 429 (honors Retry-After), 5xx, network errors
- Does NOT retry on 4xx (wrong path, retrying won't help)
- 3 attempts, exponential backoff capped at 8s
- Request timeout bumped 15s → 20s (archive-node slow paths)
• fetchProvidersWithSpecs and fetchAllProvidersImpl drop `.catch(...)`:
if a spec fetch fails after retries, the whole request throws.
Fastify returns 500, cache plugin skips 4xx/5xx so no bad response
gets cached. Next caller retries from scratch.
Verified against block 4697952: all 40 common providers match jsinfo
byte-exact. Remaining 8 will come in after this ships (retries will
recover the transient drops that were previously silently lost).
114 tests passing.
nimrod-teich
force-pushed
the
feat/rpc-retry-fail-loud
branch
from
April 20, 2026 13:04
fef46c9 to
21cf0b3
Compare
Cold-fetch cost for /provider-estimated-rewards?block=N used to be 70-95s
and often 500'd on blocks older than a week. Two-part fix:
1. rest.ts: chain LB routes historical queries to non-archive replicas
which return a 200 OK body with `{success:false, "version does not
exist"}`. Previously that silently parsed as "empty rewards" and
dropped providers from the response. Now:
- classify body-level errors: "retry" (pruned state — version
missing) vs "passthrough" (legitimate empty, e.g. provider not
found) so we don't spin through retries on real empties.
- set `lava-extension: archive` header on queries for blocks older
than ~7 days (auto-detected via cached latest-height).
- retry on a fresh no-keepalive undici dispatcher so each attempt
breaks out of CF's socket-level replica pinning and gets a fresh
routing decision.
- 10 attempts with jittered backoff; Connection: close after first.
2. provider-estimated-rewards.ts: swap the 25-30s of per-spec stake-list
chain RPCs for a single indexer GraphQL query.
- address list comes from indexer `app.providers` (every provider
ever paid a relay — superset of any address with rewards at a
historical block).
- specNames from fetchAllSpecs (block-agnostic, cached).
- monikers from fetchAllProviderMonikers (currently-staked; unstaked
ghosts render as "-" in the FE).
- LAVA historical price fetch kicked off in parallel with the chain
fan-out instead of serially after it. Non-LAVA denoms (rare) are
fetched lazily if they appear in the raw rewards.
Verified against jsinfo: 50/50 providers, 0 USD mismatches at block
4895283. Cold fetch 70-95s → ~54s. The remaining cost is almost entirely
the per-provider estimated_provider_rewards fan-out, which can only be
eliminated by a lava-indexer snapshot job.
Route no longer calls fetchProvidersWithSpecs / fetchAllProviders — address list comes from the indexer's app.providers table via gqlSafe. Tests now mock fetchAllSpecs + fetchAllProviderMonikers + gqlSafe. Also split the historical-prices assertion into two calls: LAVA upfront (kicked off in parallel with the chain fan-out) + any non-LAVA denoms that actually appear after fan-out. Replaces the "snapshots the provider set at the historical block" test with one asserting that indexer ghost providers (unstaked but historically active) still get queried.
…ockTime through fetchRest
Two related correctness bugs for historical /provider-estimated-rewards:
1. Back-to-back cold fetches of the same block produced different
provider counts and total_usd (32 / \$35k vs 33 / \$45k for block
3928945). Root cause: a bare "collections: not found" can come from
two places:
- legit: provider has no accrued rewards at this block
- degraded replica: partially-pruned state can't look up the key
classifyChainUpstreamError used to treat both as passthrough,
silently dropping providers from the response under cold-fetch LB
variance. Now only an explicit
"cannot get claimable rewards after distribution" marker qualifies
as passthrough — everything else retries. Degraded replicas now
force a fail-loud 5xx (which the cache plugin skips), matching the
same "never cache incomplete data" discipline as the LAVA price
503 path.
2. fetchBlockTime hit tendermint-RPC directly (/block?height=N) which
gateway-times-out on pruned replicas and returns
{result:{success:false, message:"gateway_timeout: ..."}} — crashing
our caller with "Cannot read properties of undefined (reading
'header')" at block.ts:21. Switched to the cosmos-REST equivalent
/cosmos/base/tendermint/v1beta1/blocks/{height} so the call goes
through fetchRest and inherits archive-header routing + retries.
Verified: block 4895283 still returns 50 providers / \$13,199.08 in 54s.
Block 4697952 (1 month old) now 5xx'es cleanly after exhausting retries
against pruned replicas, instead of serving a partial 200.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes the "8 providers silently missing from historical rewards responses" observation from the latest comparison. Two layered fixes:
The bug
Against block 4697952 the latest deployed code gave us 40/48 providers (byte-exact match on all 40 — the historical-pricing + IMMUTABLE-cache work is landed correctly). The 8 missing providers are all in chain's pairing list at that block; I verified by iterating the 66 specs sequentially. The 25-wide concurrent fan-out against the public archive node occasionally drops a batch (timeout, 429, transient network), and `.catch(() => [])` in `providers.ts` silently swallows the error — so that spec's providers vanish from the response, which then gets cached for a year under `CACHE_TTL.IMMUTABLE`.
Fix 1 — retry in `fetchRest`
New `fetchWithRetry` helper:
Fix 2 — fail loud
`fetchProvidersWithSpecs` and `fetchAllProvidersImpl` no longer wrap their `Promise.all` entries in `.catch(() => [])`. With the retries in place, the remaining error modes are genuine (not transient) and we want Fastify to 500 rather than return a silently-incomplete response. The cache plugin already skips 4xx/5xx so a failed request can't poison Redis.
Not touched
Per follow-up, `RPC_BATCH_SIZE` stays at 25. The retries handle the concurrency pressure; lowering the batch would slow cold queries unnecessarily.
Test plan