Skip to content

feat(api): retry transient chain-RPC failures + fail loud in provider fan-out - #42

Merged
nimrod-teich merged 4 commits into
mainfrom
feat/rpc-retry-fail-loud
Apr 21, 2026
Merged

nimrod-teich merged 4 commits into
mainfrom
feat/rpc-retry-fail-loud

Conversation

@nimrod-teich

Copy link
Copy Markdown
Contributor

Summary

Fixes the "8 providers silently missing from historical rewards responses" observation from the latest comparison. Two layered fixes:

  1. Retry transient chain-RPC failures in `fetchRest`
  2. Fail loud in the provider fan-out — drop `.catch(() => [])` in `fetchProvidersWithSpecs` and `fetchAllProvidersImpl`

The bug

Against block 4697952 the latest deployed code gave us 40/48 providers (byte-exact match on all 40 — the historical-pricing + IMMUTABLE-cache work is landed correctly). The 8 missing providers are all in chain's pairing list at that block; I verified by iterating the 66 specs sequentially. The 25-wide concurrent fan-out against the public archive node occasionally drops a batch (timeout, 429, transient network), and `.catch(() => [])` in `providers.ts` silently swallows the error — so that spec's providers vanish from the response, which then gets cached for a year under `CACHE_TTL.IMMUTABLE`.

Fix 1 — retry in `fetchRest`

New `fetchWithRetry` helper:

  • 3 attempts, exponential backoff capped at 8s per try
  • Honors `Retry-After` header on 429
  • Retries on 408, 429, 5xx, and network-level errors (timeout, connection reset)
  • Does NOT retry on 4xx (wrong path / bad params)
  • Request timeout bumped 15s → 20s to tolerate archive-node slow paths

Fix 2 — fail loud

`fetchProvidersWithSpecs` and `fetchAllProvidersImpl` no longer wrap their `Promise.all` entries in `.catch(() => [])`. With the retries in place, the remaining error modes are genuine (not transient) and we want Fastify to 500 rather than return a silently-incomplete response. The cache plugin already skips 4xx/5xx so a failed request can't poison Redis.

Not touched

Per follow-up, `RPC_BATCH_SIZE` stays at 25. The retries handle the concurrency pressure; lowering the batch would slow cold queries unnecessarily.

Test plan

  • `pnpm --filter @info/api typecheck` clean
  • `pnpm --filter @info/api test` — 114 passing (no test changes needed; existing tests don't mock the retry path)
  • Deploy + re-run the rewards-breakdown comparison against block 4697952. Expected: 48/48 providers, byte-exact USD match on all of them.

…der fan-out

Two layered fixes for the "8 providers silently missing from historical
rewards responses" bug: fetchRest now retries transient failures with
exponential backoff, and fetchProvidersWithSpecs / fetchAllProvidersImpl
no longer swallow errors via .catch(() => []).

Before:
  • 25-wide concurrent chain-RPC fan-out against the public archive node
    occasionally triggered rate-limits / slow paths / connection drops
  • Errors were caught with `.catch(() => [])`, dropping an entire spec's
    providers without any signal to the caller or in logs
  • A historical /provider-estimated-rewards?block=N response could be
    missing 8-20% of providers vs jsinfo; the wrong response would then
    be cached for a year under CACHE_TTL.IMMUTABLE

After:
  • fetchRest wraps its fetch() in fetchWithRetry:
      - Retries on 408, 429 (honors Retry-After), 5xx, network errors
      - Does NOT retry on 4xx (wrong path, retrying won't help)
      - 3 attempts, exponential backoff capped at 8s
      - Request timeout bumped 15s → 20s (archive-node slow paths)
  • fetchProvidersWithSpecs and fetchAllProvidersImpl drop `.catch(...)`:
    if a spec fetch fails after retries, the whole request throws.
    Fastify returns 500, cache plugin skips 4xx/5xx so no bad response
    gets cached. Next caller retries from scratch.

Verified against block 4697952: all 40 common providers match jsinfo
byte-exact. Remaining 8 will come in after this ships (retries will
recover the transient drops that were previously silently lost).

114 tests passing.
@nimrod-teich
nimrod-teich force-pushed the feat/rpc-retry-fail-loud branch from fef46c9 to 21cf0b3 Compare April 20, 2026 13:04
Cold-fetch cost for /provider-estimated-rewards?block=N used to be 70-95s
and often 500'd on blocks older than a week. Two-part fix:

1. rest.ts: chain LB routes historical queries to non-archive replicas
   which return a 200 OK body with `{success:false, "version does not
   exist"}`. Previously that silently parsed as "empty rewards" and
   dropped providers from the response. Now:
    - classify body-level errors: "retry" (pruned state — version
      missing) vs "passthrough" (legitimate empty, e.g. provider not
      found) so we don't spin through retries on real empties.
    - set `lava-extension: archive` header on queries for blocks older
      than ~7 days (auto-detected via cached latest-height).
    - retry on a fresh no-keepalive undici dispatcher so each attempt
      breaks out of CF's socket-level replica pinning and gets a fresh
      routing decision.
    - 10 attempts with jittered backoff; Connection: close after first.

2. provider-estimated-rewards.ts: swap the 25-30s of per-spec stake-list
   chain RPCs for a single indexer GraphQL query.
    - address list comes from indexer `app.providers` (every provider
      ever paid a relay — superset of any address with rewards at a
      historical block).
    - specNames from fetchAllSpecs (block-agnostic, cached).
    - monikers from fetchAllProviderMonikers (currently-staked; unstaked
      ghosts render as "-" in the FE).
    - LAVA historical price fetch kicked off in parallel with the chain
      fan-out instead of serially after it. Non-LAVA denoms (rare) are
      fetched lazily if they appear in the raw rewards.

Verified against jsinfo: 50/50 providers, 0 USD mismatches at block
4895283. Cold fetch 70-95s → ~54s. The remaining cost is almost entirely
the per-provider estimated_provider_rewards fan-out, which can only be
eliminated by a lava-indexer snapshot job.
Route no longer calls fetchProvidersWithSpecs / fetchAllProviders —
address list comes from the indexer's app.providers table via gqlSafe.
Tests now mock fetchAllSpecs + fetchAllProviderMonikers + gqlSafe.

Also split the historical-prices assertion into two calls: LAVA upfront
(kicked off in parallel with the chain fan-out) + any non-LAVA denoms
that actually appear after fan-out.

Replaces the "snapshots the provider set at the historical block" test
with one asserting that indexer ghost providers (unstaked but historically
active) still get queried.
…ockTime through fetchRest

Two related correctness bugs for historical /provider-estimated-rewards:

1. Back-to-back cold fetches of the same block produced different
   provider counts and total_usd (32 / \$35k vs 33 / \$45k for block
   3928945). Root cause: a bare "collections: not found" can come from
   two places:
    - legit: provider has no accrued rewards at this block
    - degraded replica: partially-pruned state can't look up the key
   classifyChainUpstreamError used to treat both as passthrough,
   silently dropping providers from the response under cold-fetch LB
   variance. Now only an explicit
   "cannot get claimable rewards after distribution" marker qualifies
   as passthrough — everything else retries. Degraded replicas now
   force a fail-loud 5xx (which the cache plugin skips), matching the
   same "never cache incomplete data" discipline as the LAVA price
   503 path.

2. fetchBlockTime hit tendermint-RPC directly (/block?height=N) which
   gateway-times-out on pruned replicas and returns
   {result:{success:false, message:"gateway_timeout: ..."}} — crashing
   our caller with "Cannot read properties of undefined (reading
   'header')" at block.ts:21. Switched to the cosmos-REST equivalent
   /cosmos/base/tendermint/v1beta1/blocks/{height} so the call goes
   through fetchRest and inherits archive-header routing + retries.

Verified: block 4895283 still returns 50 providers / \$13,199.08 in 54s.
Block 4697952 (1 month old) now 5xx'es cleanly after exhausting retries
against pruned replicas, instead of serving a partial 200.
@nimrod-teich
nimrod-teich merged commit f9f5e5a into main Apr 21, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant