Repository navigation
feat(snapshotters): provider_rewards + denom_prices + supply + priced_rewards MV - #15
Merged
Merged
Conversation
New ingestion mechanism: monthly-17th@15:00-UTC snapshots of
estimated_provider_rewards for every known provider, stored in
Postgres so downstream consumers can query historical rewards via
GraphQL instead of fanning out to chain RPC at request time.
## Architecture
Added a sibling Snapshotter seam (internal/snapshotters/) parallel to
the existing event-driven Handler, since estimated_provider_rewards
is chain state not events. Snapshotter interface mirrors Handler's
shape: Name + DDL + BlocksDue + Snapshot (tx-scoped).
Registry.RunLoop ticks on a time.Ticker (default 10m), runs once
immediately on startup for backfill, and isolates each target in its
own pgx.Tx so one failure doesn't starve others. Jittered sleep
between targets to avoid bursting chain RPC.
## Provider rewards snapshotter
Concrete snapshotter calls
GET {rest}/lavanet/lava/subscription/estimated_provider_rewards/{addr}
Headers: x-cosmos-block-height={block}, lava-extension=archive
at monthly-17th@15:00UTC block heights resolved via binary search
against /cosmos/base/tendermint/v1beta1/blocks/{h}.
Response classification:
- 200 + info[] → parse "Boost: ETH1" → source_kind (0/1/2) + spec,
upsert into app.providers / app.chains via Dict, insert rows.
- success:false + "version does not exist" / "version mismatch" /
"no commit info found" → replica pruned, retry with jittered
exp-backoff up to 10 attempts.
- success:false + "cannot get claimable rewards after distribution"
→ legit empty, treat as empty info[].
- Any other success:false → conservative retry.
All of (insert snapshot row + provider_rewards rows) happens in the
passed pgx.Tx; a single failed provider rolls the whole snapshot
back. Next tick retries from scratch.
## Schema
Two new tables using raw on-chain amounts (NUMERIC(40,0) base units,
no USD conversion — reader's job):
app.provider_rewards_snapshots (block_height PK, snapshot_date UNIQUE,
block_time, provider_count, status)
app.provider_rewards (block_height FK, provider_id FK, spec_id FK,
source_kind, denom, amount, composite PK)
Reuses app.providers + app.chains via the existing Dict helper.
## Web UI
GET /api/snapshotters returns expected/covered/missing/failed dates
per registered snapshotter. A new "Snapshotters" card renders a
timeline strip alongside the existing handler card — different shape
(discrete points vs contiguous ranges), so no attempt to unify.
## Config toggle
snapshotters.provider_rewards.enabled=false fully disables: no
goroutine spawned, no DDL applied, no log lines. Verified by the
errgroup guard on len(snapReg.All()).
Two fixes the integration tests surfaced:
1. The `estimated_provider_rewards` endpoint takes two path segments
(provider, amount_delegator), not one. Calling the one-arg form
returns 501 Not Implemented from every Lava LCD gateway. Use
`/lavanet/lava/pairing/estimated_provider_rewards/{provider}/1ulava`
— the minimum valid coin, which gets us the per-provider rewards
breakdown without requiring a real delegation.
2. 501/404 now route into the pruned-retry loop with structured
httpStatusErr. Lava's public gateway load-balances across replicas
and the rewards handler isn't always registered on every replica;
a fresh connection usually rotates to a working one. Also added
"not implemented" to the JSON-level classifyChainMessage branches
for the same reason.
3. Snapshotter selection now mirrors the handler pattern exactly:
`snapshotters.list: ["all"]` (or empty) enables every compiled-in
snapshotter; explicit names turn on just those. Dropped the
per-snapshotter `enabled: bool` flag. Env override is
`SNAPSHOTTERS=<comma-separated>`, matching `INDEXER_HANDLERS`.
Rename snapshotters.list → snapshotters.handlers so the yaml key name reads identically to indexer.handlers (both are string-lists with the same "all" semantics). Default is now ["all"] explicitly (was nil). Env override is SNAPSHOTTERS_HANDLERS (was SNAPSHOTTERS), matching INDEXER_HANDLERS.
The snapshotter's REST endpoint is separate from network.endpoints
(it's typically an archive LCD, not a tendermint RPC), so it was
invisible to the dashboard. Surface it via /api/snapshotters as
rest_url and render it on the snapshotter card meta line.
Exposed via a local interface{ RESTURL() string } rather than adding
to the Snapshotter interface proper, so the core interface stays
focused on DDL + schedule + snapshot. Other snapshotters can opt in
by implementing the same method.
Fixes the 1 critical + 4 highs from _workspace/report.md on this branch: - [critical] Long-held tx across chain fan-out: Snapshotter.Snapshot now takes *pgxpool.Pool, not pgx.Tx. Each snapshotter owns its own tx boundary. provider_rewards.Snapshot does HTTP fan-out OUTSIDE any tx; opens a short tx for dict ID resolution; opens a second short tx for the snapshot row + CopyFrom. Connection held for DB work only — single-digit seconds instead of minutes. - [high] fetchAll no sibling cancel: replaced the jobs/replies channel pattern with errgroup.WithContext + g.SetLimit. First error cancels siblings so a doomed snapshot doesn't run the remaining ~2000 retries. - [high] HTTPCaller uses http.DefaultTransport: now uses rpc.SharedTransport (HTTP/2, 256 idle conns per host, TLS session cache), same as the main pipeline's RPC clients. - [high] doGET doesn't drain body on non-2xx: read body via io.ReadAll(io.LimitReader(..., 16 MiB)) BEFORE checking status. Drains on all paths so net/http can return the conn to keep-alive pool. Also caps body size to bound a compromised upstream. - [high] FK coupling to relay_payment dict tables: provider_rewards .DDL() now includes providers + chains CREATE IF NOT EXISTS so the snapshotter can start standalone. Idempotent — harmless when relay_payment is also registered. - [high] Web concretely imports provider_rewards: added Status(ctx, pool) to Snapshotter interface; provider_rewards implements it with its own schema query + ExpectedDates. web drops the concrete import and the name-match dispatch. Also dropped redundant Server.SnapshotterSchema (duplicated State.Schema()).
Diagnosed the "hours with zero snapshots" problem: the chain returns
HTTP 501 Not Implemented consistently at historical blocks where
EstimatedProviderRewards wasn't yet registered, and intermittently
even on recent blocks because Lava's public LB rotates through
replicas that haven't all picked up the handler. With fail-on-first-
error, one bad provider killed every snapshot.
Primary fix: per-provider tolerance.
- fetchAll now collects all results (entries or per-provider err)
instead of failing on the first error. Per-provider errors do not
cancel siblings — only outer ctx cancel aborts the fan-out.
- Snapshot computes success_fraction and persists:
* status='ok' with the successful subset when >=80% succeeded
(the cluster the operator can actually analyse);
* status='failed' with the error message when <80% succeeded
(so BlocksDue stops re-retrying this date forever).
- coveredDates now excludes both 'ok' and 'failed' statuses, so
BlocksDue skips failed dates too — operators retry by deleting the
failed row manually.
Also classify empty-body and malformed-JSON responses as pruned-retry
(they were bubbling up as terminal parse errors, which is why some
dates failed with zero retries).
Medium + low review findings rolled into this PR:
- [medium] snapshotters.list → snapshotters.handlers doc refs in
main.go, config.go, README.md updated (code was already renamed).
- [medium] full-jitter backoff (was narrow ±50% band → clumped 25
workers onto the same retry window).
- [medium] Registry.LastTickErr() accessor now exposes the lastTickErr
the field was carrying — used to be set but never read.
- [low] URLSurface named interface in snapshotters/ (was a local
type-assert in web/).
- [low] redactURLUserinfo before surfacing rest_url on the dashboard
(basic-auth creds would otherwise leak via unauthenticated
/api/snapshotters).
- [low] url.PathEscape for the provider addr in the REST path.
- [low] strconv.FormatInt / strconv.ParseInt replacing hand-rolled
digit loops in doGET and Tip.
- [low] time.NewTimer + Stop instead of time.After in the registry's
per-target jitter sleep (no leaked timers on ctx cancel).
- gofmt -w on the new files.
…lava
Per operator: the canonical endpoint is
/lavanet/lava/subscription/estimated_provider_rewards/{addr}/10000000000ulava
not the `pairing` module, and the amount_delegator must be
10^10 ulava (= 10 LAVA), not 1ulava. Earlier trial-and-error against
lava.rest.lava.build's LB was misleading because replicas rotate and
inconsistently register the handler — empirical probes returned 501
for both paths, but the subscription path is the authoritative one
downstream consumers expect.
Also classify "cannot estimate rewards for delegator/delegation" as
empty (no accrued rewards), same as "cannot get claimable rewards
after distribution". Observed in the wild on providers whose
delegation state isn't present at the queried height — deterministic
per provider, no point retrying.
Per operator feedback: when iterating across every provider ever
seen, most will legitimately have NO rewards at a historical block
(not staked yet, unstaked, no delegation state). The chain returns
success=false with various messages for these, and the previous
classification treated them the same as real fetch failures (HTTP
5xx, decode errors) — causing historical dates to falsely show as
"failed" when most providers simply weren't applicable.
New classification rules:
- Only replica/version issues retry as pruned:
"version does not exist" / "version mismatch" /
"no commit info found" / "not implemented"
- Every other success=false is chain-authoritative "no rewards for
this (provider, block) pair" — treated as outcomeNoRewards, does
NOT count toward the failed-snapshot threshold.
- Only HTTP 5xx (after retry exhaustion), empty-body / decode errors
(after retry), or exhausted pruned-retry count toward outcomeFailed.
Snapshot log line now breaks down fetched / no_rewards / failed so
operators can see what each snapshot actually did, e.g.:
total=276 fetched=48 no_rewards=220 failed=8
Threshold check now uses (fetched + no_rewards) / total >= 0.8 so a
date where most providers weren't staked yet still lands as
status='ok' with only the successful subset written to
provider_rewards. Only dates where the upstream is genuinely broken
get marked status='failed'.
Also: UI timeline dots now show the pinned block height in the
tooltip alongside the date (per operator request), so each circle
identifies the exact chain block the snapshot was taken at.
SnapshotterStatus now carries a blocks map; the dashboard renders
"{date} · block {N}" on hover.
Two follow-up fixes from iteration:
1. Reference time is now the chain genesis timestamp, not a
hardcoded monthly-17th@15:00 UTC. Slot N = genesis + N months,
preserving genesis hour/minute/second. For Lava mainnet (genesis
2024-01-17T15:00:00Z) this reproduces the existing cadence without
magic numbers; for any other chain the anchor is correct by
construction.
ExpectedDates signature: (genesis, earliest, now). earliest_date
config is now just a floor — no longer constrained to a 17th,
since the day is derived from genesis.
2. New outcomeStatePruned distinguishes "archive pruned state at this
height" from real fetch failures. When EstimatedRewards exhausts
its pruned-retries AND the final error is a state-availability
issue (version does not exist / pruned / 501 from a rotated
replica), we wrap with errStatePruned instead of letting it bubble
as outcomeFailed.
The snapshot threshold now applies ONLY to outcomeFailed — archive
unavailability is a chain-infrastructure limitation, not something
retries fix, so dates where most providers come back as
statePruned can still land as status='ok' with the subset we DID
get.
Progress log now shows the full breakdown:
fetched=N no_rewards=N state_pruned=N failed=N
Verified that `lava-extension: archive` header IS being sent (caller
loops over configured headers), but the public `lava.rest.lava.build`
gateway's archive backing only retains state ~2 months deep, so the
header has no effect on historical dates. That's a gateway-
infrastructure constraint, not fixable here.
Chains that forked or were renamed still report the upstream origin's timestamp via /genesis — e.g. lava-testnet-2 reports 2022-12-26 (the testnet-1 origin) despite its operational start being 2023-08-17 10:02 UTC. The snapshot anchor needs to be the real operational start so the monthly cadence lands on the intended day-of-month. snapshot_anchor is an RFC3339 override; empty = fall back to client.GenesisTime(). Lava mainnet leaves it empty (its /genesis reports the correct 2024-01-17T15:00 start); testnet sets "2023-08-17T10:02:00Z".
The floor was silently masking all pre-2025 slots on mainnet + all pre-2025 slots on testnet, producing fewer timeline dots than expected. Making it optional lets the genesis-anchored cadence surface every slot from genesis+1mo through now by default.
New internal/denoms package derives app.priced_denoms from
app.provider_rewards on a 5m ticker. Seeds 23 known Cosmos token
mappings (base_denom → coingecko_id) at startup via ON CONFLICT DO
UPDATE. New denoms land with coingecko_id=NULL — operators fill
via one SQL UPDATE, no redeploy.
Resolver handles:
- microdenom strip (ulava → lava, uatom → atom)
- IBC trace lookup against REST /ibc/apps/transfer/v1/denom_traces/{hash}
with in-process cache (traces are immutable once issued)
- transient vs permanent classification so transient chain errors
retry next tick, persistent ones (404 on a trace) don't
Deriver exposes an in-memory Status() snapshot the web layer reads
via /api/denoms. Dashboard renders a "Priced denoms" card with
mapped/unmapped counts and an expandable list of denoms needing a
coingecko_id — actionable signal for operators.
Config: denoms.deriver.enabled/interval/resolve_timeout/rest_url/
rest_headers. REST URL inherits from snapshotters.provider_rewards
if not set (same archive-backed endpoint).
Diagnosed "27 circles green, zero fact rows, zero provider_count":
the chain's actual response shape for our simulated-delegation query
is
{"info":[], "total":[{"denom":"ulava","amount":"…"}]}
not
{"info":[{"source":"Boost: …", "amount":[…]}]}
as the spec assumed. info[] carries the source breakdown but the
chain only populates it for query shapes we aren't sending; total[]
has the aggregate amount.
Fix: parse both fields. Prefer info[] when present (authoritative
breakdown); fall back to a synthesised entry with SourceKind=3
(SourceTotal) and spec="" from total[] so the amount isn't dropped.
Downstream consumers differentiate total-aggregate rows from
per-source rows by source_kind.
Also persists the seeded enum gap — SourceTotal=3 extends the
existing 0=Boost / 1=Pools / 2=Subscription mapping; the DB column
is SMALLINT so no schema change.
…ice snapshotter
Replaces the standalone priced_denoms deriver with the right shape:
the denom is a Dict-managed table (parallel to providers/chains), IBC
hashes are resolved at snapshot-write time so the dict stores only
chain-canonical microdenoms (uatom, ulava), and a new snapshotter
populates historical USD prices via CoinGecko on the same monthly
cadence as rewards.
Schema (DROP + recreate, no prod data):
app.denoms — (id, denom) Dict pattern, resolved-form only
app.denom_metadata — (denom_id PK, base_denom, factor, coingecko_id,
suppress, first/last_seen_at). Seeded with the
24-token map ported verbatim from info's
DENOM_CONVERSIONS + DENOM_COINGECKO_ID.
app.provider_rewards — denom TEXT → denom_id INT + source_denom TEXT
(raw preserved for info's source_denom API field).
PK now includes source_denom so two IBC hashes
resolving to the same base don't collide.
app.denom_price_snapshots — one row per priced date. Parent of:
app.denom_prices — (snapshot_date, denom_id) PK, price_usd NUMERIC.
price_usd=0 is a legitimate "coin not listed /
no market_data on date" signal.
app.priced_rewards MV — rewards × metadata × prices JOIN with
display_amount + value_usd computed in SQL.
Refreshed CONCURRENTLY after every successful
denom_prices snapshot. This is the surface
info (and anything else) reads.
provider_rewards snapshotter:
- h.denoms *events.Dict added (third Dict alongside providers/chains).
- HTTPCaller.ResolveIBC(hash) — trace lookup via
/ibc/apps/transfer/v1/denom_traces/{hash}, in-process cache forever
(IBC traces are immutable). 404 → permanent sentinel, 5xx → transient
(provider retried). Raw ibc/ hashes never enter app.denoms.
- Suppressed denoms (denom_metadata.suppress=TRUE) are cached per tick
and dropped before Dict.IDs resolution — test denom blacklist now
editable via SQL instead of hardcoded in info.
denom_prices snapshotter (new internal/snapshotters/denom_prices):
- BlocksDue shares provider_rewards.ExpectedDates for the same
genesis-anchored monthly cadence; binary-searches block_height at
each slot via the shared HTTPCaller.
- CoinGecko /coins/{id}/history?date=DD-MM-YYYY, sequential per date,
LAVA first. 5-attempt retry with Retry-After + jittered exponential
backoff, 60s cap. LAVA failure → status='failed' (no point without
the anchor); non-LAVA failure → log + skip (best-effort). 404 and
no-market_data → persist price_usd=0.
- REFRESH MATERIALIZED VIEW CONCURRENTLY app.priced_rewards after
each successful write. Unique index on the MV enables CONCURRENTLY.
Deleted:
- internal/denoms/ (whole package — old Deriver pattern was wrong shape)
- /api/denoms handler + DenomDeriver field on web.Server
- "Priced denoms" dashboard card
- cfg.Denoms struct + DENOMS_DERIVER_* envs
Config added:
snapshotters:
denom_prices:
coingecko_api_url: "" # default: https://api.coingecko.com/api/v3
rest_headers: {} # e.g. x-cg-pro-api-key for Pro plan
On deploy: drops + recreates all snapshotter tables (CASCADE catches
the MV). Snapshotters re-run from genesis on the next tick.
Mainnet snapshots 2024-10 → 2026-03 were all landing as status='failed' because 36-52 of 118 providers returned HTTP 500 (not 501) per snapshot — the Lava LB routes historical queries across archive replicas with varying state depth, and the refusal layer returns 500/502/503/504 about as often as 501. Only 501 and 404 were routed into the pruned-retry path; everything else fell through to outcomeFailed, pushed the failure fraction past the 20% threshold, and caused the entire date's 70+ successful providers to be discarded. Fix: 5xx + 404 all route into the pruned-retry loop and exhaust to outcomeStatePruned on max attempts. The snapshot threshold only counts real fetch failures (non-route-able errors), so the 70 providers that DID succeed at each date now land as rows with status='ok'. Other 4xx (401, 403, 429) still fall through as real errors — those are authentication / rate-limit issues we should surface, not silence.
…sh from rewards side Two issues surfaced during deploy: 1. CoinGecko rate limiting: the snapshotter was fetching prices for EVERY mapped denom in denom_metadata (24 on a fresh seed), regardless of whether that denom appears in rewards at the snapshot block. Free-tier CoinGecko limits made this hit 5-attempt exhaustion on 15/27 mainnet dates, marking them status='failed' and preventing any MV refresh. Fix: selectDenomsForBlock joins provider_rewards + denom_metadata filtered on the target block_height. Mainnet observes 7 distinct denoms across 27 dates (vs the 24 from the seed) — a ~3x reduction in CoinGecko budget that eliminates the rate-limit pressure. 2. MV refresh only happened on denom_prices success. When every denom_prices snapshot failed (see #1), the priced_rewards MV stayed empty even though provider_rewards had 1615 rows populated. Fix: provider_rewards.Snapshot also REFRESH MATERIALIZED VIEW CONCURRENTLY on success. MV LEFT JOINs denom_prices, so rows appear with price_usd = NULL when the price snapshot hasn't landed yet — better than waiting behind a failing price snapshotter. Bonus: dates with provider_count=0 (legitimate empty, e.g. pre-rewards blocks) now short-circuit the price snapshot — the block has no denoms to price, record 0-count row, done. Was previously calling CoinGecko 24 times per empty date.
Agent's original DDL included DROP TABLE IF EXISTS app.provider_rewards CASCADE; DROP TABLE IF EXISTS app.provider_rewards_snapshots CASCADE; as a migration helper. Problem: DDL runs on EVERY startup, so every restart was wiping the accumulated snapshot data. Kept the priced_denoms drop (cleanup of the obsolete deriver table) but swapped the main tables to CREATE IF NOT EXISTS only. Schema migration has already run on every live environment; no future DROP needed.
3 of 4 tasks
Root cause of the 40× undercount on mainnet rewards: the Lava chain's
/estimated_provider_rewards endpoint has two semantically distinct
response shapes depending on the path:
/{addr}/ → actual accrued rewards per source
(info[] populated with "Boost: ARBITRUM",
"Pools: ETH1", "Subscription: COSMOSHUB"
entries, each with ulava amount)
/{addr}/<coin-amount> → simulated rewards for a delegation of
that size (info=[], total=[{ulava, N}])
I wired the indexer to the second shape (`/10000000000ulava`), believing
the arg was required. That returned a tiny synthetic total ~40× smaller
than actual accrual, and with no source attribution — which is why our
priced_rewards rows show ~$2 per provider vs prod's ~$99+, and why every
row's spec is empty (the `total[]` fallback I added as a coping
mechanism for empty info[] is what we were persisting).
Fix: match prod info's path verbatim
(apps/api/src/rpc/rewards.ts::fetchRawProviderRewards) — trailing slash,
no amount arg. `total[]` fallback stays in ParseEstimatedRewards as
defence in depth but should now rarely fire.
Memory updated to reflect the right path. Tests updated to mock the
corrected path.
Operator action required: wipe app.provider_rewards on mainnet so the
next tick re-snapshots with the correct values. The priced_rewards MV
will auto-refresh.
Third sibling snapshotter alongside provider_rewards and denom_prices.
Captures Lava's total ulava supply at each genesis-anchored monthly
slot; downstream consumers (info's /burn-rate) read the table via
GraphQL and compute burn = prev_supply - curr_supply per month,
replacing the ~2N chain-RPC calls they make today on every cold
request.
Schema:
app.supply_snapshots (snapshot_date PK, block_height, block_time,
snapshot_at, total_supply NUMERIC(40,0),
status, error)
Flow per Snapshot():
1. x-cosmos-block-height pinned REST call to
/cosmos/bank/v1beta1/supply?pagination.limit=10000
2. Extract `ulava` amount (missing ulava → status='failed' with
clear error — don't silently zero)
3. One-row INSERT; short tx; block-time lookup shares the
provider_rewards RESTCaller for pool + TLS reuse
BlocksDue shares provider_rewards.ExpectedDates for the same
genesis-anchored monthly cadence; subtracts dates already covered or
explicitly failed so operators can retry by deleting the failed row.
No UI changes — the dashboard iterates `Snapshotters.All()`
generically via the Snapshotter + URLSurface interfaces, so the
supply card appears automatically on /api/snapshotters.
Config: snapshotters.supply { rest_url, rest_headers, earliest_date }.
Env: SUPPLY_REST_URL. Falls back to the main REST endpoint when
rest_url is empty (same resolver pattern as the other snapshotters).
Build + vet + test green; 13 new tests cover happy path, missing
ulava, transient 5xx, bad JSON, bad amount string, large-supply
roundtrip, cache hit, interface pin, URL surface, DDL idempotency.
Bake the mainnet-useful floor into the code so new operators don't have to discover it: provider_rewards + denom_prices both default to 2024-11-17 (the earliest monthly-17th where the chain's estimated_provider_rewards handler returns populated info[]). Supply stays empty — bank-module queries work from genesis, so there's no reason to hide early samples. Also: - Add DenomPrices.EarliestDate + ParsedEarliestDate(); main.go now reads it for denom_prices wiring (was incorrectly sharing the provider_rewards field). - New DENOM_PRICES_EARLIEST_DATE env override. - config.example.yml: document the new default, list compiled-in handlers/snapshotters next to handlers: ["all"] so operators can see what "all" expands to without grepping.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Three monthly-17th-anchored snapshotters plus a consolidated materialized view.
app.denoms— Dict-managed (same pattern asapp.providers/app.chains). IBC hashes resolved at snapshot-write time so the dict only holds chain-canonical microdenoms.app.denom_metadata— operator-editable overlay (base_denom, factor, coingecko_id, suppress). Seeded with the 24 tokens from info'sDENOM_CONVERSIONS+DENOM_COINGECKO_ID.provider_rewardssnapshotter —estimated_provider_rewardsat each monthly-17th block. Uses the trailing-slash path (/{addr}/, noamount_delegator), matching info'sfetchRawProviderRewards, so the chain returns actual per-source accrual (info[] populated).denom_pricessnapshotter — CoinGecko/coins/{id}/historyper denom-appearing-in-rewards at the snapshot's block_height. Rate-limit-aware, LAVA-first, zero-result caching.supplysnapshotter —ulavatotal supply from/cosmos/bank/v1beta1/supply. Downstream (info's/burn-rate, burn-ui) reads and computes burn locally.app.priced_rewardsMV — joins rewards × metadata × prices. Refreshed CONCURRENTLY after every successful snapshot (both sides).Retry semantics
outcomeStatePruned, up to 10 pruned-retries, doesn't count against the failure threshold.success:falsewithversion does not exist/version mismatch/no commit info found/not implemented→ pruned retry.success:false→outcomeNoRewards(chain-authoritative "nothing to compute for this provider at this block").errgroup— per-provider errors never abort the whole snapshot; dates landokas long as the real-failure rate stays below threshold.Verification
go build ./...,go vet ./...,go test ./...clean$13,199.08total at block 4895283 — byte-identical toapi-info.lavapro.xyz/provider-estimated-rewards?block=4895283).Consumers
/provider-estimated-rewardsreadsallPricedRewards./burn-ratereadsallSupplySnapshots.Folded from former PR #16
The supply snapshotter was on a stacked PR (#16); closed and its single commit added here since it's the same shape as the other two snapshotters.