Repository navigation
feat: initial lava-indexer - #1
Merged
Merged
Conversation
nimrod-teich
force-pushed
the
init
branch
4 times, most recently
from
April 18, 2026 11:31
d9c6f4e to
d1ca982
Compare
nimrod-teich
added a commit
that referenced
this pull request
Apr 19, 2026
Experimental round chasing a 3× throughput goal uncovered four real bottlenecks + one bug. All shipped here. 1. Writer pool was half the DB pool. pipeline.go: writerCount = pool_size / 2 → pool_size - 4. DB write was the bottleneck and we had half the pool deliberately unused. Reserving 4 connections for status handler + heartbeat + sweep + fetcher's HandlerNeedsRange still gives ~95% of pool to writers. Mainnet 53.6 → 70.6 bps (+32%) standalone. 2. DB pool was too small for the new worker count. Operator config bumped pool_size 32 → 80 (deploy-time, via .env). max_connections default is 100 in our postgres image; 80 indexer + ~10 postgraphile + ~5 internal fits comfortably. Writers went from saturated at pool_size to comfortably below. With #1 together: mainnet 53.6 → 208 bps (+289%), hit 3× target. 3. Adaptive batch sizer kept shrinking on tail-latency alone. sizer.go: drop the p99/p50 > 10 hot signal and the p99/p50 < 3 calm signal. For owned-infra operators, tail latency is a "slow but healthy" signal — routing handles it. ONLY error rate drives sizer shrink now. This matches the same decision made earlier in the AIMD-cap removal. pipeline.go: when operator sets fetch_batch_size > 0 explicitly, skip the sizer's batch state and use the static value. Previously the sizer ran whenever fetch_workers was adaptive (a separate signal), and its batch-state override silently ignored the operator's explicit batch-size config. With #1 + #2 + batch=50 static: mainnet 53.6 → 353 bps (+559%, observed up to 625 bps on log rolling average). 4. aggregates.yml: MV refresh schedule */5 → */30. On a 400M+ row relay_payments table, REFRESH MATERIALIZED VIEW CONCURRENTLY takes 5-10 min. At a 5-min schedule refreshes piled up and serialised on MV row-level locks, starving the writer pool. Dashboards read the MV for daily aggregates; 30 min freshness is acceptable and eliminates the lock contention. 5. Bug fix: processWithSplit treated per-request HTTP timeouts as fatal. pipeline.go: Go's http.Client.Timeout wraps context.Canceled when it fires on a slow upstream, so errors.Is(err, context.Canceled) returned true for BOTH "we were shut down" and "one request took too long." The clean-shutdown branch propagated the err and errgroup cancelled the WHOLE indexer. Bigger batches triggered this more often — went from latent to catastrophic on batch=100. Distinguish by checking ctx.Err() on the OUTER context; anything else falls through to bisect/dead-letter like any other transient fetch failure. 6. transport.go: ResponseHeaderTimeout 60s → 120s. Bigger batches against archive nodes occasionally take > 60s to respond; legit latency, not a failure signal. Deployed to both stacks; mainnet at ~350-600 bps steady state, up from ~53. Testnet has higher per-block row density and is now DB-write-bound at ~47-230 bps depending on which historical segment it's processing — still 2-4× baseline on a good window.
nimrod-teich
added a commit
that referenced
this pull request
Apr 19, 2026
…tes (#10) * fix(web): DB-size cache + handler "blocks done / total" line /api/status used to run SUM(pg_total_relation_size(...)) on every poll. On multi-hundred-million-row tables that query can spike to multi- second under checkpoint or autovacuum pressure, blocking the dashboard (observed: /api/status going from ~150 ms to >10 s with HTTP 500 "context deadline exceeded"). Mirror the SchemaInfo 30 s cache pattern: recompute at most once per 30 s, fall back to the last cached value on query failure. DB size drifts by a few MB per second under normal indexing load, so a 30 s staleness is invisible to operators. Handler card: new "X / Y blocks" line under the window/chain percentages. X = sum of (range.to − range.from + 1) clipped to the configured window; Y = windowEnd − windowStart + 1. Matches the same denominator window_pct uses, so the percent and the absolute count agree at a glance. Also fixed a JS regression: introducing `const ranges = h.ranges || []` at the top of updateHandlerCard collided with a pre-existing declaration later in the same function — duplicate-const is a parse-time SyntaxError which kills the whole inline <script> and renders the dashboard blank. Removed the duplicate declaration. * perf(rpc): shared http.Transport with HTTP/2 + TLS session cache Every RPCClient and RESTClient used to build its own http.Transport. Two concrete costs: 1. TCP keep-alive pool wasn't shared. Two clients pointing at the same host couldn't reuse each other's idle connections. 2. TLS session tickets / session IDs weren't cached anywhere — every reconnect paid a full TLS handshake. With 96 fetch workers hitting one host and Go's default MaxIdleConnsPerHost of 2, we were doing TCP+TLS handshakes on roughly every request. That's tens of milliseconds of CPU + round trips burned per batch, repeated across every worker. Introduce internal/rpc/transport.go with a package-level SharedTransport that every client wraps. Tuning: MaxIdleConnsPerHost = 256 (was Go default 2) MaxIdleConns = 1024 (pool ceiling) ForceAttemptHTTP2 = true (single multiplexed connection when supported) ClientSessionCache = LRU(512) (TLS session resumption, skip handshake on reconnect) IdleConnTimeout = 90s (keep-alives stay warm between bursts) DialContext = 5s connect + 30s tcp keep-alive TLSHandshakeTimeout = 5s ResponseHeaderTimeout = 60s (archive queries can be slow on cold cache) http.Client stays per-endpoint because Timeout is a Client-level field, not Transport — but all Clients share the Transport, so the connection pool and session cache are process-global. Should substantially reduce handshake churn and enable HTTP/2 multiplexing on supporting upstreams. Expected: lower p50 latency on every request, fewer concurrent TCP connections needed per worker count, and a much faster first-request-after-idle. * perf(rpc): drop per-endpoint AIMD cap; route by p50 × in-flight The AIMD controller capped each endpoint's concurrency based on p99/p50 latency ratio + error rate. For operators running shared public gateways, this kept us from being a bad citizen — essential. For operators running owned RPC nodes they're comfortable overwhelming, the cap became throughput on the table: budgets pinned at Min=4 with 90+ fetch workers idle, because the grow heuristic required 70% utilization we never reached with uneven tier coverage. For the owned-infra case the right primitive is NOT a cap, but a routing signal — always send the next request to the fastest currently-least-busy endpoint. If one slows down, its score rises and traffic naturally shifts; if it recovers, traffic comes back. No explicit cap, no stale floor pinning. Changes: - Delete internal/rpc/adaptive.go. AdaptiveController + its Tick / Grow / Shrink semantics are gone. - Remove the `ctrl` field from Endpoint, `ConcurrencyBudget()` method, `InitialConcurrency` / `MinConcurrency` / `MaxConcurrency` / `TargetP99` from MultiClientOptions. MultiClientOptions now only carries `HistorySeconds`. - Rename `StartController` → `StartMetricsRecorder`. Same 3s tick, but it only samples metrics into the history ring; no controller side-effects. - Rewrite `MultiClient.FetchBlocks` routing: rank eligible endpoints by `p50 × (1 + in_flight)` (lower is better). A slow-or-queued endpoint gets deranked; a fast idle endpoint wins. Unmeasured p50 is treated as 100 ms so new endpoints get exploration traffic. Ties break via the existing round-robin counter. - Remove the 5-second saturation-wait loop. Without a per-endpoint cap, "saturated" is no longer a real state at the routing layer — the score function handles pressure instead. - Drop the AIMD-driven worker-cap term from pipeline/sizer.go: the worker cap is now purely host-resource-limited (CPU × 8, mem / 2 MiB, operator hard-max). `EndpointSignal.Budget` field removed. - Retain `MetricSample` shape (drop Budget), remove Budget from `EndpointMetricsJSON`. - Dashboard: drop the Budget stat card and Adaptive-budget sparkline. Replace with a p50 latency sparkline (directly useful now that p50 is the routing signal). Update tooltip text to describe the new routing rule and note there's no concurrency cap. Expected impact: throughput climbs until the DB or the host run out of room — whichever comes first. Bad upstreams don't cause a retry-storm because the routing layer already routes around them via the p50/in-flight score; errors still trigger failover via the unchanged fast-fail path (429/5xx → next endpoint immediately). Guidance for operators still on shared public gateways: set `fetch_workers` modestly in config (or let the host-based cap do it implicitly). If a specific endpoint needs a hard per-connection limit, re-introduce a per-endpoint `max_concurrency` config field — the structure of MultiClient supports it cleanly, just wasn't needed for the owned-infra target. * perf(pipeline): split fetch + persist into decoupled worker pools Each worker used to do fetch + persist serially: pull a job, call FetchBlocks, open a tx, COPY, commit, advance. Under write-saturated load — which is normal on a 486M-row table with FKs + BRIN + index maintenance — fetchers sat idle behind COPYs. With 96 fetch workers and pool_size=32 there was nothing the concurrency could even do: half the workers were asleep waiting for their peer to finish writing. Split into two decoupled pools connected by a small buffered channel: producer → jobs chan → fetchers (N = fetch_workers) ↓ writeCh (buf 2 × writerCount) ↓ writers (M = pool_size / 2, min 8) → DB Fetchers fetch, Dispatch events, compute HandlerNeedsRange, and hand the parsed batch to writeCh. No DB tx is held by a fetcher — the brief pool connection for HandlerNeedsRange releases before enqueue. Writers pull from writeCh, open one tx per batch, Persist + RecordRange + Heartbeat, commit. If persist fails, the writer records the whole batch as a dead-letter (no bisect — persist errors are rarely block-specific) and moves on. The dead-letter sweep will refetch. Why this helps: - Fetch throughput is now bounded by the upstream ceiling + fetch worker count, not by DB write speed. Previously a slow commit blocked the fetch it was paired with. - Write throughput is bounded by pool_size (via writerCount), which is the correct DB-side constraint. Writers always have work queued (up to 2 × writerCount) so they never idle during producer gaps. - Progress log + blocks/sec rate now reflect COMMITTED blocks, not fetched-but-maybe-not-written. Honest dashboard numbers. - Bisect on fetch failure stays in the fetcher (unchanged). - processWithSplit signature gains a writeCh chan<-; dead-letter sweep plumbs the same one through so retries go through the split flow too. Default fetch_batch_size bumped 10 → 50. With per-endpoint caps gone and the split pipeline smoothing bursts, batch overhead matters more than queue depth; 50 heights per JSON-RPC batch cuts per-request overhead ~5× on the wire. Tendermint batch responses at this size are still under 2 MB, comfortably within any upstream's buffers. Shutdown ordering: producer closes jobs chan → fetchers drain, signal via WaitGroup → separate goroutine closes writeCh after last fetcher exits → writers drain, exit. errgroup.Wait() resolves when all participants have returned. * perf(rpc)+ui: moderate error-rate routing penalty; ETA on handler card Two small refinements after deploying the uncap + latency-routing change: 1. rpc: reduce the error-rate penalty on the routing score from ×10 to ×3. The 10× version was aggressively quarantining any endpoint that once had errors — because errors flow into a cumulative TotalErrors/TotalRequests ratio that decays slowly even as recent errors stop. A briefly-degraded endpoint ended up pinned off rotation until its 500-entry latency ring rotated out its old slow measurements, which at low traffic took minutes. ×3 is a gentler weighting: 50% err = 2.5× penalty, 100% err = 4× penalty — enough to bias away from a broken endpoint, not enough to keep a recovering one off the roster. 2. ui: handler card adds ETA to the "X / Y blocks" line, e.g. "3.21M / 4.91M blocks · ETA 2h 15m". Computed as (windowTotal − doneInWindow) / blocks_per_sec, formatted via the existing formatEta helper. Shows "done" when remaining is zero and "—" when bps is zero (warming up). * perf: writer pool sizing, static-batch, 30m MV, ctx-err classification Experimental round chasing a 3× throughput goal uncovered four real bottlenecks + one bug. All shipped here. 1. Writer pool was half the DB pool. pipeline.go: writerCount = pool_size / 2 → pool_size - 4. DB write was the bottleneck and we had half the pool deliberately unused. Reserving 4 connections for status handler + heartbeat + sweep + fetcher's HandlerNeedsRange still gives ~95% of pool to writers. Mainnet 53.6 → 70.6 bps (+32%) standalone. 2. DB pool was too small for the new worker count. Operator config bumped pool_size 32 → 80 (deploy-time, via .env). max_connections default is 100 in our postgres image; 80 indexer + ~10 postgraphile + ~5 internal fits comfortably. Writers went from saturated at pool_size to comfortably below. With #1 together: mainnet 53.6 → 208 bps (+289%), hit 3× target. 3. Adaptive batch sizer kept shrinking on tail-latency alone. sizer.go: drop the p99/p50 > 10 hot signal and the p99/p50 < 3 calm signal. For owned-infra operators, tail latency is a "slow but healthy" signal — routing handles it. ONLY error rate drives sizer shrink now. This matches the same decision made earlier in the AIMD-cap removal. pipeline.go: when operator sets fetch_batch_size > 0 explicitly, skip the sizer's batch state and use the static value. Previously the sizer ran whenever fetch_workers was adaptive (a separate signal), and its batch-state override silently ignored the operator's explicit batch-size config. With #1 + #2 + batch=50 static: mainnet 53.6 → 353 bps (+559%, observed up to 625 bps on log rolling average). 4. aggregates.yml: MV refresh schedule */5 → */30. On a 400M+ row relay_payments table, REFRESH MATERIALIZED VIEW CONCURRENTLY takes 5-10 min. At a 5-min schedule refreshes piled up and serialised on MV row-level locks, starving the writer pool. Dashboards read the MV for daily aggregates; 30 min freshness is acceptable and eliminates the lock contention. 5. Bug fix: processWithSplit treated per-request HTTP timeouts as fatal. pipeline.go: Go's http.Client.Timeout wraps context.Canceled when it fires on a slow upstream, so errors.Is(err, context.Canceled) returned true for BOTH "we were shut down" and "one request took too long." The clean-shutdown branch propagated the err and errgroup cancelled the WHOLE indexer. Bigger batches triggered this more often — went from latent to catastrophic on batch=100. Distinguish by checking ctx.Err() on the OUTER context; anything else falls through to bisect/dead-letter like any other transient fetch failure. 6. transport.go: ResponseHeaderTimeout 60s → 120s. Bigger batches against archive nodes occasionally take > 60s to respond; legit latency, not a failure signal. Deployed to both stacks; mainnet at ~350-600 bps steady state, up from ~53. Testnet has higher per-block row density and is now DB-write-bound at ~47-230 bps depending on which historical segment it's processing — still 2-4× baseline on a good window. * perf(writer): lock-free ranges + aggregate writes + empty-skip + compactor Four changes shipped together (#2, #3, #6 from the optimisation list): ### #3 — Remove advisory lock from RecordRange Before: every writer acquired pg_advisory_xact_lock(hashtext(handler)), then SELECT FOR UPDATE'd touching ranges, then DELETE+INSERT the merged row. All 76 writers on the same handler (lava_relay_payment) serialised on that lock. On pg_stat_activity under load the lock wait dominated. After: RecordRange is a single INSERT … ON CONFLICT DO UPDATE. No lock, no merge, no coordination between writers. Writers insert raw ranges (possibly overlapping), which accumulate until... ### Async range compactor A new goroutine per indexer process wakes every 30 s and for each handler runs `range_agg(int8range(...))` server-side to compute the minimum-rows representation, then swaps the rows in a single tx. Writers that race with compaction insert rows that survive and are picked up next cycle — one tick of extra fragmentation max. State.CompactRanges is public so ops can trigger it manually if needed. Logs `range compactor handler=X rows_merged=N` on non-zero reductions. ### #2 — Aggregate N writeJobs per tx Before: one writer = one goroutine = one tx per writeJob. With per-tx overhead (BEGIN, heartbeat, COMMIT) around 5-10 ms even with synchronous_commit=off, this caps throughput at ~writerCount × 100 txs/s no matter how fast individual batches are. After: each writer drains up to 32 jobs from writeCh with a 50 ms wait budget, then commits them in one tx. For each job: per-handler Persist (one COPY per handler), per-job RecordRange call, one shared Heartbeat at the tx tail. Amortises the per-tx cost across up to 32 batches. On failure the whole aggregate gets recorded as dead-letter — a coarser blast radius than per-batch, but persist errors are rare post-Dict-fix and the dead-letter sweep re-fetches anyway. ### #6 — Skip Persist call on empty-event batches For historical segments with no relay_payment events, each handler's dispatched event slice is empty. Persist is a no-op in that case but still costs interface dispatch + tx bookkeeping. Skip directly to RecordRange so the range still gets marked covered without the Persist call. ### Dashboard fix: >100% display The "X / Y blocks" line summed range lengths client-side; with overlapping raw ranges from the new lock-free RecordRange, sum > actual coverage → displayed >100% (user-reported testnet showing 100.00% when server-side window_pct was 78.6%). Switch to deriving absolute counts from the server's authoritative window_pct: doneInWindow = windowTotal × (window_pct / 100). Correct regardless of fragmentation. ### Observed Mainnet (near tip): status bps 270, row delta 12,395 rows/s. At historical segments seen peaks of 600+ bps. Throughput no longer capped by advisory-lock serialisation — writers can run fully parallel across the 76-worker pool. Testnet (heavy historical with upstream 502s): 30-75 bps depending on segment, but row delta ~93k rows/s at times — DB-write-bound, not lock-bound. Aggregation actual factor observed: `agg:1` in many progress logs. With 76 writers and a 50 ms window, the first writer often drains 1 job before the window expires — job arrival rate / writers ≫ 1 per 50 ms. Aggregation becomes visible under burst arrivals; under steady-state uniform arrival rate the factor stays ~1. Kept as the structural change it enables (one-tx-many-jobs is correct under any rate, the ceiling just rises with burstiness). * fix(web): restore const ranges = h.ranges declaration Previous commit removed the `const ranges` from updateHandlerCard because the new blocksLine logic (derived from window_pct) no longer needed it. But segment rendering later in the same function still references `ranges.length` and `ranges.forEach` — removing the declaration made those lines throw ReferenceError at the first handler render, and the whole dashboard went blank. Symmetrical to the earlier 'duplicate const' bug where I ADDED a redundant declaration. This time I removed a still-needed one. Added a JS parse check to the deploy path but it only catches static syntax errors, not runtime ReferenceErrors — worth adding a smoke test on the rendered page in a follow-up. * docs(readme): update to match the throughput-overhaul architecture Touches that track the code changes in this branch: - Tip-first parallel gap filling: add a "Fetch / persist are decoupled" subsection explaining the two-pool architecture and the endpoint-coverage tier slicing. Remove the sentence about writers taking pg_advisory_xact_lock — now lock-free with the async compactor. - "How much load are we pushing": update defaults (batch 10 → 50), drop the "live headroom" language (now p50-based routing), remove references to the AIMD budget in the endpoint-row example. - Delete the "Adaptive concurrency (AIMD)" section. Replace with "Endpoint routing" explaining the score = p50 × (1 + in_flight) × (1 + errRate × 3) rule + the fast-fail failover path on 429/5xx. - Retry layer map: redraw as the fetch+write split with the writeCh handoff between pools. Score-order routing in place of headroom-sorted. - Retry failover: add a paragraph on the ctx-err classification fix (per-request HTTP timeouts no longer crash the indexer via a mis-matched context.Canceled unwrap). - Worst-case amplification: update to N=3 endpoints, batch=50 giving ~108 requests (was 60 at batch=20, N=2). Reflects the common owned-infra deployment shape. - "Finding the capacity of a node": clarify that in owned-infra mode there's no per-endpoint cap to pin; the curve just informs process-wide fetch_workers sizing. - "Performance knobs": rewrite fetch_workers and fetch_batch_size descriptions (new defaults, new adaptive behaviour — only errors trigger shrink now). Add database.pool_size as a top-level perf knob since the writer pool derives from it, with guidance to watch pg_stat_activity for saturation.
nimrod-teich
added a commit
that referenced
this pull request
Apr 22, 2026
…sh from rewards side Two issues surfaced during deploy: 1. CoinGecko rate limiting: the snapshotter was fetching prices for EVERY mapped denom in denom_metadata (24 on a fresh seed), regardless of whether that denom appears in rewards at the snapshot block. Free-tier CoinGecko limits made this hit 5-attempt exhaustion on 15/27 mainnet dates, marking them status='failed' and preventing any MV refresh. Fix: selectDenomsForBlock joins provider_rewards + denom_metadata filtered on the target block_height. Mainnet observes 7 distinct denoms across 27 dates (vs the 24 from the seed) — a ~3x reduction in CoinGecko budget that eliminates the rate-limit pressure. 2. MV refresh only happened on denom_prices success. When every denom_prices snapshot failed (see #1), the priced_rewards MV stayed empty even though provider_rewards had 1615 rows populated. Fix: provider_rewards.Snapshot also REFRESH MATERIALIZED VIEW CONCURRENTLY on success. MV LEFT JOINs denom_prices, so rows appear with price_usd = NULL when the price snapshot hasn't landed yet — better than waiting behind a failing price snapshotter. Bonus: dates with provider_count=0 (legitimate empty, e.g. pre-rewards blocks) now short-circuit the price snapshot — the block has no denoms to price, record 0-count row, done. Was previously calling CoinGecko 24 times per empty date.
nimrod-teich
added a commit
that referenced
this pull request
Apr 22, 2026
…_rewards MV (#15) * feat(snapshotters): add provider_rewards monthly snapshotter New ingestion mechanism: monthly-17th@15:00-UTC snapshots of estimated_provider_rewards for every known provider, stored in Postgres so downstream consumers can query historical rewards via GraphQL instead of fanning out to chain RPC at request time. ## Architecture Added a sibling Snapshotter seam (internal/snapshotters/) parallel to the existing event-driven Handler, since estimated_provider_rewards is chain state not events. Snapshotter interface mirrors Handler's shape: Name + DDL + BlocksDue + Snapshot (tx-scoped). Registry.RunLoop ticks on a time.Ticker (default 10m), runs once immediately on startup for backfill, and isolates each target in its own pgx.Tx so one failure doesn't starve others. Jittered sleep between targets to avoid bursting chain RPC. ## Provider rewards snapshotter Concrete snapshotter calls GET {rest}/lavanet/lava/subscription/estimated_provider_rewards/{addr} Headers: x-cosmos-block-height={block}, lava-extension=archive at monthly-17th@15:00UTC block heights resolved via binary search against /cosmos/base/tendermint/v1beta1/blocks/{h}. Response classification: - 200 + info[] → parse "Boost: ETH1" → source_kind (0/1/2) + spec, upsert into app.providers / app.chains via Dict, insert rows. - success:false + "version does not exist" / "version mismatch" / "no commit info found" → replica pruned, retry with jittered exp-backoff up to 10 attempts. - success:false + "cannot get claimable rewards after distribution" → legit empty, treat as empty info[]. - Any other success:false → conservative retry. All of (insert snapshot row + provider_rewards rows) happens in the passed pgx.Tx; a single failed provider rolls the whole snapshot back. Next tick retries from scratch. ## Schema Two new tables using raw on-chain amounts (NUMERIC(40,0) base units, no USD conversion — reader's job): app.provider_rewards_snapshots (block_height PK, snapshot_date UNIQUE, block_time, provider_count, status) app.provider_rewards (block_height FK, provider_id FK, spec_id FK, source_kind, denom, amount, composite PK) Reuses app.providers + app.chains via the existing Dict helper. ## Web UI GET /api/snapshotters returns expected/covered/missing/failed dates per registered snapshotter. A new "Snapshotters" card renders a timeline strip alongside the existing handler card — different shape (discrete points vs contiguous ranges), so no attempt to unify. ## Config toggle snapshotters.provider_rewards.enabled=false fully disables: no goroutine spawned, no DDL applied, no log lines. Verified by the errgroup guard on len(snapReg.All()). * fix(snapshotters): correct REST path; mirror handlers:["all"] selection Two fixes the integration tests surfaced: 1. The `estimated_provider_rewards` endpoint takes two path segments (provider, amount_delegator), not one. Calling the one-arg form returns 501 Not Implemented from every Lava LCD gateway. Use `/lavanet/lava/pairing/estimated_provider_rewards/{provider}/1ulava` — the minimum valid coin, which gets us the per-provider rewards breakdown without requiring a real delegation. 2. 501/404 now route into the pruned-retry loop with structured httpStatusErr. Lava's public gateway load-balances across replicas and the rewards handler isn't always registered on every replica; a fresh connection usually rotates to a working one. Also added "not implemented" to the JSON-level classifyChainMessage branches for the same reason. 3. Snapshotter selection now mirrors the handler pattern exactly: `snapshotters.list: ["all"]` (or empty) enables every compiled-in snapshotter; explicit names turn on just those. Dropped the per-snapshotter `enabled: bool` flag. Env override is `SNAPSHOTTERS=<comma-separated>`, matching `INDEXER_HANDLERS`. * refactor(config): snapshotters.handlers mirrors indexer.handlers Rename snapshotters.list → snapshotters.handlers so the yaml key name reads identically to indexer.handlers (both are string-lists with the same "all" semantics). Default is now ["all"] explicitly (was nil). Env override is SNAPSHOTTERS_HANDLERS (was SNAPSHOTTERS), matching INDEXER_HANDLERS. * feat(web): show snapshotter REST URL on the dashboard The snapshotter's REST endpoint is separate from network.endpoints (it's typically an archive LCD, not a tendermint RPC), so it was invisible to the dashboard. Surface it via /api/snapshotters as rest_url and render it on the snapshotter card meta line. Exposed via a local interface{ RESTURL() string } rather than adding to the Snapshotter interface proper, so the core interface stays focused on DDL + schedule + snapshot. Other snapshotters can opt in by implementing the same method. * fix(snapshotters): address review criticals + highs Fixes the 1 critical + 4 highs from _workspace/report.md on this branch: - [critical] Long-held tx across chain fan-out: Snapshotter.Snapshot now takes *pgxpool.Pool, not pgx.Tx. Each snapshotter owns its own tx boundary. provider_rewards.Snapshot does HTTP fan-out OUTSIDE any tx; opens a short tx for dict ID resolution; opens a second short tx for the snapshot row + CopyFrom. Connection held for DB work only — single-digit seconds instead of minutes. - [high] fetchAll no sibling cancel: replaced the jobs/replies channel pattern with errgroup.WithContext + g.SetLimit. First error cancels siblings so a doomed snapshot doesn't run the remaining ~2000 retries. - [high] HTTPCaller uses http.DefaultTransport: now uses rpc.SharedTransport (HTTP/2, 256 idle conns per host, TLS session cache), same as the main pipeline's RPC clients. - [high] doGET doesn't drain body on non-2xx: read body via io.ReadAll(io.LimitReader(..., 16 MiB)) BEFORE checking status. Drains on all paths so net/http can return the conn to keep-alive pool. Also caps body size to bound a compromised upstream. - [high] FK coupling to relay_payment dict tables: provider_rewards .DDL() now includes providers + chains CREATE IF NOT EXISTS so the snapshotter can start standalone. Idempotent — harmless when relay_payment is also registered. - [high] Web concretely imports provider_rewards: added Status(ctx, pool) to Snapshotter interface; provider_rewards implements it with its own schema query + ExpectedDates. web drops the concrete import and the name-match dispatch. Also dropped redundant Server.SnapshotterSchema (duplicated State.Schema()). * fix(snapshotters): per-provider tolerance + mediums + lows Diagnosed the "hours with zero snapshots" problem: the chain returns HTTP 501 Not Implemented consistently at historical blocks where EstimatedProviderRewards wasn't yet registered, and intermittently even on recent blocks because Lava's public LB rotates through replicas that haven't all picked up the handler. With fail-on-first- error, one bad provider killed every snapshot. Primary fix: per-provider tolerance. - fetchAll now collects all results (entries or per-provider err) instead of failing on the first error. Per-provider errors do not cancel siblings — only outer ctx cancel aborts the fan-out. - Snapshot computes success_fraction and persists: * status='ok' with the successful subset when >=80% succeeded (the cluster the operator can actually analyse); * status='failed' with the error message when <80% succeeded (so BlocksDue stops re-retrying this date forever). - coveredDates now excludes both 'ok' and 'failed' statuses, so BlocksDue skips failed dates too — operators retry by deleting the failed row manually. Also classify empty-body and malformed-JSON responses as pruned-retry (they were bubbling up as terminal parse errors, which is why some dates failed with zero retries). Medium + low review findings rolled into this PR: - [medium] snapshotters.list → snapshotters.handlers doc refs in main.go, config.go, README.md updated (code was already renamed). - [medium] full-jitter backoff (was narrow ±50% band → clumped 25 workers onto the same retry window). - [medium] Registry.LastTickErr() accessor now exposes the lastTickErr the field was carrying — used to be set but never read. - [low] URLSurface named interface in snapshotters/ (was a local type-assert in web/). - [low] redactURLUserinfo before surfacing rest_url on the dashboard (basic-auth creds would otherwise leak via unauthenticated /api/snapshotters). - [low] url.PathEscape for the provider addr in the REST path. - [low] strconv.FormatInt / strconv.ParseInt replacing hand-rolled digit loops in doGET and Tip. - [low] time.NewTimer + Stop instead of time.After in the registry's per-target jitter sleep (no leaked timers on ctx cancel). - gofmt -w on the new files. * fix(snapshotters): correct REST path to subscription/...+10000000000ulava Per operator: the canonical endpoint is /lavanet/lava/subscription/estimated_provider_rewards/{addr}/10000000000ulava not the `pairing` module, and the amount_delegator must be 10^10 ulava (= 10 LAVA), not 1ulava. Earlier trial-and-error against lava.rest.lava.build's LB was misleading because replicas rotate and inconsistently register the handler — empirical probes returned 501 for both paths, but the subscription path is the authoritative one downstream consumers expect. Also classify "cannot estimate rewards for delegator/delegation" as empty (no accrued rewards), same as "cannot get claimable rewards after distribution". Observed in the wild on providers whose delegation state isn't present at the queried height — deterministic per provider, no point retrying. * fix(snapshotters): distinguish real failures from chain no-rewards Per operator feedback: when iterating across every provider ever seen, most will legitimately have NO rewards at a historical block (not staked yet, unstaked, no delegation state). The chain returns success=false with various messages for these, and the previous classification treated them the same as real fetch failures (HTTP 5xx, decode errors) — causing historical dates to falsely show as "failed" when most providers simply weren't applicable. New classification rules: - Only replica/version issues retry as pruned: "version does not exist" / "version mismatch" / "no commit info found" / "not implemented" - Every other success=false is chain-authoritative "no rewards for this (provider, block) pair" — treated as outcomeNoRewards, does NOT count toward the failed-snapshot threshold. - Only HTTP 5xx (after retry exhaustion), empty-body / decode errors (after retry), or exhausted pruned-retry count toward outcomeFailed. Snapshot log line now breaks down fetched / no_rewards / failed so operators can see what each snapshot actually did, e.g.: total=276 fetched=48 no_rewards=220 failed=8 Threshold check now uses (fetched + no_rewards) / total >= 0.8 so a date where most providers weren't staked yet still lands as status='ok' with only the successful subset written to provider_rewards. Only dates where the upstream is genuinely broken get marked status='failed'. Also: UI timeline dots now show the pinned block height in the tooltip alongside the date (per operator request), so each circle identifies the exact chain block the snapshot was taken at. SnapshotterStatus now carries a blocks map; the dashboard renders "{date} · block {N}" on hover. * fix(snapshotters): genesis-anchored slots + state-pruned outcome Two follow-up fixes from iteration: 1. Reference time is now the chain genesis timestamp, not a hardcoded monthly-17th@15:00 UTC. Slot N = genesis + N months, preserving genesis hour/minute/second. For Lava mainnet (genesis 2024-01-17T15:00:00Z) this reproduces the existing cadence without magic numbers; for any other chain the anchor is correct by construction. ExpectedDates signature: (genesis, earliest, now). earliest_date config is now just a floor — no longer constrained to a 17th, since the day is derived from genesis. 2. New outcomeStatePruned distinguishes "archive pruned state at this height" from real fetch failures. When EstimatedRewards exhausts its pruned-retries AND the final error is a state-availability issue (version does not exist / pruned / 501 from a rotated replica), we wrap with errStatePruned instead of letting it bubble as outcomeFailed. The snapshot threshold now applies ONLY to outcomeFailed — archive unavailability is a chain-infrastructure limitation, not something retries fix, so dates where most providers come back as statePruned can still land as status='ok' with the subset we DID get. Progress log now shows the full breakdown: fetched=N no_rewards=N state_pruned=N failed=N Verified that `lava-extension: archive` header IS being sent (caller loops over configured headers), but the public `lava.rest.lava.build` gateway's archive backing only retains state ~2 months deep, so the header has no effect on historical dates. That's a gateway- infrastructure constraint, not fixable here. * feat(config): snapshot_anchor override for forked chains Chains that forked or were renamed still report the upstream origin's timestamp via /genesis — e.g. lava-testnet-2 reports 2022-12-26 (the testnet-1 origin) despite its operational start being 2023-08-17 10:02 UTC. The snapshot anchor needs to be the real operational start so the monthly cadence lands on the intended day-of-month. snapshot_anchor is an RFC3339 override; empty = fall back to client.GenesisTime(). Lava mainnet leaves it empty (its /genesis reports the correct 2024-01-17T15:00 start); testnet sets "2023-08-17T10:02:00Z". * fix(config): earliest_date is now optional — empty = from genesis The floor was silently masking all pre-2025 slots on mainnet + all pre-2025 slots on testnet, producing fewer timeline dots than expected. Making it optional lets the genesis-anchored cadence surface every slot from genesis+1mo through now by default. * feat(denoms): priced_denoms deriver + dashboard card New internal/denoms package derives app.priced_denoms from app.provider_rewards on a 5m ticker. Seeds 23 known Cosmos token mappings (base_denom → coingecko_id) at startup via ON CONFLICT DO UPDATE. New denoms land with coingecko_id=NULL — operators fill via one SQL UPDATE, no redeploy. Resolver handles: - microdenom strip (ulava → lava, uatom → atom) - IBC trace lookup against REST /ibc/apps/transfer/v1/denom_traces/{hash} with in-process cache (traces are immutable once issued) - transient vs permanent classification so transient chain errors retry next tick, persistent ones (404 on a trace) don't Deriver exposes an in-memory Status() snapshot the web layer reads via /api/denoms. Dashboard renders a "Priced denoms" card with mapped/unmapped counts and an expandable list of denoms needing a coingecko_id — actionable signal for operators. Config: denoms.deriver.enabled/interval/resolve_timeout/rest_url/ rest_headers. REST URL inherits from snapshotters.provider_rewards if not set (same archive-backed endpoint). * fix(snapshotters): read total[] fallback when chain leaves info[] empty Diagnosed "27 circles green, zero fact rows, zero provider_count": the chain's actual response shape for our simulated-delegation query is {"info":[], "total":[{"denom":"ulava","amount":"…"}]} not {"info":[{"source":"Boost: …", "amount":[…]}]} as the spec assumed. info[] carries the source breakdown but the chain only populates it for query shapes we aren't sending; total[] has the aggregate amount. Fix: parse both fields. Prefer info[] when present (authoritative breakdown); fall back to a synthesised entry with SourceKind=3 (SourceTotal) and spec="" from total[] so the amount isn't dropped. Downstream consumers differentiate total-aggregate rows from per-source rows by source_kind. Also persists the seeded enum gap — SourceTotal=3 extends the existing 0=Boost / 1=Pools / 2=Subscription mapping; the DB column is SMALLINT so no schema change. * feat(denoms): dict-managed denoms + priced_rewards MV + historical price snapshotter Replaces the standalone priced_denoms deriver with the right shape: the denom is a Dict-managed table (parallel to providers/chains), IBC hashes are resolved at snapshot-write time so the dict stores only chain-canonical microdenoms (uatom, ulava), and a new snapshotter populates historical USD prices via CoinGecko on the same monthly cadence as rewards. Schema (DROP + recreate, no prod data): app.denoms — (id, denom) Dict pattern, resolved-form only app.denom_metadata — (denom_id PK, base_denom, factor, coingecko_id, suppress, first/last_seen_at). Seeded with the 24-token map ported verbatim from info's DENOM_CONVERSIONS + DENOM_COINGECKO_ID. app.provider_rewards — denom TEXT → denom_id INT + source_denom TEXT (raw preserved for info's source_denom API field). PK now includes source_denom so two IBC hashes resolving to the same base don't collide. app.denom_price_snapshots — one row per priced date. Parent of: app.denom_prices — (snapshot_date, denom_id) PK, price_usd NUMERIC. price_usd=0 is a legitimate "coin not listed / no market_data on date" signal. app.priced_rewards MV — rewards × metadata × prices JOIN with display_amount + value_usd computed in SQL. Refreshed CONCURRENTLY after every successful denom_prices snapshot. This is the surface info (and anything else) reads. provider_rewards snapshotter: - h.denoms *events.Dict added (third Dict alongside providers/chains). - HTTPCaller.ResolveIBC(hash) — trace lookup via /ibc/apps/transfer/v1/denom_traces/{hash}, in-process cache forever (IBC traces are immutable). 404 → permanent sentinel, 5xx → transient (provider retried). Raw ibc/ hashes never enter app.denoms. - Suppressed denoms (denom_metadata.suppress=TRUE) are cached per tick and dropped before Dict.IDs resolution — test denom blacklist now editable via SQL instead of hardcoded in info. denom_prices snapshotter (new internal/snapshotters/denom_prices): - BlocksDue shares provider_rewards.ExpectedDates for the same genesis-anchored monthly cadence; binary-searches block_height at each slot via the shared HTTPCaller. - CoinGecko /coins/{id}/history?date=DD-MM-YYYY, sequential per date, LAVA first. 5-attempt retry with Retry-After + jittered exponential backoff, 60s cap. LAVA failure → status='failed' (no point without the anchor); non-LAVA failure → log + skip (best-effort). 404 and no-market_data → persist price_usd=0. - REFRESH MATERIALIZED VIEW CONCURRENTLY app.priced_rewards after each successful write. Unique index on the MV enables CONCURRENTLY. Deleted: - internal/denoms/ (whole package — old Deriver pattern was wrong shape) - /api/denoms handler + DenomDeriver field on web.Server - "Priced denoms" dashboard card - cfg.Denoms struct + DENOMS_DERIVER_* envs Config added: snapshotters: denom_prices: coingecko_api_url: "" # default: https://api.coingecko.com/api/v3 rest_headers: {} # e.g. x-cg-pro-api-key for Pro plan On deploy: drops + recreates all snapshotter tables (CASCADE catches the MV). Snapshotters re-run from genesis on the next tick. * fix(snapshotters): treat HTTP 5xx as state-pruned, not real failure Mainnet snapshots 2024-10 → 2026-03 were all landing as status='failed' because 36-52 of 118 providers returned HTTP 500 (not 501) per snapshot — the Lava LB routes historical queries across archive replicas with varying state depth, and the refusal layer returns 500/502/503/504 about as often as 501. Only 501 and 404 were routed into the pruned-retry path; everything else fell through to outcomeFailed, pushed the failure fraction past the 20% threshold, and caused the entire date's 70+ successful providers to be discarded. Fix: 5xx + 404 all route into the pruned-retry loop and exhaust to outcomeStatePruned on max attempts. The snapshot threshold only counts real fetch failures (non-route-able errors), so the 70 providers that DID succeed at each date now land as rows with status='ok'. Other 4xx (401, 403, 429) still fall through as real errors — those are authentication / rate-limit issues we should surface, not silence. * test(snapshotters): update HTTP test to match new 5xx-as-pruned contract * fix(denom_prices): only fetch prices for denoms in rewards + MV refresh from rewards side Two issues surfaced during deploy: 1. CoinGecko rate limiting: the snapshotter was fetching prices for EVERY mapped denom in denom_metadata (24 on a fresh seed), regardless of whether that denom appears in rewards at the snapshot block. Free-tier CoinGecko limits made this hit 5-attempt exhaustion on 15/27 mainnet dates, marking them status='failed' and preventing any MV refresh. Fix: selectDenomsForBlock joins provider_rewards + denom_metadata filtered on the target block_height. Mainnet observes 7 distinct denoms across 27 dates (vs the 24 from the seed) — a ~3x reduction in CoinGecko budget that eliminates the rate-limit pressure. 2. MV refresh only happened on denom_prices success. When every denom_prices snapshot failed (see #1), the priced_rewards MV stayed empty even though provider_rewards had 1615 rows populated. Fix: provider_rewards.Snapshot also REFRESH MATERIALIZED VIEW CONCURRENTLY on success. MV LEFT JOINs denom_prices, so rows appear with price_usd = NULL when the price snapshot hasn't landed yet — better than waiting behind a failing price snapshotter. Bonus: dates with provider_count=0 (legitimate empty, e.g. pre-rewards blocks) now short-circuit the price snapshot — the block has no denoms to price, record 0-count row, done. Was previously calling CoinGecko 24 times per empty date. * fix(snapshotters): drop one-shot DROP CASCADE from provider_rewards DDL Agent's original DDL included DROP TABLE IF EXISTS app.provider_rewards CASCADE; DROP TABLE IF EXISTS app.provider_rewards_snapshots CASCADE; as a migration helper. Problem: DDL runs on EVERY startup, so every restart was wiping the accumulated snapshot data. Kept the priced_denoms drop (cleanup of the obsolete deriver table) but swapped the main tables to CREATE IF NOT EXISTS only. Schema migration has already run on every live environment; no future DROP needed. * fix(snapshotters): drop /10000000000ulava path arg — use bare /{addr}/ Root cause of the 40× undercount on mainnet rewards: the Lava chain's /estimated_provider_rewards endpoint has two semantically distinct response shapes depending on the path: /{addr}/ → actual accrued rewards per source (info[] populated with "Boost: ARBITRUM", "Pools: ETH1", "Subscription: COSMOSHUB" entries, each with ulava amount) /{addr}/<coin-amount> → simulated rewards for a delegation of that size (info=[], total=[{ulava, N}]) I wired the indexer to the second shape (`/10000000000ulava`), believing the arg was required. That returned a tiny synthetic total ~40× smaller than actual accrual, and with no source attribution — which is why our priced_rewards rows show ~$2 per provider vs prod's ~$99+, and why every row's spec is empty (the `total[]` fallback I added as a coping mechanism for empty info[] is what we were persisting). Fix: match prod info's path verbatim (apps/api/src/rpc/rewards.ts::fetchRawProviderRewards) — trailing slash, no amount arg. `total[]` fallback stays in ParseEstimatedRewards as defence in depth but should now rarely fire. Memory updated to reflect the right path. Tests updated to mock the corrected path. Operator action required: wipe app.provider_rewards on mainnet so the next tick re-snapshots with the correct values. The priced_rewards MV will auto-refresh. * feat(snapshotters): add supply snapshotter — monthly-17th total supply Third sibling snapshotter alongside provider_rewards and denom_prices. Captures Lava's total ulava supply at each genesis-anchored monthly slot; downstream consumers (info's /burn-rate) read the table via GraphQL and compute burn = prev_supply - curr_supply per month, replacing the ~2N chain-RPC calls they make today on every cold request. Schema: app.supply_snapshots (snapshot_date PK, block_height, block_time, snapshot_at, total_supply NUMERIC(40,0), status, error) Flow per Snapshot(): 1. x-cosmos-block-height pinned REST call to /cosmos/bank/v1beta1/supply?pagination.limit=10000 2. Extract `ulava` amount (missing ulava → status='failed' with clear error — don't silently zero) 3. One-row INSERT; short tx; block-time lookup shares the provider_rewards RESTCaller for pool + TLS reuse BlocksDue shares provider_rewards.ExpectedDates for the same genesis-anchored monthly cadence; subtracts dates already covered or explicitly failed so operators can retry by deleting the failed row. No UI changes — the dashboard iterates `Snapshotters.All()` generically via the Snapshotter + URLSurface interfaces, so the supply card appears automatically on /api/snapshotters. Config: snapshotters.supply { rest_url, rest_headers, earliest_date }. Env: SUPPLY_REST_URL. Falls back to the main REST endpoint when rest_url is empty (same resolver pattern as the other snapshotters). Build + vet + test green; 13 new tests cover happy path, missing ulava, transient 5xx, bad JSON, bad amount string, large-supply roundtrip, cache hit, interface pin, URL surface, DDL idempotency. * feat(config): default earliest_date for rewards/denoms to 2024-11-17 Bake the mainnet-useful floor into the code so new operators don't have to discover it: provider_rewards + denom_prices both default to 2024-11-17 (the earliest monthly-17th where the chain's estimated_provider_rewards handler returns populated info[]). Supply stays empty — bank-module queries work from genesis, so there's no reason to hide early samples. Also: - Add DenomPrices.EarliestDate + ParsedEarliestDate(); main.go now reads it for denom_prices wiring (was incorrectly sharing the provider_rewards field). - New DENOM_PRICES_EARLIEST_DATE env override. - config.example.yml: document the new default, list compiled-in handlers/snapshotters next to handlers: ["all"] so operators can see what "all" expands to without grepping.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
First commit of the lava-indexer — a lightweight Go. Reads Lava blocks from Tendermint RPC (and/or Cosmos REST), extracts the events we care about, and writes them to Postgres
with a space-efficient normalised schema. PostGraphile exposes a GraphQL
API over the same schema; a built-in web UI shows live progress.
What's inside
p99/p50 + utilisation, shrinks on tail-latency blowups or 5 %+ errors,
with a 20-request warmup so cold TLS can't collapse the cap.
eligible endpoint; 429 triggers instant failover instead of burning
retry budget.
exponential backoff + jitter,
Retry-Afterhonoured; when allendpoints have failed a batch we bisect it and persistently-failing
single heights land in
app.indexer_failuresso one corrupt blockcan't stall the indexer.
app.indexer_rangeswith advisory-lockmerging, so re-running with a different
start_heightonly fetchesthe gaps. Disjoint gaps are filled in parallel (backfill and
tip-follow don't serialise).
lava-extension: archive,Authorization: Bearer ...,x-api-key, ...) routedthrough for each endpoint.
provider / chain strings,
REALfor QoS fractions, BRIN on(block_height, timestamp). ~100 bytes/row.one pgx.Tx, so "indexed" always means "on disk".
0= adaptive bydefault).
indexed, red = dead-letter), Endpoints tab with AIMD / rps / latency
sparklines, GraphQL tab via same-origin reverse-proxy. In-place DOM
updates, no flicker.
mv_relay_dailyrefreshedevery 5 min, with delta-compatible
qos_*_sum / qos_count / qos_cucolumns so the consumer can re-derive averages at arbitrary
granularity.
Test plan
go build ./...passesdocker compose up -d --buildbrings postgres + indexer + graphqlup and starts indexing
:8080renders Overview / Endpoints / GraphQL tabs andreflects live progress
curl http://localhost:8080/healthz→ 200mvRelayDailiesGraphQL query returns rows after first refresh