Repository navigation
Throughput overhaul: uncap AIMD, split pipelines, lock-free range writes - #10
Merged
Merged
Conversation
/api/status used to run SUM(pg_total_relation_size(...)) on every poll. On multi-hundred-million-row tables that query can spike to multi- second under checkpoint or autovacuum pressure, blocking the dashboard (observed: /api/status going from ~150 ms to >10 s with HTTP 500 "context deadline exceeded"). Mirror the SchemaInfo 30 s cache pattern: recompute at most once per 30 s, fall back to the last cached value on query failure. DB size drifts by a few MB per second under normal indexing load, so a 30 s staleness is invisible to operators. Handler card: new "X / Y blocks" line under the window/chain percentages. X = sum of (range.to − range.from + 1) clipped to the configured window; Y = windowEnd − windowStart + 1. Matches the same denominator window_pct uses, so the percent and the absolute count agree at a glance. Also fixed a JS regression: introducing `const ranges = h.ranges || []` at the top of updateHandlerCard collided with a pre-existing declaration later in the same function — duplicate-const is a parse-time SyntaxError which kills the whole inline <script> and renders the dashboard blank. Removed the duplicate declaration.
Every RPCClient and RESTClient used to build its own http.Transport. Two concrete costs: 1. TCP keep-alive pool wasn't shared. Two clients pointing at the same host couldn't reuse each other's idle connections. 2. TLS session tickets / session IDs weren't cached anywhere — every reconnect paid a full TLS handshake. With 96 fetch workers hitting one host and Go's default MaxIdleConnsPerHost of 2, we were doing TCP+TLS handshakes on roughly every request. That's tens of milliseconds of CPU + round trips burned per batch, repeated across every worker. Introduce internal/rpc/transport.go with a package-level SharedTransport that every client wraps. Tuning: MaxIdleConnsPerHost = 256 (was Go default 2) MaxIdleConns = 1024 (pool ceiling) ForceAttemptHTTP2 = true (single multiplexed connection when supported) ClientSessionCache = LRU(512) (TLS session resumption, skip handshake on reconnect) IdleConnTimeout = 90s (keep-alives stay warm between bursts) DialContext = 5s connect + 30s tcp keep-alive TLSHandshakeTimeout = 5s ResponseHeaderTimeout = 60s (archive queries can be slow on cold cache) http.Client stays per-endpoint because Timeout is a Client-level field, not Transport — but all Clients share the Transport, so the connection pool and session cache are process-global. Should substantially reduce handshake churn and enable HTTP/2 multiplexing on supporting upstreams. Expected: lower p50 latency on every request, fewer concurrent TCP connections needed per worker count, and a much faster first-request-after-idle.
The AIMD controller capped each endpoint's concurrency based on p99/p50 latency ratio + error rate. For operators running shared public gateways, this kept us from being a bad citizen — essential. For operators running owned RPC nodes they're comfortable overwhelming, the cap became throughput on the table: budgets pinned at Min=4 with 90+ fetch workers idle, because the grow heuristic required 70% utilization we never reached with uneven tier coverage. For the owned-infra case the right primitive is NOT a cap, but a routing signal — always send the next request to the fastest currently-least-busy endpoint. If one slows down, its score rises and traffic naturally shifts; if it recovers, traffic comes back. No explicit cap, no stale floor pinning. Changes: - Delete internal/rpc/adaptive.go. AdaptiveController + its Tick / Grow / Shrink semantics are gone. - Remove the `ctrl` field from Endpoint, `ConcurrencyBudget()` method, `InitialConcurrency` / `MinConcurrency` / `MaxConcurrency` / `TargetP99` from MultiClientOptions. MultiClientOptions now only carries `HistorySeconds`. - Rename `StartController` → `StartMetricsRecorder`. Same 3s tick, but it only samples metrics into the history ring; no controller side-effects. - Rewrite `MultiClient.FetchBlocks` routing: rank eligible endpoints by `p50 × (1 + in_flight)` (lower is better). A slow-or-queued endpoint gets deranked; a fast idle endpoint wins. Unmeasured p50 is treated as 100 ms so new endpoints get exploration traffic. Ties break via the existing round-robin counter. - Remove the 5-second saturation-wait loop. Without a per-endpoint cap, "saturated" is no longer a real state at the routing layer — the score function handles pressure instead. - Drop the AIMD-driven worker-cap term from pipeline/sizer.go: the worker cap is now purely host-resource-limited (CPU × 8, mem / 2 MiB, operator hard-max). `EndpointSignal.Budget` field removed. - Retain `MetricSample` shape (drop Budget), remove Budget from `EndpointMetricsJSON`. - Dashboard: drop the Budget stat card and Adaptive-budget sparkline. Replace with a p50 latency sparkline (directly useful now that p50 is the routing signal). Update tooltip text to describe the new routing rule and note there's no concurrency cap. Expected impact: throughput climbs until the DB or the host run out of room — whichever comes first. Bad upstreams don't cause a retry-storm because the routing layer already routes around them via the p50/in-flight score; errors still trigger failover via the unchanged fast-fail path (429/5xx → next endpoint immediately). Guidance for operators still on shared public gateways: set `fetch_workers` modestly in config (or let the host-based cap do it implicitly). If a specific endpoint needs a hard per-connection limit, re-introduce a per-endpoint `max_concurrency` config field — the structure of MultiClient supports it cleanly, just wasn't needed for the owned-infra target.
Each worker used to do fetch + persist serially: pull a job, call
FetchBlocks, open a tx, COPY, commit, advance. Under write-saturated
load — which is normal on a 486M-row table with FKs + BRIN + index
maintenance — fetchers sat idle behind COPYs. With 96 fetch workers
and pool_size=32 there was nothing the concurrency could even do:
half the workers were asleep waiting for their peer to finish writing.
Split into two decoupled pools connected by a small buffered channel:
producer → jobs chan → fetchers (N = fetch_workers)
↓
writeCh (buf 2 × writerCount)
↓
writers (M = pool_size / 2, min 8) → DB
Fetchers fetch, Dispatch events, compute HandlerNeedsRange, and hand
the parsed batch to writeCh. No DB tx is held by a fetcher — the
brief pool connection for HandlerNeedsRange releases before enqueue.
Writers pull from writeCh, open one tx per batch, Persist + RecordRange
+ Heartbeat, commit. If persist fails, the writer records the whole
batch as a dead-letter (no bisect — persist errors are rarely
block-specific) and moves on. The dead-letter sweep will refetch.
Why this helps:
- Fetch throughput is now bounded by the upstream ceiling + fetch
worker count, not by DB write speed. Previously a slow commit
blocked the fetch it was paired with.
- Write throughput is bounded by pool_size (via writerCount), which
is the correct DB-side constraint. Writers always have work queued
(up to 2 × writerCount) so they never idle during producer gaps.
- Progress log + blocks/sec rate now reflect COMMITTED blocks, not
fetched-but-maybe-not-written. Honest dashboard numbers.
- Bisect on fetch failure stays in the fetcher (unchanged).
- processWithSplit signature gains a writeCh chan<-; dead-letter sweep
plumbs the same one through so retries go through the split flow
too.
Default fetch_batch_size bumped 10 → 50. With per-endpoint caps gone
and the split pipeline smoothing bursts, batch overhead matters more
than queue depth; 50 heights per JSON-RPC batch cuts per-request
overhead ~5× on the wire. Tendermint batch responses at this size
are still under 2 MB, comfortably within any upstream's buffers.
Shutdown ordering: producer closes jobs chan → fetchers drain,
signal via WaitGroup → separate goroutine closes writeCh after last
fetcher exits → writers drain, exit. errgroup.Wait() resolves when
all participants have returned.
Two small refinements after deploying the uncap + latency-routing change: 1. rpc: reduce the error-rate penalty on the routing score from ×10 to ×3. The 10× version was aggressively quarantining any endpoint that once had errors — because errors flow into a cumulative TotalErrors/TotalRequests ratio that decays slowly even as recent errors stop. A briefly-degraded endpoint ended up pinned off rotation until its 500-entry latency ring rotated out its old slow measurements, which at low traffic took minutes. ×3 is a gentler weighting: 50% err = 2.5× penalty, 100% err = 4× penalty — enough to bias away from a broken endpoint, not enough to keep a recovering one off the roster. 2. ui: handler card adds ETA to the "X / Y blocks" line, e.g. "3.21M / 4.91M blocks · ETA 2h 15m". Computed as (windowTotal − doneInWindow) / blocks_per_sec, formatted via the existing formatEta helper. Shows "done" when remaining is zero and "—" when bps is zero (warming up).
Experimental round chasing a 3× throughput goal uncovered four real bottlenecks + one bug. All shipped here. 1. Writer pool was half the DB pool. pipeline.go: writerCount = pool_size / 2 → pool_size - 4. DB write was the bottleneck and we had half the pool deliberately unused. Reserving 4 connections for status handler + heartbeat + sweep + fetcher's HandlerNeedsRange still gives ~95% of pool to writers. Mainnet 53.6 → 70.6 bps (+32%) standalone. 2. DB pool was too small for the new worker count. Operator config bumped pool_size 32 → 80 (deploy-time, via .env). max_connections default is 100 in our postgres image; 80 indexer + ~10 postgraphile + ~5 internal fits comfortably. Writers went from saturated at pool_size to comfortably below. With #1 together: mainnet 53.6 → 208 bps (+289%), hit 3× target. 3. Adaptive batch sizer kept shrinking on tail-latency alone. sizer.go: drop the p99/p50 > 10 hot signal and the p99/p50 < 3 calm signal. For owned-infra operators, tail latency is a "slow but healthy" signal — routing handles it. ONLY error rate drives sizer shrink now. This matches the same decision made earlier in the AIMD-cap removal. pipeline.go: when operator sets fetch_batch_size > 0 explicitly, skip the sizer's batch state and use the static value. Previously the sizer ran whenever fetch_workers was adaptive (a separate signal), and its batch-state override silently ignored the operator's explicit batch-size config. With #1 + #2 + batch=50 static: mainnet 53.6 → 353 bps (+559%, observed up to 625 bps on log rolling average). 4. aggregates.yml: MV refresh schedule */5 → */30. On a 400M+ row relay_payments table, REFRESH MATERIALIZED VIEW CONCURRENTLY takes 5-10 min. At a 5-min schedule refreshes piled up and serialised on MV row-level locks, starving the writer pool. Dashboards read the MV for daily aggregates; 30 min freshness is acceptable and eliminates the lock contention. 5. Bug fix: processWithSplit treated per-request HTTP timeouts as fatal. pipeline.go: Go's http.Client.Timeout wraps context.Canceled when it fires on a slow upstream, so errors.Is(err, context.Canceled) returned true for BOTH "we were shut down" and "one request took too long." The clean-shutdown branch propagated the err and errgroup cancelled the WHOLE indexer. Bigger batches triggered this more often — went from latent to catastrophic on batch=100. Distinguish by checking ctx.Err() on the OUTER context; anything else falls through to bisect/dead-letter like any other transient fetch failure. 6. transport.go: ResponseHeaderTimeout 60s → 120s. Bigger batches against archive nodes occasionally take > 60s to respond; legit latency, not a failure signal. Deployed to both stacks; mainnet at ~350-600 bps steady state, up from ~53. Testnet has higher per-block row density and is now DB-write-bound at ~47-230 bps depending on which historical segment it's processing — still 2-4× baseline on a good window.
…actor Four changes shipped together (#2, #3, #6 from the optimisation list): ### #3 — Remove advisory lock from RecordRange Before: every writer acquired pg_advisory_xact_lock(hashtext(handler)), then SELECT FOR UPDATE'd touching ranges, then DELETE+INSERT the merged row. All 76 writers on the same handler (lava_relay_payment) serialised on that lock. On pg_stat_activity under load the lock wait dominated. After: RecordRange is a single INSERT … ON CONFLICT DO UPDATE. No lock, no merge, no coordination between writers. Writers insert raw ranges (possibly overlapping), which accumulate until... ### Async range compactor A new goroutine per indexer process wakes every 30 s and for each handler runs `range_agg(int8range(...))` server-side to compute the minimum-rows representation, then swaps the rows in a single tx. Writers that race with compaction insert rows that survive and are picked up next cycle — one tick of extra fragmentation max. State.CompactRanges is public so ops can trigger it manually if needed. Logs `range compactor handler=X rows_merged=N` on non-zero reductions. ### #2 — Aggregate N writeJobs per tx Before: one writer = one goroutine = one tx per writeJob. With per-tx overhead (BEGIN, heartbeat, COMMIT) around 5-10 ms even with synchronous_commit=off, this caps throughput at ~writerCount × 100 txs/s no matter how fast individual batches are. After: each writer drains up to 32 jobs from writeCh with a 50 ms wait budget, then commits them in one tx. For each job: per-handler Persist (one COPY per handler), per-job RecordRange call, one shared Heartbeat at the tx tail. Amortises the per-tx cost across up to 32 batches. On failure the whole aggregate gets recorded as dead-letter — a coarser blast radius than per-batch, but persist errors are rare post-Dict-fix and the dead-letter sweep re-fetches anyway. ### #6 — Skip Persist call on empty-event batches For historical segments with no relay_payment events, each handler's dispatched event slice is empty. Persist is a no-op in that case but still costs interface dispatch + tx bookkeeping. Skip directly to RecordRange so the range still gets marked covered without the Persist call. ### Dashboard fix: >100% display The "X / Y blocks" line summed range lengths client-side; with overlapping raw ranges from the new lock-free RecordRange, sum > actual coverage → displayed >100% (user-reported testnet showing 100.00% when server-side window_pct was 78.6%). Switch to deriving absolute counts from the server's authoritative window_pct: doneInWindow = windowTotal × (window_pct / 100). Correct regardless of fragmentation. ### Observed Mainnet (near tip): status bps 270, row delta 12,395 rows/s. At historical segments seen peaks of 600+ bps. Throughput no longer capped by advisory-lock serialisation — writers can run fully parallel across the 76-worker pool. Testnet (heavy historical with upstream 502s): 30-75 bps depending on segment, but row delta ~93k rows/s at times — DB-write-bound, not lock-bound. Aggregation actual factor observed: `agg:1` in many progress logs. With 76 writers and a 50 ms window, the first writer often drains 1 job before the window expires — job arrival rate / writers ≫ 1 per 50 ms. Aggregation becomes visible under burst arrivals; under steady-state uniform arrival rate the factor stays ~1. Kept as the structural change it enables (one-tx-many-jobs is correct under any rate, the ceiling just rises with burstiness).
Previous commit removed the `const ranges` from updateHandlerCard because the new blocksLine logic (derived from window_pct) no longer needed it. But segment rendering later in the same function still references `ranges.length` and `ranges.forEach` — removing the declaration made those lines throw ReferenceError at the first handler render, and the whole dashboard went blank. Symmetrical to the earlier 'duplicate const' bug where I ADDED a redundant declaration. This time I removed a still-needed one. Added a JS parse check to the deploy path but it only catches static syntax errors, not runtime ReferenceErrors — worth adding a smoke test on the rendered page in a follow-up.
Touches that track the code changes in this branch: - Tip-first parallel gap filling: add a "Fetch / persist are decoupled" subsection explaining the two-pool architecture and the endpoint-coverage tier slicing. Remove the sentence about writers taking pg_advisory_xact_lock — now lock-free with the async compactor. - "How much load are we pushing": update defaults (batch 10 → 50), drop the "live headroom" language (now p50-based routing), remove references to the AIMD budget in the endpoint-row example. - Delete the "Adaptive concurrency (AIMD)" section. Replace with "Endpoint routing" explaining the score = p50 × (1 + in_flight) × (1 + errRate × 3) rule + the fast-fail failover path on 429/5xx. - Retry layer map: redraw as the fetch+write split with the writeCh handoff between pools. Score-order routing in place of headroom-sorted. - Retry failover: add a paragraph on the ctx-err classification fix (per-request HTTP timeouts no longer crash the indexer via a mis-matched context.Canceled unwrap). - Worst-case amplification: update to N=3 endpoints, batch=50 giving ~108 requests (was 60 at batch=20, N=2). Reflects the common owned-infra deployment shape. - "Finding the capacity of a node": clarify that in owned-infra mode there's no per-endpoint cap to pin; the curve just informs process-wide fetch_workers sizing. - "Performance knobs": rewrite fetch_workers and fetch_batch_size descriptions (new defaults, new adaptive behaviour — only errors trigger shrink now). Add database.pool_size as a top-level perf knob since the writer pool derives from it, with guidance to watch pg_stat_activity for saturation.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A multi-layer throughput push for operators running owned RPC nodes who don't need good-citizen throttling. Baseline → steady state:
Eight commits, grouped into five workstreams below. Each commit message has the full rationale + before/after metrics; this description is the tour.
1. Good-citizen constraints were leaving throughput on the table
The indexer had three layers of throttling designed for shared/public RPCs: AIMD per-endpoint concurrency caps, a 5-second saturation wait, and a sizer that shrunk batches on tail-latency alone. For operators running owned infra they're comfortable overwhelming, all three pinned throughput at a tiny fraction of upstream capacity.
perf(rpc): drop per-endpoint AIMD cap; route by p50 × in-flight— remove the AIMD controller, the saturation wait, theConcurrencyBudgetfield. Route byp50 × (1 + in_flight)score — fastest least-busy endpoint wins naturally. Unmeasured endpoints get a default p50=100ms so new nodes get exploration traffic.perf(rpc)+ui: moderate error-rate routing penalty; ETA on handler card— multiply the routing score by(1 + errRate × 3)so a broken endpoint biases traffic away without quarantining a briefly-degraded one. First tried ×10, over-corrected; ×3 sticks. Handler card also grows an ETA field.perf: writer pool sizing, static-batch, 30m MV, ctx-err classification— also drops the adaptive sizer's tail-latency hot signal. Only error rate shrinks now.2. Connection layer was costing a handshake per request
Each RPC/REST client used its own
http.Transport. DefaultMaxIdleConnsPerHost = 2meant 94 of every 96 fetches paid a TCP+TLS handshake. TLS session tickets weren't cached anywhere.perf(rpc): shared http.Transport with HTTP/2 + TLS session cache— newinternal/rpc/transport.gowith a process-wideSharedTransport:MaxIdleConnsPerHost=256, HTTP/2 forced,ClientSessionCache=LRU(512), tuned dial/keepalive timeouts. Every client wraps the same Transport now.3. Fetch and persist were gating each other
A worker used to do fetch → tx → persist → commit serially. Under write pressure, fetchers sat idle behind COPYs; under fetch pressure, writers sat idle waiting on producer.
perf(pipeline): split fetch + persist into decoupled worker pools— producer → jobs → N fetchers → writeCh → M writers → DB. Fetch runs up to the upstream ceiling, write runs up to the DB ceiling, neither gates the other. Bisect on fetch failure stays; persist failures record a dead-letter (no bisect; persist errors aren't block-specific).4. Write path was itself the bottleneck once fetch was unblocked
After #3, writes became the limit. Three internal bottlenecks:
pool_size = 32,writerCount = pool_size / 2 = 16. Half the pool was reserved but unused. Bumped writer count topool_size - 4and pool to 80 (via config) in commitperf: writer pool sizing, ....pg_advisory_xact_lock(hashtext(handler))during range merge. Inperf(writer): lock-free ranges + aggregate writes + empty-skip + compactorthis became a singleINSERT … ON CONFLICT DO UPDATEper writer (no lock), plus an async background compactor that runsrange_aggevery 30s to merge fragmented rows.synchronous_commit=off. Same commit adds a writer that drains up to 32 writeJobs in a 50ms window and commits them as one tx.Also in the same commit: skip
handler.Persistwhen the dispatched event slice is empty — historical-no-event blocks still mark their range covered viaRecordRangebut don't pay the no-op Persist call.5. Misc: MV schedule, ctx-err bug, dashboard bugs
MV cron */5 → */30. On 400M+ rowrelay_payments,REFRESH MATERIALIZED VIEW CONCURRENTLYtakes 5-10 min; a 5-min schedule piled up refreshes that serialised on MV row locks, starving writers. Downstream consumers accept 30-min freshness.processWithSplittreated per-request HTTP timeouts as fatal. Go'shttp.Client.Timeoutwrapscontext.Canceled, soerrors.Is(err, context.Canceled)was true for BOTH shutdown and request timeout — the "shutdown" branch propagated and crashed the whole indexer. Fix: checkctx.Err()on the OUTER context instead. BumpedResponseHeaderTimeout 60s → 120sfor slow archive queries.fix(web): DB-size cache + handler "blocks done / total" line— status endpoint'spg_total_relation_sizespiked to multi-second under checkpoint/autovacuum; 30s cache mirrors SchemaInfo. New "X / Y blocks · ETA" line per handler card.fix(web): restore const ranges = h.ranges declaration— removed therangesdeclaration in a later edit; later code still referenced it → runtime ReferenceError → blank page. Restored.Files touched
Test plan
go build ./...,go vet ./...,go test ./...all green.RecordRange+ compactor verified: indexer_ranges row count peaks, then drops ~10× after the 30-second compactor tick.Deliberately not in this PR
Four levers that would push further but aren't included:
relay_paymentsby block_height — partitioned PK scales sub-linearly. Requires migration of existing rows. Biggest remaining unshipped win.tx_searchbefore fetch — skip blocks known to have no matching events. Requires producer rewrite.max_concurrencyconfig for operators running against shared public gateways who still need a hard cap. Routing handles uneven per-node speed but won't defend an upstream from flooding.