Skip to content

Parallel-tier backfill, duplicate-URL UI, Dict FK-race fix - #6

Merged
nimrod-teich merged 3 commits into
mainfrom
feat/parallel-backfill-and-dict-fix
Apr 18, 2026
Merged

nimrod-teich merged 3 commits into
mainfrom
feat/parallel-backfill-and-dict-fix

Conversation

@nimrod-teich

Copy link
Copy Markdown
Contributor

Summary

Three production-driven fixes discovered while running two lava-indexer stacks (mainnet + testnet) side-by-side on one host. Split into three commits so each can be reviewed / reverted independently.

1. feat(pipeline): split gaps at endpoint coverage boundaries

On a fresh backfill the producer walked a single giant gap bottom-up, so the archive endpoint did all the work while pruning endpoints sat at rps=0 until the cursor crossed their Earliest horizon.

runWindow now slices each gap at every endpoint's Earliest, so the existing multi-gap interleave producer can dispatch across tiers in parallel from block zero. Endpoints with Earliest=0 or Disabled don't contribute boundaries; non-*MultiClient clients (tests) fall through unchanged.

2. feat(web): disambiguate duplicate-URL endpoints

The dashboard keyed endpoint rows by URL and /api/history took ?url= — so a config with the same URL declared twice (e.g. one plain, one with lava-extension: archive header, which makes the node report a different earliest_block) rendered as a single row with mixed metrics.

overviewRows and detailCards are now keyed by array index, and /api/history?idx=N is the new canonical query. ?url= is kept for backward compat but returns the first match. Collapse-state localStorage keys include the index so same-URL rows remember their own layout.

3. fix(events): remove Dict in-memory cache

Dict.IDs cached the INSERT…RETURNING id before the caller's tx committed. Under parallel workers: worker A inserts provider P and caches id=5 inside tx_A; worker B's IDs() hits the cache and returns 5; tx_A rolls back; worker B's COPY relay_payments fails with violates foreign key constraint relay_payments_provider_id_fkey.

Observed: 14,892 persist failures clustered in heights 498632..696785 on testnet bootstrap, all FK violations on provider_id (+150 on chain_id). Every retry sweep churned them for no gain because the cached id kept pointing at a rolled-back row.

Fix: drop the cache + its sync.RWMutex; always round-trip INSERT…ON CONFLICT…RETURNING. With ~107 unique values across the chain's entire history and the UPSERT being idempotent and cheap, the correctness win dwarfs the saved round-trip.

Test plan

  • go build ./... clean, go vet ./... clean (done locally).
  • Deploy to mainnet + testnet, observe all pruning-endpoints go from in_flight=0/0 rps=0 to actively serving their coverage tier (verified in-session).
  • With a config that declares the same URL twice, dashboard renders two distinct rows with independent metrics / collapse state (verified in-session).
  • After fix-3 deploy, no new insert or update on table "relay_payments" violates foreign key constraint errors for blocks fetched post-deploy.
  • Pre-existing indexer_failures rows with FK-violation reasons may still churn until they hit retries=3 and flip permanent, or can be cleared manually to let the producer re-fetch.

On a fresh backfill the producer walks a single giant gap bottom-up,
so every batch below a pruning node's `Earliest` horizon is eligible
only to the archive endpoint. Pruning endpoints sit at rps=0 until
the cursor crosses their horizon — which for a multi-year chain can
take hours or days.

Slice each gap at each endpoint's `Earliest` into per-tier sub-gaps
so the existing multi-gap interleave producer can dispatch across
tiers in parallel from block zero. Endpoints whose Earliest is 0 or
which are Disabled do not contribute boundaries. Non-MultiClient
clients (tests) fall through unchanged.

Observed effect on testnet: the two pruning endpoints went from
`in_flight=0/0 rps=0` to actively serving the upper-coverage tier
while the archive endpoint churns the historical tail.
The dashboard keyed endpoint rows and cards by URL, and
/api/history took ?url= for lookup. If an operator configured the
same URL twice — e.g. one entry with and one without a
lava-extension: archive header, which makes the node report a
different `earliest_block` horizon — the two entries collided into
a single UI row with metrics mixed across the two backend endpoint
objects.

Key the two DOM maps (overviewRows, detailCards) by array index,
and add ?idx=N to /api/history. The URL form of history is kept
as a backward-compat fallback but returns the first matching
endpoint, which is the pre-existing ambiguity. Collapse-state
localStorage keys now include the index, so two same-URL rows
remember their own layout independently.

No visual distinguisher between duplicates — the config author
chose to declare the same URL twice, so both rows just render
normally with whatever coverage window each reports.
Dict cached the INSERT…RETURNING id before the caller's transaction
committed. Under parallel batch workers this produced foreign-key
violations: worker A inserts provider 'P' and caches id=5 inside
tx_A; worker B's IDs() call hits the cache and returns id=5; tx_A
rolls back (batch error, deadlock, retry); worker B's COPY into
relay_payments references provider_id=5 and fails with
"violates foreign key constraint relay_payments_provider_id_fkey".

Observed scale on a testnet bootstrap: 14,892 persist failures
clustered in heights 498632..696785, all FK violations on
provider_id (and 150 on chain_id). All stuck as retrying at
retries≤3, churning the dead-letter sweep each cycle for no gain.

Fix: always round-trip INSERT…ON CONFLICT…RETURNING; drop the
cache and its sync.RWMutex. With ~107 unique values across the
chain's history and the UPSERT already being idempotent and cheap,
the correctness win dwarfs the saved round-trip.

Accumulated indexer_failures from before this fix will keep
FK-failing until retries=3 flips them permanent. Operators who
want them cleared immediately can DELETE them and let the producer
re-fetch into the corrected handler.
@nimrod-teich
nimrod-teich merged commit 1e42ba2 into main Apr 18, 2026
1 check passed
@nimrod-teich
nimrod-teich deleted the feat/parallel-backfill-and-dict-fix branch April 18, 2026 23:20
nimrod-teich added a commit that referenced this pull request Apr 18, 2026
…deadlock

processBatch's BeginTxFunc callback called p.state.HandlerNeedsRange,
which issues a pool query. With a pool connection already held for
the tx, every worker also tried to acquire a SECOND connection for
the gap read. Under N workers > pool_size the pool self-deadlocked —
every connection pinned to a tx waiting on a gap-read connection
that would never come.

Observed after PR #6's gap-split landed on a fresh backfill: at
pool_size=16, workers=64, pg_stat_activity accumulated 8+ rows of
"idle in transaction, wait_event=ClientRead, duration > 6 min" and
/api/status timed out on its own pool acquire, leaving the dashboard
blank.

The read is idempotent and doesn't need a tx — any state drift
between the read and the subsequent tx is handled by the advisory
lock + merge semantics in RecordRange. Compute the needs-map up
front; the tx body then just applies the writes + heartbeat.
nimrod-teich added a commit that referenced this pull request Apr 19, 2026
PR #6 removed Dict's in-memory cache to fix an FK race: worker A
inserted a row + cached its id pre-commit, B grabbed the id, A
rolled back, B's COPY referenced a non-existent id. The fix made
every IDs() call hit Postgres for the full set every batch — at 50
batches/sec × ~100 unique providers, that's hundreds of dictionary
upserts per second sitting in the critical path of every
relay_payment commit.

Restore the cache with strict commit-only discipline:

- New Warmup(ctx, pool) loads every committed (id, value) pair into
  the cache once at startup, after DDL.
- IDs() checks the cache first; misses go through the existing
  INSERT…DO NOTHING + SELECT path.
- ONLY rows returned from the SELECT fallback (already-committed in
  another tx, made visible by PG's conflict-resolution wait) are
  added to the cache. INSERT…RETURNING rows belong to the current,
  uncommitted tx and are NEVER cached — eliminating the rollback
  race that motivated the original cache removal.

Wire the optional events.Warmer interface so Pipeline-style
callers run Warmup once after DDL apply. relay_payment.Handler now
implements Warmer to warm both providers and chains.

Newly-introduced values pay one extra round-trip on the *next*
batch that needs them; from then on they hit the cache. Steady
state for a chain with ~100 providers/chains is ~99% hit rate, so
IDs() becomes a no-op for the hot path while the FK-race
correctness invariant is preserved.
nimrod-teich added a commit that referenced this pull request Apr 19, 2026
* fix(pipeline): hoist HandlerNeedsRange out of batch tx to break pool deadlock

processBatch's BeginTxFunc callback called p.state.HandlerNeedsRange,
which issues a pool query. With a pool connection already held for
the tx, every worker also tried to acquire a SECOND connection for
the gap read. Under N workers > pool_size the pool self-deadlocked —
every connection pinned to a tx waiting on a gap-read connection
that would never come.

Observed after PR #6's gap-split landed on a fresh backfill: at
pool_size=16, workers=64, pg_stat_activity accumulated 8+ rows of
"idle in transaction, wait_event=ClientRead, duration > 6 min" and
/api/status timed out on its own pool acquire, leaving the dashboard
blank.

The read is idempotent and doesn't need a tx — any state drift
between the read and the subsequent tx is handled by the advisory
lock + merge semantics in RecordRange. Compute the needs-map up
front; the tx body then just applies the writes + heartbeat.

* fix(events): switch Dict to ON CONFLICT DO NOTHING + fallback SELECT

Previous form used ON CONFLICT DO UPDATE SET col = EXCLUDED.col to
force RETURNING on conflict rows. The no-op UPDATE still takes a
row-level exclusive lock that's held until the caller's tx commits
— which in our case includes a heavy COPY into relay_payments
right after.

Under N parallel batch workers that all upsert the same ~60
provider addresses, every worker serialised on those 60 row locks
for the full batch duration. Observed in pg_stat_activity on
testnet: 8+ rows stuck at 30-48 seconds each, all on the providers
upsert, while the status handler starved on pool acquire and the
dashboard rendered blank.

DO NOTHING grabs the row lock only for the conflict check and
releases immediately. We follow with a plain SELECT for the
conflicted rows — MVCC-safe because Step 1's INSERT waits on any
concurrent uncommitted insert before returning, so by the time we
run Step 2 the committed row is visible. Two queries in the common
conflict case, one when all rows are new.

Drops /api/status latency from >10 s (timing out) to sub-500 ms
under fresh-backfill load.
nimrod-teich added a commit that referenced this pull request Apr 19, 2026
PR #6 removed Dict's in-memory cache to fix an FK race: worker A
inserted a row + cached its id pre-commit, B grabbed the id, A
rolled back, B's COPY referenced a non-existent id. The fix made
every IDs() call hit Postgres for the full set every batch — at 50
batches/sec × ~100 unique providers, that's hundreds of dictionary
upserts per second sitting in the critical path of every
relay_payment commit.

Restore the cache with strict commit-only discipline:

- New Warmup(ctx, pool) loads every committed (id, value) pair into
  the cache once at startup, after DDL.
- IDs() checks the cache first; misses go through the existing
  INSERT…DO NOTHING + SELECT path.
- ONLY rows returned from the SELECT fallback (already-committed in
  another tx, made visible by PG's conflict-resolution wait) are
  added to the cache. INSERT…RETURNING rows belong to the current,
  uncommitted tx and are NEVER cached — eliminating the rollback
  race that motivated the original cache removal.

Wire the optional events.Warmer interface so Pipeline-style
callers run Warmup once after DDL apply. relay_payment.Handler now
implements Warmer to warm both providers and chains.

Newly-introduced values pay one extra round-trip on the *next*
batch that needs them; from then on they hit the cache. Steady
state for a chain with ~100 providers/chains is ~99% hit rate, so
IDs() becomes a no-op for the hot path while the FK-race
correctness invariant is preserved.
nimrod-teich added a commit that referenced this pull request Apr 19, 2026
* fix(docker): require scram-sha-256 on pg_hba bridge subnets

The shipped pg_hba.conf granted passwordless `trust` to 0.0.0.0/0 and
::/0. Combined with the host-only port-forward, off-host attackers
were blocked but any other container on the docker bridge network
(graphql, a future side-car, an attacker-controlled image pulled via
supply-chain) could open a superuser session with no credentials.

Drop the global rules. Keep loopback `trust` so a host-side
`psql -h 127.0.0.1` via the compose port-forward still works without
a password. Bridge-subnet ranges (172.16/12, 192.168/16, 10/8) now
require scram-sha-256 — the indexer/graphql containers already
receive DB_PASSWORD via env so they authenticate normally.

Strengthen the file header to make the dev-only scope explicit and
to direct operators to ship a separate pg_hba.prod.conf rather than
copying this file into a production image.

* fix(config): drop default DB password from every shipped surface

Four places shipped `postgres` as the password default: the
example config that the Dockerfile bakes into the image, the
.env.example, the compose interpolation defaults for both the
postgres and indexer services, and the Database defaults in
internal/config. An operator running `docker compose up` with no
.env brought up Postgres listening with the most-guessed default
credential pair. With the new pg_hba enforcing scram-sha-256 on the
bridge subnets, the password is now actually load-bearing — and a
shipped default would defeat the HBA fix from the previous commit.

- config.example.yml: drop the literal `password:` line; comment
  explains it's env-only.
- .env.example: blank DB_PASSWORD with a "set this" comment.
- docker-compose.yml (postgres + indexer + graphql): replace
  `${DB_PASSWORD:-postgres}` with `${DB_PASSWORD:?must be set}` so
  compose fails fast instead of bringing the stack up with weak
  creds.
- internal/config/config.go: drop the `Password: "postgres"` default
  from the Database struct.

The Dockerfile still bakes config.example.yml at build time, but
without the password line — operators must inject DB_PASSWORD via
env or a mounted secret.

* fix(pool): size pgxpool to resolved fetch_workers + web headroom

Previously newPool ran before the adaptive sizer resolved the worker
count, so MaxConns came from the config default of 8 even when
fetch_workers ended up at 100+. Each worker holds a pool connection
for its full per-batch transaction, so anything below fetch_workers
was a hard ceiling: at fetch_workers=16 / pool_size=8 throughput
was capped at 50% of nominal; at adaptive ~100 workers it fell to
8%. The dashboard and aggregate refreshes shared the same starved
pool.

Move newPool down past the adaptive-sizer block so it sees the
resolved cfg.Indexer.FetchWorkers. In newPool, derive
MaxConns = max(configured_pool_size, fetch_workers + headroom),
where headroom (4) covers the embedded web UI and aggregates.
Operators who explicitly under-size the pool get a WARN log and
the auto-grown value, not a silent throughput collapse.

* fix(events): commit-only Dict cache restores hot-path savings

PR #6 removed Dict's in-memory cache to fix an FK race: worker A
inserted a row + cached its id pre-commit, B grabbed the id, A
rolled back, B's COPY referenced a non-existent id. The fix made
every IDs() call hit Postgres for the full set every batch — at 50
batches/sec × ~100 unique providers, that's hundreds of dictionary
upserts per second sitting in the critical path of every
relay_payment commit.

Restore the cache with strict commit-only discipline:

- New Warmup(ctx, pool) loads every committed (id, value) pair into
  the cache once at startup, after DDL.
- IDs() checks the cache first; misses go through the existing
  INSERT…DO NOTHING + SELECT path.
- ONLY rows returned from the SELECT fallback (already-committed in
  another tx, made visible by PG's conflict-resolution wait) are
  added to the cache. INSERT…RETURNING rows belong to the current,
  uncommitted tx and are NEVER cached — eliminating the rollback
  race that motivated the original cache removal.

Wire the optional events.Warmer interface so Pipeline-style
callers run Warmup once after DDL apply. relay_payment.Handler now
implements Warmer to warm both providers and chains.

Newly-introduced values pay one extra round-trip on the *next*
batch that needs them; from then on they hit the cache. Steady
state for a chain with ~100 providers/chains is ~99% hit rate, so
IDs() becomes a no-op for the hot path while the FK-race
correctness invariant is preserved.

* fix(compose): bind unauthenticated UI to loopback by default

docker-compose.yml exposed the indexer's web port on every host
interface (`${INDEXER_PORT:-8080}:8080` with no bind address),
unlike the postgres binding which is already loopback-only. The web
UI / /api/status / GraphQL proxy are UNAUTHENTICATED and disclose
endpoint URLs (which can carry tokens), dead-letter reasons, and
per-handler row counts. Anyone reachable on the host's network
could enumerate the indexer fleet.

Default the bind to 127.0.0.1 via a new INDEXER_BIND env (matching
the postgres pattern). Operators who front the indexer with a
reverse proxy + auth can set INDEXER_BIND=0.0.0.0; everyone else
gets the safe default.

Adding actual auth (basic / token) is intentionally NOT in this
commit — it's a behavioural change that wants a config decision
about credential storage. Loopback binding is the immediate
non-breaking mitigation.

* fix(graphql): default proxy to disabled

The indexer's web server reverse-proxied /graphql and /graphiql to
the upstream Postgraphile by default, and Postgraphile launches
with --no-ignore-rbac --enhance-graphiql over the entire `app`
schema with no auth headers required. Combined with the web UI
being unauthenticated, an attacker who reached the indexer's port
could browse /graphiql and read every table — event history,
dead-letter reasons, run telemetry — with no challenge.

Flip the default to disabled in both internal/config defaults() and
the example config. Operators opt in (graphql.enabled: true or
GRAPHQL_ENABLED=true) after deciding how to gate access.

* fix(web): validate GraphQL proxy upstream against SSRF

The GraphQL reverse-proxy passed s.GraphQLUpstream straight through
url.Parse with no scheme or host validation, then handed the result
to httputil.NewSingleHostReverseProxy. A misconfigured
GRAPHQL_UPSTREAM (or any code-path that lets an attacker influence
that env) turned the indexer's web port into a SSRF proxy: the same
unauthenticated /graphql route that's intended for Postgraphile
could be aimed at the cloud metadata service, internal management
APIs, or an external attacker host.

Add parseProxyUpstream that requires http/https and rejects any
host that resolves to a non-private, non-loopback address.
host.IsPrivate covers RFC1918 + ULA; we additionally accept
loopback and link-local. Public IPs (including the metadata range
169.254.169.254 — link-local, allowed for safety because
file://etc/passwd was already blocked by the scheme check, and
169.254 is intentionally allowed by isPrivateOrLoopback only when
the operator deliberately configures a link-local target) are
refused with a WARN log and the proxy is skipped. The UI renders
without GraphQL when the upstream is rejected — no silent fallback.

* fix(web): tighten SSRF allowlist — drop link-local

The previous commit allowed link-local addresses, which includes
the IPv4 169.254.0.0/16 range that holds the cloud metadata
service (169.254.169.254 on AWS/GCP/Azure) — the canonical SSRF
target. Loopback + RFC1918 + IPv6 ULA (covered by net.IP.IsPrivate)
are sufficient for legitimate same-stack proxy targets. Drop
link-local from the allowlist.

* fix(pool): redact DSN password from pgxpool errors + escape ConnString

Two related leakage paths:

(1) pgxpool.ParseConfig errors include the raw DSN — and therefore
the password — in their messages (well-known pgx behaviour:
"cannot parse url postgres://user:hunter2@…"). newPool returned
that error verbatim and main.go fed it to slog.Error("connect
postgres", "err", err), landing the password in the JSON log
stream. Common trigger: a malformed DB_PORT or DB_HOST env var.

(2) Database.ConnString built the URL via fmt.Sprintf, so a
password containing `@`, `?`, `/`, `#`, `%` (legal and common in
managed-DB credentials) corrupted the URL parse — at best
producing the leak in (1), at worst silently downgrading SSL when
a password ended in `?sslmode=`.

Switch ConnString to net/url with url.UserPassword(), which
percent-escapes the userinfo correctly. Wrap the ParseConfig error
with a redactSecret helper that strips both the literal and
URL-escaped forms of the password before the error reaches any log
sink.

* fix(pipeline): redact URLs and bodies from dead-letter reasons

processWithSplit wrote truncate(err.Error(), 250) into
indexer_failures.reason. Two typed RPC errors carry sensitive
material in their Error() output: HTTPStatusError includes up to
300 chars of upstream response body (which can echo query-string
tokens or hostnames), and HeightPrunedError carries the full
endpoint URL — paid LCD providers commonly tokenise auth via path
(`https://lava.example.com/v1/<api-key>/...`). The reason column
is then surfaced unauthenticated on /api/status, so any DB reader
or any unauthenticated UI viewer could harvest those tokens.

Add safeFailureReason that recognises the typed errors and
returns status-only forms; generic errors (network classifications,
context errors) fall through unchanged and don't carry creds.
Replace both truncate(err.Error(), 250) call sites in
processWithSplit with safeFailureReason.

* fix(rpc): delete dead EndpointSem (per-acquire goroutine leak risk)

EndpointSem was a sync.Cond+goroutine-based budget gate intended to
mediate per-endpoint concurrency. It carries two real defects:

(1) Acquire spawns a fresh forwarder goroutine on every wait
iteration just to translate ctx.Done into cond.Broadcast. Only the
acquirer that returns from cond.Wait closes its `done` channel —
every other waiter still holds a live forwarder until ITS Wait
returns. Under sustained budget shrinks (AIMD 64→4 under errors)
with N waiting workers, every wake-up leaks N-1 goroutines per
cycle.

(2) The whole type is dead code. multi.go's FetchBlocks does its
own poll-based budget gate; nothing in the production path calls
Acquire/Release. Resize is the only method called externally
(every controller tick). Future readers see the type, can't find a
caller, and either delete it (correct) or assume it's wired
(wrong).

Delete EndpointSem entirely from adaptive.go. Drop the `sem`
field on Endpoint, the NewEndpointSem call in
NewMultiWithOptions, and the Resize() call in the controller tick
loop. Tests still pass green.

* fix(state): wrap startup DDL in transactions

state.Ensure ran the schema/ranges DDL, the runs DDL, and the
failures DDL in three separate pool.Exec calls — each its own
implicit autocommit. A failure mid-bring-up (network blip, lock
contention, syntax mismatch on a future PG version) left the DB
in a half-migrated state that the next startup didn't detect.
The handler-DDL loop in main.go had the same shape: per-statement
autocommit with no boundary.

Move both into pgx.BeginTxFunc so each handler's DDL slice — and
the state-bring-up trio — apply atomically. CREATE TABLE/INDEX IF
NOT EXISTS and ALTER TABLE … ADD COLUMN IF NOT EXISTS are all
transaction-safe in PG. Renamed ensureRunsTable/ensureFailuresTable
to *Tx variants that take an existing pgx.Tx so Ensure can drive
the whole bring-up off one tx.

This is the smallest step toward proper migration discipline; a
schema_migrations table is still TODO once a non-additive change
arrives.

* fix(pipeline): pass resolved knobs explicitly; stop mutating cfg

main.go used to write back the adaptive sizer's resolved
fetch_workers and fetch_batch_size into cfg.Indexer so the pipeline
could see them via its config read path. That mutation made
cfg.Indexer lie about what the operator typed: anyone reading the
config (logs, /api/status, future "show effective config" handler)
saw the resolved value, not the YAML one. The pattern compounds —
a future "config drift detector" would compare against values that
main silently overwrote.

Pipeline now carries explicit `workers` and `batchMax` fields,
defaulted from cfg.Indexer at construction and overrideable via
WithRuntimeKnobs(workers, batchMax). main.go computes the
resolved values into local variables, passes them in via
WithRuntimeKnobs, and stops touching cfg. newPool similarly takes
resolvedWorkers as an explicit param instead of reading the
mutated cfg.

cfg.Indexer is now treated as input-only after Load — operator
intent stays observable, and the pipeline's runtime knobs are an
independent thing reachable via Pipeline.Workers/BatchMax.

* fix(pipeline): cap dead-letter sweep at one FailureRetryInterval

Default FailureRetryBatch=200 + ~500 ms per retry = 100 s of work,
but FailureRetryInterval defaults to 60 s. The previous code drained
PopFailures() heights serially without watching the clock, so every
sweep that drew a full batch overlapped the next tick. Two
consequences:

1. Producer + sweep both took the per-handler advisory lock inside
   RecordRange. Two long-running tx's sharing one
   pg_advisory_xact_lock degrade to FCFS — the entire commit loop
   slowed by 10-60 ms per sweep retry, throttling the main indexer
   to ~1 bps for the duration.
2. A chronic 10 k-row dead-letter pool cycled the same heights
   forever — sweeps overlapped indefinitely, and the user-supplied
   "7 bps process ceiling" memory note explicitly mentions the
   regime where dead-letter amplification dominates.

Cap each sweep at one FailureRetryInterval. When the deadline
expires mid-sweep, log a WARN with `remaining` and bail; the next
tick picks up the leftovers cleanly. Heights stay in
indexer_failures across the deferral — PopFailures only removes
ones we actually retried — so no progress is lost.

* fix(state): cache SchemaInfo for 30s to dampen UI poll cost

internal/state/schema.go runs four pg_catalog queries per
SchemaInfo call. The schemaIndexes one is the heaviest: a row-level
subquery calls pg_get_indexdef once per indexed column, ~750 calls
on a ~30-table schema = ~40 ms of pg-side CPU. As soon as this is
wired to a polled UI endpoint (the obvious next step), every
browser tab adds another 12-20 polls/min, all hitting the same
shared 8-conn pgxpool that the indexer needs.

Add a 30 s in-process cache on State (sync-protected). The schema
doesn't change inside a runtime, so cache invalidation is purely
time-based — no callback needed. Steady-state cost: one
sub-millisecond map read per call, with the heavy work deferred
until the next 30 s window starts.

A deeper fix (rewriting the indexes query to fold the per-column
pg_get_indexdef into a single call, and giving schema queries a
separate read-only pool) is the bigger structural change the
review flagged; the cache is the immediate win.

* fix(state): bound HandlerNeedsRange scan to overlapping ranges only

HandlerNeedsRange called Gaps→Ranges, which read every
indexer_ranges row for the handler and ran the gap math
client-side. Inside processBatch this fires once per handler per
batch — at 50 batches/sec on a year-long backfill with ~10 k
handler ranges, that's ~8 MB/sec of repeated scans (and the bytes
sit on a pgxpool conn for the duration of every per-batch tx).
Per-query latency grows linearly with the ranges table.

Replace with a bounded scan that pulls only ranges overlapping
[from, to] (`from_height <= to AND to_height >= from`), uses the
existing (handler, from_height) PK / (handler, to_height) index,
and walks the typically-1-row result client-side to detect gaps.
For the common cases (fully covered → 1 row, fully uncovered → 0
rows) the cost is now constant in the total ranges count.
nimrod-teich added a commit that referenced this pull request Apr 19, 2026
…actor

Four changes shipped together (#2, #3, #6 from the optimisation list):

### #3 — Remove advisory lock from RecordRange

Before: every writer acquired pg_advisory_xact_lock(hashtext(handler)),
then SELECT FOR UPDATE'd touching ranges, then DELETE+INSERT the merged
row. All 76 writers on the same handler (lava_relay_payment) serialised
on that lock. On pg_stat_activity under load the lock wait dominated.

After: RecordRange is a single INSERT … ON CONFLICT DO UPDATE. No lock,
no merge, no coordination between writers. Writers insert raw ranges
(possibly overlapping), which accumulate until...

### Async range compactor

A new goroutine per indexer process wakes every 30 s and for each
handler runs `range_agg(int8range(...))` server-side to compute the
minimum-rows representation, then swaps the rows in a single tx. Writers
that race with compaction insert rows that survive and are picked up
next cycle — one tick of extra fragmentation max.

State.CompactRanges is public so ops can trigger it manually if
needed. Logs `range compactor handler=X rows_merged=N` on non-zero
reductions.

### #2 — Aggregate N writeJobs per tx

Before: one writer = one goroutine = one tx per writeJob. With per-tx
overhead (BEGIN, heartbeat, COMMIT) around 5-10 ms even with
synchronous_commit=off, this caps throughput at ~writerCount × 100 txs/s
no matter how fast individual batches are.

After: each writer drains up to 32 jobs from writeCh with a 50 ms wait
budget, then commits them in one tx. For each job: per-handler Persist
(one COPY per handler), per-job RecordRange call, one shared Heartbeat
at the tx tail. Amortises the per-tx cost across up to 32 batches.

On failure the whole aggregate gets recorded as dead-letter — a
coarser blast radius than per-batch, but persist errors are rare
post-Dict-fix and the dead-letter sweep re-fetches anyway.

### #6 — Skip Persist call on empty-event batches

For historical segments with no relay_payment events, each handler's
dispatched event slice is empty. Persist is a no-op in that case but
still costs interface dispatch + tx bookkeeping. Skip directly to
RecordRange so the range still gets marked covered without the Persist
call.

### Dashboard fix: >100% display

The "X / Y blocks" line summed range lengths client-side; with
overlapping raw ranges from the new lock-free RecordRange, sum >
actual coverage → displayed >100% (user-reported testnet showing
100.00% when server-side window_pct was 78.6%). Switch to deriving
absolute counts from the server's authoritative window_pct:
doneInWindow = windowTotal × (window_pct / 100). Correct regardless
of fragmentation.

### Observed

Mainnet (near tip): status bps 270, row delta 12,395 rows/s. At
historical segments seen peaks of 600+ bps. Throughput no longer
capped by advisory-lock serialisation — writers can run fully
parallel across the 76-worker pool.

Testnet (heavy historical with upstream 502s): 30-75 bps depending on
segment, but row delta ~93k rows/s at times — DB-write-bound, not
lock-bound.

Aggregation actual factor observed: `agg:1` in many progress logs.
With 76 writers and a 50 ms window, the first writer often drains 1
job before the window expires — job arrival rate / writers ≫ 1 per
50 ms. Aggregation becomes visible under burst arrivals; under
steady-state uniform arrival rate the factor stays ~1. Kept as the
structural change it enables (one-tx-many-jobs is correct under any
rate, the ceiling just rises with burstiness).
nimrod-teich added a commit that referenced this pull request Apr 19, 2026
…tes (#10)

* fix(web): DB-size cache + handler "blocks done / total" line

/api/status used to run SUM(pg_total_relation_size(...)) on every poll.
On multi-hundred-million-row tables that query can spike to multi-
second under checkpoint or autovacuum pressure, blocking the dashboard
(observed: /api/status going from ~150 ms to >10 s with HTTP 500
"context deadline exceeded"). Mirror the SchemaInfo 30 s cache
pattern: recompute at most once per 30 s, fall back to the last
cached value on query failure. DB size drifts by a few MB per second
under normal indexing load, so a 30 s staleness is invisible to
operators.

Handler card: new "X / Y blocks" line under the window/chain
percentages. X = sum of (range.to − range.from + 1) clipped to the
configured window; Y = windowEnd − windowStart + 1. Matches the same
denominator window_pct uses, so the percent and the absolute count
agree at a glance.

Also fixed a JS regression: introducing `const ranges = h.ranges ||
[]` at the top of updateHandlerCard collided with a pre-existing
declaration later in the same function — duplicate-const is a
parse-time SyntaxError which kills the whole inline <script> and
renders the dashboard blank. Removed the duplicate declaration.

* perf(rpc): shared http.Transport with HTTP/2 + TLS session cache

Every RPCClient and RESTClient used to build its own http.Transport.
Two concrete costs:

1. TCP keep-alive pool wasn't shared. Two clients pointing at the same
   host couldn't reuse each other's idle connections.
2. TLS session tickets / session IDs weren't cached anywhere — every
   reconnect paid a full TLS handshake.

With 96 fetch workers hitting one host and Go's default
MaxIdleConnsPerHost of 2, we were doing TCP+TLS handshakes on roughly
every request. That's tens of milliseconds of CPU + round trips
burned per batch, repeated across every worker.

Introduce internal/rpc/transport.go with a package-level
SharedTransport that every client wraps. Tuning:

  MaxIdleConnsPerHost = 256        (was Go default 2)
  MaxIdleConns        = 1024       (pool ceiling)
  ForceAttemptHTTP2   = true       (single multiplexed connection when supported)
  ClientSessionCache  = LRU(512)   (TLS session resumption, skip handshake on reconnect)
  IdleConnTimeout     = 90s        (keep-alives stay warm between bursts)
  DialContext         = 5s connect + 30s tcp keep-alive
  TLSHandshakeTimeout = 5s
  ResponseHeaderTimeout = 60s      (archive queries can be slow on cold cache)

http.Client stays per-endpoint because Timeout is a Client-level
field, not Transport — but all Clients share the Transport, so the
connection pool and session cache are process-global.

Should substantially reduce handshake churn and enable HTTP/2
multiplexing on supporting upstreams. Expected: lower p50 latency on
every request, fewer concurrent TCP connections needed per worker
count, and a much faster first-request-after-idle.

* perf(rpc): drop per-endpoint AIMD cap; route by p50 × in-flight

The AIMD controller capped each endpoint's concurrency based on
p99/p50 latency ratio + error rate. For operators running shared
public gateways, this kept us from being a bad citizen — essential.
For operators running owned RPC nodes they're comfortable
overwhelming, the cap became throughput on the table: budgets pinned
at Min=4 with 90+ fetch workers idle, because the grow heuristic
required 70% utilization we never reached with uneven tier coverage.

For the owned-infra case the right primitive is NOT a cap, but a
routing signal — always send the next request to the fastest
currently-least-busy endpoint. If one slows down, its score rises
and traffic naturally shifts; if it recovers, traffic comes back.
No explicit cap, no stale floor pinning.

Changes:

- Delete internal/rpc/adaptive.go. AdaptiveController + its Tick /
  Grow / Shrink semantics are gone.
- Remove the `ctrl` field from Endpoint, `ConcurrencyBudget()` method,
  `InitialConcurrency` / `MinConcurrency` / `MaxConcurrency` /
  `TargetP99` from MultiClientOptions. MultiClientOptions now only
  carries `HistorySeconds`.
- Rename `StartController` → `StartMetricsRecorder`. Same 3s tick,
  but it only samples metrics into the history ring; no controller
  side-effects.
- Rewrite `MultiClient.FetchBlocks` routing: rank eligible endpoints
  by `p50 × (1 + in_flight)` (lower is better). A slow-or-queued
  endpoint gets deranked; a fast idle endpoint wins. Unmeasured p50
  is treated as 100 ms so new endpoints get exploration traffic. Ties
  break via the existing round-robin counter.
- Remove the 5-second saturation-wait loop. Without a per-endpoint
  cap, "saturated" is no longer a real state at the routing layer —
  the score function handles pressure instead.
- Drop the AIMD-driven worker-cap term from pipeline/sizer.go: the
  worker cap is now purely host-resource-limited (CPU × 8, mem /
  2 MiB, operator hard-max). `EndpointSignal.Budget` field removed.
- Retain `MetricSample` shape (drop Budget), remove Budget from
  `EndpointMetricsJSON`.
- Dashboard: drop the Budget stat card and Adaptive-budget sparkline.
  Replace with a p50 latency sparkline (directly useful now that p50
  is the routing signal). Update tooltip text to describe the new
  routing rule and note there's no concurrency cap.

Expected impact: throughput climbs until the DB or the host run
out of room — whichever comes first. Bad upstreams don't cause a
retry-storm because the routing layer already routes around them
via the p50/in-flight score; errors still trigger failover via the
unchanged fast-fail path (429/5xx → next endpoint immediately).

Guidance for operators still on shared public gateways: set
`fetch_workers` modestly in config (or let the host-based cap do it
implicitly). If a specific endpoint needs a hard per-connection
limit, re-introduce a per-endpoint `max_concurrency` config field —
the structure of MultiClient supports it cleanly, just wasn't needed
for the owned-infra target.

* perf(pipeline): split fetch + persist into decoupled worker pools

Each worker used to do fetch + persist serially: pull a job, call
FetchBlocks, open a tx, COPY, commit, advance. Under write-saturated
load — which is normal on a 486M-row table with FKs + BRIN + index
maintenance — fetchers sat idle behind COPYs. With 96 fetch workers
and pool_size=32 there was nothing the concurrency could even do:
half the workers were asleep waiting for their peer to finish writing.

Split into two decoupled pools connected by a small buffered channel:

    producer → jobs chan → fetchers (N = fetch_workers)
                              ↓
                          writeCh (buf 2 × writerCount)
                              ↓
                          writers (M = pool_size / 2, min 8) → DB

Fetchers fetch, Dispatch events, compute HandlerNeedsRange, and hand
the parsed batch to writeCh. No DB tx is held by a fetcher — the
brief pool connection for HandlerNeedsRange releases before enqueue.

Writers pull from writeCh, open one tx per batch, Persist + RecordRange
+ Heartbeat, commit. If persist fails, the writer records the whole
batch as a dead-letter (no bisect — persist errors are rarely
block-specific) and moves on. The dead-letter sweep will refetch.

Why this helps:

- Fetch throughput is now bounded by the upstream ceiling + fetch
  worker count, not by DB write speed. Previously a slow commit
  blocked the fetch it was paired with.
- Write throughput is bounded by pool_size (via writerCount), which
  is the correct DB-side constraint. Writers always have work queued
  (up to 2 × writerCount) so they never idle during producer gaps.
- Progress log + blocks/sec rate now reflect COMMITTED blocks, not
  fetched-but-maybe-not-written. Honest dashboard numbers.
- Bisect on fetch failure stays in the fetcher (unchanged).
- processWithSplit signature gains a writeCh chan<-; dead-letter sweep
  plumbs the same one through so retries go through the split flow
  too.

Default fetch_batch_size bumped 10 → 50. With per-endpoint caps gone
and the split pipeline smoothing bursts, batch overhead matters more
than queue depth; 50 heights per JSON-RPC batch cuts per-request
overhead ~5× on the wire. Tendermint batch responses at this size
are still under 2 MB, comfortably within any upstream's buffers.

Shutdown ordering: producer closes jobs chan → fetchers drain,
signal via WaitGroup → separate goroutine closes writeCh after last
fetcher exits → writers drain, exit. errgroup.Wait() resolves when
all participants have returned.

* perf(rpc)+ui: moderate error-rate routing penalty; ETA on handler card

Two small refinements after deploying the uncap + latency-routing
change:

1. rpc: reduce the error-rate penalty on the routing score from
   ×10 to ×3. The 10× version was aggressively quarantining any
   endpoint that once had errors — because errors flow into a
   cumulative TotalErrors/TotalRequests ratio that decays slowly
   even as recent errors stop. A briefly-degraded endpoint ended
   up pinned off rotation until its 500-entry latency ring rotated
   out its old slow measurements, which at low traffic took
   minutes. ×3 is a gentler weighting: 50% err = 2.5× penalty,
   100% err = 4× penalty — enough to bias away from a broken
   endpoint, not enough to keep a recovering one off the roster.

2. ui: handler card adds ETA to the "X / Y blocks" line, e.g.
   "3.21M / 4.91M blocks · ETA 2h 15m". Computed as
   (windowTotal − doneInWindow) / blocks_per_sec, formatted via
   the existing formatEta helper. Shows "done" when remaining is
   zero and "—" when bps is zero (warming up).

* perf: writer pool sizing, static-batch, 30m MV, ctx-err classification

Experimental round chasing a 3× throughput goal uncovered four real
bottlenecks + one bug. All shipped here.

1. Writer pool was half the DB pool.
   pipeline.go: writerCount = pool_size / 2 → pool_size - 4. DB write
   was the bottleneck and we had half the pool deliberately unused.
   Reserving 4 connections for status handler + heartbeat + sweep +
   fetcher's HandlerNeedsRange still gives ~95% of pool to writers.
   Mainnet 53.6 → 70.6 bps (+32%) standalone.

2. DB pool was too small for the new worker count.
   Operator config bumped pool_size 32 → 80 (deploy-time, via .env).
   max_connections default is 100 in our postgres image; 80 indexer +
   ~10 postgraphile + ~5 internal fits comfortably. Writers went from
   saturated at pool_size to comfortably below.
   With #1 together: mainnet 53.6 → 208 bps (+289%), hit 3× target.

3. Adaptive batch sizer kept shrinking on tail-latency alone.
   sizer.go: drop the p99/p50 > 10 hot signal and the p99/p50 < 3 calm
   signal. For owned-infra operators, tail latency is a "slow but
   healthy" signal — routing handles it. ONLY error rate drives
   sizer shrink now. This matches the same decision made earlier in
   the AIMD-cap removal.

   pipeline.go: when operator sets fetch_batch_size > 0 explicitly,
   skip the sizer's batch state and use the static value. Previously
   the sizer ran whenever fetch_workers was adaptive (a separate
   signal), and its batch-state override silently ignored the
   operator's explicit batch-size config.

   With #1 + #2 + batch=50 static: mainnet 53.6 → 353 bps (+559%,
   observed up to 625 bps on log rolling average).

4. aggregates.yml: MV refresh schedule */5 → */30.
   On a 400M+ row relay_payments table, REFRESH MATERIALIZED VIEW
   CONCURRENTLY takes 5-10 min. At a 5-min schedule refreshes piled
   up and serialised on MV row-level locks, starving the writer pool.
   Dashboards read the MV for daily aggregates; 30 min freshness is
   acceptable and eliminates the lock contention.

5. Bug fix: processWithSplit treated per-request HTTP timeouts as
   fatal.
   pipeline.go: Go's http.Client.Timeout wraps context.Canceled when
   it fires on a slow upstream, so errors.Is(err, context.Canceled)
   returned true for BOTH "we were shut down" and "one request took
   too long." The clean-shutdown branch propagated the err and
   errgroup cancelled the WHOLE indexer. Bigger batches triggered
   this more often — went from latent to catastrophic on batch=100.
   Distinguish by checking ctx.Err() on the OUTER context; anything
   else falls through to bisect/dead-letter like any other transient
   fetch failure.

6. transport.go: ResponseHeaderTimeout 60s → 120s. Bigger batches
   against archive nodes occasionally take > 60s to respond; legit
   latency, not a failure signal.

Deployed to both stacks; mainnet at ~350-600 bps steady state, up
from ~53. Testnet has higher per-block row density and is now
DB-write-bound at ~47-230 bps depending on which historical segment
it's processing — still 2-4× baseline on a good window.

* perf(writer): lock-free ranges + aggregate writes + empty-skip + compactor

Four changes shipped together (#2, #3, #6 from the optimisation list):

### #3 — Remove advisory lock from RecordRange

Before: every writer acquired pg_advisory_xact_lock(hashtext(handler)),
then SELECT FOR UPDATE'd touching ranges, then DELETE+INSERT the merged
row. All 76 writers on the same handler (lava_relay_payment) serialised
on that lock. On pg_stat_activity under load the lock wait dominated.

After: RecordRange is a single INSERT … ON CONFLICT DO UPDATE. No lock,
no merge, no coordination between writers. Writers insert raw ranges
(possibly overlapping), which accumulate until...

### Async range compactor

A new goroutine per indexer process wakes every 30 s and for each
handler runs `range_agg(int8range(...))` server-side to compute the
minimum-rows representation, then swaps the rows in a single tx. Writers
that race with compaction insert rows that survive and are picked up
next cycle — one tick of extra fragmentation max.

State.CompactRanges is public so ops can trigger it manually if
needed. Logs `range compactor handler=X rows_merged=N` on non-zero
reductions.

### #2 — Aggregate N writeJobs per tx

Before: one writer = one goroutine = one tx per writeJob. With per-tx
overhead (BEGIN, heartbeat, COMMIT) around 5-10 ms even with
synchronous_commit=off, this caps throughput at ~writerCount × 100 txs/s
no matter how fast individual batches are.

After: each writer drains up to 32 jobs from writeCh with a 50 ms wait
budget, then commits them in one tx. For each job: per-handler Persist
(one COPY per handler), per-job RecordRange call, one shared Heartbeat
at the tx tail. Amortises the per-tx cost across up to 32 batches.

On failure the whole aggregate gets recorded as dead-letter — a
coarser blast radius than per-batch, but persist errors are rare
post-Dict-fix and the dead-letter sweep re-fetches anyway.

### #6 — Skip Persist call on empty-event batches

For historical segments with no relay_payment events, each handler's
dispatched event slice is empty. Persist is a no-op in that case but
still costs interface dispatch + tx bookkeeping. Skip directly to
RecordRange so the range still gets marked covered without the Persist
call.

### Dashboard fix: >100% display

The "X / Y blocks" line summed range lengths client-side; with
overlapping raw ranges from the new lock-free RecordRange, sum >
actual coverage → displayed >100% (user-reported testnet showing
100.00% when server-side window_pct was 78.6%). Switch to deriving
absolute counts from the server's authoritative window_pct:
doneInWindow = windowTotal × (window_pct / 100). Correct regardless
of fragmentation.

### Observed

Mainnet (near tip): status bps 270, row delta 12,395 rows/s. At
historical segments seen peaks of 600+ bps. Throughput no longer
capped by advisory-lock serialisation — writers can run fully
parallel across the 76-worker pool.

Testnet (heavy historical with upstream 502s): 30-75 bps depending on
segment, but row delta ~93k rows/s at times — DB-write-bound, not
lock-bound.

Aggregation actual factor observed: `agg:1` in many progress logs.
With 76 writers and a 50 ms window, the first writer often drains 1
job before the window expires — job arrival rate / writers ≫ 1 per
50 ms. Aggregation becomes visible under burst arrivals; under
steady-state uniform arrival rate the factor stays ~1. Kept as the
structural change it enables (one-tx-many-jobs is correct under any
rate, the ceiling just rises with burstiness).

* fix(web): restore const ranges = h.ranges declaration

Previous commit removed the `const ranges` from updateHandlerCard
because the new blocksLine logic (derived from window_pct) no longer
needed it. But segment rendering later in the same function still
references `ranges.length` and `ranges.forEach` — removing the
declaration made those lines throw ReferenceError at the first
handler render, and the whole dashboard went blank.

Symmetrical to the earlier 'duplicate const' bug where I ADDED a
redundant declaration. This time I removed a still-needed one. Added
a JS parse check to the deploy path but it only catches static syntax
errors, not runtime ReferenceErrors — worth adding a smoke test on
the rendered page in a follow-up.

* docs(readme): update to match the throughput-overhaul architecture

Touches that track the code changes in this branch:

- Tip-first parallel gap filling: add a "Fetch / persist are
  decoupled" subsection explaining the two-pool architecture and the
  endpoint-coverage tier slicing. Remove the sentence about writers
  taking pg_advisory_xact_lock — now lock-free with the async
  compactor.

- "How much load are we pushing": update defaults (batch 10 → 50),
  drop the "live headroom" language (now p50-based routing), remove
  references to the AIMD budget in the endpoint-row example.

- Delete the "Adaptive concurrency (AIMD)" section. Replace with
  "Endpoint routing" explaining the
  score = p50 × (1 + in_flight) × (1 + errRate × 3) rule + the
  fast-fail failover path on 429/5xx.

- Retry layer map: redraw as the fetch+write split with the writeCh
  handoff between pools. Score-order routing in place of
  headroom-sorted.

- Retry failover: add a paragraph on the ctx-err classification fix
  (per-request HTTP timeouts no longer crash the indexer via a
  mis-matched context.Canceled unwrap).

- Worst-case amplification: update to N=3 endpoints, batch=50 giving
  ~108 requests (was 60 at batch=20, N=2). Reflects the common
  owned-infra deployment shape.

- "Finding the capacity of a node": clarify that in owned-infra mode
  there's no per-endpoint cap to pin; the curve just informs
  process-wide fetch_workers sizing.

- "Performance knobs": rewrite fetch_workers and fetch_batch_size
  descriptions (new defaults, new adaptive behaviour — only errors
  trigger shrink now). Add database.pool_size as a top-level perf
  knob since the writer pool derives from it, with guidance to watch
  pg_stat_activity for saturation.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant