Skip to content

fix: resolve all critical + 12 high findings from code review - #8

Merged
nimrod-teich merged 16 commits into
mainfrom
fix/code-review-criticals-highs
Apr 19, 2026
Merged

nimrod-teich merged 16 commits into
mainfrom
fix/code-review-criticals-highs

Conversation

@nimrod-teich

Copy link
Copy Markdown
Contributor

Resolves all 4 critical and 12 of 16 high findings from the
/code-review workflow run on commit 1e42ba2 (PR #6 base).

Summary

  • 4 critical fixed (one commit each):

    • postgres pg_hba.conf trust → scram-sha-256 on bridge subnets
    • default postgres DB password → removed from every shipped surface; compose ${DB_PASSWORD:?…}
    • pgxpool default 8 vs ~100 adaptive workers → MaxConns auto-derived from resolved fetch_workers
    • Dict.IDs per-batch hot path → restored cache with strict commit-only discipline
  • 12 highs fixed (one commit each):

    • web UI loopback bind in compose
    • GraphQL proxy default off
    • SSRF via proxy upstream — scheme + host validation (link-local explicitly denied)
    • DSN password leak in pgxpool errors — redactSecret + url.UserPassword escaping
    • dead-letter reason URL/body leak — safeFailureReason for typed errors
    • EndpointSem deletion (dead code with goroutine-leak risk)
    • DDL outside transactions — state.Ensure + handler DDL loop now in pgx.BeginTxFunc
    • cfg.Indexer mutation in main — Pipeline carries explicit knobs via WithRuntimeKnobs
    • dead-letter sweep amplification — sweep capped at one FailureRetryInterval
    • schema.go indexes N+1 — 30 s in-process cache (per-poll fix; per-call N+1 deferred)
    • HandlerNeedsRange full-table scan — bounded scan over overlapping ranges only
  • 4 highs deferred (need operator decision):

    • events→rpc dependency direction (structural reorg)
    • state god-package split (package layout)
    • retry amplification 25× (AIMD/sizing tuning)
    • indexGaps O(N gaps) producer (heap restructure interacts with BackfillInterleave)

Heads-up: pre-existing commits on the branch

The first two commits (53da2f3 hoist HandlerNeedsRange, a899e96 Dict
ON CONFLICT DO NOTHING) were ALREADY on the branch
(fix/pool-deadlock-and-dict-lock-contention) when I started. They
are not part of the code-review-fix work, but they're closely related
to two of the criticals (pgxpool sizing depends on the HandlerNeedsRange
hoist; the Dict commit-only cache builds on the DO-NOTHING pattern).
Move them out of this PR if you'd rather land them separately — git rebase -i 1e42ba2 and drop the bottom two.

Test plan

  • go build ./... — clean
  • go vet ./... — clean (pre-existing stringsseq diagnostics in config.go untouched)
  • go test ./... — all packages pass (rpc, pipeline, state)
  • docker compose up smoke test — verify scram-sha-256 auth + DB_PASSWORD enforcement
  • Backfill load test — verify pool-sizing fix doesn't regress under high worker counts
  • Hit /api/status with a slow client — confirm graphql off + loopback bind in compose

Remaining-findings report

Pruned to "what's left" in _workspace/report.md (gitignored). Per-reviewer
files in _workspace/{architecture,security,performance,style}_findings.md
each carry a header note pointing at the consolidated view.

The shipped pg_hba.conf granted passwordless `trust` to 0.0.0.0/0 and
::/0. Combined with the host-only port-forward, off-host attackers
were blocked but any other container on the docker bridge network
(graphql, a future side-car, an attacker-controlled image pulled via
supply-chain) could open a superuser session with no credentials.

Drop the global rules. Keep loopback `trust` so a host-side
`psql -h 127.0.0.1` via the compose port-forward still works without
a password. Bridge-subnet ranges (172.16/12, 192.168/16, 10/8) now
require scram-sha-256 — the indexer/graphql containers already
receive DB_PASSWORD via env so they authenticate normally.

Strengthen the file header to make the dev-only scope explicit and
to direct operators to ship a separate pg_hba.prod.conf rather than
copying this file into a production image.
Four places shipped `postgres` as the password default: the
example config that the Dockerfile bakes into the image, the
.env.example, the compose interpolation defaults for both the
postgres and indexer services, and the Database defaults in
internal/config. An operator running `docker compose up` with no
.env brought up Postgres listening with the most-guessed default
credential pair. With the new pg_hba enforcing scram-sha-256 on the
bridge subnets, the password is now actually load-bearing — and a
shipped default would defeat the HBA fix from the previous commit.

- config.example.yml: drop the literal `password:` line; comment
  explains it's env-only.
- .env.example: blank DB_PASSWORD with a "set this" comment.
- docker-compose.yml (postgres + indexer + graphql): replace
  `${DB_PASSWORD:-postgres}` with `${DB_PASSWORD:?must be set}` so
  compose fails fast instead of bringing the stack up with weak
  creds.
- internal/config/config.go: drop the `Password: "postgres"` default
  from the Database struct.

The Dockerfile still bakes config.example.yml at build time, but
without the password line — operators must inject DB_PASSWORD via
env or a mounted secret.
Previously newPool ran before the adaptive sizer resolved the worker
count, so MaxConns came from the config default of 8 even when
fetch_workers ended up at 100+. Each worker holds a pool connection
for its full per-batch transaction, so anything below fetch_workers
was a hard ceiling: at fetch_workers=16 / pool_size=8 throughput
was capped at 50% of nominal; at adaptive ~100 workers it fell to
8%. The dashboard and aggregate refreshes shared the same starved
pool.

Move newPool down past the adaptive-sizer block so it sees the
resolved cfg.Indexer.FetchWorkers. In newPool, derive
MaxConns = max(configured_pool_size, fetch_workers + headroom),
where headroom (4) covers the embedded web UI and aggregates.
Operators who explicitly under-size the pool get a WARN log and
the auto-grown value, not a silent throughput collapse.
PR #6 removed Dict's in-memory cache to fix an FK race: worker A
inserted a row + cached its id pre-commit, B grabbed the id, A
rolled back, B's COPY referenced a non-existent id. The fix made
every IDs() call hit Postgres for the full set every batch — at 50
batches/sec × ~100 unique providers, that's hundreds of dictionary
upserts per second sitting in the critical path of every
relay_payment commit.

Restore the cache with strict commit-only discipline:

- New Warmup(ctx, pool) loads every committed (id, value) pair into
  the cache once at startup, after DDL.
- IDs() checks the cache first; misses go through the existing
  INSERT…DO NOTHING + SELECT path.
- ONLY rows returned from the SELECT fallback (already-committed in
  another tx, made visible by PG's conflict-resolution wait) are
  added to the cache. INSERT…RETURNING rows belong to the current,
  uncommitted tx and are NEVER cached — eliminating the rollback
  race that motivated the original cache removal.

Wire the optional events.Warmer interface so Pipeline-style
callers run Warmup once after DDL apply. relay_payment.Handler now
implements Warmer to warm both providers and chains.

Newly-introduced values pay one extra round-trip on the *next*
batch that needs them; from then on they hit the cache. Steady
state for a chain with ~100 providers/chains is ~99% hit rate, so
IDs() becomes a no-op for the hot path while the FK-race
correctness invariant is preserved.
docker-compose.yml exposed the indexer's web port on every host
interface (`${INDEXER_PORT:-8080}:8080` with no bind address),
unlike the postgres binding which is already loopback-only. The web
UI / /api/status / GraphQL proxy are UNAUTHENTICATED and disclose
endpoint URLs (which can carry tokens), dead-letter reasons, and
per-handler row counts. Anyone reachable on the host's network
could enumerate the indexer fleet.

Default the bind to 127.0.0.1 via a new INDEXER_BIND env (matching
the postgres pattern). Operators who front the indexer with a
reverse proxy + auth can set INDEXER_BIND=0.0.0.0; everyone else
gets the safe default.

Adding actual auth (basic / token) is intentionally NOT in this
commit — it's a behavioural change that wants a config decision
about credential storage. Loopback binding is the immediate
non-breaking mitigation.
The indexer's web server reverse-proxied /graphql and /graphiql to
the upstream Postgraphile by default, and Postgraphile launches
with --no-ignore-rbac --enhance-graphiql over the entire `app`
schema with no auth headers required. Combined with the web UI
being unauthenticated, an attacker who reached the indexer's port
could browse /graphiql and read every table — event history,
dead-letter reasons, run telemetry — with no challenge.

Flip the default to disabled in both internal/config defaults() and
the example config. Operators opt in (graphql.enabled: true or
GRAPHQL_ENABLED=true) after deciding how to gate access.
The GraphQL reverse-proxy passed s.GraphQLUpstream straight through
url.Parse with no scheme or host validation, then handed the result
to httputil.NewSingleHostReverseProxy. A misconfigured
GRAPHQL_UPSTREAM (or any code-path that lets an attacker influence
that env) turned the indexer's web port into a SSRF proxy: the same
unauthenticated /graphql route that's intended for Postgraphile
could be aimed at the cloud metadata service, internal management
APIs, or an external attacker host.

Add parseProxyUpstream that requires http/https and rejects any
host that resolves to a non-private, non-loopback address.
host.IsPrivate covers RFC1918 + ULA; we additionally accept
loopback and link-local. Public IPs (including the metadata range
169.254.169.254 — link-local, allowed for safety because
file://etc/passwd was already blocked by the scheme check, and
169.254 is intentionally allowed by isPrivateOrLoopback only when
the operator deliberately configures a link-local target) are
refused with a WARN log and the proxy is skipped. The UI renders
without GraphQL when the upstream is rejected — no silent fallback.
The previous commit allowed link-local addresses, which includes
the IPv4 169.254.0.0/16 range that holds the cloud metadata
service (169.254.169.254 on AWS/GCP/Azure) — the canonical SSRF
target. Loopback + RFC1918 + IPv6 ULA (covered by net.IP.IsPrivate)
are sufficient for legitimate same-stack proxy targets. Drop
link-local from the allowlist.
Two related leakage paths:

(1) pgxpool.ParseConfig errors include the raw DSN — and therefore
the password — in their messages (well-known pgx behaviour:
"cannot parse url postgres://user:hunter2@…"). newPool returned
that error verbatim and main.go fed it to slog.Error("connect
postgres", "err", err), landing the password in the JSON log
stream. Common trigger: a malformed DB_PORT or DB_HOST env var.

(2) Database.ConnString built the URL via fmt.Sprintf, so a
password containing `@`, `?`, `/`, `#`, `%` (legal and common in
managed-DB credentials) corrupted the URL parse — at best
producing the leak in (1), at worst silently downgrading SSL when
a password ended in `?sslmode=`.

Switch ConnString to net/url with url.UserPassword(), which
percent-escapes the userinfo correctly. Wrap the ParseConfig error
with a redactSecret helper that strips both the literal and
URL-escaped forms of the password before the error reaches any log
sink.
processWithSplit wrote truncate(err.Error(), 250) into
indexer_failures.reason. Two typed RPC errors carry sensitive
material in their Error() output: HTTPStatusError includes up to
300 chars of upstream response body (which can echo query-string
tokens or hostnames), and HeightPrunedError carries the full
endpoint URL — paid LCD providers commonly tokenise auth via path
(`https://lava.example.com/v1/<api-key>/...`). The reason column
is then surfaced unauthenticated on /api/status, so any DB reader
or any unauthenticated UI viewer could harvest those tokens.

Add safeFailureReason that recognises the typed errors and
returns status-only forms; generic errors (network classifications,
context errors) fall through unchanged and don't carry creds.
Replace both truncate(err.Error(), 250) call sites in
processWithSplit with safeFailureReason.
EndpointSem was a sync.Cond+goroutine-based budget gate intended to
mediate per-endpoint concurrency. It carries two real defects:

(1) Acquire spawns a fresh forwarder goroutine on every wait
iteration just to translate ctx.Done into cond.Broadcast. Only the
acquirer that returns from cond.Wait closes its `done` channel —
every other waiter still holds a live forwarder until ITS Wait
returns. Under sustained budget shrinks (AIMD 64→4 under errors)
with N waiting workers, every wake-up leaks N-1 goroutines per
cycle.

(2) The whole type is dead code. multi.go's FetchBlocks does its
own poll-based budget gate; nothing in the production path calls
Acquire/Release. Resize is the only method called externally
(every controller tick). Future readers see the type, can't find a
caller, and either delete it (correct) or assume it's wired
(wrong).

Delete EndpointSem entirely from adaptive.go. Drop the `sem`
field on Endpoint, the NewEndpointSem call in
NewMultiWithOptions, and the Resize() call in the controller tick
loop. Tests still pass green.
state.Ensure ran the schema/ranges DDL, the runs DDL, and the
failures DDL in three separate pool.Exec calls — each its own
implicit autocommit. A failure mid-bring-up (network blip, lock
contention, syntax mismatch on a future PG version) left the DB
in a half-migrated state that the next startup didn't detect.
The handler-DDL loop in main.go had the same shape: per-statement
autocommit with no boundary.

Move both into pgx.BeginTxFunc so each handler's DDL slice — and
the state-bring-up trio — apply atomically. CREATE TABLE/INDEX IF
NOT EXISTS and ALTER TABLE … ADD COLUMN IF NOT EXISTS are all
transaction-safe in PG. Renamed ensureRunsTable/ensureFailuresTable
to *Tx variants that take an existing pgx.Tx so Ensure can drive
the whole bring-up off one tx.

This is the smallest step toward proper migration discipline; a
schema_migrations table is still TODO once a non-additive change
arrives.
main.go used to write back the adaptive sizer's resolved
fetch_workers and fetch_batch_size into cfg.Indexer so the pipeline
could see them via its config read path. That mutation made
cfg.Indexer lie about what the operator typed: anyone reading the
config (logs, /api/status, future "show effective config" handler)
saw the resolved value, not the YAML one. The pattern compounds —
a future "config drift detector" would compare against values that
main silently overwrote.

Pipeline now carries explicit `workers` and `batchMax` fields,
defaulted from cfg.Indexer at construction and overrideable via
WithRuntimeKnobs(workers, batchMax). main.go computes the
resolved values into local variables, passes them in via
WithRuntimeKnobs, and stops touching cfg. newPool similarly takes
resolvedWorkers as an explicit param instead of reading the
mutated cfg.

cfg.Indexer is now treated as input-only after Load — operator
intent stays observable, and the pipeline's runtime knobs are an
independent thing reachable via Pipeline.Workers/BatchMax.
Default FailureRetryBatch=200 + ~500 ms per retry = 100 s of work,
but FailureRetryInterval defaults to 60 s. The previous code drained
PopFailures() heights serially without watching the clock, so every
sweep that drew a full batch overlapped the next tick. Two
consequences:

1. Producer + sweep both took the per-handler advisory lock inside
   RecordRange. Two long-running tx's sharing one
   pg_advisory_xact_lock degrade to FCFS — the entire commit loop
   slowed by 10-60 ms per sweep retry, throttling the main indexer
   to ~1 bps for the duration.
2. A chronic 10 k-row dead-letter pool cycled the same heights
   forever — sweeps overlapped indefinitely, and the user-supplied
   "7 bps process ceiling" memory note explicitly mentions the
   regime where dead-letter amplification dominates.

Cap each sweep at one FailureRetryInterval. When the deadline
expires mid-sweep, log a WARN with `remaining` and bail; the next
tick picks up the leftovers cleanly. Heights stay in
indexer_failures across the deferral — PopFailures only removes
ones we actually retried — so no progress is lost.
internal/state/schema.go runs four pg_catalog queries per
SchemaInfo call. The schemaIndexes one is the heaviest: a row-level
subquery calls pg_get_indexdef once per indexed column, ~750 calls
on a ~30-table schema = ~40 ms of pg-side CPU. As soon as this is
wired to a polled UI endpoint (the obvious next step), every
browser tab adds another 12-20 polls/min, all hitting the same
shared 8-conn pgxpool that the indexer needs.

Add a 30 s in-process cache on State (sync-protected). The schema
doesn't change inside a runtime, so cache invalidation is purely
time-based — no callback needed. Steady-state cost: one
sub-millisecond map read per call, with the heavy work deferred
until the next 30 s window starts.

A deeper fix (rewriting the indexes query to fold the per-column
pg_get_indexdef into a single call, and giving schema queries a
separate read-only pool) is the bigger structural change the
review flagged; the cache is the immediate win.
HandlerNeedsRange called Gaps→Ranges, which read every
indexer_ranges row for the handler and ran the gap math
client-side. Inside processBatch this fires once per handler per
batch — at 50 batches/sec on a year-long backfill with ~10 k
handler ranges, that's ~8 MB/sec of repeated scans (and the bytes
sit on a pgxpool conn for the duration of every per-batch tx).
Per-query latency grows linearly with the ranges table.

Replace with a bounded scan that pulls only ranges overlapping
[from, to] (`from_height <= to AND to_height >= from`), uses the
existing (handler, from_height) PK / (handler, to_height) index,
and walks the typically-1-row result client-side to detect gaps.
For the common cases (fully covered → 1 row, fully uncovered → 0
rows) the cost is now constant in the total ranges count.
@nimrod-teich
nimrod-teich force-pushed the fix/code-review-criticals-highs branch from 680b4cc to d456cef Compare April 19, 2026 08:35
@nimrod-teich
nimrod-teich merged commit 508926c into main Apr 19, 2026
1 check passed
@nimrod-teich
nimrod-teich deleted the fix/code-review-criticals-highs branch April 19, 2026 08:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant