Repository navigation
fix: resolve all critical + 12 high findings from code review - #8
Merged
Merged
Conversation
The shipped pg_hba.conf granted passwordless `trust` to 0.0.0.0/0 and ::/0. Combined with the host-only port-forward, off-host attackers were blocked but any other container on the docker bridge network (graphql, a future side-car, an attacker-controlled image pulled via supply-chain) could open a superuser session with no credentials. Drop the global rules. Keep loopback `trust` so a host-side `psql -h 127.0.0.1` via the compose port-forward still works without a password. Bridge-subnet ranges (172.16/12, 192.168/16, 10/8) now require scram-sha-256 — the indexer/graphql containers already receive DB_PASSWORD via env so they authenticate normally. Strengthen the file header to make the dev-only scope explicit and to direct operators to ship a separate pg_hba.prod.conf rather than copying this file into a production image.
Four places shipped `postgres` as the password default: the
example config that the Dockerfile bakes into the image, the
.env.example, the compose interpolation defaults for both the
postgres and indexer services, and the Database defaults in
internal/config. An operator running `docker compose up` with no
.env brought up Postgres listening with the most-guessed default
credential pair. With the new pg_hba enforcing scram-sha-256 on the
bridge subnets, the password is now actually load-bearing — and a
shipped default would defeat the HBA fix from the previous commit.
- config.example.yml: drop the literal `password:` line; comment
explains it's env-only.
- .env.example: blank DB_PASSWORD with a "set this" comment.
- docker-compose.yml (postgres + indexer + graphql): replace
`${DB_PASSWORD:-postgres}` with `${DB_PASSWORD:?must be set}` so
compose fails fast instead of bringing the stack up with weak
creds.
- internal/config/config.go: drop the `Password: "postgres"` default
from the Database struct.
The Dockerfile still bakes config.example.yml at build time, but
without the password line — operators must inject DB_PASSWORD via
env or a mounted secret.
Previously newPool ran before the adaptive sizer resolved the worker count, so MaxConns came from the config default of 8 even when fetch_workers ended up at 100+. Each worker holds a pool connection for its full per-batch transaction, so anything below fetch_workers was a hard ceiling: at fetch_workers=16 / pool_size=8 throughput was capped at 50% of nominal; at adaptive ~100 workers it fell to 8%. The dashboard and aggregate refreshes shared the same starved pool. Move newPool down past the adaptive-sizer block so it sees the resolved cfg.Indexer.FetchWorkers. In newPool, derive MaxConns = max(configured_pool_size, fetch_workers + headroom), where headroom (4) covers the embedded web UI and aggregates. Operators who explicitly under-size the pool get a WARN log and the auto-grown value, not a silent throughput collapse.
PR #6 removed Dict's in-memory cache to fix an FK race: worker A inserted a row + cached its id pre-commit, B grabbed the id, A rolled back, B's COPY referenced a non-existent id. The fix made every IDs() call hit Postgres for the full set every batch — at 50 batches/sec × ~100 unique providers, that's hundreds of dictionary upserts per second sitting in the critical path of every relay_payment commit. Restore the cache with strict commit-only discipline: - New Warmup(ctx, pool) loads every committed (id, value) pair into the cache once at startup, after DDL. - IDs() checks the cache first; misses go through the existing INSERT…DO NOTHING + SELECT path. - ONLY rows returned from the SELECT fallback (already-committed in another tx, made visible by PG's conflict-resolution wait) are added to the cache. INSERT…RETURNING rows belong to the current, uncommitted tx and are NEVER cached — eliminating the rollback race that motivated the original cache removal. Wire the optional events.Warmer interface so Pipeline-style callers run Warmup once after DDL apply. relay_payment.Handler now implements Warmer to warm both providers and chains. Newly-introduced values pay one extra round-trip on the *next* batch that needs them; from then on they hit the cache. Steady state for a chain with ~100 providers/chains is ~99% hit rate, so IDs() becomes a no-op for the hot path while the FK-race correctness invariant is preserved.
docker-compose.yml exposed the indexer's web port on every host
interface (`${INDEXER_PORT:-8080}:8080` with no bind address),
unlike the postgres binding which is already loopback-only. The web
UI / /api/status / GraphQL proxy are UNAUTHENTICATED and disclose
endpoint URLs (which can carry tokens), dead-letter reasons, and
per-handler row counts. Anyone reachable on the host's network
could enumerate the indexer fleet.
Default the bind to 127.0.0.1 via a new INDEXER_BIND env (matching
the postgres pattern). Operators who front the indexer with a
reverse proxy + auth can set INDEXER_BIND=0.0.0.0; everyone else
gets the safe default.
Adding actual auth (basic / token) is intentionally NOT in this
commit — it's a behavioural change that wants a config decision
about credential storage. Loopback binding is the immediate
non-breaking mitigation.
The indexer's web server reverse-proxied /graphql and /graphiql to the upstream Postgraphile by default, and Postgraphile launches with --no-ignore-rbac --enhance-graphiql over the entire `app` schema with no auth headers required. Combined with the web UI being unauthenticated, an attacker who reached the indexer's port could browse /graphiql and read every table — event history, dead-letter reasons, run telemetry — with no challenge. Flip the default to disabled in both internal/config defaults() and the example config. Operators opt in (graphql.enabled: true or GRAPHQL_ENABLED=true) after deciding how to gate access.
The GraphQL reverse-proxy passed s.GraphQLUpstream straight through url.Parse with no scheme or host validation, then handed the result to httputil.NewSingleHostReverseProxy. A misconfigured GRAPHQL_UPSTREAM (or any code-path that lets an attacker influence that env) turned the indexer's web port into a SSRF proxy: the same unauthenticated /graphql route that's intended for Postgraphile could be aimed at the cloud metadata service, internal management APIs, or an external attacker host. Add parseProxyUpstream that requires http/https and rejects any host that resolves to a non-private, non-loopback address. host.IsPrivate covers RFC1918 + ULA; we additionally accept loopback and link-local. Public IPs (including the metadata range 169.254.169.254 — link-local, allowed for safety because file://etc/passwd was already blocked by the scheme check, and 169.254 is intentionally allowed by isPrivateOrLoopback only when the operator deliberately configures a link-local target) are refused with a WARN log and the proxy is skipped. The UI renders without GraphQL when the upstream is rejected — no silent fallback.
The previous commit allowed link-local addresses, which includes the IPv4 169.254.0.0/16 range that holds the cloud metadata service (169.254.169.254 on AWS/GCP/Azure) — the canonical SSRF target. Loopback + RFC1918 + IPv6 ULA (covered by net.IP.IsPrivate) are sufficient for legitimate same-stack proxy targets. Drop link-local from the allowlist.
Two related leakage paths:
(1) pgxpool.ParseConfig errors include the raw DSN — and therefore
the password — in their messages (well-known pgx behaviour:
"cannot parse url postgres://user:hunter2@…"). newPool returned
that error verbatim and main.go fed it to slog.Error("connect
postgres", "err", err), landing the password in the JSON log
stream. Common trigger: a malformed DB_PORT or DB_HOST env var.
(2) Database.ConnString built the URL via fmt.Sprintf, so a
password containing `@`, `?`, `/`, `#`, `%` (legal and common in
managed-DB credentials) corrupted the URL parse — at best
producing the leak in (1), at worst silently downgrading SSL when
a password ended in `?sslmode=`.
Switch ConnString to net/url with url.UserPassword(), which
percent-escapes the userinfo correctly. Wrap the ParseConfig error
with a redactSecret helper that strips both the literal and
URL-escaped forms of the password before the error reaches any log
sink.
processWithSplit wrote truncate(err.Error(), 250) into indexer_failures.reason. Two typed RPC errors carry sensitive material in their Error() output: HTTPStatusError includes up to 300 chars of upstream response body (which can echo query-string tokens or hostnames), and HeightPrunedError carries the full endpoint URL — paid LCD providers commonly tokenise auth via path (`https://lava.example.com/v1/<api-key>/...`). The reason column is then surfaced unauthenticated on /api/status, so any DB reader or any unauthenticated UI viewer could harvest those tokens. Add safeFailureReason that recognises the typed errors and returns status-only forms; generic errors (network classifications, context errors) fall through unchanged and don't carry creds. Replace both truncate(err.Error(), 250) call sites in processWithSplit with safeFailureReason.
EndpointSem was a sync.Cond+goroutine-based budget gate intended to mediate per-endpoint concurrency. It carries two real defects: (1) Acquire spawns a fresh forwarder goroutine on every wait iteration just to translate ctx.Done into cond.Broadcast. Only the acquirer that returns from cond.Wait closes its `done` channel — every other waiter still holds a live forwarder until ITS Wait returns. Under sustained budget shrinks (AIMD 64→4 under errors) with N waiting workers, every wake-up leaks N-1 goroutines per cycle. (2) The whole type is dead code. multi.go's FetchBlocks does its own poll-based budget gate; nothing in the production path calls Acquire/Release. Resize is the only method called externally (every controller tick). Future readers see the type, can't find a caller, and either delete it (correct) or assume it's wired (wrong). Delete EndpointSem entirely from adaptive.go. Drop the `sem` field on Endpoint, the NewEndpointSem call in NewMultiWithOptions, and the Resize() call in the controller tick loop. Tests still pass green.
state.Ensure ran the schema/ranges DDL, the runs DDL, and the failures DDL in three separate pool.Exec calls — each its own implicit autocommit. A failure mid-bring-up (network blip, lock contention, syntax mismatch on a future PG version) left the DB in a half-migrated state that the next startup didn't detect. The handler-DDL loop in main.go had the same shape: per-statement autocommit with no boundary. Move both into pgx.BeginTxFunc so each handler's DDL slice — and the state-bring-up trio — apply atomically. CREATE TABLE/INDEX IF NOT EXISTS and ALTER TABLE … ADD COLUMN IF NOT EXISTS are all transaction-safe in PG. Renamed ensureRunsTable/ensureFailuresTable to *Tx variants that take an existing pgx.Tx so Ensure can drive the whole bring-up off one tx. This is the smallest step toward proper migration discipline; a schema_migrations table is still TODO once a non-additive change arrives.
main.go used to write back the adaptive sizer's resolved fetch_workers and fetch_batch_size into cfg.Indexer so the pipeline could see them via its config read path. That mutation made cfg.Indexer lie about what the operator typed: anyone reading the config (logs, /api/status, future "show effective config" handler) saw the resolved value, not the YAML one. The pattern compounds — a future "config drift detector" would compare against values that main silently overwrote. Pipeline now carries explicit `workers` and `batchMax` fields, defaulted from cfg.Indexer at construction and overrideable via WithRuntimeKnobs(workers, batchMax). main.go computes the resolved values into local variables, passes them in via WithRuntimeKnobs, and stops touching cfg. newPool similarly takes resolvedWorkers as an explicit param instead of reading the mutated cfg. cfg.Indexer is now treated as input-only after Load — operator intent stays observable, and the pipeline's runtime knobs are an independent thing reachable via Pipeline.Workers/BatchMax.
Default FailureRetryBatch=200 + ~500 ms per retry = 100 s of work, but FailureRetryInterval defaults to 60 s. The previous code drained PopFailures() heights serially without watching the clock, so every sweep that drew a full batch overlapped the next tick. Two consequences: 1. Producer + sweep both took the per-handler advisory lock inside RecordRange. Two long-running tx's sharing one pg_advisory_xact_lock degrade to FCFS — the entire commit loop slowed by 10-60 ms per sweep retry, throttling the main indexer to ~1 bps for the duration. 2. A chronic 10 k-row dead-letter pool cycled the same heights forever — sweeps overlapped indefinitely, and the user-supplied "7 bps process ceiling" memory note explicitly mentions the regime where dead-letter amplification dominates. Cap each sweep at one FailureRetryInterval. When the deadline expires mid-sweep, log a WARN with `remaining` and bail; the next tick picks up the leftovers cleanly. Heights stay in indexer_failures across the deferral — PopFailures only removes ones we actually retried — so no progress is lost.
internal/state/schema.go runs four pg_catalog queries per SchemaInfo call. The schemaIndexes one is the heaviest: a row-level subquery calls pg_get_indexdef once per indexed column, ~750 calls on a ~30-table schema = ~40 ms of pg-side CPU. As soon as this is wired to a polled UI endpoint (the obvious next step), every browser tab adds another 12-20 polls/min, all hitting the same shared 8-conn pgxpool that the indexer needs. Add a 30 s in-process cache on State (sync-protected). The schema doesn't change inside a runtime, so cache invalidation is purely time-based — no callback needed. Steady-state cost: one sub-millisecond map read per call, with the heavy work deferred until the next 30 s window starts. A deeper fix (rewriting the indexes query to fold the per-column pg_get_indexdef into a single call, and giving schema queries a separate read-only pool) is the bigger structural change the review flagged; the cache is the immediate win.
HandlerNeedsRange called Gaps→Ranges, which read every indexer_ranges row for the handler and ran the gap math client-side. Inside processBatch this fires once per handler per batch — at 50 batches/sec on a year-long backfill with ~10 k handler ranges, that's ~8 MB/sec of repeated scans (and the bytes sit on a pgxpool conn for the duration of every per-batch tx). Per-query latency grows linearly with the ranges table. Replace with a bounded scan that pulls only ranges overlapping [from, to] (`from_height <= to AND to_height >= from`), uses the existing (handler, from_height) PK / (handler, to_height) index, and walks the typically-1-row result client-side to detect gaps. For the common cases (fully covered → 1 row, fully uncovered → 0 rows) the cost is now constant in the total ranges count.
nimrod-teich
force-pushed
the
fix/code-review-criticals-highs
branch
from
April 19, 2026 08:35
680b4cc to
d456cef
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Resolves all 4 critical and 12 of 16 high findings from the
/code-reviewworkflow run on commit 1e42ba2 (PR #6 base).Summary
4 critical fixed (one commit each):
pg_hba.conftrust→ scram-sha-256 on bridge subnetspostgresDB password → removed from every shipped surface; compose${DB_PASSWORD:?…}Dict.IDsper-batch hot path → restored cache with strict commit-only discipline12 highs fixed (one commit each):
redactSecret+url.UserPasswordescapingreasonURL/body leak —safeFailureReasonfor typed errorsEndpointSemdeletion (dead code with goroutine-leak risk)state.Ensure+ handler DDL loop now inpgx.BeginTxFuncWithRuntimeKnobsFailureRetryInterval4 highs deferred (need operator decision):
Heads-up: pre-existing commits on the branch
The first two commits (53da2f3 hoist HandlerNeedsRange, a899e96 Dict
ON CONFLICT DO NOTHING) were ALREADY on the branch
(
fix/pool-deadlock-and-dict-lock-contention) when I started. Theyare not part of the code-review-fix work, but they're closely related
to two of the criticals (pgxpool sizing depends on the HandlerNeedsRange
hoist; the Dict commit-only cache builds on the DO-NOTHING pattern).
Move them out of this PR if you'd rather land them separately —
git rebase -i 1e42ba2and drop the bottom two.Test plan
go build ./...— cleango vet ./...— clean (pre-existing stringsseq diagnostics in config.go untouched)go test ./...— all packages pass (rpc, pipeline, state)docker compose upsmoke test — verify scram-sha-256 auth + DB_PASSWORD enforcement/api/statuswith a slow client — confirm graphql off + loopback bind in composeRemaining-findings report
Pruned to "what's left" in
_workspace/report.md(gitignored). Per-reviewerfiles in
_workspace/{architecture,security,performance,style}_findings.mdeach carry a header note pointing at the consolidated view.