Skip to content

Unblock dashboard: pool-deadlock + Dict lock contention - #7

Merged
nimrod-teich merged 2 commits into
mainfrom
fix/pool-deadlock-and-dict-lock-contention
Apr 19, 2026
Merged

nimrod-teich merged 2 commits into
mainfrom
fix/pool-deadlock-and-dict-lock-contention

Conversation

@nimrod-teich

Copy link
Copy Markdown
Contributor

Summary

Follow-up to #6. After that PR merged and both stacks were redeployed to fresh empty volumes, the dashboards rendered blank — /api/status timed out on both. Two bugs in a chain, diagnosed via pg_stat_activity.

1. fix(pipeline): hoist HandlerNeedsRange out of batch tx

processBatch's BeginTxFunc callback called p.state.HandlerNeedsRange, which issues a pool query. Each worker held its tx connection AND tried to grab a second connection for the gap read. At pool_size=16, workers=64, all 16 connections self-deadlocked holding BEGIN, waiting on a gap-read connection that would never come.

Fix: compute the needs-map before BeginTxFunc. Read is idempotent; state drift between read and tx is handled by the advisory lock + merge in RecordRange.

2. fix(events): Dict DO NOTHING + fallback SELECT

After unwinding the pool deadlock, 8+ workers were still stuck 30-48s each on the providers/chains UPSERT. The previous form ON CONFLICT DO UPDATE SET col = EXCLUDED.col takes a row-level exclusive lock on every conflicting row and holds it until the caller's tx commits — which includes the heavy COPY relay_payments right after. Every worker upserting the same ~60 dictionary rows serialised on those locks for the full batch duration.

Fix: ON CONFLICT DO NOTHING (no lock held past the conflict check) + a plain SELECT for the conflicted rows. MVCC-safe because Step 1's INSERT waits on any concurrent uncommitted insert, so Step 2's read sees committed state.

Observed impact

Before → after on testnet (verified in-session, post-deploy):

metric before after
/api/status latency >10s (timeout, HTTP 500) ~500ms
idle in transaction rows in pg_stat_activity 8+, dur 6+ min 0-1
Longest-running query 48s Dict UPSERT ~200ms COPY

Mainnet /api/status now ~180ms (was HTTP 000).

Test plan

  • go build ./... clean, go vet ./... clean.
  • Both stacks deployed, /api/status responds sub-second.
  • pg_stat_activity clean (no idle-in-tx, no multi-minute queries).
  • Dashboards render with endpoint metrics, ranges, failure counts.

Still open (separate follow-ups)

  • Dynamic gap-split: split should weight tiers by the per-endpoint AIMD budget (today it's static by Earliest boundary only, round-robin across tiers).
  • status handler queries could be cheaper — pg_total_relation_size + Totals() are probably the heaviest part of the remaining 500ms on testnet; worth cheapening but not blocking.

…deadlock

processBatch's BeginTxFunc callback called p.state.HandlerNeedsRange,
which issues a pool query. With a pool connection already held for
the tx, every worker also tried to acquire a SECOND connection for
the gap read. Under N workers > pool_size the pool self-deadlocked —
every connection pinned to a tx waiting on a gap-read connection
that would never come.

Observed after PR #6's gap-split landed on a fresh backfill: at
pool_size=16, workers=64, pg_stat_activity accumulated 8+ rows of
"idle in transaction, wait_event=ClientRead, duration > 6 min" and
/api/status timed out on its own pool acquire, leaving the dashboard
blank.

The read is idempotent and doesn't need a tx — any state drift
between the read and the subsequent tx is handled by the advisory
lock + merge semantics in RecordRange. Compute the needs-map up
front; the tx body then just applies the writes + heartbeat.
Previous form used ON CONFLICT DO UPDATE SET col = EXCLUDED.col to
force RETURNING on conflict rows. The no-op UPDATE still takes a
row-level exclusive lock that's held until the caller's tx commits
— which in our case includes a heavy COPY into relay_payments
right after.

Under N parallel batch workers that all upsert the same ~60
provider addresses, every worker serialised on those 60 row locks
for the full batch duration. Observed in pg_stat_activity on
testnet: 8+ rows stuck at 30-48 seconds each, all on the providers
upsert, while the status handler starved on pool acquire and the
dashboard rendered blank.

DO NOTHING grabs the row lock only for the conflict check and
releases immediately. We follow with a plain SELECT for the
conflicted rows — MVCC-safe because Step 1's INSERT waits on any
concurrent uncommitted insert before returning, so by the time we
run Step 2 the committed row is visible. Two queries in the common
conflict case, one when all rows are new.

Drops /api/status latency from >10 s (timing out) to sub-500 ms
under fresh-backfill load.
@nimrod-teich
nimrod-teich merged commit 8c40d3c into main Apr 19, 2026
1 check passed
@nimrod-teich
nimrod-teich deleted the fix/pool-deadlock-and-dict-lock-contention branch April 19, 2026 08:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant