Repository navigation
Unblock dashboard: pool-deadlock + Dict lock contention - #7
Merged
Merged
Conversation
…deadlock processBatch's BeginTxFunc callback called p.state.HandlerNeedsRange, which issues a pool query. With a pool connection already held for the tx, every worker also tried to acquire a SECOND connection for the gap read. Under N workers > pool_size the pool self-deadlocked — every connection pinned to a tx waiting on a gap-read connection that would never come. Observed after PR #6's gap-split landed on a fresh backfill: at pool_size=16, workers=64, pg_stat_activity accumulated 8+ rows of "idle in transaction, wait_event=ClientRead, duration > 6 min" and /api/status timed out on its own pool acquire, leaving the dashboard blank. The read is idempotent and doesn't need a tx — any state drift between the read and the subsequent tx is handled by the advisory lock + merge semantics in RecordRange. Compute the needs-map up front; the tx body then just applies the writes + heartbeat.
Previous form used ON CONFLICT DO UPDATE SET col = EXCLUDED.col to force RETURNING on conflict rows. The no-op UPDATE still takes a row-level exclusive lock that's held until the caller's tx commits — which in our case includes a heavy COPY into relay_payments right after. Under N parallel batch workers that all upsert the same ~60 provider addresses, every worker serialised on those 60 row locks for the full batch duration. Observed in pg_stat_activity on testnet: 8+ rows stuck at 30-48 seconds each, all on the providers upsert, while the status handler starved on pool acquire and the dashboard rendered blank. DO NOTHING grabs the row lock only for the conflict check and releases immediately. We follow with a plain SELECT for the conflicted rows — MVCC-safe because Step 1's INSERT waits on any concurrent uncommitted insert before returning, so by the time we run Step 2 the committed row is visible. Two queries in the common conflict case, one when all rows are new. Drops /api/status latency from >10 s (timing out) to sub-500 ms under fresh-backfill load.
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-up to #6. After that PR merged and both stacks were redeployed to fresh empty volumes, the dashboards rendered blank —
/api/statustimed out on both. Two bugs in a chain, diagnosed viapg_stat_activity.1.
fix(pipeline): hoist HandlerNeedsRange out of batch txprocessBatch'sBeginTxFunccallback calledp.state.HandlerNeedsRange, which issues a pool query. Each worker held its tx connection AND tried to grab a second connection for the gap read. Atpool_size=16, workers=64, all 16 connections self-deadlocked holding BEGIN, waiting on a gap-read connection that would never come.Fix: compute the needs-map before
BeginTxFunc. Read is idempotent; state drift between read and tx is handled by the advisory lock + merge inRecordRange.2.
fix(events): Dict DO NOTHING + fallback SELECTAfter unwinding the pool deadlock, 8+ workers were still stuck 30-48s each on the providers/chains UPSERT. The previous form
ON CONFLICT DO UPDATE SET col = EXCLUDED.coltakes a row-level exclusive lock on every conflicting row and holds it until the caller's tx commits — which includes the heavyCOPY relay_paymentsright after. Every worker upserting the same ~60 dictionary rows serialised on those locks for the full batch duration.Fix:
ON CONFLICT DO NOTHING(no lock held past the conflict check) + a plainSELECTfor the conflicted rows. MVCC-safe because Step 1's INSERT waits on any concurrent uncommitted insert, so Step 2's read sees committed state.Observed impact
Before → after on testnet (verified in-session, post-deploy):
/api/statuslatencyidle in transactionrows inpg_stat_activityCOPYMainnet
/api/statusnow ~180ms (was HTTP 000).Test plan
go build ./...clean,go vet ./...clean./api/statusresponds sub-second.pg_stat_activityclean (no idle-in-tx, no multi-minute queries).Still open (separate follow-ups)
Earliestboundary only, round-robin across tiers).status handler queries could be cheaper—pg_total_relation_size+Totals()are probably the heaviest part of the remaining 500ms on testnet; worth cheapening but not blocking.