Skip to content

fix: stop losing relay payments and re-emitting dead-lettered blocks (MAG-5084) - #19

Merged
nimrod-teich merged 3 commits into
mainfrom
mag-5084
Nov 10, 2026
Merged

nimrod-teich merged 3 commits into
mainfrom
mag-5084

Conversation

@nimrod-teich

Copy link
Copy Markdown
Contributor

#Closes MAG-5084

Jira ticket: MAG-5084

Why

The mainnet indexer has 70 dead-lettered lava_relay_payment heights (since 2026-04-22, all permanent):

  • Two loop forever: 5,599,322 and 6,126,966 are retried ~35 times a minute (2.28M retries so far), each time failing with duplicate key value violates unique constraint "relay_payments_pkey". Their rows are already stored, but no range covers them.
  • Missing data hidden as covered: 68 sit inside ranges recorded as indexed, and 36 of those have no rows. Of the 18 still servable by an endpoint, 13 have relay payments on chain: 9,231 records are missing while the dashboard shows 100% coverage. Example: block 6,015,512 has 111 records on chain and 0 stored.

What changed

# Where Defect Fix
1 state.CompactRanges The swap re-selected every row of the handler for its DELETE. In READ COMMITTED, a range a writer committed between the compactor's read and that DELETE was deleted and never re-inserted. The block's rows stayed committed; the block became a gap again. Merge in Go, then delete only the exact (from_height, to_height) pairs that were read. A row committed or extended mid-compaction survives to the next tick.
2 relay_payment.Persist Plain COPY into relay_payments: re-processing a stored block aborts the tx. COPY into a session temp table, then move the rows with one DELETE … RETURNING / INSERT … ON CONFLICT DO NOTHING statement. A block's rows are fully determined by the chain, so an existing row with the same key is the same row.
3 pipeline writer One failing job rolled back the whole aggregated tx (up to 32 jobs) and dead-lettered all of them. Innocent neighbours then shared a gap with the bad block and failed with it forever (6,015,511 stored + 6,015,512 lost). persistWithFallback: if the aggregated tx fails, retry each job in its own tx; only jobs that fail alone are dead-lettered.
4 state.UnionGaps Permanent dead-letters stayed gaps, so the producer re-emitted them on every tip poll. Skip permanent heights per handler. Gaps (the UI's view) still reports them as uncovered. Retrying one takes deleting its indexer_failures row.
5 rpc.decodeBlock A null/empty result, a header for another height, or fewer txs_results than txs decoded as a block with missing events, and the range was recorded as covered. Reject those responses so MultiClient fails over. CometBFT always returns one txs_results entry per tx.

About fix 5: the logs from when the empty blocks were recorded are gone (the container was recreated on 2026-11-05). Today all three mainnet endpoints return complete results for those blocks, so #5 is the only code path I found that records a block as covered without its rows. It is not proven to be how they were lost. It's a guard: a correct node always passes it.

The README's write-path, gap and retry sections now describe the new behaviour.

Tests

Run with a real Postgres: TEST_DATABASE_URL=… go test ./.... CI has no TEST_DATABASE_URL, so it skips the integration tests.

  • state/ranges_test.go:
    • A range committed mid-compaction survives.
    • A range extended mid-compaction keeps its extension.
    • UnionGaps skips permanent heights, per handler only; Gaps still reports them.
    • subtractHeights table test.
  • relay_payment/handler_test.go:
    • Re-persisting a stored block is a no-op.
    • A stored block mixed with a new one in one tx still stores the new rows.
    • Several Persist calls in one tx each leave the stage empty.
  • pipeline/persist_test.go: one bad job dead-letters only itself; a single job; a cancelled ctx dead-letters nothing.
  • rpc/decode_test.go: a good block, an empty block, and six rejected shapes.

Each new test was checked against the old behaviour, which I put back one fix at a time; every test fails there. With the old compaction SQL, the race tests show the lost coverage exactly: [{1 20}] instead of [{1 20} {30 40}].

go vet, go build and go test ./... pass locally (Go 1.26.2, Postgres 16).

Deploy and repair (done right after merge)

  1. Build and restart only the indexer service on the mainnet host.
  2. Re-index the dead-lettered heights ≥ 5,318,001: delete their indexer_failures rows and remove them from indexer_ranges, so the producer refetches them. Stored rows are kept; the idempotent insert adds only what is missing.
  3. Heights below 5,318,001 can't be refetched: no current endpoint serves them, since the public archive dropped that history this week. They are left as they are.

…(MAG-5084)

Mainnet had 70 dead-lettered lava_relay_payment heights: two looped
forever at ~35 errors/min on a relay_payments_pkey duplicate, and blocks
recorded as indexed were missing their relay payments.

- state: CompactRanges deleted every row of the handler between its read
  and its swap, so a range committed in that window was lost while its
  rows stayed committed. It now deletes only the exact rows it read.
- relay_payment: persist via a staging table and INSERT ... ON CONFLICT
  DO NOTHING, so re-processing a stored block is a no-op instead of a
  duplicate-key abort.
- pipeline: when an aggregated write tx fails, retry each job in its own
  tx, so one bad job dead-letters only its own heights.
- state: UnionGaps skips permanent dead-letters, so the producer stops
  re-emitting them every tip poll. Gaps (the UI) still reports them.
- rpc: decodeBlock rejects a null/empty result, a header for another
  height, or a txs_results count that differs from the block's tx count,
  so MultiClient fails over instead of a block with missing events being
  recorded as indexed.
@nimrod-teich
nimrod-teich merged commit 82e1527 into main Nov 10, 2026
1 check passed
@nimrod-teich
nimrod-teich deleted the mag-5084 branch November 10, 2026 12:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant