Repository navigation
fix: stop losing relay payments and re-emitting dead-lettered blocks (MAG-5084) - #19
Merged
Merged
Conversation
…(MAG-5084) Mainnet had 70 dead-lettered lava_relay_payment heights: two looped forever at ~35 errors/min on a relay_payments_pkey duplicate, and blocks recorded as indexed were missing their relay payments. - state: CompactRanges deleted every row of the handler between its read and its swap, so a range committed in that window was lost while its rows stayed committed. It now deletes only the exact rows it read. - relay_payment: persist via a staging table and INSERT ... ON CONFLICT DO NOTHING, so re-processing a stored block is a no-op instead of a duplicate-key abort. - pipeline: when an aggregated write tx fails, retry each job in its own tx, so one bad job dead-letters only its own heights. - state: UnionGaps skips permanent dead-letters, so the producer stops re-emitting them every tip poll. Gaps (the UI) still reports them. - rpc: decodeBlock rejects a null/empty result, a header for another height, or a txs_results count that differs from the block's tx count, so MultiClient fails over instead of a block with missing events being recorded as indexed.
… of TRUNCATE (MAG-5084)
…ermanent dead-letters (MAG-5084)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
#Closes MAG-5084
Jira ticket: MAG-5084
Why
The mainnet indexer has 70 dead-lettered
lava_relay_paymentheights (since 2026-04-22, allpermanent):duplicate key value violates unique constraint "relay_payments_pkey". Their rows are already stored, but no range covers them.What changed
state.CompactRangesDELETE. In READ COMMITTED, a range a writer committed between the compactor's read and thatDELETEwas deleted and never re-inserted. The block's rows stayed committed; the block became a gap again.(from_height, to_height)pairs that were read. A row committed or extended mid-compaction survives to the next tick.relay_payment.PersistCOPYintorelay_payments: re-processing a stored block aborts the tx.COPYinto a session temp table, then move the rows with oneDELETE … RETURNING/INSERT … ON CONFLICT DO NOTHINGstatement. A block's rows are fully determined by the chain, so an existing row with the same key is the same row.persistWithFallback: if the aggregated tx fails, retry each job in its own tx; only jobs that fail alone are dead-lettered.state.UnionGapsGaps(the UI's view) still reports them as uncovered. Retrying one takes deleting itsindexer_failuresrow.rpc.decodeBlocktxs_resultsthan txs decoded as a block with missing events, and the range was recorded as covered.MultiClientfails over. CometBFT always returns onetxs_resultsentry per tx.About fix 5: the logs from when the empty blocks were recorded are gone (the container was recreated on 2026-11-05). Today all three mainnet endpoints return complete results for those blocks, so #5 is the only code path I found that records a block as covered without its rows. It is not proven to be how they were lost. It's a guard: a correct node always passes it.
The README's write-path, gap and retry sections now describe the new behaviour.
Tests
Run with a real Postgres:
TEST_DATABASE_URL=… go test ./.... CI has noTEST_DATABASE_URL, so it skips the integration tests.state/ranges_test.go:UnionGapsskips permanent heights, per handler only;Gapsstill reports them.subtractHeightstable test.relay_payment/handler_test.go:Persistcalls in one tx each leave the stage empty.pipeline/persist_test.go: one bad job dead-letters only itself; a single job; a cancelled ctx dead-letters nothing.rpc/decode_test.go: a good block, an empty block, and six rejected shapes.Each new test was checked against the old behaviour, which I put back one fix at a time; every test fails there. With the old compaction SQL, the race tests show the lost coverage exactly:
[{1 20}]instead of[{1 20} {30 40}].go vet,go buildandgo test ./...pass locally (Go 1.26.2, Postgres 16).Deploy and repair (done right after merge)
indexerservice on the mainnet host.indexer_failuresrows and remove them fromindexer_ranges, so the producer refetches them. Stored rows are kept; the idempotent insert adds only what is missing.