Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 16 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -98,8 +98,10 @@ refetched on next run.

When you rerun with a lower `start_height`, the missing window is split into
disjoint gaps: the _old_ gap between the new start and the prior earliest,
plus every gap between previously-indexed ranges (including dead-letter
holes) and the tip. All gaps are worked in parallel by a shared pool, but
plus every gap between previously-indexed ranges (including transient
dead-letter holes) and the tip. Permanent dead-letters are not gaps for the
producer — it would re-emit them every tip poll — so retrying one takes
deleting its `indexer_failures` row. All gaps are worked in parallel by a shared pool, but
the producer is **tip-first**: on every emit it picks the gap whose cursor
is currently highest — i.e. closest to the chain tip — and dispatches the
next batch from there. Consequences:
Expand Down Expand Up @@ -135,18 +137,24 @@ producer → jobs → N fetchers → writeCh → M writers → DB
holds a DB tx.
- **M writers** (`pool_size - 4`) drain up to 32 `writeJob`s per 50 ms
window and commit them in a single Postgres tx — amortising the
BEGIN/Heartbeat/COMMIT overhead across many batches. Persist failures
record the batch as dead-letter; the sweep re-fetches.
BEGIN/Heartbeat/COMMIT overhead across many batches. If that tx fails,
each `writeJob` is retried in its own tx, so only a job that fails alone
is recorded as dead-letter; the sweep re-fetches it. Event persists are
idempotent (`relay_payments` goes through a staging table and
`INSERT … ON CONFLICT DO NOTHING`), so re-processing an already-stored
block is a no-op.

This lets fetch run up to the upstream ceiling and write run up to the DB
ceiling independently — whichever is slower becomes the bottleneck, not
both. No more "fast upstream gated on slow commit" or vice versa.

Writers to `indexer_ranges` are **lock-free**: each writer does an
`INSERT … ON CONFLICT DO UPDATE` and exits. A background compactor ticks
every 30 s and runs `range_agg(int8range(...))` to collapse overlapping
or touching raw ranges into the minimum-row representation. Writers never
block on each other; fragmentation is bounded by one compaction cycle.
every 30 s and collapses overlapping or touching raw ranges into the
minimum-row representation. It swaps out only the exact rows it read, so
a range a writer commits or extends mid-compaction survives to the next
tick. Writers never block on each other; fragmentation is bounded by one
compaction cycle.

## Space-efficient schema

Expand Down Expand Up @@ -270,6 +278,7 @@ runWindow (gap detection + coverage-tier slicing)
| `processWithSplit` | multi-block batch failure (transient) | **bisect halves until size 1** | isolate one bad block from an otherwise-healthy batch |
| Dead-letter sweep | size-1 failures still in `indexer_failures` | **`failure_max_retries` × 60 s ticks** (default 3) | slow-time recovery for whole-infra outages; flips `permanent=true` once exhausted |
| Permanent classifier (`HeightPrunedError`, `NoEndpointCoversError`) | nothing | **0** — short-circuits the bisect, records every height as `permanent=true` in one shot | retrying a pruned height across all endpoints is the same answer every time |
| Writer fallback | an aggregated write tx that fails | **each `writeJob` once, in its own tx** | one bad job must not dead-letter the jobs it was batched with |

### Why we don't retry the whole `MultiClient` call

Expand Down
43 changes: 35 additions & 8 deletions internal/events/relay_payment/handler.go
Original file line number Diff line number Diff line change
Expand Up @@ -197,14 +197,41 @@ func (h *Handler) Persist(ctx context.Context, tx pgx.Tx, es []events.HandledEve
}
}

_, err = tx.CopyFrom(
ctx,
pgx.Identifier{h.schema, "relay_payments"},
copyCols,
pgx.CopyFromRows(final),
)
if err != nil {
return fmt.Errorf("COPY relay_payments: %w", err)
return h.insertIdempotent(ctx, tx, final)
}

// stageTable is a session-local temp table the rows are COPYed into before
// they are merged into relay_payments. The merge moves rows out with
// DELETE … RETURNING, so the stage is empty again after every Persist call
// (the writer runs several per tx); ON COMMIT DELETE ROWS is a backstop.
const stageTable = "relay_payments_stage"

// insertIdempotent writes rows so that a block whose rows already exist is
// a no-op instead of a failed tx. A plain COPY into relay_payments aborts
// on the first duplicate key, and the writer then dead-letters every job in
// the aggregated tx (MAG-5084). The rows of a block are fully determined by
// the chain, so an existing row with the same primary key is the same row:
// ON CONFLICT DO NOTHING is safe. COPY into the stage table keeps the bulk
// path fast.
func (h *Handler) insertIdempotent(ctx context.Context, tx pgx.Tx, rows [][]any) error {
cols := strings.Join(copyCols, ", ")
if _, err := tx.Exec(ctx, fmt.Sprintf(`
CREATE TEMP TABLE IF NOT EXISTS %s (LIKE %s.relay_payments INCLUDING DEFAULTS)
ON COMMIT DELETE ROWS`, stageTable, h.schema)); err != nil {
return fmt.Errorf("create %s: %w", stageTable, err)
}
if _, err := tx.CopyFrom(ctx, pgx.Identifier{stageTable}, copyCols, pgx.CopyFromRows(rows)); err != nil {
return fmt.Errorf("COPY %s: %w", stageTable, err)
}
// One statement moves the staged rows: it empties the stage for the
// next Persist call in this tx without a TRUNCATE (which would swap the
// table's file, i.e. catalog churn, on every call).
if _, err := tx.Exec(ctx, fmt.Sprintf(`
WITH moved AS (DELETE FROM %[3]s RETURNING %[2]s)
INSERT INTO %[1]s.relay_payments (%[2]s)
SELECT %[2]s FROM moved
ON CONFLICT DO NOTHING`, h.schema, cols, stageTable)); err != nil {
return fmt.Errorf("insert relay_payments: %w", err)
}
return nil
}
Expand Down
162 changes: 162 additions & 0 deletions internal/events/relay_payment/handler_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,162 @@
package relay_payment

import (
"context"
"fmt"
"os"
"strings"
"testing"
"time"

"github.com/jackc/pgx/v5"
"github.com/jackc/pgx/v5/pgxpool"
"github.com/magma-devs/lava-indexer/internal/events"
"github.com/magma-devs/lava-indexer/internal/rpc"
)

// testHandler dials TEST_DATABASE_URL and returns a Handler on a fresh
// schema with its DDL applied. Skips without TEST_DATABASE_URL.
func testHandler(t *testing.T) (*Handler, *pgxpool.Pool) {
t.Helper()
dsn := os.Getenv("TEST_DATABASE_URL")
if dsn == "" {
t.Skip("TEST_DATABASE_URL not set; skipping relay_payment integration test")
}
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
pool, err := pgxpool.New(ctx, dsn)
if err != nil {
t.Fatalf("connect: %v", err)
}
schema := fmt.Sprintf("test_relay_%d", time.Now().UnixNano())
t.Cleanup(func() {
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
_, _ = pool.Exec(ctx, fmt.Sprintf("DROP SCHEMA IF EXISTS %s CASCADE", schema))
pool.Close()
})
if _, err := pool.Exec(ctx, "CREATE SCHEMA "+schema); err != nil {
t.Fatalf("create schema: %v", err)
}
h := New(schema)
for _, ddl := range h.DDL() {
if _, err := pool.Exec(ctx, ddl); err != nil {
t.Fatalf("ddl: %v", err)
}
}
return h, pool
}

// paymentEvents builds one lava_relay_payment event per block, each carrying
// `providers` provider entries.
func paymentEvents(heights []int64, providers int) []events.HandledEvent {
var out []events.HandledEvent
for _, height := range heights {
attrs := map[string]string{}
for i := 0; i < providers; i++ {
n := fmt.Sprint(i)
attrs["provider."+n] = fmt.Sprintf("lava@provider%d", i)
attrs["chainID."+n] = "ETH1"
attrs["CU."+n] = "10"
attrs["relayNumber."+n] = "3"
}
blk := &rpc.Block{Height: height, Time: time.Unix(1_700_000_000+height, 0).UTC()}
out = append(out, events.HandledEvent{
Block: blk,
Event: rpc.Event{
TxHash: strings.Repeat(fmt.Sprintf("%02X", height%256), 32),
Type: EventType,
Attrs: attrs,
EventIdx: 0,
},
})
}
return out
}

func persist(t *testing.T, ctx context.Context, h *Handler, pool *pgxpool.Pool, batches ...[]events.HandledEvent) {
t.Helper()
err := pgx.BeginTxFunc(ctx, pool, pgx.TxOptions{}, func(tx pgx.Tx) error {
for _, es := range batches {
if err := h.Persist(ctx, tx, es); err != nil {
return err
}
}
return nil
})
if err != nil {
t.Fatalf("persist: %v", err)
}
}

func countRows(t *testing.T, ctx context.Context, h *Handler, pool *pgxpool.Pool) int {
t.Helper()
var n int
if err := pool.QueryRow(ctx, fmt.Sprintf("SELECT count(*) FROM %s.relay_payments", h.schema)).Scan(&n); err != nil {
t.Fatalf("count: %v", err)
}
return n
}

// Persisting a block whose rows already exist is a no-op, not a duplicate-key
// error that aborts the writer's tx (MAG-5084).
func TestPersistIsIdempotent(t *testing.T) {
h, pool := testHandler(t)
ctx := context.Background()
es := paymentEvents([]int64{100, 101}, 3)

persist(t, ctx, h, pool, es)
if got := countRows(t, ctx, h, pool); got != 6 {
t.Fatalf("rows after first persist = %d, want 6", got)
}
persist(t, ctx, h, pool, es)
if got := countRows(t, ctx, h, pool); got != 6 {
t.Fatalf("rows after re-persist = %d, want 6", got)
}
}

// A re-processed block that rides with a new block in the same tx must not
// stop the new block's rows from landing.
func TestPersistMixedExistingAndNewBlocks(t *testing.T) {
h, pool := testHandler(t)
ctx := context.Background()
persist(t, ctx, h, pool, paymentEvents([]int64{100}, 2))

persist(t, ctx, h, pool, paymentEvents([]int64{100, 200}, 2))
if got := countRows(t, ctx, h, pool); got != 4 {
t.Fatalf("rows = %d, want 4", got)
}
}

// The writer aggregates several jobs into one tx, so Persist runs several
// times per tx. Each call must insert only its own rows and leave the stage
// empty for the next call.
func TestPersistSeveralCallsInOneTx(t *testing.T) {
h, pool := testHandler(t)
ctx := context.Background()
err := pgx.BeginTxFunc(ctx, pool, pgx.TxOptions{}, func(tx pgx.Tx) error {
for _, es := range [][]events.HandledEvent{
paymentEvents([]int64{300}, 2),
paymentEvents([]int64{301}, 1),
paymentEvents([]int64{300}, 2),
} {
if err := h.Persist(ctx, tx, es); err != nil {
return err
}
var staged int
if err := tx.QueryRow(ctx, "SELECT count(*) FROM "+stageTable).Scan(&staged); err != nil {
return err
}
if staged != 0 {
t.Errorf("%d rows left in %s after Persist", staged, stageTable)
}
}
return nil
})
if err != nil {
t.Fatalf("persist: %v", err)
}
if got := countRows(t, ctx, h, pool); got != 3 {
t.Fatalf("rows = %d, want 3", got)
}
}
103 changes: 103 additions & 0 deletions internal/pipeline/persist_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,103 @@
package pipeline

import (
"context"
"errors"
"testing"
)

func jobs(froms ...int64) []writeJob {
out := make([]writeJob, len(froms))
for i, f := range froms {
out[i] = writeJob{from: f, to: f + 9, blocksThisBatch: 10}
}
return out
}

func froms(js []writeJob) []int64 {
out := make([]int64, len(js))
for i, j := range js {
out[i] = j.from
}
return out
}

func equal(a, b []int64) bool {
if len(a) != len(b) {
return false
}
for i := range a {
if a[i] != b[i] {
return false
}
}
return true
}

// persistFailing fails any persist call whose batch contains a job starting
// at one of bad, mimicking a tx that aborts on that job.
func persistFailing(calls *int, bad ...int64) func([]writeJob) error {
return func(js []writeJob) error {
*calls++
for _, j := range js {
for _, b := range bad {
if j.from == b {
return errors.New("duplicate key")
}
}
}
return nil
}
}

func TestPersistWithFallbackAllCommit(t *testing.T) {
var calls int
var failed []int64
got := persistWithFallback(context.Background(), jobs(1, 11, 21), persistFailing(&calls),
func(j writeJob, _ error) { failed = append(failed, j.from) })
if !equal(froms(got), []int64{1, 11, 21}) || len(failed) != 0 || calls != 1 {
t.Fatalf("committed=%v failed=%v calls=%d, want [1 11 21] [] 1", froms(got), failed, calls)
}
}

// One bad job dead-letters only itself; its batch-mates still commit
// (MAG-5084: previously every job in the aggregated tx was dead-lettered).
func TestPersistWithFallbackIsolatesBadJob(t *testing.T) {
var calls int
var failed []int64
got := persistWithFallback(context.Background(), jobs(1, 11, 21), persistFailing(&calls, 11),
func(j writeJob, _ error) { failed = append(failed, j.from) })
if !equal(froms(got), []int64{1, 21}) {
t.Fatalf("committed = %v, want [1 21]", froms(got))
}
if !equal(failed, []int64{11}) {
t.Fatalf("failed = %v, want [11]", failed)
}
if calls != 4 {
t.Fatalf("persist calls = %d, want 4 (1 aggregated + 3 single)", calls)
}
}

func TestPersistWithFallbackSingleJobFails(t *testing.T) {
var calls int
var failed []int64
got := persistWithFallback(context.Background(), jobs(5), persistFailing(&calls, 5),
func(j writeJob, _ error) { failed = append(failed, j.from) })
if len(got) != 0 || !equal(failed, []int64{5}) || calls != 1 {
t.Fatalf("committed=%v failed=%v calls=%d, want [] [5] 1", froms(got), failed, calls)
}
}

// On shutdown the failure is the cancelled ctx, not the jobs: nothing is
// retried or dead-lettered.
func TestPersistWithFallbackCancelledCtx(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background())
cancel()
var calls int
var failed []int64
got := persistWithFallback(ctx, jobs(1, 11), persistFailing(&calls, 1),
func(j writeJob, _ error) { failed = append(failed, j.from) })
if len(got) != 0 || len(failed) != 0 || calls != 1 {
t.Fatalf("committed=%v failed=%v calls=%d, want [] [] 1", froms(got), failed, calls)
}
}
Loading
Loading