Skip to content

fix: cap dead-letter retries and classify permanent fetch errors - #5

Merged
nimrod-teich merged 1 commit into
mainfrom
fix/dead-letter-carousel
Apr 18, 2026
Merged

nimrod-teich merged 1 commit into
mainfrom
fix/dead-letter-carousel

Conversation

@nimrod-teich

Copy link
Copy Markdown
Contributor

Summary

  • Fixes the dead-letter retry carousel: ~10k failed heights were cycling through the sweep every ~55 min with no give-up threshold. Every fetch error was treated the same — permanent errors (no endpoint coverage, LCD pruning) rejoined the carousel forever.
  • Adds error classification (permanent vs transient), a failure_max_retries cap (default 5) that promotes rows to permanent = true server-side in the RecordFailure UPSERT, and an additive schema migration on indexer_failures.
  • PopFailures excludes permanent rows so the sweep stops spending fetch budget on unrecoverable heights; permanent rows stay in the table for operator review. UI splits "N dead-letter ranges" into "N retrying · M permanent".

How the classification works

Error Classified as Behaviour
rpc.ErrNoEndpointCovers (no eligible endpoint for requested range) permanent Fast-path: whole range recorded permanent in one pass, no bisection
rpc.ErrHeightPruned (LCD 400/404 on block path) permanent Same
ErrRateLimited, 5xx, timeouts, network errors transient Existing behaviour: bisect → retry → dead-letter → sweep
context cancel/deadline propagated Unchanged

Schema migration

ALTER TABLE indexer_failures ADD COLUMN IF NOT EXISTS permanent BOOLEAN NOT NULL DEFAULT FALSE;
ALTER TABLE indexer_failures ADD COLUMN IF NOT EXISTS max_retries_reached_at TIMESTAMPTZ;
-- + partial-friendly index

Applied on startup via ensureFailuresTable. Existing rows default to permanent = false and continue behaving exactly as before.

Config

indexer:
  failure_max_retries: 5   # new; 0 disables the cap

Test plan

  • go build ./... clean
  • go vet ./... clean
  • Unit tests pass without a DB (classify, error sentinels)
  • Integration tests pass against a real Postgres with TEST_DATABASE_URL (pop-filter, cap-promotion, no-cap-never-promotes, legacy-row migration, RecordPermanentFailure insert+upgrade)
  • Deploy to a staging indexer and confirm the permanent count climbs while the retrying count drains
  • Verify the overview card renders "N retrying · M permanent" against real data
  • Clear permanent rows manually after adding an archive endpoint — confirm they re-index cleanly on next run

The in-run dead-letter sweep re-queued failed heights forever because
nothing ever consulted the `retries` counter and every error looked the
same. On a live indexer this manifested as 10k+ heights cycling through
the sweep every ~55 min with zero convergence.

Changes:

* Typed sentinels `rpc.ErrNoEndpointCovers` and `rpc.ErrHeightPruned`
  carry structured context and match via `errors.Is` through wrapping.
  LCD 400/404 on block paths are promoted to `HeightPrunedError` via a
  new `HTTPStatusError` wrapper.
* `internal/pipeline/classify.go` splits fetch errors into permanent vs
  transient. Permanent errors fast-path in `processWithSplit` — no
  bisection; every height in the range is recorded as permanent in a
  single pass.
* Additive migration on `indexer_failures`: `permanent BOOLEAN NOT NULL
  DEFAULT FALSE`, `max_retries_reached_at TIMESTAMPTZ`, plus a
  partial-friendly index. `ALTER TABLE … ADD COLUMN IF NOT EXISTS` so
  existing deployments upgrade on startup.
* `failure_max_retries` (default 5): promotion to `permanent = true`
  happens server-side in the `RecordFailure` UPSERT via a CASE, no RMW
  race.
* `PopFailures` filters `permanent = FALSE` — permanent rows stay for
  operator review but never rejoin the carousel.
* `FailureCount` returns `(retrying, permanent, maxRetries)`; the UI
  surfaces both counts as "N retrying · M permanent".
@nimrod-teich
nimrod-teich merged commit d9a7553 into main Apr 18, 2026
1 check passed
@nimrod-teich
nimrod-teich deleted the fix/dead-letter-carousel branch April 18, 2026 18:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant