solana-indexer: advance the finalized watermark from slot statuses

[squadgazzz]

Description

The yellowstone subscription only delivered confirmed slot statuses, so the indexer never saw finality: solana.indexer_state.finalized_slot and solana.chain_tip had no writer, and the last-indexed watermark only advanced when a settlement flushed. Since we eventually need to support resume/backfill and reorgs functionality for older slots, this needs to be changed.

This is the first half of BE-203. The rollback cascade for rows above the finalized watermark follows in its own PR.

solana.chain_tip existed to answer "is the indexer keeping up with the chain". The watermark could not answer that alone, since it only moved on settlements, so the planned monitoring needed a second row that tracked the chain to compare against. With this PR, the watermark moves on every confirmed slot, so "keeping up" is simply "the watermark keeps moving", and one row answers the question. That leaves chain_tip with no reader and no purpose, so it goes away.

Changes

• The slot filter subscribes to every status transition. The ingester forwards confirmed statuses (flush signal and tip counter) and finalized ones (watermark), and drops the rest: a status ahead of the transaction stream's commitment must not flush a buffer whose transactions are still in flight.
• finalized_slot advances from finalized statuses, monotone and update-only.
• The last-indexed watermark advances on quiet slots up to the flush hold-back, so the resume point tracks the stream instead of the last settlement.
• solana.chain_tip is dropped (V2 migration), see above: the quiet-slot watermark replaces it as the freshness signal.

How to test

New unit tests and ignored postgres tests, including the pipeline test driving finalized and quiet-slot statuses end to end.

Related issues

BE-203

[linear-code]

BE-203

[claude]

Claude finished @squadgazzz's task in 4m 32s —— View job

---

PR Review: advance the finalized watermark from slot statuses

• Read the PR diff and understand the changes
• Review the ingester/decoder slot-status routing logic
• Review the persistence + migration changes
• Review tests
• Post findings

---

Verdict: looks good — no blocking issues found. This is a clean, well-documented change and the reasoning in the comments matches the code.

I traced the core correctness concerns and they hold up:

• Quiet-slot watermark advance (decoder.rs:171-179) rests on the same in-order + hold-back invariant the buffer flush already relied on: once a confirmed status for observed is seen, no transaction for a slot ≤ observed - FLUSH_HOLDBACK_SLOTS will still arrive. So advancing last_indexed_slot to cutoff on quiet slots doesn't introduce a resume gap that the existing flush path didn't already assume away. The cutoff > 0 guard correctly avoids writing slot 0 near genesis, and flushed_through stays monotone across the flush loop and the quiet branch.
• finalized_slot can legitimately run ahead of last_indexed_slot at cold start (finalized statuses replay from from_slot before the confirmed frontier catches up). That's harmless here — no rows exist for those slots yet, and any later-inserted row at a slot ≤ finalized_slot is genuinely final. Worth keeping in mind for the rollback-cascade PR, which will need to tolerate finalized_slot > slot.
• write_finalized_slot is correctly update-only + monotone via GREATEST, and the no-state-row no-op is covered by a test. Dropping chain_tip is safe — confirmed there is no remaining reader or writer in the codebase (latest_chain_slot is now log-only, and the table never had a DB writer).
• Status routing in handle_slot correctly forwards only Confirmed/Finalized and drops the rest (including Processed, which is the proto zero-default, and Dead), matching the filter_by_commitment: false subscription. Tests cover confirmed / finalized / dropped-processed paths.

Optional (non-blocking): write_finalized_slot (persistence.rs:348-356) issues an unconditional UPDATE on every finalized status (~2.5/s). GREATEST already guarantees monotonicity, but adding a WHERE finalized_slot < $1 would skip the WAL write on reconnect replays, mirroring the guard in upsert_last_indexed_slot. Marginal given finalized slots normally only move forward — mentioning for symmetry, not asking for a change.

Note: I couldn't run cargo check/tests in this environment (sandbox denied the command), so compilation/test verification is by inspection only. The imports (SlotStatus from types::wire) and helper signatures all line up.
· branch solana-indexer/be-203-finalized-watermark

[MartinquaXD]

how does this check behave if flushed_through is None? Is None < Some(Slot(cutoff))?
Would be nice to make this less ambiguous.

[squadgazzz]

That comparison is gone, since flushing now keys off the confirmed slot status, and the watermark update is an explicit watermark.is_none_or(|w| w < confirmed), so no Option ordering is involved anymore.

[MartinquaXD]

Why do we only write_last_indexed_slot() when we didn't flush to the cutoff yet?
It's very unclear to me what purpose cutoff and flushed_through have in relation to the last indexed block watermark.

[squadgazzz]

Simplified and gone now.

[MartinquaXD]

What happens if we encounter an error after we already replaced the data in the line above?
Seems like the data would be lost forever. Do we have to restart the indexer to not lose any data?

[squadgazzz]

Yes, that slot is gone from memory, but the error kills the indexer, and the watermark only advances after a successful flush. So the restart resumes below the failed slot and the stream re-delivers it. Updated the doc.