Skip to content

fix(cannon): unstick four stalled backfill derivers - #914

Merged
samcm merged 7 commits into
masterfrom
fix/cannon-deriver-stalls
Oct 20, 2026
Merged

samcm merged 7 commits into
masterfrom
fix/cannon-deriver-stalls

Conversation

@samcm

@samcm samcm commented Oct 20, 2026 •

Copy link
Copy Markdown
Member

Bumps go-eth2-client so the attester slashing deriver handles Altair/Bellatrix blocks (mainnet slot 6110523) and the attestation reward deriver decodes negative inactivity penalties (mainnet epoch 200759). The sync committee and sync aggregate derivers read the committee from the Altair fork state when a sync committee period began before Altair (sepolia period 0, Altair at epoch 50). The execution transaction deriver still derives transactions when blobs are past the beacon node's retention window (400, 404 or a short sidecar list), storing NULL blob sizes for blob transactions; inside the window those cases and any other fetch error fail the slot. GetGasPrice handles Capella. Tests use real mainnet blocks (slots 6110523, 6316823, 13795744, 15416308), a real blob sidecar and a trimmed lighthouse rewards response.

samcm added 7 commits October 20, 2026 11:40
…estation rewards

Bumps go-eth2-client for two decoding bugs that stalled mainnet backfill:
- Altair/Bellatrix attester slashings came back labelled Phase0, so the
  deriver failed with "no Phase0 indexed attestation" at slot 6110523.
- Attestation rewards decoded inactivity (and head) as unsigned; mainnet
  epoch 200759 has inactivity penalties of -5722 from the May 2023 leak.

Tests use the real slot 6110523 block and a trimmed lighthouse response
for epoch 200759.
Sync committee period 0 on sepolia spans epochs 0-255 but Altair forked at
epoch 50, so the sync committee and sync aggregate derivers asked for the
pre-Altair state at slot 0 and the node rejected it. A period that began
before Altair now reads its committee from the fork epoch's state.
The execution transaction backfill stalled at mainnet slot 13795744 and
sepolia slot 9722304: the beacon node answers 400 "Insufficient data
columns to reconstruct blobs" for blobs older than the data column
retention window, and every non-404 error was fatal.

Blob sidecars only feed the blob size columns, so when the slot is past
MIN_EPOCHS_FOR_BLOB_SIDECARS_REQUESTS (MIN_EPOCHS_FOR_DATA_COLUMN_SIDECARS_REQUESTS
from Fulu) the transactions are still derived and the blob sizes of blob
transactions are left NULL. Inside the window errors still fail the slot.
A blob transaction with any missing sidecar (404, Gloas reconstruction
failures, pruned Deneb responses) now gets NULL sizes instead of 0.

GetGasPrice had no Capella case, so the backfill would have stalled again
at the first Capella block with a type-2 transaction.
Inside the retention window a 404 or a sidecar list shorter than the
block's commitments now fails the slot. Past it, only 400, 404 and short
lists leave blob sizes NULL; timeouts, 5xx and connection errors still
fail and retry. Gloas reconstruction failures keep NULL.

The decision moves into resolveBlobSidecars, tested per status code and
retention side, and the available-blob test uses mainnet slot 15416308
with its real blob sidecar.
@samcm
samcm merged commit 81bb024 into master Oct 20, 2026
6 checks passed
@samcm
samcm deleted the fix/cannon-deriver-stalls branch October 20, 2026 02:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant