feat(l1): pruning of historical blocks/receipts/txs - #6673
Conversation
Adds opt-in rolling-window pruning of block bodies, receipts, and
transaction-location entries for canonical blocks below a configurable
wall-clock retention window. Canonical headers are preserved; orphan
block data at finalized heights is also deleted.
Architecture:
- New BLOCK_HASHES_BY_NUMBER CF (`BE block_number || block_hash` -> empty)
indexes every known hash per height. Written on all header-insertion
paths, consumed by the pruner for per-height enumeration.
- Schema v1 -> v2 migration: backfills the new CF from HEADERS via
StorageReadView::full_scan (streaming, bounded heap, no Keccak
recomputation). Also fixes EarliestBlockNumber on snap-synced v1 DBs
by binary-searching for the lowest canonical block with a body.
- HistoryPruner background task: every 12s, computes
`target = min(finalized, find_canonical_block_by_timestamp(now - retention))`
and atomically prunes a contiguous range via Store::prune_block_heights.
Each pass: rayon-parallel gather phase (one read txn per height) +
single WriteBatch with DeleteRange on the per-height index and
body-conditional bodies/receipts/tx_locations deletes (no pointless
tombstones for snap-synced pre-pivot heights). Bounded by a 2s pass
timeout and 4096-block batch. Feature-gated metrics
(earliest/target/lag gauges, pass-duration/blocks histograms,
per-category deletion counters).
- Snap-sync completion sets EarliestBlockNumber to the pivot from
persisted chain metadata (before clearing snap state, so a crash in
the window leaves the operation retryable).
- For chains without engine-API finality (or any node before its first
FCU) the pruner falls back to `head - 256` as the prune floor so it
still does useful work.
- RPC: eth_getBlockBy{Number,Hash} with include_txs=true returns the
header with an empty transactions list for pruned blocks;
eth_getLogs clamps fromBlock to EarliestBlockNumber while preserving
the malformed-range error; tx-by-hash, receipt, and getBlockReceipts
return null naturally.
- CLI: `--history.retention=<duration>` (e.g. 1d, 12h, 365d) enables
the pruner; omitted by default.
# Conflicts: # cmd/ethrex/cli.rs # crates/storage/api/tables.rs # crates/storage/store.rs
Lines of code reportTotal lines added: Detailed view |
Resolved conflicts from main's TRANSACTION_LOCATIONS schema change and overlapping schema-version bump: - Schema bumped to v4. main's tx-locations rewrite stays v2->v3 (migrate_2_to_3); this branch's BLOCK_HASHES_BY_NUMBER backfill becomes v3->v4 (migrate_3_to_4), so released-main nodes (already at v3) only run the new backfill. - write_block_indices now uses the merge-based tx-location write (tx.merge / encode_tx_location_operand) instead of composite-key puts. - prune_block_height reworked for the tx-hash-keyed Vec schema: read-modify-write per tx (trim pruned blocks' locations, delete key if empty), accumulated across the batch so a tx in multiple pruned blocks is handled once. - Kept both api/backend trait methods (delete_range + merge) and both test modules (pruning_index_tests + merge_tests).
# Conflicts: # crates/storage/migrations.rs
# Conflicts: # crates/storage/store.rs
|
Benchmark Results ComparisonNo significant difference was registered for any benchmark run. Detailed ResultsBenchmark Results: BubbleSort
Benchmark Results: ERC20Approval
Benchmark Results: ERC20Mint
Benchmark Results: ERC20Transfer
Benchmark Results: Factorial
Benchmark Results: FactorialRecursive
Benchmark Results: Fibonacci
Benchmark Results: FibonacciRecursive
Benchmark Results: ManyHashes
Benchmark Results: MstoreBench
Benchmark Results: Push
Benchmark Results: SstoreBench_no_opt
|
|
Wall clock is an interesting decision. Every other client does either blocks or epochs. Ofc slot time is deterministic and wall clock can be computed from epochs by the user |
get_receipts_for_block returns a bare Vec, so "this block's receipts are not stored" and "this block had no transactions" are the same value. get_all_block_receipts passed that straight through, so debug_getRawReceipts answered an empty list for a block that has transactions — a wrong answer rather than a reported failure. Check the receipt count against the block's own transaction count, and report a block whose body is absent rather than validating against nothing. This mirrors the mismatch check the by-index receipt path already performs. Genesis keeps its existing short circuit, since it legitimately has none. This matters ahead of history pruning (#6673): an absent receipt set stops being a corruption signal and becomes a normal steady-state outcome, so every path that conflates it with emptiness starts returning wrong answers on every node rather than on a corrupted one.
Conflicts and API drift from 47 commits of main: - chain_data_key: this branch hoists it into utils.rs as the single source of truth shared with migrations.rs, which its own code depends on at five call sites; main kept it in store.rs and added decode_block_number beside it. Kept the move, kept main's new decode_block_number, and dropped the From<u8> impl the branch re-added — main deleted it as dead in #7187 and it is unused here too. - get_latest_block_number is sync on main now; dropped three stale .awaits. - InMemoryPrefixIter was removed by #7186; full_scan returns the iterator directly, matching its sibling. - update_earliest_block_number was renamed here to advance_earliest_block_number with monotone-up semantics, which main's descending writers cannot use. Added set_earliest_block_number for them: the pruner's floor only rises, but backfill fills history in and must be able to move the pointer down. - BlockRangeUpdate::new awaited without being async, so the branch did not build.
Test setup sites establish a starting frontier rather than advancing a floor, so the monotone-up guard is wrong for them.
The window a consensus client must serve blocks over is the natural lower bound for execution history: the EL has to supply payloads at least as long as the CL has to serve them. That window is MIN_EPOCHS_FOR_BLOCK_REQUESTS, 33024 epochs (MIN_VALIDATOR_WITHDRAWABILITY_DELAY 256 + CHURN_LIMIT_QUOTIENT 65536 / 2), and it is now the default. The flag becomes `cl-window | all | <N>epochs` instead of a wall-clock duration. Bare numbers are rejected: --history.chain already takes an absolute block number, so a neighbouring flag where a bare number meant a distance would read identically and mean something else. The barrier is now one subtraction on block numbers rather than a timestamp search from the host clock. A block distance is a sound proxy for a slot distance because EIP-3675 permits at most one block per slot and a missed slot produces no block, so the first canonical block inside the window is always above head - window - slack; the error is one-sided toward keeping more. This removes two hazards: a host clock skewed a year forward pruned a year of history, and the timestamp bisection assumed every canonical height in range had a header, which is false on a database with a header hole. HistoryRetention::Blocks carries the underlying unit, since an epoch is 32 blocks but the slack alone is 7200, so no epoch value can produce a barrier above zero on a test-sized chain. A default that prunes must not delete history an existing datadir already holds, so the gate is: an explicit flag prunes, the default only prunes what it would have kept anyway. A node holding pre-window history keeps it and warns every boot until the operator opts in. The staleness probe reads block 1, never 0 — genesis always has a body, so probing 0 always succeeds and would conclude a stale earliest-block pointer was sound, grandfathering every node that snap-synced on an older build. --history.chain disables pruning by default rather than fighting it: backfilled history would otherwise be deleted as fast as it arrives.
Covers the default equalling MIN_EPOCHS_FOR_BLOCK_REQUESTS, All never pruning, a chain shorter than the window keeping everything rather than underflowing, and the one-sided property that makes a block distance a safe proxy for a slot distance.
Two read paths treated an absent receipt set as something other than absent history, which becomes the normal outcome below the prune barrier rather than a corruption signal. log_index_base summed whatever receipts came back with no count check, so a short read silently under-counted and shifted every log index in the trace — a wrong answer rather than a reported failure. It now checks the count against the block's transaction count. eth_getLogs reported a missing per-transaction receipt as an internal error. A caller can act on 4444 by asking a node that still holds the range; it cannot act on a 500.
🤖 Kimi Code ReviewThis is a substantial and well-engineered PR introducing history pruning to ethrex. The implementation is conservative, safety-focused, and includes comprehensive tests. Below are specific findings and suggestions. Critical Safety Checks (All Clear)
Issues Found1. Documentation Typo (Minor)File: 2. Redundant String Allocation (Minor)File: // Current:
return Err(eyre::eyre!(format!(...)));
// Better:
return Err(eyre::eyre!(...)); // eyre! accepts format args directly3. Transaction Location Race DocumentationFile: 4. Migration Header DecodingFile: 5. Test-Only Code ExposureFile: Performance Observations1. Gather Phase ParallelismThe dedicated 4-thread pool for pruning gather operations ( 2. Uncached ReadsUsing 3. Range TombstonesThe Architecture Suggestions1. L2 Pruning Cap FallbackFile: 2. Metrics InitializationFile: 3. Earliest Block Pointer ValidationFile: Testing CoverageExcellent coverage including:
ConclusionThe PR is ready for merge with minor nits addressed. The safety architecture (multiple floors, atomic commits, no pointer regression) is sound, and the test coverage is comprehensive. The separation of concerns between L1 (time/epoch based) and L2 (commitment-based) retention policies is well-designed. Recommended fixes before merge:
Automated review by Kimi (Moonshot AI) · kimi-k2.5 · custom prompt |
🤖 Claude Code ReviewWaiting for the review agents to complete. Automated review by Claude (Anthropic) · sonnet · custom prompt |
🤖 Codex Code Review
Aside from those points, the storage-side pruning work and the new coverage around pruned RPC behavior look careful. I couldn’t verify with Automated review by OpenAI Codex · gpt-5.4 · custom prompt |
Four failing checks, three root causes. The L2 build could not compile: EthClient and StoreError were used without being imported, and the L2 initializer has its own pruner wiring that still logged the retention as seconds. Neither showed up under a default `cargo check` because the L2 code sits behind a feature flag, so `--features l2` is the check that matters here. humantime became an unused dependency when the flag stopped taking a wall-clock duration; removing it left three lockfiles stale, which failed `Check Cargo.lock` and, through it, `Lint zisk backend` and the `Lint` aggregator that reports it. docs/CLI.md is compared against `--help` output verbatim, so hand-editing the flag text could not match. Regenerated it from the binary.
Regenerating docs/CLI.md from a local --help baked this machine's datadir default into it. The committed doc carries the Linux form CI compares against, so the machine-dependent line has to be substituted rather than copied.
Codex flagged the L2 path enabling pruning only on an explicit flag as a mismatch with L1, where an omitted flag resolves to the CL block-retention window. The divergence is deliberate: that window is the range a beacon chain must serve blocks over, and no beacon chain serves an L2's blocks, so inheriting the default would delete history for a reason that does not apply. It was undocumented, which is what made it look like an oversight. Also warn when cl-window is passed explicitly on an L2, since the operator has then asked for a window defined by something that is not involved.
The wall-clock form was dropped along with the mechanism that made it dangerous, but only the mechanism had to go. The hazard was never the unit: the old code read SystemTime on every pruning pass and bisected the chain by timestamp, so a host clock a year fast deleted a year of real history, and the bisection additionally assumed every canonical height in range had a header. Converting a duration to a block distance once, at startup, keeps all of that fixed — no clock is read while running and the barrier is still one subtraction — while giving back the spelling an operator budgeting disk actually thinks in. `<N>d`, `<N>h` and `<N>m` are accepted and converted assuming ASSUMED_SECONDS_PER_SLOT. That assumption is the cost, so the help text says `<N>epochs` is exact and survives a slot-time change, and the constant is the one place to edit if it ever does. Parsed by suffix rather than reintroducing humantime.
This has shifted now, as of what's been discussed in the ACDEs recently, but without removing the wall-clock variant. We'll now support CL block retention window as the default, and either |
The error still said only cl-window, all and <N>epochs were accepted, so an operator who mistyped a duration was told durations are unsupported when they are.
Pruning is irreversible without a resync, and tokio's interval fires its first tick immediately, so a mis-gated pruner starts deleting at startup rather than after the interval. That makes 'try it and watch the logs' an unsafe way to validate the gate on a datadir you cannot replace. Dry run resolves the policy, reports the barrier and the exact number of block heights that would be deleted, and returns without spawning the pruner. It also reports the grandfathered case, so an operator can see both what the default decided and what an explicit flag would do.
…ass#7210) **Motivation** Three places hardcoded the advertised earliest block to `0`: | Site | What it feeds | |---|---| | `crates/networking/p2p/rlpx/eth/status.rs:69` | shared eth/69+ handshake status (eth/69, eth/71) | | `crates/networking/p2p/rlpx/eth/eth70/status.rs:108` | eth/70's own status struct | | `crates/networking/p2p/rlpx/eth/update.rs:33` | `BlockRangeUpdate` — the ongoing half of the same advertisement | So every eth/69+ peer was told we hold history from genesis. That is already untrue on any snap-synced node, where the earliest retained block is the pivot, and peers use the advertised range to decide what to request from us — meaning we invite requests we then answer with empty replies. It gets materially worse with history pruning (lambdaclass#6673): the earliest block moves continuously, so a hardcoded handshake value is wrong within hours of starting, and `BlockRangeUpdate` exists precisely to keep peers current as the range moves. **Description** Reads the value from the store in all three places. `get_earliest_block_number` is async (`crates/storage/store.rs:1256`) while these constructors were sync — there is no cached sync accessor for it the way `get_latest_block_number` has one (`:1275`, backed by `latest_block_header`). So the constructors and their four handshake call sites in `rlpx/connection/server.rs` become async. The alternative, caching earliest in memory alongside latest, is a larger change for no benefit on a path that runs once per handshake. **Tests** Three tests in `crates/networking/p2p/rlpx/eth/status.rs`: - the eth/69, eth/70 and eth/71 handshake statuses all advertise a non-zero earliest block set on the store - `BlockRangeUpdate` reads the field from the store rather than hardcoding it, and the result still satisfies `validate`'s `earliest <= latest` invariant - a genesis-synced node still advertises `0`, so this is a no-op for full-history nodes Note the second test deliberately uses `earliest = 0` on a genesis-only store: setting a non-zero earliest above the latest block manufactures a range that `validate` correctly rejects, which is a state a real node cannot reach.
…g empty (lambdaclass#7208) **Motivation** `get_receipts_for_block` returns a bare `Vec` (`crates/storage/store.rs:1491-1497`), so "this block's receipts are not stored" and "this block had no transactions" are the same value. `get_all_block_receipts` passed that straight through with no length check, so `debug_getRawReceipts` answered an **empty list** for a block that has transactions — a wrong answer rather than a reported failure. The sibling by-index path already guards against this (`crates/networking/rpc/eth/block.rs:408-414`, "Return 500 on receipt count mismatch"); the all-receipts path did not. This matters ahead of history pruning (lambdaclass#6673). Today an absent receipt set means corruption, which is rare. Once pruning lands it becomes a normal steady-state outcome, so every path that conflates "absent" with "empty" starts returning wrong answers on every node rather than on a corrupted one. Making these paths honest is a prerequisite for pruning being safe to enable, and it is worth doing on its own merits regardless. **Description** `get_all_block_receipts` now checks the receipt count against the block's own transaction count, and reports a block whose body is absent rather than validating against nothing. Genesis keeps its existing short circuit, since it legitimately has no receipts. The error is `RpcErr::Internal` to match the sibling path. Once lambdaclass#7069 lands its `PrunedHistoryUnavailable` variant (JSON-RPC code 4444), both sites should move to it together — an absent-because-pruned block deserves a distinct code from a corrupt one, and splitting that out keeps this change reviewable. **Tests** Two tests in `test/tests/rpc/raw_receipts_completeness_tests.rs`. The harness stores blocks and their transactions but no receipts, which is exactly the shape a pruned block presents — body present, receipts gone: - a block with one transaction and no stored receipts must report a failure rather than return an empty list. Verified to fail without the change (`test result: FAILED`) and pass with it - genesis must still answer with an empty list, so the completeness check does not break the one block that legitimately has none
|
|
…ult in warnings Both defects were found by running the gate tests against a mainnet node holding history from the first post-merge block. --history.chain with an explicit finite --history.retention was silently accepted and pruning proceeded. Backfill fills history downward toward an absolute floor while pruning deletes upward from a rolling barrier, so with the floor below the barrier the node downloads blocks only to delete them, re-detects the gap, and downloads again. On the node under test that is 9.2M blocks of churn against 84 GB of free disk. It is now a startup error naming both flags. The grandfathered-node warning said only that pruning was not enabled. It now states how many block heights an explicit flag would delete, so an operator can see the stakes without having to run anything.
3 participants
Motivation
Block bodies, receipts and transaction locations grow without bound, and after Glamsterdam they grow faster. The consensus layer is only required to serve blocks over
MIN_EPOCHS_FOR_BLOCK_REQUESTS— 33024 epochs, roughly 4.8 months — and that is the natural lower bound for execution history: the EL must be able to supply payloads at least as long as the CL has to serve them, and anything beyond that is optional. Following the ACDE discussion (ethereum/pm#2197), that window is now the default retention.Description
Background pruning of block bodies, receipts and transaction locations below a retention barrier. Canonical headers are always kept. A new block-number → known-hashes-at-that-height table lets orphaned blocks be pruned too.
--history.retentioncl-window|all|<N>epochscl-window--history.retention.below-cl-windowfalseBare numbers and wall-clock durations are rejected.
--history.chainalready takes an absolute block number, so a neighbouring flag where a bare number meant a distance would read identically and mean something else.The barrier is one subtraction on block numbers, not a timestamp search:
A block distance is a sound proxy for a slot distance because EIP-3675 permits at most one block per slot and a missed slot produces no block, so the first canonical block inside the window always sits above the barrier. The error is one-sided toward over-retention — missed slots make us keep more, never less. This removes two hazards the earlier timestamp-based cutoff had: it derived the cutoff from the host clock, so a machine skewed a year forward pruned a year of real history, and the bisection assumed every canonical height in range had a header, which is false on a database with a header hole.
HistoryRetention::Blocksexposes the underlying unit, since an epoch is 32 blocks but the slack alone is 7200, so no epoch value can produce a barrier above zero on a test-sized chain.A default that prunes must never delete history a datadir already holds. So: an explicit flag prunes, the default only prunes what it would have kept anyway. A node holding pre-window history keeps it and warns on every boot until the operator opts in. The staleness probe for the earliest-block pointer reads block 1, never 0 — genesis always has a body, so probing 0 always succeeds and would conclude a stale pointer was sound, grandfathering every node that snap-synced on a build predating #7024.
--history.chaindisables pruning by default rather than fighting it: backfilled history would otherwise be deleted as fast as it arrives.Read paths that could conflate "absent" with "empty" now report unavailable history with JSON-RPC code 4444 instead of returning a plausible-but-wrong value:
eth_getLogs(both theblockHashand range branches, plus the per-transaction receipt fetch) andlog_index_base, which previously summed a short receipt read and silently shifted every log index in a trace.Two setters now exist where there was one. The pruner's floor only ever rises, so
advance_earliest_block_numberrefuses to regress; backfill fills history in, so reconciliation moves the pointer down and needsset_earliest_block_number. A single monotone-up setter silently swallowed the second case.Not in scope, deliberately
engine_getPayloadBodiesBy*V2through a full block re-execution before degrading, so pruning them would put repeated failed re-executions on the consensus-critical path. They grow below the barrier; that is a known, smaller cost than the alternative.trace_*namespace andeth_gasPrice's 20-block lookback are untouched; the latter is safe only because it sits under the pruner's 256-block near-head floor, which is a constant coincidence worth a follow-up.Depends on
#7207 (
eth_feeHistoryrange inversion — an unauthenticated remote abort that pruning makes far easier to hit), #7208 (debug_getRawReceiptsanswering empty for absent receipts) and #7210 (advertising the earliest block we can actually serve, which peers use to decide what to request). This branch'supdate.rschange overlaps #7210 and should keep this version — its error handling is stricter.Tests
MIN_EPOCHS_FOR_BLOCK_REQUESTS;allnever prunes; a chain shorter than the window keeps everything rather than underflowing; and the one-sided property that makes the block-distance proxy safe.cl-window, since only an explicit one may delete existing history.eth_getLogsreturning 4444 rather than a wrong answer on pruned ranges.Whole workspace is clean under
cargo clippy --all-targets -- -D warnings; 127 storage, 121 rpc, 58 blockchain and 967 integration tests pass.