perf: move blocking work off the async runtime

AGENTS.md

@@ -55,6 +55,7 @@ Both binaries share a generic three-stage pipeline defined in `stateless-core::p
 1. **Fetch** — `block_fetcher` streams blocks + witnesses from a `BlockFetcher` via a bounded in-flight window (concurrency capped by `fetcher_max_in_flight`).
 2. **Process** — N workers run `BlockProcessor::process` (validator: EVM execution; trace server: pass-through).
 3. **Advance** — `chain_advancer` reorders out-of-order results, verifies parent-hash continuity, detects reorgs, and persists via `ChainStore::advance_chain`.
+   The hooks' `pre_advance` and that commit are synchronous multi-millisecond disk work, so they run on the blocking pool rather than on an async worker (the trace server shares its runtime with the RPC handlers); a panic in either still unwinds the advancer unchanged, via `try_into_panic` → `resume_unwind`.
 
 The outer loop (`run_pipeline`) handles reorg rollback + restart, stale-data anchor reset, and transient vs fatal error classification.
 On a detected reorg, the rollback floor comes from a pluggable `ReorgResolver`: the core `BisectResolver` walks local history via `find_divergence_point`, while an embedder with an externally-supplied floor (e.g. the mega-reth FullNode) provides its own.
@@ -155,7 +156,7 @@ The validator splits the two the same way: `--r2-max-concurrent-requests` caps R
 Client-side routing, budgets, and fallback match the S3 target, but edge behavior is zone configuration: **a cache rule making these objects cacheable must set 404s to bypass cache**, or a pre-upload frontier miss gets pinned for the negative-cache TTL (stalling the validator's tip-following in its fallback-less R2 mode) and a cached 404 can false-fire the below-band `kind="missing"` bucket-integrity alarm.
 The bucket is the same store the public gateway reads and can lead the generator at the frontier (uploader and generator RPC server publish from different files), so frontier hits are real; the frontier band is a small near-tip window (`R2_FRONTIER_WINDOW`, 32 blocks of uploader-lag grace on either side of the local tip — deliberately far narrower than the 4096-block routing window, so a stale catching-up tip cannot silence holes above it), hits there are labeled `witness_r2_frontier` (vs `witness_r2` past the band), the speculative frontier probe runs on an eighth of the remaining stage (vs half for blocks R2 must hold, so degraded R2 cannot burn half of every near-tip request's budget), and a `missing` classifies by band: in-band is the expected probe-ahead outcome (excluded from the alarm), below-band feeds `debug_trace_r2_witness_errors_total{kind="missing"}` (the bucket-integrity alarm, still covering recent-but-below-tip holes), and above-band — only reachable behind a stale catching-up tip — lands on its own `kind="missing_above_tip"` series, visible without flooding the alarm on every catch-up.
 Any witness-chain RPC attempt under a deadline is capped at the tightest of three bounds — half the full witness stage (`RpcClientConfig::witness_per_attempt_timeout`, derived from `--witness-timeout`), the global `--rpc-per-attempt-timeout-ms` (an explicitly stricter operator setting is honored, never loosened), and — only while the round still has an untried provider to rotate to — half of what the call still has as the attempt starts (recomputed after any concurrency-permit wait, so neither an old-block-clamped stage, a post-R2 remainder, nor a long permit queue defeats the reserve).
-The round's last hop, and every hop of a single-provider chain, takes the remainder whole under the ceiling instead: rotation stays protected without structurally condemning a slow-but-honest transfer, and the witness decode runs outside the attempt window (bounded by the deadline alone), so CPU-bound decode neither burns the reserve nor reads as a provider stall while a corrupt payload still rotates as the provider's error; deadline-less chain-sync fetches keep the general 20s cap so a slower-than-cap transfer still completes.
+The round's last hop, and every hop of a single-provider chain, takes the remainder whole under the ceiling instead: rotation stays protected without structurally condemning a slow-but-honest transfer, and the witness decode — like `eth_getBlock`'s integrity verification, which runs on the blocking pool — runs outside the attempt window (bounded by the deadline alone), so CPU-bound work neither burns the reserve nor reads as a provider stall while a corrupt payload still rotates as the provider's error; deadline-less chain-sync fetches keep the general 20s cap so a slower-than-cap transfer still completes.
 When a logical upstream call gives up on its deadline it logs one WARN naming the `phase` it died in (`before_attempt` / `permit_wait_clamped` / `attempt_clamped` / `before_backoff`) with `provider` / `round` / `permit_wait_ms` / `attempt_ms`, and the abandoned attempt is recorded as `outcome="deadline_clamped"` rather than dropped; best-effort internal probes (the throttled upstream tip seed) demote that give-up log to debug while the deadline metric still fires, so a probe whose failure is already degraded cannot page as a user-visible incident.
 Permit wait is timed separately (`debug_trace_upstream_permit_wait_seconds{method}`) and the acquire is clamped to the deadline (phase `permit_wait_clamped`, cut-short wait still sampled), so queueing behind our own `--witness-max-concurrent-requests` stays distinguishable from endpoint slowness and a saturated queue cannot block a call past its budget unobserved.
 The background chain-sync prefetch routes by freshness against the last observed remote head: frontier-fresh blocks give the generator a short exclusive grace (its "witness not found" means "not generated yet" — fallbacks are fed by the same pipeline and cannot be ahead) before falling back to the full endpoint chain, while deep catch-up blocks — and any block classified against a stale head observation (older than the grace, as during a long catch-up stretch when the tip is not re-polled) — use the full chain from the first attempt.

README.md

@@ -209,7 +209,7 @@ A count of zero, a non-numeric or blank one, one set alongside the S3 endpoint (
 The configured target is published as the constant-1 gauge `debug_trace_r2_target_info{target}` (`r2_target_info` on the validator), so the target-less R2 series can be attributed to one target or the other during a rollout.
 Whether the domain actually delivered HTTP/2 is a separate question — selection is pure ALPN, so a misconfigured zone degrades to HTTP/1.1 with the h2 tuning inert — and is answered by `..._r2_negotiated_http_version_info{version}` plus a one-time warning naming the protocol that was negotiated.
 Any single witness-chain RPC attempt under a deadline is additionally capped at the tightest of: half the witness stage budget, the global per-attempt timeout, and — only while the round still has an untried provider to rotate to — half of what the call still has as the attempt starts (recomputed after any permit wait).
-The round's last hop, and every hop of a single-provider chain, takes the remainder whole under the ceiling instead, so a stalled endpoint (or a saturated concurrency permit — waits are deadline-bounded too) can never consume the stage while a rotation is still worth reserving for, and a slow-but-honest transfer is never structurally condemned; the witness decode runs outside the attempt window, bounded by the request deadline alone.
+The round's last hop, and every hop of a single-provider chain, takes the remainder whole under the ceiling instead, so a stalled endpoint (or a saturated concurrency permit — waits are deadline-bounded too) can never consume the stage while a rotation is still worth reserving for, and a slow-but-honest transfer is never structurally condemned; the witness decode — and `eth_getBlock`'s integrity verification, which runs on the blocking pool — runs outside the attempt window, bounded by the request deadline alone.
 `--r2-max-concurrent-requests` caps in-flight GETs separately from `--witness-max-concurrent-requests` — the RPC cap sizes a shared gateway, R2 tolerates far more.
 
 **Admission and response-size knobs** (each also settable via its `DEBUG_TRACE_SERVER_*` env var):
@@ -320,7 +320,7 @@ Both binaries share a generic three-stage pipeline defined in `stateless-core`:
                           Reorders out-of-order results (BTreeMap)
                           Verifies parent-hash continuity
                           Detects reorgs → rollback + restart
-                          Persists via ChainStore::advance_chain()
+                          Persists via ChainStore::advance_chain() (on the blocking pool)
 
  Outer loop (run_pipeline):
    Reorg → ReorgResolver decides floor → rollback → restart pipeline

crates/stateless-common/src/rpc_client.rs

@@ -452,15 +452,24 @@ impl RpcClient {
         })
     }
 
+    /// The data provider this call's first round starts at: rotated per call so healthy
+    /// endpoints share load evenly. Within a round the order is fixed (start → start+1 → …).
+    ///
+    /// Safety: the constructor guarantees at least one data provider. The atomic op is
+    /// skipped for a single provider — pointless contention otherwise.
+    fn next_data_rr_start(&self) -> usize {
+        let n = self.data_providers.len();
+        if n > 1 { self.data_rr_counter.fetch_add(1, Ordering::Relaxed) % n } else { 0 }
+    }
+
     /// Deadline-aware counterpart of [`Self::call`].
     ///
     /// With `deadline = Some(..)` the retry loop returns [`RpcDeadlineExceeded`] once the
     /// deadline passes, clamping each inter-round sleep so it doesn't overshoot. With
     /// `None` this is equivalent to [`Self::call`] and never returns `Err`.
     ///
-    /// Each call performs rounds of "try every data provider once in round-robin order".
-    /// The starting provider rotates per call via an atomic counter so healthy endpoints
-    /// share load evenly; within a round the order is fixed (start → start+1 → …).
+    /// Each call performs rounds of "try every data provider once in round-robin order",
+    /// starting at [`Self::next_data_rr_start`].
     async fn call_with_deadline<T: Send + 'static>(
         &self,
         method: RpcMethod,
@@ -481,18 +490,13 @@ impl RpcClient {
         best_effort: bool,
         f: impl Fn(RootProvider<Optimism>) -> BoxFuture<Result<T>>,
     ) -> std::result::Result<T, RpcDeadlineExceeded> {
-        // Safety: constructor guarantees at least one data provider.
-        let n = self.data_providers.len();
-        // Skip the atomic op when there's a single provider — avoids pointless contention.
-        let rr_start =
-            if n > 1 { self.data_rr_counter.fetch_add(1, Ordering::Relaxed) % n } else { 0 };
         round_robin_with_backoff(
             &self.data_providers,
             &self.data_provider_labels,
             &self.data_concurrency,
             &self.config.rpc_retry,
             AttemptCap::Fixed(self.config.per_attempt_timeout),
-            rr_start,
+            self.next_data_rr_start(),
             method,
             self.config.metrics.as_ref(),
             deadline,
@@ -554,22 +558,43 @@ impl RpcClient {
     }
 
     /// Deadline-aware counterpart of [`Self::get_block`].
+    ///
+    /// Verification is the retry loop's *finalize* step, not part of the attempt window: it
+    /// is per-transaction ECDSA recovery and re-encoding over a whole block — CPU-bound work
+    /// that would otherwise both burn the rotation reserve and read as a provider stall,
+    /// classifying a healthy endpoint serving a large block as stalled (see
+    /// [`round_robin_with_backoff`]). A verification failure still counts as that provider's
+    /// error, so a tampered block rotates exactly as a transport failure does.
     pub async fn get_block_with_deadline(
         &self,
         block_id: BlockId,
         full_txs: bool,
         deadline: Option<Instant>,
     ) -> std::result::Result<Block<Transaction>, RpcDeadlineExceeded> {
         let verify = !self.config.skip_block_verification;
-        self.call_with_deadline(RpcMethod::EthGetBlock, deadline, move |provider| {
-            Box::pin(async move {
-                let block = do_get_block_unchecked(&provider, block_id, full_txs).await?;
-                if verify {
-                    verify_block_integrity(&block)?;
-                }
-                Ok(block)
-            })
-        })
+        round_robin_with_backoff(
+            &self.data_providers,
+            &self.data_provider_labels,
+            &self.data_concurrency,
+            &self.config.rpc_retry,
+            AttemptCap::Fixed(self.config.per_attempt_timeout),
+            self.next_data_rr_start(),
+            RpcMethod::EthGetBlock,
+            self.config.metrics.as_ref(),
+            deadline,
+            false,
+            move |provider, _provider_label| {
+                Box::pin(async move { do_get_block_unchecked(&provider, block_id, full_txs).await })
+            },
+            move |block, _provider_label| {
+                Box::pin(async move {
+                    if !verify {
+                        return Ok(block);
+                    }
+                    verify_block_on_blocking_pool(block).await
+                })
+            },
+        )
         .await
     }
 
@@ -1573,6 +1598,21 @@ async fn decode_witness_wire<T: Send + 'static>(
     Ok(result)
 }
 
+/// [`RpcClient::get_block_with_deadline`]'s finalize half: [`verify_block_integrity`] on the
+/// blocking pool (per-transaction ECDSA recovery plus a re-encode of every envelope is
+/// CPU-bound over a full block), handing the block back untouched on success.
+///
+/// A failure here is an integrity failure from this provider — the retry loop records it as
+/// that provider's `Error` and rotates, exactly like a transport error.
+async fn verify_block_on_blocking_pool(block: Block<Transaction>) -> Result<Block<Transaction>> {
+    tokio::task::spawn_blocking(move || -> Result<Block<Transaction>> {
+        verify_block_integrity(&block)?;
+        Ok(block)
+    })
+    .await
+    .context("block verification task panicked")?
+}
+
 /// Verifies structural integrity of a block fetched from RPC.
 ///
 /// Checks:
@@ -1660,7 +1700,7 @@ mod tests {
         find_divergence_point,
         pipeline::{BlockFetcher, DivergenceLookups},
     };
-    use stateless_test_utils::mock_rpc::{header_stub, parse_hex_u64, serve};
+    use stateless_test_utils::mock_rpc::{consistent_header, header_stub, parse_hex_u64, serve};
     use tokio_util::sync::CancellationToken;
 
     use super::*;
@@ -1979,6 +2019,71 @@ mod tests {
         hb.stop().unwrap();
     }
 
+    /// A block with no transactions, so [`verify_block_integrity`] reduces to its header-hash
+    /// check and the fixture needs no signed transactions.
+    fn block_stub(header: alloy_rpc_types_eth::Header) -> Block<Transaction> {
+        Block {
+            header,
+            uncles: Vec::new(),
+            transactions: alloy_rpc_types_eth::BlockTransactions::Hashes(Vec::new()),
+            withdrawals: None,
+        }
+    }
+
+    /// Block verification is the retry loop's finalize step rather than part of the attempt
+    /// window (so a large block cannot read as a provider stall), and this pins the property
+    /// that move must not cost: an integrity failure is still that provider's error, so the
+    /// call rotates to the next provider instead of surfacing the bad block.
+    #[tokio::test]
+    async fn block_verification_failure_rotates_to_the_next_provider() {
+        let bad_hits = Arc::new(AtomicUsize::new(0));
+        let (bad_handle, bad_url) = serve(Arc::clone(&bad_hits), |m| {
+            m.register_method("eth_getBlockByNumber", |_params, hits, _| {
+                hits.fetch_add(1, Ordering::Relaxed);
+                // A hash the header does not actually hash to — the first integrity check.
+                Ok::<_, ErrorObjectOwned>(block_stub(header_stub(7, BlockHash::from([9u8; 32]))))
+            })
+            .unwrap();
+        })
+        .await;
+        let good_hits = Arc::new(AtomicUsize::new(0));
+        let (good_handle, good_url) = serve(Arc::clone(&good_hits), |m| {
+            m.register_method("eth_getBlockByNumber", |_params, hits, _| {
+                hits.fetch_add(1, Ordering::Relaxed);
+                Ok::<_, ErrorObjectOwned>(block_stub(consistent_header(7)))
+            })
+            .unwrap();
+        })
+        .await;
+
+        let config = RpcClientConfig {
+            rpc_retry: BackoffPolicy::new(Duration::from_millis(1), Duration::from_millis(2)),
+            ..Default::default()
+        };
+        let client = RpcClient::new_with_config(
+            &[bad_url.as_str(), good_url.as_str()],
+            &[good_url.as_str()],
+            config,
+            None,
+        )
+        .unwrap();
+
+        let block = client
+            .get_block_with_deadline(BlockId::number(7), false, None)
+            .await
+            .expect("None deadline cannot time out");
+        assert_eq!(
+            block.header.hash,
+            consistent_header(7).hash,
+            "the verified block must come from the second provider"
+        );
+        assert_eq!(bad_hits.load(Ordering::Relaxed), 1, "the tampering provider must be tried");
+        assert_eq!(good_hits.load(Ordering::Relaxed), 1, "rotation must reach the good provider");
+
+        bad_handle.stop().unwrap();
+        good_handle.stop().unwrap();
+    }
+
     /// After every provider in a round fails, the helper sleeps and starts a new round.
     /// Observable via two providers that each fail one request then recover: the call must
     /// succeed after at least one full round of failures.