feat(consensus): add batched forward epoch sync

[Lchangliang]

Summary

• add a versioned, block-number-anchored forward epoch sync RPC
• fetch, persist, and replay one bounded batch before requesting the next
• authenticate every batch with signed blocks, QCs, ledger infos, parent links, and a pinned manifest-serving peer
• bound server work with a four-request semaphore and return Busy under load
• keep the legacy reverse-sync path as the rolling-upgrade fallback

Compatibility

• appends the new ConsensusMsg variants so existing BCS tags remain stable
• falls back when peers do not support or reject the forward prepare request
• uses block numbers only as cursors; hashes and consensus proofs remain authoritative
• does not add or modify any ConsensusDB schema
• does not require gravity-aptos changes

Validation

• cargo +nightly fmt --all --check
• 4 focused aptos-consensus tests pass
• gravity_node quick-release build passes
• 3-validator plus delayed-VFN test: VFN caught up to the exact live height
• 4-validator plus VFN test: three validators produced blocks first, then the empty fourth validator and VFN caught up from block 330 to 841; the delayed validator subsequently proposed and voted

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d95dd942b9

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Allow replay targets that precede the fetch cursor

When a node has already ordered the epoch-change target but has not committed it, fast_forward_sync_by_epoch_forward uses that ordered block as the fetch anchor and the older commit root as the replay anchor. The server then starts after the ordered block, but this predicate rejects the epoch-ending ledger-info boundary because its target position is before start, even though its certifying suffix QC is in the requested batch. The fetch returns BatchBoundaryNotFound, and the caller propagates that error rather than using the legacy fallback, so a validator/fullnode in this common non-blocking epoch-change state cannot finish the epoch transition. Reproduce by setting the ordered root to the epoch-change target, leaving the commit root behind, and requesting its certifying suffix; permit an authenticated replay target at or before the fetch cursor and validate it against the existing replay anchor/local block instead.

AGENTS.md reference: AGENTS.md:L5-L12

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: efc06ea146

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Require block numbers on non-suffix records

A Byzantine serving peer can mark an ordinary executable block as unnumbered, and this branch accepts it as long as randomness is also omitted, even though only reconfiguration suffixes may legitimately lack execution block numbers. persist_and_replay_forward_epoch_sync_batch then saves and inserts that block without a number, and recover_blocks_checked fails at block_number().ok_or(...); an honest retry repairs the DB mapping but insert_block returns the already-cached PipelinedBlock without copying the corrected number, so synchronization remains stuck until the block store is rebuilt or the node restarts. Validate None against is_reconfiguration_suffix() before persistence; this can be reproduced by changing one non-suffix record's number and randomness to None.

AGENTS.md reference: AGENTS.md:L5-L8

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Authenticate fetched randomness before persisting it

For numbered records, the verifier accepts any record.randomness bytes without checking that they are the certified randomness for the block's epoch and round. The forward path persists those bytes before insertion, try_set_randomness_from_db installs them on the PipelinedBlock, and recovery feeds them into block metadata during execution; a Byzantine selected peer can therefore force an execution/hash mismatch and abort epoch sync. Retrying against an honest peer overwrites the DB value but does not repair the in-memory block because try_set_randomness_from_db refuses to replace randomness already present, leaving the node stuck until restart; the protocol needs an authenticated randomness proof or equivalent validation before persistence.

AGENTS.md reference: AGENTS.md:L5-L8

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 913a733fe5

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Allow batches without a nearby commit boundary

When a valid epoch path has more than batch_size_blocks consensus blocks between replayable ledger infos—such as after a long sequence of certified but noncommitting rounds—select_forward_batch_end caps end at that size, this search finds no boundary, and the server permanently returns BatchBoundaryNotFound. The call path fast_forward_sync_by_epoch_forward → fetch_forward_epoch_sync_batch propagates that fetch error without invoking the legacy fallback, so the node cannot complete the epoch transition even though all records are valid. This can be reproduced by persisting batch_size_blocks + 1 chained certified records between two LedgerInfoSchema entries; support a persistence-only batch with an unchanged replay anchor, or otherwise continue fetching until a replay boundary without violating the server work bound.

AGENTS.md reference: AGENTS.md:L7-L8

Useful? React with 👍 / 👎.

[nekomoto911]

Opened #820: prepare timeout should be longer by default and configurable.

[nekomoto911]

Suggestion: lengthen prepare RPC timeout by default and make it configurable

In try_prepare_forward_epoch_sync, the prepare probe uses a hardcoded Duration::from_millis(1_000) with 2 retries, while fetch uses RPC_TIMEOUT_MSEC (5s):

Duration::from_millis(1_000),  // prepare — magic literal
Duration::from_millis(RPC_TIMEOUT_MSEC),  // fetch — 5s

Why this matters

Cold build_forward_epoch_sync_index is O(epoch): get_all(EpochByBlockNumber) + get_qc_range + parent walk + full LedgerInfoSchema scan. On large epochs / loaded peers, a similar QC scan alone has been observed around ~1.3s. A 1s prepare timeout then often becomes Ok(None) → legacy reverse sync (HashValue::zero()), which re-enters the all-or-nothing zero-hash path this PR is meant to avoid.

Recommendation

1. Raise the default prepare timeout substantially (at least align with fetch / RPC_TIMEOUT_MSEC 5s; prefer ~10s+ pending cold-index measurement).
2. Expose a node-configurable knob (e.g. under consensus local config alongside existing sync settings), not a magic literal.
3. Optionally add metrics: prepare success / timeout / legacy fallback.

Happy to track this as a follow-up in this PR or a small follow-on change.

[nekomoto911]

Question for author: BatchBoundaryNotFound when ordered is past epoch target but commit is not

I'm having trouble understanding whether this is a real stuck window or if I'm missing an invariant. Could you walk through the intended behavior?

What I think the code does

In fetch_forward_epoch_sync, a boundary is only selected when:

boundary.certifying_position >= start &&
boundary.certifying_position < end &&
index.positions[&boundary.target_block_id] >= start &&  // target T must still be in this batch
boundary.target_block_number > request.replay_anchor_block_number

Client fetch/replay anchors come from ordered_root() / commit_root(), and on fetch error we bail! (no legacy fallback — only prepare Ok(false) falls back).

Scenario I can't map cleanly

Suppose locally:

• ordered_root is already past the epoch-ending target T (e.g. into the certifying suffix S)
• commit_root is still before T (so we still need the epoch-ending LI + certifying QC to advance commit / finish the epoch)

Then start is after T, so target_position >= start fails → server returns BatchBoundaryNotFound → client hard-fails the forward attempt and retries into the same local root relationship.

What I'm unsure about

1. Is this state reachable in practice (e.g. crash between save_tree and writing the epoch-ending LI, non-blocking epoch-change with execution lag, restart resume from divergent ordered/commit roots)? Or does some invariant guarantee we never Fetch with ordered > T while commit < T?
2. If it is reachable, how is the client supposed to obtain the epoch-ending LI / certifying QC in that window? The early-exit path only triggers when commit >= target and ordered >= terminal.
3. If Fetch returns BatchBoundaryNotFound here, is hard-fail (no legacy fallback) intentional?

A short intended-state diagram or “this can’t happen because …” note would help a lot — thanks!

[Lchangliang]

@nekomoto911 Thanks — this is a real reachable window; we should not assume ordered == commit.

Addressed in 1dad95a6:

• Removed the requirement that the epoch target T and its certifying proof must be in the same batch.
• Removed the special “final batch” / replay-anchor semantics. Every fetch is now an ordinary bounded page from the current block-number + block-ID cursor.
• A page may carry zero or more LedgerInfos. The client persists and processes each page, then continues fetching. If T has already been fetched but the epoch-ending LI is not available yet, it keeps fetching the certifying suffix until the LI can be verified and applied.
• Completion is based only on the epoch-change target T being durably committed.
• If Prepare finds that the actual commit-decision target is already on the local ordered path, the client persists the LI and re-sends the commit proof; if it still cannot commit within 5s, it falls back to the legacy path.
• BatchBoundaryNotFound is no longer returned merely because a bounded page has no replayable boundary; such pages are valid persistence-only pages.
• The Prepare RPC timeout is now 5s per attempt.

This keeps ordered > T > commit recoverable without changing any ConsensusDB schema.

[nekomoto911]

[High] Batch paging loop issues one more Fetch after consuming the last batch instead of waiting for the commit to land — the whole forward path then fails with Err and never falls back to legacy

Problem

In fast_forward_sync_by_epoch_forward, after a batch is persisted and handed to execution, the loop checks forward_epoch_sync_target_committed() and, if false, immediately sends the next Fetch (

gravity-sdk/aptos-core/consensus/src/block_storage/sync_manager/forward_epoch_sync.rs

Lines 574 to 608 in 1dad95a

	loop {
	let request = ForwardEpochSyncFetchRequest {
	epoch,
	manifest_id: manifest.manifest_id,
	anchor_block_number: fetch_anchor_block_number,
	anchor_block_id: fetch_anchor_block_id,
	batch_size_blocks,
	};
	let batch =
	retriever.fetch_forward_epoch_sync_batch(request.clone(), serving_peer).await?;
	BLOCKS_FETCHED_FROM_NETWORK_WHILE_FAST_FORWARD_SYNC.inc_by(batch.records.len() as u64);
	self.persist_and_process_forward_epoch_sync_batch(&batch).await?;
	let committed = self.commit_root();
	let committed_number = committed
	.block()
	.block_number()
	.ok_or_else(|| anyhow!("Commit root has no block number after forward replay"))?;
	fetch_anchor_block_number = batch.next_anchor_block_number;
	fetch_anchor_block_id = batch.next_anchor_block_id;
	info!(
	epoch = epoch,
	fetch_anchor_block_number = fetch_anchor_block_number,
	committed_block_number = committed_number,
	ledger_info_count = batch.ledger_infos.len(),
	"Forward epoch sync batch persisted and processed"
	);
	if self.forward_epoch_sync_target_committed(&manifest)? {
	self.send_committed_epoch_change(retriever, &manifest.target_ledger_info).await?;
	return Ok(true);
	}
	}

). Once the final batch has been consumed, the anchor already sits at the manifest tail, so that extra Fetch is guaranteed to fail:

1. Server side: validate_forward_anchor maps a tail anchor to start == entries.len(), so fetch_forward_epoch_sync returns BatchBoundaryNotFound (

   gravity-sdk/aptos-core/consensus/src/block_storage/sync_manager/forward_epoch_sync.rs

   Lines 381 to 383 in 1dad95a

   	if start >= index.entries.len() {
   	return ForwardEpochSyncResponseV1::Error(ForwardEpochSyncError::BatchBoundaryNotFound);
   	}

   ).
2. Client side: fetch_forward_epoch_sync_batch bail!s on any Error response (

   gravity-sdk/aptos-core/consensus/src/block_storage/sync_manager/forward_epoch_sync.rs

   Lines 887 to 895 in 1dad95a

   	let batch = match response {
   	ForwardEpochSyncResponseV1::Batch(batch) => batch,
   	ForwardEpochSyncResponseV1::Error(error) => {
   	bail!("Forward epoch sync fetch rejected: {error:?}")
   	}
   	ForwardEpochSyncResponseV1::Prepared(_) => {
   	bail!("Forward epoch sync fetch returned a manifest")
   	}
   	};

   ).
3. The Err propagates to fast_forward_sync_by_epoch, which only falls back to legacy on Ok(false) — an Err is a hard failure (

   gravity-sdk/aptos-core/consensus/src/block_storage/sync_manager/forward_epoch_sync.rs

   Lines 506 to 516 in 1dad95a

   	match self
   	.fast_forward_sync_by_epoch_forward(&mut retriever, epoch, batch_size_blocks)
   	.await
   	{
   	Ok(true) => Ok(()),
   	Ok(false) => {
   	info!(epoch = epoch, "Falling back to legacy reverse epoch sync");
   	self.fast_forward_sync_by_epoch_legacy(retriever, epoch).await
   	}
   	Err(error) => Err(error),
   	}

   ).

So the sync fails even though every block, QC, and ledger info of the epoch has already been fetched, verified, and persisted.

Why "target not yet committed after the last batch" is the common case, not a race

Commit-root advancement is asynchronous with respect to batch processing: persist_and_process_forward_epoch_sync_batch → append_blocks_for_sync_checked → recover_blocks_checked → send_for_execution(recovery=true), and that path has several early returns that leave commit_root unadvanced. The most notable one is the defer when the execution layer has not yet written the ledger hash for the batch tail (

gravity-sdk/aptos-core/consensus/src/block_storage/block_store.rs

Lines 621 to 634 in 1dad95a

	if self
	.storage
	.consensus_db()
	.ledger_db
	.metadata_db()
	.get_block_hash(last_block_number)
	.is_none()
	{
	info!(
	"send_for_execution(recovery): defer batch because last block has no ledger hash: block_number={}",
	last_block_number,
	);
	return Ok(());
	}

). Whenever execution lags consensus fetching — the normal situation for a node that is catching up — the loop reaches the manifest tail with forward_epoch_sync_target_committed() == false and fires the doomed Fetch.

Notably, the sibling recovery branch already handles this exact situation correctly: forward_epoch_sync_commit_proof_target_is_ordered polls with wait_for_forward_epoch_sync_target and returns Ok(false) (legacy fallback) on timeout (

gravity-sdk/aptos-core/consensus/src/block_storage/sync_manager/forward_epoch_sync.rs

Lines 557 to 572 in 1dad95a

	if self.forward_epoch_sync_commit_proof_target_is_ordered(&manifest) {
	self.persist_forward_epoch_sync_ledger_infos(std::slice::from_ref(
	&manifest.target_ledger_info,
	))?;
	retriever.network.send_commit_proof(manifest.target_ledger_info.clone()).await;
	if self.wait_for_forward_epoch_sync_target(&manifest).await? {
	self.send_committed_epoch_change(retriever, &manifest.target_ledger_info).await?;
	return Ok(true);
	}
	info!(
	epoch = epoch,
	target_block_number = manifest.target_block_number,
	"Local ordered epoch target did not commit in time; use legacy fallback"
	);
	return Ok(false);
	}

). The paging loop is missing the same treatment.

Impact

Not a permanent stall, but a reliability regression on the new default path:

• The failure surfaces as an Err from fast_forward_sync_by_epoch; RoundManager only logs it and leaves wait_change_epoch_flag unset, so the sync is retried on the next EpochChange event (

  gravity-sdk/aptos-core/consensus/src/round_manager.rs

  Lines 1722 to 1741 in 1dad95a

  	VerifiedEvent::EpochChange(epoch) => {
  	if !self.wait_change_epoch_flag {
  	let batch_size_blocks = self.local_config
  	.max_blocks_per_sending_request(
  	self.onchain_config.quorum_store_enabled(),
  	);
  	if let Err(e) = self.block_store.fast_forward_sync_by_epoch(
  	self.create_block_retriever(peer_id),
  	epoch,
  	batch_size_blocks,
  	).await {
  	Err(e)
  	} else {
  	self.wait_change_epoch_flag = true;
  	Ok(())
  	}
  	} else {
  	Ok(())
  	}
  	}

  ).
• Because neither ordered_root nor commit_root advanced past the persisted-but-uncommitted tail, each retry re-runs Prepare and re-fetches batches that were already persisted. While execution is still catching up, every retry ends the same way: full batch traffic + BatchBoundaryNotFound + error logs, repeatedly, until execution finally commits the target.

Suggested fix

The manifest intentionally carries no terminal/tail metadata (a certifying suffix past target_block_number may span any number of unnumbered blocks), so the client cannot predict when its anchor reaches the tail — a BatchBoundaryNotFound returned for an anchor that passed validation is effectively the server's authoritative "no more pages" signal. A fix that needs no protocol change: when a Fetch fails with BatchBoundaryNotFound and the locally persisted chain already covers manifest.target_block_number, treat it as end-of-data instead of a hard error — poll wait_for_forward_epoch_sync_target, and on timeout return Ok(false) so the caller falls back to the legacy reverse path, mirroring the commit_proof_target_is_ordered branch.