From a627b6bcf616f0834dfd62c21a344c5ac3e40dca Mon Sep 17 00:00:00 2001 From: ilitteri Date: Mon, 24 Aug 2026 11:52:21 -0300 Subject: [PATCH 1/4] docs(l2): run the upgrade test with an SP1 GPU prover, not the exec backend MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The upgrade test ran locally with `--backend exec`, which produces no proof and therefore never reads a verification key — and the verification key is precisely what a release bump changes. Batches are committed under `keccak(VERGEN_GIT_SHA)`, baked into the binary at build time, and the OnChainProposer resolves `verificationKeys[commitHash][SP1_VERIFIER_ID]` when verifying, so a new release's batches commit fine and then never verify until its key is registered. Under `exec` the test passes while the real upgrade stalls at the first batch. Move the test to `l2-gpu` with `--backend sp1` and the `-gpu` asset, deploy `VERSION_FROM` with `--sp1 true` so proofs are actually required, and add the step that registers `VERSION_TO`'s key through the Timelock. The same requirement applies to any deployment verifying real proofs, not just to the release test, so it is also documented as a standing step in the L2 upgrade guide — which had no mention of verification keys at all. --- docs/developers/l2/upgrade-test.md | 85 ++++++++++++++++++++++++++---- docs/developers/release-process.md | 19 ++++++- docs/l2/deployment/upgrades.md | 31 +++++++++++ 3 files changed, 123 insertions(+), 12 deletions(-) diff --git a/docs/developers/l2/upgrade-test.md b/docs/developers/l2/upgrade-test.md index a357b760669..cc9f760badc 100644 --- a/docs/developers/l2/upgrade-test.md +++ b/docs/developers/l2/upgrade-test.md @@ -5,6 +5,7 @@ This is a per-release acceptance test. The goal is to verify that: 1. A node running the **previous release** (``) can be cleanly stopped. 2. The L1 contracts can be upgraded following the per-release migration guide. 3. The **new release** (``) sequencer and prover resume operation against the upgraded contracts without re-deploying from scratch. +4. Batches produced by `` and committed by `` are **proved and verified on L1** under the new release's verification key. The commands below are ready to copy-paste. The only thing you need to substitute is the version tags. Every other value (private keys, addresses, ports, fee parameters) is taken from the defaults in `crates/l2/Makefile`; feel free to override them, but the values below give you a working baseline. @@ -15,13 +16,25 @@ Replace these once at the top of your shell session: ```bash export VERSION_FROM= export VERSION_TO= -export ARCH=linux-x86_64 # or linux-aarch64, macos-aarch64 +export ARCH=linux-x86_64 export WORK=$HOME/upgrade-test # workspace mkdir -p "$WORK" ``` +> [!IMPORTANT] +> Run this on the GPU server (`l2-gpu`) with the **SP1** prover, not locally with +> `--backend exec`. The `exec` backend produces no proof, so it never consults a +> verification key — and the verification key is the thing an upgrade changes. +> Batches are committed under `keccak(VERGEN_GIT_SHA)`, baked into the binary at +> build time, and `OnChainProposer.verificationKeys[commitHash][SP1_VERIFIER_ID]` +> is read at verification, so `$VERSION_TO`'s batches cannot be verified until +> its key is registered ([Step 3.6](#36-register-the-version_to-verification-key)). +> Under `exec` that failure is invisible: the test passes and the real upgrade +> stalls at the first batch. + ## Prerequisites +- A CUDA-capable GPU (this is why it runs on `l2-gpu`), and the `ethrex-l2-$ARCH-gpu` release asset - [`rex`](https://github.com/lambdaclass/rex) on `$PATH` (used to call `upgradeToAndCall` and read contract state) - `curl`, `jq` - `git` (to fetch genesis / fixture files for each version) @@ -71,16 +84,26 @@ cd "$WORK" git clone --branch "$VERSION_FROM" --depth 1 https://github.com/lambdaclass/ethrex.git "ethrex-$VERSION_FROM" git clone --branch "$VERSION_TO" --depth 1 https://github.com/lambdaclass/ethrex.git "ethrex-$VERSION_TO" -# Binaries -curl -L "https://github.com/lambdaclass/ethrex/releases/download/$VERSION_FROM/ethrex-l2-$ARCH" -o "ethrex-$VERSION_FROM/ethrex" -curl -L "https://github.com/lambdaclass/ethrex/releases/download/$VERSION_TO/ethrex-l2-$ARCH" -o "ethrex-$VERSION_TO/ethrex" -chmod +x "ethrex-$VERSION_FROM/ethrex" "ethrex-$VERSION_TO/ethrex" +# Binaries. The `-gpu` asset is the one that can prove with SP1 on a GPU. +# Remove before downloading: curl-ing over a binary that is currently running +# reuses the inode and gets the live process killed. +for V in "$VERSION_FROM" "$VERSION_TO"; do + rm -f "ethrex-$V/ethrex" + curl -fL "https://github.com/lambdaclass/ethrex/releases/download/$V/ethrex-l2-$ARCH-gpu" -o "ethrex-$V/ethrex" + chmod +x "ethrex-$V/ethrex" + "ethrex-$V/ethrex" --version +done -"ethrex-$VERSION_FROM/ethrex" --version -"ethrex-$VERSION_TO/ethrex" --version +# Verification keys ship in each release's contracts tarball; both versions' +# keys are needed — VERSION_FROM's to deploy, VERSION_TO's for Step 3.6. +for V in "$VERSION_FROM" "$VERSION_TO"; do + mkdir -p "contracts-$V" + curl -fL "https://github.com/lambdaclass/ethrex/releases/download/$V/ethrex-contracts.tar.gz" \ + | tar xz -C "contracts-$V" +done ``` -> Use the `ethrex-l2-*` asset (the plain `ethrex-*` asset is an L1-only build and does not have the `l2` subcommand). +> Use an `ethrex-l2-*` asset (the plain `ethrex-*` asset is an L1-only build and does not have the `l2` subcommand), and specifically the `-gpu` variant — the others carry no CUDA support, so `--backend sp1` falls back to CPU proving and a batch takes long enough to look like a hang. ### 0.1 Save the `$VERSION_FROM` L2 genesis @@ -125,6 +148,8 @@ COMPILE_CONTRACTS=true ./ethrex l2 deploy \ --on-chain-proposer-owner 0x4417092b70a3e5f10dc504d0947dd256b965fc62 \ --bridge-owner 0x4417092b70a3e5f10dc504d0947dd256b965fc62 \ --bridge-owner-pk 0x941e103320615d394a55708be13e45994c7d93b932b064dbcb2b511fe3254e2e \ + --sp1 true \ + --sp1-vk-path "$WORK/contracts-$VERSION_FROM/ethrex-riscv32im-succinct-zkvm-vk-bn254" \ --deposit-rich \ --private-keys-file-path fixtures/keys/private_keys_l1.txt \ --genesis-l1-path fixtures/genesis/l1.json \ @@ -132,6 +157,12 @@ COMPILE_CONTRACTS=true ./ethrex l2 deploy \ --env-file-path cmd/.env ``` +`--sp1 true` is what makes the OnChainProposer require a real proof, and +`--sp1-vk-path` registers `$VERSION_FROM`'s verification key against +`$VERSION_FROM`'s commit hash. Both matter: without them the deployment accepts +anything and [Step 3.6](#36-register-the-version_to-verification-key) has nothing +to prove. + After this finishes, capture the addresses we'll need later: ```bash @@ -176,7 +207,7 @@ set -a; source cmd/.env; set +a cd "$WORK/ethrex-$VERSION_FROM" ./ethrex l2 prover \ --proof-coordinators tcp://127.0.0.1:3900 \ - --backend exec + --backend sp1 ``` ### 1.5 Confirm the stack is healthy @@ -405,6 +436,40 @@ rex send "$ETHREX_WATCHER_BRIDGE_ADDRESS" 'setL2GasLimit(uint256)' Date: Mon, 24 Aug 2026 11:59:29 -0300 Subject: [PATCH 2/4] docs(l2): add the commands that verify a registered verification key The derivation was checked against a live deployment: the deployer registers `VERSION_FROM`'s key under `keccak(ascii git sha)`, and reading the public `verificationKeys` mapping back returns exactly the key shipped in that release's contracts tarball. Include both the `emergencyExecute` call and the read that confirms it, so the commit hash can be sanity-checked against the key the deployment already holds instead of being taken on faith. --- docs/developers/l2/upgrade-test.md | 21 ++++++++++++++++++++- docs/l2/deployment/upgrades.md | 9 +++++++++ 2 files changed, 29 insertions(+), 1 deletion(-) diff --git a/docs/developers/l2/upgrade-test.md b/docs/developers/l2/upgrade-test.md index cc9f760badc..b231a8a1c80 100644 --- a/docs/developers/l2/upgrade-test.md +++ b/docs/developers/l2/upgrade-test.md @@ -468,7 +468,26 @@ OnChainProposer.upgradeSP1VerificationKey(bytes32 commit_hash, bytes32 new_vk) `upgradeSP1VerificationKey` is `onlyOwner` and that owner is the Timelock, so it routes like any other privileged call: Governance `schedule` + `execute` after the delay, or Security Council `emergencyExecute` to skip it. See -[Timelock](../../l2/fundamentals/timelock.md). +[Timelock](../../l2/fundamentals/timelock.md). In this test both roles are the +account passed as `--on-chain-proposer-owner`, so `emergencyExecute` from that +key needs no delay: + +```bash +cast send "$ETHREX_TIMELOCK_ADDRESS" 'emergencyExecute(address,uint256,bytes)' \ + "$ETHREX_COMMITTER_ON_CHAIN_PROPOSER_ADDRESS" 0 \ + "$(cast calldata 'upgradeSP1VerificationKey(bytes32,bytes32)' "$TO_COMMIT_HASH" "$TO_VK")" \ + --private-key 0x941e103320615d394a55708be13e45994c7d93b932b064dbcb2b511fe3254e2e \ + --rpc-url http://localhost:8545 +``` + +Confirm it landed, and sanity-check the derivation against the key the +`$VERSION_FROM` deploy registered for itself (`1` is the SP1 verifier id): + +```bash +cast call "$ETHREX_COMMITTER_ON_CHAIN_PROPOSER_ADDRESS" \ + 'verificationKeys(bytes32,uint8)(bytes32)' "$TO_COMMIT_HASH" 1 \ + --rpc-url http://localhost:8545 +``` --- diff --git a/docs/l2/deployment/upgrades.md b/docs/l2/deployment/upgrades.md index f974b7b3ec3..9f6a6b7852a 100644 --- a/docs/l2/deployment/upgrades.md +++ b/docs/l2/deployment/upgrades.md @@ -28,6 +28,15 @@ The function is `onlyOwner` and that owner is the Timelock, so route it as Governance `schedule` + `execute`, or Security Council `emergencyExecute` to skip the delay. Use `upgradeRISC0VerificationKey` for a RISC0 deployment. +`verificationKeys` is public, so you can confirm the mapping before and after — +and confirm the deployment's existing key was derived the same way, which is the +cheapest way to check you have the right commit hash (`1` is the SP1 verifier id): + +```bash +cast call "$ON_CHAIN_PROPOSER" 'verificationKeys(bytes32,uint8)(bytes32)' \ + "$(cast keccak "$SHA")" 1 --rpc-url "$L1_RPC" +``` + Keys are per commit hash rather than per version, so the old key stays valid — which is what lets a rollback keep verifying. From 366ad44a1f1ae1bd5865ec130f7e9dac2b2f5e1f Mon Sep 17 00:00:00 2001 From: ilitteri Date: Mon, 24 Aug 2026 12:09:11 -0300 Subject: [PATCH 3/4] docs(l2): correct the symptom of an unregistered verification key Running the upgrade test on a GPU with a real prover showed the previous description was wrong. The key is not checked at verification time but inside `commitBatch`, which reverts with `MissingVerificationKeyForCommit()` (selector `0xf6b9798e`) before storing anything. An upgraded sequencer therefore commits nothing at all, rather than committing batches that later fail to verify: `lastCommittedBatch` and `lastVerifiedBatch` both stand still while the L2 keeps producing blocks, which reads as a stuck committer. Observed on a v24.0.0 -> v25.0.0 upgrade: commits reverted with that selector until the key was registered through `Timelock.emergencyExecute`, after which the committer's own retry recommitted the pending batch and it verified with a real SP1 proof. Nothing is lost by hitting this, which is worth saying so operators do not go looking for damage. --- docs/developers/l2/upgrade-test.md | 31 +++++++++++++++++++----------- docs/developers/release-process.md | 8 ++++---- docs/l2/deployment/upgrades.md | 18 ++++++++++++----- 3 files changed, 37 insertions(+), 20 deletions(-) diff --git a/docs/developers/l2/upgrade-test.md b/docs/developers/l2/upgrade-test.md index b231a8a1c80..1d7ddb219a4 100644 --- a/docs/developers/l2/upgrade-test.md +++ b/docs/developers/l2/upgrade-test.md @@ -26,11 +26,13 @@ mkdir -p "$WORK" > `--backend exec`. The `exec` backend produces no proof, so it never consults a > verification key — and the verification key is the thing an upgrade changes. > Batches are committed under `keccak(VERGEN_GIT_SHA)`, baked into the binary at -> build time, and `OnChainProposer.verificationKeys[commitHash][SP1_VERIFIER_ID]` -> is read at verification, so `$VERSION_TO`'s batches cannot be verified until -> its key is registered ([Step 3.6](#36-register-the-version_to-verification-key)). -> Under `exec` that failure is invisible: the test passes and the real upgrade -> stalls at the first batch. +> build time, and `commitBatch` refuses a commit hash it holds no key for. Until +> `$VERSION_TO`'s key is registered +> ([Step 3.6](#36-register-the-version_to-verification-key)) the upgraded +> sequencer cannot commit **anything**: every `commitBatch` reverts with +> `MissingVerificationKeyForCommit()` (selector `0xf6b9798e`) and the L2 keeps +> producing blocks locally while nothing reaches L1. Under `exec` that failure is +> invisible — the test passes and the real upgrade stops at the first batch. ## Prerequisites @@ -440,12 +442,19 @@ rex send "$ETHREX_WATCHER_BRIDGE_ADDRESS" 'setL2GasLimit(uint256)' Date: Mon, 24 Aug 2026 12:23:36 -0300 Subject: [PATCH 4/4] docs(l2): correct the stale verification-key instructions and use rex for it MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `docs/l2/fundamentals/upgrades.md` has documented this since the mechanism landed, but both of its instructions stopped working in the same release that introduced it: - It says to hash the "reduced" git commit, with a worked example that does not reproduce: keccak of that short string is not the hash shown. Measured against a live deployment, the value is the full 40-char sha as ASCII — hashing v24's full sha yields exactly the key its own deployer registered. - It says to send the call from the OnChainProposer owner account, which the Timelock became in v9.0.0, the same release as per-commit-hash keys. A direct EOA call now reverts with `OwnableUnauthorizedAccount` (`0x118cdaa7`), reproduced here. Point all three docs at `rex l2 register-vk`, which derives the hash, routes through the Timelock and reads the key back, and keep one canonical explanation in the fundamentals page rather than a second copy in the deployment guide — two divergent copies are what let this go unnoticed. --- docs/developers/l2/upgrade-test.md | 38 ++++++++----------- docs/l2/deployment/upgrades.md | 47 ++++++++--------------- docs/l2/fundamentals/upgrades.md | 60 ++++++++++++++++++++---------- 3 files changed, 73 insertions(+), 72 deletions(-) diff --git a/docs/developers/l2/upgrade-test.md b/docs/developers/l2/upgrade-test.md index 1d7ddb219a4..a9472d0aad5 100644 --- a/docs/developers/l2/upgrade-test.md +++ b/docs/developers/l2/upgrade-test.md @@ -468,35 +468,29 @@ TO_COMMIT_HASH=$(cast keccak "$TO_SHA") TO_VK=0x$(tr -d '\n' < ethrex-riscv32im-succinct-zkvm-vk-bn254 | sed 's/^0x//') ``` -The call itself is: - -```solidity -OnChainProposer.upgradeSP1VerificationKey(bytes32 commit_hash, bytes32 new_vk) -``` - -`upgradeSP1VerificationKey` is `onlyOwner` and that owner is the Timelock, so it -routes like any other privileged call: Governance `schedule` + `execute` after -the delay, or Security Council `emergencyExecute` to skip it. See -[Timelock](../../l2/fundamentals/timelock.md). In this test both roles are the -account passed as `--on-chain-proposer-owner`, so `emergencyExecute` from that -key needs no delay: +`rex l2 register-vk` derives the commit hash, routes the call through the +Timelock, and reads the key back afterwards: ```bash -cast send "$ETHREX_TIMELOCK_ADDRESS" 'emergencyExecute(address,uint256,bytes)' \ - "$ETHREX_COMMITTER_ON_CHAIN_PROPOSER_ADDRESS" 0 \ - "$(cast calldata 'upgradeSP1VerificationKey(bytes32,bytes32)' "$TO_COMMIT_HASH" "$TO_VK")" \ +rex l2 register-vk \ + --commit "$TO_SHA" \ + --vk "$TO_VK" \ + --on-chain-proposer "$ETHREX_COMMITTER_ON_CHAIN_PROPOSER_ADDRESS" \ + --timelock "$ETHREX_TIMELOCK_ADDRESS" \ --private-key 0x941e103320615d394a55708be13e45994c7d93b932b064dbcb2b511fe3254e2e \ --rpc-url http://localhost:8545 ``` -Confirm it landed, and sanity-check the derivation against the key the -`$VERSION_FROM` deploy registered for itself (`1` is the SP1 verifier id): +The `--timelock` flag is what makes this work: `upgradeSP1VerificationKey` is +`onlyOwner`, that owner is the Timelock, and the command routes through +`emergencyExecute` — which needs no delay because in this test the Security +Council is the account passed as `--on-chain-proposer-owner`. Sending straight to +the OnChainProposer instead reverts with `OwnableUnauthorizedAccount`. See +[Timelock](../../l2/fundamentals/timelock.md) for the Governance +`schedule` + `execute` route. -```bash -cast call "$ETHREX_COMMITTER_ON_CHAIN_PROPOSER_ADDRESS" \ - 'verificationKeys(bytes32,uint8)(bytes32)' "$TO_COMMIT_HASH" 1 \ - --rpc-url http://localhost:8545 -``` +Run it with `--dry-run` first if you want to see the derived commit hash and the +key currently on chain without sending anything. --- diff --git a/docs/l2/deployment/upgrades.md b/docs/l2/deployment/upgrades.md index 90bbb62b80f..4c0ee7d6102 100644 --- a/docs/l2/deployment/upgrades.md +++ b/docs/l2/deployment/upgrades.md @@ -5,47 +5,32 @@ Applies to any deployment that verifies real proofs (`--sp1 true` at deploy time), whether or not the release changes a contract. -A batch is committed under `keccak(VERGEN_GIT_SHA)` of the binary that committed -it, and `commitBatch` rejects a commit hash it holds no key for. Your deployment -only holds the key of the version that deployed it, so an upgraded sequencer -commits **nothing** until its key is registered — every commit reverts with -`MissingVerificationKeyForCommit()`, selector `0xf6b9798e`, surfacing in the -committer log as: +A batch is committed under the git sha of the binary that committed it, and +`commitBatch` rejects a commit hash the deployment holds no key for. Your +deployment only holds the key of the version that deployed it, so an upgraded +sequencer commits **nothing** until the new key is registered — every commit +reverts with `MissingVerificationKeyForCommit()` (`0xf6b9798e`), surfacing as: ``` Failed to send commitment for batch N ... execution reverted: 0xf6b9798e ``` -The L2 keeps producing blocks the whole time, so it reads as a stuck committer +The L2 keeps producing blocks throughout, so it reads as a stuck committer rather than a missing key. Nothing is lost: register the key and the pending -batches commit and verify on the next attempt. +batch commits on the next attempt. -Register the new release's key before or right after swapping the binary: - -```solidity -OnChainProposer.upgradeSP1VerificationKey(bytes32 commit_hash, bytes32 new_vk) ``` - -- `commit_hash` — `keccak256` of the ASCII git sha the new binary reports in - `ethrex --version` (the segment after `HEAD-`), not the release tag. -- `new_vk` — the `ethrex-riscv32im-succinct-zkvm-vk-bn254` file from the new - release's `ethrex-contracts.tar.gz`, the same hex-text file the deployer takes - via `--sp1-vk-path`. - -The function is `onlyOwner` and that owner is the Timelock, so route it as -Governance `schedule` + `execute`, or Security Council `emergencyExecute` to skip -the delay. Use `upgradeRISC0VerificationKey` for a RISC0 deployment. - -`verificationKeys` is public, so you can confirm the mapping before and after — -and confirm the deployment's existing key was derived the same way, which is the -cheapest way to check you have the right commit hash (`1` is the SP1 verifier id): - -```bash -cast call "$ON_CHAIN_PROPOSER" 'verificationKeys(bytes32,uint8)(bytes32)' \ - "$(cast keccak "$SHA")" 1 --rpc-url "$L1_RPC" +rex l2 register-vk --commit --vk \ + --on-chain-proposer --timelock --private-key ``` -Keys are per commit hash rather than per version, so the old key stays valid — +The sha is the one the new binary prints after `HEAD-` in `ethrex --version`; the +key is the `ethrex-riscv32im-succinct-zkvm-vk-bn254` file from the new release's +`ethrex-contracts.tar.gz`. See +[Registering a new verification key](../fundamentals/upgrades.md#registering-a-new-verification-key) +for what the command does and why the Timelock is in the path. + +Keys are stored per commit hash, not per version, so the old key stays valid — which is what lets a rollback keep verifying. ## From v7 to v8 diff --git a/docs/l2/fundamentals/upgrades.md b/docs/l2/fundamentals/upgrades.md index 39b2ce99ec6..ad3705fd0c1 100644 --- a/docs/l2/fundamentals/upgrades.md +++ b/docs/l2/fundamentals/upgrades.md @@ -6,25 +6,47 @@ Each committed batch stores the git commit hash of the sequencer build that prod ## Registering a new verification key -To allow proofs from a new sequencer/prover build, register its verification key against the commit hash: - -1. Compute the commit hash as the Keccak-256 of the (reduced) git commit. For example, the commit `9219410` produces `b9105485bc4ba523201eaaf76478a47b259fa7399bbed795cf19294861b7fc57`. -2. From the OnChainProposer owner account, send the upgrade transaction. Example (replace addresses and keys with your values): - ``` - rex send \ - "upgradeSP1VerificationKey(bytes32,bytes32)" \ - \ - \ - --private-key - ``` -3. (Optional) Verify the mapping entry: - ``` - rex call \ - "verificationKeys(bytes32,uint8)(bytes32)" \ - \ - - ``` - `1` is the SP1 verifier ID, `2` is RISC0. +This is not optional on an upgrade. `commitBatch` rejects a commit hash it holds +no key for, so an upgraded sequencer commits **nothing** until its key is +registered: every commit reverts with `MissingVerificationKeyForCommit()` +(selector `0xf6b9798e`) while the L2 keeps producing blocks, which reads as a +stuck committer rather than a missing key. Nothing is lost — register the key and +the pending batch is committed on the next attempt. + +Use `rex l2 register-vk`, which derives the commit hash, routes the call, and +reads the mapping back: + +``` +rex l2 register-vk \ + --commit \ + --vk \ + --on-chain-proposer \ + --timelock \ + --private-key +``` + +Add `--prover risc0` for a RISC0 deployment, and `--dry-run` to print the derived +commit hash and the key currently on chain without sending. + +Two details this gets right, and which are easy to get wrong by hand: + +- **The hashed value is the full git sha the binary reports**, as ASCII — the + segment after `HEAD-` in `ethrex --version`. Hashing an abbreviated sha + produces a different key, and the only symptom is that commits keep reverting. +- **`upgradeSP1VerificationKey` is `onlyOwner`, and that owner is the Timelock** + in any deployment made since v9.0.0, so the call must be routed through + `emergencyExecute` (Security Council) or Governance `schedule` + `execute`. + Sending it straight to the OnChainProposer from an EOA reverts with + `OwnableUnauthorizedAccount`. + +To read a mapping entry directly (`1` is the SP1 verifier ID, `2` is RISC0): + +``` +rex l2 call \ + "verificationKeys(bytes32,uint8)(bytes32)" \ + \ + +``` ### Verification key artifacts