docs(l2): run the release upgrade test with a real prover, and fix the verification-key instructions - #7203
docs(l2): run the release upgrade test with a real prover, and fix the verification-key instructions#7203ilitteri wants to merge 4 commits into
Conversation
…ackend The upgrade test ran locally with `--backend exec`, which produces no proof and therefore never reads a verification key — and the verification key is precisely what a release bump changes. Batches are committed under `keccak(VERGEN_GIT_SHA)`, baked into the binary at build time, and the OnChainProposer resolves `verificationKeys[commitHash][SP1_VERIFIER_ID]` when verifying, so a new release's batches commit fine and then never verify until its key is registered. Under `exec` the test passes while the real upgrade stalls at the first batch. Move the test to `l2-gpu` with `--backend sp1` and the `-gpu` asset, deploy `VERSION_FROM` with `--sp1 true` so proofs are actually required, and add the step that registers `VERSION_TO`'s key through the Timelock. The same requirement applies to any deployment verifying real proofs, not just to the release test, so it is also documented as a standing step in the L2 upgrade guide — which had no mention of verification keys at all.
The derivation was checked against a live deployment: the deployer registers `VERSION_FROM`'s key under `keccak(ascii git sha)`, and reading the public `verificationKeys` mapping back returns exactly the key shipped in that release's contracts tarball. Include both the `emergencyExecute` call and the read that confirms it, so the commit hash can be sanity-checked against the key the deployment already holds instead of being taken on faith.
Running the upgrade test on a GPU with a real prover showed the previous description was wrong. The key is not checked at verification time but inside `commitBatch`, which reverts with `MissingVerificationKeyForCommit()` (selector `0xf6b9798e`) before storing anything. An upgraded sequencer therefore commits nothing at all, rather than committing batches that later fail to verify: `lastCommittedBatch` and `lastVerifiedBatch` both stand still while the L2 keeps producing blocks, which reads as a stuck committer. Observed on a v24.0.0 -> v25.0.0 upgrade: commits reverted with that selector until the key was registered through `Timelock.emergencyExecute`, after which the committer's own retry recommitted the pending batch and it verified with a real SP1 proof. Nothing is lost by hitting this, which is worth saying so operators do not go looking for damage.
… for it `docs/l2/fundamentals/upgrades.md` has documented this since the mechanism landed, but both of its instructions stopped working in the same release that introduced it: - It says to hash the "reduced" git commit, with a worked example that does not reproduce: keccak of that short string is not the hash shown. Measured against a live deployment, the value is the full 40-char sha as ASCII — hashing v24's full sha yields exactly the key its own deployer registered. - It says to send the call from the OnChainProposer owner account, which the Timelock became in v9.0.0, the same release as per-commit-hash keys. A direct EOA call now reverts with `OwnableUnauthorizedAccount` (`0x118cdaa7`), reproduced here. Point all three docs at `rex l2 register-vk`, which derives the hash, routes through the Timelock and reads the key back, and keep one canonical explanation in the fundamentals page rather than a second copy in the deployment guide — two divergent copies are what let this go unnoticed.
|
🤖 Codex Code Review
Aside from those two doc regressions, the contract-side rationale looks correct: Automated review by OpenAI Codex · gpt-5.4 · custom prompt |
… key (#249) **Motivation** Every ethrex upgrade of an L2 that verifies real proofs needs the new build's verification key registered against the commit hash it commits batches under. This has been true since ethrex v9.0.0, when verification keys became per-commit-hash and the Timelock became the OnChainProposer's owner. It is not optional. `commitBatch` rejects a commit hash the deployment holds no key for, so an upgraded sequencer commits **nothing** until the key is registered — every commit reverts with `MissingVerificationKeyForCommit()` (`0xf6b9798e`) while the L2 keeps producing blocks. The symptom looks like a stuck committer, not a missing key. Doing it by hand is easy to get wrong in two ways, and both failures look the same from the outside: - **The hashed value is the full git sha the binary reports, as ASCII** — the segment after `HEAD-` in `ethrex --version`. Hashing an abbreviated sha silently produces a different key, and the only symptom is that commits keep reverting. The ethrex docs carried a worked example using a shortened sha that does not reproduce; that is fixed in lambdaclass/ethrex#7203. - **`upgradeSP1VerificationKey` is `onlyOwner`, and that owner is the Timelock**, so the call has to be routed through `emergencyExecute` (Security Council) or Governance `schedule` + `execute`. Sending it straight to the OnChainProposer from an EOA reverts with `OwnableUnauthorizedAccount` (`0x118cdaa7`), which surfaces as an opaque abi-decode failure. **Description** Adds `rex l2 register-vk` (alias `vk`) and the SDK functions behind it. ``` rex l2 register-vk --commit <GIT_SHA> --vk <VERIFICATION_KEY> \ --on-chain-proposer <OCP> --timelock <TIMELOCK> --private-key <SECURITY_COUNCIL_PK> ``` - `--commit` takes the git sha and hashes it, or accepts an already-hashed 32-byte value, so the caller never has to know which form the contract wants. - `--timelock` routes through `emergencyExecute`. Omit it only when an EOA still owns the contract. If the direct call is rejected as not-owner, the revert is translated into a message saying to pass the flag rather than an abi-decode error. - `--prover risc0` targets `upgradeRISC0VerificationKey`; the default is SP1. - `--dry-run` prints the derived commit hash and the key currently on chain without sending. - The key is read back after the transaction is mined, because a call routed to the wrong owner can be mined without taking effect. A re-run when the key already matches is a no-op. New: `sdk/src/l2/verification_key.rs` (`commit_hash_from_git_sha`, `get_verification_key`, `register_verification_key`, `Prover`), plus the four contract signatures in `sdk/src/l2/constants.rs`. **How to test** Against a dev L2 deployed with `--sp1 true` (this is the flow lambdaclass/ethrex#7203 documents as Step 3.6 of the release upgrade test): ```bash # what would happen, nothing sent rex l2 register-vk --commit "$(./ethrex --version | sed 's|.*HEAD-||; s|/.*||')" \ --vk 0x<key from the release's ethrex-contracts.tar.gz> \ --on-chain-proposer "$OCP" --timelock "$TIMELOCK" --private-key "$PK" --dry-run # register, then re-run: the second call reports nothing to do rex l2 register-vk --commit ... --vk ... --on-chain-proposer "$OCP" --timelock "$TIMELOCK" --private-key "$PK" ``` Verified end to end against a live v24.0.0 → v25.0.0 upgrade: the dry run reproduced a hand-computed commit hash, the registration went through the Timelock and read back correctly, a repeat run reported nothing to do, and omitting `--timelock` produced the translated error. `cargo check --all-targets`, `clippy` and `fmt` are clean.
**Motivation** Merges the `release/v25.0.0` branch back into `main` now that [v25.0.0](https://github.com/lambdaclass/ethrex/releases/tag/v25.0.0) is published, per the last step of the [release process](https://github.com/lambdaclass/ethrex/blob/main/docs/developers/release-process.md). The second commit matters on its own: **`main`'s release workflow is currently broken**, and this is what fixes it. Any tag cut from `main` before this merges would build every artifact and then fail to publish a release. **Description** Two commits, no conflicts with `main`: - `chore(l1,l2): bump version to 25.0.0` — the 25 internal version pins across 9 `Cargo.toml` files, the regenerated lockfiles, and both `--builder.extra-data` defaults in `docs/CLI.md`. Includes the four `crates/guest-program/stateless-validator/**/Cargo.lock` files, which are each their own workspace and reach the ethrex crates through path dependencies, so they pin the workspace version and go stale on a bump. `make update-cargo-lock` does not cover them while `check-cargo-lock` does, so they were refreshed by hand — worth closing that asymmetry in a follow-up, since every future bump hits it. - `ci(l1,l2): stop the release asset download from matching docker build records` — `finalize-release` downloaded artifacts with `pattern: "*ethrex*"`, which also matches the build records the docker jobs upload as `<org>~ethrex~<id>.dockerbuild`. Those are not zip archives, so `download-artifact` exhausted its retries trying to extract them and the job died before publishing anything. This is a regression from lambdaclass#7115, which widened the pattern to pick up the new `stateless-validator-ethrex-*` artifacts; the previous anchored pattern never matched the docker records. v25.0.0-rc.1 hit it — all 13 build jobs green, no release created. The dry-run rehearsal could not have caught it either: that job only waits on the guest builds, so it runs before any docker record exists. Fixed with two anchored patterns, `ethrex*` and `stateless-validator-ethrex-*`, verified against the run's real 17 artifacts to select exactly the 12 release assets. rc.2 published all 22 assets correctly. **Validation** v25.0.0 went through the full release checklist: three mainnet canaries paired with Prysm/Teku/Grandine, an eth-docker mainnet node, 13 consecutive successful multisync runs across hoodi/sepolia/mainnet, the SP1 GPU integration suite, and the L2 upgrade test — the last of which ran with a real SP1 GPU prover for the first time (see lambdaclass#7203). **Checklist** - [ ] Updated `STORE_SCHEMA_VERSION` (crates/storage/lib.rs) if the PR includes breaking changes to the `Store` requiring a re-sync.
1 participant
Motivation
The release checklist's L2 upgrade test ran locally with
--backend exec. That backend produces no proof, so it never consults a verification key — and the verification key is precisely what an ethrex upgrade changes. The test could not observe the main way an upgrade breaks.Running it with a real SP1 prover for the first time (v24.0.0 → v25.0.0) surfaced a requirement that has applied since v9.0.0 and whose written procedure has been unusable for just as long. Two changes landed together in that release: per-commit-hash verification keys, and the Timelock becoming the OnChainProposer's owner.
Each batch is committed under
keccakof the git sha of the binary that committed it, andcommitBatchrejects a commit hash it holds no key for. So after upgrading, a deployment that verifies real proofs commits nothing until the new build's key is registered — every commit reverts withMissingVerificationKeyForCommit()(0xf6b9798e) while the L2 keeps producing blocks, which reads as a stuck committer rather than a missing key.docs/l2/fundamentals/upgrades.mdhas documented the mechanism since it landed, but both of its instructions stopped working in that same release:9219410→b9105485…4732af28…, so the example does not reproduce. The value is the full 40-char sha as ASCII — hashing v24.0.0's full sha yields exactly the key its own deployer registeredupgradeSP1VerificationKeyfrom the OnChainProposer owner accountOwnableUnauthorizedAccount(0x118cdaa7); the Timelock has owned the contract since v9.0.0, so the call has to be routed through itDescription
Move the release upgrade test onto the GPU server with
--backend sp1, and document the step it was missing.docs/developers/release-process.md— the checklist item now namesl2-gpuand SP1, explains whyexechides this failure class, and notes both L2 checks share ports and datadirs so they run sequentially. Keeps the fresh-deploy SP1 test as well: a release can deploy cleanly and still fail to upgrade, and the reverse.docs/developers/l2/upgrade-test.md— deployVERSION_FROMwith--sp1 trueso proofs are actually required, use the-gpuasset (the others have no CUDA, so--backend sp1silently falls back to CPU and a batch looks like a hang), both provers on--backend sp1, a fourth stated goal covering on-L1 verification, and a new Step 3.6 that registersVERSION_TO's key.docs/l2/deployment/upgrades.md— a standing per-release entry, since this applies to any real-proof deployment and not only to the release test.docs/l2/fundamentals/upgrades.md— corrected. This page stays the single canonical explanation and the other two link to it; two divergent copies are what let the stale version go unnoticed.All four now point at
rex l2 register-vk(lambdaclass/rex#249), which derives the commit hash, routes through the Timelock and reads the key back, rather than a hand-assembledcast/rex sendcall.Validation
Every claim here was measured on a v24.0.0 → v25.0.0 upgrade on
l2-gpu, not derived from reading the contracts:keccak(full ascii sha)reproduces the key theVERSION_FROMdeployer registered for itself — confirming the derivation--sp1 true: batches verified with real GPU proofs0xf6b9798eTimelock.emergencyExecute: the committer's own retry recommitted the pending batch, and the batch carrying the v24.0.0-produced blocks verified under v25.0.0's key1 passed; 0 failedChecklist
STORE_SCHEMA_VERSION(crates/storage/lib.rs) if the PR includes breaking changes to theStorerequiring a re-sync.