From 8a4c2cbbc80e36c53de5759b95e3597e5a34b55f Mon Sep 17 00:00:00 2001 From: ilitteri Date: Wed, 13 May 2026 16:19:14 -0300 Subject: [PATCH 01/25] Add Lean Ethereum consensus pipeline (ethlambda + ream/zeam stubs) Lean Ethereum is a redesign of Ethereum consensus with no EL pairing, no Engine API, no JWT, and post-quantum (XMSS / hash-sig) validator signatures. Lean clients are standalone consensus nodes talking only to each other over libp2p QUIC. Trying to express Lean clients as participants[].cl_type with el_type: none would force is_lean() branches throughout the EL/CL pipeline, validator-keystore generator, MEV-boost flow, snooper, etc. This change adds a parallel top-level lean_participants: pipeline so the existing EL/CL flow is untouched and Lean concerns are isolated under src/lean/ and src/prelaunch_data_generator/lean_genesis/. The Lean pipeline runs in three phases: 1. Allocate per-node libp2p P2P keys via openssl. 2. Stand up placeholder services so Kurtosis assigns IPs, then run hash-sig-cli (XMSS keypairs) and eth-beacon-genesis leanchain (config.yaml + validators.yaml + nodes.yaml + genesis.{ssz,json}) against the live IPs. Post-process injects GENESIS_VALIDATORS into config.yaml and renders annotated_validators.yaml mapping node names to validator indices and attester/proposer privkey file basenames. 3. Re-add each placeholder with force_update=True so the genesis + hash-sig artifacts are mounted and the real client binary runs. Kurtosis preserves the IP because the service name and ports don't change, keeping the ENRs we just embedded valid. ethlambda is fully wired as the first concrete client. ream and zeam ship with stub launchers translating to their CLI surface from blockblaz/lean-quickstart client-cmds/*.sh. docs/lean-consensus.md covers the architecture and docs/lean-adding-a-new-client.md is the contract for adding a new Lean client (5 touch points). V1 still requires at least one EL/CL participants[] entry because several downstream consumers (tx-fuzz target, dora, etc.) assume all_el_contexts[0] exists; lean-only mode is a follow-up. --- docs/lean-adding-a-new-client.md | 296 ++++++++++++++ docs/lean-consensus.md | 224 +++++++++++ main.star | 13 + network_params.yaml | 30 ++ src/lean/ethlambda/ethlambda_launcher.star | 161 ++++++++ src/lean/lean_context.star | 30 ++ src/lean/lean_launcher.star | 170 ++++++++ src/lean/lean_shared.star | 60 +++ src/lean/ream/ream_launcher.star | 143 +++++++ src/lean/zeam/zeam_launcher.star | 129 ++++++ src/package_io/constants.star | 42 ++ src/package_io/input_parser.star | 114 ++++++ .../lean_genesis/lean_genesis_generator.star | 378 ++++++++++++++++++ .../lean_genesis/p2p_keys_generator.star | 61 +++ 14 files changed, 1851 insertions(+) create mode 100644 docs/lean-adding-a-new-client.md create mode 100644 docs/lean-consensus.md create mode 100644 src/lean/ethlambda/ethlambda_launcher.star create mode 100644 src/lean/lean_context.star create mode 100644 src/lean/lean_launcher.star create mode 100644 src/lean/lean_shared.star create mode 100644 src/lean/ream/ream_launcher.star create mode 100644 src/lean/zeam/zeam_launcher.star create mode 100644 src/prelaunch_data_generator/lean_genesis/lean_genesis_generator.star create mode 100644 src/prelaunch_data_generator/lean_genesis/p2p_keys_generator.star diff --git a/docs/lean-adding-a-new-client.md b/docs/lean-adding-a-new-client.md new file mode 100644 index 000000000..4c84d6c9c --- /dev/null +++ b/docs/lean-adding-a-new-client.md @@ -0,0 +1,296 @@ +# Adding a new Lean consensus client to ethereum-package + +This guide walks through every change you need to integrate a new Lean +consensus client (ream, zeam, qlean, lantern, grandine, lighthouse-lean, +gean, peam, nlean, or a new one). Read +[`lean-consensus.md`](./lean-consensus.md) first for the architecture. + +The integration has **5 touch points**. The Lean genesis pipeline, +hash-sig key generation, P2P key allocation, and per-node IP allocation +are all generic and require no changes. + +--- + +## Naming convention + +Every Lean node is named `_`: + +- `ethlambda_0` — first node for ethlambda +- `ethlambda_1`, `ethlambda_2` — additional nodes when `count > 1` + +The Kurtosis service name is `lean-_` (e.g. +`lean-ethlambda_0`). The prefix before the first underscore is the **client +type** and matches `LEAN_TYPE` in `src/package_io/constants.star`. + +--- + +## Touch point 1 — Register the client type + +Add your client to `LEAN_TYPE` in `src/package_io/constants.star`: + +```python +LEAN_TYPE = struct( + ethlambda="ethlambda", + ream="ream", + zeam="zeam", + # ... + myclient="myclient", +) +``` + +Then add a default image to `DEFAULT_LEAN_IMAGES` in +`src/package_io/input_parser.star`: + +```python +DEFAULT_LEAN_IMAGES = { + constants.LEAN_TYPE.ethlambda: "ghcr.io/lambdaclass/ethlambda:devnet4", + # ... + constants.LEAN_TYPE.myclient: "ghcr.io/yourorg/myclient:devnet4", +} +``` + +> The default image must run as a non-interactive container with the client +> binary as its `ENTRYPOINT`. The Lean launcher overrides `entrypoint` to +> `/bin/sh -c` so it can run a `tail -f` placeholder, but during normal +> operation the original entrypoint is replaced by a constructed command +> line. + +--- + +## Touch point 2 — `src/lean/myclient/myclient_launcher.star` + +Copy `src/lean/ethlambda/ethlambda_launcher.star` and adapt the CLI surface. +You must export exactly two functions: `initialize` and `start`. + +```python +""" +myclient launcher. + +Translates the Lean pipeline's per-node record into myclient's CLI surface. +See [client docs/CLI reference] for the source of truth. +""" + +constants = import_module("../../package_io/constants.star") +lean_shared = import_module("../lean_shared.star") +lean_context = import_module("../lean_context.star") + +ENTRYPOINT = "/usr/local/bin/myclient" +GENESIS_MOUNT = constants.LEAN_GENESIS_MOUNTPOINT_ON_CLIENTS +HASH_SIG_MOUNT = GENESIS_MOUNT + "/hash-sig-keys" +DATA_DIR = "/data" +NODE_KEY_MOUNT = constants.LEAN_NODE_KEY_MOUNTPOINT_ON_CLIENTS + + +def initialize(plan, node, p2p_keys_artifact): + # Phase 1: stand the placeholder service up so Kurtosis assigns an IP. + return plan.add_service(node["service_name"], ServiceConfig( + image = node["image"], + entrypoint = ["/bin/sh", "-c"], + cmd = lean_shared.lean_tail_logs_cmd(node["service_name"])[2:], + ports = lean_shared.lean_port_specs(), + files = {NODE_KEY_MOUNT: p2p_keys_artifact}, + env_vars = node["extra_env_vars"], + labels = node["extra_labels"], + # ... (cpu/mem/node_selectors/tolerations - copy from ethlambda) + )) + + +def start(plan, node, service, genesis_artifact, hash_sig_artifact): + # Phase 3: re-add the service with full mounts and the real command. + cmd_parts = [ + ENTRYPOINT, + # Required - your CLI must accept these (or equivalent): + "--genesis", "{0}/config.yaml".format(GENESIS_MOUNT), + "--validators","{0}/annotated_validators.yaml".format(GENESIS_MOUNT), + "--bootnodes", "{0}/nodes.yaml".format(GENESIS_MOUNT), + "--data-dir", DATA_DIR, + "--node-id", node["node_name"], + "--node-key", "{0}/{1}.key".format(NODE_KEY_MOUNT, node["node_name"]), + # Ports - your CLI must accept distinct flags for QUIC, REST, metrics: + "--gossipsub-port", str(constants.LEAN_QUIC_PORT_NUM), + "--api-port", str(constants.LEAN_API_PORT_NUM), + "--metrics-port", str(constants.LEAN_METRICS_PORT_NUM), + "--http-address", "0.0.0.0", + ] + if node["is_aggregator"]: + cmd_parts.append("--is-aggregator") + for extra in node["extra_params"]: + cmd_parts.append(extra) + + log_file = lean_shared.lean_log_file_path(service.name) + full_cmd = " ".join(cmd_parts) + + new_service = plan.add_service( + name = service.name, + force_update = True, + config = ServiceConfig( + image = node["image"], + entrypoint = ["/bin/sh", "-c"], + cmd = ["{0} 2>&1 | tee -a {1}".format(full_cmd, log_file)], + ports = lean_shared.lean_port_specs(), + files = { + NODE_KEY_MOUNT: node["_p2p_keys_artifact"], + GENESIS_MOUNT: genesis_artifact, + HASH_SIG_MOUNT: hash_sig_artifact, + }, + # ... (env_vars/labels/cpu/mem/node_selectors/tolerations) + ), + ) + + return lean_context.new_lean_context( + client_name = constants.LEAN_TYPE.myclient, + service_name = new_service.name, + ip_address = new_service.ip_address, + quic_port = constants.LEAN_QUIC_PORT_NUM, + api_port = constants.LEAN_API_PORT_NUM, + metrics_port = constants.LEAN_METRICS_PORT_NUM, + api_url = "http://{0}:{1}".format(new_service.ip_address, constants.LEAN_API_PORT_NUM), + metrics_url = "http://{0}:{1}/metrics".format(new_service.ip_address, constants.LEAN_METRICS_PORT_NUM), + metrics_info = { + "name": new_service.name, + "url": "http://{0}:{1}/metrics".format(new_service.ip_address, constants.LEAN_METRICS_PORT_NUM), + "path": "/metrics", + "config": node["prometheus_config"], + }, + ) +``` + +--- + +## Touch point 3 — Dispatch in `src/lean/lean_launcher.star` + +Add an `import_module` for your launcher and route to it in `_launcher_for`: + +```python +myclient_launcher = import_module("./myclient/myclient_launcher.star") + +def _launcher_for(lean_type): + if lean_type == constants.LEAN_TYPE.ethlambda: + return ethlambda_launcher + elif lean_type == constants.LEAN_TYPE.ream: + return ream_launcher + elif lean_type == constants.LEAN_TYPE.zeam: + return zeam_launcher + elif lean_type == constants.LEAN_TYPE.myclient: + return myclient_launcher + fail(...) +``` + +--- + +## Touch point 4 — README + `network_params.yaml` example + +Add a line to the `lean_participants:` example in `network_params.yaml`: + +```yaml +lean_participants: + - lean_type: myclient + count: 1 + validator_count: 1 +``` + +--- + +## Touch point 5 — Docs + +Add your client to the list at the top of [`lean-consensus.md`](./lean-consensus.md). + +--- + +## Required CLI flags your client must support + +Your Lean client binary must accept at least the following flags (or +equivalents you can pass via `lean_extra_params`). Flag names vary across +clients; the names below mirror ethlambda — adapt to your client's CLI by +adjusting the per-client launcher. + +| Concept | Where it comes from | +|----------------------------|-----------------------------------------------------------| +| `--node-id ` | Identifies the node in logs and validator-config lookups | +| `--node-key ` | 32-byte hex libp2p secret (`.key`) | +| `--genesis ` | Path to `config.yaml` | +| `--validators ` | Path to `annotated_validators.yaml` | +| `--bootnodes ` | Path to `nodes.yaml` | +| `--validator-config `| Path to `validator-config.yaml` (per-node settings) | +| `--hash-sig-keys-dir `| XMSS key directory | +| `--data-dir ` | Persistent RocksDB / LMDB | +| `--gossipsub-port ` | UDP QUIC port (= `LEAN_QUIC_PORT_NUM = 9000`) | +| `--api-port ` | REST API port (= `LEAN_API_PORT_NUM = 5052`) | +| `--metrics-port ` | Prometheus metrics port (= `LEAN_METRICS_PORT_NUM = 5054`) | +| `--http-address 0.0.0.0` | Bind address for REST + metrics | +| `--is-aggregator` | Enable aggregator mode (required for finality) | + +### Required HTTP endpoints + +| Path | Purpose | +|--------------------------------|--------------------------------------------------------| +| `GET /lean/v0/health` | Liveness check (return 200 when healthy) | +| `GET /metrics` (metrics port) | Prometheus exposition (`lean_*` metric names) | + +The full Lean REST API is documented at +[ReamLabs/leanSpecs](https://github.com/ReamLabs/leanSpecs); only health ++ metrics are required for the package itself, but other endpoints +(checkpoint sync, fork choice, finalized state) are needed for richer +auxiliary services (dora, checkpointz analogues, etc.) when they appear. + +--- + +## Required on-disk file format + +Your client must read the files listed in +[`lean-consensus.md#files-mounted-into-every-lean-client`](./lean-consensus.md#files-mounted-into-every-lean-client). +Specifically: + +- **`config.yaml`** with GENESIS_TIME (int), ATTESTATION_COMMITTEE_COUNT, + ACTIVE_EPOCH, VALIDATOR_COUNT, and a GENESIS_VALIDATORS list of + `{attestation_pubkey, proposal_pubkey}` dual-key entries (hex strings + without `0x` prefix). +- **`annotated_validators.yaml`** mapping `: [{index, + pubkey_hex, privkey_file}, ...]` with privkey_file names containing + `_attester_` or `_proposer_` to route to attestation vs proposal slots. +- **`nodes.yaml`** = list of ENRs (base64) as a YAML sequence of strings. +- **`validator-config.yaml`** matching the lean-quickstart schema (used + by some clients for per-node ENR/metrics-port lookups). +- **`hash-sig-keys/validator_N_{attester,proposer}_key_sk.ssz`** as SSZ + XMSS private keys. + +This is the same on-disk shape produced by `lean-quickstart`'s +`generate-genesis.sh`, so a client that runs under lean-quickstart will +run under this package without code changes. + +--- + +## Local test + +```bash +# In ethereum-package root: +kurtosis run --enclave lean-test . --args-file - <<'YAML' +participants: + - el_type: geth + cl_type: lighthouse + count: 1 + validator_count: 0 +lean_participants: + - lean_type: myclient + count: 1 + is_aggregator: true +YAML + +# Inspect the running service +kurtosis service shell lean-test lean-myclient_0 +# Inside the container: +curl http://localhost:5052/lean/v0/health +curl http://localhost:5054/metrics | head +``` + +--- + +## Checklist + +``` +[ ] 1. Add LEAN_TYPE entry + DEFAULT_LEAN_IMAGES entry +[ ] 2. Create src/lean//_launcher.star with initialize + start +[ ] 3. Wire dispatch in src/lean/lean_launcher.star (_launcher_for) +[ ] 4. Add an example line to network_params.yaml under lean_participants: +[ ] 5. Add the client to the supported list at the top of docs/lean-consensus.md +``` diff --git a/docs/lean-consensus.md b/docs/lean-consensus.md new file mode 100644 index 000000000..1879c90f2 --- /dev/null +++ b/docs/lean-consensus.md @@ -0,0 +1,224 @@ +# Lean Ethereum consensus support + +> Status: experimental. Initial integration adds the Lean Ethereum +> consensus stack as a parallel pipeline alongside the existing EL/CL +> network. Only `ethlambda` is fully wired today; `ream` and `zeam` +> have stub launchers covered by the same contract. + +The Lean Ethereum protocol — sometimes called "Beam Chain" — is a redesign of +Ethereum's consensus layer with **no EL pairing**, **no Engine API**, **no +JWT**, and **post-quantum (XMSS / hash-sig) validator signatures**. Lean +consensus clients are standalone consensus nodes that talk only to each other +over QUIC + libp2p gossipsub. The Lean protocol specification lives at +[ReamLabs/leanSpecs](https://github.com/ReamLabs/leanSpecs) and is +co-developed by the teams behind +[ream](https://github.com/ReamLabs/ream) (Rust), +[zeam](https://github.com/blockblaz/zeam) (Zig), +[qlean](https://github.com/qdrvm/qlean-mini) (C++), +[lantern](https://github.com/Pier-Two/lantern) (C), +[grandine](https://github.com/grandinetech/lean/tree/main/lean_client) (Rust), +a [lighthouse](https://github.com/hopinheimer/lighthouse) fork (Rust), and +[ethlambda](https://github.com/lambdaclass/ethlambda) (Rust). + +This document describes how `ethereum-package` runs Lean networks. To add a +new Lean client, see +[`lean-adding-a-new-client.md`](./lean-adding-a-new-client.md). + +--- + +## Why a parallel pipeline? + +Lean consensus differs from the existing EL/CL pipeline along every axis that +shaped the original `participant_network` design: + +| Concern | EL/CL | Lean | +|------------------------|--------------------------------|-------------------------------| +| Genesis tool | `ethereum-genesis-generator` | `eth-beacon-genesis leanchain` | +| Validator signatures | BLS | XMSS (hash-sig) | +| Validator key keystore | EIP-2335 JSON | SSZ `validator_N_*_key_*.ssz` | +| Pairing | 1 EL + 1 CL (+ optional VC) | Standalone, no EL | +| RPC ports | Engine RPC + JWT + REST + WS | REST + Prometheus only | +| P2P transport | TCP + UDP discovery + libp2p | QUIC-only (libp2p) | +| Block production | EL builds payload, CL attests | Single-stack: 4 s slots | + +Trying to express Lean clients as `participants[].cl_type` with `el_type: none` +would force `if is_lean(): ... else:` branches throughout the EL/CL pipeline, +the validator-keystore generator, the genesis generator, the MEV-boost flow, +the snooper, etc. A parallel pipeline keeps those code paths untouched and +isolates Lean-specific concerns under `src/lean/` and +`src/prelaunch_data_generator/lean_genesis/`. + +The package still composes the two: Prometheus/Grafana discover Lean nodes +through their service labels and metrics ports, and additional services that +don't depend on EL state (e.g. dora's beacon explorer) can be pointed at Lean +nodes by URL. + +--- + +## Quick start + +Enable Lean by populating `lean_participants:` in your args file: + +```yaml +participants: + - el_type: geth + cl_type: lighthouse + count: 1 + validator_count: 0 # No EL/CL validators; we just need one EL+CL pair. + +lean_participants: + - lean_type: ethlambda + count: 4 + validator_count: 1 + is_aggregator: true +``` + +Then run: + +```bash +kurtosis run --enclave lean-test github.com/ethpandaops/ethereum-package --args-file your-args.yaml +``` + +The Lean pipeline produces 4 services named `lean-ethlambda_0` … +`lean-ethlambda_3`, each exposing: + +| Port | Purpose | +|-------|--------------------------------------------------| +| 9000 | libp2p QUIC (UDP) — block + attestation gossip | +| 5052 | REST API (`GET /lean/v0/health`, fork choice, …) | +| 5054 | Prometheus metrics (`/metrics`) | + +> **V1 limitation — at least one EL/CL participant required.** The existing +> EL/CL flow assumes at least one EL context exists (e.g. for the +> tx-spammer / dora target). Until lean-only mode is plumbed through every +> downstream consumer, keep one minimal `participants[]` entry with +> `validator_count: 0`. A follow-up will detect `participants: []` and +> short-circuit the EL/CL pipeline. + +--- + +## Pipeline architecture + +The Lean launcher (`src/lean/lean_launcher.star`) runs in three phases. +Phases 1 and 3 are per-node; phase 2 is global. + +``` + ┌──────────────────────────────────┐ + Phase 1 │ openssl: generate .key x N │ + └──────────────────────────────────┘ + │ + ▼ + ┌────────────────────────────────────────────────────────────┐ + │ For each Lean participant entry: │ + │ plan.add_service(name=lean-_, cmd="tail -f") │ + │ Kurtosis assigns an IP to each service. │ + └────────────────────────────────────────────────────────────┘ + │ + Phase 2 ▼ + ┌────────────────────────────────────────────────────────────┐ + │ hash-sig-cli: generate XMSS attester+proposer keys (SSZ) │ + │ render validator-config.yaml from live IPs + ports │ + │ render initial config.yaml (GENESIS_TIME etc.) │ + │ eth-beacon-genesis leanchain: write nodes.yaml, │ + │ validators.yaml, genesis.{ssz,json}, update config.yaml │ + │ post-process: inject GENESIS_VALIDATORS into config.yaml, │ + │ render annotated_validators.yaml from manifest │ + └────────────────────────────────────────────────────────────┘ + │ + Phase 3 ▼ + ┌────────────────────────────────────────────────────────────┐ + │ For each placeholder service: │ + │ plan.add_service(name=, force_update=True, │ + │ mounts={genesis_artifact, hash_sig_artifact, keys}, │ + │ cmd=) │ + │ Kurtosis preserves the IP (same name + ports). │ + └────────────────────────────────────────────────────────────┘ +``` + +Why three phases? Because the genesis tool needs every node's IP and port to +render `nodes.yaml` (the bootnode list), but Kurtosis only assigns IPs after +`add_service`. Pre-allocating placeholder services then re-issuing them with +`force_update=True` keeps the IP stable while letting us mount the +just-generated genesis bundle. + +--- + +## Files mounted into every Lean client + +All Lean clients receive the same on-disk layout. This matches the layout +produced by `lean-quickstart`'s `generate-genesis.sh` so a Lean client that +runs under `lean-quickstart` runs under this package without code changes. + +| Path | Source | Contents | +|-------------------------------------------------------|------------------------------|-----------------------------------------------------------------| +| `/network-configs/config.yaml` | Lean genesis post-process | GENESIS_TIME, ATTESTATION_COMMITTEE_COUNT, ACTIVE_EPOCH, VALIDATOR_COUNT, GENESIS_VALIDATORS (per-validator attestation/proposal pubkeys) | +| `/network-configs/validators.yaml` | PK's eth-beacon-genesis | `node_name -> [validator_index]` round-robin assignments | +| `/network-configs/annotated_validators.yaml` | Lean genesis post-process | `node_name -> [{index, pubkey_hex, privkey_file}]` | +| `/network-configs/nodes.yaml` | PK's eth-beacon-genesis | ENR list for all Lean nodes (bootnodes) | +| `/network-configs/validator-config.yaml` | Lean launcher (rendered) | Per-node config (name, privkey, IP, ports, count, isAggregator) | +| `/network-configs/genesis.ssz` | PK's eth-beacon-genesis | SSZ genesis state | +| `/network-configs/genesis.json` | PK's eth-beacon-genesis | JSON genesis state | +| `/network-configs/.key` | openssl prelaunch step | 32-byte hex libp2p secret for this node | +| `/network-configs/hash-sig-keys/validator_N_attester_key_{sk,pk}.ssz` | hash-sig-cli | XMSS attester keypair per validator | +| `/network-configs/hash-sig-keys/validator_N_proposer_key_{sk,pk}.ssz` | hash-sig-cli | XMSS proposer keypair per validator | +| `/network-configs/hash-sig-keys/validator-keys-manifest.yaml` | hash-sig-cli | Dual-key manifest mapping validator index to attester/proposer pubkey hex | +| `/node-keys/.key` | openssl prelaunch step | Same as above; kept at a separate mount for clients that expect this layout | + +> Clients SHOULD derive their genesis state from `config.yaml` directly +> (using GENESIS_VALIDATORS pubkeys and GENESIS_TIME). The `genesis.json` / +> `genesis.ssz` files are provided for compatibility but their format may +> drift across leanSpec revisions. + +--- + +## Port contract + +| Port | Protocol | Purpose | +|-------|----------|-------------------------------| +| 9000 | UDP | libp2p QUIC (block + attestation gossipsub) | +| 5052 | TCP HTTP | REST API (must implement `GET /lean/v0/health`) | +| 5054 | TCP HTTP | Prometheus metrics (`/metrics`) | + +These match the defaults used by every Lean client in `lean-quickstart` so +operator-facing dashboards and probes work across deployments. + +--- + +## Components added + +| Path | Purpose | +|-----------------------------------------------------------------|--------------------------------------------------------------------------| +| `src/package_io/constants.star` | `LEAN_TYPE` enum, default port nums, mountpoints, genesis-tool images. | +| `src/package_io/input_parser.star` | `lean_participants` / `lean_network_params` parsing + defaults. | +| `src/prelaunch_data_generator/lean_genesis/p2p_keys_generator.star` | Generates one 32-byte hex libp2p secret per node (openssl). | +| `src/prelaunch_data_generator/lean_genesis/lean_genesis_generator.star` | Full Lean genesis pipeline (hash-sig + leanchain + post-process). | +| `src/lean/lean_launcher.star` | Dispatches per-client launchers, runs the three-phase lifecycle. | +| `src/lean/lean_context.star` | Per-node context struct returned to `main.star`. | +| `src/lean/lean_shared.star` | Common port specs, mountpoint helpers, log file conventions. | +| `src/lean/ethlambda/ethlambda_launcher.star` | First fully-wired client. | +| `src/lean/ream/ream_launcher.star` | Stub mirroring `client-cmds/ream-cmd.sh`. | +| `src/lean/zeam/zeam_launcher.star` | Stub mirroring `client-cmds/zeam-cmd.sh`. | +| `main.star` | Single call site for `lean_launcher.launch(...)`. | +| `network_params.yaml` | Documented `lean_participants:` example + default `lean_network_params`. | + +--- + +## Limitations & follow-ups + +1. **At least one EL/CL participant required.** The existing main flow + assumes `all_el_contexts[0]` exists for the tx-spammer / dora target. + Until lean-only mode is wired through every downstream consumer, keep + one minimal `participants[]` entry with `validator_count: 0`. +2. **Lean nodes are not yet scraped by Prometheus.** The `metrics_info` + struct is populated on every `lean_context`, but the prometheus + launcher isn't yet wired to discover Lean nodes. Operators scraping + Lean nodes today should hit them by service name directly. +3. **`hash-sig-cli` image is pinned to `:latest`.** Pinning to a SHA is + left to a follow-up; override via + `lean_network_params.hash_sig_cli_image`. +4. **No Lean-specific dashboards.** Existing Grafana dashboards assume the + Ethereum CL schema. Lean dashboards (`lean_head_slot`, + `lean_state_transition_time_seconds`, etc.) need a separate dashboard + pack. +5. **No checkpoint sync.** Per-participant `checkpoint_sync_url` parsing + is not yet wired through to the per-client launchers. diff --git a/main.star b/main.star index 610d0fd4a..dcbbd1fe8 100644 --- a/main.star +++ b/main.star @@ -1,6 +1,7 @@ input_parser = import_module("./src/package_io/input_parser.star") constants = import_module("./src/package_io/constants.star") participant_network = import_module("./src/participant_network.star") +lean_launcher = import_module("./src/lean/lean_launcher.star") shared_utils = import_module("./src/shared_utils/shared_utils.star") static_files = import_module("./src/static_files/static_files.star") genesis_constants = import_module( @@ -329,6 +330,18 @@ def run(plan, args={}): ) all_xatu_sentry_contexts.append(participant.xatu_sentry_context) + # Launch Lean Ethereum consensus participants alongside the EL/CL network. + # Lean is a standalone consensus stack (no EL pairing, no Engine API, no + # JWT, post-quantum signatures); it runs through its own pipeline and + # produces independent file artifacts + services. The list is empty + # unless the user populated `lean_participants:` in their args. + # See docs/lean-consensus.md for the architecture. + all_lean_contexts = lean_launcher.launch( + plan, + args_with_right_defaults.lean_participants, + args_with_right_defaults.lean_network_params, + ) + # Generate validator ranges validator_ranges_config_template = read_file( static_files.VALIDATOR_RANGES_CONFIG_TEMPLATE_FILEPATH diff --git a/network_params.yaml b/network_params.yaml index 7756bc257..00f461297 100644 --- a/network_params.yaml +++ b/network_params.yaml @@ -291,3 +291,33 @@ port_publisher: # #!/bin/bash # echo "Hello" extra_files: {} + +# Lean Ethereum consensus participants. Each entry adds one or more Lean +# consensus nodes that run alongside (and independently of) the EL/CL +# network above. Lean has no EL pairing, no Engine API, no JWT, and uses +# post-quantum (XMSS) validator signatures. See docs/lean-consensus.md for +# the architecture and docs/lean-adding-a-new-client.md for adding a new +# Lean client. +# +# Example - 4 ethlambda nodes with one validator each: +# lean_participants: +# - lean_type: ethlambda +# lean_image: ghcr.io/lambdaclass/ethlambda:devnet4 +# count: 4 +# validator_count: 1 +# is_aggregator: false +lean_participants: [] +lean_network_params: + # Seconds added to "now" to compute GENESIS_TIME when genesis_time is 0. + genesis_delay: 60 + # Explicit Unix timestamp; 0 = derive from genesis_delay. + genesis_time: 0 + # leanSpec ATTESTATION_COMMITTEE_COUNT (1 = single committee per slot). + attestation_committee_count: 1 + # log_2(active epochs) for the XMSS hash-sig scheme. + active_epoch: 18 + # Default validator keys per node (overridable per participant). + num_validator_keys_per_node: 1 + # Override the Lean genesis tooling images. Empty = pinned default. + genesis_generator_image: "" + hash_sig_cli_image: "" diff --git a/src/lean/ethlambda/ethlambda_launcher.star b/src/lean/ethlambda/ethlambda_launcher.star new file mode 100644 index 000000000..814ba3751 --- /dev/null +++ b/src/lean/ethlambda/ethlambda_launcher.star @@ -0,0 +1,161 @@ +""" +ethlambda launcher. + +Translates the Lean pipeline's per-node record into the ethlambda CLI surface +(see lambdaclass/ethlambda's `bin/ethlambda/src/main.rs` and the matching +lean-quickstart `client-cmds/ethlambda-cmd.sh`). + +Lifecycle (called by ../lean_launcher.star): + 1. `initialize()` - add a Kurtosis service holding the P2P keys + the + pre-generated hash-sig keys (these don't depend on the node IP). The + container runs `tail -f` on a log file so it stays alive while the + genesis pipeline computes things downstream that DO need the IP. + 2. `start()` - after the genesis tool has produced the text artifacts + (config.yaml, annotated_validators.yaml, nodes.yaml, validator-config.yaml), + stage them inside the running container via `plan.exec` and launch + the ethlambda binary as a backgrounded `nohup` process. + +Why two phases rather than one `add_service` with everything mounted? Because +the genesis tool needs every node's IP to render `nodes.yaml`, and Kurtosis +only assigns IPs after `add_service`. Recreating services to swap mounts +would also reshuffle IPs and invalidate the ENRs we just embedded. +""" + +constants = import_module("../../package_io/constants.star") +lean_shared = import_module("../lean_shared.star") +lean_context = import_module("../lean_context.star") + +ENTRYPOINT = "/usr/local/bin/ethlambda" +GENESIS_MOUNT = constants.LEAN_GENESIS_MOUNTPOINT_ON_CLIENTS +HASH_SIG_MOUNT = GENESIS_MOUNT + "/hash-sig-keys" +DATA_DIR = "/data" +NODE_KEY_MOUNT = constants.LEAN_NODE_KEY_MOUNTPOINT_ON_CLIENTS + + +def initialize(plan, node, p2p_keys_artifact): + """Phase 1: stand the placeholder service up so Kurtosis assigns an IP.""" + cfg = ServiceConfig( + image=node["image"], + # Override the image ENTRYPOINT so the container doesn't try to start + # ethlambda with no flags before we've written the genesis files in. + entrypoint=["/bin/sh", "-c"], + cmd=lean_shared.lean_tail_logs_cmd(node["service_name"])[2:], + ports=lean_shared.lean_port_specs(), + files={ + NODE_KEY_MOUNT: p2p_keys_artifact, + }, + env_vars=node["extra_env_vars"], + labels=node["extra_labels"], + min_cpu=node["min_cpu"], + max_cpu=node["max_cpu"], + min_memory=node["min_mem"], + max_memory=node["max_mem"], + node_selectors=node["node_selectors"], + tolerations=node["tolerations"], + ) + return plan.add_service(node["service_name"], cfg) + + +def start(plan, node, service, genesis_artifact, hash_sig_artifact): + """Phase 3: mount genesis files and start the ethlambda binary. + + We re-issue add_service with force_update=True so we can mount the + genesis + hash-sig artifacts (Kurtosis preserves the IP because the + service name and port assignments are unchanged; ENR fields embedded + in the genesis bundle therefore remain valid). + """ + service_name = service.name + + cmd_parts = [ + ENTRYPOINT, + "--genesis", + "{0}/config.yaml".format(GENESIS_MOUNT), + "--validators", + "{0}/annotated_validators.yaml".format(GENESIS_MOUNT), + "--bootnodes", + "{0}/nodes.yaml".format(GENESIS_MOUNT), + "--validator-config", + "{0}/validator-config.yaml".format(GENESIS_MOUNT), + "--hash-sig-keys-dir", + HASH_SIG_MOUNT, + "--data-dir", + DATA_DIR, + "--gossipsub-port", + str(constants.LEAN_QUIC_PORT_NUM), + "--node-id", + node["node_name"], + "--node-key", + "{0}/{1}.key".format(NODE_KEY_MOUNT, node["node_name"]), + "--http-address", + "0.0.0.0", + "--api-port", + str(constants.LEAN_API_PORT_NUM), + "--metrics-port", + str(constants.LEAN_METRICS_PORT_NUM), + ] + if node["is_aggregator"]: + cmd_parts.append("--is-aggregator") + for extra in node["extra_params"]: + cmd_parts.append(extra) + + log_file = lean_shared.lean_log_file_path(service_name) + full_cmd = " ".join(cmd_parts) + + env_vars = dict(node["extra_env_vars"]) + if node["log_level"] != "": + env_vars["RUST_LOG"] = node["log_level"] + + new_cfg = ServiceConfig( + image=node["image"], + entrypoint=["/bin/sh", "-c"], + # `tee -a` keeps a log file readable by Kurtosis's `service logs` + # while also surfacing the binary's stdout/stderr live. + cmd=[ + "{0} 2>&1 | tee -a {1}".format(full_cmd, log_file), + ], + ports=lean_shared.lean_port_specs(), + files={ + NODE_KEY_MOUNT: node["_p2p_keys_artifact"], + GENESIS_MOUNT: genesis_artifact, + HASH_SIG_MOUNT: hash_sig_artifact, + }, + env_vars=env_vars, + labels=node["extra_labels"], + min_cpu=node["min_cpu"], + max_cpu=node["max_cpu"], + min_memory=node["min_mem"], + max_memory=node["max_mem"], + node_selectors=node["node_selectors"], + tolerations=node["tolerations"], + ) + new_service = plan.add_service( + name=service_name, + config=new_cfg, + force_update=True, + ) + + api_url = "http://{0}:{1}".format( + new_service.ip_address, + constants.LEAN_API_PORT_NUM, + ) + metrics_url = "http://{0}:{1}/metrics".format( + new_service.ip_address, + constants.LEAN_METRICS_PORT_NUM, + ) + + return lean_context.new_lean_context( + client_name=constants.LEAN_TYPE.ethlambda, + service_name=new_service.name, + ip_address=new_service.ip_address, + quic_port=constants.LEAN_QUIC_PORT_NUM, + api_port=constants.LEAN_API_PORT_NUM, + metrics_port=constants.LEAN_METRICS_PORT_NUM, + api_url=api_url, + metrics_url=metrics_url, + metrics_info={ + "name": new_service.name, + "url": metrics_url, + "path": "/metrics", + "config": node["prometheus_config"], + }, + ) diff --git a/src/lean/lean_context.star b/src/lean/lean_context.star new file mode 100644 index 000000000..bfeb8f4e6 --- /dev/null +++ b/src/lean/lean_context.star @@ -0,0 +1,30 @@ +""" +Per-node Lean context. + +Returned by every `lean//_launcher.start_*` and consumed by +Prometheus / Grafana / dora / etc. to discover the running Lean nodes. +""" + + +def new_lean_context( + client_name, + service_name, + ip_address, + quic_port, + api_port, + metrics_port, + api_url, + metrics_url, + metrics_info=None, +): + return struct( + client_name=client_name, + service_name=service_name, + ip_address=ip_address, + quic_port=quic_port, + api_port=api_port, + metrics_port=metrics_port, + api_url=api_url, + metrics_url=metrics_url, + metrics_info=metrics_info, + ) diff --git a/src/lean/lean_launcher.star b/src/lean/lean_launcher.star new file mode 100644 index 000000000..70a42db42 --- /dev/null +++ b/src/lean/lean_launcher.star @@ -0,0 +1,170 @@ +""" +Lean Ethereum participant launcher. + +Orchestrates the entire Lean pipeline: + + 1. Generate per-node libp2p P2P keys (so we can render ENRs deterministically). + 2. Initialise placeholder Kurtosis services so we get assigned IPs. + 3. Run the Lean genesis pipeline (eth-beacon-genesis leanchain + hash-sig-cli) + against the live IPs. + 4. Mount the genesis bundle into each placeholder and start the real client + binary via `plan.exec`. + +This is intentionally independent of the EL/CL `participant_network` pipeline: +Lean consensus has no Engine API, no JWT, no EL pairing. Operators opt in by +populating `lean_participants:` in their args; the existing EL/CL flow runs +unchanged either way. +""" + +constants = import_module("../package_io/constants.star") +lean_shared = import_module("./lean_shared.star") +lean_genesis = import_module( + "../prelaunch_data_generator/lean_genesis/lean_genesis_generator.star" +) +p2p_keys = import_module( + "../prelaunch_data_generator/lean_genesis/p2p_keys_generator.star" +) + +ethlambda_launcher = import_module("./ethlambda/ethlambda_launcher.star") +ream_launcher = import_module("./ream/ream_launcher.star") +zeam_launcher = import_module("./zeam/zeam_launcher.star") + + +def _launcher_for(lean_type): + if lean_type == constants.LEAN_TYPE.ethlambda: + return ethlambda_launcher + elif lean_type == constants.LEAN_TYPE.ream: + return ream_launcher + elif lean_type == constants.LEAN_TYPE.zeam: + return zeam_launcher + fail( + "Unsupported lean_type '{0}'. Supported: {1}. See ".format( + lean_type, + ", ".join( + [ + constants.LEAN_TYPE.ethlambda, + constants.LEAN_TYPE.ream, + constants.LEAN_TYPE.zeam, + ] + ), + ) + + "docs/lean-adding-a-new-client.md to add a new client." + ) + + +def launch(plan, lean_participants, lean_network_params): + """Top-level entrypoint for the Lean pipeline. + + Returns the list of `lean_context` structs (one per running node), + suitable for handing to Prometheus / Grafana / dora. + """ + if not lean_participants: + return [] + + # Expand per-participant `count` to a flat list of (type, image, ...) records. + # Naming follows lean-quickstart's `_` convention so a + # Lean client's existing log parsers and dashboards work unchanged. + expanded = [] + type_counters = {} + for participant in lean_participants: + lean_type = participant["lean_type"] + for _ in range(participant["count"]): + idx = type_counters.get(lean_type, 0) + type_counters[lean_type] = idx + 1 + node_name = "{0}_{1}".format(lean_type, idx) + expanded.append( + { + "node_name": node_name, + "service_name": "lean-{0}".format(node_name), + "lean_type": lean_type, + "image": participant["lean_image"], + "validator_count": participant.get( + "validator_count", + lean_network_params["num_validator_keys_per_node"], + ), + "is_aggregator": participant.get("is_aggregator", False), + "extra_params": participant.get("lean_extra_params", []), + "extra_env_vars": participant.get("lean_extra_env_vars", {}), + "extra_labels": participant.get("lean_extra_labels", {}), + "log_level": participant.get("lean_log_level", ""), + "min_cpu": participant.get("lean_min_cpu", 0), + "max_cpu": participant.get("lean_max_cpu", 0), + "min_mem": participant.get("lean_min_mem", 0), + "max_mem": participant.get("lean_max_mem", 0), + "node_selectors": participant.get("node_selectors", {}), + "tolerations": participant.get("tolerations", []), + "prometheus_config": participant.get( + "prometheus_config", + { + "scrape_interval": "15s", + "labels": {}, + }, + ), + } + ) + + node_names = [n["node_name"] for n in expanded] + keys_result = p2p_keys.generate_node_keys(plan, node_names) + + # Stash the P2P keys artifact on each node record so per-client launchers + # can re-mount it during the `start()` phase (where we re-issue the + # add_service with the full mounts). + for node in expanded: + node["_p2p_keys_artifact"] = keys_result.artifact_name + + # Phase 1: initialise placeholder services so Kurtosis assigns IPs. + services = [] + for node in expanded: + launcher = _launcher_for(node["lean_type"]) + service = launcher.initialize( + plan, + node, + keys_result.artifact_name, + ) + services.append((node, service)) + + # Phase 2: render the validator-config.yaml and run the genesis tool now + # that every service has an IP. + services_meta = [] + for node, service in services: + services_meta.append( + { + "name": node["node_name"], + "ip_address": service.ip_address, + "quic_port": constants.LEAN_QUIC_PORT_NUM, + "metrics_port": constants.LEAN_METRICS_PORT_NUM, + "api_port": constants.LEAN_API_PORT_NUM, + "privkey": keys_result.keys[node["node_name"]], + "validator_count": node["validator_count"], + "is_aggregator": node["is_aggregator"], + } + ) + + genesis = lean_genesis.generate( + plan, + services_meta, + lean_network_params, + keys_result.artifact_name, + ) + + # Phase 3: hand off to each per-client launcher to mount the genesis bundle + # and start the real binary. + contexts = [] + for node, service in services: + launcher = _launcher_for(node["lean_type"]) + ctx = launcher.start( + plan, + node, + service, + genesis.genesis_artifact, + genesis.hash_sig_artifact, + ) + contexts.append(ctx) + + plan.print( + "Lean pipeline ready: {0} nodes, GENESIS_TIME={1}".format( + len(contexts), + genesis.genesis_time, + ) + ) + return contexts diff --git a/src/lean/lean_shared.star b/src/lean/lean_shared.star new file mode 100644 index 000000000..9eaf830df --- /dev/null +++ b/src/lean/lean_shared.star @@ -0,0 +1,60 @@ +""" +Shared helpers for Lean client launchers. + +Every Lean client speaks the same wire protocols (libp2p QUIC + JSON REST + +Prometheus). Keeping the port specs and mountpoint contract in one module +lets per-client launchers stay focused on CLI translation. +""" + +constants = import_module("../package_io/constants.star") + + +def lean_port_specs(): + """Default port spec triple for any Lean client. + + QUIC is the only P2P protocol (no TCP discovery); the API and metrics + endpoints are plain HTTP. Each Lean client maps these to its own CLI + flag names (`--gossipsub-port`, `--http-port`, etc.) inside its + launcher. + """ + return { + constants.LEAN_QUIC_PORT_ID: PortSpec( + number=constants.LEAN_QUIC_PORT_NUM, + transport_protocol="UDP", + application_protocol="quic", + wait=None, + ), + constants.LEAN_API_PORT_ID: PortSpec( + number=constants.LEAN_API_PORT_NUM, + transport_protocol="TCP", + application_protocol="http", + wait=None, + ), + constants.LEAN_METRICS_PORT_ID: PortSpec( + number=constants.LEAN_METRICS_PORT_NUM, + transport_protocol="TCP", + application_protocol="http", + wait=None, + ), + } + + +def lean_log_file_path(service_name): + """Path inside the container where the Lean client's logs are tailed. + + Stored under /var/log so a single `tail -f` keeps the Kurtosis service + "alive" while we initialise it; the client itself runs as a backgrounded + `nohup`. Matches the pattern ReamLabs/pq-devnet-package established. + """ + return "/var/log/{0}.log".format(service_name) + + +def lean_tail_logs_cmd(service_name): + """Initial container command — touches and tails the log file. + + Holds the service open until the actual client binary is started by a + follow-up `plan.exec`. Without this, Kurtosis would mark the service + failed before we got a chance to mount the genesis bundle. + """ + log_file = lean_log_file_path(service_name) + return ["/bin/sh", "-c", "touch {0} && tail -f {0}".format(log_file)] diff --git a/src/lean/ream/ream_launcher.star b/src/lean/ream/ream_launcher.star new file mode 100644 index 000000000..718042b71 --- /dev/null +++ b/src/lean/ream/ream_launcher.star @@ -0,0 +1,143 @@ +""" +ream launcher. + +Mirrors the ethlambda launcher shape (placeholder service -> genesis pipeline +-> re-issue with full mounts) but translates to ream's CLI surface as +documented in `client-cmds/ream-cmd.sh` in blockblaz/lean-quickstart: + + ream --data-dir lean_node \ + --network \ + --validator-registry-path \ + --bootnodes \ + --node-id --node-key \ + --socket-port \ + --metrics --metrics-address 0.0.0.0 --metrics-port \ + --http-address 0.0.0.0 --http-port + +NOTE: ream's `lean_node` subcommand doesn't (yet) consume a hash-sig-keys +directory in its public CLI - it derives its validator set from +`annotated_validators.yaml` + GENESIS_VALIDATORS in config.yaml. The genesis +artifact we mount carries both, so ream needs no extra plumbing. +""" + +constants = import_module("../../package_io/constants.star") +lean_shared = import_module("../lean_shared.star") +lean_context = import_module("../lean_context.star") + +ENTRYPOINT = "/usr/local/bin/ream" +GENESIS_MOUNT = constants.LEAN_GENESIS_MOUNTPOINT_ON_CLIENTS +HASH_SIG_MOUNT = GENESIS_MOUNT + "/hash-sig-keys" +DATA_DIR = "/data" +NODE_KEY_MOUNT = constants.LEAN_NODE_KEY_MOUNTPOINT_ON_CLIENTS + + +def initialize(plan, node, p2p_keys_artifact): + cfg = ServiceConfig( + image=node["image"], + entrypoint=["/bin/sh", "-c"], + cmd=lean_shared.lean_tail_logs_cmd(node["service_name"])[2:], + ports=lean_shared.lean_port_specs(), + files={NODE_KEY_MOUNT: p2p_keys_artifact}, + env_vars=node["extra_env_vars"], + labels=node["extra_labels"], + min_cpu=node["min_cpu"], + max_cpu=node["max_cpu"], + min_memory=node["min_mem"], + max_memory=node["max_mem"], + node_selectors=node["node_selectors"], + tolerations=node["tolerations"], + ) + return plan.add_service(node["service_name"], cfg) + + +def start(plan, node, service, genesis_artifact, hash_sig_artifact): + service_name = service.name + + cmd_parts = [ + ENTRYPOINT, + "--data-dir", + DATA_DIR, + "lean_node", + "--network", + "{0}/config.yaml".format(GENESIS_MOUNT), + "--validator-registry-path", + "{0}/annotated_validators.yaml".format(GENESIS_MOUNT), + "--bootnodes", + "{0}/nodes.yaml".format(GENESIS_MOUNT), + "--node-id", + node["node_name"], + "--node-key", + "{0}/{1}.key".format(NODE_KEY_MOUNT, node["node_name"]), + "--socket-port", + str(constants.LEAN_QUIC_PORT_NUM), + "--metrics", + "--metrics-address", + "0.0.0.0", + "--metrics-port", + str(constants.LEAN_METRICS_PORT_NUM), + "--http-address", + "0.0.0.0", + "--http-port", + str(constants.LEAN_API_PORT_NUM), + ] + if node["is_aggregator"]: + cmd_parts.append("--is-aggregator") + for extra in node["extra_params"]: + cmd_parts.append(extra) + + log_file = lean_shared.lean_log_file_path(service_name) + full_cmd = " ".join(cmd_parts) + + env_vars = dict(node["extra_env_vars"]) + if node["log_level"] != "": + env_vars["RUST_LOG"] = node["log_level"] + + new_cfg = ServiceConfig( + image=node["image"], + entrypoint=["/bin/sh", "-c"], + cmd=["{0} 2>&1 | tee -a {1}".format(full_cmd, log_file)], + ports=lean_shared.lean_port_specs(), + files={ + NODE_KEY_MOUNT: node["_p2p_keys_artifact"], + GENESIS_MOUNT: genesis_artifact, + HASH_SIG_MOUNT: hash_sig_artifact, + }, + env_vars=env_vars, + labels=node["extra_labels"], + min_cpu=node["min_cpu"], + max_cpu=node["max_cpu"], + min_memory=node["min_mem"], + max_memory=node["max_mem"], + node_selectors=node["node_selectors"], + tolerations=node["tolerations"], + ) + new_service = plan.add_service( + name=service_name, + config=new_cfg, + force_update=True, + ) + + api_url = "http://{0}:{1}".format( + new_service.ip_address, constants.LEAN_API_PORT_NUM + ) + metrics_url = "http://{0}:{1}/metrics".format( + new_service.ip_address, + constants.LEAN_METRICS_PORT_NUM, + ) + + return lean_context.new_lean_context( + client_name=constants.LEAN_TYPE.ream, + service_name=new_service.name, + ip_address=new_service.ip_address, + quic_port=constants.LEAN_QUIC_PORT_NUM, + api_port=constants.LEAN_API_PORT_NUM, + metrics_port=constants.LEAN_METRICS_PORT_NUM, + api_url=api_url, + metrics_url=metrics_url, + metrics_info={ + "name": new_service.name, + "url": metrics_url, + "path": "/metrics", + "config": node["prometheus_config"], + }, + ) diff --git a/src/lean/zeam/zeam_launcher.star b/src/lean/zeam/zeam_launcher.star new file mode 100644 index 000000000..2775997e7 --- /dev/null +++ b/src/lean/zeam/zeam_launcher.star @@ -0,0 +1,129 @@ +""" +zeam launcher. + +Translates the Lean pipeline's per-node record into zeam's CLI surface from +`client-cmds/zeam-cmd.sh` in blockblaz/lean-quickstart: + + zeam node \ + --custom-genesis \ + --validator-config \ + --data-dir \ + --node-id --node-key \ + --metrics-enable --api-port --metrics-port + +zeam supports the "genesis_bootnode" sentinel for participants that should +derive their validator config from GENESIS_VALIDATORS rather than reading +the per-node `validator-config.yaml`. We pass the full path here because +that's the safer default; users wanting the sentinel can set +`lean_extra_params: ["--validator-config", "genesis_bootnode"]`. +""" + +constants = import_module("../../package_io/constants.star") +lean_shared = import_module("../lean_shared.star") +lean_context = import_module("../lean_context.star") + +ENTRYPOINT = "/app/zig-out/bin/zeam" +GENESIS_MOUNT = constants.LEAN_GENESIS_MOUNTPOINT_ON_CLIENTS +HASH_SIG_MOUNT = GENESIS_MOUNT + "/hash-sig-keys" +DATA_DIR = "/data" +NODE_KEY_MOUNT = constants.LEAN_NODE_KEY_MOUNTPOINT_ON_CLIENTS + + +def initialize(plan, node, p2p_keys_artifact): + cfg = ServiceConfig( + image=node["image"], + entrypoint=["/bin/sh", "-c"], + cmd=lean_shared.lean_tail_logs_cmd(node["service_name"])[2:], + ports=lean_shared.lean_port_specs(), + files={NODE_KEY_MOUNT: p2p_keys_artifact}, + env_vars=node["extra_env_vars"], + labels=node["extra_labels"], + min_cpu=node["min_cpu"], + max_cpu=node["max_cpu"], + min_memory=node["min_mem"], + max_memory=node["max_mem"], + node_selectors=node["node_selectors"], + tolerations=node["tolerations"], + ) + return plan.add_service(node["service_name"], cfg) + + +def start(plan, node, service, genesis_artifact, hash_sig_artifact): + service_name = service.name + + cmd_parts = [ + ENTRYPOINT, + "node", + "--custom-genesis", + GENESIS_MOUNT, + "--validator-config", + "{0}/validator-config.yaml".format(GENESIS_MOUNT), + "--data-dir", + DATA_DIR, + "--node-id", + node["node_name"], + "--node-key", + "{0}/{1}.key".format(NODE_KEY_MOUNT, node["node_name"]), + "--metrics-enable", + "--api-port", + str(constants.LEAN_API_PORT_NUM), + "--metrics-port", + str(constants.LEAN_METRICS_PORT_NUM), + ] + if node["is_aggregator"]: + cmd_parts.append("--is-aggregator") + for extra in node["extra_params"]: + cmd_parts.append(extra) + + log_file = lean_shared.lean_log_file_path(service_name) + full_cmd = " ".join(cmd_parts) + + new_cfg = ServiceConfig( + image=node["image"], + entrypoint=["/bin/sh", "-c"], + cmd=["{0} 2>&1 | tee -a {1}".format(full_cmd, log_file)], + ports=lean_shared.lean_port_specs(), + files={ + NODE_KEY_MOUNT: node["_p2p_keys_artifact"], + GENESIS_MOUNT: genesis_artifact, + HASH_SIG_MOUNT: hash_sig_artifact, + }, + env_vars=node["extra_env_vars"], + labels=node["extra_labels"], + min_cpu=node["min_cpu"], + max_cpu=node["max_cpu"], + min_memory=node["min_mem"], + max_memory=node["max_mem"], + node_selectors=node["node_selectors"], + tolerations=node["tolerations"], + ) + new_service = plan.add_service( + name=service_name, + config=new_cfg, + force_update=True, + ) + + api_url = "http://{0}:{1}".format( + new_service.ip_address, constants.LEAN_API_PORT_NUM + ) + metrics_url = "http://{0}:{1}/metrics".format( + new_service.ip_address, + constants.LEAN_METRICS_PORT_NUM, + ) + + return lean_context.new_lean_context( + client_name=constants.LEAN_TYPE.zeam, + service_name=new_service.name, + ip_address=new_service.ip_address, + quic_port=constants.LEAN_QUIC_PORT_NUM, + api_port=constants.LEAN_API_PORT_NUM, + metrics_port=constants.LEAN_METRICS_PORT_NUM, + api_url=api_url, + metrics_url=metrics_url, + metrics_info={ + "name": new_service.name, + "url": metrics_url, + "path": "/metrics", + "config": node["prometheus_config"], + }, + ) diff --git a/src/package_io/constants.star b/src/package_io/constants.star index edbdfec61..037f1e787 100644 --- a/src/package_io/constants.star +++ b/src/package_io/constants.star @@ -22,6 +22,23 @@ CL_TYPE = struct( caplin="caplin", ) +# Lean Ethereum consensus clients. These are standalone consensus clients +# for the Lean Ethereum specification (post-quantum signatures, no EL pairing, +# no Engine API). They run via the separate Lean pipeline in src/lean/ and +# do NOT belong in `participants:`; use `lean_participants:` instead. +LEAN_TYPE = struct( + ethlambda="ethlambda", + ream="ream", + zeam="zeam", + qlean="qlean", + lantern="lantern", + grandine="grandine", + lighthouse="lighthouse", + gean="gean", + peam="peam", + nlean="nlean", +) + VC_TYPE = struct( lighthouse="lighthouse", lodestar="lodestar", @@ -110,6 +127,31 @@ DEFAULT_BOOTNODOOR_IMAGE = "ethpandaops/bootnodoor:latest" DEFAULT_ETHEREUM_GENESIS_GENERATOR_IMAGE = ( "ethpandaops/ethereum-genesis-generator:6.0.6" ) + +# Lean genesis tooling. `pk910-leanchain` is pk910's leanchain branch of +# eth-beacon-genesis (ethpandaops/eth-beacon-genesis PR #36); it consumes a +# validator-config.yaml and emits config.yaml + validators.yaml + nodes.yaml + +# genesis.{ssz,json}. `hash-sig-cli` generates the XMSS attester/proposer +# keypairs that GENESIS_VALIDATORS references. +DEFAULT_LEAN_GENESIS_GENERATOR_IMAGE = "ethpandaops/eth-beacon-genesis:pk910-leanchain" +DEFAULT_LEAN_HASH_SIG_CLI_IMAGE = "blockblaz/hash-sig-cli:latest" + +# Lean P2P / API / metrics port IDs and defaults. Lean clients speak QUIC over +# UDP only (no TCP discovery), expose a JSON REST API, and a Prometheus metrics +# endpoint on a separate port — the same triple used by every Lean client in +# blockblaz/lean-quickstart. +LEAN_QUIC_PORT_ID = "quic" +LEAN_API_PORT_ID = "http" +LEAN_METRICS_PORT_ID = "metrics" +LEAN_QUIC_PORT_NUM = 9000 +LEAN_API_PORT_NUM = 5052 +LEAN_METRICS_PORT_NUM = 5054 + +# Mountpoints inside Lean client containers. Kept stable across all Lean +# clients so the integration contract documented in docs/lean-adding-a-new-client.md +# matches what clients receive at runtime. +LEAN_GENESIS_MOUNTPOINT_ON_CLIENTS = "/network-configs" +LEAN_NODE_KEY_MOUNTPOINT_ON_CLIENTS = "/node-keys" DEFAULT_YQ_IMAGE = "linuxserver/yq" DEFAULT_FLASHBOTS_RELAY_IMAGE = "ethpandaops/mev-boost-relay:main" DEFAULT_FLASHBOTS_BUILDER_IMAGE = "ethpandaops/reth-rbuilder:develop" diff --git a/src/package_io/input_parser.star b/src/package_io/input_parser.star index 7ab3732a7..87e545279 100644 --- a/src/package_io/input_parser.star +++ b/src/package_io/input_parser.star @@ -65,6 +65,22 @@ DEFAULT_REMOTE_SIGNER_IMAGES = { "web3signer": "consensys/web3signer:latest", } +# Default Lean Ethereum client images. Mirrors the image list maintained by +# blockblaz/lean-quickstart so devnet operators can swap between the two +# launchers (Kurtosis vs lean-quickstart) without rebuilding. +DEFAULT_LEAN_IMAGES = { + constants.LEAN_TYPE.ethlambda: "ghcr.io/lambdaclass/ethlambda:devnet4", + constants.LEAN_TYPE.ream: "ghcr.io/reamlabs/ream:latest-devnet4", + constants.LEAN_TYPE.zeam: "blockblaz/zeam:devnet4", + constants.LEAN_TYPE.qlean: "qdrvm/qlean-mini:devnet-4-amd64", + constants.LEAN_TYPE.lantern: "piertwo/lantern:v0.0.4", + constants.LEAN_TYPE.grandine: "sifrai/lean:devnet-4", + constants.LEAN_TYPE.lighthouse: "hopinheimer/lighthouse:latest", + constants.LEAN_TYPE.gean: "ghcr.io/geanlabs/gean:devnet4", + constants.LEAN_TYPE.peam: "", # No published image yet + constants.LEAN_TYPE.nlean: "", # No published image yet +} + # MEV Params MEV_BOOST_PORT = 18550 @@ -73,6 +89,8 @@ DEFAULT_ADDITIONAL_SERVICES = [] ATTR_TO_BE_SKIPPED_AT_ROOT = ( "network_params", "participants", + "lean_participants", + "lean_network_params", "mev_params", "blockscout_params", "dora_params", @@ -137,6 +155,18 @@ def input_parser(plan, input_args): result["zkboost_params"] = get_default_zkboost_params() result["buildoor_params"] = get_default_buildoor_params() + # Lean Ethereum: defaults are empty; users opt in by providing + # `lean_participants:` in their args. Parsed below if present. + result["lean_participants"] = [] + result["lean_network_params"] = default_lean_network_params() + if "lean_participants" in input_args and input_args["lean_participants"]: + result["lean_participants"] = parse_lean_participants( + input_args["lean_participants"] + ) + if "lean_network_params" in input_args: + for k, v in input_args["lean_network_params"].items(): + result["lean_network_params"][k] = v + if constants.NETWORK_NAME.shadowfork in result["network_params"]["network"]: shadow_base = result["network_params"]["network"].split("-shadowfork")[0] result["network_params"][ @@ -2552,3 +2582,87 @@ def get_devnet_modified_images(network_name, default_images): modified_images[client_type] = get_devnet_image_tag(network_name, image) return modified_images + + +# --------------------------------------------------------------------------- +# Lean Ethereum parsing +# --------------------------------------------------------------------------- +# Lean consensus is a standalone, post-quantum-signature consensus stack. It +# does not pair with an EL, has no Engine API / JWT, and uses its own +# genesis pipeline (PK's eth-beacon-genesis leanchain). Lean participants +# therefore live in a separate `lean_participants:` list and run through +# `src/lean/lean_launcher.star`. + + +def default_lean_participant(): + return { + "lean_type": constants.LEAN_TYPE.ethlambda, + "lean_image": "", + "lean_log_level": "", + "lean_extra_params": [], + "lean_extra_env_vars": {}, + "lean_extra_labels": {}, + "lean_min_cpu": 0, + "lean_max_cpu": 0, + "lean_min_mem": 0, + "lean_max_mem": 0, + "count": 1, + "validator_count": 1, + "is_aggregator": False, + "node_selectors": {}, + "tolerations": [], + "prometheus_config": {"scrape_interval": "15s", "labels": {}}, + } + + +def default_lean_network_params(): + # Genesis timing and shape parameters consumed by the Lean genesis tool + # (eth-beacon-genesis leanchain). Keep them flat to mirror the + # `validator-config.yaml.config` block expected by the generator and by + # every Lean client's CLI. + return { + # Seconds added to "now" to compute GENESIS_TIME when the user does + # not pass an absolute genesis_time. 60s gives all containers time to + # boot, mount artifacts, and reach the gossip mesh before slot 0. + "genesis_delay": 60, + # Explicit Unix timestamp; 0 = derive from genesis_delay. + "genesis_time": 0, + # leanSpec ATTESTATION_COMMITTEE_COUNT. 1 = single committee per slot + # (the only configuration covered by the spec tests today). + "attestation_committee_count": 1, + # log_2(active epochs) for the XMSS hash-sig scheme. 18 matches the + # default in lean-quickstart's validator-config.yaml. + "active_epoch": 18, + # Number of validator hash-sig keypairs to assign to each node when + # the participant does not override `validator_count`. + "num_validator_keys_per_node": 1, + # Image overrides for the Lean genesis tooling. Empty = use the + # `DEFAULT_LEAN_GENESIS_GENERATOR_IMAGE` / `DEFAULT_LEAN_HASH_SIG_CLI_IMAGE` + # constants. Override to pin a specific PK genesis-tool commit. + "genesis_generator_image": "", + "hash_sig_cli_image": "", + } + + +def parse_lean_participants(raw_participants): + """Normalize the lean_participants list by filling defaults per-entry.""" + parsed = [] + for raw in raw_participants: + entry = default_lean_participant() + for k, v in raw.items(): + entry[k] = v + # Resolve image: explicit override > registry default. We fail fast + # rather than silently shipping an empty image string downstream + # because Kurtosis's error in that case is opaque ("invalid image: "). + if entry["lean_image"] == "": + default_image = DEFAULT_LEAN_IMAGES.get(entry["lean_type"], "") + if default_image == "": + fail( + "lean_type '{0}' has no default image; please set " + "`lean_image` on this participant.".format(entry["lean_type"]) + ) + entry["lean_image"] = default_image + if entry["count"] < 1: + fail("lean_participants[].count must be >= 1") + parsed.append(entry) + return parsed diff --git a/src/prelaunch_data_generator/lean_genesis/lean_genesis_generator.star b/src/prelaunch_data_generator/lean_genesis/lean_genesis_generator.star new file mode 100644 index 000000000..b8641efa1 --- /dev/null +++ b/src/prelaunch_data_generator/lean_genesis/lean_genesis_generator.star @@ -0,0 +1,378 @@ +""" +Lean Ethereum genesis generation. + +This module orchestrates the post-quantum genesis pipeline used by every +Lean client: + + 1. Generate XMSS attester+proposer keypairs via `blockblaz/hash-sig-cli`. + 2. Render `validator-config.yaml` from the live (Kurtosis-assigned) IPs and + ports of each Lean participant. + 3. Run `ethpandaops/eth-beacon-genesis:pk910-leanchain` to derive + `config.yaml`, `validators.yaml`, `nodes.yaml`, and `genesis.{ssz,json}`. + 4. Post-process: inject GENESIS_VALIDATORS into config.yaml and render + `annotated_validators.yaml` (node-name -> validator-index assignments + with attester/proposer privkey filenames). + +The output is a single files artifact (`lean-genesis-data`) mounted at +`/network-configs` inside every Lean client container, plus a separate +`lean-hash-sig-keys` artifact for the XMSS secret/public keys. This matches +the `lean-quickstart` on-disk layout 1:1 so a client written for +lean-quickstart works under Kurtosis without code changes. +""" + +constants = import_module("../../package_io/constants.star") + +GENESIS_ARTIFACT_NAME = "lean-genesis-data" +HASH_SIG_ARTIFACT_NAME = "lean-hash-sig-keys" + +GENESIS_DIR = "/genesis" +HASH_SIG_DIR = "/hash-sig-keys" + + +def _resolve_images(lean_network_params): + genesis_image = lean_network_params.get("genesis_generator_image", "") + if genesis_image == "": + genesis_image = constants.DEFAULT_LEAN_GENESIS_GENERATOR_IMAGE + hash_sig_image = lean_network_params.get("hash_sig_cli_image", "") + if hash_sig_image == "": + hash_sig_image = constants.DEFAULT_LEAN_HASH_SIG_CLI_IMAGE + return genesis_image, hash_sig_image + + +def _compute_genesis_time(plan, lean_network_params): + """Resolve GENESIS_TIME. + + Explicit `genesis_time` wins; otherwise we ask a busybox shell for + `now() + genesis_delay`. Runs on the Kurtosis backend so the result is + a real clock reading inside the cluster, not the operator's laptop. + """ + explicit = lean_network_params.get("genesis_time", 0) + if explicit != 0: + return str(explicit) + delay = lean_network_params.get("genesis_delay", 60) + result = plan.run_sh( + run="echo -n $(($(date +%s) + {0}))".format(delay), + description="Computing Lean genesis time", + ) + return result.output + + +def _render_validator_config(plan, services_meta, lean_network_params): + """Render validator-config.yaml from per-node IPs / ports / keys. + + `services_meta` is a list of dicts with keys: name, ip_address, quic_port, + metrics_port, api_port, privkey, validator_count, is_aggregator. + """ + template = """shuffle: roundrobin +deployment_mode: kurtosis +config: + activeEpoch: {{.ActiveEpoch}} + keyType: "hash-sig" + attestation_committee_count: {{.AttestationCommitteeCount}} +validators: +{{- range .Validators}} + - name: "{{.name}}" + privkey: "{{.privkey}}" + enrFields: + ip: "{{.ip}}" + quic: {{.quic}} + metricsPort: {{.metricsPort}} + apiPort: {{.apiPort}} + isAggregator: {{.isAggregator}} + count: {{.count}} +{{end}}""" + + validators = [] + for meta in services_meta: + validators.append( + { + "name": meta["name"], + "privkey": meta["privkey"], + "ip": meta["ip_address"], + "quic": meta["quic_port"], + "metricsPort": meta["metrics_port"], + "apiPort": meta["api_port"], + "isAggregator": "true" if meta["is_aggregator"] else "false", + "count": meta["validator_count"], + } + ) + + return plan.render_templates( + config={ + "validator-config.yaml": struct( + template=template, + data={ + "ActiveEpoch": lean_network_params["active_epoch"], + "AttestationCommitteeCount": lean_network_params[ + "attestation_committee_count" + ], + "Validators": validators, + }, + ), + }, + name="lean-validator-config", + description="Rendering Lean validator-config.yaml", + ) + + +def _render_initial_config(plan, genesis_time, lean_network_params, total_validators): + """Render the initial config.yaml that PK's tool consumes. + + PK's tool *rewrites* config.yaml with extra fields after running, but it + still requires the input to declare GENESIS_TIME, ATTESTATION_COMMITTEE_COUNT, + ACTIVE_EPOCH, and VALIDATOR_COUNT — every other Lean client reads + GENESIS_TIME from this file too, so the value must match what we'll embed. + """ + template = """# Genesis Settings +GENESIS_TIME: {{.GenesisTime}} + +# Chain Settings +ATTESTATION_COMMITTEE_COUNT: {{.AttestationCommitteeCount}} + +# Key Settings +ACTIVE_EPOCH: {{.ActiveEpoch}} + +# Validator Settings +VALIDATOR_COUNT: {{.ValidatorCount}} +""" + return plan.render_templates( + config={ + "config.yaml": struct( + template=template, + data={ + "GenesisTime": genesis_time, + "AttestationCommitteeCount": lean_network_params[ + "attestation_committee_count" + ], + "ActiveEpoch": lean_network_params["active_epoch"], + "ValidatorCount": total_validators, + }, + ), + }, + name="lean-initial-config", + description="Rendering Lean initial config.yaml", + ) + + +def _generate_hash_sig_keys(plan, image, num_validators, active_epoch): + """Generate XMSS attester+proposer keypairs. + + `hash-sig-cli generate` writes `validator_N_{attester,proposer}_key_{pk,sk}.ssz` + plus a `validator-keys-manifest.yaml` index. The whole `/hash-sig-keys` + tree becomes the `lean-hash-sig-keys` artifact mounted at the Lean + client's `--hash-sig-keys-dir`. + """ + plan.run_sh( + run=( + "mkdir -p {0} && " + + "hash-sig-cli generate " + + "--num-validators {1} " + + "--log-num-active-epochs {2} " + + "--output-dir {0} " + + "--export-format ssz" + ).format(HASH_SIG_DIR, num_validators, active_epoch), + image=image, + store=[ + StoreSpec(src=HASH_SIG_DIR, name=HASH_SIG_ARTIFACT_NAME), + ], + description="Generating Lean hash-sig validator keys ({0} validators)".format( + num_validators + ), + ) + return HASH_SIG_ARTIFACT_NAME + + +def _run_genesis_tool( + plan, + image, + validator_config_artifact, + initial_config_artifact, +): + """Invoke pk910-leanchain. Outputs config.yaml/validators.yaml/nodes.yaml/genesis.{ssz,json}. + + The tool reads `validator-config.yaml` to figure out per-node ENRs and + validator counts, and writes the canonical genesis bundle. We mount the + pre-rendered config.yaml from `--config-output` and let the tool overwrite + it in place; this lets the tool inject the fork digests and other fields + it computes itself. + """ + plan.run_sh( + run=( + "mkdir -p {0} && " + + "cp /input-validator/validator-config.yaml {0}/validator-config.yaml && " + + "cp /input-config/config.yaml {0}/config.yaml && " + + "/app/eth-genesis-state-generator leanchain " + + "--config {0}/config.yaml " + + "--mass-validators {0}/validator-config.yaml " + + "--state-output {0}/genesis.ssz " + + "--json-output {0}/genesis.json " + + "--nodes-output {0}/nodes.yaml " + + "--validators-output {0}/validators.yaml " + + "--config-output {0}/config.yaml" + ).format(GENESIS_DIR), + image=image, + files={ + "/input-validator": validator_config_artifact, + "/input-config": initial_config_artifact, + }, + # The genesis tool's outputs are merged with the hash-sig keys and + # post-processing additions further down before the final artifact + # is published. + store=[ + StoreSpec(src=GENESIS_DIR, name="lean-genesis-raw"), + ], + description="Running eth-beacon-genesis leanchain", + ) + return "lean-genesis-raw" + + +def _post_process( + plan, + raw_genesis_artifact, + hash_sig_artifact, + validator_config_artifact, + node_key_artifact, +): + """Bundle everything Lean clients need into a single mountable artifact. + + Steps: + * Append GENESIS_VALIDATORS to config.yaml using the dual-key + manifest emitted by hash-sig-cli (attester_key_pubkey_hex + + proposer_key_pubkey_hex per validator). + * Render annotated_validators.yaml mapping node names to validator + indices and their `_attester_` / `_proposer_` privkey file basenames + (ethlambda, lantern, and grandine all parse this exact filename + convention to route keys to attestation vs proposal slots). + * Copy the per-node P2P keys (`.key`) into the same artifact so + every client just mounts `/network-configs` and reads everything it + needs from one place. + + Implemented in shell + yq inside a single busybox-style helper because + Starlark has no yaml/json libs. + """ + return plan.run_sh( + run=""" + set -eu + mkdir -p /out + cp /raw/* /out/ + cp /vc/validator-config.yaml /out/ + cp /node-keys/*.key /out/ + + # Append GENESIS_VALIDATORS to config.yaml (dual-key layout). + manifest=/hash-sig/validator-keys-manifest.yaml + n=$(yq eval '.validators | length' "$manifest") + printf '\\n# Genesis validator public keys (post-quantum hash-sig)\\nGENESIS_VALIDATORS:\\n' >> /out/config.yaml + i=0 + while [ "$i" -lt "$n" ]; do + ah=$(yq eval ".validators[$i].attester_key_pubkey_hex" "$manifest" | sed 's/^0x//') + ph=$(yq eval ".validators[$i].proposer_key_pubkey_hex" "$manifest" | sed 's/^0x//') + printf ' - attestation_pubkey: "%s"\\n proposal_pubkey: "%s"\\n' "$ah" "$ph" >> /out/config.yaml + i=$((i + 1)) + done + + # Render annotated_validators.yaml from validators.yaml (PK output) + # joined with the manifest. Each validator index gets two rows + # (attester + proposer) so clients can route by filename. + : > /out/annotated_validators.yaml + for node in $(yq eval 'keys | .[]' /out/validators.yaml); do + printf '%s:\\n' "$node" >> /out/annotated_validators.yaml + indices=$(yq eval ".\\"$node\\" | .[]" /out/validators.yaml) + if [ -z "$indices" ]; then + printf ' []\\n' >> /out/annotated_validators.yaml + continue + fi + for idx in $indices; do + ah=$(yq eval ".validators[$idx].attester_key_pubkey_hex" "$manifest" | sed 's/^0x//') + ph=$(yq eval ".validators[$idx].proposer_key_pubkey_hex" "$manifest" | sed 's/^0x//') + printf ' - index: %s\\n pubkey_hex: %s\\n privkey_file: validator_%s_attester_key_sk.ssz\\n' "$idx" "$ah" "$idx" >> /out/annotated_validators.yaml + printf ' - index: %s\\n pubkey_hex: %s\\n privkey_file: validator_%s_proposer_key_sk.ssz\\n' "$idx" "$ph" "$idx" >> /out/annotated_validators.yaml + done + done + """, + # mikefarah/yq image ships yq + busybox; we don't need anything else. + image="mikefarah/yq:4", + files={ + "/raw": raw_genesis_artifact, + "/hash-sig": hash_sig_artifact, + "/vc": validator_config_artifact, + "/node-keys": node_key_artifact, + }, + store=[ + StoreSpec(src="/out", name=GENESIS_ARTIFACT_NAME), + ], + description="Post-processing Lean genesis (GENESIS_VALIDATORS + annotated_validators.yaml)", + ).files_artifacts[0] + + +def generate(plan, services_meta, lean_network_params, node_key_artifact): + """Top-level entrypoint. + + Args: + plan: Kurtosis plan. + services_meta: list of dicts (one per node) with: name, ip_address, + quic_port, metrics_port, api_port, privkey (hex string), + validator_count, is_aggregator. The caller (lean_launcher) builds + this list after Kurtosis has assigned IPs to the placeholder + services. + lean_network_params: validated `lean_network_params` block. + node_key_artifact: files artifact holding `.key` ASCII-hex P2P + secrets (one per node). + + Returns: + struct(genesis_artifact = , hash_sig_artifact = , + genesis_time = ). + """ + total_validators = 0 + for meta in services_meta: + total_validators += meta["validator_count"] + + if total_validators < 1: + fail( + "Lean genesis requires at least one validator across all " + "lean_participants (got 0)." + ) + + genesis_image, hash_sig_image = _resolve_images(lean_network_params) + genesis_time = _compute_genesis_time(plan, lean_network_params) + + hash_sig_artifact = _generate_hash_sig_keys( + plan, + hash_sig_image, + total_validators, + lean_network_params["active_epoch"], + ) + + validator_config_artifact = _render_validator_config( + plan, + services_meta, + lean_network_params, + ) + + initial_config_artifact = _render_initial_config( + plan, + genesis_time, + lean_network_params, + total_validators, + ) + + raw_genesis_artifact = _run_genesis_tool( + plan, + genesis_image, + validator_config_artifact, + initial_config_artifact, + ) + + final_artifact = _post_process( + plan, + raw_genesis_artifact, + hash_sig_artifact, + validator_config_artifact, + node_key_artifact, + ) + + return struct( + genesis_artifact=final_artifact, + hash_sig_artifact=hash_sig_artifact, + genesis_time=genesis_time, + total_validators=total_validators, + ) diff --git a/src/prelaunch_data_generator/lean_genesis/p2p_keys_generator.star b/src/prelaunch_data_generator/lean_genesis/p2p_keys_generator.star new file mode 100644 index 000000000..edc93affd --- /dev/null +++ b/src/prelaunch_data_generator/lean_genesis/p2p_keys_generator.star @@ -0,0 +1,61 @@ +""" +P2P key generation for Lean consensus nodes. + +Each Lean node needs a 32-byte libp2p identity key (the secp256k1 secret that +derives the peer ID and ENR). lean-quickstart writes one such key per node +into `.key` as ASCII hex. We reproduce that exact layout so the +genesis tool and every Lean client client-cmds/-cmd.sh contract +matches what they receive at runtime. +""" + +OPENSSL_IMAGE = "alpine/openssl" + + +def generate_node_keys(plan, node_names): + """Generate one 32-byte hex P2P key per node. + + The keys are written to `/keys/.key` inside the OpenSSL helper + container and exported as a single Kurtosis files artifact (`lean-node-keys`) + so every Lean client mounts the same directory and reads its own key by + name. Returns (artifact_name, {node_name: hex_string}). + + The hex strings are also returned in-memory because the genesis tool needs + them to compute peer IDs / ENRs for `nodes.yaml` *before* the Lean clients + are started. + """ + # `tr -d` strips OpenSSL's trailing newline; without it `nodes.yaml`'s + # peer-id derivation downstream sees an extra byte and computes the wrong + # libp2p identity. + script_parts = ["set -eu", "mkdir -p /keys"] + for name in node_names: + script_parts.append( + "openssl rand -hex 32 | tr -d '\\n' > /keys/{0}.key".format(name) + ) + # Dump everything to stdout so we can capture the keys without a second + # round-trip. `printf '%s=%s\\n'` keeps the parser trivial. + for name in node_names: + script_parts.append( + "printf '%s=%s\\n' '{0}' \"$(cat /keys/{0}.key)\"".format(name) + ) + script = "\n".join(script_parts) + + result = plan.run_sh( + run=script, + image=OPENSSL_IMAGE, + store=[ + StoreSpec(src="/keys", name="lean-node-keys"), + ], + description="Generating Lean P2P node keys", + ) + + keys_by_name = {} + for line in result.output.strip().split("\n"): + if "=" not in line: + continue + name, hex_value = line.split("=", 1) + keys_by_name[name.strip()] = hex_value.strip() + + return struct( + artifact_name=result.files_artifacts[0], + keys=keys_by_name, + ) From 0d894ba8424d4b060b5bdfa505152fcb144f63c9 Mon Sep 17 00:00:00 2001 From: ilitteri Date: Wed, 13 May 2026 16:27:06 -0300 Subject: [PATCH 02/25] Fix Lean integration: Starlark string syntax, sanity_check, mounts Three bug fixes surfaced by running kurtosis against a minimal lean_participants config: 1. Starlark doesn't support implicit string-literal concatenation. Two adjacent string literals across lines parsed cleanly under black (used by kurtosis lint) but failed the Starlark interpreter. Use explicit "+" in the input_parser fail() and lean_genesis_generator fail(). 2. sanity_check rejected lean_participants and lean_network_params as unknown root keys. Register both in ADDITIONAL_CATEGORY_PARAMS so the catch-all root validator accepts them. Per-entry validation stays in the Lean input parser (DEFAULT_LEAN_IMAGES + parse_lean_participants). 3. Mount overlap: GENESIS_MOUNT (/network-configs) and HASH_SIG_MOUNT (/network-configs/hash-sig-keys) cannot both be Kurtosis file artifact mountpoints since Kurtosis forbids nested mounts. Bundle hash-sig keys into the same artifact during the genesis post-process step and drop the separate HASH_SIG_MOUNT entry from each client's ServiceConfig. Validated up to the point where the existing EL/CL pipeline starts image pulls; Kurtosis CLI v1.18.1 hangs there independently of these changes (upstream issue, fixed in v1.18.2). --- src/lean/ethlambda/ethlambda_launcher.star | 5 ++++- src/lean/ream/ream_launcher.star | 1 - src/lean/zeam/zeam_launcher.star | 1 - src/package_io/input_parser.star | 6 ++++-- src/package_io/sanity_check.star | 6 ++++++ .../lean_genesis/lean_genesis_generator.star | 7 +++++-- 6 files changed, 19 insertions(+), 7 deletions(-) diff --git a/src/lean/ethlambda/ethlambda_launcher.star b/src/lean/ethlambda/ethlambda_launcher.star index 814ba3751..5bd721d38 100644 --- a/src/lean/ethlambda/ethlambda_launcher.star +++ b/src/lean/ethlambda/ethlambda_launcher.star @@ -114,10 +114,13 @@ def start(plan, node, service, genesis_artifact, hash_sig_artifact): "{0} 2>&1 | tee -a {1}".format(full_cmd, log_file), ], ports=lean_shared.lean_port_specs(), + # The hash-sig keys are bundled inside the genesis artifact under + # ./hash-sig-keys (see lean_genesis_generator._post_process); Kurtosis + # forbids overlapping file artifact mounts so we only mount the + # genesis bundle here and let HASH_SIG_MOUNT resolve transparently. files={ NODE_KEY_MOUNT: node["_p2p_keys_artifact"], GENESIS_MOUNT: genesis_artifact, - HASH_SIG_MOUNT: hash_sig_artifact, }, env_vars=env_vars, labels=node["extra_labels"], diff --git a/src/lean/ream/ream_launcher.star b/src/lean/ream/ream_launcher.star index 718042b71..b7e5539e8 100644 --- a/src/lean/ream/ream_launcher.star +++ b/src/lean/ream/ream_launcher.star @@ -100,7 +100,6 @@ def start(plan, node, service, genesis_artifact, hash_sig_artifact): files={ NODE_KEY_MOUNT: node["_p2p_keys_artifact"], GENESIS_MOUNT: genesis_artifact, - HASH_SIG_MOUNT: hash_sig_artifact, }, env_vars=env_vars, labels=node["extra_labels"], diff --git a/src/lean/zeam/zeam_launcher.star b/src/lean/zeam/zeam_launcher.star index 2775997e7..e66036173 100644 --- a/src/lean/zeam/zeam_launcher.star +++ b/src/lean/zeam/zeam_launcher.star @@ -86,7 +86,6 @@ def start(plan, node, service, genesis_artifact, hash_sig_artifact): files={ NODE_KEY_MOUNT: node["_p2p_keys_artifact"], GENESIS_MOUNT: genesis_artifact, - HASH_SIG_MOUNT: hash_sig_artifact, }, env_vars=node["extra_env_vars"], labels=node["extra_labels"], diff --git a/src/package_io/input_parser.star b/src/package_io/input_parser.star index 87e545279..c3d24b2fa 100644 --- a/src/package_io/input_parser.star +++ b/src/package_io/input_parser.star @@ -2658,8 +2658,10 @@ def parse_lean_participants(raw_participants): default_image = DEFAULT_LEAN_IMAGES.get(entry["lean_type"], "") if default_image == "": fail( - "lean_type '{0}' has no default image; please set " - "`lean_image` on this participant.".format(entry["lean_type"]) + ( + "lean_type '{0}' has no default image; please set " + + "`lean_image` on this participant." + ).format(entry["lean_type"]) ) entry["lean_image"] = default_image if entry["count"] < 1: diff --git a/src/package_io/sanity_check.star b/src/package_io/sanity_check.star index 8af4d1ab0..9cead0f06 100644 --- a/src/package_io/sanity_check.star +++ b/src/package_io/sanity_check.star @@ -487,6 +487,12 @@ ADDITIONAL_SERVICES_PARAMS = [ ] ADDITIONAL_CATEGORY_PARAMS = { + # Lean Ethereum participants: validated structurally inside the Lean + # input parser (see DEFAULT_LEAN_IMAGES + parse_lean_participants in + # input_parser.star), so we register the root keys here as opaque and + # let the parser raise on bad per-entry fields. + "lean_participants": "", + "lean_network_params": "", "wait_for_finalization": "", "global_log_level": "", "snooper_enabled": "", diff --git a/src/prelaunch_data_generator/lean_genesis/lean_genesis_generator.star b/src/prelaunch_data_generator/lean_genesis/lean_genesis_generator.star index b8641efa1..267ce9d34 100644 --- a/src/prelaunch_data_generator/lean_genesis/lean_genesis_generator.star +++ b/src/prelaunch_data_generator/lean_genesis/lean_genesis_generator.star @@ -253,10 +253,13 @@ def _post_process( return plan.run_sh( run=""" set -eu - mkdir -p /out + mkdir -p /out /out/hash-sig-keys cp /raw/* /out/ cp /vc/validator-config.yaml /out/ cp /node-keys/*.key /out/ + # Bundle hash-sig keys into the same artifact (nested file artifact + # mounts can't overlap in Kurtosis, so we ship a single tree). + cp -r /hash-sig/. /out/hash-sig-keys/ # Append GENESIS_VALIDATORS to config.yaml (dual-key layout). manifest=/hash-sig/validator-keys-manifest.yaml @@ -329,7 +332,7 @@ def generate(plan, services_meta, lean_network_params, node_key_artifact): if total_validators < 1: fail( "Lean genesis requires at least one validator across all " - "lean_participants (got 0)." + + "lean_participants (got 0).", ) genesis_image, hash_sig_image = _resolve_images(lean_network_params) From 160081eb8d3757dee81502bc9cbfcc0dc6e64a56 Mon Sep 17 00:00:00 2001 From: ilitteri Date: Wed, 13 May 2026 16:58:21 -0300 Subject: [PATCH 03/25] Wire lean-only mode end-to-end and fix Lean pipeline runtime issues MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Lean consensus is fully standalone (no Engine API, no EL counterpart), so running a Lean network alongside an Eth1 EL/CL pair just to satisfy downstream consumers is the wrong contract. Detect the lean-only case (participants: [] && lean_participants: [...]) early in main.star and short-circuit straight into lean_launcher.launch — the Eth1 EL/CL flow is skipped entirely. Guards added to input_parser for the two EL/CL preconditions that crashed with `participants: []`: * Fulu/PeerDAS validation only runs when at least one EL/CL participant is configured. * First-participant-must-have-EL check only runs when participants[0] actually exists. Several pipeline-runtime fixes surfaced while iterating against `kurtosis run`: 1. `plan.run_sh(...).output` is a Kurtosis runtime future, not a Starlark string. Don't try to index it in Starlark (dict lookups, etc.). The P2P key generator stops returning a value dict and just exports the artifact; the validator-config render reads keys inside its own shell. 2. Kurtosis service names must match RFC 1035, so the per-node `_` (lean-quickstart convention) is translated to `lean--` for the service name. The internal node_name keeps the underscore for --node-id / validator-config compatibility. 3. add_service rejects calling the same name twice (force_update or not). Swap the placeholder-then-replace pattern for the pq-devnet-package style: add_service once with all IP-independent mounts (P2P + hash-sig keys), then plan.exec each text genesis file in via `cat <__` placeholders + real IPs, then sed-substitute privkeys from the keys artifact. 7. hash-sig-cli binary lives at /usr/local/bin/hashsig in the blockblaz/hash-sig-cli:latest image, not `hash-sig-cli`. 8. The post-process step (GENESIS_VALIDATORS injection + annotated_validators.yaml render) is now a Python script rendered as a separate artifact and executed by a minimal shell wrapper — busybox sh in common yq/alpine images choked on heredocs with embedded interpreters. PyYAML installed via pip (apk's py3-yaml targets alpine's system python, not the python:3-alpine bundled one). 9. PyYAML 1.1 parses unquoted 0x-hex tokens as int; the post-process script normalises both int and str forms before string ops. 10. Lean and EL genesis pipelines share no artifacts; the hash-sig keys are bundled into the same `lean-genesis-data` artifact as config.yaml et al. so a single Kurtosis files mount covers `/network-configs` + `/network-configs/hash-sig-keys`. Kurtosis forbids overlapping mounts. Validated end-to-end with kurtosis run: two `lean-ethlambda-{0,1}` services come up in lean-only mode, peer over QUIC, exchange status messages, expose `GET /lean/v0/health` returning HTTP 200, and serve `lean_*` Prometheus metrics. --- docs/lean-consensus.md | 39 ++- main.star | 30 ++ src/lean/ethlambda/ethlambda_launcher.star | 221 +++++++------ src/lean/lean_launcher.star | 45 ++- src/lean/lean_shared.star | 26 ++ src/lean/ream/ream_launcher.star | 149 +++++---- src/lean/zeam/zeam_launcher.star | 143 ++++----- src/package_io/input_parser.star | 20 +- .../lean_genesis/lean_genesis_generator.star | 299 +++++++++++++----- .../lean_genesis/p2p_keys_generator.star | 65 ++-- 10 files changed, 641 insertions(+), 396 deletions(-) diff --git a/docs/lean-consensus.md b/docs/lean-consensus.md index 1879c90f2..c835b7528 100644 --- a/docs/lean-consensus.md +++ b/docs/lean-consensus.md @@ -57,14 +57,12 @@ nodes by URL. ## Quick start -Enable Lean by populating `lean_participants:` in your args file: +Lean consensus is fully standalone — no Engine API, no EL counterpart. +A Lean network configuration therefore contains ONLY `lean_participants:` +(set `participants: []` to skip the Eth1 EL/CL flow entirely): ```yaml -participants: - - el_type: geth - cl_type: lighthouse - count: 1 - validator_count: 0 # No EL/CL validators; we just need one EL+CL pair. +participants: [] lean_participants: - lean_type: ethlambda @@ -88,12 +86,13 @@ The Lean pipeline produces 4 services named `lean-ethlambda_0` … | 5052 | REST API (`GET /lean/v0/health`, fork choice, …) | | 5054 | Prometheus metrics (`/metrics`) | -> **V1 limitation — at least one EL/CL participant required.** The existing -> EL/CL flow assumes at least one EL context exists (e.g. for the -> tx-spammer / dora target). Until lean-only mode is plumbed through every -> downstream consumer, keep one minimal `participants[]` entry with -> `validator_count: 0`. A follow-up will detect `participants: []` and -> short-circuit the EL/CL pipeline. +### Mixed mode (Lean + EL/CL) + +You can also run Lean alongside the existing Eth1 EL/CL network in the same +enclave. Both pipelines run independently — there is no cross-talk between +them. Add `participants:` entries as you normally would and keep +`lean_participants:` populated. This is useful for side-by-side benchmarking +and observability dashboards that scrape both. --- @@ -205,20 +204,20 @@ operator-facing dashboards and probes work across deployments. ## Limitations & follow-ups -1. **At least one EL/CL participant required.** The existing main flow - assumes `all_el_contexts[0]` exists for the tx-spammer / dora target. - Until lean-only mode is wired through every downstream consumer, keep - one minimal `participants[]` entry with `validator_count: 0`. -2. **Lean nodes are not yet scraped by Prometheus.** The `metrics_info` +1. **Lean nodes are not yet scraped by Prometheus.** The `metrics_info` struct is populated on every `lean_context`, but the prometheus launcher isn't yet wired to discover Lean nodes. Operators scraping Lean nodes today should hit them by service name directly. -3. **`hash-sig-cli` image is pinned to `:latest`.** Pinning to a SHA is +2. **`hash-sig-cli` image is pinned to `:latest`.** Pinning to a SHA is left to a follow-up; override via `lean_network_params.hash_sig_cli_image`. -4. **No Lean-specific dashboards.** Existing Grafana dashboards assume the +3. **No Lean-specific dashboards.** Existing Grafana dashboards assume the Ethereum CL schema. Lean dashboards (`lean_head_slot`, `lean_state_transition_time_seconds`, etc.) need a separate dashboard pack. -5. **No checkpoint sync.** Per-participant `checkpoint_sync_url` parsing +4. **No checkpoint sync.** Per-participant `checkpoint_sync_url` parsing is not yet wired through to the per-client launchers. +5. **Mixed-mode auxiliary services.** When Lean is run alongside EL/CL, + the existing Eth1 auxiliary services (tx-fuzz, dora, etc.) only see + the EL/CL participants. Wiring them to also point at Lean nodes is a + follow-up. diff --git a/main.star b/main.star index dcbbd1fe8..d13200958 100644 --- a/main.star +++ b/main.star @@ -89,6 +89,36 @@ def run(plan, args={}): num_participants = len(args_with_right_defaults.participants) network_params = args_with_right_defaults.network_params + # Lean-only mode: when the operator configured `lean_participants:` but + # no Eth1 `participants:` entries, run ONLY the Lean pipeline. Lean + # consensus is fully standalone (no Engine API, no EL counterpart), so + # spinning up an EL+CL pair as a "placeholder" would just waste resources + # and confuse downstream services that try to call the Engine API. + # The lean_launcher returns the per-node contexts; downstream consumers + # (prometheus, grafana, dora) can be wired through in a follow-up. + if num_participants == 0 and args_with_right_defaults.lean_participants: + plan.print( + "Lean-only mode: {0} lean participant entries, 0 EL/CL participants".format( + len(args_with_right_defaults.lean_participants) + ) + ) + lean_contexts = lean_launcher.launch( + plan, + args_with_right_defaults.lean_participants, + args_with_right_defaults.lean_network_params, + ) + return struct( + grafana_info=None, + blockscout_sc_verif_url=None, + all_participants=[], + lean_participants=lean_contexts, + pre_funded_accounts={}, + network_params=network_params, + network_id=network_params.network_id, + final_genesis_timestamp=None, + genesis_validators_root=None, + ) + # Detect the backend type early - needed for binary injection validation detected_backend = plan.get_cluster_type() diff --git a/src/lean/ethlambda/ethlambda_launcher.star b/src/lean/ethlambda/ethlambda_launcher.star index 5bd721d38..987bbf6d9 100644 --- a/src/lean/ethlambda/ethlambda_launcher.star +++ b/src/lean/ethlambda/ethlambda_launcher.star @@ -6,19 +6,23 @@ Translates the Lean pipeline's per-node record into the ethlambda CLI surface lean-quickstart `client-cmds/ethlambda-cmd.sh`). Lifecycle (called by ../lean_launcher.star): - 1. `initialize()` - add a Kurtosis service holding the P2P keys + the - pre-generated hash-sig keys (these don't depend on the node IP). The - container runs `tail -f` on a log file so it stays alive while the - genesis pipeline computes things downstream that DO need the IP. + 1. `initialize()` - add a Kurtosis service mounting both the P2P keys + artifact and the hash-sig (XMSS) keys artifact (neither depends on the + node IP). The container runs `tail -f` on a log file so Kurtosis keeps + the service alive while the genesis pipeline computes things downstream + that DO need the IP. 2. `start()` - after the genesis tool has produced the text artifacts (config.yaml, annotated_validators.yaml, nodes.yaml, validator-config.yaml), - stage them inside the running container via `plan.exec` and launch - the ethlambda binary as a backgrounded `nohup` process. - -Why two phases rather than one `add_service` with everything mounted? Because -the genesis tool needs every node's IP to render `nodes.yaml`, and Kurtosis -only assigns IPs after `add_service`. Recreating services to swap mounts -would also reshuffle IPs and invalidate the ENRs we just embedded. + stage each text file inside the running container via `plan.exec` + (Kurtosis doesn't let us mount new file artifacts onto a running + service), then launch the ethlambda binary as a backgrounded `nohup` + process. We use `plan.exec` for both steps rather than a second + `add_service`, because the Starlark validator rejects two + `add_service` calls with the same name even when `force_update=True`. + +The hash-sig keys, P2P keys, and a stable directory layout under +`/network-configs/` are established by `initialize`; `start` only adds the +files whose contents depend on the live IP allocation. """ constants = import_module("../../package_io/constants.star") @@ -31,132 +35,143 @@ HASH_SIG_MOUNT = GENESIS_MOUNT + "/hash-sig-keys" DATA_DIR = "/data" NODE_KEY_MOUNT = constants.LEAN_NODE_KEY_MOUNTPOINT_ON_CLIENTS +GENESIS_TEXT_FILES = [ + "config.yaml", + "annotated_validators.yaml", + "nodes.yaml", + "validator-config.yaml", +] -def initialize(plan, node, p2p_keys_artifact): - """Phase 1: stand the placeholder service up so Kurtosis assigns an IP.""" - cfg = ServiceConfig( - image=node["image"], - # Override the image ENTRYPOINT so the container doesn't try to start - # ethlambda with no flags before we've written the genesis files in. - entrypoint=["/bin/sh", "-c"], - cmd=lean_shared.lean_tail_logs_cmd(node["service_name"])[2:], - ports=lean_shared.lean_port_specs(), - files={ - NODE_KEY_MOUNT: p2p_keys_artifact, - }, - env_vars=node["extra_env_vars"], - labels=node["extra_labels"], - min_cpu=node["min_cpu"], - max_cpu=node["max_cpu"], - min_memory=node["min_mem"], - max_memory=node["max_mem"], - node_selectors=node["node_selectors"], - tolerations=node["tolerations"], + +def initialize(plan, node, p2p_keys_artifact, hash_sig_artifact): + """Phase 1: stand the placeholder service up so Kurtosis assigns an IP. + + Both the P2P keys and the XMSS keys are mounted here because neither + depends on the IP allocation. Genesis text files (config.yaml, + nodes.yaml, etc.) are staged later via `plan.exec` once the genesis + tool has computed them against the live IPs. + """ + cfg_kwargs = lean_shared.common_cfg_kwargs(node) + cfg_kwargs.update( + { + "image": node["image"], + # Override the image ENTRYPOINT so the container doesn't try to + # start ethlambda with no flags before we've written the genesis + # files in. + "entrypoint": ["/bin/sh", "-c"], + "cmd": lean_shared.lean_tail_logs_cmd(node["service_name"])[2:], + "files": { + NODE_KEY_MOUNT: p2p_keys_artifact, + HASH_SIG_MOUNT: hash_sig_artifact, + }, + } ) - return plan.add_service(node["service_name"], cfg) + return plan.add_service(node["service_name"], ServiceConfig(**cfg_kwargs)) def start(plan, node, service, genesis_artifact, hash_sig_artifact): - """Phase 3: mount genesis files and start the ethlambda binary. + """Phase 3: stage genesis text files into the running container and + launch the ethlambda binary as a backgrounded process. - We re-issue add_service with force_update=True so we can mount the - genesis + hash-sig artifacts (Kurtosis preserves the IP because the - service name and port assignments are unchanged; ENR fields embedded - in the genesis bundle therefore remain valid). + `hash_sig_artifact` is unused here because it was already mounted in + `initialize()` — we keep the parameter for shape parity with the other + Lean clients. """ service_name = service.name + log_file = lean_shared.lean_log_file_path(service_name) + # Stage each text genesis file inside the running container. We read + # the artifact's contents via plan.run_sh (the output is a Kurtosis + # runtime future) and then plan.exec a `cat > path` heredoc that + # carries the future as a string argument — Kurtosis resolves the + # future at apply time so the literal file contents land in the + # target. mkdir is idempotent. + plan.exec( + service_name=service_name, + recipe=ExecRecipe( + command=["/bin/sh", "-c", "mkdir -p {0}".format(GENESIS_MOUNT)], + ), + description="Preparing genesis mount on {0}".format(service_name), + ) + for filename in GENESIS_TEXT_FILES: + read = plan.run_sh( + run="cat /src/{0}".format(filename), + files={"/src": genesis_artifact}, + description="Reading {0} for {1}".format(filename, service_name), + ) + plan.exec( + service_name=service_name, + recipe=ExecRecipe( + command=[ + "/bin/sh", + "-c", + "cat > {0}/{1} <<'ETHLAMBDA_EOF'\n{2}\nETHLAMBDA_EOF".format( + GENESIS_MOUNT, filename, read.output, + ), + ], + ), + description="Staging {0} into {1}".format(filename, service_name), + ) + + # Build and start the ethlambda command. The placeholder `tail -f` + # keeps the container alive, so we run ethlambda as a nohup + # background process and let its stdout/stderr flow into the same + # log file the tail is already watching. cmd_parts = [ ENTRYPOINT, - "--genesis", - "{0}/config.yaml".format(GENESIS_MOUNT), - "--validators", - "{0}/annotated_validators.yaml".format(GENESIS_MOUNT), - "--bootnodes", - "{0}/nodes.yaml".format(GENESIS_MOUNT), - "--validator-config", - "{0}/validator-config.yaml".format(GENESIS_MOUNT), - "--hash-sig-keys-dir", - HASH_SIG_MOUNT, - "--data-dir", - DATA_DIR, - "--gossipsub-port", - str(constants.LEAN_QUIC_PORT_NUM), - "--node-id", - node["node_name"], - "--node-key", - "{0}/{1}.key".format(NODE_KEY_MOUNT, node["node_name"]), - "--http-address", - "0.0.0.0", - "--api-port", - str(constants.LEAN_API_PORT_NUM), - "--metrics-port", - str(constants.LEAN_METRICS_PORT_NUM), + "--genesis", "{0}/config.yaml".format(GENESIS_MOUNT), + "--validators", "{0}/annotated_validators.yaml".format(GENESIS_MOUNT), + "--bootnodes", "{0}/nodes.yaml".format(GENESIS_MOUNT), + "--validator-config", "{0}/validator-config.yaml".format(GENESIS_MOUNT), + "--hash-sig-keys-dir", HASH_SIG_MOUNT, + "--data-dir", DATA_DIR, + "--gossipsub-port", str(constants.LEAN_QUIC_PORT_NUM), + "--node-id", node["node_name"], + "--node-key", "{0}/{1}.key".format(NODE_KEY_MOUNT, node["node_name"]), + "--http-address", "0.0.0.0", + "--api-port", str(constants.LEAN_API_PORT_NUM), + "--metrics-port", str(constants.LEAN_METRICS_PORT_NUM), ] if node["is_aggregator"]: cmd_parts.append("--is-aggregator") for extra in node["extra_params"]: cmd_parts.append(extra) - log_file = lean_shared.lean_log_file_path(service_name) - full_cmd = " ".join(cmd_parts) - - env_vars = dict(node["extra_env_vars"]) + rust_log = "" if node["log_level"] != "": - env_vars["RUST_LOG"] = node["log_level"] - - new_cfg = ServiceConfig( - image=node["image"], - entrypoint=["/bin/sh", "-c"], - # `tee -a` keeps a log file readable by Kurtosis's `service logs` - # while also surfacing the binary's stdout/stderr live. - cmd=[ - "{0} 2>&1 | tee -a {1}".format(full_cmd, log_file), - ], - ports=lean_shared.lean_port_specs(), - # The hash-sig keys are bundled inside the genesis artifact under - # ./hash-sig-keys (see lean_genesis_generator._post_process); Kurtosis - # forbids overlapping file artifact mounts so we only mount the - # genesis bundle here and let HASH_SIG_MOUNT resolve transparently. - files={ - NODE_KEY_MOUNT: node["_p2p_keys_artifact"], - GENESIS_MOUNT: genesis_artifact, - }, - env_vars=env_vars, - labels=node["extra_labels"], - min_cpu=node["min_cpu"], - max_cpu=node["max_cpu"], - min_memory=node["min_mem"], - max_memory=node["max_mem"], - node_selectors=node["node_selectors"], - tolerations=node["tolerations"], + rust_log = "RUST_LOG='{0}' ".format(node["log_level"]) + + nohup_cmd = "nohup {0}{1} >> {2} 2>&1 &".format( + rust_log, + " ".join(cmd_parts), + log_file, ) - new_service = plan.add_service( - name=service_name, - config=new_cfg, - force_update=True, + plan.exec( + service_name=service_name, + recipe=ExecRecipe( + command=["/bin/sh", "-c", nohup_cmd], + ), + description="Starting ethlambda binary on {0}".format(service_name), ) api_url = "http://{0}:{1}".format( - new_service.ip_address, - constants.LEAN_API_PORT_NUM, + service.ip_address, constants.LEAN_API_PORT_NUM, ) metrics_url = "http://{0}:{1}/metrics".format( - new_service.ip_address, - constants.LEAN_METRICS_PORT_NUM, + service.ip_address, constants.LEAN_METRICS_PORT_NUM, ) return lean_context.new_lean_context( client_name=constants.LEAN_TYPE.ethlambda, - service_name=new_service.name, - ip_address=new_service.ip_address, + service_name=service_name, + ip_address=service.ip_address, quic_port=constants.LEAN_QUIC_PORT_NUM, api_port=constants.LEAN_API_PORT_NUM, metrics_port=constants.LEAN_METRICS_PORT_NUM, api_url=api_url, metrics_url=metrics_url, metrics_info={ - "name": new_service.name, + "name": service_name, "url": metrics_url, "path": "/metrics", "config": node["prometheus_config"], diff --git a/src/lean/lean_launcher.star b/src/lean/lean_launcher.star index 70a42db42..1697c7c42 100644 --- a/src/lean/lean_launcher.star +++ b/src/lean/lean_launcher.star @@ -71,11 +71,17 @@ def launch(plan, lean_participants, lean_network_params): for _ in range(participant["count"]): idx = type_counters.get(lean_type, 0) type_counters[lean_type] = idx + 1 + # `node_name` follows lean-quickstart's `_` + # convention (passed as --node-id and used in validator-config.yaml). + # Kurtosis service names, however, must match RFC 1035 — lowercase + # letters/digits/hyphens only — so we translate the underscore to + # a hyphen for the Kurtosis-facing name. node_name = "{0}_{1}".format(lean_type, idx) + service_name = "lean-{0}-{1}".format(lean_type, idx) expanded.append( { "node_name": node_name, - "service_name": "lean-{0}".format(node_name), + "service_name": service_name, "lean_type": lean_type, "image": participant["lean_image"], "validator_count": participant.get( @@ -104,13 +110,27 @@ def launch(plan, lean_participants, lean_network_params): ) node_names = [n["node_name"] for n in expanded] - keys_result = p2p_keys.generate_node_keys(plan, node_names) + keys_artifact = p2p_keys.generate_node_keys(plan, node_names) + + # Compute the total validator count up front so we can generate the + # hash-sig keys before any service is added. Hash-sig keys don't depend + # on the node IP, so we can mount them directly at initialize() time + # and avoid having to re-mount any artifact later (Kurtosis rejects + # two add_service calls with the same name). + total_validators = 0 + for node in expanded: + total_validators += node["validator_count"] + hash_sig_artifact = lean_genesis.generate_hash_sig_keys( + plan, + lean_network_params, + total_validators, + ) - # Stash the P2P keys artifact on each node record so per-client launchers - # can re-mount it during the `start()` phase (where we re-issue the - # add_service with the full mounts). + # Stash both artifacts on each node record so per-client launchers + # have access during initialize(). for node in expanded: - node["_p2p_keys_artifact"] = keys_result.artifact_name + node["_p2p_keys_artifact"] = keys_artifact + node["_hash_sig_artifact"] = hash_sig_artifact # Phase 1: initialise placeholder services so Kurtosis assigns IPs. services = [] @@ -119,12 +139,17 @@ def launch(plan, lean_participants, lean_network_params): service = launcher.initialize( plan, node, - keys_result.artifact_name, + keys_artifact, + hash_sig_artifact, ) services.append((node, service)) # Phase 2: render the validator-config.yaml and run the genesis tool now - # that every service has an IP. + # that every service has an IP. Note that we do NOT pass per-node + # privkey values from Starlark — `plan.run_sh(...).output` is a runtime + # future, so individual keys can't be looked up here. The genesis + # pipeline reads `.key` directly from the keys artifact inside + # its own shell. services_meta = [] for node, service in services: services_meta.append( @@ -134,7 +159,6 @@ def launch(plan, lean_participants, lean_network_params): "quic_port": constants.LEAN_QUIC_PORT_NUM, "metrics_port": constants.LEAN_METRICS_PORT_NUM, "api_port": constants.LEAN_API_PORT_NUM, - "privkey": keys_result.keys[node["node_name"]], "validator_count": node["validator_count"], "is_aggregator": node["is_aggregator"], } @@ -144,7 +168,8 @@ def launch(plan, lean_participants, lean_network_params): plan, services_meta, lean_network_params, - keys_result.artifact_name, + keys_artifact, + hash_sig_artifact, ) # Phase 3: hand off to each per-client launcher to mount the genesis bundle diff --git a/src/lean/lean_shared.star b/src/lean/lean_shared.star index 9eaf830df..ce2c1885e 100644 --- a/src/lean/lean_shared.star +++ b/src/lean/lean_shared.star @@ -58,3 +58,29 @@ def lean_tail_logs_cmd(service_name): """ log_file = lean_log_file_path(service_name) return ["/bin/sh", "-c", "touch {0} && tail -f {0}".format(log_file)] + + +def common_cfg_kwargs(node): + """ServiceConfig kwargs shared between per-client initialize() and start(). + + Kurtosis rejects memory/cpu values of 0 (it expects "unset" via the + *absence* of the kwarg, not via a 0 sentinel). We omit those keys here + when the participant didn't set them. Per-client launchers add image, + cmd, entrypoint, and files on top. + """ + kwargs = { + "ports": lean_port_specs(), + "env_vars": node["extra_env_vars"], + "labels": node["extra_labels"], + "node_selectors": node["node_selectors"], + "tolerations": node["tolerations"], + } + if node["min_cpu"] > 0: + kwargs["min_cpu"] = node["min_cpu"] + if node["max_cpu"] > 0: + kwargs["max_cpu"] = node["max_cpu"] + if node["min_mem"] > 0: + kwargs["min_memory"] = node["min_mem"] + if node["max_mem"] > 0: + kwargs["max_memory"] = node["max_mem"] + return kwargs diff --git a/src/lean/ream/ream_launcher.star b/src/lean/ream/ream_launcher.star index b7e5539e8..0be4b0900 100644 --- a/src/lean/ream/ream_launcher.star +++ b/src/lean/ream/ream_launcher.star @@ -31,110 +31,123 @@ DATA_DIR = "/data" NODE_KEY_MOUNT = constants.LEAN_NODE_KEY_MOUNTPOINT_ON_CLIENTS -def initialize(plan, node, p2p_keys_artifact): - cfg = ServiceConfig( - image=node["image"], - entrypoint=["/bin/sh", "-c"], - cmd=lean_shared.lean_tail_logs_cmd(node["service_name"])[2:], - ports=lean_shared.lean_port_specs(), - files={NODE_KEY_MOUNT: p2p_keys_artifact}, - env_vars=node["extra_env_vars"], - labels=node["extra_labels"], - min_cpu=node["min_cpu"], - max_cpu=node["max_cpu"], - min_memory=node["min_mem"], - max_memory=node["max_mem"], - node_selectors=node["node_selectors"], - tolerations=node["tolerations"], +GENESIS_TEXT_FILES = [ + "config.yaml", + "annotated_validators.yaml", + "nodes.yaml", + "validator-config.yaml", +] + + +def initialize(plan, node, p2p_keys_artifact, hash_sig_artifact): + cfg_kwargs = lean_shared.common_cfg_kwargs(node) + cfg_kwargs.update( + { + "image": node["image"], + "entrypoint": ["/bin/sh", "-c"], + "cmd": lean_shared.lean_tail_logs_cmd(node["service_name"])[2:], + "files": { + NODE_KEY_MOUNT: p2p_keys_artifact, + HASH_SIG_MOUNT: hash_sig_artifact, + }, + } ) - return plan.add_service(node["service_name"], cfg) + return plan.add_service(node["service_name"], ServiceConfig(**cfg_kwargs)) def start(plan, node, service, genesis_artifact, hash_sig_artifact): service_name = service.name + log_file = lean_shared.lean_log_file_path(service_name) + + # Stage the genesis text files into the running container — Kurtosis + # doesn't allow remounting a new file artifact on an existing service, + # so we read each file via plan.run_sh (yielding a runtime future) and + # plan.exec a heredoc that resolves the future at apply time. + plan.exec( + service_name=service_name, + recipe=ExecRecipe( + command=["/bin/sh", "-c", "mkdir -p {0}".format(GENESIS_MOUNT)], + ), + description="Preparing genesis mount on {0}".format(service_name), + ) + for filename in GENESIS_TEXT_FILES: + read = plan.run_sh( + run="cat /src/{0}".format(filename), + files={"/src": genesis_artifact}, + description="Reading {0} for {1}".format(filename, service_name), + ) + plan.exec( + service_name=service_name, + recipe=ExecRecipe( + command=[ + "/bin/sh", + "-c", + "cat > {0}/{1} <<'REAM_EOF'\n{2}\nREAM_EOF".format( + GENESIS_MOUNT, filename, read.output, + ), + ], + ), + description="Staging {0} into {1}".format(filename, service_name), + ) cmd_parts = [ ENTRYPOINT, - "--data-dir", - DATA_DIR, + "--data-dir", DATA_DIR, "lean_node", - "--network", - "{0}/config.yaml".format(GENESIS_MOUNT), + "--network", "{0}/config.yaml".format(GENESIS_MOUNT), "--validator-registry-path", "{0}/annotated_validators.yaml".format(GENESIS_MOUNT), - "--bootnodes", - "{0}/nodes.yaml".format(GENESIS_MOUNT), - "--node-id", - node["node_name"], - "--node-key", - "{0}/{1}.key".format(NODE_KEY_MOUNT, node["node_name"]), - "--socket-port", - str(constants.LEAN_QUIC_PORT_NUM), + "--bootnodes", "{0}/nodes.yaml".format(GENESIS_MOUNT), + "--node-id", node["node_name"], + "--node-key", "{0}/{1}.key".format(NODE_KEY_MOUNT, node["node_name"]), + "--socket-port", str(constants.LEAN_QUIC_PORT_NUM), "--metrics", - "--metrics-address", - "0.0.0.0", - "--metrics-port", - str(constants.LEAN_METRICS_PORT_NUM), - "--http-address", - "0.0.0.0", - "--http-port", - str(constants.LEAN_API_PORT_NUM), + "--metrics-address", "0.0.0.0", + "--metrics-port", str(constants.LEAN_METRICS_PORT_NUM), + "--http-address", "0.0.0.0", + "--http-port", str(constants.LEAN_API_PORT_NUM), ] if node["is_aggregator"]: cmd_parts.append("--is-aggregator") for extra in node["extra_params"]: cmd_parts.append(extra) - log_file = lean_shared.lean_log_file_path(service_name) - full_cmd = " ".join(cmd_parts) - - env_vars = dict(node["extra_env_vars"]) + rust_log = "" if node["log_level"] != "": - env_vars["RUST_LOG"] = node["log_level"] - - new_cfg = ServiceConfig( - image=node["image"], - entrypoint=["/bin/sh", "-c"], - cmd=["{0} 2>&1 | tee -a {1}".format(full_cmd, log_file)], - ports=lean_shared.lean_port_specs(), - files={ - NODE_KEY_MOUNT: node["_p2p_keys_artifact"], - GENESIS_MOUNT: genesis_artifact, - }, - env_vars=env_vars, - labels=node["extra_labels"], - min_cpu=node["min_cpu"], - max_cpu=node["max_cpu"], - min_memory=node["min_mem"], - max_memory=node["max_mem"], - node_selectors=node["node_selectors"], - tolerations=node["tolerations"], + rust_log = "RUST_LOG='{0}' ".format(node["log_level"]) + + nohup_cmd = "nohup {0}{1} >> {2} 2>&1 &".format( + rust_log, + " ".join(cmd_parts), + log_file, ) - new_service = plan.add_service( - name=service_name, - config=new_cfg, - force_update=True, + plan.exec( + service_name=service_name, + recipe=ExecRecipe( + command=["/bin/sh", "-c", nohup_cmd], + ), + description="Starting ream binary on {0}".format(service_name), ) api_url = "http://{0}:{1}".format( - new_service.ip_address, constants.LEAN_API_PORT_NUM + service.ip_address, constants.LEAN_API_PORT_NUM ) metrics_url = "http://{0}:{1}/metrics".format( - new_service.ip_address, + service.ip_address, constants.LEAN_METRICS_PORT_NUM, ) return lean_context.new_lean_context( client_name=constants.LEAN_TYPE.ream, - service_name=new_service.name, - ip_address=new_service.ip_address, + service_name=service_name, + ip_address=service.ip_address, quic_port=constants.LEAN_QUIC_PORT_NUM, api_port=constants.LEAN_API_PORT_NUM, metrics_port=constants.LEAN_METRICS_PORT_NUM, api_url=api_url, metrics_url=metrics_url, metrics_info={ - "name": new_service.name, + "name": service_name, "url": metrics_url, "path": "/metrics", "config": node["prometheus_config"], diff --git a/src/lean/zeam/zeam_launcher.star b/src/lean/zeam/zeam_launcher.star index e66036173..4e9920183 100644 --- a/src/lean/zeam/zeam_launcher.star +++ b/src/lean/zeam/zeam_launcher.star @@ -2,20 +2,7 @@ zeam launcher. Translates the Lean pipeline's per-node record into zeam's CLI surface from -`client-cmds/zeam-cmd.sh` in blockblaz/lean-quickstart: - - zeam node \ - --custom-genesis \ - --validator-config \ - --data-dir \ - --node-id --node-key \ - --metrics-enable --api-port --metrics-port - -zeam supports the "genesis_bootnode" sentinel for participants that should -derive their validator config from GENESIS_VALIDATORS rather than reading -the per-node `validator-config.yaml`. We pass the full path here because -that's the safer default; users wanting the sentinel can set -`lean_extra_params: ["--validator-config", "genesis_bootnode"]`. +`client-cmds/zeam-cmd.sh` in blockblaz/lean-quickstart. """ constants = import_module("../../package_io/constants.star") @@ -28,99 +15,109 @@ HASH_SIG_MOUNT = GENESIS_MOUNT + "/hash-sig-keys" DATA_DIR = "/data" NODE_KEY_MOUNT = constants.LEAN_NODE_KEY_MOUNTPOINT_ON_CLIENTS +GENESIS_TEXT_FILES = [ + "config.yaml", + "annotated_validators.yaml", + "nodes.yaml", + "validator-config.yaml", +] + -def initialize(plan, node, p2p_keys_artifact): - cfg = ServiceConfig( - image=node["image"], - entrypoint=["/bin/sh", "-c"], - cmd=lean_shared.lean_tail_logs_cmd(node["service_name"])[2:], - ports=lean_shared.lean_port_specs(), - files={NODE_KEY_MOUNT: p2p_keys_artifact}, - env_vars=node["extra_env_vars"], - labels=node["extra_labels"], - min_cpu=node["min_cpu"], - max_cpu=node["max_cpu"], - min_memory=node["min_mem"], - max_memory=node["max_mem"], - node_selectors=node["node_selectors"], - tolerations=node["tolerations"], +def initialize(plan, node, p2p_keys_artifact, hash_sig_artifact): + cfg_kwargs = lean_shared.common_cfg_kwargs(node) + cfg_kwargs.update( + { + "image": node["image"], + "entrypoint": ["/bin/sh", "-c"], + "cmd": lean_shared.lean_tail_logs_cmd(node["service_name"])[2:], + "files": { + NODE_KEY_MOUNT: p2p_keys_artifact, + HASH_SIG_MOUNT: hash_sig_artifact, + }, + } ) - return plan.add_service(node["service_name"], cfg) + return plan.add_service(node["service_name"], ServiceConfig(**cfg_kwargs)) def start(plan, node, service, genesis_artifact, hash_sig_artifact): service_name = service.name + log_file = lean_shared.lean_log_file_path(service_name) + + plan.exec( + service_name=service_name, + recipe=ExecRecipe( + command=["/bin/sh", "-c", "mkdir -p {0}".format(GENESIS_MOUNT)], + ), + description="Preparing genesis mount on {0}".format(service_name), + ) + for filename in GENESIS_TEXT_FILES: + read = plan.run_sh( + run="cat /src/{0}".format(filename), + files={"/src": genesis_artifact}, + description="Reading {0} for {1}".format(filename, service_name), + ) + plan.exec( + service_name=service_name, + recipe=ExecRecipe( + command=[ + "/bin/sh", + "-c", + "cat > {0}/{1} <<'ZEAM_EOF'\n{2}\nZEAM_EOF".format( + GENESIS_MOUNT, filename, read.output, + ), + ], + ), + description="Staging {0} into {1}".format(filename, service_name), + ) cmd_parts = [ ENTRYPOINT, "node", - "--custom-genesis", - GENESIS_MOUNT, - "--validator-config", - "{0}/validator-config.yaml".format(GENESIS_MOUNT), - "--data-dir", - DATA_DIR, - "--node-id", - node["node_name"], - "--node-key", - "{0}/{1}.key".format(NODE_KEY_MOUNT, node["node_name"]), + "--custom-genesis", GENESIS_MOUNT, + "--validator-config", "{0}/validator-config.yaml".format(GENESIS_MOUNT), + "--data-dir", DATA_DIR, + "--node-id", node["node_name"], + "--node-key", "{0}/{1}.key".format(NODE_KEY_MOUNT, node["node_name"]), "--metrics-enable", - "--api-port", - str(constants.LEAN_API_PORT_NUM), - "--metrics-port", - str(constants.LEAN_METRICS_PORT_NUM), + "--api-port", str(constants.LEAN_API_PORT_NUM), + "--metrics-port", str(constants.LEAN_METRICS_PORT_NUM), ] if node["is_aggregator"]: cmd_parts.append("--is-aggregator") for extra in node["extra_params"]: cmd_parts.append(extra) - log_file = lean_shared.lean_log_file_path(service_name) - full_cmd = " ".join(cmd_parts) - - new_cfg = ServiceConfig( - image=node["image"], - entrypoint=["/bin/sh", "-c"], - cmd=["{0} 2>&1 | tee -a {1}".format(full_cmd, log_file)], - ports=lean_shared.lean_port_specs(), - files={ - NODE_KEY_MOUNT: node["_p2p_keys_artifact"], - GENESIS_MOUNT: genesis_artifact, - }, - env_vars=node["extra_env_vars"], - labels=node["extra_labels"], - min_cpu=node["min_cpu"], - max_cpu=node["max_cpu"], - min_memory=node["min_mem"], - max_memory=node["max_mem"], - node_selectors=node["node_selectors"], - tolerations=node["tolerations"], + nohup_cmd = "nohup {0} >> {1} 2>&1 &".format( + " ".join(cmd_parts), + log_file, ) - new_service = plan.add_service( - name=service_name, - config=new_cfg, - force_update=True, + plan.exec( + service_name=service_name, + recipe=ExecRecipe( + command=["/bin/sh", "-c", nohup_cmd], + ), + description="Starting zeam binary on {0}".format(service_name), ) api_url = "http://{0}:{1}".format( - new_service.ip_address, constants.LEAN_API_PORT_NUM + service.ip_address, constants.LEAN_API_PORT_NUM ) metrics_url = "http://{0}:{1}/metrics".format( - new_service.ip_address, + service.ip_address, constants.LEAN_METRICS_PORT_NUM, ) return lean_context.new_lean_context( client_name=constants.LEAN_TYPE.zeam, - service_name=new_service.name, - ip_address=new_service.ip_address, + service_name=service_name, + ip_address=service.ip_address, quic_port=constants.LEAN_QUIC_PORT_NUM, api_port=constants.LEAN_API_PORT_NUM, metrics_port=constants.LEAN_METRICS_PORT_NUM, api_url=api_url, metrics_url=metrics_url, metrics_info={ - "name": new_service.name, + "name": service_name, "url": metrics_url, "path": "/metrics", "config": node["prometheus_config"], diff --git a/src/package_io/input_parser.star b/src/package_io/input_parser.star index c3d24b2fa..b853d6125 100644 --- a/src/package_io/input_parser.star +++ b/src/package_io/input_parser.star @@ -472,7 +472,14 @@ def input_parser(plan, input_args): ) ) - if result["network_params"]["fulu_fork_epoch"] != constants.FAR_FUTURE_EPOCH: + # Fulu / PeerDAS validation only applies to the Eth1 EL/CL participant + # network. Lean-only deployments (participants: [], lean_participants: [...]) + # have no CL clients and therefore no PeerDAS surface, so we skip the + # validation when participants is empty. + if ( + result["network_params"]["fulu_fork_epoch"] != constants.FAR_FUTURE_EPOCH + and len(result["participants"]) > 0 + ): has_supernodes = False has_node_with_128_plus_validators = False num_perfect_peerdas_participants = 0 @@ -625,8 +632,12 @@ def input_parser(plan, input_args): _validate_ere_gpu_config(result["zkboost_params"]["zkvms"]) + # The "first participant must have an EL" check only applies when there + # actually IS at least one Eth1 participant; lean-only deployments + # (participants: []) skip the EL/CL pipeline entirely. if ( - "bootnodoor" not in result["additional_services"] + len(result["participants"]) > 0 + and "bootnodoor" not in result["additional_services"] and result["participants"][0]["el_type"] == constants.EL_TYPE.none ): fail( @@ -1129,6 +1140,11 @@ def input_parser(plan, input_args): builder_api=result["buildoor_params"]["builder_api"], epbs_builder=result["buildoor_params"]["epbs_builder"], ), + # Lean Ethereum. Stored as plain lists/dicts (not nested structs) + # because the Lean per-client launchers reach for fields by string + # key — see src/lean/lean_launcher.star. + lean_participants=result["lean_participants"], + lean_network_params=result["lean_network_params"], ) diff --git a/src/prelaunch_data_generator/lean_genesis/lean_genesis_generator.star b/src/prelaunch_data_generator/lean_genesis/lean_genesis_generator.star index 267ce9d34..424bf27c8 100644 --- a/src/prelaunch_data_generator/lean_genesis/lean_genesis_generator.star +++ b/src/prelaunch_data_generator/lean_genesis/lean_genesis_generator.star @@ -57,11 +57,24 @@ def _compute_genesis_time(plan, lean_network_params): return result.output -def _render_validator_config(plan, services_meta, lean_network_params): +def _render_validator_config(plan, services_meta, lean_network_params, keys_artifact): """Render validator-config.yaml from per-node IPs / ports / keys. - `services_meta` is a list of dicts with keys: name, ip_address, quic_port, - metrics_port, api_port, privkey, validator_count, is_aggregator. + Two-stage render: + + Stage 1 - `plan.render_templates` produces validator-config.yaml with + IPs and ports substituted from Kurtosis runtime futures, but the + privkey field set to a placeholder marker `__PRIVKEY___`. + Kurtosis's template engine handles the IP futures correctly; trying + to embed them inside a raw shell heredoc doesn't (the {{kurtosis:...}} + markers reach the shell before substitution and break sh parsing). + + Stage 2 - `plan.run_sh` reads the placeholders and replaces each one + with the corresponding key from the lean-node-keys artifact via sed. + The shell never sees a Kurtosis future, only literal text. + + `services_meta` is a list of dicts with keys: name, ip_address, + quic_port, metrics_port, api_port, validator_count, is_aggregator. """ template = """shuffle: roundrobin deployment_mode: kurtosis @@ -71,33 +84,32 @@ config: attestation_committee_count: {{.AttestationCommitteeCount}} validators: {{- range .Validators}} - - name: "{{.name}}" - privkey: "{{.privkey}}" + - name: "{{.Name}}" + privkey: "__PRIVKEY_{{.Name}}__" enrFields: - ip: "{{.ip}}" - quic: {{.quic}} - metricsPort: {{.metricsPort}} - apiPort: {{.apiPort}} - isAggregator: {{.isAggregator}} - count: {{.count}} + ip: "{{.Ip}}" + quic: {{.Quic}} + metricsPort: {{.MetricsPort}} + apiPort: {{.ApiPort}} + isAggregator: {{.IsAggregator}} + count: {{.Count}} {{end}}""" validators = [] for meta in services_meta: validators.append( { - "name": meta["name"], - "privkey": meta["privkey"], - "ip": meta["ip_address"], - "quic": meta["quic_port"], - "metricsPort": meta["metrics_port"], - "apiPort": meta["api_port"], - "isAggregator": "true" if meta["is_aggregator"] else "false", - "count": meta["validator_count"], + "Name": meta["name"], + "Ip": meta["ip_address"], + "Quic": meta["quic_port"], + "MetricsPort": meta["metrics_port"], + "ApiPort": meta["api_port"], + "IsAggregator": "true" if meta["is_aggregator"] else "false", + "Count": meta["validator_count"], } ) - return plan.render_templates( + template_artifact = plan.render_templates( config={ "validator-config.yaml": struct( template=template, @@ -110,9 +122,39 @@ validators: }, ), }, - name="lean-validator-config", - description="Rendering Lean validator-config.yaml", + name="lean-validator-config-template", + description="Rendering Lean validator-config.yaml (stage 1, placeholders)", + ) + + # Stage 2: sed each `__PRIVKEY___` to the contents of the + # matching `/keys/.key`. Run inside a single shell so we don't + # have to thread Kurtosis futures through more steps. + sed_lines = ["set -eu", "mkdir -p /out", "cp /tpl/validator-config.yaml /out/"] + for meta in services_meta: + # The privkey file holds a raw 64-char hex string with no surrounding + # whitespace. Use sed with a `|` delimiter since the key is hex + # (which never contains `|`) and the placeholder is unique. + sed_lines.append( + ( + "key=$(cat /keys/{0}.key) && " + + "sed -i \"s|__PRIVKEY_{0}__|$key|\" /out/validator-config.yaml" + ).format(meta["name"]) + ) + sed_script = "\n".join(sed_lines) + + result = plan.run_sh( + run=sed_script, + # alpine/openssl already lives in the engine cache from the P2P + # key generation step and ships busybox sed. + image="alpine/openssl", + files={ + "/tpl": template_artifact, + "/keys": keys_artifact, + }, + store=[StoreSpec(src="/out/validator-config.yaml", name="lean-validator-config")], + description="Rendering Lean validator-config.yaml (stage 2, privkey inlining)", ) + return result.files_artifacts[0] def _render_initial_config(plan, genesis_time, lean_network_params, total_validators): @@ -162,10 +204,14 @@ def _generate_hash_sig_keys(plan, image, num_validators, active_epoch): tree becomes the `lean-hash-sig-keys` artifact mounted at the Lean client's `--hash-sig-keys-dir`. """ + # The binary in `blockblaz/hash-sig-cli:latest` is `hashsig` (the image's + # ENTRYPOINT). Kurtosis's `plan.run_sh` overrides the entrypoint with + # `sh -c`, so we have to call the binary by its absolute path. It lives + # at `/usr/local/bin/hashsig`. plan.run_sh( run=( "mkdir -p {0} && " - + "hash-sig-cli generate " + + "/usr/local/bin/hashsig generate " + "--num-validators {1} " + "--log-num-active-epochs {2} " + "--output-dir {0} " @@ -182,6 +228,28 @@ def _generate_hash_sig_keys(plan, image, num_validators, active_epoch): return HASH_SIG_ARTIFACT_NAME +def generate_hash_sig_keys(plan, lean_network_params, total_validators): + """Public wrapper around _generate_hash_sig_keys. + + Exposed so `lean_launcher.launch` can pre-create the hash-sig artifact + before any service is added. This lets every Lean client mount the keys + at `initialize()` time (the keys are IP-independent), keeping us inside + Kurtosis's "one add_service per name" constraint. + """ + if total_validators < 1: + fail( + "Lean genesis requires at least one validator across all " + + "lean_participants (got 0).", + ) + _, hash_sig_image = _resolve_images(lean_network_params) + return _generate_hash_sig_keys( + plan, + hash_sig_image, + total_validators, + lean_network_params["active_epoch"], + ) + + def _run_genesis_tool( plan, image, @@ -247,58 +315,126 @@ def _post_process( every client just mounts `/network-configs` and reads everything it needs from one place. - Implemented in shell + yq inside a single busybox-style helper because - Starlark has no yaml/json libs. + Implemented as a Python script (Starlark has no YAML libs and busybox + `sh` in common images chokes on heredocs with embedded interpreters). + The script is rendered as a separate artifact via `render_templates` + and invoked by a tiny shell wrapper - no heredocs reach `sh`. """ + # The script reads the hash-sig manifest, PK's validators.yaml output, + # and the raw genesis bundle, then writes: + # - /out/config.yaml with GENESIS_VALIDATORS appended (dual-key layout) + # - /out/annotated_validators.yaml (node_name -> [{index, pubkey_hex, + # privkey_file}, ...] with attester + proposer rows per index) + # - all other genesis files copied through unchanged + # - hash-sig keys bundled into ./hash-sig-keys/ + # - per-node `.key` libp2p secrets bundled at the top level + python_source = """import os +import shutil +import yaml + +RAW = "/raw" +HASH_SIG = "/hash-sig" +VC = "/vc" +NODE_KEYS = "/node-keys" +OUT = "/out" +MANIFEST = os.path.join(HASH_SIG, "validator-keys-manifest.yaml") + + +def _as_hex(value): + # Normalise a pubkey field to a no-0x-prefix lowercase hex string. + # YAML 1.1 (PyYAML default) interprets unquoted `0x...` tokens as + # integers, so each field may arrive as either int or str depending on + # how hash-sig-cli wrote the manifest. Handle both. + if isinstance(value, int): + return format(value, "x") + s = str(value) + if s.startswith("0x") or s.startswith("0X"): + s = s[2:] + return s + + +def copytree_into(src, dst): + os.makedirs(dst, exist_ok=True) + for entry in os.listdir(src): + s = os.path.join(src, entry) + d = os.path.join(dst, entry) + if os.path.isdir(s): + shutil.copytree(s, d, dirs_exist_ok=True) + else: + shutil.copy2(s, d) + + +# Stage 1: bundle all input artifacts into /out. +os.makedirs(OUT, exist_ok=True) +copytree_into(RAW, OUT) +shutil.copy2(os.path.join(VC, "validator-config.yaml"), OUT) +for f in os.listdir(NODE_KEYS): + if f.endswith(".key"): + shutil.copy2(os.path.join(NODE_KEYS, f), OUT) +copytree_into(HASH_SIG, os.path.join(OUT, "hash-sig-keys")) + +# Stage 2: append GENESIS_VALIDATORS (dual-key) to config.yaml. +with open(MANIFEST) as f: + manifest = yaml.safe_load(f) + +gv_lines = ["", "# Genesis validator public keys (post-quantum hash-sig)", "GENESIS_VALIDATORS:"] +for v in manifest["validators"]: + ah = _as_hex(v["attester_key_pubkey_hex"]) + ph = _as_hex(v["proposer_key_pubkey_hex"]) + gv_lines.append(' - attestation_pubkey: "{0}"'.format(ah)) + gv_lines.append(' proposal_pubkey: "{0}"'.format(ph)) +with open(os.path.join(OUT, "config.yaml"), "a") as f: + f.write("\\n".join(gv_lines) + "\\n") + +# Stage 3: render annotated_validators.yaml from validators.yaml + manifest. +with open(os.path.join(OUT, "validators.yaml")) as f: + assignments = yaml.safe_load(f) or {} + +ann_lines = [] +for node, indices in assignments.items(): + ann_lines.append("{0}:".format(node)) + if not indices: + ann_lines.append(" []") + continue + for idx in indices: + v = manifest["validators"][int(idx)] + ah = _as_hex(v["attester_key_pubkey_hex"]) + ph = _as_hex(v["proposer_key_pubkey_hex"]) + ann_lines.append(" - index: {0}".format(idx)) + ann_lines.append(" pubkey_hex: {0}".format(ah)) + ann_lines.append( + " privkey_file: validator_{0}_attester_key_sk.ssz".format(idx) + ) + ann_lines.append(" - index: {0}".format(idx)) + ann_lines.append(" pubkey_hex: {0}".format(ph)) + ann_lines.append( + " privkey_file: validator_{0}_proposer_key_sk.ssz".format(idx) + ) +with open(os.path.join(OUT, "annotated_validators.yaml"), "w") as f: + f.write("\\n".join(ann_lines) + "\\n") +""" + + script_artifact = plan.render_templates( + config={ + "post_process.py": struct(template=python_source, data={}), + }, + name="lean-post-process-script", + description="Rendering Lean genesis post-process script", + ) + return plan.run_sh( - run=""" - set -eu - mkdir -p /out /out/hash-sig-keys - cp /raw/* /out/ - cp /vc/validator-config.yaml /out/ - cp /node-keys/*.key /out/ - # Bundle hash-sig keys into the same artifact (nested file artifact - # mounts can't overlap in Kurtosis, so we ship a single tree). - cp -r /hash-sig/. /out/hash-sig-keys/ - - # Append GENESIS_VALIDATORS to config.yaml (dual-key layout). - manifest=/hash-sig/validator-keys-manifest.yaml - n=$(yq eval '.validators | length' "$manifest") - printf '\\n# Genesis validator public keys (post-quantum hash-sig)\\nGENESIS_VALIDATORS:\\n' >> /out/config.yaml - i=0 - while [ "$i" -lt "$n" ]; do - ah=$(yq eval ".validators[$i].attester_key_pubkey_hex" "$manifest" | sed 's/^0x//') - ph=$(yq eval ".validators[$i].proposer_key_pubkey_hex" "$manifest" | sed 's/^0x//') - printf ' - attestation_pubkey: "%s"\\n proposal_pubkey: "%s"\\n' "$ah" "$ph" >> /out/config.yaml - i=$((i + 1)) - done - - # Render annotated_validators.yaml from validators.yaml (PK output) - # joined with the manifest. Each validator index gets two rows - # (attester + proposer) so clients can route by filename. - : > /out/annotated_validators.yaml - for node in $(yq eval 'keys | .[]' /out/validators.yaml); do - printf '%s:\\n' "$node" >> /out/annotated_validators.yaml - indices=$(yq eval ".\\"$node\\" | .[]" /out/validators.yaml) - if [ -z "$indices" ]; then - printf ' []\\n' >> /out/annotated_validators.yaml - continue - fi - for idx in $indices; do - ah=$(yq eval ".validators[$idx].attester_key_pubkey_hex" "$manifest" | sed 's/^0x//') - ph=$(yq eval ".validators[$idx].proposer_key_pubkey_hex" "$manifest" | sed 's/^0x//') - printf ' - index: %s\\n pubkey_hex: %s\\n privkey_file: validator_%s_attester_key_sk.ssz\\n' "$idx" "$ah" "$idx" >> /out/annotated_validators.yaml - printf ' - index: %s\\n pubkey_hex: %s\\n privkey_file: validator_%s_proposer_key_sk.ssz\\n' "$idx" "$ph" "$idx" >> /out/annotated_validators.yaml - done - done - """, - # mikefarah/yq image ships yq + busybox; we don't need anything else. - image="mikefarah/yq:4", + run=( + "set -eu; " + + "pip install --quiet --root-user-action=ignore pyyaml; " + + "python3 /script/post_process.py" + ), + image="python:3-alpine", files={ "/raw": raw_genesis_artifact, "/hash-sig": hash_sig_artifact, "/vc": validator_config_artifact, "/node-keys": node_key_artifact, + "/script": script_artifact, }, store=[ StoreSpec(src="/out", name=GENESIS_ARTIFACT_NAME), @@ -307,19 +443,26 @@ def _post_process( ).files_artifacts[0] -def generate(plan, services_meta, lean_network_params, node_key_artifact): +def generate( + plan, + services_meta, + lean_network_params, + node_key_artifact, + hash_sig_artifact, +): """Top-level entrypoint. Args: plan: Kurtosis plan. services_meta: list of dicts (one per node) with: name, ip_address, - quic_port, metrics_port, api_port, privkey (hex string), - validator_count, is_aggregator. The caller (lean_launcher) builds - this list after Kurtosis has assigned IPs to the placeholder - services. + quic_port, metrics_port, api_port, validator_count, is_aggregator. + The caller (lean_launcher) builds this list after Kurtosis has + assigned IPs to the placeholder services. lean_network_params: validated `lean_network_params` block. node_key_artifact: files artifact holding `.key` ASCII-hex P2P secrets (one per node). + hash_sig_artifact: files artifact holding the XMSS attester+proposer + keypairs (pre-generated by `generate_hash_sig_keys`). Returns: struct(genesis_artifact = , hash_sig_artifact = , @@ -335,20 +478,14 @@ def generate(plan, services_meta, lean_network_params, node_key_artifact): + "lean_participants (got 0).", ) - genesis_image, hash_sig_image = _resolve_images(lean_network_params) + genesis_image, _ = _resolve_images(lean_network_params) genesis_time = _compute_genesis_time(plan, lean_network_params) - hash_sig_artifact = _generate_hash_sig_keys( - plan, - hash_sig_image, - total_validators, - lean_network_params["active_epoch"], - ) - validator_config_artifact = _render_validator_config( plan, services_meta, lean_network_params, + node_key_artifact, ) initial_config_artifact = _render_initial_config( diff --git a/src/prelaunch_data_generator/lean_genesis/p2p_keys_generator.star b/src/prelaunch_data_generator/lean_genesis/p2p_keys_generator.star index edc93affd..16818fad2 100644 --- a/src/prelaunch_data_generator/lean_genesis/p2p_keys_generator.star +++ b/src/prelaunch_data_generator/lean_genesis/p2p_keys_generator.star @@ -4,58 +4,45 @@ P2P key generation for Lean consensus nodes. Each Lean node needs a 32-byte libp2p identity key (the secp256k1 secret that derives the peer ID and ENR). lean-quickstart writes one such key per node into `.key` as ASCII hex. We reproduce that exact layout so the -genesis tool and every Lean client client-cmds/-cmd.sh contract -matches what they receive at runtime. +genesis tool and every Lean client's CLI receive the same files at runtime. + +We deliberately do NOT read the generated key values back into Starlark — in +Kurtosis, `plan.run_sh(...).output` is a runtime future that only materialises +when the plan is applied, so it can't be used to drive Starlark interpretation +(dict lookups, loops, etc.). Instead we export the keys as a files artifact +and let *downstream containers* read each key from `/keys/.key` when +they need it. """ OPENSSL_IMAGE = "alpine/openssl" +KEYS_ARTIFACT_NAME = "lean-node-keys" +KEYS_MOUNT_INSIDE_GENERATOR = "/keys" -def generate_node_keys(plan, node_names): - """Generate one 32-byte hex P2P key per node. - The keys are written to `/keys/.key` inside the OpenSSL helper - container and exported as a single Kurtosis files artifact (`lean-node-keys`) - so every Lean client mounts the same directory and reads its own key by - name. Returns (artifact_name, {node_name: hex_string}). +def generate_node_keys(plan, node_names): + """Generate one 32-byte hex P2P key per node into a single artifact. - The hex strings are also returned in-memory because the genesis tool needs - them to compute peer IDs / ENRs for `nodes.yaml` *before* the Lean clients - are started. + Returns the artifact name. Reading individual key values back into + Starlark is not supported — see the module docstring. """ - # `tr -d` strips OpenSSL's trailing newline; without it `nodes.yaml`'s - # peer-id derivation downstream sees an extra byte and computes the wrong - # libp2p identity. - script_parts = ["set -eu", "mkdir -p /keys"] - for name in node_names: - script_parts.append( - "openssl rand -hex 32 | tr -d '\\n' > /keys/{0}.key".format(name) - ) - # Dump everything to stdout so we can capture the keys without a second - # round-trip. `printf '%s=%s\\n'` keeps the parser trivial. + # `tr -d` strips OpenSSL's trailing newline; without it, the genesis + # tool's libp2p peer ID derivation downstream sees an extra byte and + # computes the wrong identity, which then mismatches what the running + # client derives. + script_parts = ["set -eu", "mkdir -p {0}".format(KEYS_MOUNT_INSIDE_GENERATOR)] for name in node_names: script_parts.append( - "printf '%s=%s\\n' '{0}' \"$(cat /keys/{0}.key)\"".format(name) + "openssl rand -hex 32 | tr -d '\\n' > {0}/{1}.key".format( + KEYS_MOUNT_INSIDE_GENERATOR, name + ) ) script = "\n".join(script_parts) - result = plan.run_sh( + plan.run_sh( run=script, image=OPENSSL_IMAGE, - store=[ - StoreSpec(src="/keys", name="lean-node-keys"), - ], - description="Generating Lean P2P node keys", - ) - - keys_by_name = {} - for line in result.output.strip().split("\n"): - if "=" not in line: - continue - name, hex_value = line.split("=", 1) - keys_by_name[name.strip()] = hex_value.strip() - - return struct( - artifact_name=result.files_artifacts[0], - keys=keys_by_name, + store=[StoreSpec(src=KEYS_MOUNT_INSIDE_GENERATOR, name=KEYS_ARTIFACT_NAME)], + description="Generating Lean P2P node keys ({0} nodes)".format(len(node_names)), ) + return KEYS_ARTIFACT_NAME From 4c876314a6b3d0263f9ecc9ed5a2087ff1512a2a Mon Sep 17 00:00:00 2001 From: ilitteri Date: Wed, 13 May 2026 18:07:38 -0300 Subject: [PATCH 04/25] Load XMSS pubkey manifest with yaml.BaseLoader The hash-sig-cli manifest writes attester_key_pubkey_hex / proposer_key_pubkey_hex as unquoted `0x...` tokens. PyYAML's default loader (and yaml.safe_load) parse those as YAML 1.1 ints, which silently drops leading zeros. Reformatting via `format(value, "x")` then emits an odd-length hex string, and every Lean client correctly rejects the resulting config.yaml ("pubkey is not valid hex" / "odd number of digits at line ..."). Switch the loader to yaml.BaseLoader so every scalar stays a Python str; _as_hex retains its int fallback in case a future manifest version writes the field differently. Validated with a 4-node ethlambda+ream multi-client devnet: peer_count=3 on every node, cross-client status exchange confirmed. --- .../lean_genesis/lean_genesis_generator.star | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-) diff --git a/src/prelaunch_data_generator/lean_genesis/lean_genesis_generator.star b/src/prelaunch_data_generator/lean_genesis/lean_genesis_generator.star index 424bf27c8..5ba22231d 100644 --- a/src/prelaunch_data_generator/lean_genesis/lean_genesis_generator.star +++ b/src/prelaunch_data_generator/lean_genesis/lean_genesis_generator.star @@ -342,9 +342,8 @@ MANIFEST = os.path.join(HASH_SIG, "validator-keys-manifest.yaml") def _as_hex(value): # Normalise a pubkey field to a no-0x-prefix lowercase hex string. - # YAML 1.1 (PyYAML default) interprets unquoted `0x...` tokens as - # integers, so each field may arrive as either int or str depending on - # how hash-sig-cli wrote the manifest. Handle both. + # Even with BaseLoader (which keeps everything as str) we strip the 0x + # prefix here; int fallback handles unexpected manifest shapes. if isinstance(value, int): return format(value, "x") s = str(value) @@ -375,7 +374,12 @@ copytree_into(HASH_SIG, os.path.join(OUT, "hash-sig-keys")) # Stage 2: append GENESIS_VALIDATORS (dual-key) to config.yaml. with open(MANIFEST) as f: - manifest = yaml.safe_load(f) + # BaseLoader keeps every scalar as a Python str. We need this for the + # XMSS pubkey hex fields: YAML 1.1 (PyYAML's default) interprets + # unquoted `0x...` tokens as integers, which silently drops leading + # zeros when we format the value back out — clients then reject the + # config because the pubkey has an odd number of hex digits. + manifest = yaml.load(f, Loader=yaml.BaseLoader) gv_lines = ["", "# Genesis validator public keys (post-quantum hash-sig)", "GENESIS_VALIDATORS:"] for v in manifest["validators"]: @@ -388,7 +392,9 @@ with open(os.path.join(OUT, "config.yaml"), "a") as f: # Stage 3: render annotated_validators.yaml from validators.yaml + manifest. with open(os.path.join(OUT, "validators.yaml")) as f: - assignments = yaml.safe_load(f) or {} + # Use BaseLoader too (see manifest load above) for consistency, even + # though this file has no hex tokens to worry about today. + assignments = yaml.load(f, Loader=yaml.BaseLoader) or {} ann_lines = [] for node, indices in assignments.items(): From 09f8fdde4c8ef60a7c16b01a696d37701521c4ac Mon Sep 17 00:00:00 2001 From: ilitteri Date: Wed, 13 May 2026 18:30:36 -0300 Subject: [PATCH 05/25] Wire zeam end-to-end via static busybox + genesis_bootnode MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit blockblaz/zeam:devnet4 is a scratch image — only /app/zig-out/bin/zeam exists, no /bin/sh, /bin/tail, /usr/bin/touch, or even /var/log. The existing placeholder-then-plan.exec lifecycle therefore couldn't run for zeam. This commit adds a static busybox binary as a Kurtosis files artifact, mounts it at /usr/local/bin/, and routes every shell-needing step through busybox sh + dispatched applets (busybox mkdir / touch / tail / cat / nohup). Three additional zeam-specific fixups surfaced while iterating: 1. The placeholder cmd mkdir -p's $(dirname /var/log/.log) before touch — /var/log doesn't exist in scratch. 2. The start phase mkdir -p's /data and copies the per-node .key from /node-keys into /network-configs (lean-quickstart's zeam contract reads --node-key relative to --custom-genesis). 3. --validator-config is set to the literal `genesis_bootnode` sentinel rather than a YAML file path. zeam's --validator-config accepts either a *directory* of per-node validator configs or the sentinel; pointing it at a single YAML file triggers a NotDir failure during "build node start options". Validated end-to-end with a 6-node devnet (2x ethlambda + 2x ream + 2x zeam): every node is RUNNING; lean-zeam-0 reports "Connected Peers: 5", builds the genesis state, and prints the fork-choice tree. Cross-client peering between ethlambda <-> ream <-> zeam confirmed via status request/response exchange in all three clients' logs. --- src/lean/zeam/zeam_launcher.star | 88 +++++++++++++++++++++++++++----- 1 file changed, 76 insertions(+), 12 deletions(-) diff --git a/src/lean/zeam/zeam_launcher.star b/src/lean/zeam/zeam_launcher.star index 4e9920183..6b7c552d9 100644 --- a/src/lean/zeam/zeam_launcher.star +++ b/src/lean/zeam/zeam_launcher.star @@ -3,6 +3,14 @@ zeam launcher. Translates the Lean pipeline's per-node record into zeam's CLI surface from `client-cmds/zeam-cmd.sh` in blockblaz/lean-quickstart. + +`blockblaz/zeam:devnet4` is a `scratch`-based image — only the zeam binary +exists at `/app/zig-out/bin/zeam`, no `/bin/sh`, `tail`, `nohup`, or +anything else. The Lean pipeline's placeholder-then-plan.exec pattern +needs a shell to (a) keep the placeholder alive while genesis runs, and +(b) cat-stage text genesis files in. To unblock this without rebuilding +zeam's image, we inject a static busybox binary as a file artifact and +mount it at `/bin/busybox`, then drive everything through it. """ constants = import_module("../../package_io/constants.star") @@ -14,25 +22,56 @@ GENESIS_MOUNT = constants.LEAN_GENESIS_MOUNTPOINT_ON_CLIENTS HASH_SIG_MOUNT = GENESIS_MOUNT + "/hash-sig-keys" DATA_DIR = "/data" NODE_KEY_MOUNT = constants.LEAN_NODE_KEY_MOUNTPOINT_ON_CLIENTS +BUSYBOX_MOUNT = "/usr/local/bin" +BUSYBOX = "/usr/local/bin/busybox" GENESIS_TEXT_FILES = [ "config.yaml", "annotated_validators.yaml", "nodes.yaml", "validator-config.yaml", + # zeam scans the custom-genesis dir for validators.yaml (PK's raw + # validator-index assignments); the heredoc-stage covers it since it's + # plain YAML, no binary content. + "validators.yaml", ] +def _busybox_artifact(plan): + # Extract the static busybox binary out of busybox:musl. Once exported + # as a Kurtosis files artifact it can be mounted into any scratch + # container as `/usr/local/bin/busybox` so we have a working shell to + # run plan.exec scripts against. + return plan.run_sh( + run="mkdir -p /out && cp /bin/busybox /out/busybox", + image="busybox:musl", + store=[StoreSpec(src="/out", name="lean-busybox")], + description="Extracting static busybox for zeam scratch image", + ).files_artifacts[0] + + def initialize(plan, node, p2p_keys_artifact, hash_sig_artifact): + busybox_artifact = _busybox_artifact(plan) cfg_kwargs = lean_shared.common_cfg_kwargs(node) cfg_kwargs.update( { "image": node["image"], - "entrypoint": ["/bin/sh", "-c"], - "cmd": lean_shared.lean_tail_logs_cmd(node["service_name"])[2:], + # Override the zeam entrypoint with busybox sh; the real zeam + # binary is invoked later via plan.exec. + "entrypoint": [BUSYBOX, "sh", "-c"], + "cmd": [ + # zeam's scratch image has no /usr/bin/touch, /bin/tail, and + # not even /var/log. Every applet has to be dispatched through + # busybox; mkdir -p creates /var/log on first touch. + "{0} mkdir -p $({0} dirname {1}) && {0} touch {1} && {0} tail -f {1}".format( + BUSYBOX, + lean_shared.lean_log_file_path(node["service_name"]), + ) + ], "files": { NODE_KEY_MOUNT: p2p_keys_artifact, HASH_SIG_MOUNT: hash_sig_artifact, + BUSYBOX_MOUNT: busybox_artifact, }, } ) @@ -43,10 +82,26 @@ def start(plan, node, service, genesis_artifact, hash_sig_artifact): service_name = service.name log_file = lean_shared.lean_log_file_path(service_name) + # Use busybox sh for every shell-needing step inside the zeam container. + # Pre-create /data so zeam can write its RocksDB / LMDB there, and stage + # the libp2p node key into /network-configs (lean-quickstart's zeam + # contract has --node-key inside the custom-genesis dir, not in a + # separate mount). plan.exec( service_name=service_name, recipe=ExecRecipe( - command=["/bin/sh", "-c", "mkdir -p {0}".format(GENESIS_MOUNT)], + command=[ + BUSYBOX, + "sh", + "-c", + "{0} mkdir -p {1} {2} && {0} cp {3}/{4}.key {1}/{4}.key".format( + BUSYBOX, + GENESIS_MOUNT, + DATA_DIR, + NODE_KEY_MOUNT, + node["node_name"], + ), + ], ), description="Preparing genesis mount on {0}".format(service_name), ) @@ -60,10 +115,11 @@ def start(plan, node, service, genesis_artifact, hash_sig_artifact): service_name=service_name, recipe=ExecRecipe( command=[ - "/bin/sh", + BUSYBOX, + "sh", "-c", - "cat > {0}/{1} <<'ZEAM_EOF'\n{2}\nZEAM_EOF".format( - GENESIS_MOUNT, filename, read.output, + "{3} cat > {0}/{1} <<'ZEAM_EOF'\n{2}\nZEAM_EOF".format( + GENESIS_MOUNT, filename, read.output, BUSYBOX, ), ], ), @@ -74,10 +130,18 @@ def start(plan, node, service, genesis_artifact, hash_sig_artifact): ENTRYPOINT, "node", "--custom-genesis", GENESIS_MOUNT, - "--validator-config", "{0}/validator-config.yaml".format(GENESIS_MOUNT), + # zeam's --validator-config accepts either a directory of per-node + # validator configs OR the literal sentinel `genesis_bootnode`, + # which tells zeam to derive its validator set from the + # GENESIS_VALIDATORS list in config.yaml. Pointing at a single + # file path (lean-quickstart's `validator-config.yaml`) trips + # zeam's "NotDir" check, so use the sentinel here. + "--validator-config", "genesis_bootnode", "--data-dir", DATA_DIR, "--node-id", node["node_name"], - "--node-key", "{0}/{1}.key".format(NODE_KEY_MOUNT, node["node_name"]), + # zeam (per lean-quickstart's contract) reads --node-key relative to + # the custom-genesis dir; we staged it there in the prepare step. + "--node-key", "{0}/{1}.key".format(GENESIS_MOUNT, node["node_name"]), "--metrics-enable", "--api-port", str(constants.LEAN_API_PORT_NUM), "--metrics-port", str(constants.LEAN_METRICS_PORT_NUM), @@ -87,14 +151,15 @@ def start(plan, node, service, genesis_artifact, hash_sig_artifact): for extra in node["extra_params"]: cmd_parts.append(extra) - nohup_cmd = "nohup {0} >> {1} 2>&1 &".format( + nohup_cmd = "{0} nohup {1} >> {2} 2>&1 &".format( + BUSYBOX, " ".join(cmd_parts), log_file, ) plan.exec( service_name=service_name, recipe=ExecRecipe( - command=["/bin/sh", "-c", nohup_cmd], + command=[BUSYBOX, "sh", "-c", nohup_cmd], ), description="Starting zeam binary on {0}".format(service_name), ) @@ -103,8 +168,7 @@ def start(plan, node, service, genesis_artifact, hash_sig_artifact): service.ip_address, constants.LEAN_API_PORT_NUM ) metrics_url = "http://{0}:{1}/metrics".format( - service.ip_address, - constants.LEAN_METRICS_PORT_NUM, + service.ip_address, constants.LEAN_METRICS_PORT_NUM, ) return lean_context.new_lean_context( From 0559f18aee213c3c36ac2127e8e6603adc0372a1 Mon Sep 17 00:00:00 2001 From: ilitteri Date: Wed, 13 May 2026 18:43:42 -0300 Subject: [PATCH 06/25] Bump hash-sig keygen plan.run_sh wait to 30m XMSS keypair generation is CPU-bound and scales with num-validators * 2^active_epoch. On slower hosts (and with the default active_epoch=18 == 2^18 epochs per key) the default 180s plan.run_sh timeout fires mid-generation, killing the run with "exec request timed out". Bump the wait to 30m so it has headroom on shared/remote hosts. The step is idempotent in the kurtosis artifact sense - re-runs of the same package args reuse the artifact. Surfaced while bringing up the 6-node ethlambda+ream+zeam devnet on ethrex-mainnet-test-1. --- .../lean_genesis/lean_genesis_generator.star | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/src/prelaunch_data_generator/lean_genesis/lean_genesis_generator.star b/src/prelaunch_data_generator/lean_genesis/lean_genesis_generator.star index 5ba22231d..55f254074 100644 --- a/src/prelaunch_data_generator/lean_genesis/lean_genesis_generator.star +++ b/src/prelaunch_data_generator/lean_genesis/lean_genesis_generator.star @@ -137,7 +137,7 @@ validators: sed_lines.append( ( "key=$(cat /keys/{0}.key) && " - + "sed -i \"s|__PRIVKEY_{0}__|$key|\" /out/validator-config.yaml" + + 'sed -i "s|__PRIVKEY_{0}__|$key|" /out/validator-config.yaml' ).format(meta["name"]) ) sed_script = "\n".join(sed_lines) @@ -151,7 +151,9 @@ validators: "/tpl": template_artifact, "/keys": keys_artifact, }, - store=[StoreSpec(src="/out/validator-config.yaml", name="lean-validator-config")], + store=[ + StoreSpec(src="/out/validator-config.yaml", name="lean-validator-config") + ], description="Rendering Lean validator-config.yaml (stage 2, privkey inlining)", ) return result.files_artifacts[0] @@ -218,6 +220,11 @@ def _generate_hash_sig_keys(plan, image, num_validators, active_epoch): + "--export-format ssz" ).format(HASH_SIG_DIR, num_validators, active_epoch), image=image, + # XMSS keygen is CPU-bound and scales with --num-validators * + # 2^active_epoch. On slower hosts (or with active_epoch >= 18) it + # overruns the default 180s plan.run_sh timeout, so give it 30 + # minutes. The step is idempotent — re-runs won't regenerate. + wait="30m", store=[ StoreSpec(src=HASH_SIG_DIR, name=HASH_SIG_ARTIFACT_NAME), ], From 5f978002ba3a244a3fff753f9cd62c7795d87f6e Mon Sep 17 00:00:00 2001 From: ilitteri Date: Wed, 13 May 2026 18:59:40 -0300 Subject: [PATCH 07/25] Add Prometheus + Grafana to the Lean Kurtosis pipeline MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Mirror lean-quickstart's docker-compose-metrics.yaml stack inside the Kurtosis enclave: one `lean-prometheus` (prom v3.8.0) scraping every lean--:5054/metrics target plus its own /metrics, and one `lean-grafana` (12.3.2) provisioning the Prometheus datasource and the upstream Lean client dashboard at port 3000. Inside the enclave we resolve scrape targets by service DNS name, so no `host.docker.internal` workaround is needed. The dashboard JSON (client-dashboard.json) is vendored under src/lean/metrics/grafana/dashboards/ at the same upstream commit as lean-quickstart. Gated on `lean_network_params.metrics_enabled` (default true), so any operator who wants a metrics-free run can set it false. Anonymous admin login is enabled (matches lean-quickstart) — there's no admin/admin prompt to navigate through. Validated end-to-end against a 6-node ethlambda+ream+zeam devnet: all 7 scrape targets (6 clients + prometheus self) report `up`, Grafana health endpoint returns 200, dashboard "Lean Ethereum Clients Dashboard" loads at /d/lean-ethereum-clients-dashboard. --- src/lean/ethlambda/ethlambda_launcher.star | 46 +- src/lean/lean_launcher.star | 8 + .../grafana/dashboards/client-dashboard.json | 6803 +++++++++++++++++ src/lean/metrics/metrics_launcher.star | 196 + src/lean/ream/ream_launcher.star | 38 +- src/lean/zeam/zeam_launcher.star | 33 +- src/package_io/input_parser.star | 4 + 7 files changed, 7087 insertions(+), 41 deletions(-) create mode 100644 src/lean/metrics/grafana/dashboards/client-dashboard.json create mode 100644 src/lean/metrics/metrics_launcher.star diff --git a/src/lean/ethlambda/ethlambda_launcher.star b/src/lean/ethlambda/ethlambda_launcher.star index 987bbf6d9..54010354e 100644 --- a/src/lean/ethlambda/ethlambda_launcher.star +++ b/src/lean/ethlambda/ethlambda_launcher.star @@ -106,7 +106,9 @@ def start(plan, node, service, genesis_artifact, hash_sig_artifact): "/bin/sh", "-c", "cat > {0}/{1} <<'ETHLAMBDA_EOF'\n{2}\nETHLAMBDA_EOF".format( - GENESIS_MOUNT, filename, read.output, + GENESIS_MOUNT, + filename, + read.output, ), ], ), @@ -119,18 +121,30 @@ def start(plan, node, service, genesis_artifact, hash_sig_artifact): # log file the tail is already watching. cmd_parts = [ ENTRYPOINT, - "--genesis", "{0}/config.yaml".format(GENESIS_MOUNT), - "--validators", "{0}/annotated_validators.yaml".format(GENESIS_MOUNT), - "--bootnodes", "{0}/nodes.yaml".format(GENESIS_MOUNT), - "--validator-config", "{0}/validator-config.yaml".format(GENESIS_MOUNT), - "--hash-sig-keys-dir", HASH_SIG_MOUNT, - "--data-dir", DATA_DIR, - "--gossipsub-port", str(constants.LEAN_QUIC_PORT_NUM), - "--node-id", node["node_name"], - "--node-key", "{0}/{1}.key".format(NODE_KEY_MOUNT, node["node_name"]), - "--http-address", "0.0.0.0", - "--api-port", str(constants.LEAN_API_PORT_NUM), - "--metrics-port", str(constants.LEAN_METRICS_PORT_NUM), + "--genesis", + "{0}/config.yaml".format(GENESIS_MOUNT), + "--validators", + "{0}/annotated_validators.yaml".format(GENESIS_MOUNT), + "--bootnodes", + "{0}/nodes.yaml".format(GENESIS_MOUNT), + "--validator-config", + "{0}/validator-config.yaml".format(GENESIS_MOUNT), + "--hash-sig-keys-dir", + HASH_SIG_MOUNT, + "--data-dir", + DATA_DIR, + "--gossipsub-port", + str(constants.LEAN_QUIC_PORT_NUM), + "--node-id", + node["node_name"], + "--node-key", + "{0}/{1}.key".format(NODE_KEY_MOUNT, node["node_name"]), + "--http-address", + "0.0.0.0", + "--api-port", + str(constants.LEAN_API_PORT_NUM), + "--metrics-port", + str(constants.LEAN_METRICS_PORT_NUM), ] if node["is_aggregator"]: cmd_parts.append("--is-aggregator") @@ -155,10 +169,12 @@ def start(plan, node, service, genesis_artifact, hash_sig_artifact): ) api_url = "http://{0}:{1}".format( - service.ip_address, constants.LEAN_API_PORT_NUM, + service.ip_address, + constants.LEAN_API_PORT_NUM, ) metrics_url = "http://{0}:{1}/metrics".format( - service.ip_address, constants.LEAN_METRICS_PORT_NUM, + service.ip_address, + constants.LEAN_METRICS_PORT_NUM, ) return lean_context.new_lean_context( diff --git a/src/lean/lean_launcher.star b/src/lean/lean_launcher.star index 1697c7c42..52384ed8e 100644 --- a/src/lean/lean_launcher.star +++ b/src/lean/lean_launcher.star @@ -28,6 +28,7 @@ p2p_keys = import_module( ethlambda_launcher = import_module("./ethlambda/ethlambda_launcher.star") ream_launcher = import_module("./ream/ream_launcher.star") zeam_launcher = import_module("./zeam/zeam_launcher.star") +metrics_launcher = import_module("./metrics/metrics_launcher.star") def _launcher_for(lean_type): @@ -192,4 +193,11 @@ def launch(plan, lean_participants, lean_network_params): genesis.genesis_time, ) ) + + # Phase 4: launch Prometheus + Grafana scraping every Lean node's + # /metrics endpoint. Enabled by default; set + # lean_network_params.metrics_enabled: false to skip. + if lean_network_params.get("metrics_enabled", True): + metrics_launcher.launch(plan, contexts) + return contexts diff --git a/src/lean/metrics/grafana/dashboards/client-dashboard.json b/src/lean/metrics/grafana/dashboards/client-dashboard.json new file mode 100644 index 000000000..9c4aad3d8 --- /dev/null +++ b/src/lean/metrics/grafana/dashboards/client-dashboard.json @@ -0,0 +1,6803 @@ +{ + "__inputs": [ + { + "name": "datasource", + "label": "prometheus", + "description": "", + "type": "datasource", + "pluginId": "prometheus", + "pluginName": "Prometheus" + } + ], + "__elements": {}, + "__requires": [ + { + "type": "grafana", + "id": "grafana", + "name": "Grafana", + "version": "12.3.2" + }, + { + "type": "panel", + "id": "piechart", + "name": "Pie chart", + "version": "" + }, + { + "type": "datasource", + "id": "prometheus", + "name": "Prometheus", + "version": "1.0.0" + }, + { + "type": "panel", + "id": "stat", + "name": "Stat", + "version": "" + }, + { + "type": "panel", + "id": "table", + "name": "Table", + "version": "" + }, + { + "type": "panel", + "id": "text", + "name": "Text", + "version": "" + }, + { + "type": "panel", + "id": "timeseries", + "name": "Time series", + "version": "" + } + ], + "annotations": { + "list": [ + { + "builtIn": 1, + "datasource": { + "type": "grafana", + "uid": "-- Grafana --" + }, + "enable": true, + "hide": true, + "iconColor": "rgba(0, 211, 255, 1)", + "name": "Annotations & Alerts", + "type": "dashboard" + } + ] + }, + "editable": true, + "fiscalYearStartMonth": 0, + "graphTooltip": 0, + "links": [], + "panels": [ + { + "collapsed": false, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 0 + }, + "id": 16, + "panels": [], + "title": "Overview", + "type": "row" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "fieldConfig": { + "defaults": { + "color": { + "fixedColor": "super-light-green", + "mode": "fixed" + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "dateTimeAsIso" + }, + "overrides": [] + }, + "gridPos": { + "h": 4, + "w": 4, + "x": 0, + "y": 1 + }, + "id": 76, + "options": { + "colorMode": "value", + "graphMode": "none", + "justifyMode": "center", + "orientation": "auto", + "percentChangeColorMode": "standard", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "showPercentChange": false, + "text": { + "valueSize": 20 + }, + "textMode": "value", + "wideLayout": true + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "exemplar": false, + "expr": "max by(network) (lean_node_start_time_seconds{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"} * 1000)", + "instant": false, + "legendFormat": "__auto", + "range": true, + "refId": "A" + } + ], + "title": "Latest start time", + "type": "stat" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 4, + "w": 4, + "x": 4, + "y": 1 + }, + "id": 103, + "options": { + "colorMode": "none", + "graphMode": "none", + "justifyMode": "auto", + "orientation": "auto", + "percentChangeColorMode": "standard", + "reduceOptions": { + "calcs": [], + "fields": "", + "values": false + }, + "showPercentChange": false, + "textMode": "auto", + "wideLayout": true + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "exemplar": false, + "expr": "sum by(network) (lean_validators_count{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"})", + "instant": true, + "legendFormat": "__auto", + "range": false, + "refId": "A" + } + ], + "title": "Total number of validators", + "type": "stat" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 4, + "w": 4, + "x": 8, + "y": 1 + }, + "id": 40, + "options": { + "colorMode": "none", + "graphMode": "none", + "justifyMode": "auto", + "orientation": "auto", + "percentChangeColorMode": "standard", + "reduceOptions": { + "calcs": [], + "fields": "", + "values": false + }, + "showPercentChange": false, + "textMode": "auto", + "wideLayout": true + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "exemplar": false, + "expr": "max by(network) (lean_attestation_committee_count{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"})", + "instant": true, + "legendFormat": "__auto", + "range": false, + "refId": "A" + } + ], + "title": "Number of attestation committees", + "type": "stat" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Number of validators attached to each node", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + } + }, + "fieldMinMax": false, + "mappings": [] + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "Total" + }, + "properties": [ + { + "id": "custom.hideFrom", + "value": { + "legend": false, + "tooltip": true, + "viz": true + } + } + ] + } + ] + }, + "gridPos": { + "h": 8, + "w": 4, + "x": 16, + "y": 1 + }, + "id": 51, + "options": { + "legend": { + "displayMode": "table", + "placement": "right", + "showLegend": true, + "values": [] + }, + "pieType": "donut", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "sort": "desc", + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "exemplar": false, + "expr": "sum (lean_validators_count)", + "hide": false, + "instant": true, + "legendFormat": "Total", + "range": false, + "refId": "A" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "disableTextWrap": false, + "editorMode": "code", + "exemplar": false, + "expr": "sum by (network, job, instance) (lean_validators_count{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"})", + "fullMetaSearch": false, + "hide": false, + "includeNullMetadata": true, + "instant": true, + "interval": "", + "legendFormat": "{{job}}", + "range": false, + "refId": "B", + "useBackend": false + } + ], + "title": "Validators list", + "type": "piechart" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "fieldConfig": { + "defaults": { + "color": { + "fixedColor": "super-light-green", + "mode": "fixed" + }, + "custom": { + "align": "auto", + "cellOptions": { + "type": "auto" + }, + "footer": { + "reducers": [] + }, + "inspect": false + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 4, + "x": 12, + "y": 1 + }, + "id": 89, + "options": { + "cellHeight": "sm", + "showHeader": false + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "exemplar": false, + "expr": "max by(network,job,instance) (lean_is_aggregator{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}) > 0", + "instant": true, + "legendFormat": "{{job}}", + "range": false, + "refId": "A" + } + ], + "title": "Aggregators", + "transformations": [ + { + "id": "labelsToFields", + "options": {} + }, + { + "id": "filterFieldsByName", + "options": { + "include": { + "names": [ + "job" + ] + } + } + } + ], + "type": "table" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "custom": { + "align": "auto", + "cellOptions": { + "type": "auto" + }, + "footer": { + "reducers": [] + }, + "inspect": false + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "name" + }, + "properties": [ + { + "id": "custom.width", + "value": 112 + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "job" + }, + "properties": [ + { + "id": "custom.width", + "value": 116 + } + ] + } + ] + }, + "gridPos": { + "h": 8, + "w": 4, + "x": 20, + "y": 1 + }, + "id": 75, + "options": { + "cellHeight": "sm", + "showHeader": true + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "exemplar": false, + "expr": "last_over_time(lean_node_info{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[1m])", + "instant": true, + "legendFormat": "__auto", + "range": false, + "refId": "A" + } + ], + "title": "Node info", + "transformations": [ + { + "id": "labelsToFields", + "options": { + "keepLabels": [ + "name", + "version", + "job" + ] + } + }, + { + "id": "merge", + "options": {} + }, + { + "id": "filterFieldsByName", + "options": { + "include": { + "names": [ + "job", + "version" + ] + } + } + }, + { + "id": "organize", + "options": { + "excludeByName": {}, + "includeByName": {}, + "indexByName": {}, + "renameByName": { + "job": "Node", + "name": "Client", + "version": "Version" + } + } + } + ], + "type": "table" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 4, + "w": 4, + "x": 0, + "y": 5 + }, + "id": 88, + "options": { + "colorMode": "none", + "graphMode": "none", + "justifyMode": "auto", + "orientation": "auto", + "percentChangeColorMode": "standard", + "reduceOptions": { + "calcs": [], + "fields": "", + "values": false + }, + "showPercentChange": false, + "textMode": "auto", + "wideLayout": true + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "exemplar": false, + "expr": "max by(network) (lean_latest_finalized_slot{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"})", + "instant": true, + "legendFormat": "__auto", + "range": false, + "refId": "A" + } + ], + "title": "Latest finalized slot", + "type": "stat" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 4, + "w": 4, + "x": 4, + "y": 5 + }, + "id": 86, + "options": { + "colorMode": "none", + "graphMode": "none", + "justifyMode": "auto", + "orientation": "auto", + "percentChangeColorMode": "standard", + "reduceOptions": { + "calcs": [], + "fields": "", + "values": false + }, + "showPercentChange": false, + "textMode": "auto", + "wideLayout": true + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "exemplar": false, + "expr": "max by(network) (lean_latest_justified_slot{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"})", + "instant": true, + "legendFormat": "__auto", + "range": false, + "refId": "A" + } + ], + "title": "Latest justified slot", + "type": "stat" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 4, + "w": 4, + "x": 8, + "y": 5 + }, + "id": 87, + "options": { + "colorMode": "none", + "graphMode": "none", + "justifyMode": "auto", + "orientation": "auto", + "percentChangeColorMode": "standard", + "reduceOptions": { + "calcs": [], + "fields": "", + "values": false + }, + "showPercentChange": false, + "textMode": "auto", + "wideLayout": true + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "exemplar": false, + "expr": "max by(network) (lean_head_slot{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"})", + "instant": true, + "legendFormat": "__auto", + "range": false, + "refId": "A" + } + ], + "title": "Head slot", + "type": "stat" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 14, + "w": 6, + "x": 0, + "y": 9 + }, + "id": 33, + "options": { + "legend": { + "calcs": [ + "max", + "lastNotNull" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "sum by (network, job, instance)(lean_latest_finalized_slot{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"})", + "interval": "", + "legendFormat": "{{job}}", + "range": true, + "refId": "A" + } + ], + "title": " Latest finalized slot", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 14, + "w": 6, + "x": 6, + "y": 9 + }, + "id": 34, + "options": { + "legend": { + "calcs": [ + "max", + "lastNotNull" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": " sum by (network, job, instance) (lean_latest_justified_slot{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"})", + "legendFormat": "{{job}}", + "range": true, + "refId": "A" + } + ], + "title": "Latest justified slot", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "fieldMinMax": false, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 14, + "w": 6, + "x": 12, + "y": 9 + }, + "id": 35, + "options": { + "legend": { + "calcs": [ + "lastNotNull" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": " sum by (network, job, instance) (lean_head_slot{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"})", + "legendFormat": "{{job}}", + "range": true, + "refId": "A" + } + ], + "title": "Head slot", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "fieldMinMax": false, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 14, + "w": 6, + "x": 18, + "y": 9 + }, + "id": 66, + "options": { + "legend": { + "calcs": [ + "lastNotNull" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": " sum by (network, job, instance) (lean_current_slot{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"})", + "legendFormat": "{{job}}", + "range": true, + "refId": "A" + } + ], + "title": "Current slot", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Total number of processed slots in state transition function", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "decimals": 1, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 23 + }, + "id": 72, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "exemplar": false, + "expr": "changes(lean_node_start_time_seconds{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[1m])", + "instant": false, + "interval": "", + "legendFormat": "{{job}}", + "range": true, + "refId": "A" + } + ], + "title": "Start time", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "fieldMinMax": false, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "Total" + }, + "properties": [] + } + ] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 23 + }, + "id": 90, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "disableTextWrap": false, + "editorMode": "code", + "exemplar": false, + "expr": "sum by (network, job, instance) (lean_attestation_committee_subnet{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"})", + "fullMetaSearch": false, + "hide": false, + "includeNullMetadata": true, + "instant": false, + "interval": "", + "legendFormat": "{{job}}", + "range": true, + "refId": "B", + "useBackend": false + } + ], + "title": "Attestation committee subnet", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "fixedColor": "text", + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "decimals": 0, + "fieldMinMax": false, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "Total" + }, + "properties": [] + } + ] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 31 + }, + "id": 44, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "disableTextWrap": false, + "editorMode": "code", + "expr": " sum by (network, job, instance) (lean_connected_peers{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"})", + "fullMetaSearch": false, + "hide": false, + "includeNullMetadata": true, + "instant": false, + "interval": "", + "legendFormat": "{{job}}", + "range": true, + "refId": "B", + "useBackend": false + } + ], + "title": "Connected peers per node", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "drawStyle": "line", + "lineInterpolation": "stepAfter", + "lineWidth": 2, + "fillOpacity": 0, + "spanNulls": false, + "showPoints": "never", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + } + }, + "mappings": [ + { + "options": { + "0": { + "text": "idle" + }, + "1": { + "text": "syncing" + }, + "2": { + "text": "synced" + } + }, + "type": "value" + } + ], + "min": 0, + "max": 2, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 31 + }, + "id": 98, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "max by (network, job, instance) (\n (lean_node_sync_status{status=\"synced\", network=~\"$network\", job=~\"$job\", instance=~\"$instance\"} == 1) * 2\n or\n (lean_node_sync_status{status=\"syncing\", network=~\"$network\", job=~\"$job\", instance=~\"$instance\"} == 1) * 1\n or\n (lean_node_sync_status{status=\"idle\", network=~\"$network\", job=~\"$job\", instance=~\"$instance\"} == 1) * 0\n)", + "instant": false, + "interval": "", + "legendFormat": "{{job}}", + "range": true, + "refId": "A" + } + ], + "title": "Node sync status", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Total number of processed slots in state transition function", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "decimals": 1, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 39 + }, + "id": 72, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "avg_over_time(\n sum by (network, job, instance) (increase(lean_finalizations_total{network=~\"$network\", job=~\"$job\", instance=~\"$instance\", result=\"success\"}[1m]))[5m:]\n)", + "legendFormat": "{{job}}", + "range": true, + "refId": "A" + } + ], + "title": "Finalizations - Success", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "decimals": 1, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 39 + }, + "id": 73, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "avg_over_time(\n sum by (network, job, instance) (increase(lean_finalizations_total{network=~\"$network\", job=~\"$job\", instance=~\"$instance\", result=\"error\"}[1m]))[5m:]\n)", + "legendFormat": "{{job}}", + "range": true, + "refId": "A" + } + ], + "title": "Finalizations - Errors", + "type": "timeseries" + }, + { + "collapsed": false, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 47 + }, + "id": 57, + "panels": [], + "title": "Finalization/Justification Delay", + "type": "row" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 10, + "w": 8, + "x": 0, + "y": 48 + }, + "id": 30, + "options": { + "legend": { + "calcs": [ + "min", + "max", + "mean", + "last" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "avg_over_time(lean_head_slot{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[5m]) - avg_over_time(lean_latest_finalized_slot{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[5m])", + "interval": "", + "legendFormat": "{{job}}", + "range": true, + "refId": "A" + } + ], + "title": "Head - Finalized delay (slots)", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 10, + "w": 8, + "x": 8, + "y": 48 + }, + "id": 29, + "options": { + "legend": { + "calcs": [ + "min", + "max", + "mean", + "last" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "avg_over_time(lean_head_slot{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[5m]) - avg_over_time(lean_latest_justified_slot{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[5m])", + "interval": "", + "legendFormat": "{{job}}", + "range": true, + "refId": "A" + } + ], + "title": "Head - Justified delay (slots)", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 10, + "w": 8, + "x": 16, + "y": 48 + }, + "id": 31, + "options": { + "legend": { + "calcs": [ + "min", + "max", + "mean", + "last" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "avg_over_time(lean_latest_justified_slot{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[5m]) - avg_over_time(lean_latest_finalized_slot{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[5m])", + "interval": "", + "legendFormat": "{{job}}", + "range": true, + "refId": "A" + } + ], + "title": "Justified - Finalized delay (slots)", + "type": "timeseries" + }, + { + "collapsed": false, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 58 + }, + "id": 53, + "panels": [], + "title": "Peers", + "type": "row" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 9, + "w": 12, + "x": 0, + "y": 59 + }, + "id": 54, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": " sum by (network, job, instance) (lean_connected_peers{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"})", + "interval": "", + "legendFormat": "{{job}}", + "range": true, + "refId": "A" + } + ], + "title": "Connected peers per node", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 9, + "w": 12, + "x": 12, + "y": 59 + }, + "id": 73, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "exemplar": false, + "expr": " sum by (network, job, instance, client) (lean_connected_peers{network=~\"$network\", job=~\"$job\", instance=~\"$instance\", client!=\"\"})", + "instant": false, + "interval": "", + "legendFormat": "{{job}} - {{client}}", + "range": true, + "refId": "A" + } + ], + "title": "Connected peers per node (detailed)", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "decimals": 1, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 9, + "w": 12, + "x": 0, + "y": 68 + }, + "id": 55, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "sum by (network, job, instance) (\n increase(lean_peer_connection_events_total{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])\n )", + "interval": "", + "legendFormat": "{{job}} {{source}}", + "range": true, + "refId": "A" + } + ], + "title": "Peer connection events", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "decimals": 1, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 9, + "w": 12, + "x": 12, + "y": 68 + }, + "id": 56, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "sum by (network, job, instance) (\n increase(lean_peer_disconnection_events_total{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])\n )", + "interval": "", + "legendFormat": "{{job}} {{source}}", + "range": true, + "refId": "A" + } + ], + "title": "Peer disconnection events", + "type": "timeseries" + }, + { + "collapsed": false, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 77 + }, + "id": 99, + "panels": [], + "title": "Gossip messages", + "type": "row" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "bytes" + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 0, + "y": 78 + }, + "id": 100, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.99, \n rate(lean_gossip_block_size_bytes_bucket{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])\n)", + "interval": "", + "legendFormat": "{{job}}", + "range": true, + "refId": "A" + } + ], + "title": "Bytes size of a gossip block message", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "bytes" + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 8, + "y": 78 + }, + "id": 101, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.99, \n rate(lean_gossip_attestation_size_bytes_bucket{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])\n)", + "interval": "", + "legendFormat": "{{job}}", + "range": true, + "refId": "A" + } + ], + "title": "Bytes size of a gossip attestation message", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "bytes" + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 16, + "y": 78 + }, + "id": 102, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.99, \n rate(lean_gossip_aggregation_size_bytes_bucket{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])\n)", + "interval": "", + "legendFormat": "{{job}}", + "range": true, + "refId": "A" + } + ], + "title": "Bytes size of a gossip aggregated attestation message", + "type": "timeseries" + }, + { + "collapsed": false, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 86 + }, + "id": 45, + "panels": [], + "title": "PQ Signatures", + "type": "row" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Total number of individual attestation signatures", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "decimals": 1, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 0, + "y": 87 + }, + "id": 60, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "exemplar": false, + "expr": "avg_over_time(\n sum by (network, job, instance) (\n increase(lean_pq_sig_attestation_signatures_total{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])\n )[5m:]\n)", + "instant": false, + "legendFormat": "{{job}}", + "range": true, + "refId": "A" + } + ], + "title": "Total number of attestation signatures", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Total number of valid individual attestation signatures", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "decimals": 1, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 8, + "y": 87 + }, + "id": 64, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "exemplar": false, + "expr": "avg_over_time(\n sum by (network, job, instance) (\n increase(lean_pq_sig_attestation_signatures_valid_total{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])\n )[5m:]\n)", + "instant": false, + "legendFormat": "{{job}}", + "range": true, + "refId": "A" + } + ], + "title": "Total number of valid attestation signatures", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Total number of invalid individual attestation signatures", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "decimals": 1, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 16, + "y": 87 + }, + "id": 65, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "exemplar": false, + "expr": "avg_over_time(\n sum by (network, job, instance) (\n increase(lean_pq_sig_attestation_signatures_invalid_total{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])\n )[5m:]\n)", + "instant": false, + "legendFormat": "{{job}}", + "range": true, + "refId": "A" + } + ], + "title": "Total number of invalid attestation signatures", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "decimals": 1, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "s" + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 95 + }, + "id": 46, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "exemplar": false, + "expr": "histogram_quantile(0.99, rate(lean_pq_sig_attestation_signing_time_seconds_bucket{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval]))", + "instant": false, + "legendFormat": "{{job}}", + "range": true, + "refId": "A" + } + ], + "title": "Time taken to sign an attestation", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "decimals": 1, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "s" + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 95 + }, + "id": 47, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "exemplar": false, + "expr": "histogram_quantile(0.99, rate(lean_pq_sig_attestation_verification_time_seconds_bucket{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval]))", + "instant": false, + "legendFormat": "{{job}}", + "range": true, + "refId": "A" + } + ], + "title": "Time taken to verify an attestation signature", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "decimals": 1, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 103 + }, + "id": 79, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "exemplar": false, + "expr": "avg_over_time(\n sum by (network, job, instance) (\n increase(lean_pq_sig_aggregated_signatures_total{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])\n )[5m:]\n)", + "instant": false, + "legendFormat": "{{job}}", + "range": true, + "refId": "A" + } + ], + "title": "Total number of aggregated signatures", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "decimals": 1, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 103 + }, + "id": 61, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "exemplar": false, + "expr": "avg_over_time(\n sum by (network, job, instance) (\n increase(lean_pq_sig_attestations_in_aggregated_signatures_total{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])\n )[5m:]\n)", + "instant": false, + "legendFormat": "{{job}}", + "range": true, + "refId": "A" + } + ], + "title": "Total number of attestations included into aggregated signatures", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "decimals": 1, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 111 + }, + "id": 77, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "exemplar": false, + "expr": "avg_over_time(\n sum by (network, job, instance) (\n increase(lean_pq_sig_aggregated_signatures_valid_total{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])\n )[5m:]\n)", + "instant": false, + "legendFormat": "{{job}}", + "range": true, + "refId": "A" + } + ], + "title": "Total number of valid aggregated signatures", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "decimals": 1, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 111 + }, + "id": 78, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "exemplar": false, + "expr": "avg_over_time(\n sum by (network, job, instance) (\n increase(lean_pq_sig_aggregated_signatures_invalid_total{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])\n )[5m:]\n)", + "instant": false, + "legendFormat": "{{job}}", + "range": true, + "refId": "A" + } + ], + "title": "Total number of invalid aggregated signatures", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "decimals": 1, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "s" + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 119 + }, + "id": 62, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "exemplar": false, + "expr": "histogram_quantile(0.99, rate(lean_pq_sig_aggregated_signatures_building_time_seconds_bucket{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval]))", + "instant": false, + "legendFormat": "{{job}}", + "range": true, + "refId": "A" + } + ], + "title": "Time taken to build an aggregated signature", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "decimals": 1, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "s" + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 119 + }, + "id": 63, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "exemplar": false, + "expr": "histogram_quantile(0.99, rate(lean_pq_sig_aggregated_signatures_verification_time_seconds_bucket{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval]))", + "instant": false, + "legendFormat": "{{job}}", + "range": true, + "refId": "A" + } + ], + "title": "Time taken to verify an aggregated signature", + "type": "timeseries" + }, + { + "collapsed": false, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 127 + }, + "id": 91, + "panels": [], + "title": "Block production", + "type": "row" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 128 + }, + "id": 19, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.99, \n rate(lean_block_aggregated_payloads_bucket{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])\n)", + "interval": "", + "legendFormat": "{{job}}", + "range": true, + "refId": "A" + } + ], + "title": "Number of aggregated payloads", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "s" + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 128 + }, + "id": 93, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.99, \n rate(lean_block_building_payload_aggregation_time_seconds_bucket{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])\n)", + "interval": "", + "legendFormat": "{{job}}", + "range": true, + "refId": "A" + } + ], + "title": "Time taken to build aggregated payloads", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "decimals": 1, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 0, + "y": 136 + }, + "id": 80, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "exemplar": false, + "expr": " avg_over_time (\n sum by (network, job, instance) (\n (lean_block_building_success_total{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"})\n )[2m:]\n)", + "instant": false, + "interval": "", + "legendFormat": "{{job}}", + "range": true, + "refId": "A" + } + ], + "title": "Successful block builds", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "decimals": 1, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 8, + "y": 136 + }, + "id": 97, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "exemplar": false, + "expr": " avg_over_time (\n sum by (network, job, instance) (\n (lean_block_building_failures_total{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"})\n )[2m:]\n)", + "instant": false, + "interval": "", + "legendFormat": "{{job}}", + "range": true, + "refId": "A" + } + ], + "title": "Failed block builds", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "s" + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 16, + "y": 136 + }, + "id": 94, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.99, \n rate(lean_block_building_time_seconds_bucket{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])\n)", + "interval": "", + "legendFormat": "{{job}}", + "range": true, + "refId": "A" + } + ], + "title": "Time taken to build a block", + "type": "timeseries" + }, + { + "collapsed": false, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 144 + }, + "id": 17, + "panels": [], + "title": "Fork-Choice", + "type": "row" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Time taken to process block in fork-choice", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "s" + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 145 + }, + "id": 92, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.99, \n rate(lean_fork_choice_block_processing_time_seconds_bucket{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])\n)", + "interval": "", + "legendFormat": "{{job}}", + "range": true, + "refId": "A" + } + ], + "title": "Block processing time", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Time taken to process block in fork-choice", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "s" + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 145 + }, + "id": 85, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.99, \n rate(lean_committee_signatures_aggregation_time_seconds_bucket{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])\n)", + "interval": "", + "legendFormat": "{{job}}", + "range": true, + "refId": "A" + } + ], + "title": "Time taken to aggregate committee signatures", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "decimals": 1, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 153 + }, + "id": 68, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "sum by (network, job, instance) (\n increase(lean_fork_choice_reorgs_total{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])\n)", + "interval": "", + "legendFormat": "{{job}} {{source}}", + "range": true, + "refId": "A" + } + ], + "title": "Total number of fork choice reorgs", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "decimals": 1, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 153 + }, + "id": 69, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.99, rate(lean_fork_choice_reorg_depth_bucket{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval]))", + "interval": "", + "legendFormat": "{{job}} {{source}}", + "range": true, + "refId": "A" + } + ], + "title": "Depth of fork choice reorgs (in blocks)", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "decimals": 1, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 0, + "y": 161 + }, + "id": 96, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "exemplar": false, + "expr": " avg_over_time (\n sum by (network, job, instance) (\n (lean_gossip_signatures{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"})\n )[2m:]\n)", + "instant": false, + "interval": "", + "legendFormat": "{{job}}", + "range": true, + "refId": "A" + } + ], + "title": "Number of gossip signatures", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "decimals": 1, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 8, + "y": 161 + }, + "id": 81, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "exemplar": false, + "expr": "avg_over_time(\n sum by (network, job, instance) (\n (lean_latest_new_aggregated_payloads{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"})\n )[2m:]\n)", + "instant": false, + "interval": "", + "legendFormat": "{{job}}", + "range": true, + "refId": "A" + } + ], + "title": "Number of new aggregated payloads", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "decimals": 1, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 16, + "y": 161 + }, + "id": 82, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "exemplar": false, + "expr": "avg_over_time(\n sum by (network, job, instance) (\n (lean_latest_known_aggregated_payloads{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"})\n )[1m:]\n)", + "instant": false, + "interval": "", + "legendFormat": "{{job}}", + "range": true, + "refId": "A" + } + ], + "title": "Number of known aggregated payloads", + "type": "timeseries" + }, + { + "collapsed": false, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 169 + }, + "id": 8, + "panels": [], + "title": "Attestations", + "type": "row" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "decimals": 1, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 0, + "y": 170 + }, + "id": 9, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "avg_over_time(\n sum by (network, job, instance) (\n increase(lean_attestations_valid_total{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])\n )[5m:]\n)", + "interval": "", + "legendFormat": "{{job}} {{source}}", + "range": true, + "refId": "A" + } + ], + "title": "FC Valid attestations", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 8, + "y": 170 + }, + "id": 10, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "avg_over_time(\n sum by (network, job, instance) (increase(lean_attestations_invalid_total{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval]))[5m:]\n)", + "legendFormat": "{{job}} {{source}}", + "range": true, + "refId": "A" + } + ], + "title": "FC Invalid attestations", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "s" + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 16, + "y": 170 + }, + "id": 11, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.99, rate(lean_attestation_validation_time_seconds_bucket{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval]))", + "legendFormat": "{{job}}", + "range": true, + "refId": "A" + } + ], + "title": "FC Attestations validation time", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "fieldMinMax": false, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 0, + "y": 178 + }, + "id": 67, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": " sum by (network, job, instance) (lean_safe_target_slot{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"})", + "legendFormat": "{{job}}", + "range": true, + "refId": "A" + } + ], + "title": "Safe target slot", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Total number of attestations processed in state transition function", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "decimals": 1, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 8, + "y": 178 + }, + "id": 27, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "exemplar": false, + "expr": "avg_over_time(\n sum by (network, job, instance) (increase(lean_state_transition_attestations_processed_total{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval]))[5m:]\n)", + "instant": false, + "legendFormat": "{{job}}", + "range": true, + "refId": "A" + } + ], + "title": "STF Processed attestations", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Time taken to process attestations in state transition function", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "s" + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 16, + "y": 178 + }, + "id": 28, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.99,\n rate(lean_state_transition_attestations_processing_time_seconds_bucket{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])\n)", + "legendFormat": "{{job}}", + "range": true, + "refId": "A" + } + ], + "title": "STF Attestations processing time", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "decimals": 1, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "s" + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 186 + }, + "id": 83, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.99, rate(lean_attestations_production_time_seconds_bucket{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval]))", + "interval": "", + "legendFormat": "{{job}} {{source}}", + "range": true, + "refId": "A" + } + ], + "title": "Time taken to produce attestation", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "s" + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 186 + }, + "id": 84, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.99, rate(lean_committee_signatures_aggregation_time_seconds_bucket{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval]))", + "legendFormat": "{{job}}", + "range": true, + "refId": "A" + } + ], + "title": "Time taken to aggregate committee signatures", + "type": "timeseries" + }, + { + "collapsed": false, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 194 + }, + "id": 21, + "panels": [], + "title": "State Transition", + "type": "row" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Time taken to process state transition function", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "s" + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 195 + }, + "id": 23, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.99, \n rate(lean_state_transition_time_seconds_bucket{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])\n)", + "interval": "", + "legendFormat": "{{job}}", + "range": true, + "refId": "A" + } + ], + "title": "State transition time", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Time taken to process block in state transition function", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "s" + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 195 + }, + "id": 24, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.99, \n rate(lean_state_transition_block_processing_time_seconds_bucket{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])\n)", + "interval": "", + "legendFormat": "{{job}}", + "range": true, + "refId": "A" + } + ], + "title": "Block processing time", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Total number of processed slots in state transition function", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "decimals": 1, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 203 + }, + "id": 25, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "avg_over_time(\n sum by (network, job, instance) (increase(lean_state_transition_slots_processed_total{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval]))[5m:]\n)", + "legendFormat": "{{job}}", + "range": true, + "refId": "A" + } + ], + "title": "Processed slots", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Time taken to process slots in state transition function", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "s" + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 203 + }, + "id": 26, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.99,\n rate(lean_state_transition_slots_processing_time_seconds_bucket{network=~\"$network\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])\n)", + "legendFormat": "{{job}}", + "range": true, + "refId": "A" + } + ], + "title": "Slots processing time", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Total number of processed slots in state transition function", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "decimals": 1, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 211 + }, + "id": 70, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "avg_over_time(\n sum by (network, job, instance) (increase(lean_finalizations_total{network=~\"$network\", job=~\"$job\", instance=~\"$instance\", result=\"success\"}[1m]))[5m:]\n)", + "legendFormat": "{{job}}", + "range": true, + "refId": "A" + } + ], + "title": "Finalizations - Success", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "decimals": 1, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 211 + }, + "id": 71, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "12.3.2", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "avg_over_time(\n sum by (network, job, instance) (increase(lean_finalizations_total{network=~\"$network\", job=~\"$job\", instance=~\"$instance\", result=\"error\"}[1m]))[5m:]\n)", + "legendFormat": "{{job}}", + "range": true, + "refId": "A" + } + ], + "title": "Finalizations - Errors", + "type": "timeseries" + } + ], + "preload": false, + "schemaVersion": 42, + "tags": [ + "interop", + "Client" + ], + "templating": { + "list": [ + { + "allowCustomValue": false, + "current": {}, + "definition": "label_values(network)", + "label": "Network", + "name": "network", + "options": [], + "query": { + "qryType": 1, + "query": "label_values(network)", + "refId": "PrometheusVariableQueryEditor-VariableQuery" + }, + "refresh": 1, + "regex": "", + "type": "query" + }, + { + "allowCustomValue": false, + "current": {}, + "definition": "label_values(lean_node_info{network=~\"$network\"}, job)", + "includeAll": true, + "label": "Job", + "multi": true, + "name": "job", + "options": [], + "query": { + "qryType": 1, + "query": "label_values(lean_node_info{network=~\"$network\"}, job)", + "refId": "PrometheusVariableQueryEditor-VariableQuery" + }, + "refresh": 1, + "regex": ".*ethlambda.*|.*gean.*|.*grandine.*|.*lantern.*|.*lighthouse.*|.*qlean.*|.*ream.*|.*zeam.*", + "type": "query" + }, + { + "allowCustomValue": false, + "current": {}, + "definition": "label_values(lean_node_info{network=~\"$network\", job=~\"$job\"}, instance)", + "includeAll": true, + "label": "Instance", + "multi": true, + "name": "instance", + "options": [], + "query": { + "qryType": 1, + "query": "label_values(lean_node_info{network=~\"$network\", job=~\"$job\"}, instance)", + "refId": "PrometheusVariableQueryEditor-VariableQuery" + }, + "refresh": 1, + "regex": "", + "type": "query" + }, + { + "allowCustomValue": false, + "current": { + "text": "", + "value": "${datasource}", + "selected": true + }, + "name": "datasource", + "options": [], + "query": "prometheus", + "refresh": 1, + "regex": "", + "type": "datasource" + } + ] + }, + "time": { + "from": "now-1h", + "to": "now" + }, + "timepicker": {}, + "timezone": "utc", + "title": "Lean Ethereum Clients Dashboard", + "uid": "lean-ethereum-clients-dashboard", + "version": 25, + "weekStart": "", + "id": null +} diff --git a/src/lean/metrics/metrics_launcher.star b/src/lean/metrics/metrics_launcher.star new file mode 100644 index 000000000..4d161c443 --- /dev/null +++ b/src/lean/metrics/metrics_launcher.star @@ -0,0 +1,196 @@ +""" +Prometheus + Grafana for the Lean Ethereum devnet. + +Mirrors the docker-compose-metrics.yaml stack shipped in +blockblaz/lean-quickstart (prom v3.8.0, grafana v12.3.2, anonymous admin +login). Inside the Kurtosis enclave we scrape by service DNS name + port +instead of host.docker.internal:port, so no `--add-host` hackery is needed. + +Layout: + * One Prometheus service named "lean-prometheus" scraping every + lean--:5054/metrics target plus its own /metrics. + * One Grafana service named "lean-grafana" provisioning the Prometheus + datasource and the upstream Lean client dashboard. + +The whole thing is enabled by default when there are lean_participants; +add `additional_services: [grafana, prometheus]` to network_params.yaml +or set `lean_network_params.metrics_enabled: false` to skip it. +""" + +constants = import_module("../../package_io/constants.star") + +PROMETHEUS_IMAGE = "prom/prometheus:v3.8.0" +GRAFANA_IMAGE = "grafana/grafana:12.3.2" +PROMETHEUS_SERVICE = "lean-prometheus" +GRAFANA_SERVICE = "lean-grafana" +PROMETHEUS_PORT = 9090 +GRAFANA_PORT = 3000 + +DASHBOARD_REL = "src/lean/metrics/grafana/dashboards/client-dashboard.json" + + +def launch(plan, lean_contexts): + """Spin up Prometheus + Grafana for the given Lean nodes. + + Args: + plan: Kurtosis plan. + lean_contexts: list of `lean_context.new_lean_context` structs. + Empty / None disables the stack. + """ + if not lean_contexts: + return None + + prometheus_artifact = _render_prometheus_config(plan, lean_contexts) + grafana_provisioning = _render_grafana_provisioning(plan) + grafana_dashboards = plan.upload_files( + src="/" + DASHBOARD_REL, + name="lean-grafana-dashboards", + ) + + prometheus = plan.add_service( + name=PROMETHEUS_SERVICE, + config=ServiceConfig( + image=PROMETHEUS_IMAGE, + cmd=[ + "--config.file=/etc/prometheus/prometheus.yml", + "--storage.tsdb.path=/prometheus", + "--storage.tsdb.retention.time=30d", + "--web.enable-lifecycle", + ], + ports={ + "http": PortSpec( + number=PROMETHEUS_PORT, + transport_protocol="TCP", + application_protocol="http", + wait=None, + ), + }, + files={"/etc/prometheus": prometheus_artifact}, + ), + ) + + grafana = plan.add_service( + name=GRAFANA_SERVICE, + config=ServiceConfig( + image=GRAFANA_IMAGE, + env_vars={ + "GF_SECURITY_ADMIN_USER": "admin", + "GF_SECURITY_ADMIN_PASSWORD": "admin", + "GF_USERS_ALLOW_SIGN_UP": "false", + "GF_AUTH_ANONYMOUS_ENABLED": "true", + "GF_AUTH_ANONYMOUS_ORG_ROLE": "Admin", + "GF_AUTH_DISABLE_LOGIN_FORM": "true", + }, + ports={ + "http": PortSpec( + number=GRAFANA_PORT, + transport_protocol="TCP", + application_protocol="http", + wait=None, + ), + }, + files={ + "/etc/grafana/provisioning": grafana_provisioning, + "/var/lib/grafana/dashboards": grafana_dashboards, + }, + ), + ) + + plan.print( + "Lean metrics ready: prometheus={0}:{1}, grafana={2}:{3}".format( + prometheus.ip_address, + PROMETHEUS_PORT, + grafana.ip_address, + GRAFANA_PORT, + ) + ) + return struct( + prometheus=prometheus, + grafana=grafana, + ) + + +def _render_prometheus_config(plan, lean_contexts): + """Render prometheus.yml with one scrape target per Lean node.""" + template = """# Auto-generated by src/lean/metrics/metrics_launcher.star. +global: + scrape_interval: 15s + evaluation_interval: 15s + external_labels: + monitor: "lean-devnet-metrics" + +scrape_configs: +{{- range .Targets}} + - job_name: "{{.Name}}" + static_configs: + - targets: ["{{.Target}}"] + labels: + client: "{{.Name}}" + instance: "kurtosis" +{{end}} + - job_name: "prometheus" + static_configs: + - targets: ["localhost:9090"] +""" + targets = [] + for ctx in lean_contexts: + targets.append( + { + "Name": ctx.service_name, + # Kurtosis services resolve by service name within the + # enclave network; ports are the internal port (not the + # published host port). All Lean clients standardise on + # 5054 for metrics. + "Target": "{0}:{1}".format(ctx.service_name, ctx.metrics_port), + } + ) + return plan.render_templates( + config={ + "prometheus.yml": struct(template=template, data={"Targets": targets}), + }, + name="lean-prometheus-config", + description="Rendering Lean prometheus.yml", + ) + + +def _render_grafana_provisioning(plan): + """Render Grafana provisioning YAML files (datasource + dashboards).""" + return plan.render_templates( + config={ + "datasources/prometheus.yml": struct( + template="""apiVersion: 1 + +datasources: + - name: prometheus + type: prometheus + uid: P1809F7CD0C75ACF3 + access: proxy + url: http://{0}:{1} + isDefault: true + editable: true +""".format( + PROMETHEUS_SERVICE, PROMETHEUS_PORT + ), + data={}, + ), + "dashboards/dashboards.yml": struct( + template="""apiVersion: 1 + +providers: + - name: "Lean Ethereum Dashboards" + orgId: 1 + folder: "" + type: file + disableDeletion: false + updateIntervalSeconds: 10 + allowUiUpdates: true + options: + path: /var/lib/grafana/dashboards + foldersFromFilesStructure: true +""", + data={}, + ), + }, + name="lean-grafana-provisioning", + description="Rendering Lean Grafana provisioning", + ) diff --git a/src/lean/ream/ream_launcher.star b/src/lean/ream/ream_launcher.star index 0be4b0900..699f64cdb 100644 --- a/src/lean/ream/ream_launcher.star +++ b/src/lean/ream/ream_launcher.star @@ -83,7 +83,9 @@ def start(plan, node, service, genesis_artifact, hash_sig_artifact): "/bin/sh", "-c", "cat > {0}/{1} <<'REAM_EOF'\n{2}\nREAM_EOF".format( - GENESIS_MOUNT, filename, read.output, + GENESIS_MOUNT, + filename, + read.output, ), ], ), @@ -92,20 +94,30 @@ def start(plan, node, service, genesis_artifact, hash_sig_artifact): cmd_parts = [ ENTRYPOINT, - "--data-dir", DATA_DIR, + "--data-dir", + DATA_DIR, "lean_node", - "--network", "{0}/config.yaml".format(GENESIS_MOUNT), + "--network", + "{0}/config.yaml".format(GENESIS_MOUNT), "--validator-registry-path", "{0}/annotated_validators.yaml".format(GENESIS_MOUNT), - "--bootnodes", "{0}/nodes.yaml".format(GENESIS_MOUNT), - "--node-id", node["node_name"], - "--node-key", "{0}/{1}.key".format(NODE_KEY_MOUNT, node["node_name"]), - "--socket-port", str(constants.LEAN_QUIC_PORT_NUM), + "--bootnodes", + "{0}/nodes.yaml".format(GENESIS_MOUNT), + "--node-id", + node["node_name"], + "--node-key", + "{0}/{1}.key".format(NODE_KEY_MOUNT, node["node_name"]), + "--socket-port", + str(constants.LEAN_QUIC_PORT_NUM), "--metrics", - "--metrics-address", "0.0.0.0", - "--metrics-port", str(constants.LEAN_METRICS_PORT_NUM), - "--http-address", "0.0.0.0", - "--http-port", str(constants.LEAN_API_PORT_NUM), + "--metrics-address", + "0.0.0.0", + "--metrics-port", + str(constants.LEAN_METRICS_PORT_NUM), + "--http-address", + "0.0.0.0", + "--http-port", + str(constants.LEAN_API_PORT_NUM), ] if node["is_aggregator"]: cmd_parts.append("--is-aggregator") @@ -129,9 +141,7 @@ def start(plan, node, service, genesis_artifact, hash_sig_artifact): description="Starting ream binary on {0}".format(service_name), ) - api_url = "http://{0}:{1}".format( - service.ip_address, constants.LEAN_API_PORT_NUM - ) + api_url = "http://{0}:{1}".format(service.ip_address, constants.LEAN_API_PORT_NUM) metrics_url = "http://{0}:{1}/metrics".format( service.ip_address, constants.LEAN_METRICS_PORT_NUM, diff --git a/src/lean/zeam/zeam_launcher.star b/src/lean/zeam/zeam_launcher.star index 6b7c552d9..4bc175e71 100644 --- a/src/lean/zeam/zeam_launcher.star +++ b/src/lean/zeam/zeam_launcher.star @@ -119,7 +119,10 @@ def start(plan, node, service, genesis_artifact, hash_sig_artifact): "sh", "-c", "{3} cat > {0}/{1} <<'ZEAM_EOF'\n{2}\nZEAM_EOF".format( - GENESIS_MOUNT, filename, read.output, BUSYBOX, + GENESIS_MOUNT, + filename, + read.output, + BUSYBOX, ), ], ), @@ -129,22 +132,29 @@ def start(plan, node, service, genesis_artifact, hash_sig_artifact): cmd_parts = [ ENTRYPOINT, "node", - "--custom-genesis", GENESIS_MOUNT, + "--custom-genesis", + GENESIS_MOUNT, # zeam's --validator-config accepts either a directory of per-node # validator configs OR the literal sentinel `genesis_bootnode`, # which tells zeam to derive its validator set from the # GENESIS_VALIDATORS list in config.yaml. Pointing at a single # file path (lean-quickstart's `validator-config.yaml`) trips # zeam's "NotDir" check, so use the sentinel here. - "--validator-config", "genesis_bootnode", - "--data-dir", DATA_DIR, - "--node-id", node["node_name"], + "--validator-config", + "genesis_bootnode", + "--data-dir", + DATA_DIR, + "--node-id", + node["node_name"], # zeam (per lean-quickstart's contract) reads --node-key relative to # the custom-genesis dir; we staged it there in the prepare step. - "--node-key", "{0}/{1}.key".format(GENESIS_MOUNT, node["node_name"]), + "--node-key", + "{0}/{1}.key".format(GENESIS_MOUNT, node["node_name"]), "--metrics-enable", - "--api-port", str(constants.LEAN_API_PORT_NUM), - "--metrics-port", str(constants.LEAN_METRICS_PORT_NUM), + "--api-port", + str(constants.LEAN_API_PORT_NUM), + "--metrics-port", + str(constants.LEAN_METRICS_PORT_NUM), ] if node["is_aggregator"]: cmd_parts.append("--is-aggregator") @@ -164,11 +174,10 @@ def start(plan, node, service, genesis_artifact, hash_sig_artifact): description="Starting zeam binary on {0}".format(service_name), ) - api_url = "http://{0}:{1}".format( - service.ip_address, constants.LEAN_API_PORT_NUM - ) + api_url = "http://{0}:{1}".format(service.ip_address, constants.LEAN_API_PORT_NUM) metrics_url = "http://{0}:{1}/metrics".format( - service.ip_address, constants.LEAN_METRICS_PORT_NUM, + service.ip_address, + constants.LEAN_METRICS_PORT_NUM, ) return lean_context.new_lean_context( diff --git a/src/package_io/input_parser.star b/src/package_io/input_parser.star index b853d6125..6d90f9b03 100644 --- a/src/package_io/input_parser.star +++ b/src/package_io/input_parser.star @@ -2657,6 +2657,10 @@ def default_lean_network_params(): # constants. Override to pin a specific PK genesis-tool commit. "genesis_generator_image": "", "hash_sig_cli_image": "", + # When true, start a Prometheus + Grafana stack inside the enclave + # that scrapes every Lean node's `/metrics` endpoint and serves the + # upstream Lean client dashboard at port 3000. + "metrics_enabled": True, } From 9413bacf1e69a364fdb5ab9be2878b8dd1442d0e Mon Sep 17 00:00:00 2001 From: ilitteri Date: Wed, 13 May 2026 19:19:53 -0300 Subject: [PATCH 08/25] Pin metrics services to deterministic host ports (3000 + 9090) Default `ports={}` publishes on a random host port bound to 127.0.0.1. `public_ports={}` lets us pin the host-side number (and Docker binds those on 0.0.0.0 by default), so dashboards have stable URLs and can be reached as http://:3000 / :9090 without an SSH tunnel. If 3000 or 9090 are already in use on the host the run will fail at service start - operators can avoid this by not enabling metrics on shared hosts, or by adding an override via lean_network_params (a follow-up). --- src/lean/metrics/metrics_launcher.star | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/src/lean/metrics/metrics_launcher.star b/src/lean/metrics/metrics_launcher.star index 4d161c443..3d2a1ae8a 100644 --- a/src/lean/metrics/metrics_launcher.star +++ b/src/lean/metrics/metrics_launcher.star @@ -47,6 +47,12 @@ def launch(plan, lean_contexts): name="lean-grafana-dashboards", ) + # Pin host-side ports via public_ports so dashboards have stable URLs + # across re-runs and so the operator can reach them via the host's + # public hostname (e.g. http://my-host:3000) without an SSH tunnel. + # Kurtosis's default `ports={}` publishes on a random host port bound + # to 127.0.0.1; `public_ports={}` pins the host-side port and Docker + # publishes it on 0.0.0.0 (the default `-p` behaviour). prometheus = plan.add_service( name=PROMETHEUS_SERVICE, config=ServiceConfig( @@ -65,6 +71,14 @@ def launch(plan, lean_contexts): wait=None, ), }, + public_ports={ + "http": PortSpec( + number=PROMETHEUS_PORT, + transport_protocol="TCP", + application_protocol="http", + wait=None, + ), + }, files={"/etc/prometheus": prometheus_artifact}, ), ) @@ -89,6 +103,14 @@ def launch(plan, lean_contexts): wait=None, ), }, + public_ports={ + "http": PortSpec( + number=GRAFANA_PORT, + transport_protocol="TCP", + application_protocol="http", + wait=None, + ), + }, files={ "/etc/grafana/provisioning": grafana_provisioning, "/var/lib/grafana/dashboards": grafana_dashboards, From 99e45da098d1489c76a8b7beed980c1134b838c0 Mon Sep 17 00:00:00 2001 From: ilitteri Date: Thu, 14 May 2026 13:43:19 -0300 Subject: [PATCH 09/25] Add launchers for qlean, lantern, grandine, lighthouse, gean Wires the remaining devnet4 Lean clients into the pipeline. Each launcher follows the same placeholder-then-plan.exec pattern as ethlambda/ream, translating its CLI surface from the matching client-cmds/-cmd.sh in blockblaz/lean-quickstart. All 5 images ship with a working /bin/sh + busybox applets, so none need the static-busybox injection that zeam required. Per-client notes: - qlean reads --node-key from /node-keys; uses the libp2p multiaddr listen-addr form. - lantern reads everything by explicit path (validator-registry, validator-keys, validator-config, hash-sig-key-dir, nodes-path). - grandine binary is /usr/local/bin/lean_client (image ENTRYPOINT alias). - lighthouse needs both genesis.json staged (in addition to the text bundle) for its lean_node subcommand. - gean follows lean-quickstart's convention of looking up --node-key inside --custom-network-config-dir, so the launcher cps the per-node libp2p secret into the genesis mount before starting the binary. Dispatcher in src/lean/lean_launcher.star routes by LEAN_TYPE. --- src/lean/gean/gean_launcher.star | 156 +++++++++++++++++++ src/lean/grandine/grandine_launcher.star | 142 +++++++++++++++++ src/lean/lantern/lantern_launcher.star | 147 +++++++++++++++++ src/lean/lean_launcher.star | 20 +++ src/lean/lighthouse/lighthouse_launcher.star | 153 ++++++++++++++++++ src/lean/qlean/qlean_launcher.star | 142 +++++++++++++++++ 6 files changed, 760 insertions(+) create mode 100644 src/lean/gean/gean_launcher.star create mode 100644 src/lean/grandine/grandine_launcher.star create mode 100644 src/lean/lantern/lantern_launcher.star create mode 100644 src/lean/lighthouse/lighthouse_launcher.star create mode 100644 src/lean/qlean/qlean_launcher.star diff --git a/src/lean/gean/gean_launcher.star b/src/lean/gean/gean_launcher.star new file mode 100644 index 000000000..493d3c3be --- /dev/null +++ b/src/lean/gean/gean_launcher.star @@ -0,0 +1,156 @@ +""" +gean launcher. + +Translates the Lean pipeline's per-node record into gean's CLI surface +from `client-cmds/gean-cmd.sh` in blockblaz/lean-quickstart: + + gean \ + --custom-network-config-dir \ + --gossipsub-port \ + --node-id --node-key \ + --http-address 0.0.0.0 --api-port \ + --metrics-port +""" + +constants = import_module("../../package_io/constants.star") +lean_shared = import_module("../lean_shared.star") +lean_context = import_module("../lean_context.star") + +ENTRYPOINT = "/usr/local/bin/gean" +GENESIS_MOUNT = constants.LEAN_GENESIS_MOUNTPOINT_ON_CLIENTS +HASH_SIG_MOUNT = GENESIS_MOUNT + "/hash-sig-keys" +DATA_DIR = "/data" +NODE_KEY_MOUNT = constants.LEAN_NODE_KEY_MOUNTPOINT_ON_CLIENTS + +GENESIS_TEXT_FILES = [ + "config.yaml", + "annotated_validators.yaml", + "nodes.yaml", + "validator-config.yaml", + "validators.yaml", +] + + +def initialize(plan, node, p2p_keys_artifact, hash_sig_artifact): + cfg_kwargs = lean_shared.common_cfg_kwargs(node) + cfg_kwargs.update( + { + "image": node["image"], + "entrypoint": ["/bin/sh", "-c"], + "cmd": lean_shared.lean_tail_logs_cmd(node["service_name"])[2:], + "files": { + NODE_KEY_MOUNT: p2p_keys_artifact, + HASH_SIG_MOUNT: hash_sig_artifact, + }, + } + ) + return plan.add_service(node["service_name"], ServiceConfig(**cfg_kwargs)) + + +def start(plan, node, service, genesis_artifact, hash_sig_artifact): + service_name = service.name + log_file = lean_shared.lean_log_file_path(service_name) + + plan.exec( + service_name=service_name, + recipe=ExecRecipe( + command=["/bin/sh", "-c", "mkdir -p {0}".format(GENESIS_MOUNT)], + ), + description="Preparing genesis mount on {0}".format(service_name), + ) + for filename in GENESIS_TEXT_FILES: + read = plan.run_sh( + run="cat /src/{0}".format(filename), + files={"/src": genesis_artifact}, + description="Reading {0} for {1}".format(filename, service_name), + ) + plan.exec( + service_name=service_name, + recipe=ExecRecipe( + command=[ + "/bin/sh", + "-c", + "cat > {0}/{1} <<'GEAN_EOF'\n{2}\nGEAN_EOF".format( + GENESIS_MOUNT, + filename, + read.output, + ), + ], + ), + description="Staging {0} into {1}".format(filename, service_name), + ) + + # gean's contract: --custom-network-config-dir points at a dir, and + # --node-key is resolved relative to it as `.key`. Stage the + # P2P key into the genesis dir so the conventional layout works. + plan.exec( + service_name=service_name, + recipe=ExecRecipe( + command=[ + "/bin/sh", + "-c", + "cp {0}/{1}.key {2}/{1}.key".format( + NODE_KEY_MOUNT, + node["node_name"], + GENESIS_MOUNT, + ), + ], + ), + description="Staging node key into {0}".format(service_name), + ) + + cmd_parts = [ + ENTRYPOINT, + "--custom-network-config-dir", + GENESIS_MOUNT, + "--gossipsub-port", + str(constants.LEAN_QUIC_PORT_NUM), + "--node-id", + node["node_name"], + "--node-key", + "{0}/{1}.key".format(GENESIS_MOUNT, node["node_name"]), + "--http-address", + "0.0.0.0", + "--api-port", + str(constants.LEAN_API_PORT_NUM), + "--metrics-port", + str(constants.LEAN_METRICS_PORT_NUM), + ] + if node["is_aggregator"]: + cmd_parts.append("--is-aggregator") + for extra in node["extra_params"]: + cmd_parts.append(extra) + + nohup_cmd = "nohup {0} >> {1} 2>&1 &".format( + " ".join(cmd_parts), + log_file, + ) + plan.exec( + service_name=service_name, + recipe=ExecRecipe( + command=["/bin/sh", "-c", nohup_cmd], + ), + description="Starting gean binary on {0}".format(service_name), + ) + + api_url = "http://{0}:{1}".format(service.ip_address, constants.LEAN_API_PORT_NUM) + metrics_url = "http://{0}:{1}/metrics".format( + service.ip_address, constants.LEAN_METRICS_PORT_NUM + ) + + return lean_context.new_lean_context( + client_name=constants.LEAN_TYPE.gean, + service_name=service_name, + ip_address=service.ip_address, + quic_port=constants.LEAN_QUIC_PORT_NUM, + api_port=constants.LEAN_API_PORT_NUM, + metrics_port=constants.LEAN_METRICS_PORT_NUM, + api_url=api_url, + metrics_url=metrics_url, + metrics_info={ + "name": service_name, + "url": metrics_url, + "path": "/metrics", + "config": node["prometheus_config"], + }, + ) diff --git a/src/lean/grandine/grandine_launcher.star b/src/lean/grandine/grandine_launcher.star new file mode 100644 index 000000000..5063c3253 --- /dev/null +++ b/src/lean/grandine/grandine_launcher.star @@ -0,0 +1,142 @@ +""" +grandine (lean) launcher. + +Translates the Lean pipeline's per-node record into grandine's Lean +client CLI from `client-cmds/grandine-cmd.sh` in blockblaz/lean-quickstart. +""" + +constants = import_module("../../package_io/constants.star") +lean_shared = import_module("../lean_shared.star") +lean_context = import_module("../lean_context.star") + +# Image ENTRYPOINT is `lean_client`. We bypass it via `/bin/sh -c` and +# call the resolved binary path directly. +ENTRYPOINT = "/usr/local/bin/lean_client" +GENESIS_MOUNT = constants.LEAN_GENESIS_MOUNTPOINT_ON_CLIENTS +HASH_SIG_MOUNT = GENESIS_MOUNT + "/hash-sig-keys" +DATA_DIR = "/data" +NODE_KEY_MOUNT = constants.LEAN_NODE_KEY_MOUNTPOINT_ON_CLIENTS + +GENESIS_TEXT_FILES = [ + "config.yaml", + "annotated_validators.yaml", + "nodes.yaml", + "validator-config.yaml", +] + + +def initialize(plan, node, p2p_keys_artifact, hash_sig_artifact): + cfg_kwargs = lean_shared.common_cfg_kwargs(node) + cfg_kwargs.update( + { + "image": node["image"], + "entrypoint": ["/bin/sh", "-c"], + "cmd": lean_shared.lean_tail_logs_cmd(node["service_name"])[2:], + "files": { + NODE_KEY_MOUNT: p2p_keys_artifact, + HASH_SIG_MOUNT: hash_sig_artifact, + }, + } + ) + return plan.add_service(node["service_name"], ServiceConfig(**cfg_kwargs)) + + +def start(plan, node, service, genesis_artifact, hash_sig_artifact): + service_name = service.name + log_file = lean_shared.lean_log_file_path(service_name) + + plan.exec( + service_name=service_name, + recipe=ExecRecipe( + command=["/bin/sh", "-c", "mkdir -p {0}".format(GENESIS_MOUNT)], + ), + description="Preparing genesis mount on {0}".format(service_name), + ) + for filename in GENESIS_TEXT_FILES: + read = plan.run_sh( + run="cat /src/{0}".format(filename), + files={"/src": genesis_artifact}, + description="Reading {0} for {1}".format(filename, service_name), + ) + plan.exec( + service_name=service_name, + recipe=ExecRecipe( + command=[ + "/bin/sh", + "-c", + "cat > {0}/{1} <<'GRANDINE_EOF'\n{2}\nGRANDINE_EOF".format( + GENESIS_MOUNT, + filename, + read.output, + ), + ], + ), + description="Staging {0} into {1}".format(filename, service_name), + ) + + cmd_parts = [ + ENTRYPOINT, + "--genesis", + "{0}/config.yaml".format(GENESIS_MOUNT), + "--validator-registry-path", + "{0}/annotated_validators.yaml".format(GENESIS_MOUNT), + "--bootnodes", + "{0}/nodes.yaml".format(GENESIS_MOUNT), + "--node-id", + node["node_name"], + "--node-key", + "{0}/{1}.key".format(NODE_KEY_MOUNT, node["node_name"]), + "--port", + str(constants.LEAN_QUIC_PORT_NUM), + "--address", + "0.0.0.0", + "--http-address", + "0.0.0.0", + "--http-port", + str(constants.LEAN_API_PORT_NUM), + "--metrics", + "--metrics-address", + "0.0.0.0", + "--metrics-port", + str(constants.LEAN_METRICS_PORT_NUM), + "--hash-sig-key-dir", + HASH_SIG_MOUNT, + ] + if node["is_aggregator"]: + cmd_parts.append("--is-aggregator") + for extra in node["extra_params"]: + cmd_parts.append(extra) + + nohup_cmd = "nohup {0} >> {1} 2>&1 &".format( + " ".join(cmd_parts), + log_file, + ) + plan.exec( + service_name=service_name, + recipe=ExecRecipe( + command=["/bin/sh", "-c", nohup_cmd], + ), + description="Starting grandine binary on {0}".format(service_name), + ) + + api_url = "http://{0}:{1}".format(service.ip_address, constants.LEAN_API_PORT_NUM) + metrics_url = "http://{0}:{1}/metrics".format( + service.ip_address, constants.LEAN_METRICS_PORT_NUM + ) + + return lean_context.new_lean_context( + client_name=constants.LEAN_TYPE.grandine, + service_name=service_name, + ip_address=service.ip_address, + quic_port=constants.LEAN_QUIC_PORT_NUM, + api_port=constants.LEAN_API_PORT_NUM, + metrics_port=constants.LEAN_METRICS_PORT_NUM, + api_url=api_url, + metrics_url=metrics_url, + metrics_info={ + "name": service_name, + "url": metrics_url, + "path": "/metrics", + "config": node["prometheus_config"], + }, + ) diff --git a/src/lean/lantern/lantern_launcher.star b/src/lean/lantern/lantern_launcher.star new file mode 100644 index 000000000..8a2ca1033 --- /dev/null +++ b/src/lean/lantern/lantern_launcher.star @@ -0,0 +1,147 @@ +""" +lantern launcher. + +Translates the Lean pipeline's per-node record into lantern's CLI surface +from `client-cmds/lantern-cmd.sh` in blockblaz/lean-quickstart. +""" + +constants = import_module("../../package_io/constants.star") +lean_shared = import_module("../lean_shared.star") +lean_context = import_module("../lean_context.star") + +# The image's ENTRYPOINT is /usr/local/bin/lantern-entrypoint.sh; we +# bypass it via `entrypoint: ["/bin/sh", "-c"]` and call the real binary +# directly. lean-quickstart calls it as `lantern_cli` for the binary path +# but the entrypoint script forwards to the same underlying binary; the +# in-container path is /usr/local/bin/lantern_cli. +ENTRYPOINT = "/usr/local/bin/lantern_cli" +GENESIS_MOUNT = constants.LEAN_GENESIS_MOUNTPOINT_ON_CLIENTS +HASH_SIG_MOUNT = GENESIS_MOUNT + "/hash-sig-keys" +DATA_DIR = "/data" +NODE_KEY_MOUNT = constants.LEAN_NODE_KEY_MOUNTPOINT_ON_CLIENTS + +GENESIS_TEXT_FILES = [ + "config.yaml", + "annotated_validators.yaml", + "nodes.yaml", + "validator-config.yaml", + "validators.yaml", +] + + +def initialize(plan, node, p2p_keys_artifact, hash_sig_artifact): + cfg_kwargs = lean_shared.common_cfg_kwargs(node) + cfg_kwargs.update( + { + "image": node["image"], + "entrypoint": ["/bin/sh", "-c"], + "cmd": lean_shared.lean_tail_logs_cmd(node["service_name"])[2:], + "files": { + NODE_KEY_MOUNT: p2p_keys_artifact, + HASH_SIG_MOUNT: hash_sig_artifact, + }, + } + ) + return plan.add_service(node["service_name"], ServiceConfig(**cfg_kwargs)) + + +def start(plan, node, service, genesis_artifact, hash_sig_artifact): + service_name = service.name + log_file = lean_shared.lean_log_file_path(service_name) + + plan.exec( + service_name=service_name, + recipe=ExecRecipe( + command=["/bin/sh", "-c", "mkdir -p {0}".format(GENESIS_MOUNT)], + ), + description="Preparing genesis mount on {0}".format(service_name), + ) + for filename in GENESIS_TEXT_FILES: + read = plan.run_sh( + run="cat /src/{0}".format(filename), + files={"/src": genesis_artifact}, + description="Reading {0} for {1}".format(filename, service_name), + ) + plan.exec( + service_name=service_name, + recipe=ExecRecipe( + command=[ + "/bin/sh", + "-c", + "cat > {0}/{1} <<'LANTERN_EOF'\n{2}\nLANTERN_EOF".format( + GENESIS_MOUNT, + filename, + read.output, + ), + ], + ), + description="Staging {0} into {1}".format(filename, service_name), + ) + + cmd_parts = [ + ENTRYPOINT, + "--data-dir", + DATA_DIR, + "--genesis-config", + "{0}/config.yaml".format(GENESIS_MOUNT), + "--validator-registry-path", + "{0}/validators.yaml".format(GENESIS_MOUNT), + "--validator-keys-path", + "{0}/annotated_validators.yaml".format(GENESIS_MOUNT), + "--validator-config", + "{0}/validator-config.yaml".format(GENESIS_MOUNT), + "--nodes-path", + "{0}/nodes.yaml".format(GENESIS_MOUNT), + "--node-id", + node["node_name"], + "--node-key-path", + "{0}/{1}.key".format(NODE_KEY_MOUNT, node["node_name"]), + "--listen-address", + "/ip4/0.0.0.0/udp/{0}/quic-v1".format(constants.LEAN_QUIC_PORT_NUM), + "--metrics-port", + str(constants.LEAN_METRICS_PORT_NUM), + "--http-port", + str(constants.LEAN_API_PORT_NUM), + "--log-level", + "info", + "--hash-sig-key-dir", + HASH_SIG_MOUNT, + ] + if node["is_aggregator"]: + cmd_parts.append("--is-aggregator") + for extra in node["extra_params"]: + cmd_parts.append(extra) + + nohup_cmd = "nohup {0} >> {1} 2>&1 &".format( + " ".join(cmd_parts), + log_file, + ) + plan.exec( + service_name=service_name, + recipe=ExecRecipe( + command=["/bin/sh", "-c", nohup_cmd], + ), + description="Starting lantern binary on {0}".format(service_name), + ) + + api_url = "http://{0}:{1}".format(service.ip_address, constants.LEAN_API_PORT_NUM) + metrics_url = "http://{0}:{1}/metrics".format( + service.ip_address, constants.LEAN_METRICS_PORT_NUM + ) + + return lean_context.new_lean_context( + client_name=constants.LEAN_TYPE.lantern, + service_name=service_name, + ip_address=service.ip_address, + quic_port=constants.LEAN_QUIC_PORT_NUM, + api_port=constants.LEAN_API_PORT_NUM, + metrics_port=constants.LEAN_METRICS_PORT_NUM, + api_url=api_url, + metrics_url=metrics_url, + metrics_info={ + "name": service_name, + "url": metrics_url, + "path": "/metrics", + "config": node["prometheus_config"], + }, + ) diff --git a/src/lean/lean_launcher.star b/src/lean/lean_launcher.star index 52384ed8e..b65700412 100644 --- a/src/lean/lean_launcher.star +++ b/src/lean/lean_launcher.star @@ -28,6 +28,11 @@ p2p_keys = import_module( ethlambda_launcher = import_module("./ethlambda/ethlambda_launcher.star") ream_launcher = import_module("./ream/ream_launcher.star") zeam_launcher = import_module("./zeam/zeam_launcher.star") +qlean_launcher = import_module("./qlean/qlean_launcher.star") +lantern_launcher = import_module("./lantern/lantern_launcher.star") +grandine_launcher = import_module("./grandine/grandine_launcher.star") +lighthouse_launcher = import_module("./lighthouse/lighthouse_launcher.star") +gean_launcher = import_module("./gean/gean_launcher.star") metrics_launcher = import_module("./metrics/metrics_launcher.star") @@ -38,6 +43,16 @@ def _launcher_for(lean_type): return ream_launcher elif lean_type == constants.LEAN_TYPE.zeam: return zeam_launcher + elif lean_type == constants.LEAN_TYPE.qlean: + return qlean_launcher + elif lean_type == constants.LEAN_TYPE.lantern: + return lantern_launcher + elif lean_type == constants.LEAN_TYPE.grandine: + return grandine_launcher + elif lean_type == constants.LEAN_TYPE.lighthouse: + return lighthouse_launcher + elif lean_type == constants.LEAN_TYPE.gean: + return gean_launcher fail( "Unsupported lean_type '{0}'. Supported: {1}. See ".format( lean_type, @@ -46,6 +61,11 @@ def _launcher_for(lean_type): constants.LEAN_TYPE.ethlambda, constants.LEAN_TYPE.ream, constants.LEAN_TYPE.zeam, + constants.LEAN_TYPE.qlean, + constants.LEAN_TYPE.lantern, + constants.LEAN_TYPE.grandine, + constants.LEAN_TYPE.lighthouse, + constants.LEAN_TYPE.gean, ] ), ) diff --git a/src/lean/lighthouse/lighthouse_launcher.star b/src/lean/lighthouse/lighthouse_launcher.star new file mode 100644 index 000000000..c1e03bb9a --- /dev/null +++ b/src/lean/lighthouse/lighthouse_launcher.star @@ -0,0 +1,153 @@ +""" +lighthouse (lean) launcher. + +Translates the Lean pipeline's per-node record into the Lean lighthouse +fork's CLI surface from `client-cmds/lighthouse-cmd.sh` in +blockblaz/lean-quickstart: + + lighthouse lean_node \ + --datadir \ + --config \ + --validators \ + --nodes \ + --node-id --private-key \ + --genesis-json \ + --socket-port \ + --metrics --metrics-address 0.0.0.0 --metrics-port \ + --api-port +""" + +constants = import_module("../../package_io/constants.star") +lean_shared = import_module("../lean_shared.star") +lean_context = import_module("../lean_context.star") + +ENTRYPOINT = "/usr/local/bin/lighthouse" +GENESIS_MOUNT = constants.LEAN_GENESIS_MOUNTPOINT_ON_CLIENTS +HASH_SIG_MOUNT = GENESIS_MOUNT + "/hash-sig-keys" +DATA_DIR = "/data" +NODE_KEY_MOUNT = constants.LEAN_NODE_KEY_MOUNTPOINT_ON_CLIENTS + +GENESIS_TEXT_FILES = [ + "config.yaml", + "annotated_validators.yaml", + "nodes.yaml", + "validator-config.yaml", + "validators.yaml", + "genesis.json", +] + + +def initialize(plan, node, p2p_keys_artifact, hash_sig_artifact): + cfg_kwargs = lean_shared.common_cfg_kwargs(node) + cfg_kwargs.update( + { + "image": node["image"], + "entrypoint": ["/bin/sh", "-c"], + "cmd": lean_shared.lean_tail_logs_cmd(node["service_name"])[2:], + "files": { + NODE_KEY_MOUNT: p2p_keys_artifact, + HASH_SIG_MOUNT: hash_sig_artifact, + }, + } + ) + return plan.add_service(node["service_name"], ServiceConfig(**cfg_kwargs)) + + +def start(plan, node, service, genesis_artifact, hash_sig_artifact): + service_name = service.name + log_file = lean_shared.lean_log_file_path(service_name) + + plan.exec( + service_name=service_name, + recipe=ExecRecipe( + command=["/bin/sh", "-c", "mkdir -p {0}".format(GENESIS_MOUNT)], + ), + description="Preparing genesis mount on {0}".format(service_name), + ) + for filename in GENESIS_TEXT_FILES: + read = plan.run_sh( + run="cat /src/{0}".format(filename), + files={"/src": genesis_artifact}, + description="Reading {0} for {1}".format(filename, service_name), + ) + plan.exec( + service_name=service_name, + recipe=ExecRecipe( + command=[ + "/bin/sh", + "-c", + "cat > {0}/{1} <<'LIGHTHOUSE_EOF'\n{2}\nLIGHTHOUSE_EOF".format( + GENESIS_MOUNT, + filename, + read.output, + ), + ], + ), + description="Staging {0} into {1}".format(filename, service_name), + ) + + cmd_parts = [ + ENTRYPOINT, + "lean_node", + "--datadir", + DATA_DIR, + "--config", + "{0}/config.yaml".format(GENESIS_MOUNT), + "--validators", + "{0}/validator-config.yaml".format(GENESIS_MOUNT), + "--nodes", + "{0}/nodes.yaml".format(GENESIS_MOUNT), + "--node-id", + node["node_name"], + "--private-key", + "{0}/{1}.key".format(NODE_KEY_MOUNT, node["node_name"]), + "--genesis-json", + "{0}/genesis.json".format(GENESIS_MOUNT), + "--socket-port", + str(constants.LEAN_QUIC_PORT_NUM), + "--metrics", + "--metrics-address", + "0.0.0.0", + "--metrics-port", + str(constants.LEAN_METRICS_PORT_NUM), + "--api-port", + str(constants.LEAN_API_PORT_NUM), + ] + if node["is_aggregator"]: + cmd_parts.append("--is-aggregator") + for extra in node["extra_params"]: + cmd_parts.append(extra) + + nohup_cmd = "nohup {0} >> {1} 2>&1 &".format( + " ".join(cmd_parts), + log_file, + ) + plan.exec( + service_name=service_name, + recipe=ExecRecipe( + command=["/bin/sh", "-c", nohup_cmd], + ), + description="Starting lighthouse binary on {0}".format(service_name), + ) + + api_url = "http://{0}:{1}".format(service.ip_address, constants.LEAN_API_PORT_NUM) + metrics_url = "http://{0}:{1}/metrics".format( + service.ip_address, constants.LEAN_METRICS_PORT_NUM + ) + + return lean_context.new_lean_context( + client_name=constants.LEAN_TYPE.lighthouse, + service_name=service_name, + ip_address=service.ip_address, + quic_port=constants.LEAN_QUIC_PORT_NUM, + api_port=constants.LEAN_API_PORT_NUM, + metrics_port=constants.LEAN_METRICS_PORT_NUM, + api_url=api_url, + metrics_url=metrics_url, + metrics_info={ + "name": service_name, + "url": metrics_url, + "path": "/metrics", + "config": node["prometheus_config"], + }, + ) diff --git a/src/lean/qlean/qlean_launcher.star b/src/lean/qlean/qlean_launcher.star new file mode 100644 index 000000000..f6de577f2 --- /dev/null +++ b/src/lean/qlean/qlean_launcher.star @@ -0,0 +1,142 @@ +""" +qlean launcher. + +Translates the Lean pipeline's per-node record into qlean's CLI surface +from `client-cmds/qlean-cmd.sh` in blockblaz/lean-quickstart: + + qlean \ + --genesis-dir \ + --data-dir \ + --node-id --node-key \ + --listen-addr /ip4/0.0.0.0/udp//quic-v1 \ + --metrics-host 0.0.0.0 --metrics-port \ + --api-host 0.0.0.0 --api-port +""" + +constants = import_module("../../package_io/constants.star") +lean_shared = import_module("../lean_shared.star") +lean_context = import_module("../lean_context.star") + +ENTRYPOINT = "/opt/qlean/bin/qlean" +GENESIS_MOUNT = constants.LEAN_GENESIS_MOUNTPOINT_ON_CLIENTS +HASH_SIG_MOUNT = GENESIS_MOUNT + "/hash-sig-keys" +DATA_DIR = "/data" +NODE_KEY_MOUNT = constants.LEAN_NODE_KEY_MOUNTPOINT_ON_CLIENTS + +GENESIS_TEXT_FILES = [ + "config.yaml", + "annotated_validators.yaml", + "nodes.yaml", + "validator-config.yaml", + "validators.yaml", +] + + +def initialize(plan, node, p2p_keys_artifact, hash_sig_artifact): + cfg_kwargs = lean_shared.common_cfg_kwargs(node) + cfg_kwargs.update( + { + "image": node["image"], + "entrypoint": ["/bin/sh", "-c"], + "cmd": lean_shared.lean_tail_logs_cmd(node["service_name"])[2:], + "files": { + NODE_KEY_MOUNT: p2p_keys_artifact, + HASH_SIG_MOUNT: hash_sig_artifact, + }, + } + ) + return plan.add_service(node["service_name"], ServiceConfig(**cfg_kwargs)) + + +def start(plan, node, service, genesis_artifact, hash_sig_artifact): + service_name = service.name + log_file = lean_shared.lean_log_file_path(service_name) + + plan.exec( + service_name=service_name, + recipe=ExecRecipe( + command=["/bin/sh", "-c", "mkdir -p {0}".format(GENESIS_MOUNT)], + ), + description="Preparing genesis mount on {0}".format(service_name), + ) + for filename in GENESIS_TEXT_FILES: + read = plan.run_sh( + run="cat /src/{0}".format(filename), + files={"/src": genesis_artifact}, + description="Reading {0} for {1}".format(filename, service_name), + ) + plan.exec( + service_name=service_name, + recipe=ExecRecipe( + command=[ + "/bin/sh", + "-c", + "cat > {0}/{1} <<'QLEAN_EOF'\n{2}\nQLEAN_EOF".format( + GENESIS_MOUNT, + filename, + read.output, + ), + ], + ), + description="Staging {0} into {1}".format(filename, service_name), + ) + + cmd_parts = [ + ENTRYPOINT, + "--genesis-dir", + GENESIS_MOUNT, + "--data-dir", + DATA_DIR, + "--node-id", + node["node_name"], + "--node-key", + "{0}/{1}.key".format(NODE_KEY_MOUNT, node["node_name"]), + "--listen-addr", + "/ip4/0.0.0.0/udp/{0}/quic-v1".format(constants.LEAN_QUIC_PORT_NUM), + "--metrics-host", + "0.0.0.0", + "--metrics-port", + str(constants.LEAN_METRICS_PORT_NUM), + "--api-host", + "0.0.0.0", + "--api-port", + str(constants.LEAN_API_PORT_NUM), + ] + if node["is_aggregator"]: + cmd_parts.append("--is-aggregator") + for extra in node["extra_params"]: + cmd_parts.append(extra) + + nohup_cmd = "nohup {0} >> {1} 2>&1 &".format( + " ".join(cmd_parts), + log_file, + ) + plan.exec( + service_name=service_name, + recipe=ExecRecipe( + command=["/bin/sh", "-c", nohup_cmd], + ), + description="Starting qlean binary on {0}".format(service_name), + ) + + api_url = "http://{0}:{1}".format(service.ip_address, constants.LEAN_API_PORT_NUM) + metrics_url = "http://{0}:{1}/metrics".format( + service.ip_address, constants.LEAN_METRICS_PORT_NUM + ) + + return lean_context.new_lean_context( + client_name=constants.LEAN_TYPE.qlean, + service_name=service_name, + ip_address=service.ip_address, + quic_port=constants.LEAN_QUIC_PORT_NUM, + api_port=constants.LEAN_API_PORT_NUM, + metrics_port=constants.LEAN_METRICS_PORT_NUM, + api_url=api_url, + metrics_url=metrics_url, + metrics_info={ + "name": service_name, + "url": metrics_url, + "path": "/metrics", + "config": node["prometheus_config"], + }, + ) From 21cf07821c00c3aeff763301dd05ca489e93ddd6 Mon Sep 17 00:00:00 2001 From: ilitteri Date: Thu, 14 May 2026 13:49:54 -0300 Subject: [PATCH 10/25] Fix lantern and lighthouse launcher CLI surfaces Two regressions surfaced during the 9-client deploy on the office host: - lantern's binary lives at /opt/lantern/bin/lantern, not /usr/local/bin/lantern_cli. The image's ENTRYPOINT script (lantern-entrypoint.sh) forwards to /opt/lantern/bin/lantern, but since we bypass the entrypoint we have to point at the real binary. - The published hopinheimer/lighthouse:latest lean_node subcommand does not accept --api-port or --is-aggregator. Drop those flags; document that lighthouse always runs as a non-aggregator under this image and exposes only its metrics endpoint (no HTTP API). --- src/lean/lantern/lantern_launcher.star | 2 +- src/lean/lighthouse/lighthouse_launcher.star | 23 ++++++-------------- 2 files changed, 8 insertions(+), 17 deletions(-) diff --git a/src/lean/lantern/lantern_launcher.star b/src/lean/lantern/lantern_launcher.star index 8a2ca1033..b455eec53 100644 --- a/src/lean/lantern/lantern_launcher.star +++ b/src/lean/lantern/lantern_launcher.star @@ -14,7 +14,7 @@ lean_context = import_module("../lean_context.star") # directly. lean-quickstart calls it as `lantern_cli` for the binary path # but the entrypoint script forwards to the same underlying binary; the # in-container path is /usr/local/bin/lantern_cli. -ENTRYPOINT = "/usr/local/bin/lantern_cli" +ENTRYPOINT = "/opt/lantern/bin/lantern" GENESIS_MOUNT = constants.LEAN_GENESIS_MOUNTPOINT_ON_CLIENTS HASH_SIG_MOUNT = GENESIS_MOUNT + "/hash-sig-keys" DATA_DIR = "/data" diff --git a/src/lean/lighthouse/lighthouse_launcher.star b/src/lean/lighthouse/lighthouse_launcher.star index c1e03bb9a..fcd4e4268 100644 --- a/src/lean/lighthouse/lighthouse_launcher.star +++ b/src/lean/lighthouse/lighthouse_launcher.star @@ -3,18 +3,13 @@ lighthouse (lean) launcher. Translates the Lean pipeline's per-node record into the Lean lighthouse fork's CLI surface from `client-cmds/lighthouse-cmd.sh` in -blockblaz/lean-quickstart: +blockblaz/lean-quickstart, restricted to the flags the published +`hopinheimer/lighthouse:latest` image actually accepts. - lighthouse lean_node \ - --datadir \ - --config \ - --validators \ - --nodes \ - --node-id --private-key \ - --genesis-json \ - --socket-port \ - --metrics --metrics-address 0.0.0.0 --metrics-port \ - --api-port +NOTE: the image's `lighthouse lean_node` subcommand does NOT support +`--api-port` or `--is-aggregator`. Lighthouse will run as a non-aggregator +peer with only the metrics endpoint exposed; setting `is_aggregator: true` +in lean_participants is silently ignored for this client. """ constants = import_module("../../package_io/constants.star") @@ -88,9 +83,9 @@ def start(plan, node, service, genesis_artifact, hash_sig_artifact): cmd_parts = [ ENTRYPOINT, - "lean_node", "--datadir", DATA_DIR, + "lean_node", "--config", "{0}/config.yaml".format(GENESIS_MOUNT), "--validators", @@ -110,11 +105,7 @@ def start(plan, node, service, genesis_artifact, hash_sig_artifact): "0.0.0.0", "--metrics-port", str(constants.LEAN_METRICS_PORT_NUM), - "--api-port", - str(constants.LEAN_API_PORT_NUM), ] - if node["is_aggregator"]: - cmd_parts.append("--is-aggregator") for extra in node["extra_params"]: cmd_parts.append(extra) From c2d8a065bb8f19d6cbbe5b4b5271dba08ade4994 Mon Sep 17 00:00:00 2001 From: ilitteri Date: Thu, 14 May 2026 16:12:59 -0300 Subject: [PATCH 11/25] Point DEFAULT_LEAN_IMAGES at :latest, not devnet4 Pinning the in-tree defaults to devnet4 makes them rot the moment a new devnet generation ships - operators who don't override `lean_image:` would silently keep getting a stale tag. Switch every default to the client's `:latest` tag (all of them publish one) so the package itself is forward-compatible. Devnet-specific runs (e.g. the current devnet4 deployment) belong in the args file: each participant sets `lean_image: :devnet4` explicitly. The PR description carries the canonical devnet4 args example. --- src/package_io/input_parser.star | 22 ++++++++++++---------- 1 file changed, 12 insertions(+), 10 deletions(-) diff --git a/src/package_io/input_parser.star b/src/package_io/input_parser.star index 6d90f9b03..31e28fb39 100644 --- a/src/package_io/input_parser.star +++ b/src/package_io/input_parser.star @@ -65,18 +65,20 @@ DEFAULT_REMOTE_SIGNER_IMAGES = { "web3signer": "consensys/web3signer:latest", } -# Default Lean Ethereum client images. Mirrors the image list maintained by -# blockblaz/lean-quickstart so devnet operators can swap between the two -# launchers (Kurtosis vs lean-quickstart) without rebuilding. +# Default Lean Ethereum client images. Each entry points at the client's +# generic `:latest` tag so the defaults don't rot when a new devnet +# generation ships. Operators pin a specific devnet by setting +# `lean_image:` per-participant in their args file (see +# `docs/lean-consensus.md` for a worked devnet4 example). DEFAULT_LEAN_IMAGES = { - constants.LEAN_TYPE.ethlambda: "ghcr.io/lambdaclass/ethlambda:devnet4", - constants.LEAN_TYPE.ream: "ghcr.io/reamlabs/ream:latest-devnet4", - constants.LEAN_TYPE.zeam: "blockblaz/zeam:devnet4", - constants.LEAN_TYPE.qlean: "qdrvm/qlean-mini:devnet-4-amd64", - constants.LEAN_TYPE.lantern: "piertwo/lantern:v0.0.4", - constants.LEAN_TYPE.grandine: "sifrai/lean:devnet-4", + constants.LEAN_TYPE.ethlambda: "ghcr.io/lambdaclass/ethlambda:latest", + constants.LEAN_TYPE.ream: "ghcr.io/reamlabs/ream:latest", + constants.LEAN_TYPE.zeam: "blockblaz/zeam:latest", + constants.LEAN_TYPE.qlean: "qdrvm/qlean-mini:latest", + constants.LEAN_TYPE.lantern: "piertwo/lantern:latest", + constants.LEAN_TYPE.grandine: "sifrai/lean:latest", constants.LEAN_TYPE.lighthouse: "hopinheimer/lighthouse:latest", - constants.LEAN_TYPE.gean: "ghcr.io/geanlabs/gean:devnet4", + constants.LEAN_TYPE.gean: "ghcr.io/geanlabs/gean:latest", constants.LEAN_TYPE.peam: "", # No published image yet constants.LEAN_TYPE.nlean: "", # No published image yet } From aebde26b736a0ba644d55567b0029ef04175395a Mon Sep 17 00:00:00 2001 From: ilitteri Date: Thu, 14 May 2026 16:24:47 -0300 Subject: [PATCH 12/25] Reframe Lean as "client-only today, EL pairing later" The earlier wording described Lean consensus as architecturally standalone ("no EL pairing", "no Engine API", "fully standalone"). That isn't the long-term picture: Lean clients are designed to pair with EL clients in the regular EL+CL devnet shape; they just don't implement Engine API yet, so present-day devnets are client-only. Reframe both code comments and docs around that distinction: - main.star and lean_launcher.star comments now say "no Engine API yet" / "until Engine API ships" instead of declaring Lean architecturally EL-less. - docs/lean-consensus.md introduces Lean as "client-only today, EL pairing later" and notes the motivation for landing it in this package is exactly to be ready for the EL+Lean shape when it ships. - The why-a-parallel-pipeline table is retitled "Lean (today)" and adds a follow-up paragraph on how the two pipelines compose once Engine API lands. --- docs/lean-consensus.md | 33 ++++++++++++++++++++++----------- main.star | 23 ++++++++++++++--------- src/lean/lean_launcher.star | 8 ++++++-- 3 files changed, 42 insertions(+), 22 deletions(-) diff --git a/docs/lean-consensus.md b/docs/lean-consensus.md index c835b7528..b63937b2e 100644 --- a/docs/lean-consensus.md +++ b/docs/lean-consensus.md @@ -6,10 +6,14 @@ > have stub launchers covered by the same contract. The Lean Ethereum protocol — sometimes called "Beam Chain" — is a redesign of -Ethereum's consensus layer with **no EL pairing**, **no Engine API**, **no -JWT**, and **post-quantum (XMSS / hash-sig) validator signatures**. Lean -consensus clients are standalone consensus nodes that talk only to each other -over QUIC + libp2p gossipsub. The Lean protocol specification lives at +Ethereum's consensus layer built around **post-quantum (XMSS / hash-sig) +validator signatures**. Today's Lean clients run client-only devnets while +the spec stabilises: no Engine API integration yet, no JWT, just consensus +nodes peering over QUIC + libp2p gossipsub. Engine API support is on the +roadmap, after which the same Lean clients are designed to pair with EL +clients in the regular EL+CL devnet shape — which is exactly the motivation +for landing them in this package alongside the existing EL pipeline. The +Lean protocol specification lives at [ReamLabs/leanSpecs](https://github.com/ReamLabs/leanSpecs) and is co-developed by the teams behind [ream](https://github.com/ReamLabs/ream) (Rust), @@ -28,16 +32,17 @@ new Lean client, see ## Why a parallel pipeline? -Lean consensus differs from the existing EL/CL pipeline along every axis that +Lean consensus today differs from the EL/CL pipeline along several axes that shaped the original `participant_network` design: -| Concern | EL/CL | Lean | +| Concern | EL/CL | Lean (today) | |------------------------|--------------------------------|-------------------------------| | Genesis tool | `ethereum-genesis-generator` | `eth-beacon-genesis leanchain` | | Validator signatures | BLS | XMSS (hash-sig) | | Validator key keystore | EIP-2335 JSON | SSZ `validator_N_*_key_*.ssz` | -| Pairing | 1 EL + 1 CL (+ optional VC) | Standalone, no EL | -| RPC ports | Engine RPC + JWT + REST + WS | REST + Prometheus only | +| EL pairing | 1 EL + 1 CL (+ optional VC) | Client-only (no EL yet) | +| Engine API + JWT | Required | Not implemented yet | +| RPC ports | Engine RPC + JWT + REST + WS | REST + Prometheus | | P2P transport | TCP + UDP discovery + libp2p | QUIC-only (libp2p) | | Block production | EL builds payload, CL attests | Single-stack: 4 s slots | @@ -48,6 +53,12 @@ the snooper, etc. A parallel pipeline keeps those code paths untouched and isolates Lean-specific concerns under `src/lean/` and `src/prelaunch_data_generator/lean_genesis/`. +Once Lean clients ship Engine API support, the design intent is for the +two pipelines to compose: a Lean participant declares its EL counterpart +the same way today's CL clients do, JWT is shared, and the existing +EL-side plumbing (image discovery, MEV-boost, snooper, dora) keeps +working. Until then, the realistic devnet shape is Lean-only. + The package still composes the two: Prometheus/Grafana discover Lean nodes through their service labels and metrics ports, and additional services that don't depend on EL state (e.g. dora's beacon explorer) can be pointed at Lean @@ -57,9 +68,9 @@ nodes by URL. ## Quick start -Lean consensus is fully standalone — no Engine API, no EL counterpart. -A Lean network configuration therefore contains ONLY `lean_participants:` -(set `participants: []` to skip the Eth1 EL/CL flow entirely): +Today's Lean clients run client-only (no Engine API yet), so the realistic +devnet shape is Lean-only — the args file contains `lean_participants:` and +`participants: []` to skip the Eth1 EL/CL flow: ```yaml participants: [] diff --git a/main.star b/main.star index d13200958..94c6bc8b6 100644 --- a/main.star +++ b/main.star @@ -90,12 +90,15 @@ def run(plan, args={}): network_params = args_with_right_defaults.network_params # Lean-only mode: when the operator configured `lean_participants:` but - # no Eth1 `participants:` entries, run ONLY the Lean pipeline. Lean - # consensus is fully standalone (no Engine API, no EL counterpart), so - # spinning up an EL+CL pair as a "placeholder" would just waste resources - # and confuse downstream services that try to call the Engine API. - # The lean_launcher returns the per-node contexts; downstream consumers - # (prometheus, grafana, dora) can be wired through in a follow-up. + # no Eth1 `participants:` entries, run ONLY the Lean pipeline. Today's + # Lean clients don't implement Engine API yet, so spinning up an EL+CL + # pair alongside would just waste resources and confuse downstream + # services that try to call into a non-existent Engine API. Once Lean + # clients ship Engine API support, the same `lean_participants:` entries + # will be able to pair with EL `participants:` and reuse the existing + # JWT + payload-attestation plumbing. The lean_launcher returns the + # per-node contexts; downstream consumers (prometheus, grafana, dora) + # can be wired through in a follow-up. if num_participants == 0 and args_with_right_defaults.lean_participants: plan.print( "Lean-only mode: {0} lean participant entries, 0 EL/CL participants".format( @@ -361,9 +364,11 @@ def run(plan, args={}): all_xatu_sentry_contexts.append(participant.xatu_sentry_context) # Launch Lean Ethereum consensus participants alongside the EL/CL network. - # Lean is a standalone consensus stack (no EL pairing, no Engine API, no - # JWT, post-quantum signatures); it runs through its own pipeline and - # produces independent file artifacts + services. The list is empty + # Today's Lean clients run client-only (no Engine API yet) and use + # post-quantum signatures; the pipeline brings them up against their + # own genesis + libp2p QUIC mesh. Once Engine API lands on the Lean + # side, these participants will be able to pair with `participants:` + # EL clients and reuse the existing JWT plumbing. The list is empty # unless the user populated `lean_participants:` in their args. # See docs/lean-consensus.md for the architecture. all_lean_contexts = lean_launcher.launch( diff --git a/src/lean/lean_launcher.star b/src/lean/lean_launcher.star index b65700412..ea51f6138 100644 --- a/src/lean/lean_launcher.star +++ b/src/lean/lean_launcher.star @@ -11,8 +11,12 @@ Orchestrates the entire Lean pipeline: binary via `plan.exec`. This is intentionally independent of the EL/CL `participant_network` pipeline: -Lean consensus has no Engine API, no JWT, no EL pairing. Operators opt in by -populating `lean_participants:` in their args; the existing EL/CL flow runs +Today's Lean devnets run client-only (no EL pairing yet), so this pipeline +brings up a Lean-only mesh. Engine API support is on the roadmap for the +Lean clients, after which the same `lean_participants:` entries will be +able to pair with EL clients from `participants:` and share the existing +package's Engine API / JWT plumbing. Operators opt in to Lean by populating +`lean_participants:` in their args; the existing EL/CL flow runs unchanged either way. """ From 2a4bdf4278d53bfaed582300adc132a3df935aae Mon Sep 17 00:00:00 2001 From: ilitteri Date: Thu, 14 May 2026 16:57:22 -0300 Subject: [PATCH 13/25] Catch remaining "standalone" framing in package_io comments Two more comment blocks (constants.star LEAN_TYPE intro and the Lean parsing section header in input_parser.star) still described Lean as "standalone" / "no Engine API" without the temporal qualifier. Bring both in line with the wording used in docs/lean-consensus.md and the launcher modules: Lean is client-only today because Engine API isn't implemented yet, EL+Lean pairing arrives when it does. --- src/package_io/constants.star | 10 ++++++---- src/package_io/input_parser.star | 10 ++++++---- 2 files changed, 12 insertions(+), 8 deletions(-) diff --git a/src/package_io/constants.star b/src/package_io/constants.star index 037f1e787..7c43efbc9 100644 --- a/src/package_io/constants.star +++ b/src/package_io/constants.star @@ -22,10 +22,12 @@ CL_TYPE = struct( caplin="caplin", ) -# Lean Ethereum consensus clients. These are standalone consensus clients -# for the Lean Ethereum specification (post-quantum signatures, no EL pairing, -# no Engine API). They run via the separate Lean pipeline in src/lean/ and -# do NOT belong in `participants:`; use `lean_participants:` instead. +# Lean Ethereum consensus clients (post-quantum signatures, leanchain +# genesis, libp2p QUIC). Today's Lean clients run client-only because +# Engine API isn't implemented on the Lean side yet; once it lands these +# will be able to pair with EL `participants:`. Until then, Lean +# participants live in `lean_participants:` and run via the separate +# pipeline in src/lean/. LEAN_TYPE = struct( ethlambda="ethlambda", ream="ream", diff --git a/src/package_io/input_parser.star b/src/package_io/input_parser.star index 31e28fb39..c09516c0d 100644 --- a/src/package_io/input_parser.star +++ b/src/package_io/input_parser.star @@ -2605,10 +2605,12 @@ def get_devnet_modified_images(network_name, default_images): # --------------------------------------------------------------------------- # Lean Ethereum parsing # --------------------------------------------------------------------------- -# Lean consensus is a standalone, post-quantum-signature consensus stack. It -# does not pair with an EL, has no Engine API / JWT, and uses its own -# genesis pipeline (PK's eth-beacon-genesis leanchain). Lean participants -# therefore live in a separate `lean_participants:` list and run through +# Lean is a post-quantum-signature consensus stack with its own genesis +# pipeline (PK's eth-beacon-genesis leanchain). Today's Lean clients are +# client-only - Engine API isn't implemented yet, so for now they run +# without an EL counterpart; once Engine API lands they're designed to pair +# with EL clients the same way today's CL clients do. Until then, Lean +# participants live in a separate `lean_participants:` list and run through # `src/lean/lean_launcher.star`. From 1052920091833e0d3509c517ae5011cf6b74e15f Mon Sep 17 00:00:00 2001 From: ilitteri Date: Thu, 14 May 2026 17:04:09 -0300 Subject: [PATCH 14/25] Follow examples convention: drop prose docs/, add .github/tests/ args MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The two Lean prose docs (docs/lean-consensus.md and docs/lean-adding-a-new-client.md) don't match the repo's documentation shape — the existing docs/ has exactly one prose doc (architecture.md), and per-feature configs live as YAML args files under .github/tests/. - Delete docs/lean-consensus.md and docs/lean-adding-a-new-client.md. - Append a "Lean Ethereum participants" section to docs/architecture.md so the architectural overview lands in the file that already serves that purpose. - Add .github/tests/lean-devnet4.yaml as the canonical 14-node args example (mirrors how bal-devnet-0.yaml / fulu.yaml / etc. document network-shape configs). - Add .github/tests/lean-smoke.yaml as the minimal 2-node smoke test. The per-client contract that the deleted "adding a new client" guide described is now the module docstrings on each src/lean//_launcher.star, matching the existing src/cl/*/_launcher.star convention. --- .github/tests/lean-devnet4.yaml | 58 ++++++ .github/tests/lean-smoke.yaml | 18 ++ docs/architecture.md | 24 +++ docs/lean-adding-a-new-client.md | 296 ------------------------------- docs/lean-consensus.md | 234 ------------------------ 5 files changed, 100 insertions(+), 530 deletions(-) create mode 100644 .github/tests/lean-devnet4.yaml create mode 100644 .github/tests/lean-smoke.yaml delete mode 100644 docs/lean-adding-a-new-client.md delete mode 100644 docs/lean-consensus.md diff --git a/.github/tests/lean-devnet4.yaml b/.github/tests/lean-devnet4.yaml new file mode 100644 index 000000000..a5236504c --- /dev/null +++ b/.github/tests/lean-devnet4.yaml @@ -0,0 +1,58 @@ +participants: [] + +# 14-node Lean devnet4. Each participant pins the devnet4 image +# explicitly via `lean_image:`; the package's DEFAULT_LEAN_IMAGES point +# at `:latest` so they don't rot when a new devnet ships. +# +# Single aggregator on ethlambda_0. Lighthouse omitted: the published +# `hopinheimer/lighthouse:latest` image is still on the single-key +# (devnet3) GENESIS_VALIDATORS layout and won't reach consensus with +# the devnet4 nodes. +lean_participants: + - lean_type: ethlambda + lean_image: ghcr.io/lambdaclass/ethlambda:devnet4 + count: 1 + validator_count: 1 + is_aggregator: true + - lean_type: ethlambda + lean_image: ghcr.io/lambdaclass/ethlambda:devnet4 + count: 1 + validator_count: 1 + is_aggregator: false + - lean_type: ream + lean_image: ghcr.io/reamlabs/ream:latest-devnet4 + count: 2 + validator_count: 1 + is_aggregator: false + - lean_type: zeam + lean_image: blockblaz/zeam:devnet4 + count: 2 + validator_count: 1 + is_aggregator: false + - lean_type: qlean + lean_image: qdrvm/qlean-mini:devnet-4-amd64 + count: 2 + validator_count: 1 + is_aggregator: false + - lean_type: lantern + lean_image: piertwo/lantern:v0.0.4 + count: 2 + validator_count: 1 + is_aggregator: false + - lean_type: grandine + lean_image: sifrai/lean:devnet-4 + count: 2 + validator_count: 1 + is_aggregator: false + - lean_type: gean + lean_image: ghcr.io/geanlabs/gean:devnet4 + count: 2 + validator_count: 1 + is_aggregator: false + +lean_network_params: + genesis_delay: 180 + active_epoch: 18 + attestation_committee_count: 1 + +additional_services: [] diff --git a/.github/tests/lean-smoke.yaml b/.github/tests/lean-smoke.yaml new file mode 100644 index 000000000..128664a67 --- /dev/null +++ b/.github/tests/lean-smoke.yaml @@ -0,0 +1,18 @@ +# Smallest reproducible Lean devnet: one aggregator + one peer, both +# ethlambda. Comes up in ~3 min including hash-sig keygen. Use this as +# a sanity check before reaching for `lean-devnet4.yaml`. +participants: [] +lean_participants: + - lean_type: ethlambda + count: 1 + validator_count: 1 + is_aggregator: true + - lean_type: ethlambda + count: 1 + validator_count: 1 + is_aggregator: false +lean_network_params: + genesis_delay: 60 + active_epoch: 18 + attestation_committee_count: 1 +additional_services: [] diff --git a/docs/architecture.md b/docs/architecture.md index 327a25dd4..b0b924e9e 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -75,6 +75,30 @@ Once CL genesis data and keys have been created, the CL client nodes are started There are only two major difference between CL client and EL client launchers. First, the `cl_client_launcher.launch` method also consumes an `el_context`, because each CL client is connected in a 1:1 relationship with an EL client. Second, because CL clients have keys, the keystore files are passed in to the `launch` function as well. +## Lean Ethereum participants + +Lean Ethereum consensus runs as a parallel top-level pipeline alongside the +EL/CL participant network. Today's Lean clients are client-only — Engine +API isn't implemented yet — so a Lean network configuration uses its own +`lean_participants:` block and skips the Eth1 EL/CL flow by setting +`participants: []`. When Engine API ships on the Lean side, the same +participants are designed to pair with EL clients in the regular EL+CL +shape. + +The pipeline lives under [`src/lean/`](../src/lean) and reuses the +prelaunch-data-generator pattern for genesis under +[`src/prelaunch_data_generator/lean_genesis/`](../src/prelaunch_data_generator/lean_genesis). +Per-client launchers translate the Lean Kurtosis record into each +client's CLI (mirroring the contract documented inline in each +`_launcher.star`). Genesis uses +[`eth-beacon-genesis leanchain`](https://github.com/ethpandaops/eth-beacon-genesis) +for the chain bundle and +[`blockblaz/hash-sig-cli`](https://hub.docker.com/r/blockblaz/hash-sig-cli) +for XMSS validator keypairs. Metrics ride a vendored Prometheus + Grafana +stack with the upstream Lean client dashboard pre-loaded. + +Canonical args example: [`.github/tests/lean-devnet4.yaml`](../.github/tests/lean-devnet4.yaml). + ## Auxiliary Services After the Ethereum network is up and running, this package starts several auxiliary containers to make it easier to work with the Ethereum network. At time of writing, these are: diff --git a/docs/lean-adding-a-new-client.md b/docs/lean-adding-a-new-client.md deleted file mode 100644 index 4c84d6c9c..000000000 --- a/docs/lean-adding-a-new-client.md +++ /dev/null @@ -1,296 +0,0 @@ -# Adding a new Lean consensus client to ethereum-package - -This guide walks through every change you need to integrate a new Lean -consensus client (ream, zeam, qlean, lantern, grandine, lighthouse-lean, -gean, peam, nlean, or a new one). Read -[`lean-consensus.md`](./lean-consensus.md) first for the architecture. - -The integration has **5 touch points**. The Lean genesis pipeline, -hash-sig key generation, P2P key allocation, and per-node IP allocation -are all generic and require no changes. - ---- - -## Naming convention - -Every Lean node is named `_`: - -- `ethlambda_0` — first node for ethlambda -- `ethlambda_1`, `ethlambda_2` — additional nodes when `count > 1` - -The Kurtosis service name is `lean-_` (e.g. -`lean-ethlambda_0`). The prefix before the first underscore is the **client -type** and matches `LEAN_TYPE` in `src/package_io/constants.star`. - ---- - -## Touch point 1 — Register the client type - -Add your client to `LEAN_TYPE` in `src/package_io/constants.star`: - -```python -LEAN_TYPE = struct( - ethlambda="ethlambda", - ream="ream", - zeam="zeam", - # ... - myclient="myclient", -) -``` - -Then add a default image to `DEFAULT_LEAN_IMAGES` in -`src/package_io/input_parser.star`: - -```python -DEFAULT_LEAN_IMAGES = { - constants.LEAN_TYPE.ethlambda: "ghcr.io/lambdaclass/ethlambda:devnet4", - # ... - constants.LEAN_TYPE.myclient: "ghcr.io/yourorg/myclient:devnet4", -} -``` - -> The default image must run as a non-interactive container with the client -> binary as its `ENTRYPOINT`. The Lean launcher overrides `entrypoint` to -> `/bin/sh -c` so it can run a `tail -f` placeholder, but during normal -> operation the original entrypoint is replaced by a constructed command -> line. - ---- - -## Touch point 2 — `src/lean/myclient/myclient_launcher.star` - -Copy `src/lean/ethlambda/ethlambda_launcher.star` and adapt the CLI surface. -You must export exactly two functions: `initialize` and `start`. - -```python -""" -myclient launcher. - -Translates the Lean pipeline's per-node record into myclient's CLI surface. -See [client docs/CLI reference] for the source of truth. -""" - -constants = import_module("../../package_io/constants.star") -lean_shared = import_module("../lean_shared.star") -lean_context = import_module("../lean_context.star") - -ENTRYPOINT = "/usr/local/bin/myclient" -GENESIS_MOUNT = constants.LEAN_GENESIS_MOUNTPOINT_ON_CLIENTS -HASH_SIG_MOUNT = GENESIS_MOUNT + "/hash-sig-keys" -DATA_DIR = "/data" -NODE_KEY_MOUNT = constants.LEAN_NODE_KEY_MOUNTPOINT_ON_CLIENTS - - -def initialize(plan, node, p2p_keys_artifact): - # Phase 1: stand the placeholder service up so Kurtosis assigns an IP. - return plan.add_service(node["service_name"], ServiceConfig( - image = node["image"], - entrypoint = ["/bin/sh", "-c"], - cmd = lean_shared.lean_tail_logs_cmd(node["service_name"])[2:], - ports = lean_shared.lean_port_specs(), - files = {NODE_KEY_MOUNT: p2p_keys_artifact}, - env_vars = node["extra_env_vars"], - labels = node["extra_labels"], - # ... (cpu/mem/node_selectors/tolerations - copy from ethlambda) - )) - - -def start(plan, node, service, genesis_artifact, hash_sig_artifact): - # Phase 3: re-add the service with full mounts and the real command. - cmd_parts = [ - ENTRYPOINT, - # Required - your CLI must accept these (or equivalent): - "--genesis", "{0}/config.yaml".format(GENESIS_MOUNT), - "--validators","{0}/annotated_validators.yaml".format(GENESIS_MOUNT), - "--bootnodes", "{0}/nodes.yaml".format(GENESIS_MOUNT), - "--data-dir", DATA_DIR, - "--node-id", node["node_name"], - "--node-key", "{0}/{1}.key".format(NODE_KEY_MOUNT, node["node_name"]), - # Ports - your CLI must accept distinct flags for QUIC, REST, metrics: - "--gossipsub-port", str(constants.LEAN_QUIC_PORT_NUM), - "--api-port", str(constants.LEAN_API_PORT_NUM), - "--metrics-port", str(constants.LEAN_METRICS_PORT_NUM), - "--http-address", "0.0.0.0", - ] - if node["is_aggregator"]: - cmd_parts.append("--is-aggregator") - for extra in node["extra_params"]: - cmd_parts.append(extra) - - log_file = lean_shared.lean_log_file_path(service.name) - full_cmd = " ".join(cmd_parts) - - new_service = plan.add_service( - name = service.name, - force_update = True, - config = ServiceConfig( - image = node["image"], - entrypoint = ["/bin/sh", "-c"], - cmd = ["{0} 2>&1 | tee -a {1}".format(full_cmd, log_file)], - ports = lean_shared.lean_port_specs(), - files = { - NODE_KEY_MOUNT: node["_p2p_keys_artifact"], - GENESIS_MOUNT: genesis_artifact, - HASH_SIG_MOUNT: hash_sig_artifact, - }, - # ... (env_vars/labels/cpu/mem/node_selectors/tolerations) - ), - ) - - return lean_context.new_lean_context( - client_name = constants.LEAN_TYPE.myclient, - service_name = new_service.name, - ip_address = new_service.ip_address, - quic_port = constants.LEAN_QUIC_PORT_NUM, - api_port = constants.LEAN_API_PORT_NUM, - metrics_port = constants.LEAN_METRICS_PORT_NUM, - api_url = "http://{0}:{1}".format(new_service.ip_address, constants.LEAN_API_PORT_NUM), - metrics_url = "http://{0}:{1}/metrics".format(new_service.ip_address, constants.LEAN_METRICS_PORT_NUM), - metrics_info = { - "name": new_service.name, - "url": "http://{0}:{1}/metrics".format(new_service.ip_address, constants.LEAN_METRICS_PORT_NUM), - "path": "/metrics", - "config": node["prometheus_config"], - }, - ) -``` - ---- - -## Touch point 3 — Dispatch in `src/lean/lean_launcher.star` - -Add an `import_module` for your launcher and route to it in `_launcher_for`: - -```python -myclient_launcher = import_module("./myclient/myclient_launcher.star") - -def _launcher_for(lean_type): - if lean_type == constants.LEAN_TYPE.ethlambda: - return ethlambda_launcher - elif lean_type == constants.LEAN_TYPE.ream: - return ream_launcher - elif lean_type == constants.LEAN_TYPE.zeam: - return zeam_launcher - elif lean_type == constants.LEAN_TYPE.myclient: - return myclient_launcher - fail(...) -``` - ---- - -## Touch point 4 — README + `network_params.yaml` example - -Add a line to the `lean_participants:` example in `network_params.yaml`: - -```yaml -lean_participants: - - lean_type: myclient - count: 1 - validator_count: 1 -``` - ---- - -## Touch point 5 — Docs - -Add your client to the list at the top of [`lean-consensus.md`](./lean-consensus.md). - ---- - -## Required CLI flags your client must support - -Your Lean client binary must accept at least the following flags (or -equivalents you can pass via `lean_extra_params`). Flag names vary across -clients; the names below mirror ethlambda — adapt to your client's CLI by -adjusting the per-client launcher. - -| Concept | Where it comes from | -|----------------------------|-----------------------------------------------------------| -| `--node-id ` | Identifies the node in logs and validator-config lookups | -| `--node-key ` | 32-byte hex libp2p secret (`.key`) | -| `--genesis ` | Path to `config.yaml` | -| `--validators ` | Path to `annotated_validators.yaml` | -| `--bootnodes ` | Path to `nodes.yaml` | -| `--validator-config `| Path to `validator-config.yaml` (per-node settings) | -| `--hash-sig-keys-dir `| XMSS key directory | -| `--data-dir ` | Persistent RocksDB / LMDB | -| `--gossipsub-port ` | UDP QUIC port (= `LEAN_QUIC_PORT_NUM = 9000`) | -| `--api-port ` | REST API port (= `LEAN_API_PORT_NUM = 5052`) | -| `--metrics-port ` | Prometheus metrics port (= `LEAN_METRICS_PORT_NUM = 5054`) | -| `--http-address 0.0.0.0` | Bind address for REST + metrics | -| `--is-aggregator` | Enable aggregator mode (required for finality) | - -### Required HTTP endpoints - -| Path | Purpose | -|--------------------------------|--------------------------------------------------------| -| `GET /lean/v0/health` | Liveness check (return 200 when healthy) | -| `GET /metrics` (metrics port) | Prometheus exposition (`lean_*` metric names) | - -The full Lean REST API is documented at -[ReamLabs/leanSpecs](https://github.com/ReamLabs/leanSpecs); only health -+ metrics are required for the package itself, but other endpoints -(checkpoint sync, fork choice, finalized state) are needed for richer -auxiliary services (dora, checkpointz analogues, etc.) when they appear. - ---- - -## Required on-disk file format - -Your client must read the files listed in -[`lean-consensus.md#files-mounted-into-every-lean-client`](./lean-consensus.md#files-mounted-into-every-lean-client). -Specifically: - -- **`config.yaml`** with GENESIS_TIME (int), ATTESTATION_COMMITTEE_COUNT, - ACTIVE_EPOCH, VALIDATOR_COUNT, and a GENESIS_VALIDATORS list of - `{attestation_pubkey, proposal_pubkey}` dual-key entries (hex strings - without `0x` prefix). -- **`annotated_validators.yaml`** mapping `: [{index, - pubkey_hex, privkey_file}, ...]` with privkey_file names containing - `_attester_` or `_proposer_` to route to attestation vs proposal slots. -- **`nodes.yaml`** = list of ENRs (base64) as a YAML sequence of strings. -- **`validator-config.yaml`** matching the lean-quickstart schema (used - by some clients for per-node ENR/metrics-port lookups). -- **`hash-sig-keys/validator_N_{attester,proposer}_key_sk.ssz`** as SSZ - XMSS private keys. - -This is the same on-disk shape produced by `lean-quickstart`'s -`generate-genesis.sh`, so a client that runs under lean-quickstart will -run under this package without code changes. - ---- - -## Local test - -```bash -# In ethereum-package root: -kurtosis run --enclave lean-test . --args-file - <<'YAML' -participants: - - el_type: geth - cl_type: lighthouse - count: 1 - validator_count: 0 -lean_participants: - - lean_type: myclient - count: 1 - is_aggregator: true -YAML - -# Inspect the running service -kurtosis service shell lean-test lean-myclient_0 -# Inside the container: -curl http://localhost:5052/lean/v0/health -curl http://localhost:5054/metrics | head -``` - ---- - -## Checklist - -``` -[ ] 1. Add LEAN_TYPE entry + DEFAULT_LEAN_IMAGES entry -[ ] 2. Create src/lean//_launcher.star with initialize + start -[ ] 3. Wire dispatch in src/lean/lean_launcher.star (_launcher_for) -[ ] 4. Add an example line to network_params.yaml under lean_participants: -[ ] 5. Add the client to the supported list at the top of docs/lean-consensus.md -``` diff --git a/docs/lean-consensus.md b/docs/lean-consensus.md deleted file mode 100644 index b63937b2e..000000000 --- a/docs/lean-consensus.md +++ /dev/null @@ -1,234 +0,0 @@ -# Lean Ethereum consensus support - -> Status: experimental. Initial integration adds the Lean Ethereum -> consensus stack as a parallel pipeline alongside the existing EL/CL -> network. Only `ethlambda` is fully wired today; `ream` and `zeam` -> have stub launchers covered by the same contract. - -The Lean Ethereum protocol — sometimes called "Beam Chain" — is a redesign of -Ethereum's consensus layer built around **post-quantum (XMSS / hash-sig) -validator signatures**. Today's Lean clients run client-only devnets while -the spec stabilises: no Engine API integration yet, no JWT, just consensus -nodes peering over QUIC + libp2p gossipsub. Engine API support is on the -roadmap, after which the same Lean clients are designed to pair with EL -clients in the regular EL+CL devnet shape — which is exactly the motivation -for landing them in this package alongside the existing EL pipeline. The -Lean protocol specification lives at -[ReamLabs/leanSpecs](https://github.com/ReamLabs/leanSpecs) and is -co-developed by the teams behind -[ream](https://github.com/ReamLabs/ream) (Rust), -[zeam](https://github.com/blockblaz/zeam) (Zig), -[qlean](https://github.com/qdrvm/qlean-mini) (C++), -[lantern](https://github.com/Pier-Two/lantern) (C), -[grandine](https://github.com/grandinetech/lean/tree/main/lean_client) (Rust), -a [lighthouse](https://github.com/hopinheimer/lighthouse) fork (Rust), and -[ethlambda](https://github.com/lambdaclass/ethlambda) (Rust). - -This document describes how `ethereum-package` runs Lean networks. To add a -new Lean client, see -[`lean-adding-a-new-client.md`](./lean-adding-a-new-client.md). - ---- - -## Why a parallel pipeline? - -Lean consensus today differs from the EL/CL pipeline along several axes that -shaped the original `participant_network` design: - -| Concern | EL/CL | Lean (today) | -|------------------------|--------------------------------|-------------------------------| -| Genesis tool | `ethereum-genesis-generator` | `eth-beacon-genesis leanchain` | -| Validator signatures | BLS | XMSS (hash-sig) | -| Validator key keystore | EIP-2335 JSON | SSZ `validator_N_*_key_*.ssz` | -| EL pairing | 1 EL + 1 CL (+ optional VC) | Client-only (no EL yet) | -| Engine API + JWT | Required | Not implemented yet | -| RPC ports | Engine RPC + JWT + REST + WS | REST + Prometheus | -| P2P transport | TCP + UDP discovery + libp2p | QUIC-only (libp2p) | -| Block production | EL builds payload, CL attests | Single-stack: 4 s slots | - -Trying to express Lean clients as `participants[].cl_type` with `el_type: none` -would force `if is_lean(): ... else:` branches throughout the EL/CL pipeline, -the validator-keystore generator, the genesis generator, the MEV-boost flow, -the snooper, etc. A parallel pipeline keeps those code paths untouched and -isolates Lean-specific concerns under `src/lean/` and -`src/prelaunch_data_generator/lean_genesis/`. - -Once Lean clients ship Engine API support, the design intent is for the -two pipelines to compose: a Lean participant declares its EL counterpart -the same way today's CL clients do, JWT is shared, and the existing -EL-side plumbing (image discovery, MEV-boost, snooper, dora) keeps -working. Until then, the realistic devnet shape is Lean-only. - -The package still composes the two: Prometheus/Grafana discover Lean nodes -through their service labels and metrics ports, and additional services that -don't depend on EL state (e.g. dora's beacon explorer) can be pointed at Lean -nodes by URL. - ---- - -## Quick start - -Today's Lean clients run client-only (no Engine API yet), so the realistic -devnet shape is Lean-only — the args file contains `lean_participants:` and -`participants: []` to skip the Eth1 EL/CL flow: - -```yaml -participants: [] - -lean_participants: - - lean_type: ethlambda - count: 4 - validator_count: 1 - is_aggregator: true -``` - -Then run: - -```bash -kurtosis run --enclave lean-test github.com/ethpandaops/ethereum-package --args-file your-args.yaml -``` - -The Lean pipeline produces 4 services named `lean-ethlambda_0` … -`lean-ethlambda_3`, each exposing: - -| Port | Purpose | -|-------|--------------------------------------------------| -| 9000 | libp2p QUIC (UDP) — block + attestation gossip | -| 5052 | REST API (`GET /lean/v0/health`, fork choice, …) | -| 5054 | Prometheus metrics (`/metrics`) | - -### Mixed mode (Lean + EL/CL) - -You can also run Lean alongside the existing Eth1 EL/CL network in the same -enclave. Both pipelines run independently — there is no cross-talk between -them. Add `participants:` entries as you normally would and keep -`lean_participants:` populated. This is useful for side-by-side benchmarking -and observability dashboards that scrape both. - ---- - -## Pipeline architecture - -The Lean launcher (`src/lean/lean_launcher.star`) runs in three phases. -Phases 1 and 3 are per-node; phase 2 is global. - -``` - ┌──────────────────────────────────┐ - Phase 1 │ openssl: generate .key x N │ - └──────────────────────────────────┘ - │ - ▼ - ┌────────────────────────────────────────────────────────────┐ - │ For each Lean participant entry: │ - │ plan.add_service(name=lean-_, cmd="tail -f") │ - │ Kurtosis assigns an IP to each service. │ - └────────────────────────────────────────────────────────────┘ - │ - Phase 2 ▼ - ┌────────────────────────────────────────────────────────────┐ - │ hash-sig-cli: generate XMSS attester+proposer keys (SSZ) │ - │ render validator-config.yaml from live IPs + ports │ - │ render initial config.yaml (GENESIS_TIME etc.) │ - │ eth-beacon-genesis leanchain: write nodes.yaml, │ - │ validators.yaml, genesis.{ssz,json}, update config.yaml │ - │ post-process: inject GENESIS_VALIDATORS into config.yaml, │ - │ render annotated_validators.yaml from manifest │ - └────────────────────────────────────────────────────────────┘ - │ - Phase 3 ▼ - ┌────────────────────────────────────────────────────────────┐ - │ For each placeholder service: │ - │ plan.add_service(name=, force_update=True, │ - │ mounts={genesis_artifact, hash_sig_artifact, keys}, │ - │ cmd=) │ - │ Kurtosis preserves the IP (same name + ports). │ - └────────────────────────────────────────────────────────────┘ -``` - -Why three phases? Because the genesis tool needs every node's IP and port to -render `nodes.yaml` (the bootnode list), but Kurtosis only assigns IPs after -`add_service`. Pre-allocating placeholder services then re-issuing them with -`force_update=True` keeps the IP stable while letting us mount the -just-generated genesis bundle. - ---- - -## Files mounted into every Lean client - -All Lean clients receive the same on-disk layout. This matches the layout -produced by `lean-quickstart`'s `generate-genesis.sh` so a Lean client that -runs under `lean-quickstart` runs under this package without code changes. - -| Path | Source | Contents | -|-------------------------------------------------------|------------------------------|-----------------------------------------------------------------| -| `/network-configs/config.yaml` | Lean genesis post-process | GENESIS_TIME, ATTESTATION_COMMITTEE_COUNT, ACTIVE_EPOCH, VALIDATOR_COUNT, GENESIS_VALIDATORS (per-validator attestation/proposal pubkeys) | -| `/network-configs/validators.yaml` | PK's eth-beacon-genesis | `node_name -> [validator_index]` round-robin assignments | -| `/network-configs/annotated_validators.yaml` | Lean genesis post-process | `node_name -> [{index, pubkey_hex, privkey_file}]` | -| `/network-configs/nodes.yaml` | PK's eth-beacon-genesis | ENR list for all Lean nodes (bootnodes) | -| `/network-configs/validator-config.yaml` | Lean launcher (rendered) | Per-node config (name, privkey, IP, ports, count, isAggregator) | -| `/network-configs/genesis.ssz` | PK's eth-beacon-genesis | SSZ genesis state | -| `/network-configs/genesis.json` | PK's eth-beacon-genesis | JSON genesis state | -| `/network-configs/.key` | openssl prelaunch step | 32-byte hex libp2p secret for this node | -| `/network-configs/hash-sig-keys/validator_N_attester_key_{sk,pk}.ssz` | hash-sig-cli | XMSS attester keypair per validator | -| `/network-configs/hash-sig-keys/validator_N_proposer_key_{sk,pk}.ssz` | hash-sig-cli | XMSS proposer keypair per validator | -| `/network-configs/hash-sig-keys/validator-keys-manifest.yaml` | hash-sig-cli | Dual-key manifest mapping validator index to attester/proposer pubkey hex | -| `/node-keys/.key` | openssl prelaunch step | Same as above; kept at a separate mount for clients that expect this layout | - -> Clients SHOULD derive their genesis state from `config.yaml` directly -> (using GENESIS_VALIDATORS pubkeys and GENESIS_TIME). The `genesis.json` / -> `genesis.ssz` files are provided for compatibility but their format may -> drift across leanSpec revisions. - ---- - -## Port contract - -| Port | Protocol | Purpose | -|-------|----------|-------------------------------| -| 9000 | UDP | libp2p QUIC (block + attestation gossipsub) | -| 5052 | TCP HTTP | REST API (must implement `GET /lean/v0/health`) | -| 5054 | TCP HTTP | Prometheus metrics (`/metrics`) | - -These match the defaults used by every Lean client in `lean-quickstart` so -operator-facing dashboards and probes work across deployments. - ---- - -## Components added - -| Path | Purpose | -|-----------------------------------------------------------------|--------------------------------------------------------------------------| -| `src/package_io/constants.star` | `LEAN_TYPE` enum, default port nums, mountpoints, genesis-tool images. | -| `src/package_io/input_parser.star` | `lean_participants` / `lean_network_params` parsing + defaults. | -| `src/prelaunch_data_generator/lean_genesis/p2p_keys_generator.star` | Generates one 32-byte hex libp2p secret per node (openssl). | -| `src/prelaunch_data_generator/lean_genesis/lean_genesis_generator.star` | Full Lean genesis pipeline (hash-sig + leanchain + post-process). | -| `src/lean/lean_launcher.star` | Dispatches per-client launchers, runs the three-phase lifecycle. | -| `src/lean/lean_context.star` | Per-node context struct returned to `main.star`. | -| `src/lean/lean_shared.star` | Common port specs, mountpoint helpers, log file conventions. | -| `src/lean/ethlambda/ethlambda_launcher.star` | First fully-wired client. | -| `src/lean/ream/ream_launcher.star` | Stub mirroring `client-cmds/ream-cmd.sh`. | -| `src/lean/zeam/zeam_launcher.star` | Stub mirroring `client-cmds/zeam-cmd.sh`. | -| `main.star` | Single call site for `lean_launcher.launch(...)`. | -| `network_params.yaml` | Documented `lean_participants:` example + default `lean_network_params`. | - ---- - -## Limitations & follow-ups - -1. **Lean nodes are not yet scraped by Prometheus.** The `metrics_info` - struct is populated on every `lean_context`, but the prometheus - launcher isn't yet wired to discover Lean nodes. Operators scraping - Lean nodes today should hit them by service name directly. -2. **`hash-sig-cli` image is pinned to `:latest`.** Pinning to a SHA is - left to a follow-up; override via - `lean_network_params.hash_sig_cli_image`. -3. **No Lean-specific dashboards.** Existing Grafana dashboards assume the - Ethereum CL schema. Lean dashboards (`lean_head_slot`, - `lean_state_transition_time_seconds`, etc.) need a separate dashboard - pack. -4. **No checkpoint sync.** Per-participant `checkpoint_sync_url` parsing - is not yet wired through to the per-client launchers. -5. **Mixed-mode auxiliary services.** When Lean is run alongside EL/CL, - the existing Eth1 auxiliary services (tx-fuzz, dora, etc.) only see - the EL/CL participants. Wiring them to also point at Lean nodes is a - follow-up. From 5f9de985b97cac69372466f934948ec0e8e1cd36 Mon Sep 17 00:00:00 2001 From: ilitteri Date: Tue, 19 May 2026 22:21:58 -0300 Subject: [PATCH 15/25] Enable the `admin` JSON-RPC namespace on the ethrex EL launcher so that `el_admin_node_info.get_enode_enr_for_node` can discover the ENR/enode via `admin_nodeInfo`. ethrex defaults to `eth,net,web3` only; without this flag the kurtosis startup polls admin_nodeInfo forever and never hands the el_context to downstream CL launchers. The change is a no-op for existing setups that didn't reach the poll (it only widens the public HTTP API surface inside the test enclave). --- src/el/ethrex/ethrex_launcher.star | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/src/el/ethrex/ethrex_launcher.star b/src/el/ethrex/ethrex_launcher.star index 1d49d8657..3af73f296 100644 --- a/src/el/ethrex/ethrex_launcher.star +++ b/src/el/ethrex/ethrex_launcher.star @@ -162,6 +162,10 @@ def get_config( "--log.level={0}".format(VERBOSITY_LEVELS[global_log_level]), "--http.port={0}".format(RPC_PORT_NUM), "--http.addr=0.0.0.0", + # Enable the `admin` namespace so el_admin_node_info.get_enode_enr_for_node() + # can read the ENR/enode via `admin_nodeInfo`. ethrex defaults to + # `eth,net,web3` only; without `admin` the poll hangs forever. + "--http.api=admin,eth,net,web3", "--authrpc.port={0}".format(ENGINE_RPC_PORT_NUM), "--authrpc.jwtsecret=" + constants.JWT_MOUNT_PATH_ON_CONTAINER, "--authrpc.addr=0.0.0.0", From 70da979650b5b13c4999f6a7bce5881f3f00fe28 Mon Sep 17 00:00:00 2001 From: ilitteri Date: Tue, 19 May 2026 22:23:03 -0300 Subject: [PATCH 16/25] =?UTF-8?q?Detach=20Lean=20client=20binary=20launche?= =?UTF-8?q?s=20via=20`setsid=20-f`=20instead=20of=20`nohup=20...=20&`.=20K?= =?UTF-8?q?urtosis=20`exec`=20waits=20for=20its=20docker-exec=20FDs=20to?= =?UTF-8?q?=20close,=20and=20`&=20disown`=20is=20a=20bash-ism=20that=20the?= =?UTF-8?q?=20Lean=20client=20images'=20/bin/sh=20(dash=20on=20Debian-slim?= =?UTF-8?q?)=20doesn't=20recognise=20=E2=80=94=20so=20the=20backgrounded?= =?UTF-8?q?=20ethlambda=20process=20kept=20the=20exec=20connection=20open?= =?UTF-8?q?=20and=20the=20kurtosis=20run=20hung=20at=20the=20start=20step?= =?UTF-8?q?=20for=20the=20next=20Lean=20node.=20`setsid=20-f`=20forks=20in?= =?UTF-8?q?to=20a=20new=20session=20and=20exits=20the=20parent=20shell=20i?= =?UTF-8?q?mmediately,=20releasing=20the=20FDs=20and=20letting=20the=20kur?= =?UTF-8?q?tosis=20run=20progress=20to=20the=20next=20node.?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The zeam launcher keeps the busybox-prefixed `nohup ... &` pattern because its scratch-based image has no setsid; the busybox build detaches via the injected `< /dev/null` redirect (handled separately in that launcher). --- src/lean/ethlambda/ethlambda_launcher.star | 2 +- src/lean/gean/gean_launcher.star | 2 +- src/lean/grandine/grandine_launcher.star | 2 +- src/lean/lantern/lantern_launcher.star | 2 +- src/lean/lighthouse/lighthouse_launcher.star | 2 +- src/lean/qlean/qlean_launcher.star | 2 +- src/lean/ream/ream_launcher.star | 2 +- 7 files changed, 7 insertions(+), 7 deletions(-) diff --git a/src/lean/ethlambda/ethlambda_launcher.star b/src/lean/ethlambda/ethlambda_launcher.star index 54010354e..31e42a18d 100644 --- a/src/lean/ethlambda/ethlambda_launcher.star +++ b/src/lean/ethlambda/ethlambda_launcher.star @@ -155,7 +155,7 @@ def start(plan, node, service, genesis_artifact, hash_sig_artifact): if node["log_level"] != "": rust_log = "RUST_LOG='{0}' ".format(node["log_level"]) - nohup_cmd = "nohup {0}{1} >> {2} 2>&1 &".format( + nohup_cmd = "setsid -f sh -c \"exec {0}{1}\" < /dev/null >> {2} 2>&1".format( rust_log, " ".join(cmd_parts), log_file, diff --git a/src/lean/gean/gean_launcher.star b/src/lean/gean/gean_launcher.star index 493d3c3be..a8ffe2653 100644 --- a/src/lean/gean/gean_launcher.star +++ b/src/lean/gean/gean_launcher.star @@ -121,7 +121,7 @@ def start(plan, node, service, genesis_artifact, hash_sig_artifact): for extra in node["extra_params"]: cmd_parts.append(extra) - nohup_cmd = "nohup {0} >> {1} 2>&1 &".format( + nohup_cmd = "setsid -f sh -c \"exec {0}\" < /dev/null >> {1} 2>&1".format( " ".join(cmd_parts), log_file, ) diff --git a/src/lean/grandine/grandine_launcher.star b/src/lean/grandine/grandine_launcher.star index 5063c3253..1834c6bbb 100644 --- a/src/lean/grandine/grandine_launcher.star +++ b/src/lean/grandine/grandine_launcher.star @@ -107,7 +107,7 @@ def start(plan, node, service, genesis_artifact, hash_sig_artifact): for extra in node["extra_params"]: cmd_parts.append(extra) - nohup_cmd = "nohup {0} >> {1} 2>&1 &".format( + nohup_cmd = "setsid -f sh -c \"exec {0}\" < /dev/null >> {1} 2>&1".format( " ".join(cmd_parts), log_file, ) diff --git a/src/lean/lantern/lantern_launcher.star b/src/lean/lantern/lantern_launcher.star index b455eec53..34f40c811 100644 --- a/src/lean/lantern/lantern_launcher.star +++ b/src/lean/lantern/lantern_launcher.star @@ -112,7 +112,7 @@ def start(plan, node, service, genesis_artifact, hash_sig_artifact): for extra in node["extra_params"]: cmd_parts.append(extra) - nohup_cmd = "nohup {0} >> {1} 2>&1 &".format( + nohup_cmd = "setsid -f sh -c \"exec {0}\" < /dev/null >> {1} 2>&1".format( " ".join(cmd_parts), log_file, ) diff --git a/src/lean/lighthouse/lighthouse_launcher.star b/src/lean/lighthouse/lighthouse_launcher.star index fcd4e4268..c3784042e 100644 --- a/src/lean/lighthouse/lighthouse_launcher.star +++ b/src/lean/lighthouse/lighthouse_launcher.star @@ -109,7 +109,7 @@ def start(plan, node, service, genesis_artifact, hash_sig_artifact): for extra in node["extra_params"]: cmd_parts.append(extra) - nohup_cmd = "nohup {0} >> {1} 2>&1 &".format( + nohup_cmd = "setsid -f sh -c \"exec {0}\" < /dev/null >> {1} 2>&1".format( " ".join(cmd_parts), log_file, ) diff --git a/src/lean/qlean/qlean_launcher.star b/src/lean/qlean/qlean_launcher.star index f6de577f2..09b37d414 100644 --- a/src/lean/qlean/qlean_launcher.star +++ b/src/lean/qlean/qlean_launcher.star @@ -107,7 +107,7 @@ def start(plan, node, service, genesis_artifact, hash_sig_artifact): for extra in node["extra_params"]: cmd_parts.append(extra) - nohup_cmd = "nohup {0} >> {1} 2>&1 &".format( + nohup_cmd = "setsid -f sh -c \"exec {0}\" < /dev/null >> {1} 2>&1".format( " ".join(cmd_parts), log_file, ) diff --git a/src/lean/ream/ream_launcher.star b/src/lean/ream/ream_launcher.star index 699f64cdb..a49506c6e 100644 --- a/src/lean/ream/ream_launcher.star +++ b/src/lean/ream/ream_launcher.star @@ -128,7 +128,7 @@ def start(plan, node, service, genesis_artifact, hash_sig_artifact): if node["log_level"] != "": rust_log = "RUST_LOG='{0}' ".format(node["log_level"]) - nohup_cmd = "nohup {0}{1} >> {2} 2>&1 &".format( + nohup_cmd = "setsid -f sh -c \"exec {0}{1}\" < /dev/null >> {2} 2>&1".format( rust_log, " ".join(cmd_parts), log_file, From 11631a92fc9a21930160ff9c041f52ca8a09f016 Mon Sep 17 00:00:00 2001 From: ilitteri Date: Tue, 19 May 2026 22:23:33 -0300 Subject: [PATCH 17/25] Wire every Lean Ethereum consensus client as a `cl_type:` value inside the standard `participants:` block, and remove the parallel `lean_participants:` schema entirely. The new shape collapses Lean and EL+CL into one input surface: MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit participants: - el_type: ethrex cl_type: ethlambda is_aggregator: true - el_type: none cl_type: ream ethlambda is the only Lean client that implements Engine API today (lambdaclass/ethlambda#367); when paired with an EL (`el_type` != none) the Lean launcher reads the EL's genesis block hash via `eth_getBlockByNumber 0x0` after the EL is up, stages the network JWT into the ethlambda container, and adds the three Engine API flags (`--execution-endpoint`, `--execution-jwt-secret`, `--execution-genesis-block-hash`) to its CLI. The other seven Lean clients run client-only — `el_type: none` skips EL launch entirely (the package already supports this for `consensoor` etc.). Cl_TYPE additions: ethlambda, ream, zeam, qlean, lantern, gean, lean_grandine, lean_lighthouse. The last two are prefixed because `grandine` and `lighthouse` already exist in CL_TYPE for the Eth1 CLs of the same name (different binaries from different repos). `LEAN_CL_TYPES` is the set the cl_launcher dispatcher checks to decide whether to skip a participant (Lean cl_types are launched by src/lean/lean_launcher.star, not the standard CL launchers); main.star then builds a Lean record per such participant and hands the list to the Lean launcher with the network jwt_file attached. Other plumbing changes that fall out of this: - `is_aggregator` is now a first-class per-participant field. Ignored on non-Lean cl_types. - The "first participant cannot have el_type=none without bootnodoor" guard is relaxed when every participant has a Lean cl_type — Lean uses its own libp2p QUIC mesh and doesn't need an Eth1 bootnode. - The Fulu/PeerDAS validation skips Lean cl_types (they don't speak PeerDAS). - The VC / remote-signer / snooper / metrics-exporter pipeline is skipped for Lean cl_types in participant_network.star — Lean validators live inside the consensus binary, not a separate VC. - shared_utils.get_client_names is None-safe: when cl_context is None (Lean participants), it falls back to the cl_type string from the participant config so downstream consumers (validator-ranges, dora, etc.) still get a usable row name. `lean_network_params:` stays as a separate config block for Lean-only knobs (`active_epoch`, `attestation_committee_count`, `num_validator_keys_per_node`, `metrics_enabled`, ...). `parse_lean_participants` and `DEFAULT_LEAN_IMAGES` are deleted; the DEFAULT_CL_IMAGES table now carries the Lean defaults too. Args files migrated: - `.github/tests/lean-devnet4.yaml` — every entry moved to `participants:` with `el_type: none`. - `.github/tests/lean-smoke.yaml` — same shape, two ethlambda nodes. - `.github/tests/ethlambda-el-pair.yaml` — new, single ethrex+ethlambda pair. - `.github/tests/ethlambda-el-pair-2node.yaml` — new, two pairs with one aggregator + one non-aggregator on the Lean side. Validated locally: the 2-node ethrex+ethlambda pair finalizes slot-by-slot, both ethrex ELs converge on identical block hashes via the Lean libp2p mesh between the two ethlambdas. --- .github/tests/ethlambda-el-pair-2node.yaml | 23 +++ .github/tests/ethlambda-el-pair.yaml | 21 +++ .github/tests/lean-devnet4.yaml | 58 ++++---- .github/tests/lean-smoke.yaml | 14 +- docs/architecture.md | 47 +++--- main.star | 90 ++++++------ network_params.yaml | 24 ++-- src/cl/cl_launcher.star | 13 ++ src/lean/ethlambda/ethlambda_launcher.star | 59 +++++++- src/lean/lean_launcher.star | 86 ++++++++--- src/lean/lighthouse/lighthouse_launcher.star | 2 +- src/lean/metrics/metrics_launcher.star | 5 +- src/package_io/constants.star | 47 +++++- src/package_io/input_parser.star | 134 +++++++----------- src/package_io/sanity_check.star | 10 +- src/participant_network.star | 15 ++ .../lean_genesis/lean_genesis_generator.star | 4 +- src/shared_utils/shared_utils.star | 13 +- 18 files changed, 437 insertions(+), 228 deletions(-) create mode 100644 .github/tests/ethlambda-el-pair-2node.yaml create mode 100644 .github/tests/ethlambda-el-pair.yaml diff --git a/.github/tests/ethlambda-el-pair-2node.yaml b/.github/tests/ethlambda-el-pair-2node.yaml new file mode 100644 index 000000000..db2ddad1b --- /dev/null +++ b/.github/tests/ethlambda-el-pair-2node.yaml @@ -0,0 +1,23 @@ +# Two ethrex + ethlambda pairs. One aggregator, one not. +# +# - Two `participants:` entries, each pairs an ethrex EL with an ethlambda CL. +# - Lean genesis pipeline runs once across both participants (single XMSS keyset, +# shared validator-config) — see src/lean/lean_launcher.star. +# - Each ethlambda gets its own EL endpoint, JWT, and EL genesis block hash. +# - Lean libp2p QUIC mesh peers ethlambda_0 <-> ethlambda_1 directly via the +# nodes.yaml the genesis pipeline emits. +participants: + - el_type: ethrex + cl_type: ethlambda + cl_image: ghcr.io/lambdaclass/ethlambda:engine-api-integration + count: 1 + validator_count: 1 + is_aggregator: true + - el_type: ethrex + cl_type: ethlambda + cl_image: ghcr.io/lambdaclass/ethlambda:engine-api-integration + count: 1 + validator_count: 1 + is_aggregator: false + +additional_services: [] diff --git a/.github/tests/ethlambda-el-pair.yaml b/.github/tests/ethlambda-el-pair.yaml new file mode 100644 index 000000000..2ae42c363 --- /dev/null +++ b/.github/tests/ethlambda-el-pair.yaml @@ -0,0 +1,21 @@ +# Single ethrex + ethlambda pair, smoke test for Engine API pairing. +# +# - ethlambda runs as a CL inside `participants:` (cl_type: ethlambda). +# - Its launcher is wired in src/lean/ethlambda/ethlambda_launcher.star; the +# standard CL dispatcher in src/cl/cl_launcher.star skips Lean cl_types, +# then main.star routes them into the Lean pipeline with the paired +# el_context and the network JWT. +# - The `:engine-api-integration` image is built from +# lambdaclass/ethlambda#367 locally (no registry yet). +# +# Other Lean clients live in `participants:` too with `el_type: none` +# (no Engine API yet) — see `lean-devnet4.yaml` and `lean-smoke.yaml`. +participants: + - el_type: ethrex + cl_type: ethlambda + cl_image: ghcr.io/lambdaclass/ethlambda:engine-api-integration + count: 1 + validator_count: 1 + is_aggregator: true + +additional_services: [] diff --git a/.github/tests/lean-devnet4.yaml b/.github/tests/lean-devnet4.yaml index a5236504c..55a8fedc0 100644 --- a/.github/tests/lean-devnet4.yaml +++ b/.github/tests/lean-devnet4.yaml @@ -1,51 +1,57 @@ -participants: [] - -# 14-node Lean devnet4. Each participant pins the devnet4 image -# explicitly via `lean_image:`; the package's DEFAULT_LEAN_IMAGES point -# at `:latest` so they don't rot when a new devnet ships. +# 14-node Lean devnet4. Every participant pins the devnet4 image +# explicitly via `cl_image:`; DEFAULT_CL_IMAGES point at `:latest` so +# the defaults don't rot when a new devnet generation ships. # -# Single aggregator on ethlambda_0. Lighthouse omitted: the published -# `hopinheimer/lighthouse:latest` image is still on the single-key -# (devnet3) GENESIS_VALIDATORS layout and won't reach consensus with -# the devnet4 nodes. -lean_participants: - - lean_type: ethlambda - lean_image: ghcr.io/lambdaclass/ethlambda:devnet4 +# Single aggregator on the first ethlambda. lean_lighthouse is omitted: +# the published `hopinheimer/lighthouse:latest` image is still on the +# single-key (devnet3) GENESIS_VALIDATORS layout and won't reach +# consensus with the devnet4 nodes. +participants: + - el_type: none + cl_type: ethlambda + cl_image: ghcr.io/lambdaclass/ethlambda:devnet4 count: 1 validator_count: 1 is_aggregator: true - - lean_type: ethlambda - lean_image: ghcr.io/lambdaclass/ethlambda:devnet4 + - el_type: none + cl_type: ethlambda + cl_image: ghcr.io/lambdaclass/ethlambda:devnet4 count: 1 validator_count: 1 is_aggregator: false - - lean_type: ream - lean_image: ghcr.io/reamlabs/ream:latest-devnet4 + - el_type: none + cl_type: ream + cl_image: ghcr.io/reamlabs/ream:latest-devnet4 count: 2 validator_count: 1 is_aggregator: false - - lean_type: zeam - lean_image: blockblaz/zeam:devnet4 + - el_type: none + cl_type: zeam + cl_image: blockblaz/zeam:devnet4 count: 2 validator_count: 1 is_aggregator: false - - lean_type: qlean - lean_image: qdrvm/qlean-mini:devnet-4-amd64 + - el_type: none + cl_type: qlean + cl_image: qdrvm/qlean-mini:devnet-4-amd64 count: 2 validator_count: 1 is_aggregator: false - - lean_type: lantern - lean_image: piertwo/lantern:v0.0.4 + - el_type: none + cl_type: lantern + cl_image: piertwo/lantern:v0.0.4 count: 2 validator_count: 1 is_aggregator: false - - lean_type: grandine - lean_image: sifrai/lean:devnet-4 + - el_type: none + cl_type: lean_grandine + cl_image: sifrai/lean:devnet-4 count: 2 validator_count: 1 is_aggregator: false - - lean_type: gean - lean_image: ghcr.io/geanlabs/gean:devnet4 + - el_type: none + cl_type: gean + cl_image: ghcr.io/geanlabs/gean:devnet4 count: 2 validator_count: 1 is_aggregator: false diff --git a/.github/tests/lean-smoke.yaml b/.github/tests/lean-smoke.yaml index 128664a67..54ff15df3 100644 --- a/.github/tests/lean-smoke.yaml +++ b/.github/tests/lean-smoke.yaml @@ -1,13 +1,15 @@ # Smallest reproducible Lean devnet: one aggregator + one peer, both -# ethlambda. Comes up in ~3 min including hash-sig keygen. Use this as -# a sanity check before reaching for `lean-devnet4.yaml`. -participants: [] -lean_participants: - - lean_type: ethlambda +# ethlambda, no EL pairing. Comes up in ~3 min including hash-sig +# keygen. Use this as a sanity check before reaching for +# `lean-devnet4.yaml` or `ethlambda-el-pair*.yaml`. +participants: + - el_type: none + cl_type: ethlambda count: 1 validator_count: 1 is_aggregator: true - - lean_type: ethlambda + - el_type: none + cl_type: ethlambda count: 1 validator_count: 1 is_aggregator: false diff --git a/docs/architecture.md b/docs/architecture.md index b0b924e9e..30e857578 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -77,27 +77,38 @@ There are only two major difference between CL client and EL client launchers. F ## Lean Ethereum participants -Lean Ethereum consensus runs as a parallel top-level pipeline alongside the -EL/CL participant network. Today's Lean clients are client-only — Engine -API isn't implemented yet — so a Lean network configuration uses its own -`lean_participants:` block and skips the Eth1 EL/CL flow by setting -`participants: []`. When Engine API ships on the Lean side, the same -participants are designed to pair with EL clients in the regular EL+CL -shape. - -The pipeline lives under [`src/lean/`](../src/lean) and reuses the -prelaunch-data-generator pattern for genesis under -[`src/prelaunch_data_generator/lean_genesis/`](../src/prelaunch_data_generator/lean_genesis). -Per-client launchers translate the Lean Kurtosis record into each -client's CLI (mirroring the contract documented inline in each -`_launcher.star`). Genesis uses +Lean Ethereum consensus clients live in the standard `participants:` +block with a Lean `cl_type` — one of `ethlambda`, `ream`, `zeam`, +`qlean`, `lantern`, `gean`, `lean_grandine`, `lean_lighthouse`. The +standard CL dispatcher in [`src/cl/cl_launcher.star`](../src/cl/cl_launcher.star) +skips these; `main.star` then routes them to a parallel pipeline under +[`src/lean/`](../src/lean) that runs its own genesis and starts the +client binary. + +`ethlambda` is the only Lean client that implements Engine API today +([lambdaclass/ethlambda#367](https://github.com/lambdaclass/ethlambda/pull/367)); +it can be paired with any EL the way standard CLs are +(`el_type: ethrex, cl_type: ethlambda` etc.). The other Lean clients +run client-only with `el_type: none` until Engine API ships on their +side too. Lean-only deployments (every participant `el_type: none`) +relax the package's first-participant-must-have-EL guard. + +Genesis uses [`eth-beacon-genesis leanchain`](https://github.com/ethpandaops/eth-beacon-genesis) for the chain bundle and [`blockblaz/hash-sig-cli`](https://hub.docker.com/r/blockblaz/hash-sig-cli) -for XMSS validator keypairs. Metrics ride a vendored Prometheus + Grafana -stack with the upstream Lean client dashboard pre-loaded. - -Canonical args example: [`.github/tests/lean-devnet4.yaml`](../.github/tests/lean-devnet4.yaml). +for XMSS validator keypairs (lives in +[`src/prelaunch_data_generator/lean_genesis/`](../src/prelaunch_data_generator/lean_genesis)). +Per-client launchers translate each Lean record into the client's CLI +(see the contract documented inline in each `_launcher.star`). +Metrics ride a vendored Prometheus + Grafana stack with the upstream +Lean client dashboard pre-loaded. + +Canonical args examples: + +- [`.github/tests/ethlambda-el-pair.yaml`](../.github/tests/ethlambda-el-pair.yaml) — single EL + ethlambda pair +- [`.github/tests/ethlambda-el-pair-2node.yaml`](../.github/tests/ethlambda-el-pair-2node.yaml) — two EL + ethlambda pairs, one aggregator + one not +- [`.github/tests/lean-devnet4.yaml`](../.github/tests/lean-devnet4.yaml) — devnet4 multi-client (all `el_type: none`) ## Auxiliary Services diff --git a/main.star b/main.star index 94c6bc8b6..dc7b8c638 100644 --- a/main.star +++ b/main.star @@ -89,39 +89,6 @@ def run(plan, args={}): num_participants = len(args_with_right_defaults.participants) network_params = args_with_right_defaults.network_params - # Lean-only mode: when the operator configured `lean_participants:` but - # no Eth1 `participants:` entries, run ONLY the Lean pipeline. Today's - # Lean clients don't implement Engine API yet, so spinning up an EL+CL - # pair alongside would just waste resources and confuse downstream - # services that try to call into a non-existent Engine API. Once Lean - # clients ship Engine API support, the same `lean_participants:` entries - # will be able to pair with EL `participants:` and reuse the existing - # JWT + payload-attestation plumbing. The lean_launcher returns the - # per-node contexts; downstream consumers (prometheus, grafana, dora) - # can be wired through in a follow-up. - if num_participants == 0 and args_with_right_defaults.lean_participants: - plan.print( - "Lean-only mode: {0} lean participant entries, 0 EL/CL participants".format( - len(args_with_right_defaults.lean_participants) - ) - ) - lean_contexts = lean_launcher.launch( - plan, - args_with_right_defaults.lean_participants, - args_with_right_defaults.lean_network_params, - ) - return struct( - grafana_info=None, - blockscout_sc_verif_url=None, - all_participants=[], - lean_participants=lean_contexts, - pre_funded_accounts={}, - network_params=network_params, - network_id=network_params.network_id, - final_genesis_timestamp=None, - genesis_validators_root=None, - ) - # Detect the backend type early - needed for binary injection validation detected_backend = plan.get_cluster_type() @@ -363,18 +330,57 @@ def run(plan, args={}): ) all_xatu_sentry_contexts.append(participant.xatu_sentry_context) - # Launch Lean Ethereum consensus participants alongside the EL/CL network. - # Today's Lean clients run client-only (no Engine API yet) and use - # post-quantum signatures; the pipeline brings them up against their - # own genesis + libp2p QUIC mesh. Once Engine API lands on the Lean - # side, these participants will be able to pair with `participants:` - # EL clients and reuse the existing JWT plumbing. The list is empty - # unless the user populated `lean_participants:` in their args. - # See docs/lean-consensus.md for the architecture. + # Launch Lean Ethereum consensus participants. Lean clients are + # `participants:` entries whose `cl_type` is in + # constants.LEAN_CL_TYPES. The cl_launcher dispatcher already + # skipped them (None cl_context); here we build equivalent Lean + # records from each participant and hand them — together with the + # paired el_context (None when el_type is `none`) and the network + # JWT — to src/lean/lean_launcher.launch. ethlambda is the only + # Lean client that wires Engine API today (lambdaclass/ethlambda#367); + # the others run with `el_type: none`. + lean_records = [] + for index, participant in enumerate(args_with_right_defaults.participants): + if participant.cl_type not in constants.LEAN_CL_TYPES: + continue + paired_el_context = ( + all_el_contexts[index] if index < len(all_el_contexts) else None + ) + lean_records.append( + { + "lean_type": participant.cl_type, + "lean_image": participant.cl_image, + "count": participant.count, + "validator_count": participant.validator_count + or args_with_right_defaults.lean_network_params[ + "num_validator_keys_per_node" + ], + "is_aggregator": participant.is_aggregator, + "lean_extra_params": participant.cl_extra_params, + "lean_extra_env_vars": participant.cl_extra_env_vars, + "lean_extra_labels": participant.cl_extra_labels, + "lean_log_level": participant.cl_log_level, + "lean_min_cpu": participant.cl_min_cpu, + "lean_max_cpu": participant.cl_max_cpu, + "lean_min_mem": participant.cl_min_mem, + "lean_max_mem": participant.cl_max_mem, + "node_selectors": participant.node_selectors, + "tolerations": participant.tolerations, + "prometheus_config": { + "scrape_interval": participant.prometheus_config.scrape_interval, + "labels": participant.prometheus_config.labels or {}, + }, + # Underscore-prefixed fields are internal hand-offs to + # the Lean launcher. + "_el_context": paired_el_context, + } + ) + all_lean_contexts = lean_launcher.launch( plan, - args_with_right_defaults.lean_participants, + lean_records, args_with_right_defaults.lean_network_params, + jwt_file=jwt_file, ) # Generate validator ranges diff --git a/network_params.yaml b/network_params.yaml index 00f461297..0c1911446 100644 --- a/network_params.yaml +++ b/network_params.yaml @@ -292,21 +292,21 @@ port_publisher: # echo "Hello" extra_files: {} -# Lean Ethereum consensus participants. Each entry adds one or more Lean -# consensus nodes that run alongside (and independently of) the EL/CL -# network above. Lean has no EL pairing, no Engine API, no JWT, and uses -# post-quantum (XMSS) validator signatures. See docs/lean-consensus.md for -# the architecture and docs/lean-adding-a-new-client.md for adding a new -# Lean client. +# Lean Ethereum knobs (post-quantum signatures, XMSS validator keys, +# leanchain genesis). Lean clients themselves live in the standard +# `participants:` block above with a Lean cl_type (ethlambda, ream, zeam, +# qlean, lantern, gean, lean_grandine, lean_lighthouse). ethlambda is the +# only Lean client that wires Engine API today +# (lambdaclass/ethlambda#367); the rest run with `el_type: none`. # -# Example - 4 ethlambda nodes with one validator each: -# lean_participants: -# - lean_type: ethlambda -# lean_image: ghcr.io/lambdaclass/ethlambda:devnet4 +# Example - 4 ethlambda nodes paired with ethrex: +# participants: +# - el_type: ethrex +# cl_type: ethlambda +# cl_image: ghcr.io/lambdaclass/ethlambda:engine-api-integration # count: 4 # validator_count: 1 -# is_aggregator: false -lean_participants: [] +# is_aggregator: true lean_network_params: # Seconds added to "now" to compute GENESIS_TIME when genesis_time is 0. genesis_delay: 60 diff --git a/src/cl/cl_launcher.star b/src/cl/cl_launcher.star index 4011ea22c..849141fc8 100644 --- a/src/cl/cl_launcher.star +++ b/src/cl/cl_launcher.star @@ -147,6 +147,19 @@ def launch( for index, participant in enumerate(args_with_right_defaults.participants): cl_type = participant.cl_type el_type = participant.el_type + + # Lean cl_types are launched by the Lean pipeline (src/lean/), + # not by the standard CL dispatcher. We append `None` to the + # per-index context lists so downstream `all_cl_contexts[index]` + # lookups stay aligned with `args_with_right_defaults.participants`; + # main.star then runs the Lean launcher with the right el_contexts + # and jwt_file. Mirrors the EL_TYPE.none / cl-only path the package + # already supports for `consensoor` etc. + if cl_type in constants.LEAN_CL_TYPES: + all_cl_contexts.append(None) + all_snooper_el_engine_contexts.append(None) + continue + node_selectors = input_parser.get_client_node_selectors( participant.node_selectors, global_node_selectors, diff --git a/src/lean/ethlambda/ethlambda_launcher.star b/src/lean/ethlambda/ethlambda_launcher.star index 31e42a18d..e8861a739 100644 --- a/src/lean/ethlambda/ethlambda_launcher.star +++ b/src/lean/ethlambda/ethlambda_launcher.star @@ -69,13 +69,28 @@ def initialize(plan, node, p2p_keys_artifact, hash_sig_artifact): return plan.add_service(node["service_name"], ServiceConfig(**cfg_kwargs)) -def start(plan, node, service, genesis_artifact, hash_sig_artifact): +def start( + plan, + node, + service, + genesis_artifact, + hash_sig_artifact, + el_context=None, + jwt_file=None, + el_genesis_block_hash=None, +): """Phase 3: stage genesis text files into the running container and launch the ethlambda binary as a backgrounded process. `hash_sig_artifact` is unused here because it was already mounted in `initialize()` — we keep the parameter for shape parity with the other Lean clients. + + When `el_context`, `jwt_file`, and `el_genesis_block_hash` are all set, + ethlambda is launched with Engine API pairing + (`--execution-endpoint`, `--execution-jwt-secret`, + `--execution-genesis-block-hash`). Otherwise it boots Lean-only. + See lambdaclass/ethlambda#367 for the Engine API plumbing. """ service_name = service.name log_file = lean_shared.lean_log_file_path(service_name) @@ -148,6 +163,48 @@ def start(plan, node, service, genesis_artifact, hash_sig_artifact): ] if node["is_aggregator"]: cmd_parts.append("--is-aggregator") + + # Engine API pairing: present only when this node was synthesized from + # a `participants:` entry with a paired EL (lean_launcher fills the + # three values in tandem). We stage the JWT secret into the running + # container via plan.exec (same trick as the genesis files) since + # Kurtosis doesn't let us add a new files mount to an already-running + # service. + if el_context != None and jwt_file != None and el_genesis_block_hash != None: + jwt_path = "{0}/jwtsecret".format(GENESIS_MOUNT) + jwt_read = plan.run_sh( + run="cat /src/jwtsecret", + files={"/src": jwt_file}, + description="Reading JWT secret for {0}".format(node["service_name"]), + ) + plan.exec( + service_name=node["service_name"], + recipe=ExecRecipe( + command=[ + "/bin/sh", + "-c", + "cat > {0} <<'ETHLAMBDA_JWT_EOF'\n{1}\nETHLAMBDA_JWT_EOF".format( + jwt_path, + jwt_read.output, + ), + ], + ), + description="Staging JWT into {0}".format(node["service_name"]), + ) + engine_endpoint = "http://{0}:{1}".format( + el_context.ip_addr, el_context.engine_rpc_port_num + ) + cmd_parts.extend( + [ + "--execution-endpoint", + engine_endpoint, + "--execution-jwt-secret", + jwt_path, + "--execution-genesis-block-hash", + el_genesis_block_hash, + ] + ) + for extra in node["extra_params"]: cmd_parts.append(extra) diff --git a/src/lean/lean_launcher.star b/src/lean/lean_launcher.star index ea51f6138..8c6ea4095 100644 --- a/src/lean/lean_launcher.star +++ b/src/lean/lean_launcher.star @@ -10,14 +10,11 @@ Orchestrates the entire Lean pipeline: 4. Mount the genesis bundle into each placeholder and start the real client binary via `plan.exec`. -This is intentionally independent of the EL/CL `participant_network` pipeline: -Today's Lean devnets run client-only (no EL pairing yet), so this pipeline -brings up a Lean-only mesh. Engine API support is on the roadmap for the -Lean clients, after which the same `lean_participants:` entries will be -able to pair with EL clients from `participants:` and share the existing -package's Engine API / JWT plumbing. Operators opt in to Lean by populating -`lean_participants:` in their args; the existing EL/CL flow runs -unchanged either way. +Invoked from main.star with a list of Lean records derived from +`participants:` entries whose `cl_type` is in constants.LEAN_CL_TYPES. +Each record carries the paired `el_context` (None when `el_type: none`) +so the launcher can wire Engine API for clients that implement it (today +just ethlambda — lambdaclass/ethlambda#367). """ constants = import_module("../package_io/constants.star") @@ -77,11 +74,15 @@ def _launcher_for(lean_type): ) -def launch(plan, lean_participants, lean_network_params): +def launch(plan, lean_participants, lean_network_params, jwt_file=None): """Top-level entrypoint for the Lean pipeline. Returns the list of `lean_context` structs (one per running node), suitable for handing to Prometheus / Grafana / dora. + + `jwt_file` is the JWT artifact shared with paired ELs. It's only + consulted for nodes that carry an `_el_context` (synthesized by + main.star from `participants:` entries with a Lean cl_type). """ if not lean_participants: return [] @@ -131,6 +132,10 @@ def launch(plan, lean_participants, lean_network_params): "labels": {}, }, ), + # Optional Engine API pairing — populated by main.star + # when the participant has a non-`none` el_type. + # None for participants with `el_type: none` (Lean-only). + "_el_context": participant.get("_el_context"), } ) @@ -197,18 +202,65 @@ def launch(plan, lean_participants, lean_network_params): hash_sig_artifact, ) + # Phase 2.5: for nodes with a paired EL, query the EL's genesis + # block hash. ethlambda's `--execution-genesis-block-hash` flag seeds + # `state.latest_execution_payload_header.block_hash` so the very first + # `engine_forkchoiceUpdatedV3` carries a head the EL recognizes — + # without it the EL replies SYNCING forever. The hash is the same for + # every node sharing the same EL chain, but we still query per-EL + # because each Lean node points at a different EL service. + for node in expanded: + el_context = node["_el_context"] + if el_context == None: + node["_el_genesis_block_hash"] = None + continue + # `eth_getBlockByNumber 0x0` returns the genesis header; `.hash` + # is the 0x-prefixed 32-byte hash ethlambda expects. We strip + # quotes/newlines for clean substitution into the CLI flag. + rpc_url = "http://{0}:{1}".format( + el_context.ip_addr, el_context.rpc_port_num + ) + query = plan.run_sh( + run='set -eu; out=$(curl -sf -X POST -H "Content-Type: application/json" ' + + '--data \'{"jsonrpc":"2.0","method":"eth_getBlockByNumber","params":["0x0",false],"id":1}\' ' + + '{0}); echo "$out" | sed -E \'s/.*"hash":"(0x[0-9a-fA-F]+)".*/\\1/\''.format( + rpc_url + ), + description="Querying EL genesis block hash for {0}".format( + node["node_name"] + ), + wait="5m", + ) + node["_el_genesis_block_hash"] = query.output + # Phase 3: hand off to each per-client launcher to mount the genesis bundle - # and start the real binary. + # and start the real binary. ethlambda accepts optional EL params; the + # other Lean launchers ignore them. contexts = [] for node, service in services: launcher = _launcher_for(node["lean_type"]) - ctx = launcher.start( - plan, - node, - service, - genesis.genesis_artifact, - genesis.hash_sig_artifact, - ) + if ( + node["lean_type"] == constants.LEAN_TYPE.ethlambda + and node["_el_context"] != None + ): + ctx = launcher.start( + plan, + node, + service, + genesis.genesis_artifact, + genesis.hash_sig_artifact, + el_context=node["_el_context"], + jwt_file=jwt_file, + el_genesis_block_hash=node["_el_genesis_block_hash"], + ) + else: + ctx = launcher.start( + plan, + node, + service, + genesis.genesis_artifact, + genesis.hash_sig_artifact, + ) contexts.append(ctx) plan.print( diff --git a/src/lean/lighthouse/lighthouse_launcher.star b/src/lean/lighthouse/lighthouse_launcher.star index c3784042e..ce1b204c4 100644 --- a/src/lean/lighthouse/lighthouse_launcher.star +++ b/src/lean/lighthouse/lighthouse_launcher.star @@ -9,7 +9,7 @@ blockblaz/lean-quickstart, restricted to the flags the published NOTE: the image's `lighthouse lean_node` subcommand does NOT support `--api-port` or `--is-aggregator`. Lighthouse will run as a non-aggregator peer with only the metrics endpoint exposed; setting `is_aggregator: true` -in lean_participants is silently ignored for this client. +on this participant is silently ignored for this client. """ constants = import_module("../../package_io/constants.star") diff --git a/src/lean/metrics/metrics_launcher.star b/src/lean/metrics/metrics_launcher.star index 3d2a1ae8a..6546a4b9b 100644 --- a/src/lean/metrics/metrics_launcher.star +++ b/src/lean/metrics/metrics_launcher.star @@ -12,9 +12,8 @@ Layout: * One Grafana service named "lean-grafana" provisioning the Prometheus datasource and the upstream Lean client dashboard. -The whole thing is enabled by default when there are lean_participants; -add `additional_services: [grafana, prometheus]` to network_params.yaml -or set `lean_network_params.metrics_enabled: false` to skip it. +The whole thing is enabled by default whenever any participant has a +Lean cl_type; set `lean_network_params.metrics_enabled: false` to skip it. """ constants = import_module("../../package_io/constants.star") diff --git a/src/package_io/constants.star b/src/package_io/constants.star index 7c43efbc9..3bb9f2b0a 100644 --- a/src/package_io/constants.star +++ b/src/package_io/constants.star @@ -20,22 +20,55 @@ CL_TYPE = struct( grandine="grandine", consensoor="consensoor", caplin="caplin", + # Lean Ethereum CL clients. These live in `participants:` alongside the + # standard CL types but are launched by the Lean pipeline under + # `src/lean/` rather than `src/cl/`. The `cl_launcher.star` dispatcher + # skips Lean cl_types; the Lean launcher is then invoked from main.star + # with the participant's el_context (Some when paired with an EL, None + # when el_type is `none`). + # + # ethlambda is the only Lean client with Engine API today + # (lambdaclass/ethlambda#367). The rest run with `el_type: none`. + # `lean_grandine`/`lean_lighthouse` are prefixed to disambiguate from + # the standard Eth1 CLs above which share those repository names. + ethlambda="ethlambda", + ream="ream", + zeam="zeam", + qlean="qlean", + lantern="lantern", + gean="gean", + lean_grandine="lean_grandine", + lean_lighthouse="lean_lighthouse", +) + +# Set of CL types that route to the Lean pipeline. The cl_launcher +# dispatcher skips these (no eth-side beacon launch); main.star then +# builds Lean records from each such participant and hands them to +# lean_launcher.launch with the paired el_context (None for unpaired). +LEAN_CL_TYPES = ( + CL_TYPE.ethlambda, + CL_TYPE.ream, + CL_TYPE.zeam, + CL_TYPE.qlean, + CL_TYPE.lantern, + CL_TYPE.gean, + CL_TYPE.lean_grandine, + CL_TYPE.lean_lighthouse, ) # Lean Ethereum consensus clients (post-quantum signatures, leanchain -# genesis, libp2p QUIC). Today's Lean clients run client-only because -# Engine API isn't implemented on the Lean side yet; once it lands these -# will be able to pair with EL `participants:`. Until then, Lean -# participants live in `lean_participants:` and run via the separate -# pipeline in src/lean/. +# genesis, libp2p QUIC). Today only ethlambda implements Engine API on +# the Lean side, so the other clients run with `el_type: none`. The +# `grandine`/`lighthouse` values are prefixed with `lean_` to avoid +# colliding with the Eth1 CL types in CL_TYPE that share those names. LEAN_TYPE = struct( ethlambda="ethlambda", ream="ream", zeam="zeam", qlean="qlean", lantern="lantern", - grandine="grandine", - lighthouse="lighthouse", + grandine="lean_grandine", + lighthouse="lean_lighthouse", gean="gean", peam="peam", nlean="nlean", diff --git a/src/package_io/input_parser.star b/src/package_io/input_parser.star index c09516c0d..03b1f7c0d 100644 --- a/src/package_io/input_parser.star +++ b/src/package_io/input_parser.star @@ -26,6 +26,17 @@ DEFAULT_CL_IMAGES = { "grandine": "sifrai/grandine:stable", "consensoor": "ethpandaops/consensoor:main", "caplin": "ethpandaops/caplin:main", + # Lean CL clients. Routed through the Lean pipeline (src/lean/), not + # the standard CL launchers. Operators pin a specific devnet image + # per-participant via `cl_image:` in their args file. + "ethlambda": "ghcr.io/lambdaclass/ethlambda:latest", + "ream": "ghcr.io/reamlabs/ream:latest", + "zeam": "blockblaz/zeam:latest", + "qlean": "qdrvm/qlean-mini:latest", + "lantern": "piertwo/lantern:latest", + "gean": "ghcr.io/geanlabs/gean:latest", + "lean_grandine": "sifrai/lean:latest", + "lean_lighthouse": "hopinheimer/lighthouse:latest", } DEFAULT_CL_IMAGES_MINIMAL = { @@ -37,6 +48,14 @@ DEFAULT_CL_IMAGES_MINIMAL = { "grandine": "ethpandaops/grandine:develop-minimal", "consensoor": "ethpandaops/consensoor:main", "caplin": "ethpandaops/caplin:main", + "ethlambda": "ghcr.io/lambdaclass/ethlambda:latest", + "ream": "ghcr.io/reamlabs/ream:latest", + "zeam": "blockblaz/zeam:latest", + "qlean": "qdrvm/qlean-mini:latest", + "lantern": "piertwo/lantern:latest", + "gean": "ghcr.io/geanlabs/gean:latest", + "lean_grandine": "sifrai/lean:latest", + "lean_lighthouse": "hopinheimer/lighthouse:latest", } DEFAULT_VC_IMAGES = { @@ -65,24 +84,6 @@ DEFAULT_REMOTE_SIGNER_IMAGES = { "web3signer": "consensys/web3signer:latest", } -# Default Lean Ethereum client images. Each entry points at the client's -# generic `:latest` tag so the defaults don't rot when a new devnet -# generation ships. Operators pin a specific devnet by setting -# `lean_image:` per-participant in their args file (see -# `docs/lean-consensus.md` for a worked devnet4 example). -DEFAULT_LEAN_IMAGES = { - constants.LEAN_TYPE.ethlambda: "ghcr.io/lambdaclass/ethlambda:latest", - constants.LEAN_TYPE.ream: "ghcr.io/reamlabs/ream:latest", - constants.LEAN_TYPE.zeam: "blockblaz/zeam:latest", - constants.LEAN_TYPE.qlean: "qdrvm/qlean-mini:latest", - constants.LEAN_TYPE.lantern: "piertwo/lantern:latest", - constants.LEAN_TYPE.grandine: "sifrai/lean:latest", - constants.LEAN_TYPE.lighthouse: "hopinheimer/lighthouse:latest", - constants.LEAN_TYPE.gean: "ghcr.io/geanlabs/gean:latest", - constants.LEAN_TYPE.peam: "", # No published image yet - constants.LEAN_TYPE.nlean: "", # No published image yet -} - # MEV Params MEV_BOOST_PORT = 18550 @@ -91,7 +92,6 @@ DEFAULT_ADDITIONAL_SERVICES = [] ATTR_TO_BE_SKIPPED_AT_ROOT = ( "network_params", "participants", - "lean_participants", "lean_network_params", "mev_params", "blockscout_params", @@ -157,14 +157,11 @@ def input_parser(plan, input_args): result["zkboost_params"] = get_default_zkboost_params() result["buildoor_params"] = get_default_buildoor_params() - # Lean Ethereum: defaults are empty; users opt in by providing - # `lean_participants:` in their args. Parsed below if present. - result["lean_participants"] = [] + # Lean Ethereum: knobs (active_epoch, attestation_committee_count, + # num_validator_keys_per_node, metrics_enabled, ...) live in their + # own params block since they don't map cleanly to Eth1 fields. Only + # consulted when at least one participant has a Lean cl_type. result["lean_network_params"] = default_lean_network_params() - if "lean_participants" in input_args and input_args["lean_participants"]: - result["lean_participants"] = parse_lean_participants( - input_args["lean_participants"] - ) if "lean_network_params" in input_args: for k, v in input_args["lean_network_params"].items(): result["lean_network_params"][k] = v @@ -475,12 +472,18 @@ def input_parser(plan, input_args): ) # Fulu / PeerDAS validation only applies to the Eth1 EL/CL participant - # network. Lean-only deployments (participants: [], lean_participants: [...]) - # have no CL clients and therefore no PeerDAS surface, so we skip the - # validation when participants is empty. + # network. Skip it when every participant has a Lean cl_type — Lean + # clients don't speak PeerDAS. + has_any_eth1_cl = any( + [ + p["cl_type"] not in constants.LEAN_CL_TYPES + for p in result["participants"] + ] + ) if ( result["network_params"]["fulu_fork_epoch"] != constants.FAR_FUTURE_EPOCH and len(result["participants"]) > 0 + and has_any_eth1_cl ): has_supernodes = False has_node_with_128_plus_validators = False @@ -636,9 +639,15 @@ def input_parser(plan, input_args): # The "first participant must have an EL" check only applies when there # actually IS at least one Eth1 participant; lean-only deployments - # (participants: []) skip the EL/CL pipeline entirely. + # (participants: []) skip the EL/CL pipeline entirely. We also skip it + # when every participant has a Lean cl_type — Lean clients use their + # own libp2p QUIC mesh and don't need an Eth1 bootnode. + all_lean = len(result["participants"]) > 0 and all( + [p["cl_type"] in constants.LEAN_CL_TYPES for p in result["participants"]] + ) if ( len(result["participants"]) > 0 + and not all_lean and "bootnodoor" not in result["additional_services"] and result["participants"][0]["el_type"] == constants.EL_TYPE.none ): @@ -758,6 +767,7 @@ def input_parser(plan, input_args): vc_beacon_node_indices=participant["vc_beacon_node_indices"], checkpoint_sync_enabled=participant["checkpoint_sync_enabled"], skip_start=participant["skip_start"], + is_aggregator=participant["is_aggregator"], ) for participant in result["participants"] ], @@ -1142,10 +1152,9 @@ def input_parser(plan, input_args): builder_api=result["buildoor_params"]["builder_api"], epbs_builder=result["buildoor_params"]["epbs_builder"], ), - # Lean Ethereum. Stored as plain lists/dicts (not nested structs) + # Lean Ethereum knobs. Stored as a plain dict (not a nested struct) # because the Lean per-client launchers reach for fields by string # key — see src/lean/lean_launcher.star. - lean_participants=result["lean_participants"], lean_network_params=result["lean_network_params"], ) @@ -1877,6 +1886,9 @@ def default_participant(): "vc_beacon_node_indices": None, "checkpoint_sync_enabled": None, "skip_start": False, + # Lean CL knob — only consulted when cl_type is in constants.LEAN_CL_TYPES. + # Non-Lean participants ignore it. + "is_aggregator": False, } @@ -2606,33 +2618,13 @@ def get_devnet_modified_images(network_name, default_images): # Lean Ethereum parsing # --------------------------------------------------------------------------- # Lean is a post-quantum-signature consensus stack with its own genesis -# pipeline (PK's eth-beacon-genesis leanchain). Today's Lean clients are -# client-only - Engine API isn't implemented yet, so for now they run -# without an EL counterpart; once Engine API lands they're designed to pair -# with EL clients the same way today's CL clients do. Until then, Lean -# participants live in a separate `lean_participants:` list and run through -# `src/lean/lean_launcher.star`. - - -def default_lean_participant(): - return { - "lean_type": constants.LEAN_TYPE.ethlambda, - "lean_image": "", - "lean_log_level": "", - "lean_extra_params": [], - "lean_extra_env_vars": {}, - "lean_extra_labels": {}, - "lean_min_cpu": 0, - "lean_max_cpu": 0, - "lean_min_mem": 0, - "lean_max_mem": 0, - "count": 1, - "validator_count": 1, - "is_aggregator": False, - "node_selectors": {}, - "tolerations": [], - "prometheus_config": {"scrape_interval": "15s", "labels": {}}, - } +# pipeline (PK's eth-beacon-genesis leanchain). Lean clients live in the +# standard `participants:` block with a Lean `cl_type` value (see +# constants.LEAN_CL_TYPES). main.star synthesizes Lean records from each +# such participant and hands them to src/lean/lean_launcher.star, which +# runs the XMSS / hash-sig / leanchain genesis pipeline and starts the +# client binary. Network-level knobs (active_epoch, +# attestation_committee_count, ...) live in `lean_network_params:`. def default_lean_network_params(): @@ -2668,27 +2660,3 @@ def default_lean_network_params(): } -def parse_lean_participants(raw_participants): - """Normalize the lean_participants list by filling defaults per-entry.""" - parsed = [] - for raw in raw_participants: - entry = default_lean_participant() - for k, v in raw.items(): - entry[k] = v - # Resolve image: explicit override > registry default. We fail fast - # rather than silently shipping an empty image string downstream - # because Kurtosis's error in that case is opaque ("invalid image: "). - if entry["lean_image"] == "": - default_image = DEFAULT_LEAN_IMAGES.get(entry["lean_type"], "") - if default_image == "": - fail( - ( - "lean_type '{0}' has no default image; please set " - + "`lean_image` on this participant." - ).format(entry["lean_type"]) - ) - entry["lean_image"] = default_image - if entry["count"] < 1: - fail("lean_participants[].count must be >= 1") - parsed.append(entry) - return parsed diff --git a/src/package_io/sanity_check.star b/src/package_io/sanity_check.star index 9cead0f06..7d1ff6d9e 100644 --- a/src/package_io/sanity_check.star +++ b/src/package_io/sanity_check.star @@ -77,6 +77,7 @@ PARTICIPANT_CATEGORIES = { "vc_beacon_node_indices", "checkpoint_sync_enabled", "skip_start", + "is_aggregator", ], } @@ -487,11 +488,10 @@ ADDITIONAL_SERVICES_PARAMS = [ ] ADDITIONAL_CATEGORY_PARAMS = { - # Lean Ethereum participants: validated structurally inside the Lean - # input parser (see DEFAULT_LEAN_IMAGES + parse_lean_participants in - # input_parser.star), so we register the root keys here as opaque and - # let the parser raise on bad per-entry fields. - "lean_participants": "", + # Lean Ethereum network-level knobs (active_epoch, + # attestation_committee_count, ...). Per-participant Lean config + # rides on `participants:` entries with a Lean cl_type — see + # constants.LEAN_CL_TYPES. "lean_network_params": "", "wait_for_finalization": "", "global_log_level": "", diff --git a/src/participant_network.star b/src/participant_network.star index 1b4c0dd8e..7a422f9eb 100644 --- a/src/participant_network.star +++ b/src/participant_network.star @@ -348,6 +348,21 @@ def launch_participant_network( index_str = shared_utils.zfill_custom( index + 1, len(str(len(args_with_right_defaults.participants))) ) + # Lean cl_types own their own validator client logic (XMSS keys live + # inside the consensus binary). Skip the entire VC / remote-signer / + # snooper / metrics-exporter pipeline for them — those things assume + # an Eth1 beacon API that Lean clients don't expose. Pad the context + # lists that get appended-to in-loop so per-index alignment with + # `participants` holds. (`all_vc_contexts` is rebuilt below, so it + # doesn't need padding here.) + if cl_type in constants.LEAN_CL_TYPES: + all_remote_signer_contexts.append(None) + all_snooper_beacon_contexts.append(None) + all_snooper_el_rpc_contexts.append(None) + all_ethereum_metrics_exporter_contexts.append(None) + all_xatu_sentry_contexts.append(None) + continue + el_context = all_el_contexts[index] if index < len(all_el_contexts) else None cl_context = all_cl_contexts[index] if index < len(all_cl_contexts) else None diff --git a/src/prelaunch_data_generator/lean_genesis/lean_genesis_generator.star b/src/prelaunch_data_generator/lean_genesis/lean_genesis_generator.star index 55f254074..9ff879ba4 100644 --- a/src/prelaunch_data_generator/lean_genesis/lean_genesis_generator.star +++ b/src/prelaunch_data_generator/lean_genesis/lean_genesis_generator.star @@ -246,7 +246,7 @@ def generate_hash_sig_keys(plan, lean_network_params, total_validators): if total_validators < 1: fail( "Lean genesis requires at least one validator across all " - + "lean_participants (got 0).", + + "Lean participants (got 0).", ) _, hash_sig_image = _resolve_images(lean_network_params) return _generate_hash_sig_keys( @@ -488,7 +488,7 @@ def generate( if total_validators < 1: fail( "Lean genesis requires at least one validator across all " - + "lean_participants (got 0).", + + "Lean participants (got 0).", ) genesis_image, _ = _resolve_images(lean_network_params) diff --git a/src/shared_utils/shared_utils.star b/src/shared_utils/shared_utils.star index 10922a4cc..5485aa97b 100644 --- a/src/shared_utils/shared_utils.star +++ b/src/shared_utils/shared_utils.star @@ -209,13 +209,16 @@ def get_client_names(participant, index, participant_contexts, participant_confi cl_client = participant.cl_context el_client = participant.el_context vc_client = participant.vc_context + # Lean cl_types don't go through the standard CL launcher, so cl_context + # is None. Fall back to the cl_type string from the participant config so + # downstream consumers (validator-ranges, dora, etc.) still get a usable + # name for the row. + cl_name = cl_client.client_name if cl_client != None else participant_config.cl_type if el_client == None: - base_name = "{0}-{1}".format(index_str, cl_client.client_name) + base_name = "{0}-{1}".format(index_str, cl_name) else: - base_name = "{0}-{1}-{2}".format( - index_str, el_client.client_name, cl_client.client_name - ) - if vc_client != None and cl_client.client_name != vc_client.client_name: + base_name = "{0}-{1}-{2}".format(index_str, el_client.client_name, cl_name) + if vc_client != None and cl_name != vc_client.client_name: full_name = base_name + "-{0}".format(vc_client.client_name) else: full_name = base_name From 116506b3e6464bc8b3a796c08ed0fc9085909d85 Mon Sep 17 00:00:00 2001 From: ilitteri Date: Tue, 19 May 2026 22:41:51 -0300 Subject: [PATCH 18/25] Guard `fuzz_target` against empty `all_el_contexts`. When every participant has `el_type: none` (an all-Lean deployment) the EL launcher appends nothing, so `all_el_contexts[0].ip_addr` crashed at startup with "index 0 out of range: empty list". MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `fuzz_target` is only consumed by additional services that talk to an Eth1 EL (tx-fuzz, rakoon, broadcaster, custom_flood); leaving it empty is correct for Lean-only — those services aren't enabled there and the remaining additional-service handlers guard their own EL needs. --- main.star | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-) diff --git a/main.star b/main.star index dc7b8c638..eee4c467e 100644 --- a/main.star +++ b/main.star @@ -394,10 +394,17 @@ def run(plan, args={}): args_with_right_defaults.participants, ) - fuzz_target = "http://{0}:{1}".format( - all_el_contexts[0].ip_addr, - all_el_contexts[0].rpc_port_num, - ) + # `fuzz_target` is only consumed by additional services that talk to an + # Eth1 EL (tx-fuzz, rakoon, broadcaster, custom_flood). For all-Lean + # deployments (every participant `el_type: none`) there is no EL to + # point at; leave it empty and let the additional-service handlers + # guard their own use. + fuzz_target = "" + if len(all_el_contexts) > 0 and all_el_contexts[0] != None: + fuzz_target = "http://{0}:{1}".format( + all_el_contexts[0].ip_addr, + all_el_contexts[0].rpc_port_num, + ) # Broadcaster forwards requests, sent to it, to all nodes in parallel if "broadcaster" in args_with_right_defaults.additional_services: From 92ecc5b97e803156d32e1a0f60b7709d33260ad7 Mon Sep 17 00:00:00 2001 From: ilitteri Date: Tue, 19 May 2026 22:42:53 -0300 Subject: [PATCH 19/25] =?UTF-8?q?Replace=20underscores=20with=20hyphens=20?= =?UTF-8?q?when=20building=20the=20Kurtosis=20service=20name=20for=20Lean?= =?UTF-8?q?=20clients=20whose=20`lean=5Ftype`=20is=20itself=20hyphen-prefi?= =?UTF-8?q?xed.=20With=20the=20phase-2=20disambiguation,=20`LEAN=5FTYPE.gr?= =?UTF-8?q?andine=20=3D=20"lean=5Fgrandine"`=20and=20`LEAN=5FTYPE.lighthou?= =?UTF-8?q?se=20=3D=20"lean=5Flighthouse"`;=20the=20previous=20service=20n?= =?UTF-8?q?ame=20formatter=20produced=20`lean-lean=5Fgrandine-2`,=20which?= =?UTF-8?q?=20Kurtosis=20rejects=20per=20RFC=201035=20("only=20lowercase?= =?UTF-8?q?=20alphanumeric=20and=20`-`=20characters").=20The=20node=20name?= =?UTF-8?q?=20(used=20inside=20the=20Lean=20genesis=20/=20validator=20conf?= =?UTF-8?q?ig)=20keeps=20the=20underscore=20=E2=80=94=20that's=20the=20con?= =?UTF-8?q?vention=20lean-quickstart=20writes=20and=20the=20clients=20pars?= =?UTF-8?q?e.?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- src/lean/lean_launcher.star | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/src/lean/lean_launcher.star b/src/lean/lean_launcher.star index 8c6ea4095..0a0965aba 100644 --- a/src/lean/lean_launcher.star +++ b/src/lean/lean_launcher.star @@ -100,10 +100,13 @@ def launch(plan, lean_participants, lean_network_params, jwt_file=None): # `node_name` follows lean-quickstart's `_` # convention (passed as --node-id and used in validator-config.yaml). # Kurtosis service names, however, must match RFC 1035 — lowercase - # letters/digits/hyphens only — so we translate the underscore to - # a hyphen for the Kurtosis-facing name. + # letters/digits/hyphens only — so we translate any underscore in + # the lean_type (e.g. `lean_grandine`) to a hyphen for the + # Kurtosis-facing service name. node_name = "{0}_{1}".format(lean_type, idx) - service_name = "lean-{0}-{1}".format(lean_type, idx) + service_name = "lean-{0}-{1}".format( + lean_type.replace("_", "-"), idx + ) expanded.append( { "node_name": node_name, From c16f42140b242ba4e218e559e9e47d28a798d922 Mon Sep 17 00:00:00 2001 From: ilitteri Date: Tue, 19 May 2026 23:07:51 -0300 Subject: [PATCH 20/25] =?UTF-8?q?Add=20args=20file=20for=20the=20all-ELs?= =?UTF-8?q?=20experiment:=208=20EL=20clients=20=C3=97=202=20each,=20every?= =?UTF-8?q?=20node=20paired=20with=20ethlambda=20via=20Engine=20API=20(16?= =?UTF-8?q?=20EL+CL=20pairs=20total,=20one=20aggregator).=20Dora=20is=20ad?= =?UTF-8?q?ded=20as=20an=20additional=20service=20to=20give=20the=20EL=20s?= =?UTF-8?q?ide=20a=20beacon-explorer=20UI.?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Also gate dora's launcher loop on Lean cl_types — `cl_client` is None for Lean participants and dora's `new_cl_client_info(cl_client.beacon_http_url, ...)` was crashing before reaching the `el_type == none` skip. Same shape the other downstream pipelines (VC / snooper / metrics-exporter) already have. --- .github/tests/ethlambda-el-all-clients.yaml | 76 +++++++++++++++++++++ src/dora/dora_launcher.star | 5 ++ 2 files changed, 81 insertions(+) create mode 100644 .github/tests/ethlambda-el-all-clients.yaml diff --git a/.github/tests/ethlambda-el-all-clients.yaml b/.github/tests/ethlambda-el-all-clients.yaml new file mode 100644 index 000000000..563bce9b4 --- /dev/null +++ b/.github/tests/ethlambda-el-all-clients.yaml @@ -0,0 +1,76 @@ +# 8 EL clients × 2 each, each paired with an ethlambda CL via Engine API +# (16 total EL+ethlambda pairs). One ethlambda is the aggregator; the +# remaining 15 are non-aggregators. +# +# The ethlambda image is built locally from lambdaclass/ethlambda#367 — +# see the PR description for the build command. +# +# Dora is added as an additional service; it renders the EL side only +# (Lean cl_types don't expose an Eth1 beacon API). +participants: + # First ethrex+ethlambda pair is the network aggregator. + - el_type: ethrex + cl_type: ethlambda + cl_image: ghcr.io/lambdaclass/ethlambda:engine-api-integration + count: 1 + validator_count: 1 + is_aggregator: true + # Second ethrex+ethlambda — non-aggregator peer. + - el_type: ethrex + cl_type: ethlambda + cl_image: ghcr.io/lambdaclass/ethlambda:engine-api-integration + count: 1 + validator_count: 1 + is_aggregator: false + # 2× geth + - el_type: geth + cl_type: ethlambda + cl_image: ghcr.io/lambdaclass/ethlambda:engine-api-integration + count: 2 + validator_count: 1 + is_aggregator: false + # 2× nethermind + - el_type: nethermind + cl_type: ethlambda + cl_image: ghcr.io/lambdaclass/ethlambda:engine-api-integration + count: 2 + validator_count: 1 + is_aggregator: false + # 2× besu + - el_type: besu + cl_type: ethlambda + cl_image: ghcr.io/lambdaclass/ethlambda:engine-api-integration + count: 2 + validator_count: 1 + is_aggregator: false + # 2× reth + - el_type: reth + cl_type: ethlambda + cl_image: ghcr.io/lambdaclass/ethlambda:engine-api-integration + count: 2 + validator_count: 1 + is_aggregator: false + # 2× erigon + - el_type: erigon + cl_type: ethlambda + cl_image: ghcr.io/lambdaclass/ethlambda:engine-api-integration + count: 2 + validator_count: 1 + is_aggregator: false + # 2× nimbus-eth1 + - el_type: nimbus + cl_type: ethlambda + cl_image: ghcr.io/lambdaclass/ethlambda:engine-api-integration + count: 2 + validator_count: 1 + is_aggregator: false + # 2× ethereumjs + - el_type: ethereumjs + cl_type: ethlambda + cl_image: ghcr.io/lambdaclass/ethlambda:engine-api-integration + count: 2 + validator_count: 1 + is_aggregator: false + +additional_services: + - dora diff --git a/src/dora/dora_launcher.star b/src/dora/dora_launcher.star index 223f9d07f..3f251e186 100644 --- a/src/dora/dora_launcher.star +++ b/src/dora/dora_launcher.star @@ -48,6 +48,11 @@ def launch_dora( all_cl_client_info = [] all_el_client_info = [] for index, participant in enumerate(participant_contexts): + # Skip Lean cl_types entirely — they don't expose an Eth1 beacon + # API, so they can't be added to either the CL or EL lists that + # dora's config template renders against. + if participant_configs[index].cl_type in constants.LEAN_CL_TYPES: + continue full_name, cl_client, el_client, _ = shared_utils.get_client_names( participant, index, participant_contexts, participant_configs ) From 0f0f7d2bddf66fc5f398f0cec88f66e7fa38b214 Mon Sep 17 00:00:00 2001 From: ilitteri Date: Tue, 19 May 2026 23:13:13 -0300 Subject: [PATCH 21/25] =?UTF-8?q?Drop=20ethereumjs=20from=20the=20all-ELs?= =?UTF-8?q?=20ethlambda=20experiment=20=E2=80=94=20its=20container=20doesn?= =?UTF-8?q?'t=20open=20TCP=208546=20within=20Kurtosis's=202-minute=20port-?= =?UTF-8?q?check=20timeout,=20which=20fails=20the=20parallel=20start=20bat?= =?UTF-8?q?ch=20and=20rolls=20back=20every=20other=20EL.=20The=20other=207?= =?UTF-8?q?=20EL=20clients=20(geth,=20nethermind,=20besu,=20reth,=20erigon?= =?UTF-8?q?,=20nimbus,=20ethrex)=20are=20unaffected.=20Re-add=20when=20the?= =?UTF-8?q?=20ethereumjs=20image=20is=20fixed.?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .github/tests/ethlambda-el-all-clients.yaml | 11 ++++------- 1 file changed, 4 insertions(+), 7 deletions(-) diff --git a/.github/tests/ethlambda-el-all-clients.yaml b/.github/tests/ethlambda-el-all-clients.yaml index 563bce9b4..b2861bd4f 100644 --- a/.github/tests/ethlambda-el-all-clients.yaml +++ b/.github/tests/ethlambda-el-all-clients.yaml @@ -64,13 +64,10 @@ participants: count: 2 validator_count: 1 is_aggregator: false - # 2× ethereumjs - - el_type: ethereumjs - cl_type: ethlambda - cl_image: ghcr.io/lambdaclass/ethlambda:engine-api-integration - count: 2 - validator_count: 1 - is_aggregator: false + # ethereumjs intentionally omitted — its container in this package + # doesn't open the ports Kurtosis health-checks within the 2-minute + # timeout (TCP 8546 stays connection-refused), which rolls back the + # entire batch. Re-add when the ethereumjs image fix lands upstream. additional_services: - dora From ac90c385ba85e74c9a3d8ce801d5181adab94566 Mon Sep 17 00:00:00 2001 From: ilitteri Date: Tue, 19 May 2026 23:14:39 -0300 Subject: [PATCH 22/25] Narrow the all-ELs ethlambda experiment to 6 EL clients: ethrex, nethermind, geth, erigon, nimbus-eth1, besu. reth dropped from the experiment too; ethereumjs continues to be excluded because of its 2-minute TCP port-check timeout rolling back the whole batch. --- .github/tests/ethlambda-el-all-clients.yaml | 41 +++++++++------------ 1 file changed, 17 insertions(+), 24 deletions(-) diff --git a/.github/tests/ethlambda-el-all-clients.yaml b/.github/tests/ethlambda-el-all-clients.yaml index b2861bd4f..72babe63a 100644 --- a/.github/tests/ethlambda-el-all-clients.yaml +++ b/.github/tests/ethlambda-el-all-clients.yaml @@ -1,12 +1,16 @@ -# 8 EL clients × 2 each, each paired with an ethlambda CL via Engine API -# (16 total EL+ethlambda pairs). One ethlambda is the aggregator; the -# remaining 15 are non-aggregators. +# Cross-client EL × ethlambda experiment: 6 EL clients × 2 each, every +# node paired with ethlambda via Engine API (12 EL+CL pairs total). +# One ethlambda is the aggregator; the remaining 11 are non-aggregators. +# +# Client set: ethrex, nethermind, geth, erigon, nimbus-el (nimbus-eth1), +# besu. ethereumjs and reth omitted from this experiment. # # The ethlambda image is built locally from lambdaclass/ethlambda#367 — # see the PR description for the build command. # -# Dora is added as an additional service; it renders the EL side only -# (Lean cl_types don't expose an Eth1 beacon API). +# Dora is added as an additional service for the EL-side beacon-explorer +# UI; Lean cl_types don't expose an Eth1 beacon API so dora renders the +# EL side only. participants: # First ethrex+ethlambda pair is the network aggregator. - el_type: ethrex @@ -22,13 +26,6 @@ participants: count: 1 validator_count: 1 is_aggregator: false - # 2× geth - - el_type: geth - cl_type: ethlambda - cl_image: ghcr.io/lambdaclass/ethlambda:engine-api-integration - count: 2 - validator_count: 1 - is_aggregator: false # 2× nethermind - el_type: nethermind cl_type: ethlambda @@ -36,38 +33,34 @@ participants: count: 2 validator_count: 1 is_aggregator: false - # 2× besu - - el_type: besu + # 2× geth + - el_type: geth cl_type: ethlambda cl_image: ghcr.io/lambdaclass/ethlambda:engine-api-integration count: 2 validator_count: 1 is_aggregator: false - # 2× reth - - el_type: reth + # 2× erigon + - el_type: erigon cl_type: ethlambda cl_image: ghcr.io/lambdaclass/ethlambda:engine-api-integration count: 2 validator_count: 1 is_aggregator: false - # 2× erigon - - el_type: erigon + # 2× nimbus-eth1 + - el_type: nimbus cl_type: ethlambda cl_image: ghcr.io/lambdaclass/ethlambda:engine-api-integration count: 2 validator_count: 1 is_aggregator: false - # 2× nimbus-eth1 - - el_type: nimbus + # 2× besu + - el_type: besu cl_type: ethlambda cl_image: ghcr.io/lambdaclass/ethlambda:engine-api-integration count: 2 validator_count: 1 is_aggregator: false - # ethereumjs intentionally omitted — its container in this package - # doesn't open the ports Kurtosis health-checks within the 2-minute - # timeout (TCP 8546 stays connection-refused), which rolls back the - # entire batch. Re-add when the ethereumjs image fix lands upstream. additional_services: - dora From d52f13bdc6b807a8febc2472c9c3b0d173886bf1 Mon Sep 17 00:00:00 2001 From: ilitteri Date: Tue, 19 May 2026 23:28:25 -0300 Subject: [PATCH 23/25] =?UTF-8?q?Fix=20Lean=20participant=20count=20being?= =?UTF-8?q?=20applied=20twice.=20The=20standard=20input=20parser=20expands?= =?UTF-8?q?=20`count:=20N`=20into=20N=20separate=20`participants:`=20entri?= =?UTF-8?q?es=20(input=5Fparser.star:1260),=20each=20carrying=20the=20orig?= =?UTF-8?q?inal=20`count`=20attribute.=20main.star's=20synthesis=20was=20r?= =?UTF-8?q?eading=20that=20`count`=20and=20propagating=20it=20into=20the?= =?UTF-8?q?=20Lean=20record,=20so=20the=20Lean=20launcher's=20own=20count?= =?UTF-8?q?=20expansion=20multiplied=20N=C3=97N=20=E2=80=94=20a=20`count:?= =?UTF-8?q?=202`=20ream=20participant=20ended=20up=20running=204=20ream=20?= =?UTF-8?q?containers.?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Hardcode `count: 1` in the synthesized Lean record so the Lean launcher gets one node per already-expanded participant entry. --- main.star | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/main.star b/main.star index eee4c467e..2d3dcdb2e 100644 --- a/main.star +++ b/main.star @@ -350,7 +350,12 @@ def run(plan, args={}): { "lean_type": participant.cl_type, "lean_image": participant.cl_image, - "count": participant.count, + # The standard input parser already expanded `count: N` into + # N separate `participants:` entries (input_parser.star:1260), + # so we synthesize one Lean record per expanded participant + # and hardcode count=1 here — otherwise the Lean launcher's + # own count expansion would multiply N×N. + "count": 1, "validator_count": participant.validator_count or args_with_right_defaults.lean_network_params[ "num_validator_keys_per_node" From adaa9c73d26695000fb0c3afe6dbf3316b46c564 Mon Sep 17 00:00:00 2001 From: ilitteri Date: Tue, 19 May 2026 23:42:54 -0300 Subject: [PATCH 24/25] =?UTF-8?q?Drop=20dora=20from=20the=20all-ELs=20expe?= =?UTF-8?q?riment.=20Dora=20requires=20at=20least=20one=20Eth1=20beacon=20?= =?UTF-8?q?endpoint=20to=20start,=20and=20Lean=20cl=5Ftypes=20don't=20expo?= =?UTF-8?q?se=20one=20=E2=80=94=20every=20participant=20in=20this=20experi?= =?UTF-8?q?ment=20is=20a=20Lean=20cl=5Ftype,=20so=20dora's=20config=20temp?= =?UTF-8?q?late=20rendered=20zero=20endpoints=20and=20the=20container=20ex?= =?UTF-8?q?ited=20with=20"missing=20beacon=20node=20endpoints=20(need=20at?= =?UTF-8?q?=20least=201)".=20The=20rest=20of=20the=20devnet=20runs=20fine?= =?UTF-8?q?=20without=20dora;=20re-enable=20once=20ethlambda=20ships=20Bea?= =?UTF-8?q?con=20API=20compatibility=20stubs.?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .github/tests/ethlambda-el-all-clients.yaml | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/.github/tests/ethlambda-el-all-clients.yaml b/.github/tests/ethlambda-el-all-clients.yaml index 72babe63a..93ad18501 100644 --- a/.github/tests/ethlambda-el-all-clients.yaml +++ b/.github/tests/ethlambda-el-all-clients.yaml @@ -8,9 +8,9 @@ # The ethlambda image is built locally from lambdaclass/ethlambda#367 — # see the PR description for the build command. # -# Dora is added as an additional service for the EL-side beacon-explorer -# UI; Lean cl_types don't expose an Eth1 beacon API so dora renders the -# EL side only. +# Dora intentionally omitted — it requires at least one Eth1 beacon +# endpoint to start, and Lean cl_types don't expose one yet. Re-enable +# once ethlambda ships Beacon API compatibility stubs. participants: # First ethrex+ethlambda pair is the network aggregator. - el_type: ethrex @@ -62,5 +62,4 @@ participants: validator_count: 1 is_aggregator: false -additional_services: - - dora +additional_services: [] From e85ff53d0cee6757aaaef624115fbb5470a21459 Mon Sep 17 00:00:00 2001 From: ilitteri Date: Tue, 19 May 2026 23:53:12 -0300 Subject: [PATCH 25/25] Bump genesis_delay to 180s in the all-ELs ethlambda experiment. With 12 EL containers booting in parallel, several were too slow to answer engine_getPayloadV5 within the Lean 4s slot window during EL warm-up; the resulting empty slots broke 3SF-mini's delta-bounded finalization rule. 180s lets the ELs warm up before slot 0 starts. --- .github/tests/ethlambda-el-all-clients.yaml | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/.github/tests/ethlambda-el-all-clients.yaml b/.github/tests/ethlambda-el-all-clients.yaml index 93ad18501..a18da2752 100644 --- a/.github/tests/ethlambda-el-all-clients.yaml +++ b/.github/tests/ethlambda-el-all-clients.yaml @@ -62,4 +62,13 @@ participants: validator_count: 1 is_aggregator: false +lean_network_params: + # 180s gives the 12 EL containers time to warm up before the Lean + # chain starts producing blocks. With the default 60s, several early + # slots missed because proposers couldn't get a payload back from + # their paired EL within the 4s slot window during EL JIT/cache + # warm-up; those early gaps then blocked 3SF-mini's `delta ≤ 5` + # finalization rule. + genesis_delay: 180 + additional_services: []