perf(aggregation): an aggregating node verifies inbound aggregates ~77% slower than a non-aggregating one

[dimka90]

Summary

On a mixed devnet where both clients verify the same gossiped aggregates, gean is consistently slower than ethlambda — +58% at p50, +32% on the mean, +57% at p99. The gap is across the whole distribution, not a tail artifact.

Aggregate verification runs per inbound gossip message, so this is hot-path work.

Measured

Local devnet, 8 gean + 6 ethlambda, 4 committees, ~2 h. Both clients on leanVM 48a90420. gean 14da17a, ethlambda 006453f4.

Both publish lean_pq_sig_aggregated_signatures_verification_time_seconds with identical bucket boundaries (0.1, 0.25, 0.5, 0.75, 1, 1.25, 1.5, 2, 4) and comparable sample counts — ethlambda 3,339 per node, gean 2,897.

	ethlambda	gean	gean slower
p50	0.0798 s	0.1261 s	+58%
p90	0.2096 s	0.2324 s	+11%
mean	0.0959 s	0.1265 s	+32%
p99	0.2462 s	0.3875 s	+57%

For context, the same run has the two clients matching to 3-4 significant figures on every other shared metric — attestation verify (4.951 ms vs 4.950 ms), attestation signing (0.531 s vs 0.526 s), state transition (0.2475 s vs 0.2475 s), gossip block size (248.5 KB both). Aggregate verification is the one place they diverge.

The measurement is clean

Checked before filing, because a same-named metric does not have to mean the same thing:

• One call site. metrics.ObservePqSigAggVerificationTime is recorded only at internal/node/gossip.go:96, wrapping attestation.VerifyAggregatedGossipAttestation. Nothing in the aggregation path feeds the histogram, so gean is not mixing in child-proof verification from recursive aggregation.
• The shadow simulator is inert. e.Shadow.SleepVerifyAggregated sits inside the timed region (gossip.go:95), but shadow-xmss-verify-aggregated-signatures-rate defaults to 0, shadow.sleep returns immediately at rate ≤ 0, and no shadow flag or env var is set on any node in the run.
• Same population. Both clients are timing verification of aggregates arriving over gossip, in a network where all four aggregators are gean, so both see the same stream of aggregates.

Why it matters

Verification is per inbound aggregate on the gossip path. At ~2,900 verifications per node over two hours, an extra ~46 ms at p50 is real CPU that competes with proving and with the engine loop. It is the same class of cost as #445 (block-proof verification delaying ticks), on a different path.

Not yet known

What ethlambda does differently. Candidates worth checking before optimising:

• whether the two spend their time in the same place (leanVM verify vs the surrounding claim/signer-set reconstruction)
• whether gean rebuilds the signer set per verification where ethlambda caches or reuses
• whether PubKeyCache is being hit on this path
• whether the difference is parallelism rather than per-call cost

A CPU profile of VerifyAggregatedGossipAttestation under devnet load would answer most of that. #443 asks for a reproducible proof benchmark; this path deserves one too.

Reproduce

Run a mixed devnet and compare the shared histogram:

histogram_quantile(0.50, sum by(job,le)(rate(lean_pq_sig_aggregated_signatures_verification_time_seconds_bucket[15m])))

[dimka90]

Correction — the headline figure in this issue is wrong

I filed this from a fleet-wide average without splitting aggregators from non-aggregators. Doing that split changes the conclusion.

Per-node p50 over 30 minutes, same run:

ethlambda_0  0.0794    gean_0  0.1568  (aggregator)    gean_4  0.0933
ethlambda_1  0.0806    gean_1  0.1582  (aggregator)    gean_5  0.0871
ethlambda_2  0.0751    gean_2  0.1598  (aggregator)    gean_6  0.0921
ethlambda_3  0.0752    gean_3  0.1590  (aggregator)    gean_7  0.0859
ethlambda_4  0.0813
ethlambda_5  0.0823

All four aggregators in this devnet are gean (lean_is_aggregator), and ethlambda aggregates nothing. So the fleet average compared gean's loaded aggregators against ethlambda's unloaded nodes.

comparison	p50	gap
gean non-aggregators vs ethlambda	0.0896 vs 0.0790	+13%
gean aggregators vs gean non-aggregators	0.1585 vs 0.0896	+77%

What the numbers actually say

• There is no large cross-client gap. gean's non-aggregating nodes verify at ~0.090 s against ethlambda's ~0.079 s. A 13% difference is worth a look eventually, but it is not the 58% claimed above and it may be noise at this sample size.
• There is a real aggregator penalty. A gean node building aggregates verifies inbound ones ~77% slower than an otherwise identical gean node that is not. Those nodes run at roughly 200% CPU against ~22% for non-aggregators, so the likely cause is contention with the proving work on the same host, not the verification path itself.

That second effect is the finding worth keeping. It says aggregation does not only cost the aggregator its own proving time; it degrades that node's handling of everything else on the gossip path, which is the same shape of problem as #445.

What would settle it

Run a devnet with aggregators split evenly between the clients, so both sides have loaded and unloaded nodes, and compare within and across clients. That separates "gean vs ethlambda" from "aggregating vs not" — which this run cannot do, because the two are perfectly correlated.

The measurement hygiene in the original report still holds: one call site (gossip.go:96), identical histogram buckets, shadow simulator inert. The error was in the aggregation of the numbers, not in what was being measured.

[dimka90]

Related: #446 covers the mechanism — aggregate verification is serialised on the gossip topic goroutine. This issue is the measurement of how much worse that gets on a node that is also aggregating (0.159 s vs 0.090 s p50), and #446 now carries the same per-node split.

Also adjacent: #433, on what the prover is doing on those nodes while this verification is waiting.