feat: sequence-store publisher e2e suite - #47
Open
cffls wants to merge 9 commits into
Open
Conversation
Kurtosis devnet config with the in-enclave sequence store (4 publishing validators + 1 RPC, bor built from the sequencing branch until the publisher merges), a test suite porting the store-outage drill — publishers live post-Rio, 300s outage with stall-free production, recovery floored at milestone finality (forward jump, zero supersessions) — and a path-scoped workflow running it in this repo's CI. Consumable from bor CI like the stateless suite: copy configs/kurtosis-sequencer-e2e.yml, run tests/sequencer_tests/. The store image pulls from ghcr.io/0xpolygon/sequence-store (private; Actions read access granted to this repo and bor), with a documented build-from-source fallback. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Comment on lines
+42
to
+66
| runs-on: ubuntu-latest | ||
| timeout-minutes: 20 | ||
| steps: | ||
| - name: Checkout bor | ||
| uses: actions/checkout@v5 | ||
| with: | ||
| repository: 0xPolygon/bor | ||
| # The sequencer publisher lives on the sequencing branch | ||
| # (PR #2355); switch this to develop once it merges there. | ||
| ref: sequencing | ||
|
|
||
| - name: Build bor docker image | ||
| run: docker build -t bor:local --file Dockerfile . | ||
|
|
||
| - name: Save bor docker image | ||
| run: docker save bor:local | gzip > bor-image.tar.gz | ||
|
|
||
| - name: Upload bor docker image | ||
| uses: actions/upload-artifact@v4 | ||
| with: | ||
| name: bor-image | ||
| path: bor-image.tar.gz | ||
| retention-days: 1 | ||
|
|
||
| build-heimdall-v2: |
Comment on lines
+67
to
+89
| runs-on: ubuntu-latest | ||
| timeout-minutes: 20 | ||
| steps: | ||
| - name: Checkout heimdall-v2 | ||
| uses: actions/checkout@v5 | ||
| with: | ||
| repository: 0xPolygon/heimdall-v2 | ||
| ref: develop | ||
|
|
||
| - name: Build heimdall-v2 docker image | ||
| run: docker build -t heimdall-v2:local --file Dockerfile . | ||
|
|
||
| - name: Save heimdall-v2 docker image | ||
| run: docker save heimdall-v2:local | gzip > heimdall-v2-image.tar.gz | ||
|
|
||
| - name: Upload heimdall-v2 docker image | ||
| uses: actions/upload-artifact@v4 | ||
| with: | ||
| name: heimdall-v2-image | ||
| path: heimdall-v2-image.tar.gz | ||
| retention-days: 1 | ||
|
|
||
| e2e-tests: |
The sequence-store package has no latest tag — its docker workflow publishes dev from the default branch and version tags on v* releases; pull dev until one is cut. shfmt 3.13.1 formatting applied per the checks workflow. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The first run built bor from sequencing, which does not carry the publisher yet (0xPolygon/bor#2355 is unmerged) — no sequencer metrics exist on such a build and test 1 reads 0/4 forever. Build from the PR branch until it merges (then sequencing, then develop), and on a test-1 failure print each validator's sequencer series count and publish state, telling a sequencer-less image apart from a wrong-state publisher. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The sequencer e2e leg was failing in the "Pre kurtosis run" setup step (foundry download) at the old 0cdede0 main pin, so the sequencer suite never ran. v1.4.2 is the first kurtosis-pos release tag that carries the sequence-store launcher (fully contains 0cdede0) and ships the setup-action fix bor's own kurtosis legs already run with. Bumping to it unblocks the run now that the publisher branch it builds bor from is rebased onto develop. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The v1.4.1 pin's shared setup action installs foundry via foundry-toolchain@c7450ba (v1.8.0), which fails the "Pre kurtosis run" step with a corrupt foundryup download (`cannot execute binary file`). v1.4.2 bumps that action to v1.9.1 (908c5403) — the same fix bor develop landed in #2377. This puts all three pos-workflows kurtosis legs (e2e, stateless, sequencer) on v1.4.2. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
cffls
marked this pull request as ready for review
August 27, 2026 20:06
Extends the sequencer e2e suite with two scenarios proven on local kurtosis devnets that the current three tests don't cover: - pending-state readable on every validator: guards the regression where a signer outside the active producer set stopped refreshing its pending snapshot, so eth_getBalance/eth_call against "pending" returned "missing trie node / layer stale" on non-producing nodes. - producer takeover: stops the active producer's bor and requires the chain to keep advancing (a store window on a displaced parent used to wedge the seal barrier), then restarts it and requires a clean rejoin. Adoption/supersede counters are logged as evidence rather than asserted, since they depend on whether a dangling window existed at the cut. Also asserts reconcile displaced-records stay 0 in steady state (one owner per height; a displacement is a revoked preconfirmation). Scenarios needing a load generator (consumer preconf receipts, dual-publisher contention) or a bigger topology (multi-broker / multi- gateway chaos) are left for a follow-up that adds those to the CI rig. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ered preconfs The suite only checked producer self-reported counters (supersede, displaced). Add a check that reads the seqstore-auditor's evidence — an independent follower of the store log that classifies every superseded generation — and fails on any revocation that dropped preconfirmations or reordered transactions. This catches a producer that drops or reorders content even when its own metrics disagree, and a supersession storm (the pre-follow-model churn regression). Known limitation: "dropped" and "reordered" are per-transaction, so they only bite when blocks carry load. This rig has no transaction generator, so today the check asserts the absence of violations rather than provokes them. Adding a load step (so windows carry txs) turns it into a real revocation/reorder check, especially across the takeover cut — tracked as the follow-up alongside the load-dependent scenarios noted earlier. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The auditor's dropped/reordered signals are per-transaction, so they only mean anything when windows carry content. Add a background load generator (polycli transfer mode, the devnet's prefunded account, sent to the rpc node so no validator we stop is the entry point) that runs through the outage, recovery, and takeover phases, and verify transactions actually land before proceeding. The takeover now revokes real preconfirmations if it regresses, which the auditor check catches as dropped transactions. The sequencer workflow gains the polycli install step the sibling kurtosis legs already have. When polycli is unavailable the load is skipped, not failed, so the suite still runs without the load-dependent coverage. Reordering is only fully provoked by independent senders; a single loader's nonce order limits it, so that signal stays best-effort. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds the sequence-store & publisher e2e integration: a kurtosis devnet config with the in-enclave sequence store (4 publishing validators + 1 RPC on
bor:local, heimdall-v2, 4s blocks), a test suite porting the store-outage drill validated on a lab devnet against bor PR 0xPolygon/bor#2355, and a workflow running the suite in this repo's own CI. Consumable from bor CI exactly like the stateless suite: copyconfigs/kurtosis-sequencer-e2e.yml, runtests/sequencer_tests/.The tests assert: all publishers live post-Rio (
sequencer_publish_state=1, entries flowing); a 300s store outage costs zero block production (≥60 blocks at 4s); recovery floors the backfill at milestone finality (Sequencer backfill jumping finalized heightsin the producer log,reconcile_forwardjump ≥ 1,reconcile_supersede == 0), and publishing resumes.Notes:
ghcr.io/0xpolygon/sequence-storepackage (Actions read access granted to this repo and bor);SEQUENCE_STORE_TAGislatestuntil a release tag is pinned. A build-from-source fallback is documented in the workflow.0cdede09(the sequence-store launcher landed after v1.3.4); re-pin at the next release.sequencingbranch until the publisher merges to develop; triggers are path-scoped to this suite's files +workflow_dispatchso the feature-branch build never gates unrelated PRs.Executed tests
The drill this suite ports ran end-to-end on a kurtosis devnet (lab host): 300s store outage at heights 151..226, 75/75 blocks produced during the outage, recovery forward-jumped the finalized debt in one reconcile (floor=232 > debt 152..229, zero rebuilt), zero revocations/reorders/generation mismatches in the store probe. Locally: all YAML parses, both scripts pass
bash -n; the CI job itself has not yet run (needs this PR's workflow on a runner).Rollout notes
CI-only change; no node or network impact. Requires the ghcr package access already granted. Follow-ups tracked in comments: pin
SEQUENCE_STORE_TAG, re-pin kurtosis-pos to a release tag, switch the bor ref to develop after 0xPolygon/bor#2355 merges.🤖 Generated with Claude Code