Every check the agent runs against a chain spec, in the order it runs them. Self-contained: everything needed to read a pipeline result is on this page.
The pipeline runs 12 phases; the ones below are the ones that verify. They run cheapest-first, so a docker boot is only paid for once the offline gates are clean.
Not every spec has this trail. Only spec files onboarded or reworked through the pipeline carry the
evidence described here; most of those at the repo root predate it entirely. CI can also be
started mid-pipeline (Start at Phase N, N ∈ {8,9,10,11}), skipping 1–7 and reconstructing from the
committed file, and a mechanical-only PR — a field rename across five specs, say — may legitimately have no
Phase 8 at all. A merged spec has not necessarily passed everything here. To tell: find the PR that
introduced the spec (git log --oneline -5 -- <chain>.json) and look for magmadevs-bot comments. None
means no recorded testing.
| Phase | What runs | What it proves | If it fails |
|---|---|---|---|
| 6 pre-flight | Orchestrator, inline | Live chain-id curl (mainnet and testnet); index/name/enabled sanity; archive ↔ pruning ↔ GET_EARLIEST_BLOCK triplet all-present-or-all-absent |
Mixed triplet → STOP |
| 6 gates | 9 static gates, parallel subagents | The file is internally consistent and matches research (see below) | Any FAIL → one fixer pass, then straight on. Only the four offline gates re-run (Phase 10a); the rest are backstopped by Phases 9/11 |
| 7 | jq + removed-field guard |
Valid JSON; none of the 15 fields deleted from the smart-router model (nine governance fields, title/description, deposit, extra_compute_units, category.local, category.subscription) |
Either fails → fixer; no progress until both pass |
| 7.5 | Endpoint discovery | A real validated upstream exists for every interface, subscription and addon the spec declares — including inherited ones | NOT_TESTABLE is a transparent gap, never a stop |
| 8 | Dockerized smart-router boot + live probe (smart-router:main, --use-static-spec) |
The spec actually works against real nodes | BOOT_FAILED → STOP. PARSE/VERIFY/TESTNET_VERIFY: FAIL → carried into 9/10 as CRITICAL |
| 9 | 3 parallel reviewers | Three independent audits, each having read the probe report | Missing TALLY → abort |
| 10 | Consolidate gaps → one fix pass | Dedup, drop MINOR, strip probe-only disable suggestions | jq broken after fix → STOP |
| 10a | Re-run the four offline gates (method-schema, hanging_api, stateful, unused-fields) + disabled-count |
The fixer introduced nothing new. It is the likeliest source of these: told "a hanging API needs a timeout_ms", the natural answer is a flat 30000, which on a CU-1000 method shortens the budget |
Any FAIL → re-run the fixer on that item alone, before 10b |
| 10b | Smoke re-boot (smart-router:latest) + 7-item re-probe |
The fix broke nothing | REGRESSION/BOOT_FAILED → STOP |
| 11 | Final reviewer, clean context, prior reports archived | A fourth pass re-derived from the file alone | Any CRITICAL or MEDIUM → CHANGES REQUESTED, no retry loop |
| 12 | Coverage checklist + run stats | Reporting only | No gate |
Phases 1–5 build the spec. Their only stop is the builder's refuse-to-write gate (SPEC: BLOCKED), which
fires before any test runs.
- methods-coverage — every method research found is in the spec or its transitive imports. Unjustified
MISSING→ FAIL;EXTRA IN SPECis informational. - parse-directive — three layers. L1:
GET_BLOCKNUMpresent across the candidate plus all parents (hard-required;GET_BLOCK_BY_NUMabsent is anINFOline and passes — that selects head-only tracking, correct for a chain with no blocks);GET_BLOCK_BY_NUMtemplate carries%d/%x; it extracts a real block identifier, not the echoed slot number; template/parser_func/parser_argmatch the canonical matrix for(interface, family). L2: directives match the upstream template bytag|api|sha256. L3: each directive issued at a live RPC parses to the right type,GET_BLOCK_BY_NUMstill resolves when fed the numberGET_BLOCKNUMjust returned (counter chaining), and arest+PARSE_CANONICALbody is an object, not a top-level array. - chain-metadata —
average_block_timenon-zero,block_distance_for_finalized_datapresent,blocks_in_finalization_proof∈ {1, 3,max(ceil(1000/abt),3)},allowed_block_lag_for_qos_sync=ceil(10000/abt). - verifications — each has a
name, non-emptyvalues[], a validseverity; everyparse_directive.function_tagresolves inside its own collection. - extensions — every extension has a
cu_multiplier;archivehasrule.block; every addon collection has ≥1 verification; no duplicate extension names. - cu-semantic — hard rule: subscribe = exactly
1000CU, unsubscribe = exactly10. Everything else is an advisory band (tx-submit 10–40, simulate 40–60, heavy 60–200, state-read 10–20) and never fails the gate. Uniform CU across methods is not a defect. - pruning —
archive.rule.blockandpruning.latest_distancewithin 3× of researched retention (unknown→ INFO + pass); archive-tierexpected_valueis a base-10 integer on the path the routerParseInts (a"*"there excludes the archive provider at boot). - enabled — advisory watch-list of still-enabled methods research called unsupported. Always PASS; never auto-disables.
- method-schema — three scripts. Structure: per API,
enabled,compute_units,block_parsing,categorypresent;parser_argall strings; no duplicate API names in a collection.hanging_api: the flag must not appear on aSUBSCRIBE-tagged API (the router's subscription path never reads it); a hanging API must carry an explicittimeout_ms; and thattimeout_msmust be at leastmax(1s, CU × 100ms)— below that it shortens the relay budget, because it replaces the CU term rather than adding to it.stateful: a curated list of broadcast methods must be1and a curated list of simulate/encode/decode helpers must not be; disagreement with ≥90% of other specs declaring the same method is reported asINFOand never fails.
- Addon capability matrix — each addon/extension probed direct to each node before boot, so only genuinely supported ones enter the router config.
- Boot — the spec loads,
importsresolve, upstreams pass startup verification. A wrongchain-id, a brokenresult_parsing, an unresolvable import chain all surface here. PARSE_BLOCKNUM/PARSE_BLOCK_BY_NUM— the chain tracker executes the directives against real nodes; verdict from router metrics or tracker logs, with a failure from either winning. APARSE_BLOCKNUMfailure is the most severe defect possible: the router cannot track the chain.- Verification runtime scan — every spec verification runs per provider at boot. This is the only place
GET_EARLIEST_BLOCK/pruningever executes. Failing on every provider is a spec defect; failing on some is an upstream capability issue. - Method probe loop — every API relayed through the router, never direct. Inputs are resolved live (never a hardcoded
0x1), and any first-attempt failure on a parameterized method is re-probed once before it counts. HTTP 429 is never a verdict — the method becomesUNPROBED, notFAIL. - Subscriptions — WebSocket subscribe → ≥1 message within 30s → unsubscribe.
- Extensions — one
lava-extension: archivecall routed through the router. - Probe-window log scan — catches router errors hidden behind a plausible-looking response body.
- Testnet pass — boot + verifications against the testnet entry, no method probe. The only place the testnet
chain-idis ever live-checked. - Empirical block time — measured on both networks; >20% off the spec's effective value →
BLOCK_TIME_MISMATCH.
Each reviewer runs a 9-part audit — provider identification, network params, API completeness,
method-by-method (block parsing / category flags / CU), parse directives, verifications, inheritance,
headers, live tests — after reading the probe report. Phase 9 runs three in parallel and feeds CRITICAL +
MEDIUM into the fix pass (MINOR is dropped). Phase 11 runs a fourth with prior reports archived so they
cannot anchor it, and additionally audits that every enabled: false entry has a positive-evidence row
with a URL in the Disabled-API Justifications PR comment. CRITICAL=0 MEDIUM=0 = approved; anything else
stops the run.
Appending a testnet to an existing chain skips Phases 2–7 entirely and jumps to 7.5 → 8, scoped to the new
testnet. Its single gate is A4: valid JSON + the removed-field guard + the preservation guard, the last
of which fails unless the only change is the added testnet index and every pre-existing spec is
canonical-identical to main. The mainnet is byte-for-byte unchanged and is not re-probed. ::notice:: and exits 0, so a
skip reads as a pass.
| Verdict | Meaning |
|---|---|
PASS |
Returned a result field |
PASS-existence |
-32602 invalid params — method exists, the probe lacked valid args. Counts as a pass |
FAIL |
-32601 method not found — absent or not routed |
WARN |
Some other error code |
TIMEOUT |
No response in 10s |
SKIP |
stateful: 1 — would broadcast a transaction, deliberately never probed |
UNPROBED |
Never probed (rate limit, budget). Not coverage — "we don't know" |
PARTIAL |
Failed on some providers — that upstream lacks the capability |
NOT_TESTABLE |
No supplied node supports it at all |
NOT_EXERCISED |
Never ran |
TESTED_OK / TESTED_FAIL |
Addon verified / addon failed despite a supporting upstream |
SINGLE_UPSTREAM |
Degraded because only one node was supplied |
WARN-DISAGREEMENT |
Upstreams returned materially different shapes for the same method |
| Not covered | Why |
|---|---|
| Stateful / broadcast methods | Always SKIP at probe time — sending one would move funds. Their declaration is checked (gate 9's stateful rules); their behaviour is not. Test on a funded testnet if it matters. |
| gRPC through the router | The router's gRPC listener has no server reflection, so those probes go direct to upstream. Rows are labelled as such — they are not through-router coverage. |
| Testnet method coverage | The testnet gets boot + verifications only. No method probe, ever. |
| Compute units under load | Never benchmarked. CU is assigned by semantic band, not measurement. |
| Per-API block parsing, fully | Existence-tested only; full validation needs production traffic. |
Both of the category flag checks that used to be listed here as "no validator covers these — they are
the reviewer's job" are now enforced by gate 9 and re-run in Phase 10a. Neither is a manual step any more.
What still needs a human on those two flags is narrow but real:
check_stateful.sh'sINFOrows. They fire when ≥90% of other specs disagree with the candidate on a method that is on neither curated list. They are usually right, but consensus is not evidence — check the chain's own docs before acting. 24 of 5,539 distinct method names in the catalogue genuinely disagree.- A broadcast method the curated write list does not know. A new chain family can introduce one under a name nobody has seen. If it submits a transaction and is not on the list, only the consensus layer will notice, and only once enough specs declare it.
A probe error never justifies disabling a method. The pipeline probes free-tier public nodes only, so a
-32601, 501, 429 or timeout may prove nothing more than that the free tier lacks it. enabled: false
is legal only with positive evidence — official docs saying unsupported/removed, or the node client not
implementing it, with a URL — recorded in the Disabled-API Justifications PR comment. Three separate
checks enforce this: the enabled gate, Phase 10's disable-suggestion filter, and Phase 11's ledger audit.
The pipeline writes its reports to docs/<chain>/, which is gitignored — those files exist only on the
machine that ran it. A spec's durable test record is the set of magmadevs-bot comments on the PR that
introduced it, and the disabled-API ledger is a PR comment, not a file. The PR body is the input (human
brief, endpoints, known gotchas); the bot comments are the results, one per phase.
gh pr view <N> --json comments --jq '.comments[].body' \
| grep -E 'PARSE:|VERIFY:|TESTNET_VERIFY:|ADDONS:|TALLY:|PASS=[0-9]'| Red flag | Why it matters |
|---|---|
TESTNET_VERIFY: SKIPPED |
The testnet entry shipped with zero live verification |
PARSE: or VERIFY: FAIL still present at Phase 11 |
A known defect survived the fix pass |
| Phase 11 said CHANGES REQUESTED but the PR merged | Someone overrode the gate — find out why |
enabled: false with no ledger row |
Phase 11's audit should have caught it as CRITICAL |
ADDONS: … not-testable with no follow-up |
Nobody ever verified that addon |
BLOCK_TIME_MISMATCH on mainnet |
Should be impossible — an input was wrong |
| A Phase 10 gap "resolved against the reviewers" | A human overrode a reviewer's CRITICAL |
| Add-testnet PR with no preservation-guard line in the log | The guard may have self-skipped |
| No bot comments at all | The spec predates the pipeline — there is no recorded testing |