Skip to content

feat(create-spec): enforce the hanging_api and stateful rules in the method-schema gate - #137

Merged
avitenzer merged 6 commits into
mainfrom
feat/hanging-api-gate
Oct 7, 2026
Merged

avitenzer merged 6 commits into
mainfrom
feat/hanging-api-gate

Conversation

@AnnaR-prog

@AnnaR-prog AnnaR-prog commented Sep 2, 2026 •

Copy link
Copy Markdown
Contributor

Adds check_hanging_api.sh and check_stateful.sh to the method-schema gate — the two category.* flags that steer routing and had no validator coverage between them. Ticket: MAG-3389. Companion to #136, which fixes the specs this gate would flag.

Why

category.hanging_api sets the relay window. GetRelayTimeout (protocol/chainlib/common.go:600 in smart-router d13051cf) bounds a caller's timeout override with BoundCallerRelayTimeout; the spec's own inputs are read by routerRelayTimeout (common.go:613):

func routerRelayTimeout(chainMessage, averageBlockTime) time.Duration {
    extraRelayTimeout := 0
    if IsHangingApi(chainMessage) { extraRelayTimeout = averageBlockTime * 2 }
    relayTimeAddition := common.GetTimePerCu(GetComputeUnits(chainMessage))
    if chainMessage.GetApi().TimeoutMs > 0 {
        relayTimeAddition = time.Millisecond * time.Duration(chainMessage.GetApi().TimeoutMs)
    }
    return extraRelayTimeout + relayTimeAddition
}

The flag has two other readers: the attempt budget (GetTimeoutInfo, common.go:803 → GetTimeoutForProcessing, which raises it to 6× DefaultTimeout for a hanging call, protocol/common/timeout.go:198) and the recovery probe (recovery_probe.go:136). The rules below concern the window. Two properties of routerRelayTimeout generate all three.

The three rules

1. hanging_api must not appear on a SUBSCRIBE-tagged API → FAIL

consumer_websocket_manager.go:525 branches on the SUBSCRIBE function tag straight into StartSubscription. Every GetRelayTimeout call site (five today: rpcsmartrouter_server.go:2305, :5334, :5507, :6069 and common.go:117) is on the unary path, and no subscription manager references it. A subscription's lifetime is its socket's — the WS manager's only timeouts are two hardcoded 10s constants for unsubscribe teardown.

So over WebSocket, where a subscription is served, hanging_api, compute_units and timeout_ms are dead inputs on a SUBSCRIBE-tagged API, and setting the flag asserts something the subscription path never reads. (A SUBSCRIBE call sent over HTTP takes the unary path like any other, but cannot deliver a subscription there.) 3 of 288 SUBSCRIBE-registered APIs in the catalogue did it when this was written. #136 cleared two and #141 the last (dydx.json); on main today (155 specs) none is left.

SUBSCRIBE names are collected inheritance-aware, walking the imports graph the way check_directive_presence.sh does — an L2 importing ETH1 has an empty parse_directives array of its own.

2. hanging_api: true requires an explicit timeout_ms → FAIL

Automates the rule already stated in SKILL.md's Phase 6 pre-flight checklist and agents/spec-builder.md:59, both of which flagged it as "no validator covers this." Now one does.

3. timeout_ms must be at least max(1s, CU × 100ms) → FAIL

timeout_ms replaces the CU term rather than adding to it, so a smaller value shortens the relay window relative to setting nothing at all — the opposite of why anyone sets it. (The window decides when the router tries another provider; the attempt budget above is separate.)

Caught live while writing #136: Acala's watch pair at CU 1000 carries a 100 000 ms implied base, so a reflexive timeout_ms: 30000 would have cut the window from 124s to 54s. Zero specs currently violate this, so it is a pure regression guard.

The 1s is the default MinimumTimePerRelayDelay. A deployment that raises it with --min-relay-timeout raises the real window too, so the check stays conservative there.

Rule 1 short-circuits — a subscription is never judged on its timeout, since neither field is read.


Part 2 — check_stateful.sh

category.stateful is a routing instruction, not a label. It maps to CONSISTENCY_SELECT_ALL_PROVIDERS, and the router acts on it in four places (rpcsmartrouter_server.go:1384, :4112, :4921, :6008): the call is fanned out to every provider and excluded from cross-validation (the ValidateNoStatefulPolicies guard, cross_validation_policy.go:572, wired at cross_validation_boot.go:107-119) and the recovery probe (recovery_probe.go:112-133).

Both directions fail silently:

Mistake Consequence
stateful: 0 on a write broadcast reaches one provider — no redundancy where it matters most
stateful: 1 on a read every call fanned out to all providers, billed accordingly, cross-validation lost

Scope caveat, verified rather than assumed. Those per-relay effects are unary-only. Like GetRelayTimeout, GetStateful is unreachable from the subscription path — it appears in rpcsmartrouter_server.go, recovery_probe.go, unified_relay_state_machine.go and common.go, and in neither subscription manager. So on a SUBSCRIBE-tagged method such as author_submitAndWatchExtrinsic, provider fan-out does not depend on this flag.

It does not follow that the flag is inert there, and this is where stateful differs from hanging_api. ApiHasStatefulCategory (base_chain_parser.go:608, via cross_validation_boot.go:107) looks the method up by name, independent of transport, and the cross-validation policy guard uses it to reject an enabled per-method CV policy on a write. That is why the write list still FAILs on subscription methods rather than short-circuiting them the way check_hanging_api.sh rule 1 does: hanging_api has no name-keyed reader, stateful does.

Why this one can't be done the same way

hanging_api's rules fall out of the router source. "Does this method change chain state" is chain knowledge instead.

And it can't be settled by vote either: before #136 author_submitAndWatchExtrinsic split 10 specs at 0 against 11 at 1, with kusama.json and polkadot.json on opposite sides. A majority rule produced no answer exactly where one was needed. (After #136 and #141, all 21 declare 1.)

So: three layers of decreasing confidence.

1. Curated write list → FAIL. The Substrate submit/submitAndWatch family, eth_sendRawTransaction/eth_sendTransaction/eth_sendRawTransactionSync, BroadcastTx, broadcast_tx_*, Solana sendTransaction, the three Starknet add*Transaction. Curated rather than voted, so it fires even when the whole catalogue agrees and is wrong.

2. Curated read list → FAIL. Simulation and codec helpers. eth_fillTransaction is on it by name — SPEC_GUIDE.md and phase3.2 both call it the classic trap: a *_fill*/*_prepare* helper whose argument shape looks like a transaction.

3. Cross-spec consensus → INFO, never FAIL. Fires when ≥90% of the other specs declaring a name disagree with the candidate, and at least 4 declare it. Advisory, because some names genuinely disagree across chains (24 of 5,539 distinct method names at the 2026-09-06 calibration).

REST entries are keyed "TYPE /path" where the verb disambiguates — GET /cosmos/tx/v1beta1/txs is a query, POST to the same path is BroadcastTx. The verb resolves a name collision; it does not classify. Treating POST as "is a write" is exactly the bug that put four read endpoints on the write path in cosmossdk.json (#138).

Calibration over all 155 specs (re-run 2026-10-07 on main)

2 FAIL rows in 2 specs, 11 INFO rows. Both FAILs are latent: monad and optimism carry eth_sendTransaction at stateful: 0 on a method that is enabled: false. They are not fixed here. They belong to the catalogue cleanup in MAG-3401.

Fixed since the first calibration (2026-09-06: 29 FAIL rows in 16 of 140 specs): avt and bsx by #136; babylon, kava, sei (decode/amino at 1) plus cosmossdk's four REST reads by #138; and the watch pair at stateful: 0 on astar, enjin, hydration, kusama, kusamaassethub, moonriver, polkadot_asset_hub and sdn by #141. All three have merged.

The 11 INFO rows are unchanged: lit.json's seven node-admin methods (author_insertKey, author_rotateKeys, offchain_localStorageSet, system_add/removeReservedPeer, system_add/resetLogFilter) at 0 against 16 specs at 1, plus bittensor/etc on eth_submitWork, eth_submitHashrate and transaction_v1_stop.

Two defects the tests caught

Worth naming because both would have made the consensus layer quietly useless rather than wrong:

  1. The dedupe key dropped the filename, so sort -u collapsed the whole catalogue to one row per distinct value and the threshold could never be reached.
  2. The candidate was counted in its own denominator, contradicting the "N/M other specs" wording and pushing 4-of-4 agreement below the 90% bar.

Phase 10a — so the gates fire where it matters

Gate 9 runs in Phase 6, before the Phase 10 fixer, and the only post-fix validation was jq .. Any defect the fixer introduces therefore shipped unguarded — and the fixer is the likeliest source, since it applies field-level instructions without surrounding context.

check_hanging_api.sh rule 3 is the sharpest example: told only "a hanging API needs a timeout_ms", the natural response is a flat 30000, which on a CU-1000 method shortens the window. That is exactly the mistake made by hand on Acala while writing #136 — so without this the guard could not fire on the case that motivated it.

This PR adds Phase 10a, re-running check_method_schema, check_hanging_api, check_stateful and check_unused_fields after the fixer — all offline, sub-second, no network. It also finally wires check_disabled_count.sh (#133) after the fixer, in its report-only form: the count it prints is the value the PR body's disabled-count marker must carry. It was left unwired for precisely this timing reason.

Established specs: --base

An update, add-testnet or add-collection run edits one of the established specs, and 28 of them still carry rule-2 rows (152 APIs; btc.json has 3), as do monad and optimism for check_stateful.sh. Judged on the whole file, both gates would fail such a run on rows it never touched, and Phase 10a would send them back to the fixer as regressions. That widens a surgical PR, and on an add-collection PR it would trip #163's guard.

So both scripts take --base <file>. A FAIL row the base already fails with identical text moves to === INFO === as [pre-existing …]; a new or changed row still FAILs.

  • Which base. Phase 6 and Phase 10a both pass the file as it is on origin/main, and nothing for a new chain. Phase 10a deliberately does not use the pre-fix snapshot: that is taken after the Phase-6 fixer, so a Phase-6 defect that survived would be excused as pre-existing. Run on a new chain with the Acala mistake, Phase 10a fails it; judged against the snapshot instead, it passed.
  • Rows name their collection (interface, REST verb, internal_path, add_on, e.g. jsonrpc@debug) and are matched by count. So a method that already fails in one collection does not excuse the same defect in a collection the run adds, which is the add-collection case.
  • The base is judged in the candidate's directory, by its own imports. An inherited SUBSCRIBE directive resolves the same way for both, and a candidate that newly imports a SUBSCRIBE parent still FAILs.

On main, zcash.json and solana.json carry one check_hanging_api.sh FAIL row each, starknet.json six, and monad.json and optimism.json one check_stateful.sh row each. All five pass against themselves as base, and across all 155 specs every row moves to INFO.

Wiring

Both folded into gate 9 (method-schema) rather than added as new gates — all three scripts are per-method field checks, and gate 7 (pruning) already sets the precedent of one gate running multiple scripts.

Updated: the gate table, TESTING.md §9, method-schema-validator.md, and two SKILL.md pre-flight bullets. Both of those bullets previously said the rule had no validator coverage; both now do, so the pre-flight section no longer carries any unchecked item.

The method-schema gate's ADVISORY block (check_stateful.sh's consensus rows, and [pre-existing …] rows) is added to Phase 6's severity routing, which now lists four gates. Those rows are printed to the user, never sent to the fixer, and never fail the gate.

Scope

Both scripts read the candidate file only, which is how the pipeline uses it. 152 APIs across 28 established specs (ethereum, cosmossdk, tendermint, solana, kusama …; re-counted 2026-10-05) predate rule 2 and would fail if this ran repo-wide — a separate cleanup, tracked in MAG-3401, deliberately not attempted here. An update of one of those specs is judged with --base, above.

Tests

test_check_hanging_api.sh — 20 cases: each of the three rules, rule-1 precedence over rules 2–3, the 1s CU floor at exactly 999 and 1000, inherited SUBSCRIBE directives resolved through imports, non-hanging APIs ignored, a spec with no hanging APIs at all, and base mode (an unchanged failing row is INFO; a changed, edited or new one FAILs; no base judges every row; imports resolved in context and by the base's own; a failing row in a new add-on collection FAILs; rows matched by count; an unreadable base exits 2).

test_check_stateful.sh — 17 cases: both curated lists in both directions, the eth_fillTransaction trap, REST verb disambiguation (POST /txs must be 1, GET /txs must not be forced), consensus being advisory rather than fatal, the 4-sibling threshold, regression cases for both defects above, and the same base-mode cases.

All 19 suites pass (2026-10-07, rebased on main) under bash 5.3 on macOS and bash 5.2 on Ubuntu 24.04, the CI runner. Under bash 3.2, 7 fail, as TESTING.md documents.

Also corrects the test-suite table, which had fallen behind the test files: it now has 19 rows, one per test file, including the node-admin, collection-addition and update-diff tests that came in from main.

@AnnaR-prog AnnaR-prog changed the title feat(create-spec): enforce the three hanging_api rules in the method-schema gate feat(create-spec): enforce the hanging_api and stateful rules in the method-schema gate Sep 2, 2026
@AnnaR-prog
AnnaR-prog force-pushed the feat/hanging-api-gate branch from 8a44f0c to 2c87d74 Compare September 6, 2026 09:34
AnnaR-prog added a commit that referenced this pull request Sep 27, 2026
…AG-3644 item 4) (#153)

* feat(create-spec): gate node-operator RPCs out of base collections (MAG-3644)

MAG-3644 item 4: should create-spec flag node-admin RPCs during synthesis?
Yes — and the first catalogue sweep proves why.

`bch.json` served BCHN's parkblock/unparkblock/finalizeblock from BCH's base
collection, enabled, at 10 CU. It reached main, two Phase 11 reviews passed
over it, and it was only caught once a reviewer was bumped to opus. Review is
not a reliable gate for this; a script is.

## The predicate

enabled == true AND add_on == "" AND the method acts on the NODE rather than
the chain.

`add_on` is part of the predicate, not a detail. An add-on is an opt-in
boundary a provider can decline; a base collection is not — every provider
serving the chain over that interface offers the method, and importing specs
inherit it through CombineCollections. MAG-3644's severity argument turns on
exactly this, so `debug_setHead` under a `debug` add-on is reported as INFO,
not failed.

The classes are grouped by what an attacker gets — consensus view, peering,
lifecycle, mining, keys, regtest mining — because someone asking "is my new
method one of these?" needs the reason, not the list. `personal_*` is matched
by prefix; enumerating it would fall behind a client release.

Out of scope on purpose: submitblock/submitblocklight. Block submission
changes CHAIN state, not the node's view of it, and is arguably legitimate for
a mining consumer. Its stateful/deterministic typing belongs to the
method-schema gate (and to PR #137, which is already in that territory).

## Why a baseline file exists

The sweep found the exposure is not bch-only:

  etc.json ETC   28 methods at add_on "" — admin_*, miner_*, and the entire
                 personal_* namespace: unlockAccount, importRawKey,
                 newAccount, sendTransaction, signTransaction. Key material
                 and signing on the PROVIDER'S node, as priced relays. Filed
                 as MAG-3808 (High) rather than buried here.
  zcash.json     generate, setgenerate — regtest block fabrication on a
                 mainnet spec.

A warn-only gate would reproduce the exact failure this is meant to stop, so
the gate hard-fails and pre-existing exposure goes in node_admin_baseline.txt
with its ticket. A row there is a debt record, not an exemption — and
test_check_node_admin_rpcs.sh asserts every shipped spec is clean or
baselined, so the ledger cannot drift.

bch.json's three rows are marked TEMPORARY: this branch is cut from main,
where they are still enabled. PR #152 disables them; delete the rows on merge.

## Item 1's measurement is in the docs, not just the ticket

Reproduced on BCHN 29.1.0 with two peered regtest nodes:

  parkblock     does nothing. The node logs "Unpark chain up to block … as it
                has accumulated enough PoW" and carries on. Self-healing on
                the most-work chain.
  finalizeblock strands the node permanently. The provider held at height 11
                while the network reached 20, rejecting honest headers with
                bad-header-finalization (code 259) and banning the peers
                offering them.

So the ticket's headline claim was wrong about which method is the weapon.
phase3.2 carries that as a worked example, with the lesson: check the
mechanism, because the obvious-looking candidate was the harmless one.

Also documented: a probe cannot decide this. Gateways whitelist methods and
geth deprecated personal_*, so probes look clean while a self-hosted node
serves them. The free-tier rule's converse — a -32601 is not evidence of
safety either.

## Mechanics

- spec_guards.yml gets a job scoped to the PR's changed root specs, matching
  how disabled-count already resolves them. Catalogue-wide assertion would
  make every spec PR answer for files it never touched.
- bash-3.2-clean (grep -Fxq against a temp file, no `declare -A`), like
  check_disabled_count.sh, because this is a guard reviewers run by hand.
  Verified under both /bin/bash 3.2.57 and bash 5.3.15.
- Emits add_on last in the jq row: tab is IFS whitespace, so an empty field in
  the middle silently shifts everything after it.

Skill-only changeset, so spec_pipeline.yml does not trigger (paths: "*.json").

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(create-spec): make the node-admin baseline self-cleaning

A baseline row whose method is no longer exposed is not harmless. It sits
there describing a fixed problem, and if the fix is ever reverted it silently
re-permits the method — the guard sees a known row and says nothing. The
ledger has to clean itself or it becomes the hiding place it was written not
to be.

So a stale row now FAILS, and the fix is to delete it.

The exception is a row tagged `TEMPORARY`, which exists precisely because it
is expected to go stale: a fix already in flight on another branch. Those are
reported loudly under "STALE, TEMPORARY (the in-flight fix landed — delete
these rows now)" but do not fail, because the merge that makes them stale is
usually someone else's and should not turn their CI red.

bch.json's three rows are tagged accordingly. The token is machine-readable
rather than a comment, which is the point — the previous version stated the
same intent in prose with nothing to enforce it, so the rows would have
outlived PR #152 silently.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* chore(create-spec): drop bch.json's TEMPORARY baseline rows — #152 landed

The three rows existed because this branch was cut from main while
parkblock / finalizeblock / unparkblock were still enabled there. #152
disabled them, so the rows now describe an exposure that no longer exists.

The mechanism worked as designed on its first real test. After merging main
the guard reported:

  === STALE, TEMPORARY (the in-flight fix landed — delete these rows now) ===
  bch.json BCH jsonrpc finalizeblock
  bch.json BCH jsonrpc parkblock
  bch.json BCH jsonrpc unparkblock

Reported, not failed — which is the whole point of the TEMPORARY tag: #152's
merge did not turn CI red on this branch or anyone else's, but the rows could
not quietly outlive their reason either. An untagged row going stale still
fails, because a stale row would silently re-permit the method if the fix were
reverted.

Suite still green, including the catalogue sweep over all shipped specs.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
@AnnaR-prog
AnnaR-prog requested a review from avitenzer October 6, 2026 08:25

@avitenzer avitenzer left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tested this at 550776ff. Both self-test suites pass (5/5 each), it still merges cleanly into main (21 commits behind), and the rule it encodes still matches the router: timeout_ms replaces the CU term, hanging_api adds 2× block time, with a 1 s floor. I also ran both checkers over all 155 specs on origin/main, plus qtum.json (#187) and dcr.json (#186).

1. Update mode will hit false FAILs, and Phase 10a calls them fixer regressions (blocking)

check_hanging_api.sh checks the whole candidate file. The scope note says the 152 pre-existing rows across ~30 established specs are "a separate cleanup, not this gate's job". The full-catalog run matches that count exactly: 28 files, 152 FAIL rows.

The problem is that in update mode (SKILL.md:91) the candidate is one of those established specs, and Phase 6's static gates still run there. Three open update PRs are already affected. On origin/main, before any change:

Spec Open update PR FAIL rows
zcash.json #150 1
solana.json #164 1
starknet.json #167 6

Each of these fails the gate on rows the updater never touched. SKILL.md Phase 10a then states "Any FAIL row here is a fixer-introduced regression, not an original defect. Feed it back as a CRITICAL item", so the fixer gets sent to edit methods outside the PR's scope. That is the drift that update mode exists to prevent (#80).

Possible fixes: fail only on rows that are new relative to the same file on origin/main (report the rest as INFO), or skip checks 2 and 3 in update mode for apis the updater did not add or change.

2. Stale line citations (non-blocking)

These have moved on smart-router main:

  • protocol/chainlib/common.go:575 is now :600. GetRelayTimeout now also bounds a caller override with BoundCallerRelayTimeout, and the CU / timeout_ms / hanging arithmetic moved into routerRelayTimeout. The rule is unchanged, but the quoted body no longer matches.
  • consumer_websocket_manager.go:298 is now :525.
  • SKILL.md:263, cited in the FAIL message and the header: this PR's own SKILL.md edit puts the rule at :331.

implied < 1000 hard-codes the default floor. That floor is now MinimumTimePerRelayDelay, configurable via --min-relay-timeout, so a deployment that raises it also raises the real budget. This doesn't break the check, but a comment saying it assumes the default would help.

3. Real catalog findings from check_stateful.sh (FYI, outside this PR)

monad.json and optimism.json declare eth_sendTransaction with stateful: 0. Those two are the only check_stateful failures in the catalog, and they look like genuine misclassifications worth a ticket.

qtum.json and dcr.json pass both checks.

AnnaR-prog and others added 5 commits October 7, 2026 16:04
… (MAG-3389)

Three rules for category.hanging_api, all derived from the one function
that reads it -- protocol/chainlib/common.go:575:

  func GetRelayTimeout(chainMessage, averageBlockTime) time.Duration {
      if chainMessage.TimeoutOverride() != 0 { return chainMessage.TimeoutOverride() }
      extraRelayTimeout := 0
      if IsHangingApi(chainMessage) { extraRelayTimeout = averageBlockTime * 2 }
      relayTimeAddition := common.GetTimePerCu(GetComputeUnits(chainMessage))
      if chainMessage.GetApi().TimeoutMs > 0 {
          relayTimeAddition = time.Millisecond * time.Duration(chainMessage.GetApi().TimeoutMs)
      }
      return extraRelayTimeout + relayTimeAddition
  }

1. hanging_api must not appear on a SUBSCRIBE-tagged API.
   consumer_websocket_manager.go:298 branches on the SUBSCRIBE function
   tag straight into StartSubscription. All four GetRelayTimeout call
   sites are on the unary path, and neither subscription manager
   references it -- so hanging_api, compute_units and timeout_ms are all
   dead inputs on a subscription. Its lifetime is its socket's; the WS
   manager's only timeouts are two hardcoded 10s constants for
   unsubscribe teardown. 3 of 288 SUBSCRIBE-registered APIs in the
   catalogue set the flag; PR #136 clears 2 of them.

   SUBSCRIBE names are collected inheritance-aware, walking the imports
   graph the way check_directive_presence.sh does, because an L2 that
   imports ETH1 carries an empty parse_directives array of its own.

2. hanging_api: true requires an explicit timeout_ms. This automates the
   rule at SKILL.md:263 and spec-builder.md:59, which both flagged it as
   having no validator coverage.

3. timeout_ms must be at least max(1s, CU*100ms). It REPLACES the CU
   term rather than adding to it, so a smaller value shortens the relay
   budget relative to setting nothing -- the opposite of the intent.
   Caught live on Acala: CU 1000 implies a 100 000ms base, so a
   reflexive timeout_ms: 30000 would have cut the budget from 124s to
   54s. Zero specs in the catalogue currently violate this, so it is a
   pure regression guard.

Rule 1 short-circuits: a subscription is never judged on its timeout,
since neither field is read.

Wired into gate 9 (method-schema) rather than added as a tenth gate --
both are per-method structural field checks, and gate 7 (pruning)
already sets the precedent of one gate running two scripts.

Scope: candidate file only, which is how the pipeline uses it. 152 APIs
across ~30 established specs (ethereum, cosmossdk, tendermint, solana,
kusama) predate rule 2 and would fail if it ran repo-wide. Separate
cleanup, tracked in MAG-3389.

Also corrects the test-suite table, stale since check_disabled_count.sh
(PR #133) and check_internal_paths.sh landed without being listed: 12 ->
15 rows.

test_check_hanging_api.sh covers all three rules, rule-1 precedence, the
1s CU floor at 999/1000, inherited SUBSCRIBE directives, and non-hanging
APIs being out of scope. All 15 suites pass under bash 5.3.
…AG-3389)

category.stateful is a routing instruction, not a label. It maps to
CONSISTENCY_SELECT_ALL_PROVIDERS and the router acts on it in four places
(rpcsmartrouter_server.go:3336, :3585, :3835, :4735): the call is fanned
out to every provider and excluded from cross-validation
(cross_validation_policy.go:422) and the recovery probe
(recovery_probe.go:112-133). Both directions fail silently --
stateful: 0 on a write sends the broadcast to a single provider, and
stateful: 1 on a read fans every call out to all of them and drops
cross-validation.

Unlike hanging_api this cannot be settled from the router source: "does
this method change chain state" is chain knowledge. Nor by vote --
author_submitAndWatchExtrinsic splits 10 specs at 0 against 11 at 1,
with kusama.json and polkadot.json on opposite sides, so a majority rule
produces no answer exactly where one is needed.

Three layers of decreasing confidence:

  1. Curated write list -> FAIL. Substrate submit/submitAndWatch,
     eth_sendRawTransaction/sendTransaction/sendRawTransactionSync,
     BroadcastTx, broadcast_tx_*, Solana sendTransaction, the three
     starknet add*Transaction. Curated rather than voted, so it fires
     even when the whole catalogue agrees and is wrong.
  2. Curated read list -> FAIL. Simulation and codec helpers.
     eth_fillTransaction is on it by name: SPEC_GUIDE.md and phase3.2
     both call it the classic trap.
  3. Cross-spec consensus -> INFO, never FAIL. Fires when >=90% of the
     OTHER specs declaring a name disagree and at least 4 declare it.
     Advisory because 24 of 5,539 distinct method names genuinely
     disagree across the catalogue.

REST entries are keyed "TYPE /path" where the verb disambiguates: GET
/cosmos/tx/v1beta1/txs is a query, POST to the same path is BroadcastTx.
The verb resolves a name collision, it does not classify -- treating
POST as "is a write" is the bug that put four read endpoints on the
write path in cosmossdk.json.

Calibration over the 140-spec catalogue: 29 FAIL rows in 16 specs and 11
INFO rows. Every FAIL is a real defect -- the 10 specs carrying the watch
pair at 0, babylon/kava/sei on decode/amino, monad/optimism on
eth_sendTransaction, and cosmossdk's four REST reads. The INFO rows are
lit.json's seven node-admin methods at 0 against 16 specs at 1, and
bittensor/etc on eth_submitWork, eth_submitHashrate and
transaction_v1_stop.

Two defects found while building it, both caught by the tests:

  - the consensus dedupe key dropped the filename, so `sort -u`
    collapsed the whole catalogue to one row per distinct value and the
    threshold could never be reached;
  - the candidate was counted in its own consensus denominator, which
    contradicted the "N/M other specs" wording and pushed 4-of-4
    agreement below the 90% bar.

Wired into gate 9 alongside check_method_schema.sh and
check_hanging_api.sh; gate 9's INFO rows are added to the advisory
routing table, which now lists four gates.

test_check_stateful.sh covers both curated lists, the fillTransaction
trap, REST verb disambiguation in both directions, consensus being
advisory rather than fatal, the sibling threshold, and both of the
defects above. All 16 suites pass under bash 5.3.
…-run

Three corrections found by re-checking claims I had asserted rather than
verified.

1. stateful is NOT read on the subscription path either.

   check_stateful.sh told the reader a stateful: 0 write "goes to a
   single provider instead of all of them". For a SUBSCRIBE-tagged
   method that is wrong, and wrong the same way the hanging_api claim
   was: GetStateful is unreachable from the subscription path. It
   appears in rpcsmartrouter_server.go, recovery_probe.go,
   unified_relay_state_machine.go and common.go -- and in neither
   subscription manager, exactly like GetRelayTimeout.

   But it does not follow that the flag is inert there, which is where
   this differs from hanging_api. ApiHasStatefulCategory
   (base_chain_parser.go:549, reached via rpcsmartrouter_server.go:456)
   looks the method up by NAME, independent of transport, and the
   cross-validation policy guard uses it to reject an enabled per-method
   CV policy on a write. So the write list still FAILs on subscription
   methods rather than short-circuiting them the way
   check_hanging_api.sh rule 1 does -- hanging_api has no equivalent
   name-keyed reader, stateful does.

   The message and header now say precisely which effects are unary-only
   and which are not.

2. Phase 10a: re-run the offline gates after the fixer.

   Gate 9 runs in Phase 6, before Phase 10. The only post-fix validation
   was `jq .`, so any defect the FIXER introduces shipped unguarded --
   and the fixer is the step most likely to introduce these, because it
   applies field-level instructions without surrounding context.

   check_hanging_api.sh rule 3 is the clearest case: told only "a
   hanging API needs a timeout_ms", the natural response is a flat
   30000, which on a CU-1000 method shortens the relay budget instead of
   lengthening it. That is the mistake made by hand while writing
   MAG-3389, and the guard against it could not fire where it mattered.

   Phase 10a re-runs check_method_schema, check_hanging_api,
   check_stateful and check_unused_fields, all offline and sub-second.
   It also finally wires check_disabled_count.sh (PR #133), which was
   left unwired for precisely this timing reason.

3. "Every FAIL is a genuine defect" was overstated.

   monad and optimism carry eth_sendTransaction at enabled: false, so
   those 2 of the 29 rows are latent, not live. Corrected to 27 live and
   2 latent -- the same distinction that applies to a disabled method
   anywhere.

Also: a rest collection with no collection_data.type could not match a
verb-keyed curated entry and skipped both lists in silence. Every rest
collection in the catalogue currently has one; if that ever stops being
true it now emits an INFO row instead of passing quietly.

All 16 suites pass.
…in update mode

Review on 550776f: on an established spec, Phase 6 and Phase 10a failed rows
the run never touched (zcash 1, solana 1, starknet 6 hanging_api rows; monad
and optimism 1 stateful row each), and Phase 10a called them fixer regressions.

- check_hanging_api.sh and check_stateful.sh take --base <file>. A FAIL row the
  base fails with identical text moves to INFO as pre-existing; new or changed
  rows still FAIL. No base, no change. An unreadable base exits 2.
- The base is judged in the candidate's directory by its own imports, so an
  inherited SUBSCRIBE directive resolves the same way for both.
- Phase 6 passes origin/main's copy in update, add-testnet and add-collection
  mode; Phase 10a passes the pre-fix snapshot.
- Citations: GetRelayTimeout is common.go:600, and the spec inputs are read
  by routerRelayTimeout (:613). The SUBSCRIBE branch is
  consumer_websocket_manager.go:525. The SKILL.md rule is cited by section,
  and the 1s floor is noted as MinimumTimePerRelayDelay's default.

All five specs above pass with themselves as base and still fail without one.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…sh 5.2

An unquoted $* in a heredoc joins with spaces, not IFS, before bash 5.3, so
the two-api base-mode fixtures were invalid JSON on ubuntu-24.04 and
test_check_stateful.sh failed in CI while passing on bash 5.3.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@avitenzer
avitenzer force-pushed the feat/hanging-api-gate branch from 944a8b0 to f819e75 Compare October 7, 2026 13:10
…collection

Two holes in the base mode added in 7fb2e84, both found in review:

- Phase 10a passed the pre-fix snapshot as --base. That snapshot is taken after
  the Phase-6 fixer, so on a new chain a Phase-6 FAIL the fixer left in place
  (the Acala timeout_ms case this gate exists for) was demoted to INFO as
  pre-existing and passed. Phase 10a now uses the same base as Phase 6: the
  file on origin/main, or none for a new chain.
- Base rows were matched as a set of "index/interface/name" lines, so a method
  already failing in one collection excused the same defect in an add-on
  collection the run added. Rows now name the collection (interface, REST verb,
  internal_path, add_on) and are demoted by count.

Also corrects what the docs and messages claim about the router: hanging_api
has three readers, not one; rule 3 shortens the relay window, not the attempt
budget; the subscription path that ignores hanging_api is the WebSocket one.
Refreshes check_stateful.sh's smart-router citations and the calibration
figures, names MAG-3401 as the cleanup ticket, routes the method-schema
ADVISORY block in Phase 6, and drops the circular --expect <N> from Phase 10a.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@avitenzer
avitenzer merged commit 84b2f47 into main Oct 7, 2026
3 of 6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants