Skip to content
Merged
Show file tree
Hide file tree
Changes from 15 commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
cc4a061
feat(rpcsmartrouter)!: serve a provider with no websocket url over ht…
AnnaR-prog Oct 29, 2026
2a6788f
docs: document the websocket service and missing-url on the alarm met…
AnnaR-prog Oct 29, 2026
5641d87
docs(wizard): a provider with no ws url serves without subscriptions …
avitenzer Nov 9, 2026
fde675d
fix(chainlib): retry a node url that could not be dialed by its dial …
AnnaR-prog Nov 10, 2026
c45939a
test(rpcsmartrouter): a websocket url that dies while it serves takes…
AnnaR-prog Nov 10, 2026
85ec45c
fix(chainlib): retry a failing check only at the node urls that faile…
AnnaR-prog Nov 10, 2026
270a179
fix(chainlib): a due check that no url failed in the kept pass runs a…
AnnaR-prog Nov 10, 2026
ad7d4f5
refactor(chainlib): one definition of a websocket url, for the verifi…
AnnaR-prog Nov 10, 2026
2ea36b3
fix(rpcsmartrouter): health says a provider serves no subscriptions w…
AnnaR-prog Nov 10, 2026
21768b7
refactor(chainlib): drop the websocket enforcement of newChainRouter,…
AnnaR-prog Nov 10, 2026
60d919b
fix(rpcsmartrouter): log a missing websocket url as a warning, not an…
AnnaR-prog Nov 10, 2026
0a27dd6
fix(rpcsmartrouter): a provider out of the pairing passes none of the…
AnnaR-prog Nov 12, 2026
948bdb0
fix(rpcsmartrouter): read the websocket part from the urls admission …
AnnaR-prog Nov 13, 2026
be4e030
fix(chainlib): retry a node url a pass left out for its hold-off by i…
avitenzer Nov 15, 2026
64fe257
docs: an HTTP-only provider is no longer excluded, in the example con…
avitenzer Nov 15, 2026
e3f556d
Merge branch 'main' into feat/MAG-3950-websocket-part
avitenzer Nov 15, 2026
de59d07
Merge branch 'main' into feat/MAG-3950-websocket-part
avitenzer Nov 15, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -455,7 +455,7 @@ For tests that only need the endpoint-health reset, **`POST /debug/reset-endpoin

`/debug/reset-pairing` and `/debug/reset-scores` remain available for targeted cleanup.

**`GET /debug/verification`** shows, per chain and configured provider, what it is failing (`scope`, `service`, `check`, `severity`, and `since`, the start `smartrouter_provider_verification_failed` publishes), whether it is in the pairing, and when each check it is failing is next retried (`retries`, one entry per check, earliest due first: `check`, `addon`, `extension`, or `wholePass` for a url that could not be dialed, with `attempts`, `pendingSince` and `nextRetryAt`), with the retry delays in force (MAG-3950). A test can shorten those with `--verification-retry-delay` and `--verification-retry-max-delay` instead of waiting 30s to 15m between retries.
**`GET /debug/verification`** shows, per chain and configured provider, what it is failing (`scope`, `service`, `check`, `severity`, and `since`, the start `smartrouter_provider_verification_failed` publishes), whether it is in the pairing, and when each check it is failing is next retried (`retries`, one entry per check, earliest due first: `check`, `addon`, `extension`, or `wholePass` for a provider whose pass gave no verdicts to read, with `attempts`, `pendingSince` and `nextRetryAt`; a url that could not be dialed, or that a pass did not ask because it is held off after a rate limit, has the entry `check: connect`, and its retry dials that url alone), with the retry delays in force (MAG-3950). A test can shorten those with `--verification-retry-delay` and `--verification-retry-max-delay` instead of waiting 30s to 15m between retries.

## Releases

Expand Down
6 changes: 3 additions & 3 deletions config/smartrouter_examples/smartrouter_bitcoin.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,12 +2,12 @@
# Mode: Direct connection to a public Bitcoin JSON-RPC endpoint (no Lava providers!)
#
# Upstream: PublicNode (bitcoin-rpc.publicnode.com), HTTP. Bitcoin Core's
# JSON-RPC has no websocket/subscription surface, so this example is HTTP-only —
# start the router with --skip-websocket-verification.
# JSON-RPC has no websocket/subscription surface, so this example is HTTP-only
# and has no subscriptions to miss.
#
# Run it:
# smartrouter config/smartrouter_examples/smartrouter_bitcoin.yml \
# --use-static-spec specs/ --skip-websocket-verification
# --use-static-spec specs/

# Prometheus metrics (scrape at http://<host>:7779/metrics). "disabled" turns it off.
metrics-listen-address: "0.0.0.0:7779"
Expand Down
4 changes: 2 additions & 2 deletions config/smartrouter_examples/smartrouter_eth.yml
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
# Smart Router Direct RPC Configuration — Ethereum (ETH1)
# Mode: Direct connections to public Ethereum JSON-RPC endpoints (no Lava providers!)
#
# 2 upstreams, each HTTP + WebSocket (the ETH1 spec requires ws for
# subscriptions; http-only providers are excluded at startup):
# 2 upstreams, each HTTP + WebSocket. The ETH1 spec subscribes: an http-only
# provider serves everything but subscriptions and publishes missing-url:
# 1. PublicNode (ethereum-rpc.publicnode.com, ws on the same host)
# 2. Tenderly public gateway (mainnet.gateway.tenderly.co)
# All verified live (eth_chainId == 0x1, eth_blockNumber + ws handshake OK).
Expand Down
4 changes: 2 additions & 2 deletions config/smartrouter_examples/smartrouter_eth_cached.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,8 @@
# docker compose -f docker/docker-compose.yml \
# -f docker/docker-compose.cache.yml up --build
#
# 2 upstreams, each HTTP + WebSocket (the ETH1 spec requires ws for
# subscriptions; http-only providers are excluded at startup):
# 2 upstreams, each HTTP + WebSocket. The ETH1 spec subscribes: an http-only
# provider serves everything but subscriptions and publishes missing-url:
# 1. PublicNode
# 2. Tenderly public gateway

Expand Down
4 changes: 2 additions & 2 deletions config/smartrouter_examples/smartrouter_eth_resp_cache.yml
Original file line number Diff line number Diff line change
Expand Up @@ -57,8 +57,8 @@ direct-rpc:

# Endpoint 2 — dRPC public endpoint (HTTP + WS). Tenderly's public WS
# gateway rejects the chain-tracker's subscription probes ("method not
# found"), and the ETH1 spec requires a websocket leg per provider — an
# HTTP-only provider is excluded at verification and fails overall-health.
# found"). Each provider here keeps a websocket leg: an HTTP-only one would
# serve no subscriptions and publish the websocket's missing-url (MAG-3950).
- name: "eth-drpc"
chain-id: "ETH1"
api-interface: "jsonrpc"
Expand Down
4 changes: 2 additions & 2 deletions config/smartrouter_examples/smartrouter_hyperliquid.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,11 +3,11 @@
#
# Upstream: the official Hyperliquid EVM RPC (rpc.hyperliquid.xyz/evm), HTTP.
# It's a standard EVM JSON-RPC (eth_chainId == 0x3e7 / 999) but serves no
# websocket leg, so start the router with --skip-websocket-verification.
# websocket leg; the HYPERLIQUID spec defines no subscriptions, so none is missed.
#
# Run it:
# smartrouter config/smartrouter_examples/smartrouter_hyperliquid.yml \
# --use-static-spec specs/ --skip-websocket-verification
# --use-static-spec specs/

# Prometheus metrics (scrape at http://<host>:7779/metrics). "disabled" turns it off.
metrics-listen-address: "0.0.0.0:7779"
Expand Down
13 changes: 8 additions & 5 deletions config/smartrouter_examples/smartrouter_multichain.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,10 +11,13 @@
# 3366 COSMOSHUB grpc cosmos-grpc.publicnode.com + cosmos-grpc.polkachu.com
# 3367 APT1 rest aptos-rest.publicnode.com + fullnode.mainnet.aptoslabs.com
#
# ETH1 requires a ws:// leg for subscriptions, so it pairs an https url with a
# wss one. SOLANA, BTC, HYPERLIQUID (jsonrpc) and APT1 (rest) are HTTP-only here,
# so start the router with --skip-websocket-verification (the docker-compose
# example does this).
# ETH1 serves subscriptions over a ws:// leg, so it pairs an https url with a
# wss one. SOLANA, BTC, HYPERLIQUID (jsonrpc) and APT1 (rest) are HTTP-only here
# and serve everything but subscriptions. A provider with no ws url on an
# interface that subscribes (SOLANA, Polkachu's tendermintrpc) publishes the
# websocket's missing-url (MAG-3950). No flag is needed for that; the
# docker-compose example passes --skip-websocket-verification, which only stops
# checks over ws(s) urls.
#
# Specs resolve from the bundled specs/ (pass --use-static-spec specs/): ETH1,
# SOLANA, BTC, HYPERLIQUID, APT1, and COSMOSHUB (imports COSMOSSDK50 + COSMOSWASM,
Expand Down Expand Up @@ -89,7 +92,7 @@ direct-rpc:
- url: "https://mainnet.gateway.tenderly.co"
- url: "wss://mainnet.gateway.tenderly.co"

# ---- Solana (SOLANA) — HTTP-only; run with --skip-websocket-verification ----
# ---- Solana (SOLANA) — HTTP-only: no subscriptions, publishes missing-url ----
- name: "sol-publicnode"
group-label: "publicnode"
chain-id: "SOLANA"
Expand Down
10 changes: 6 additions & 4 deletions config/smartrouter_examples/smartrouter_multichain_cached.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,10 +13,12 @@
# 3361 SOLANA jsonrpc solana-rpc.publicnode.com + api.mainnet-beta.solana.com
# 3362 COSMOSHUB rest cosmos-rest.publicnode.com + cosmos-api.polkachu.com
#
# ETH1 requires websocket support, so its providers pair an http url with a wss
# url; http-only providers are excluded at startup. SOLANA (jsonrpc) and
# COSMOSHUB (rest) are HTTP-only here, so run with --skip-websocket-verification
# (the docker-compose example does this).
# ETH1 serves subscriptions over websocket, so its providers pair an http url
# with a wss url. SOLANA (jsonrpc) and COSMOSHUB (rest) are HTTP-only here and
# serve everything but subscriptions; SOLANA's spec subscribes, so its providers
# publish the websocket's missing-url (MAG-3950). No flag is needed for that;
# the docker-compose example passes --skip-websocket-verification, which only
# stops checks over ws(s) urls.
#
# Specs required (resolve via --use-static-spec specs/):
# ETH1 -> specs/ethereum.json (already in repo)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,9 +17,12 @@
# 3366 COSMOSHUB grpc cosmos-grpc.publicnode.com + cosmos-grpc.polkachu.com
# 3367 APT1 rest aptos-rest.publicnode.com + fullnode.mainnet.aptoslabs.com
#
# ETH1 requires a ws:// leg, so it pairs an https url with a wss one. SOLANA,
# BTC, HYPERLIQUID (jsonrpc) and APT1 (rest) are HTTP-only here, so start the
# router with --skip-websocket-verification (the docker-compose example does this).
# ETH1 serves subscriptions over a ws:// leg, so it pairs an https url with a wss
# one. SOLANA, BTC, HYPERLIQUID (jsonrpc) and APT1 (rest) are HTTP-only here and
# serve everything but subscriptions. A provider with no ws url on an interface
# that subscribes (SOLANA, Polkachu's tendermintrpc) publishes the websocket's
# missing-url (MAG-3950). No flag is needed for that; the docker-compose example
# passes --skip-websocket-verification, which only stops checks over ws(s) urls.
#
# Specs resolve from the bundled specs/ (pass --use-static-spec specs/): ETH1,
# SOLANA, BTC, HYPERLIQUID, APT1, and COSMOSHUB (imports COSMOSSDK50 + COSMOSWASM,
Expand Down
5 changes: 3 additions & 2 deletions config/smartrouter_examples/smartrouter_solana.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,11 +5,12 @@
# 1. PublicNode (solana-rpc.publicnode.com)
# 2. Solana Labs public RPC (api.mainnet-beta.solana.com)
# Solana serves subscriptions over a separate ws endpoint; this example is
# HTTP-only, so start the router with --skip-websocket-verification.
# HTTP-only, so the router serves it without subscriptions and publishes the
# websocket's missing-url (MAG-3950).
#
# Run it:
# smartrouter config/smartrouter_examples/smartrouter_solana.yml \
# --use-static-spec specs/ --skip-websocket-verification
# --use-static-spec specs/

# Prometheus metrics (scrape at http://<host>:7779/metrics). "disabled" turns it off.
metrics-listen-address: "0.0.0.0:7779"
Expand Down
10 changes: 6 additions & 4 deletions docker/docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -62,10 +62,12 @@ services:
- "${SR_LOG_LEVEL:-info}"
- "--log-format"
- "${SR_LOG_FORMAT:-json}"
# Some bundled examples have HTTP-only listeners (e.g. SOLANA in the solana
# and multichain configs). Skip the startup websocket reachability check so
# those boot cleanly; ws subscriptions still work at runtime where a wss url
# is configured (the EVM examples pair https + wss).
# Dial ws(s) node urls without running the spec's checks over them, as the
# helm chart does. HTTP-only listeners (e.g. SOLANA in the solana and
# multichain configs) boot with or without it since MAG-3950: they serve
# without subscriptions and publish the websocket's missing-url. ws
# subscriptions work where a wss url is configured (the EVM examples pair
# https + wss).
- "--skip-websocket-verification"
ports:
- "3360:3360" # ETH1 jsonrpc
Expand Down
13 changes: 8 additions & 5 deletions docs/LOCAL-COMPOSE.md
Original file line number Diff line number Diff line change
Expand Up @@ -172,8 +172,10 @@ the cache overlay (see [Enabling the cache](#enabling-the-cache)).
### `smartrouter_solana.yml` / `smartrouter_bitcoin.yml` / `smartrouter_hyperliquid.yml` / `smartrouter_aptos.yml`

Single-chain JSON-RPC (Solana, Bitcoin, Hyperliquid) or REST (Aptos) examples,
each on port `3360` and HTTP-only — run with `--skip-websocket-verification`
(the compose command already passes it). Upstreams are PublicNode and each
each on port `3360` and HTTP-only. The compose command passes
`--skip-websocket-verification`, which only matters for ws(s) urls: since MAG-3950 an
HTTP-only provider serves without subscriptions whatever the flag says, and on a chain
that subscribes (Solana) it publishes `service=websocket`, `check=missing-url`. Upstreams are PublicNode and each
chain's official/community endpoint; no Lava endpoints.

### `smartrouter_cosmos.yml` — Cosmos Hub (REST + gRPC + Tendermint RPC)
Expand Down Expand Up @@ -205,9 +207,10 @@ import closure), all sourced from
[`Magma-Devs/lava-specs`](https://github.com/Magma-Devs/lava-specs) and already
bundled in `specs/`.

> Only ETH1 requires websocket support (its providers pair an `http` url with a
> `wss` one). Solana, Bitcoin, Hyperliquid (jsonrpc) and Aptos (rest) are
> HTTP-only, so the compose command runs with `--skip-websocket-verification`.
> Only the ETH1 example pairs an `http` url with a `wss` one, so only it serves
> subscriptions. Solana, Bitcoin, Hyperliquid (jsonrpc) and Aptos (rest) are
> HTTP-only; Solana's spec subscribes, so its providers publish the websocket's
> `missing-url` alarm.

### `smartrouter_multichain_cached.yml` — multi-chain with cache

Expand Down
19 changes: 11 additions & 8 deletions docs/METRICS.md
Original file line number Diff line number Diff line change
Expand Up @@ -656,8 +656,8 @@ self-heals in seconds. Alert windows should assume the 3m ceiling.

| Metric | Type | Labels | Description |
| --- | --- | --- | --- |
| `smartrouter_provider_verification_failed` | Gauge | `spec`, `apiInterface`, `provider`, `tier`, `scope`, `service`, `check`, `severity` | While a configured provider is failing a verification: the unix time in seconds the failure started. The series exists only while the failure lasts, and a provider that moves to another check in the same `scope` and `service` keeps its start (a `scope=none` warning never borrows another's). `tier` is `static` or `backup`. |
| `smartrouter_provider_verification_check_failures_total` | Counter | same as above | One per failed check run: at boot, each verification retry, and each 15m epoch pass (a grace run before a demotion included). A rate-limited run is inconclusive and not counted, on any path. `increase(...[$__interval]) > 0` draws one mark per failed run. |
| `smartrouter_provider_verification_failed` | Gauge | `spec`, `apiInterface`, `provider`, `tier`, `scope`, `service`, `check`, `severity` | While a configured provider is failing a verification: the unix time in seconds the failure started. The series exists only while the failure lasts, and a provider that moves to another check in the same `scope` and `service` keeps its start (a `scope=none` warning never borrows another's). While a provider is out of the pairing it has only its `scope=provider` series; a failure it had before it went out comes back with its own start when it is readmitted and still fails, and no "passes again" line is logged for it at the demote. `tier` is `static` or `backup`. |
| `smartrouter_provider_verification_check_failures_total` | Counter | same as above | One per failed check run: at boot, each verification retry, and each 15m epoch pass (a grace run before a demotion included). A rate-limited run is inconclusive and not counted, on any path, and a missing websocket url (`missing-url`) ran nothing, so it is not counted either. `increase(...[$__interval]) > 0` draws one mark per failed run. |

The serving tier only drops when a whole tier is gone, so a single provider that fails
verification while others serve leaves it at `2` (or `1`). Such a provider also has no
Expand All @@ -669,9 +669,9 @@ too.

| Label | Values |
| --- | --- |
| `scope` | `provider`: the provider is out of the pairing. `service`: the provider still serves, but lost one add-on or extension: no url of it serves that service any more. `url`: the provider still serves, but one of its node urls was refused (MAG-4231 verifies every url on its own): whole when `service` is empty, else for that one service, which another url of the provider still serves. `none`: a check the spec gives severity `Warning` is failing; nothing was switched off. |
| `service` | The refused add-on or extension (`archive`, `debug`, ...), or for `scope=none` the one the failing check covers; empty for `scope=provider`, for a url refused whole and for a basic check. |
| `check` | The spec verification that failed (`chain-id`, `pruning`, ...), or one of: `latest-block` (the head probe before any verification), `unsupported-addon` (a node url's addons name something the spec does not define: `scope=service` with that name as `service`, while the url serves the rest; only a `standalone-addons` url that names no add-on the spec defines is refused whole), `connect` (no connection to the node), `config` (the configured urls cannot work as given: no http(s) url, or no websocket url on a chain that subscribes), `not-run` (the provider's verification deadline passed before this check ran), `rate-limited` (a provider out at boot whose only answer so far is a 429; gauge only, never counted), `unknown`. |
| `scope` | `provider`: the provider is out of the pairing. `service`: the provider still serves, but lost one add-on or extension: no url of it serves that service any more. `url`: the provider still serves, but one of its node urls was refused (MAG-4231 verifies every url on its own): whole when `service` is empty, else for that one service, which another url of the provider still serves. The websocket counts as a service: a ws(s) url carries only subscriptions, so a refused one is published with `service=websocket`, as `scope=service` when the provider has no ws(s) url left and `scope=url` while another still serves (MAG-3950). `none`: a check the spec gives severity `Warning` is failing; nothing was switched off. |
| `service` | The refused add-on or extension (`archive`, `debug`, ...), `websocket` for the provider's subscriptions (a refused ws(s) url, or none configured), or for `scope=none` the one the failing check covers; empty for `scope=provider`, for an http(s) url refused whole and for a basic check. |
| `check` | The spec verification that failed (`chain-id`, `pruning`, ...), or one of: `latest-block` (the head probe before any verification), `unsupported-addon` (a node url's addons name something the spec does not define: `scope=service` with that name as `service`, while the url serves the rest; only a `standalone-addons` url that names no add-on the spec defines is refused whole), `connect` (no connection to the node), `config` (the configured urls cannot work as given: no http(s) url), `missing-url` (`service=websocket`, `scope=service`, `PartialFail`: the provider has no ws(s) url on a chain that subscribes, so it serves everything over http but subscriptions; nothing ran and no retry is scheduled, since only a config change fixes it; gauge only, never counted. It is published with `--skip-websocket-verification` too, which the helm chart sets on every router), `not-run` (the provider's verification deadline passed before this check ran), `rate-limited` (a provider out at boot whose only answer so far is a 429; gauge only, never counted), `unknown`. |
| `severity` | As MAG-3950 names it: `Fail` when the failure takes a url or the provider out, `PartialFail` when it switches off only `service`, `Warning` when it switches off nothing (`scope=none`). A spec value that sets no severity takes the default for where its check sits: `Fail` for a basic check (the base collection's own, no add-on or extension), `PartialFail` for an add-on or extension check. An explicit `Fail` on an add-on check makes the add-on mandatory: its failure takes the url out. A rate-limited `Warning` check, or one the deadline never ran, is not counted: one that failed the last pass keeps its series, and a new one gets none. |

Each check a provider is failing that costs something (MAG-3950) is retried on its own
Expand All @@ -680,10 +680,13 @@ doubling for every pass in a row that still does, up to `--verification-retry-ma
(15m), plus up to 20% jitter, and never before the provider's rate-limit hold-off ends.
That covers a provider out of the pairing (failed at boot, or demoted), a refused service
or url, and a url on its first strike. A retry re-runs only the provider's checks that are
due, on each node url that has them, and keeps every other verdict from the pass before:
due, at each node url that failed them, and keeps every other verdict from the pass before:
with `debug` and `archive` both failing, each has its own schedule, and the checks that
pass (`chain-id` and the rest) are re-run by the epoch alone. A url that could not be
dialed is retried whole. Checks that failed in the same pass stay due together until
pass (`chain-id` and the rest) are re-run by the epoch alone, as is a due check at a url
that passed it. A url that could not be dialed is retried by its dial alone (`connect`), and
so is a url a pass did not ask because it was held off after a rate limit (MAG-4733): once
the hold-off ends, that url alone is dialed and verified, and its siblings keep their verdicts.
Checks that failed in the same pass stay due together until
their attempts part. While its chain is dark, every provider out of the pairing is
verified whole from 2s instead, doubling to 3m, and those passes do not push a check's
own retry back. The epoch pass counts as an attempt for every check it finds failing, so
Expand Down
Loading