Skip to content

feat!: serve a provider with no websocket url over http, and publish the websocket as its own part (MAG-3950) - #602

Merged
avitenzer merged 17 commits into
mainfrom
feat/MAG-3950-websocket-part
Nov 15, 2026
Merged

avitenzer merged 17 commits into
mainfrom
feat/MAG-3950-websocket-part

Conversation

@AnnaR-prog

@AnnaR-prog AnnaR-prog commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Was stacked on #582 (retries) and #580 (severities). Both squash-merged on 2026-11-09. This branch is rebased on main (11878b1f, #700, 2026-11-15) and holds this PR's own thirteen commits plus avitenzer's be4e0304 (item 9) and 64fe2574 (item 10), all signed. The rebase onto #700 resolved three conflicts in chain_fetcher.go, all around the skip parameter #700 added to ValidateCollectionsAgain; nothing else changed (range-diff).

Problem

MAG-3950 asks that one broken part cost only that part, and that the alarm say which part: "an add-on or the websocket". MAG-4505 comment 30715 point 5 decided the case with no socket: "the upstream serves over https without subscriptions, and the alarm says the wss url is missing."

On main:

  • A provider with no ws(s) url, on a chain that subscribes, is refused whole by ProviderShapeError with check config. Only --skip-websocket-verification admits it.
    • The helm chart passes that flag to every router (routers_deployment.yaml, since chart chore(deps): bump goreleaser/goreleaser-action from 6 to 7 #69). So in production such a provider already serves over http, and nothing says it serves no subscriptions: no series, no log line.
    • Without the flag, as in a local run, a QA setup or smartrouter health, it is refused.
    • That is most chains: 217 of the 302 specs on lava-specs main subscribe on at least one interface, directly or through an import. Among them are ETH1 and the EVM specs that import it, Solana, and the tendermint interface of the Cosmos chains.
  • A refused ws(s) url publishes scope=url and its check, with no service. A refused http url publishes the same series, so the alarm can't tell a broken socket from a broken http url.

Change

1. A provider with no ws(s) url serves over http, and raises the alarm.

  • ProviderShapeError keeps only its http rule. The new chainlib.MissingWebsocket reports a provider that has no ws(s) url while its spec subscribes over a socket, meaning an enabled collection that isn't gRPC.
  • Served: the provider is verified over http like any other, and serves everything except subscriptions. Subscriptions go to the providers that have a socket; collectWSEndpoints never listed this one.
  • The alarm:
  • The flag doesn't hide it. --skip-websocket-verification stops the router running checks over a socket; it doesn't give a provider a socket it lacks. The chart sets the flag on every router, so honouring it would hide every missing url.
  • Not retried. No retry can give a provider a url. The epoch pass keeps the series current.

2. A refused ws(s) url is published as the websocket service. A ws(s) url carries no relay, only subscriptions (MAG-3477), so losing one costs the provider its websocket:

  • scope=service, service=websocket when the provider has no ws(s) url left;
  • scope=url, service=websocket while another of its ws(s) urls still serves;
  • either way it is PartialFail, because only subscriptions are switched off. Before, it was Fail with no service.
  • An http url refused whole is unchanged: scope=url, no service, Fail.

3. smartrouter health admits such a provider. Each admitted row's router.reason says no ws(s) node url: serves no subscriptions. The row's ok is unchanged. infra #748 parses router.verdict, but none of its scripts acts on it.

4. Wording:

  • the router's --skip-websocket-verification help now says what the flag does: dial ws(s) urls without running any check over them;
  • config now means only "no http(s) url".

5. Docs: docs/METRICS.md covers the websocket service and missing-url. docs/LOCAL-COMPOSE.md no longer says the flag is what lets an HTTP-only provider boot.

6. A url that could not be dialed is retried by its dial alone (MAG-4505 finding 14, b0f5a460).

  • Before: on feat!: retry each failing verification on its own backoff, log its transitions, serve GET /debug/verification (MAG-3950) #582, such a url put the whole pass on the retry schedule. Each retry re-ran every check of every url of the provider, so a dead socket made the router ask the http url its checks again on the fast schedule, although they had passed. MAG-4505 C4 rules that out for a check that passes.
  • Now: ProviderAdmission.Retries gives the url its own entry, its dial (check connect). ValidateCollectionsAgain keeps every other url's verdicts, as it does for a check. So the retry dials only the urls that could not be dialed, and runs their checks once they answer.
  • GET /debug/verification lists the entry as check connect, the same name as the failing row; the README says so. The whole pass stays for a provider whose pass gave no verdicts to read.

7. A failing check is retried only at the urls that failed it (MAG-4505 finding 16, 72274356 and 3095a699).

  • Before: a check is due for the provider, and feat!: retry each failing verification on its own backoff, log its transitions, serve GET /debug/verification (MAG-3950) #582's retry ran it at every url that declares it, the urls that had passed it too. While one address of a provider failed the archive check, each retry also asked the archive checks at its other address.
  • Now: verify runs a due check where the kept verdict failed. A url that passed it keeps that verdict when another url failed it in the kept pass, since the retry is for that url. A url with nothing to run is not dialed.
  • A due check that no url failed in the kept pass runs at every url, as before. That is the retry after a rate limit: the 429 answers a later pass, which decides nothing and is not recorded, so the kept pass holds the check passed everywhere. 72274356 alone kept those verdicts and asked nothing; the cluster test of a 429 on the chain-id check (MAG-4505 C5) caught it, and 3095a699 fixes it.
  • The schedule is unchanged: one entry per provider and check, which leaves once the urls that failed it pass. docs/METRICS.md says so.

8. Review follow-ups (avitenzer's review at 924ad1ac).

  • One definition of a websocket url (ff2d8d86): chainlib exports IsWebSocketURL, and refusalsOf and websocketLeft call it instead of their own copy, so the verifier and the publisher of the websocket part cannot drift.
  • health and a refused socket (b78eb847): a provider whose ws urls the router refuses, a dead socket for one, now gets no ws(s) node url admitted: serves no subscriptions on its admitted rows, read as websocketLeft reads the admission. It matches the metric, which publishes it as the websocket service. A provider with no ws url keeps no ws(s) node url: serves no subscriptions.
  • missing-url logs as a warning (39e02e5a): a provider with no ws url on a chain that subscribes logs verification check failing once at warning level, not error, since the config sets it and no retry can fix it. The alert by severity leaves missing-url out. Both answer the open questions of the review.
  • A provider out of the pairing passes nothing it was failing (60154d20): it is listed by the check that took it out alone, so everything else it failed left the list at the demote and logged verification check passes again, though nothing had checked it. A provider with no ws url logged its missing url passing at each demote, and failing again at each readmit. Those failures are now held while it is out: no line at the demote, and at the readmit each goes on with its start, or passes again then.
  • On top of fix(chainlib,rpcsmartrouter,lavasession): verify each generated per-path url on its own path, and let a failed path cost only that path (MAG-4214) #659 and refactor: rename the Lava-named Go internals (MAG-2972) #695 (a6e7a2b9): refactor: rename the Lava-named Go internals (MAG-2972) #695 (MAG-2972) renamed lavasession to routersession and utils.LavaFormat* to utils.Format*, and this PR's code follows it. fix(chainlib,rpcsmartrouter,lavasession): verify each generated per-path url on its own path, and let a failed path cost only that path (MAG-4214) #659 (MAG-4214) verifies the urls the router dials, generated per-path urls included, so a refused url's position is into that list. refusalsOf now reads the refused url from the admission (URLAt), websocketLeft finds each dialed url's verdict by value as servedElsewhere does, and health passes its dialed list. Before this, on a chain with per-path urls such as STRK, a dead socket was published as a refused http url, and health did not say the provider serves no subscriptions. The per-url retry keeps its positions: boot, epoch and retry passes all verify the same dialed list.
  • The old websocket enforcement is gone (5e29ac49): newChainRouter refused an endpoint with no ws url on a spec that subscribes, unless IgnoreWsEnforcementForTestCommands was set. Its only production caller, smartrouter test, set it, so it ran in chainlib tests alone. The check and the flag are removed.

9. A url a pass left out for its hold-off is retried by its dial alone (be4e0304, on the rebase onto #700). #700 (MAG-4733) leaves a url held off after a rate limit out of the pass, with no verdicts. Retries read that as the whole pass, so the retry that followed asked the siblings that had passed everything their checks again, and while the url stayed held each whole-pass retry asked them again. It now gets the connect entry a url that could not be dialed gets: its retry verifies that url alone, and every other url keeps its verdicts. While the url is still held, the retry asks nothing and backs off; the first retry after the hold-off ends verifies it. TestVerificationRetries_AHeldOffUrlIsRetriedAloneOnceItsHoldOffEnds and a case in TestRetries_AreTheChecksThatFailed pin it; both fail with the one-line change reverted. README and docs/METRICS.md say so.

10. The example configs and the gauge's help (64fe2574). Nine example configs, docker/docker-compose.yml and two setup scripts still said that an http-only provider on a chain that subscribes is excluded at startup, or that --skip-websocket-verification is what lets one boot. They now say it serves without subscriptions and publishes missing-url, and the single-chain HTTP-only examples drop the flag from their run command, where it changes nothing. The gauge's help text lists missing-url and says what service=websocket means. Comments and help text only.

What operators will see change

  • Every http-only provider on a chain that subscribes gets a series once this ships, in every tenant, on 217 of 302 specs, since the flag no longer hides it: service=websocket, check=missing-url, PartialFail. Nothing pages on it, because #978's gauge alert reads scope=provider only. #978's panel draws failed runs from the counter, so missing-url draws no mark there; the gauge holds it. The alert by severity (MAG-3950 item 5) leaves missing-url out, as decided on this PR: it is set by config and lasts as long as the config does.
  • A broken wss url changes series, from scope=url, no service, Fail to service=websocket, PartialFail. feat(metrics): publish which configured providers failed verification, and which check (MAG-4594) #572 shipped in v1.6.5, and infra #978, now merged, reads its series. On #978's panel the broken socket moves from the "one url refused" lane to "lost websocket" ("one url lost websocket" while another socket serves). The severity alerts built on #978 (branch feat/MAG-3950-alert-by-severity) page on Fail and post on PartialFail, so a broken wss url posts instead of paging, and missing-url posts nothing.
  • Without the flag, a provider with no ws(s) url is no longer refused.
  • A dead socket costs one dial per retry, not a whole pass. The retry delays are unchanged (30s doubling to 15m by default), but the http checks that passed are not asked again. /debug/verification lists that retry as check: connect, where feat!: retry each failing verification on its own backoff, log its transitions, serve GET /debug/verification (MAG-3950) #582 listed wholePass.
  • A url held off after a rate limit is retried alone once its hold-off ends, listed as check: connect on /debug/verification, where fix(rpcsmartrouter,lavasession): only let a blocked provider back in when it has a usable url (MAG-4737) #700 on its own scheduled wholePass.
  • A check that one url fails is asked again at that url alone, not at the provider's other urls that passed it.
  • smartrouter health says a provider serves no subscriptions when the router refuses each of its ws urls, as it already did when it has none.
  • A part failure keeps its start through a demote. While a provider is out of the pairing it has only its scope=provider series, as before. When it is readmitted, a failure it had before comes back with its own start, not the readmit's, and no passes again line is logged for it at the demote. The PartialFail alert of the alert-by-severity branch then posts at the readmit when that start is older than 30 minutes; sr-verify-failed reads scope=provider only and is unchanged.

Tests

  • New tests, each checked to fail with its fix reverted:
    • No ws(s) url: the provider is admitted, and its http url still runs every base check: a wrong chain id takes it out on chain-id. Run with the flag on and off.
    • The websocket's series: missing-url; the only socket refused gives scope=service; with another socket left it gives scope=url; published as PartialFail; BTC subscribes nowhere, so it has no websocket to miss.
    • From boot to the epoch: the provider is in the pairing, logs one error line, shows on /debug/verification, and is not put on the retry schedule. The epoch pass logs nothing new.
    • health: the provider is admitted with the reason, with the flag on and off. A socket clears the reason, and a wrong chain is still excluded.
    • The counter leaves out missing-url and still counts the refused archive beside it.
    • A url that could not be dialed (TestVerificationRetries_AUrlThatCannotBeDialedRetriesOnlyItsDial): its dial is on the schedule, the retry loop dials it again with no request to the http url, and the url is admitted once its socket answers. It fails with feat!: retry each failing verification on its own backoff, log its transitions, serve GET /debug/verification (MAG-3950) #582's Retries.
    • A socket that dies while it serves (TestVerificationRetries_ASocketThatDiesTakesItsSecondStrikeFromTheRetryOfItsDial): the epoch's failed dial is graced and puts the dial on the schedule, and the retry of the dial is the second strike that refuses the url, with no request to the http url. It fails with feat!: retry each failing verification on its own backoff, log its transitions, serve GET /debug/verification (MAG-3950) #582's Retries too.
    • A check that one url fails and another passes (TestVerificationRetries_ACheckIsRetriedOnlyAtTheUrlThatFailedIt): the retry asks the url that failed it, the url that passed is asked nothing, and the check leaves the schedule once the first url passes. It fails with feat!: retry each failing verification on its own backoff, log its transitions, serve GET /debug/verification (MAG-3950) #582's rule in verify.
    • A due check that every url passed in the kept pass (TestVerificationRetries_ADueCheckThatEveryUrlPassedRunsAtEveryUrl): the retry asks it at every url. It fails with 72274356's rule.
    • A dead socket on a chain with per-path urls (TestRefusalsOf_ADeadSocketOnAChainWithPerPathUrlsIsTheWebsocket, TestHealth_EveryWebsocketURLRefusedOnAChainWithPerPathUrls): on STRK a bare https url and a bare ws url are nine dialed urls. The dead socket is the websocket service, scope service, with no http url refused, and every admitted http row says the provider serves no subscriptions. Both fail on the rebased code before the fix, and reverting any one of the three fixed reads turns one of them red.
    • A provider out of the pairing (TestTrackFailures_AProviderOutOfThePairingPassesNothingItFailed): a missing ws url logs no passes again at the demote and no second line at the readmit, and keeps its start; an archive check it passes at the readmit passes again then, counted from its own start. Dropping either half of the fix turns it red.
    • health with every ws url refused (TestHealth_EveryWebsocketURLRefusedServesNoSubscriptions): with the flag on and off, the admitted http row names the reason and the dead ws url is refused; a socket that answers leaves the reason out. It fails without the new case.
  • Updated tests:
    • the shape test drops the no-socket refusal;
    • three tests with http-only providers on ETH1 and Solana now also see missing-url. Each gets a socket, or its expectation names it.
    • TestRetries_AreTheChecksThatFailed expects the url it could not dial as connect, not the whole pass.
    • TestSkipWebsocketVerificationIsPerParser keeps pinning that a parser's flag does not leak into another parser or into the process default; with the enforcement gone, both routers accept the http-only endpoint.
  • At 64fe2574, rebased on main 11878b1f: go build ./..., go vet ./... and the metrics tests pass, and CI is green. The full chainlib/..., metrics, common, holdoff and rpcsmartrouter suites, and -race -count=2 on the retry, websocket-part, health, transition-log and hold-off tests, passed on the same code at 7779bdb3; 64fe2574 changes only comments and the gauge's help string, which no test reads.
  • At a6e7a2b9, rebased on main 36b0cd82: go build ./... (the wizard too), every chainlib package, rpcsmartrouter and metrics pass, and -race on the websocket, health, transition-log, retry and per-path tests passes three runs in a row. Each of the thirteen commits builds and passes go vet on its own.
  • Before the rebase onto 36b0cd82, on k3d (smart-router-automation mag-4505, row A3, on eth-failover-real-sim-router, with the chart's flag and without it): with provider 1's ws port closed, its retries dialed the socket alone, and its http checks had the start pass alone, before and after the port opened. Subscriptions went to the other providers while it was closed, and reached it again after.
  • Before the rebase onto 36b0cd82, go test -race on chainlib, metrics and common: passes. rpcsmartrouter passes except TestHealthCommand_RejectsUnbootableCrossValidation, which fails the same way on main under -race once other tests have run (the health command writes the global log level), and passes alone.
  • golangci-lint 2.15 (--new-from-rev feat!: retry each failing verification on its own backoff, log its transitions, serve GET /debug/verification (MAG-3950) #582's head) on chainlib, metrics and rpcsmartrouter: 0 issues.
  • A real binary, run locally, measured before item 6 (ETH1 from lava-specs main, a fake http and ws upstream, --epoch-duration 1m --verification-retry-delay 2s). Item 6 changes what those retries run, not what they decide: the two tests above cover the second strike and the re-admission. The three providers: https-only with one http url, with-ws with http and ws urls, and wrong-chain with one http url answering chain id 0x89.
Run Seen
#582's head, no flag https-only and wrong-chain are both out on config: "subscriptions are applicable for this chain, but websocket is not provided". The refusal comes before any check, so it also hides that wrong-chain is on the wrong chain.
This head, no flag https-only is in the pairing, with service=websocket, check=missing-url, PartialFail, no retry scheduled and one error line. wrong-chain is out on chain-id, verified over http. Only with-ws's socket is listed for subscriptions, and 10 of 10 relays were answered.
This head, flag on (as the chart runs it) Boot is the same as with no flag. with-ws's socket was killed right after boot, at 12:59:45. The epoch at 13:00:01 gave it the grace's first strike, and the retry at 13:00:03 refused it. It was published as scope=service, service=websocket, check=connect, PartialFail, with one error line, while the provider stayed in the pairing and relays kept being answered. Once the socket was back, the next retry cleared it at 13:00:07 (passes again, failedFor=4s).

Release

Unreleased, as #580 and #582 are. #463 and #572 shipped in v1.6.5.

Jira ticket: MAG-3950

🤖 Generated with Claude Code

@AnnaR-prog
AnnaR-prog requested a review from a team as a code owner October 29, 2026 11:09
avitenzer added a commit that referenced this pull request Nov 9, 2026
…(MAG-3950)

The emitter's comment still said a spec that subscribes needs a ws url
unless the router runs with --skip-websocket-verification. Since #602 a
provider with no ws(s) url serves everything else over http and
publishes the websocket's missing-url, whatever the flag says.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

Took 2 minutes

@avitenzer avitenzer left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at bbf9478e (this PR's own diff against #582's branch). I found no correctness bug. I pushed one doc commit (924ad1ac); everything else is notes.

1. Stale wizard comment (fixed in 924ad1ac). tools/wizard/internal/emit/emit.go still said a spec that subscribes needs a ws url unless the router runs with --skip-websocket-verification. After this PR that is false, so the comment now says the provider serves everything else and publishes missing-url. Comment only, no code change.

Checked and fine:

  • What "subscribes" means per interface. I ran MissingWebsocket with one http url against the bundled specs. COSMOSHUB tendermintrpc → true, rest → false, grpc → false (MAG-2643 streaming). ETH1, BASE (through its import) and SOLANA → true, BTC → false. taggedApis is filled only from enabled collections, so a disabled subscribing collection can't raise the alarm.
  • Without the flag, an http-only provider is still really verified. getExtensionsForVerification assigns the base checks of a subscribing collection to the websocket extension when the flag is off. I traced that the http url still runs them, and TestValidateProviderCollections_NoWebsocketURLServesOverHTTP pins it: eth_chainId is sent, and a wrong chain is refused on chain-id either way.
  • A subscription with no ws url on the whole chain still gets the NoOp manager's "no endpoint" error. wsConfigured reads the configured lists, so this PR adds no hang or panic path. With the chart's flag on, production already ran this way.
  • A provider with only ws urls is still refused by ProviderShapeError ("HTTP/HTTPS is mandatory"), and dials nothing.
  • Indexes: RefusedURLs(), URLRefused and websocketLeft index provider.NodeUrls, as servedElsewhere and admission_registry already do.
  • The counter: every countFailedCheck call that publishes a refusal goes through countRefusals, so the missing-url skip there covers boot, epoch and retry. Retries: degraded reads admission.Any(), which a missing url never sets, so nothing schedules it.
  • missing-url and a refused ws url can't both appear for one provider (one needs no ws url, the other needs one), so previousCheck never mixes them.

Open decision (product, not code): once this ships, every http-only provider on a chain that subscribes (217 of 302 lava-specs specs, in every tenant, since the chart sets the flag) gets a PartialFail series plus one error-level verification check failing line per boot, and again after each demote → readmit. That is what 30715 point 5 asks for. Two things should be settled before release: whether feat/MAG-3950-alert-by-severity posts on missing-url, and whether a condition set by config should log at error level.

Nits (not changed): see the two inline comments. Also, chain_router.go:335 still has the old ws enforcement. Its only production caller (smartrouter test) sets IgnoreWsEnforcementForTestCommands, so it is dead outside chainlib tests and could be removed in a follow-up.

Verification on 924ad1ac (local):

  • go test -count=1 passes for chainlib/..., metrics/..., common/... and rpcsmartrouter/... (rpcsmartrouter took 290 s).
  • -race -count=2 passes on the new and updated tests.
  • gofmt is clean, go vet is clean, and golangci-lint --new-from-rev=origin/feat/MAG-3950-verification-retries reports 0 issues; tools/wizard builds.

}

// isWebsocketURL reports whether a node url is ws(s), as verification reads it.
func isWebsocketURL(raw string) bool {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: this is chainlib.isWebSocketUrl (chain_fetcher.go:131) again, with the same body over IsUrlWebSocket. Exporting that one and calling it here keeps a single definition of "ws(s) url, as verification reads it", so the publisher and the verifier can't drift. Up to you.

@AnnaR-prog AnnaR-prog Nov 10, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in ff2d8d8: chainlib exports it as IsWebSocketURL, and refusalsOf and websocketLeft call it, so there is one definition.

Comment thread protocol/rpcsmartrouter/health_cmd.go Outdated
row.SpecValid = true
applyValidation(&row, validations[i])
row.Router = routerVerdict(validations[i], excluded)
if missingWebsocket {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A question, not a bug. The reason is added only when no ws(s) url is configured. A provider with an http url and a ws url that is refused (dead socket) also serves no subscriptions, but its admitted http row has an empty reason. Only the ws row, refused, shows it. The metric treats both cases the same (service=websocket, scope=service). If health should match, the condition here would be "no ws(s) url admitted", using the same check as websocketLeft. It's fine to leave as is if the refused row is enough for #748.

@AnnaR-prog AnnaR-prog Nov 10, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed in b78eb84: when the router refuses every ws url of a provider, its admitted rows now say no ws(s) node url admitted: serves no subscriptions, read the same way as websocketLeft, so health matches the metric. With no ws url, the reason stays as it was. Pinned by TestHealth_EveryWebsocketURLRefusedServesNoSubscriptions.

Base automatically changed from feat/MAG-3950-verification-retries to main November 9, 2026 12:25
AnnaR-prog pushed a commit that referenced this pull request Nov 10, 2026
…(MAG-3950)

The emitter's comment still said a spec that subscribes needs a ws url
unless the router runs with --skip-websocket-verification. Since #602 a
provider with no ws(s) url serves everything else over http and
publishes the websocket's missing-url, whatever the flag says.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

Took 2 minutes
@AnnaR-prog
AnnaR-prog force-pushed the feat/MAG-3950-websocket-part branch from bd544e4 to d6b6cd5 Compare November 10, 2026 14:29
AnnaR-prog added a commit that referenced this pull request Nov 10, 2026
…er and the publisher (MAG-3950)

refusalsOf and websocketLeft read a node url as a websocket through their
own copy of chainlib's isWebSocketUrl. chainlib exports it as
IsWebSocketURL, and rpcsmartrouter calls it, so the verifier and the
publisher of the websocket part read a url the same way and cannot drift
(review of #602).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
AnnaR-prog added a commit that referenced this pull request Nov 10, 2026
…hen every ws url is refused (MAG-3950)

health added "serves no subscriptions" to a provider's admitted rows only
when it had no ws(s) url. A provider whose ws urls the router refuses, a
dead socket for one, serves no subscriptions either, and the metric
publishes both cases as the websocket service. Its admitted rows now say
"no ws(s) node url admitted: serves no subscriptions", read as
websocketLeft reads the admission. The case with no ws(s) url keeps its
reason (review of #602).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
AnnaR-prog added a commit that referenced this pull request Nov 10, 2026
… dead outside its tests (MAG-3950)

newChainRouter refused an endpoint with no ws url on a spec that
subscribes, unless IgnoreWsEnforcementForTestCommands was set. Its only
production caller, smartrouter test, sets it, so the check ran in chainlib
tests alone, and it said the opposite of what the router now does: a
provider with no ws url serves over http. The check and the flag are gone.
TestSkipWebsocketVerificationIsPerParser still pins that a parser's flag
does not leak into another parser or into the process default (review of
#602).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
AnnaR-prog added a commit that referenced this pull request Nov 10, 2026
… error (MAG-3950)

A provider with no ws(s) url on a chain that subscribes logged
"verification check failing" at error level, once per boot and again
after each demote and readmit. The config sets it and no retry can fix
it, so the line is a warning now, and error lines are left for checks
that fail (decided on #602). The alarm metric and GET /debug/verification
are unchanged, and the alert by severity leaves missing-url out.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@AnnaR-prog

AnnaR-prog commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor Author

Also from your review: the old websocket enforcement in newChainRouter and IgnoreWsEnforcementForTestCommands are removed in 5e29ac4.

On the two open questions: missing-url does not alert (the alert-by-severity rule leaves it out), and it now logs as a warning, not an error (39e02e5).

AnnaR-prog pushed a commit that referenced this pull request Nov 11, 2026
…(MAG-3950)

The emitter's comment still said a spec that subscribes needs a ws url
unless the router runs with --skip-websocket-verification. Since #602 a
provider with no ws(s) url serves everything else over http and
publishes the websocket's missing-url, whatever the flag says.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

Took 2 minutes
AnnaR-prog added a commit that referenced this pull request Nov 11, 2026
…er and the publisher (MAG-3950)

refusalsOf and websocketLeft read a node url as a websocket through their
own copy of chainlib's isWebSocketUrl. chainlib exports it as
IsWebSocketURL, and rpcsmartrouter calls it, so the verifier and the
publisher of the websocket part read a url the same way and cannot drift
(review of #602).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@AnnaR-prog
AnnaR-prog force-pushed the feat/MAG-3950-websocket-part branch from 8c4f953 to deef149 Compare November 11, 2026 05:55
AnnaR-prog added a commit that referenced this pull request Nov 11, 2026
…hen every ws url is refused (MAG-3950)

health added "serves no subscriptions" to a provider's admitted rows only
when it had no ws(s) url. A provider whose ws urls the router refuses, a
dead socket for one, serves no subscriptions either, and the metric
publishes both cases as the websocket service. Its admitted rows now say
"no ws(s) node url admitted: serves no subscriptions", read as
websocketLeft reads the admission. The case with no ws(s) url keeps its
reason (review of #602).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
AnnaR-prog added a commit that referenced this pull request Nov 11, 2026
… dead outside its tests (MAG-3950)

newChainRouter refused an endpoint with no ws url on a spec that
subscribes, unless IgnoreWsEnforcementForTestCommands was set. Its only
production caller, smartrouter test, sets it, so the check ran in chainlib
tests alone, and it said the opposite of what the router now does: a
provider with no ws url serves over http. The check and the flag are gone.
TestSkipWebsocketVerificationIsPerParser still pins that a parser's flag
does not leak into another parser or into the process default (review of
#602).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
AnnaR-prog added a commit that referenced this pull request Nov 11, 2026
… error (MAG-3950)

A provider with no ws(s) url on a chain that subscribes logged
"verification check failing" at error level, once per boot and again
after each demote and readmit. The config sets it and no retry can fix
it, so the line is a warning now, and error lines are left for checks
that fail (decided on #602). The alarm metric and GET /debug/verification
are unchanged, and the alert by severity leaves missing-url out.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
AnnaR-prog added a commit that referenced this pull request Nov 12, 2026
… checks it was failing (MAG-3950)

A provider out of the pairing is listed by the check that took it out
alone, so everything else it failed left the list at the demote and
logged "verification check passes again", though nothing had checked
it. A provider with no ws url logged its missing url passing at each
demote, and failing again at each readmit (review of #602).

trackFailures now holds those failures while the provider is out: no
line at the demote, and at the readmit each goes on with its start, or
passes again then, counted from its start. The gauge publishes the
start the logs name, so a part failure that outlasts a demote comes back
with its own start, not the readmit's: the PartialFail alert of the
alert-by-severity branch posts at the readmit when that start is older
than 30 minutes. sr-verify-failed reads scope=provider alone and is
unchanged.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
AnnaR-prog pushed a commit that referenced this pull request Nov 13, 2026
…(MAG-3950)

The emitter's comment still said a spec that subscribes needs a ws url
unless the router runs with --skip-websocket-verification. Since #602 a
provider with no ws(s) url serves everything else over http and
publishes the websocket's missing-url, whatever the flag says.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

Took 2 minutes
AnnaR-prog added a commit that referenced this pull request Nov 13, 2026
…er and the publisher (MAG-3950)

refusalsOf and websocketLeft read a node url as a websocket through their
own copy of chainlib's isWebSocketUrl. chainlib exports it as
IsWebSocketURL, and rpcsmartrouter calls it, so the verifier and the
publisher of the websocket part read a url the same way and cannot drift
(review of #602).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
AnnaR-prog added a commit that referenced this pull request Nov 13, 2026
…hen every ws url is refused (MAG-3950)

health added "serves no subscriptions" to a provider's admitted rows only
when it had no ws(s) url. A provider whose ws urls the router refuses, a
dead socket for one, serves no subscriptions either, and the metric
publishes both cases as the websocket service. Its admitted rows now say
"no ws(s) node url admitted: serves no subscriptions", read as
websocketLeft reads the admission. The case with no ws(s) url keeps its
reason (review of #602).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@AnnaR-prog
AnnaR-prog force-pushed the feat/MAG-3950-websocket-part branch from 23f1ae2 to a6e7a2b Compare November 13, 2026 03:42
AnnaR-prog added a commit that referenced this pull request Nov 13, 2026
… error (MAG-3950)

A provider with no ws(s) url on a chain that subscribes logged
"verification check failing" at error level, once per boot and again
after each demote and readmit. The config sets it and no retry can fix
it, so the line is a warning now, and error lines are left for checks
that fail (decided on #602). The alarm metric and GET /debug/verification
are unchanged, and the alert by severity leaves missing-url out.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
AnnaR-prog added a commit that referenced this pull request Nov 13, 2026
… checks it was failing (MAG-3950)

A provider out of the pairing is listed by the check that took it out
alone, so everything else it failed left the list at the demote and
logged "verification check passes again", though nothing had checked
it. A provider with no ws url logged its missing url passing at each
demote, and failing again at each readmit (review of #602).

trackFailures now holds those failures while the provider is out: no
line at the demote, and at the readmit each goes on with its start, or
passes again then, counted from its start. The gauge publishes the
start the logs name, so a part failure that outlasts a demote comes back
with its own start, not the readmit's: the PartialFail alert of the
alert-by-severity branch posts at the readmit when that start is older
than 30 minutes. sr-verify-failed reads scope=provider alone and is
unchanged.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@AnnaR-prog
AnnaR-prog requested a review from avitenzer November 13, 2026 08:37
AnnaR-prog and others added 2 commits November 15, 2026 12:55
…tp, and publish the websocket as its own part (MAG-3950)

A provider with no ws(s) node url on a chain that subscribes was refused
whole unless the router ran with --skip-websocket-verification. It now
serves over http without subscriptions, and the alarm metric publishes
scope=service, service=websocket, check=missing-url (PartialFail), as
MAG-4505 comment 30715 point 5 decided. The alarm does not read the
flag: the helm chart sets it on every router, so honouring it would hide
every missing url.

A ws(s) url carries no relay, only subscriptions (MAG-3477), so a
refused one is published as the websocket service too: scope=service
when the provider has no ws(s) url left, scope=url while another still
serves. Before, it read as scope=url with no service, the same series
a refused http url gives, so the alarm could not say "the websocket".

health admits such a provider, and each admitted row says it serves no
subscriptions.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ric (MAG-3950)

docs/METRICS.md: a refused ws(s) url is published as service=websocket,
and a provider with no ws(s) url on a chain that subscribes as its
missing-url; config is now only a provider with no http(s) url.

docs/LOCAL-COMPOSE.md: --skip-websocket-verification is no longer what
lets an HTTP-only provider boot.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
avitenzer and others added 11 commits November 15, 2026 12:55
…(MAG-3950)

The emitter's comment still said a spec that subscribes needs a ws url
unless the router runs with --skip-websocket-verification. Since #602 a
provider with no ws(s) url serves everything else over http and
publishes the websocket's missing-url, whatever the flag says.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

Took 2 minutes
…alone, not the whole pass (MAG-3950)

A url that could not be dialed put the whole pass on the retry schedule.
Each retry then re-ran every check of every url of the provider, so a
dead websocket url made the router ask the http url its checks again on
the fast schedule, although they had passed. MAG-4505 C4 rules that out
for a failing check, and the cluster test of a closed ws port (MAG-4505
A3) showed it: the debug check over http was asked again on each retry.

Retries now gives such a url its own entry, its dial (check connect).
ValidateCollectionsAgain keeps every other url's verdicts, as it does for
a check, so the retry dials only the urls that could not be dialed, and
runs their checks once they answer. /debug/verification lists the entry
as check connect, the same name as the failing row. The whole pass stays
for a provider whose pass gave no verdicts to read.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… its second strike from the retry of its dial (MAG-3950)

The tests of 041364a close the socket before boot. A socket that dies while
its provider serves goes through the url grace first: the epoch pass that
cannot dial it gives it a strike and keeps it, and puts its dial on the
schedule. The new test checks that the retry of that dial is the second
strike and refuses the url, and that it asks the http url nothing. The
recorded admission keeps the epoch's pass, so the retry finds the failed dial
and dials the url again.

refusingSocket returns its switch, so a test can turn the socket off after
boot. The README sentence on GET /debug/verification lists attempts,
pendingSince and nextRetryAt again; 041364a split that list.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…d it (MAG-3950)

A check is due for a provider, and its retry ran it at every node url that
declares it, the urls that had passed it too. While one address of a
provider failed the archive check, each retry also asked the archive checks
at its other address, which had passed them (MAG-4505 finding 16).

verify now runs a due check where the kept verdict failed, and keeps the
verdict of a url that passed it. A check with no verdict to keep, or an
inconclusive one, still runs. A url with nothing to run is not dialed, so
the url that passed is asked nothing. The schedule is unchanged: the check
leaves it once the urls that failed it pass.

docs/METRICS.md says so, and that a url that could not be dialed is retried
by its dial alone, which the commit of that change left out.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…t every url (MAG-3950)

d6b6cd5 kept the passed verdict of every url for a due check. That holds
when the kept pass is the one that failed the check at another url. A check
can also be due with the kept pass holding it passed everywhere: a rate limit
answers a later pass, which decides nothing and is not recorded, and its
retry keeps the older pass. Each url kept its verdict, so the retry asked
nothing, and the check stayed unasked until the epoch pass. The cluster test
of a 429 on the chain-id check (MAG-4505 C5) showed it: the retry due at the
end of the hold-off sent no eth_chainId, and the next one came 60 s after
the 429, at the epoch.

ValidateCollectionsAgain now gives verify the due checks that the kept pass
holds failing, or undecided, at some url. A url that passed one of those
keeps its verdict; a due check that no url failed runs at every url, as
before d6b6cd5.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…er and the publisher (MAG-3950)

refusalsOf and websocketLeft read a node url as a websocket through their
own copy of chainlib's isWebSocketUrl. chainlib exports it as
IsWebSocketURL, and rpcsmartrouter calls it, so the verifier and the
publisher of the websocket part read a url the same way and cannot drift
(review of #602).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…hen every ws url is refused (MAG-3950)

health added "serves no subscriptions" to a provider's admitted rows only
when it had no ws(s) url. A provider whose ws urls the router refuses, a
dead socket for one, serves no subscriptions either, and the metric
publishes both cases as the websocket service. Its admitted rows now say
"no ws(s) node url admitted: serves no subscriptions", read as
websocketLeft reads the admission. The case with no ws(s) url keeps its
reason (review of #602).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… dead outside its tests (MAG-3950)

newChainRouter refused an endpoint with no ws url on a spec that
subscribes, unless IgnoreWsEnforcementForTestCommands was set. Its only
production caller, smartrouter test, sets it, so the check ran in chainlib
tests alone, and it said the opposite of what the router now does: a
provider with no ws url serves over http. The check and the flag are gone.
TestSkipWebsocketVerificationIsPerParser still pins that a parser's flag
does not leak into another parser or into the process default (review of

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… error (MAG-3950)

A provider with no ws(s) url on a chain that subscribes logged
"verification check failing" at error level, once per boot and again
after each demote and readmit. The config sets it and no retry can fix
it, so the line is a warning now, and error lines are left for checks
that fail (decided on #602). The alarm metric and GET /debug/verification
are unchanged, and the alert by severity leaves missing-url out.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… checks it was failing (MAG-3950)

A provider out of the pairing is listed by the check that took it out
alone, so everything else it failed left the list at the demote and
logged "verification check passes again", though nothing had checked
it. A provider with no ws url logged its missing url passing at each
demote, and failing again at each readmit (review of #602).

trackFailures now holds those failures while the provider is out: no
line at the demote, and at the readmit each goes on with its start, or
passes again then, counted from its start. The gauge publishes the
start the logs name, so a part failure that outlasts a demote comes back
with its own start, not the readmit's: the PartialFail alert of the
alert-by-severity branch posts at the readmit when that start is older
than 30 minutes. sr-verify-failed reads scope=provider alone and is
unchanged.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…verified (MAG-3950)

Since #659 (MAG-4214) admission verifies the urls the router dials,
generated per-path urls included, and a refused url's position is into
that list. refusalsOf, websocketLeft and health read the position in
the configured list: on a chain with per-path urls, such as STRK and
AVAX, a dead socket was published as a refused http url, scope=url with
no service, and health did not say that the provider serves no
subscriptions.

refusalsOf now reads the refused url from the admission (URLAt),
websocketLeft finds each dialed url's verdict by value, as
servedElsewhere does, and health passes the dialed list its validations
are by position in. The per-url retry keeps its positions: boot, epoch
and retry passes all verify the dialed list through
validateProviderCollections. missing-url still reads the configured
urls, since expanding a url never changes its transport.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@avitenzer
avitenzer force-pushed the feat/MAG-3950-websocket-part branch from a6e7a2b to 7779bdb Compare November 15, 2026 11:19
avitenzer and others added 2 commits November 15, 2026 13:54
…ts dial alone (MAG-3950)

#700 (MAG-4733) leaves a url held off after a rate limit out of the pass:
its outcome is ErrNotAskedHeldOff, with no verdicts. Retries read any such
outcome as the whole pass. The retry that followed re-verified every url
of the provider, so a sibling that had passed everything was asked its
checks again: the cost b0f5a46 removes for a url that could not be
dialed. While the url stayed held, each of those whole-pass retries asked
the siblings again.

A held-off url now gets the entry a url that could not be dialed gets,
check connect. Its retry verifies that url alone, and every other url
keeps its verdicts. While the url is still held, the retry asks nothing
and backs off; the first retry after its hold-off ends verifies it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…figs and the gauge's help (MAG-3950)

The example configs, the compose file and two setup scripts still said
that an http-only provider on a chain that subscribes is excluded at
startup, or that --skip-websocket-verification is what lets one boot.
With this PR such a provider serves without subscriptions and publishes
the websocket's missing-url, with the flag or without it. Each comment now
says so, and the single-chain HTTP-only examples drop the flag from their
run command, where it changes nothing. The RESP cache lane's header says
that overall-health no longer catches a missing WS leg.

The gauge's help text lists missing-url among the checks, says what
service=websocket means, and that missing-url is not retried.

Comments and help text only.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@avitenzer
avitenzer force-pushed the feat/MAG-3950-websocket-part branch from 7779bdb to 64fe257 Compare November 15, 2026 11:56

@avitenzer avitenzer left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at 64fe2574: this PR's thirteen commits rebased onto main 11878b1f (#700), plus two commits of mine. I found no correctness bug in the PR's own diff. One of my commits fixes how it composes with #700; the other fixes docs this PR makes stale.

Rebase. Three conflicts, all in chain_fetcher.go and all textual. #700 added a skip parameter to ValidateCollectionsAgain and a skip field to verifyOptions, where commits 4, 6 and 7 rewrote the doc comment and added failing. Both sides are kept. git range-diff against the old head shows 9 of the 13 commits byte-identical, and the other 4 differ only in those hunks. The rebase re-signed every commit.

Pushed:

  • be4e0304: a url held off after a rate limit no longer puts the whole pass on the schedule. It gets the connect entry a url that could not be dialed gets, so its retry verifies it alone and asks nothing while it is still held. Two tests pin it; both fail with the one-line change reverted. Details inline.
  • 64fe2574: docs. Nine example configs, docker/docker-compose.yml and two setup scripts still said an http-only provider on a chain that subscribes is excluded at startup, or needs --skip-websocket-verification to boot. The gauge's help text listed every check but missing-url. Comments and help text only. I checked which interfaces publish missing-url with MissingWebsocket against the bundled specs: SOLANA, ETH1 and COSMOSHUB tendermintrpc do; BTC, HYPERLIQUID, APT1 and COSMOSHUB rest and grpc do not.

Checked and fine where #700 and this PR meet:

  • URLAt and RefusesURL read urls, which refusals() copies, so refusalsOf and websocketLeft read the right url after Rerun(), KeepingThrottledFrom and WithoutURLRefusals.
  • An http url the new per-url rule keeps whole has every verdict kept, so URLAsked is false and urlPassResults records neither an answer nor a 429 for it. #700's TestProbeReverification_ARetryLeavesAURLItDidNotAskAsItWas still pins that. A url with no checks is the exception; see the first follow-up.
  • failing and skip compose: a held url is notAsked, its siblings keep their verdicts, and KeepingThrottledFrom keeps the held url's refusals.
  • health builds its rows from the dialed list, and its validations are by position in that list, so everyWebsocketRefused reads the right rows on STRK.

Design notes, not changed: three inline comments. Two are about a retry keeping verdicts from the last recorded pass while a rate-limited pass of a paired provider is never recorded. One is about a dead url being re-dialed by every retry, which predates this PR. Also, config and missing-url are both set by config and no retry fixes either, but only missing-url is kept off the schedule and the counter. A provider with no http(s) url is retried whole and counts check=config on every retry (noteFailure). That predates this PR, which now states the rule for one of the two.

For follow-ups, not this PR:

  • #700's URLAsked answers true for a url with no checks that a retry kept whole without dialing. That is every ws url when the router runs with --skip-websocket-verification, as the chart sets. urlPassResults then records an answer, and RecordAnswer clears the hold-off the WS subscription path put on that socket. Every retry that keeps a socket whole does it, and this PR adds more such retries. Worth a ticket.
  • docs#91 updates cli.md and examples/index.md, but reference/chains/{bitcoin,aptos,hyperliquid,solana}.md still say to run each example with --skip-websocket-verification because it has no websocket leg. 64fe2574 drops the flag from this repo's single-chain examples.
  • scripts/pre_setups/cache/init_smartrouter_eth_resp_cache.sh relied on overall-health to catch a missing WS leg, which this PR ends. Asserting that no missing-url series exists would restore the check.

Verification on 64fe2574: go build ./... (wizard included), go vet ./... and gofmt are clean. go test -count=1 passes on chainlib/..., metrics, common, holdoff and rpcsmartrouter, run on 7779bdb3, whose code differs only in the gauge's help string. -race -count=2 passes on the retry, websocket-part, health, transition-log and hold-off tests. CI: unit-tests, unit-tests-race, govulncheck and lint are green.

if FailedCheck(outcome.err) != CheckUnsupportedAddon {
switch {
case FailedCheck(outcome.err) == CheckUnsupportedAddon:
case FailedCheck(outcome.err) == CheckConnect, errors.Is(outcome.err, ErrNotAskedHeldOff):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

be4e0304: ErrNotAskedHeldOff (#700) is a url outcome with an error and no verdicts, so before this line it fell to default and scheduled the whole pass. The retry that followed asked every sibling that had passed its checks again, and while the url stayed held, each whole-pass retry asked them again. The held url now gets the connect entry. ValidateCollectionsAgain with only=[connect] verifies it alone, because kept is nil for an outcome with an error, and keeps every sibling whole. While the url is still held, skip leaves it notAsked and the entry backs off. Pinned by TestVerificationRetries_AHeldOffUrlIsRetriedAloneOnceItsHoldOffEnds and the new case in TestRetries_AreTheChecksThatFailed.

_, failedElsewhere := opts.failing[checkIDOf(verification)]
verdict, known := keptVerdicts[verdictKey(verification)]
if (due || !known || inconclusive(verdict)) && !skipVerification(url, verification.Name) {
if (!known || inconclusive(verdict) || (due && (verdict != nil || !failedElsewhere))) && !skipVerification(url, verification.Name) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Design note, not changed. failing is read from the kept pass, the last one recorded. A rate-limited pass of a paired provider is never recorded: decideReverification reschedules from it, then continues before noteFailure, so lastPass keeps the older pass. Say the older pass failed this check at url B, and the rate-limited pass got a 429 for it at url A. The retry then asks B alone, and A's rate-limited check is not asked again before the epoch, which is what 3095a69 set out to prevent. It needs a pass in which no url served, so it is narrow. Recording that pass as lastPass would remove it, and failing with it, but it touches the demote grace.

// so is every url when previous is no pass of this provider's urls or only names the
// whole pass. The node urls at the positions in skip are not asked, as in
// ValidateCollectionsExcept.
// whole pass. A retry of a dial (CheckConnect) names no check of any url, so it dials

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two notes on this rule, neither changed.

  1. kept() returns nil for a url the kept pass could not dial, so every retry re-dials it, not only its connect retry. With archive failing too, each archive retry dials the dead socket again and waits on it: up to about 20 s against a black-holed host (ten connect attempts, 0.6 s to 3.3 s each). This predates the PR.
  2. The converse, from the note on failedElsewhere: when the pass that could not dial a url, or left it out for its hold-off, was rate-limited, it was not recorded. The kept pass is then older and holds verdicts for that url, so the connect retry keeps them and dials nothing there. A dead url stays admitted until the epoch re-dials it.

failure := metrics.ProviderVerificationFailure{Scope: metrics.VerificationScopeURL, Check: failureCheck(admission.URLRefusal(index))}
if url, ok := admission.URLAt(index); ok && chainlib.IsWebSocketURL(url.Url) {
failure.Service = chainlib.WebSocketExtension
if !websocketLeft(chainParser, provider, admission) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit, not changed: websocketLeft gives the same answer for every refused ws url, and each call re-expands dialedNodeUrls and matches every dialed url by reflect.DeepEqual, on the publish path under rpsr.mu. The admission already holds the list it verified, so one loop over URLAt and URLRefused before the url loop would do.

@avitenzer avitenzer left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving at de59d070: 64fe2574 with main 5a230c64 (#687, #688) merged in by Update branch. Each merge's tree is identical to a local trial merge.

Checked on top of the review above:

  • CI is green on de59d070: unit-tests, govulncheck and lint.
  • On e3f556d0, the merge with #687, build, vet and gofmt are clean, and chainlib, metrics, common, routersession and rpcsmartrouter pass. They passed twice at 64fe2574 too. de59d070 adds only #688's chainlib test; build, vet and chainlib pass on it.
  • The rebase onto #700 changed Anna's commits only in the rename and skip hunks.
  • Reverting 948bdb0's reads in refusalsOf, websocketLeft or health, or be4e030's held-off case, fails a test each time.

Not this PR, found while checking it: on main, an AVAX provider configured with bare urls answers no C-chain relay ("No pairings available") once it has any ws url. A live socket's generated /C/rpc endpoint claims the path, which predates #659. A dead socket marks /C/rpc unserved, which is #659. This PR changes only the dead socket's label, from Fail to PartialFail. It needs its own ticket.

@avitenzer
avitenzer merged commit b6a9278 into main Nov 15, 2026
@avitenzer
avitenzer deleted the feat/MAG-3950-websocket-part branch November 15, 2026 13:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants