Repository navigation
feat!: serve a provider with no websocket url over http, and publish the websocket as its own part (MAG-3950) - #602
Conversation
…(MAG-3950) The emitter's comment still said a spec that subscribes needs a ws url unless the router runs with --skip-websocket-verification. Since #602 a provider with no ws(s) url serves everything else over http and publishes the websocket's missing-url, whatever the flag says. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Took 2 minutes
avitenzer
left a comment
There was a problem hiding this comment.
Reviewed at bbf9478e (this PR's own diff against #582's branch). I found no correctness bug. I pushed one doc commit (924ad1ac); everything else is notes.
1. Stale wizard comment (fixed in 924ad1ac). tools/wizard/internal/emit/emit.go still said a spec that subscribes needs a ws url unless the router runs with --skip-websocket-verification. After this PR that is false, so the comment now says the provider serves everything else and publishes missing-url. Comment only, no code change.
Checked and fine:
- What "subscribes" means per interface. I ran
MissingWebsocketwith one http url against the bundled specs. COSMOSHUBtendermintrpc→ true,rest→ false,grpc→ false (MAG-2643 streaming). ETH1, BASE (through its import) and SOLANA → true, BTC → false.taggedApisis filled only from enabled collections, so a disabled subscribing collection can't raise the alarm. - Without the flag, an http-only provider is still really verified.
getExtensionsForVerificationassigns the base checks of a subscribing collection to thewebsocketextension when the flag is off. I traced that the http url still runs them, andTestValidateProviderCollections_NoWebsocketURLServesOverHTTPpins it:eth_chainIdis sent, and a wrong chain is refused onchain-ideither way. - A subscription with no ws url on the whole chain still gets the NoOp manager's "no endpoint" error.
wsConfiguredreads the configured lists, so this PR adds no hang or panic path. With the chart's flag on, production already ran this way. - A provider with only ws urls is still refused by
ProviderShapeError("HTTP/HTTPS is mandatory"), and dials nothing. - Indexes:
RefusedURLs(),URLRefusedandwebsocketLeftindexprovider.NodeUrls, asservedElsewhereandadmission_registryalready do. - The counter: every
countFailedCheckcall that publishes a refusal goes throughcountRefusals, so themissing-urlskip there covers boot, epoch and retry. Retries:degradedreadsadmission.Any(), which a missing url never sets, so nothing schedules it. missing-urland a refused ws url can't both appear for one provider (one needs no ws url, the other needs one), sopreviousChecknever mixes them.
Open decision (product, not code): once this ships, every http-only provider on a chain that subscribes (217 of 302 lava-specs specs, in every tenant, since the chart sets the flag) gets a PartialFail series plus one error-level verification check failing line per boot, and again after each demote → readmit. That is what 30715 point 5 asks for. Two things should be settled before release: whether feat/MAG-3950-alert-by-severity posts on missing-url, and whether a condition set by config should log at error level.
Nits (not changed): see the two inline comments. Also, chain_router.go:335 still has the old ws enforcement. Its only production caller (smartrouter test) sets IgnoreWsEnforcementForTestCommands, so it is dead outside chainlib tests and could be removed in a follow-up.
Verification on 924ad1ac (local):
go test -count=1passes forchainlib/...,metrics/...,common/...andrpcsmartrouter/...(rpcsmartroutertook 290 s).-race -count=2passes on the new and updated tests.- gofmt is clean,
go vetis clean, andgolangci-lint --new-from-rev=origin/feat/MAG-3950-verification-retriesreports 0 issues;tools/wizardbuilds.
| } | ||
|
|
||
| // isWebsocketURL reports whether a node url is ws(s), as verification reads it. | ||
| func isWebsocketURL(raw string) bool { |
There was a problem hiding this comment.
Nit: this is chainlib.isWebSocketUrl (chain_fetcher.go:131) again, with the same body over IsUrlWebSocket. Exporting that one and calling it here keeps a single definition of "ws(s) url, as verification reads it", so the publisher and the verifier can't drift. Up to you.
There was a problem hiding this comment.
Done in ff2d8d8: chainlib exports it as IsWebSocketURL, and refusalsOf and websocketLeft call it, so there is one definition.
| row.SpecValid = true | ||
| applyValidation(&row, validations[i]) | ||
| row.Router = routerVerdict(validations[i], excluded) | ||
| if missingWebsocket { |
There was a problem hiding this comment.
A question, not a bug. The reason is added only when no ws(s) url is configured. A provider with an http url and a ws url that is refused (dead socket) also serves no subscriptions, but its admitted http row has an empty reason. Only the ws row, refused, shows it. The metric treats both cases the same (service=websocket, scope=service). If health should match, the condition here would be "no ws(s) url admitted", using the same check as websocketLeft. It's fine to leave as is if the refused row is enough for #748.
There was a problem hiding this comment.
Changed in b78eb84: when the router refuses every ws url of a provider, its admitted rows now say no ws(s) node url admitted: serves no subscriptions, read the same way as websocketLeft, so health matches the metric. With no ws url, the reason stays as it was. Pinned by TestHealth_EveryWebsocketURLRefusedServesNoSubscriptions.
…(MAG-3950) The emitter's comment still said a spec that subscribes needs a ws url unless the router runs with --skip-websocket-verification. Since #602 a provider with no ws(s) url serves everything else over http and publishes the websocket's missing-url, whatever the flag says. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Took 2 minutes
bd544e4 to
d6b6cd5
Compare
…er and the publisher (MAG-3950) refusalsOf and websocketLeft read a node url as a websocket through their own copy of chainlib's isWebSocketUrl. chainlib exports it as IsWebSocketURL, and rpcsmartrouter calls it, so the verifier and the publisher of the websocket part read a url the same way and cannot drift (review of #602). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…hen every ws url is refused (MAG-3950) health added "serves no subscriptions" to a provider's admitted rows only when it had no ws(s) url. A provider whose ws urls the router refuses, a dead socket for one, serves no subscriptions either, and the metric publishes both cases as the websocket service. Its admitted rows now say "no ws(s) node url admitted: serves no subscriptions", read as websocketLeft reads the admission. The case with no ws(s) url keeps its reason (review of #602). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… dead outside its tests (MAG-3950) newChainRouter refused an endpoint with no ws url on a spec that subscribes, unless IgnoreWsEnforcementForTestCommands was set. Its only production caller, smartrouter test, sets it, so the check ran in chainlib tests alone, and it said the opposite of what the router now does: a provider with no ws url serves over http. The check and the flag are gone. TestSkipWebsocketVerificationIsPerParser still pins that a parser's flag does not leak into another parser or into the process default (review of #602). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… error (MAG-3950) A provider with no ws(s) url on a chain that subscribes logged "verification check failing" at error level, once per boot and again after each demote and readmit. The config sets it and no retry can fix it, so the line is a warning now, and error lines are left for checks that fail (decided on #602). The alarm metric and GET /debug/verification are unchanged, and the alert by severity leaves missing-url out. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…(MAG-3950) The emitter's comment still said a spec that subscribes needs a ws url unless the router runs with --skip-websocket-verification. Since #602 a provider with no ws(s) url serves everything else over http and publishes the websocket's missing-url, whatever the flag says. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Took 2 minutes
…er and the publisher (MAG-3950) refusalsOf and websocketLeft read a node url as a websocket through their own copy of chainlib's isWebSocketUrl. chainlib exports it as IsWebSocketURL, and rpcsmartrouter calls it, so the verifier and the publisher of the websocket part read a url the same way and cannot drift (review of #602). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
8c4f953 to
deef149
Compare
…hen every ws url is refused (MAG-3950) health added "serves no subscriptions" to a provider's admitted rows only when it had no ws(s) url. A provider whose ws urls the router refuses, a dead socket for one, serves no subscriptions either, and the metric publishes both cases as the websocket service. Its admitted rows now say "no ws(s) node url admitted: serves no subscriptions", read as websocketLeft reads the admission. The case with no ws(s) url keeps its reason (review of #602). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… dead outside its tests (MAG-3950) newChainRouter refused an endpoint with no ws url on a spec that subscribes, unless IgnoreWsEnforcementForTestCommands was set. Its only production caller, smartrouter test, sets it, so the check ran in chainlib tests alone, and it said the opposite of what the router now does: a provider with no ws url serves over http. The check and the flag are gone. TestSkipWebsocketVerificationIsPerParser still pins that a parser's flag does not leak into another parser or into the process default (review of #602). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… error (MAG-3950) A provider with no ws(s) url on a chain that subscribes logged "verification check failing" at error level, once per boot and again after each demote and readmit. The config sets it and no retry can fix it, so the line is a warning now, and error lines are left for checks that fail (decided on #602). The alarm metric and GET /debug/verification are unchanged, and the alert by severity leaves missing-url out. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… checks it was failing (MAG-3950) A provider out of the pairing is listed by the check that took it out alone, so everything else it failed left the list at the demote and logged "verification check passes again", though nothing had checked it. A provider with no ws url logged its missing url passing at each demote, and failing again at each readmit (review of #602). trackFailures now holds those failures while the provider is out: no line at the demote, and at the readmit each goes on with its start, or passes again then, counted from its start. The gauge publishes the start the logs name, so a part failure that outlasts a demote comes back with its own start, not the readmit's: the PartialFail alert of the alert-by-severity branch posts at the readmit when that start is older than 30 minutes. sr-verify-failed reads scope=provider alone and is unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…(MAG-3950) The emitter's comment still said a spec that subscribes needs a ws url unless the router runs with --skip-websocket-verification. Since #602 a provider with no ws(s) url serves everything else over http and publishes the websocket's missing-url, whatever the flag says. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Took 2 minutes
…er and the publisher (MAG-3950) refusalsOf and websocketLeft read a node url as a websocket through their own copy of chainlib's isWebSocketUrl. chainlib exports it as IsWebSocketURL, and rpcsmartrouter calls it, so the verifier and the publisher of the websocket part read a url the same way and cannot drift (review of #602). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…hen every ws url is refused (MAG-3950) health added "serves no subscriptions" to a provider's admitted rows only when it had no ws(s) url. A provider whose ws urls the router refuses, a dead socket for one, serves no subscriptions either, and the metric publishes both cases as the websocket service. Its admitted rows now say "no ws(s) node url admitted: serves no subscriptions", read as websocketLeft reads the admission. The case with no ws(s) url keeps its reason (review of #602). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
23f1ae2 to
a6e7a2b
Compare
… error (MAG-3950) A provider with no ws(s) url on a chain that subscribes logged "verification check failing" at error level, once per boot and again after each demote and readmit. The config sets it and no retry can fix it, so the line is a warning now, and error lines are left for checks that fail (decided on #602). The alarm metric and GET /debug/verification are unchanged, and the alert by severity leaves missing-url out. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… checks it was failing (MAG-3950) A provider out of the pairing is listed by the check that took it out alone, so everything else it failed left the list at the demote and logged "verification check passes again", though nothing had checked it. A provider with no ws url logged its missing url passing at each demote, and failing again at each readmit (review of #602). trackFailures now holds those failures while the provider is out: no line at the demote, and at the readmit each goes on with its start, or passes again then, counted from its start. The gauge publishes the start the logs name, so a part failure that outlasts a demote comes back with its own start, not the readmit's: the PartialFail alert of the alert-by-severity branch posts at the readmit when that start is older than 30 minutes. sr-verify-failed reads scope=provider alone and is unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…tp, and publish the websocket as its own part (MAG-3950) A provider with no ws(s) node url on a chain that subscribes was refused whole unless the router ran with --skip-websocket-verification. It now serves over http without subscriptions, and the alarm metric publishes scope=service, service=websocket, check=missing-url (PartialFail), as MAG-4505 comment 30715 point 5 decided. The alarm does not read the flag: the helm chart sets it on every router, so honouring it would hide every missing url. A ws(s) url carries no relay, only subscriptions (MAG-3477), so a refused one is published as the websocket service too: scope=service when the provider has no ws(s) url left, scope=url while another still serves. Before, it read as scope=url with no service, the same series a refused http url gives, so the alarm could not say "the websocket". health admits such a provider, and each admitted row says it serves no subscriptions. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ric (MAG-3950) docs/METRICS.md: a refused ws(s) url is published as service=websocket, and a provider with no ws(s) url on a chain that subscribes as its missing-url; config is now only a provider with no http(s) url. docs/LOCAL-COMPOSE.md: --skip-websocket-verification is no longer what lets an HTTP-only provider boot. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…(MAG-3950) The emitter's comment still said a spec that subscribes needs a ws url unless the router runs with --skip-websocket-verification. Since #602 a provider with no ws(s) url serves everything else over http and publishes the websocket's missing-url, whatever the flag says. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Took 2 minutes
…alone, not the whole pass (MAG-3950) A url that could not be dialed put the whole pass on the retry schedule. Each retry then re-ran every check of every url of the provider, so a dead websocket url made the router ask the http url its checks again on the fast schedule, although they had passed. MAG-4505 C4 rules that out for a failing check, and the cluster test of a closed ws port (MAG-4505 A3) showed it: the debug check over http was asked again on each retry. Retries now gives such a url its own entry, its dial (check connect). ValidateCollectionsAgain keeps every other url's verdicts, as it does for a check, so the retry dials only the urls that could not be dialed, and runs their checks once they answer. /debug/verification lists the entry as check connect, the same name as the failing row. The whole pass stays for a provider whose pass gave no verdicts to read. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… its second strike from the retry of its dial (MAG-3950) The tests of 041364a close the socket before boot. A socket that dies while its provider serves goes through the url grace first: the epoch pass that cannot dial it gives it a strike and keeps it, and puts its dial on the schedule. The new test checks that the retry of that dial is the second strike and refuses the url, and that it asks the http url nothing. The recorded admission keeps the epoch's pass, so the retry finds the failed dial and dials the url again. refusingSocket returns its switch, so a test can turn the socket off after boot. The README sentence on GET /debug/verification lists attempts, pendingSince and nextRetryAt again; 041364a split that list. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…d it (MAG-3950) A check is due for a provider, and its retry ran it at every node url that declares it, the urls that had passed it too. While one address of a provider failed the archive check, each retry also asked the archive checks at its other address, which had passed them (MAG-4505 finding 16). verify now runs a due check where the kept verdict failed, and keeps the verdict of a url that passed it. A check with no verdict to keep, or an inconclusive one, still runs. A url with nothing to run is not dialed, so the url that passed is asked nothing. The schedule is unchanged: the check leaves it once the urls that failed it pass. docs/METRICS.md says so, and that a url that could not be dialed is retried by its dial alone, which the commit of that change left out. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…t every url (MAG-3950) d6b6cd5 kept the passed verdict of every url for a due check. That holds when the kept pass is the one that failed the check at another url. A check can also be due with the kept pass holding it passed everywhere: a rate limit answers a later pass, which decides nothing and is not recorded, and its retry keeps the older pass. Each url kept its verdict, so the retry asked nothing, and the check stayed unasked until the epoch pass. The cluster test of a 429 on the chain-id check (MAG-4505 C5) showed it: the retry due at the end of the hold-off sent no eth_chainId, and the next one came 60 s after the 429, at the epoch. ValidateCollectionsAgain now gives verify the due checks that the kept pass holds failing, or undecided, at some url. A url that passed one of those keeps its verdict; a due check that no url failed runs at every url, as before d6b6cd5. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…er and the publisher (MAG-3950) refusalsOf and websocketLeft read a node url as a websocket through their own copy of chainlib's isWebSocketUrl. chainlib exports it as IsWebSocketURL, and rpcsmartrouter calls it, so the verifier and the publisher of the websocket part read a url the same way and cannot drift (review of #602). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…hen every ws url is refused (MAG-3950) health added "serves no subscriptions" to a provider's admitted rows only when it had no ws(s) url. A provider whose ws urls the router refuses, a dead socket for one, serves no subscriptions either, and the metric publishes both cases as the websocket service. Its admitted rows now say "no ws(s) node url admitted: serves no subscriptions", read as websocketLeft reads the admission. The case with no ws(s) url keeps its reason (review of #602). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… dead outside its tests (MAG-3950) newChainRouter refused an endpoint with no ws url on a spec that subscribes, unless IgnoreWsEnforcementForTestCommands was set. Its only production caller, smartrouter test, sets it, so the check ran in chainlib tests alone, and it said the opposite of what the router now does: a provider with no ws url serves over http. The check and the flag are gone. TestSkipWebsocketVerificationIsPerParser still pins that a parser's flag does not leak into another parser or into the process default (review of Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… error (MAG-3950) A provider with no ws(s) url on a chain that subscribes logged "verification check failing" at error level, once per boot and again after each demote and readmit. The config sets it and no retry can fix it, so the line is a warning now, and error lines are left for checks that fail (decided on #602). The alarm metric and GET /debug/verification are unchanged, and the alert by severity leaves missing-url out. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… checks it was failing (MAG-3950) A provider out of the pairing is listed by the check that took it out alone, so everything else it failed left the list at the demote and logged "verification check passes again", though nothing had checked it. A provider with no ws url logged its missing url passing at each demote, and failing again at each readmit (review of #602). trackFailures now holds those failures while the provider is out: no line at the demote, and at the readmit each goes on with its start, or passes again then, counted from its start. The gauge publishes the start the logs name, so a part failure that outlasts a demote comes back with its own start, not the readmit's: the PartialFail alert of the alert-by-severity branch posts at the readmit when that start is older than 30 minutes. sr-verify-failed reads scope=provider alone and is unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…verified (MAG-3950) Since #659 (MAG-4214) admission verifies the urls the router dials, generated per-path urls included, and a refused url's position is into that list. refusalsOf, websocketLeft and health read the position in the configured list: on a chain with per-path urls, such as STRK and AVAX, a dead socket was published as a refused http url, scope=url with no service, and health did not say that the provider serves no subscriptions. refusalsOf now reads the refused url from the admission (URLAt), websocketLeft finds each dialed url's verdict by value, as servedElsewhere does, and health passes the dialed list its validations are by position in. The per-url retry keeps its positions: boot, epoch and retry passes all verify the dialed list through validateProviderCollections. missing-url still reads the configured urls, since expanding a url never changes its transport. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
a6e7a2b to
7779bdb
Compare
…ts dial alone (MAG-3950) #700 (MAG-4733) leaves a url held off after a rate limit out of the pass: its outcome is ErrNotAskedHeldOff, with no verdicts. Retries read any such outcome as the whole pass. The retry that followed re-verified every url of the provider, so a sibling that had passed everything was asked its checks again: the cost b0f5a46 removes for a url that could not be dialed. While the url stayed held, each of those whole-pass retries asked the siblings again. A held-off url now gets the entry a url that could not be dialed gets, check connect. Its retry verifies that url alone, and every other url keeps its verdicts. While the url is still held, the retry asks nothing and backs off; the first retry after its hold-off ends verifies it. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…figs and the gauge's help (MAG-3950) The example configs, the compose file and two setup scripts still said that an http-only provider on a chain that subscribes is excluded at startup, or that --skip-websocket-verification is what lets one boot. With this PR such a provider serves without subscriptions and publishes the websocket's missing-url, with the flag or without it. Each comment now says so, and the single-chain HTTP-only examples drop the flag from their run command, where it changes nothing. The RESP cache lane's header says that overall-health no longer catches a missing WS leg. The gauge's help text lists missing-url among the checks, says what service=websocket means, and that missing-url is not retried. Comments and help text only. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
7779bdb to
64fe257
Compare
avitenzer
left a comment
There was a problem hiding this comment.
Reviewed at 64fe2574: this PR's thirteen commits rebased onto main 11878b1f (#700), plus two commits of mine. I found no correctness bug in the PR's own diff. One of my commits fixes how it composes with #700; the other fixes docs this PR makes stale.
Rebase. Three conflicts, all in chain_fetcher.go and all textual. #700 added a skip parameter to ValidateCollectionsAgain and a skip field to verifyOptions, where commits 4, 6 and 7 rewrote the doc comment and added failing. Both sides are kept. git range-diff against the old head shows 9 of the 13 commits byte-identical, and the other 4 differ only in those hunks. The rebase re-signed every commit.
Pushed:
be4e0304: a url held off after a rate limit no longer puts the whole pass on the schedule. It gets theconnectentry a url that could not be dialed gets, so its retry verifies it alone and asks nothing while it is still held. Two tests pin it; both fail with the one-line change reverted. Details inline.64fe2574: docs. Nine example configs,docker/docker-compose.ymland two setup scripts still said an http-only provider on a chain that subscribes is excluded at startup, or needs--skip-websocket-verificationto boot. The gauge's help text listed every check butmissing-url. Comments and help text only. I checked which interfaces publishmissing-urlwithMissingWebsocketagainst the bundled specs: SOLANA, ETH1 and COSMOSHUB tendermintrpc do; BTC, HYPERLIQUID, APT1 and COSMOSHUB rest and grpc do not.
Checked and fine where #700 and this PR meet:
URLAtandRefusesURLreadurls, whichrefusals()copies, sorefusalsOfandwebsocketLeftread the right url afterRerun(),KeepingThrottledFromandWithoutURLRefusals.- An http url the new per-url rule keeps whole has every verdict
kept, soURLAskedis false andurlPassResultsrecords neither an answer nor a 429 for it. #700'sTestProbeReverification_ARetryLeavesAURLItDidNotAskAsItWasstill pins that. A url with no checks is the exception; see the first follow-up. failingandskipcompose: a held url isnotAsked, its siblings keep their verdicts, andKeepingThrottledFromkeeps the held url's refusals.healthbuilds its rows from the dialed list, and its validations are by position in that list, soeveryWebsocketRefusedreads the right rows on STRK.
Design notes, not changed: three inline comments. Two are about a retry keeping verdicts from the last recorded pass while a rate-limited pass of a paired provider is never recorded. One is about a dead url being re-dialed by every retry, which predates this PR. Also, config and missing-url are both set by config and no retry fixes either, but only missing-url is kept off the schedule and the counter. A provider with no http(s) url is retried whole and counts check=config on every retry (noteFailure). That predates this PR, which now states the rule for one of the two.
For follow-ups, not this PR:
- #700's
URLAskedanswers true for a url with no checks that a retry kept whole without dialing. That is every ws url when the router runs with--skip-websocket-verification, as the chart sets.urlPassResultsthen records an answer, andRecordAnswerclears the hold-off the WS subscription path put on that socket. Every retry that keeps a socket whole does it, and this PR adds more such retries. Worth a ticket. - docs#91 updates
cli.mdandexamples/index.md, butreference/chains/{bitcoin,aptos,hyperliquid,solana}.mdstill say to run each example with--skip-websocket-verificationbecause it has no websocket leg.64fe2574drops the flag from this repo's single-chain examples. scripts/pre_setups/cache/init_smartrouter_eth_resp_cache.shrelied on overall-health to catch a missing WS leg, which this PR ends. Asserting that nomissing-urlseries exists would restore the check.
Verification on 64fe2574: go build ./... (wizard included), go vet ./... and gofmt are clean. go test -count=1 passes on chainlib/..., metrics, common, holdoff and rpcsmartrouter, run on 7779bdb3, whose code differs only in the gauge's help string. -race -count=2 passes on the retry, websocket-part, health, transition-log and hold-off tests. CI: unit-tests, unit-tests-race, govulncheck and lint are green.
| if FailedCheck(outcome.err) != CheckUnsupportedAddon { | ||
| switch { | ||
| case FailedCheck(outcome.err) == CheckUnsupportedAddon: | ||
| case FailedCheck(outcome.err) == CheckConnect, errors.Is(outcome.err, ErrNotAskedHeldOff): |
There was a problem hiding this comment.
be4e0304: ErrNotAskedHeldOff (#700) is a url outcome with an error and no verdicts, so before this line it fell to default and scheduled the whole pass. The retry that followed asked every sibling that had passed its checks again, and while the url stayed held, each whole-pass retry asked them again. The held url now gets the connect entry. ValidateCollectionsAgain with only=[connect] verifies it alone, because kept is nil for an outcome with an error, and keeps every sibling whole. While the url is still held, skip leaves it notAsked and the entry backs off. Pinned by TestVerificationRetries_AHeldOffUrlIsRetriedAloneOnceItsHoldOffEnds and the new case in TestRetries_AreTheChecksThatFailed.
| _, failedElsewhere := opts.failing[checkIDOf(verification)] | ||
| verdict, known := keptVerdicts[verdictKey(verification)] | ||
| if (due || !known || inconclusive(verdict)) && !skipVerification(url, verification.Name) { | ||
| if (!known || inconclusive(verdict) || (due && (verdict != nil || !failedElsewhere))) && !skipVerification(url, verification.Name) { |
There was a problem hiding this comment.
Design note, not changed. failing is read from the kept pass, the last one recorded. A rate-limited pass of a paired provider is never recorded: decideReverification reschedules from it, then continues before noteFailure, so lastPass keeps the older pass. Say the older pass failed this check at url B, and the rate-limited pass got a 429 for it at url A. The retry then asks B alone, and A's rate-limited check is not asked again before the epoch, which is what 3095a69 set out to prevent. It needs a pass in which no url served, so it is narrow. Recording that pass as lastPass would remove it, and failing with it, but it touches the demote grace.
| // so is every url when previous is no pass of this provider's urls or only names the | ||
| // whole pass. The node urls at the positions in skip are not asked, as in | ||
| // ValidateCollectionsExcept. | ||
| // whole pass. A retry of a dial (CheckConnect) names no check of any url, so it dials |
There was a problem hiding this comment.
Two notes on this rule, neither changed.
kept()returns nil for a url the kept pass could not dial, so every retry re-dials it, not only itsconnectretry. With archive failing too, each archive retry dials the dead socket again and waits on it: up to about 20 s against a black-holed host (ten connect attempts, 0.6 s to 3.3 s each). This predates the PR.- The converse, from the note on
failedElsewhere: when the pass that could not dial a url, or left it out for its hold-off, was rate-limited, it was not recorded. The kept pass is then older and holds verdicts for that url, so theconnectretry keeps them and dials nothing there. A dead url stays admitted until the epoch re-dials it.
| failure := metrics.ProviderVerificationFailure{Scope: metrics.VerificationScopeURL, Check: failureCheck(admission.URLRefusal(index))} | ||
| if url, ok := admission.URLAt(index); ok && chainlib.IsWebSocketURL(url.Url) { | ||
| failure.Service = chainlib.WebSocketExtension | ||
| if !websocketLeft(chainParser, provider, admission) { |
There was a problem hiding this comment.
Nit, not changed: websocketLeft gives the same answer for every refused ws url, and each call re-expands dialedNodeUrls and matches every dialed url by reflect.DeepEqual, on the publish path under rpsr.mu. The admission already holds the list it verified, so one loop over URLAt and URLRefused before the url loop would do.
avitenzer
left a comment
There was a problem hiding this comment.
Approving at de59d070: 64fe2574 with main 5a230c64 (#687, #688) merged in by Update branch. Each merge's tree is identical to a local trial merge.
Checked on top of the review above:
- CI is green on
de59d070: unit-tests, govulncheck and lint. - On
e3f556d0, the merge with #687, build, vet and gofmt are clean, and chainlib, metrics, common, routersession and rpcsmartrouter pass. They passed twice at64fe2574too.de59d070adds only #688's chainlib test; build, vet and chainlib pass on it. - The rebase onto #700 changed Anna's commits only in the rename and
skiphunks. - Reverting 948bdb0's reads in
refusalsOf,websocketLeftor health, or be4e030's held-off case, fails a test each time.
Not this PR, found while checking it: on main, an AVAX provider configured with bare urls answers no C-chain relay ("No pairings available") once it has any ws url. A live socket's generated /C/rpc endpoint claims the path, which predates #659. A dead socket marks /C/rpc unserved, which is #659. This PR changes only the dead socket's label, from Fail to PartialFail. It needs its own ticket.
Problem
MAG-3950 asks that one broken part cost only that part, and that the alarm say which part: "an add-on or the websocket". MAG-4505 comment 30715 point 5 decided the case with no socket: "the upstream serves over https without subscriptions, and the alarm says the wss url is missing."
On main:
ProviderShapeErrorwith checkconfig. Only--skip-websocket-verificationadmits it.routers_deployment.yaml, since chart chore(deps): bump goreleaser/goreleaser-action from 6 to 7 #69). So in production such a provider already serves over http, and nothing says it serves no subscriptions: no series, no log line.smartrouter health, it is refused.scope=urland its check, with no service. A refused http url publishes the same series, so the alarm can't tell a broken socket from a broken http url.Change
1. A provider with no ws(s) url serves over http, and raises the alarm.
ProviderShapeErrorkeeps only its http rule. The newchainlib.MissingWebsocketreports a provider that has no ws(s) url while its spec subscribes over a socket, meaning an enabled collection that isn't gRPC.collectWSEndpointsnever listed this one.smartrouter_provider_verification_failedpublishesscope=service,service=websocket,check=missing-url,severity=PartialFail. The failed-run counter leaves it out, since no check ran;verification check failingline logs it once per router start, at warning level: the config sets it, and no retry can fix it (39e02e5a, decided on this PR). A demote and readmit of the provider do not log it again (60154d20);GET /debug/verificationlists it.--skip-websocket-verificationstops the router running checks over a socket; it doesn't give a provider a socket it lacks. The chart sets the flag on every router, so honouring it would hide every missing url.2. A refused ws(s) url is published as the
websocketservice. A ws(s) url carries no relay, only subscriptions (MAG-3477), so losing one costs the provider its websocket:scope=service,service=websocketwhen the provider has no ws(s) url left;scope=url,service=websocketwhile another of its ws(s) urls still serves;PartialFail, because only subscriptions are switched off. Before, it wasFailwith no service.scope=url, no service,Fail.3.
smartrouter healthadmits such a provider. Each admitted row'srouter.reasonsaysno ws(s) node url: serves no subscriptions. The row'sokis unchanged. infra #748 parsesrouter.verdict, but none of its scripts acts on it.4. Wording:
--skip-websocket-verificationhelp now says what the flag does: dial ws(s) urls without running any check over them;confignow means only "no http(s) url".5. Docs:
docs/METRICS.mdcovers thewebsocketservice andmissing-url.docs/LOCAL-COMPOSE.mdno longer says the flag is what lets an HTTP-only provider boot.6. A url that could not be dialed is retried by its dial alone (MAG-4505 finding 14,
b0f5a460).ProviderAdmission.Retriesgives the url its own entry, its dial (checkconnect).ValidateCollectionsAgainkeeps every other url's verdicts, as it does for a check. So the retry dials only the urls that could not be dialed, and runs their checks once they answer.GET /debug/verificationlists the entry as checkconnect, the same name as the failing row; the README says so. The whole pass stays for a provider whose pass gave no verdicts to read.7. A failing check is retried only at the urls that failed it (MAG-4505 finding 16,
72274356and3095a699).verifyruns a due check where the kept verdict failed. A url that passed it keeps that verdict when another url failed it in the kept pass, since the retry is for that url. A url with nothing to run is not dialed.72274356alone kept those verdicts and asked nothing; the cluster test of a 429 on the chain-id check (MAG-4505 C5) caught it, and3095a699fixes it.docs/METRICS.mdsays so.8. Review follow-ups (avitenzer's review at
924ad1ac).ff2d8d86): chainlib exportsIsWebSocketURL, andrefusalsOfandwebsocketLeftcall it instead of their own copy, so the verifier and the publisher of the websocket part cannot drift.healthand a refused socket (b78eb847): a provider whose ws urls the router refuses, a dead socket for one, now getsno ws(s) node url admitted: serves no subscriptionson its admitted rows, read aswebsocketLeftreads the admission. It matches the metric, which publishes it as the websocket service. A provider with no ws url keepsno ws(s) node url: serves no subscriptions.missing-urllogs as a warning (39e02e5a): a provider with no ws url on a chain that subscribes logsverification check failingonce at warning level, not error, since the config sets it and no retry can fix it. The alert by severity leavesmissing-urlout. Both answer the open questions of the review.60154d20): it is listed by the check that took it out alone, so everything else it failed left the list at the demote and loggedverification check passes again, though nothing had checked it. A provider with no ws url logged its missing url passing at each demote, and failing again at each readmit. Those failures are now held while it is out: no line at the demote, and at the readmit each goes on with its start, or passes again then.a6e7a2b9): refactor: rename the Lava-named Go internals (MAG-2972) #695 (MAG-2972) renamedlavasessiontoroutersessionandutils.LavaFormat*toutils.Format*, and this PR's code follows it. fix(chainlib,rpcsmartrouter,lavasession): verify each generated per-path url on its own path, and let a failed path cost only that path (MAG-4214) #659 (MAG-4214) verifies the urls the router dials, generated per-path urls included, so a refused url's position is into that list.refusalsOfnow reads the refused url from the admission (URLAt),websocketLeftfinds each dialed url's verdict by value asservedElsewheredoes, andhealthpasses its dialed list. Before this, on a chain with per-path urls such as STRK, a dead socket was published as a refused http url, andhealthdid not say the provider serves no subscriptions. The per-url retry keeps its positions: boot, epoch and retry passes all verify the same dialed list.5e29ac49):newChainRouterrefused an endpoint with no ws url on a spec that subscribes, unlessIgnoreWsEnforcementForTestCommandswas set. Its only production caller,smartrouter test, set it, so it ran in chainlib tests alone. The check and the flag are removed.9. A url a pass left out for its hold-off is retried by its dial alone (
be4e0304, on the rebase onto #700). #700 (MAG-4733) leaves a url held off after a rate limit out of the pass, with no verdicts.Retriesread that as the whole pass, so the retry that followed asked the siblings that had passed everything their checks again, and while the url stayed held each whole-pass retry asked them again. It now gets theconnectentry a url that could not be dialed gets: its retry verifies that url alone, and every other url keeps its verdicts. While the url is still held, the retry asks nothing and backs off; the first retry after the hold-off ends verifies it.TestVerificationRetries_AHeldOffUrlIsRetriedAloneOnceItsHoldOffEndsand a case inTestRetries_AreTheChecksThatFailedpin it; both fail with the one-line change reverted. README anddocs/METRICS.mdsay so.10. The example configs and the gauge's help (
64fe2574). Nine example configs,docker/docker-compose.ymland two setup scripts still said that an http-only provider on a chain that subscribes is excluded at startup, or that--skip-websocket-verificationis what lets one boot. They now say it serves without subscriptions and publishesmissing-url, and the single-chain HTTP-only examples drop the flag from their run command, where it changes nothing. The gauge's help text listsmissing-urland says whatservice=websocketmeans. Comments and help text only.What operators will see change
service=websocket,check=missing-url,PartialFail. Nothing pages on it, because #978's gauge alert readsscope=provideronly. #978's panel draws failed runs from the counter, somissing-urldraws no mark there; the gauge holds it. The alert by severity (MAG-3950 item 5) leavesmissing-urlout, as decided on this PR: it is set by config and lasts as long as the config does.scope=url, no service,Failtoservice=websocket,PartialFail. feat(metrics): publish which configured providers failed verification, and which check (MAG-4594) #572 shipped in v1.6.5, and infra #978, now merged, reads its series. On #978's panel the broken socket moves from the "one url refused" lane to "lost websocket" ("one url lost websocket" while another socket serves). The severity alerts built on #978 (branchfeat/MAG-3950-alert-by-severity) page onFailand post onPartialFail, so a broken wss url posts instead of paging, andmissing-urlposts nothing./debug/verificationlists that retry ascheck: connect, where feat!: retry each failing verification on its own backoff, log its transitions, serve GET /debug/verification (MAG-3950) #582 listedwholePass.check: connecton/debug/verification, where fix(rpcsmartrouter,lavasession): only let a blocked provider back in when it has a usable url (MAG-4737) #700 on its own scheduledwholePass.smartrouter healthsays a provider serves no subscriptions when the router refuses each of its ws urls, as it already did when it has none.scope=providerseries, as before. When it is readmitted, a failure it had before comes back with its own start, not the readmit's, and nopasses againline is logged for it at the demote. The PartialFail alert of the alert-by-severity branch then posts at the readmit when that start is older than 30 minutes;sr-verify-failedreadsscope=provideronly and is unchanged.Tests
chain-id. Run with the flag on and off.missing-url; the only socket refused givesscope=service; with another socket left it givesscope=url; published asPartialFail; BTC subscribes nowhere, so it has no websocket to miss./debug/verification, and is not put on the retry schedule. The epoch pass logs nothing new.missing-urland still counts the refused archive beside it.TestVerificationRetries_AUrlThatCannotBeDialedRetriesOnlyItsDial): its dial is on the schedule, the retry loop dials it again with no request to the http url, and the url is admitted once its socket answers. It fails with feat!: retry each failing verification on its own backoff, log its transitions, serve GET /debug/verification (MAG-3950) #582'sRetries.TestVerificationRetries_ASocketThatDiesTakesItsSecondStrikeFromTheRetryOfItsDial): the epoch's failed dial is graced and puts the dial on the schedule, and the retry of the dial is the second strike that refuses the url, with no request to the http url. It fails with feat!: retry each failing verification on its own backoff, log its transitions, serve GET /debug/verification (MAG-3950) #582'sRetriestoo.TestVerificationRetries_ACheckIsRetriedOnlyAtTheUrlThatFailedIt): the retry asks the url that failed it, the url that passed is asked nothing, and the check leaves the schedule once the first url passes. It fails with feat!: retry each failing verification on its own backoff, log its transitions, serve GET /debug/verification (MAG-3950) #582's rule inverify.TestVerificationRetries_ADueCheckThatEveryUrlPassedRunsAtEveryUrl): the retry asks it at every url. It fails with72274356's rule.TestRefusalsOf_ADeadSocketOnAChainWithPerPathUrlsIsTheWebsocket,TestHealth_EveryWebsocketURLRefusedOnAChainWithPerPathUrls): on STRK a bare https url and a bare ws url are nine dialed urls. The dead socket is the websocket service, scopeservice, with no http url refused, and every admitted http row says the provider serves no subscriptions. Both fail on the rebased code before the fix, and reverting any one of the three fixed reads turns one of them red.TestTrackFailures_AProviderOutOfThePairingPassesNothingItFailed): a missing ws url logs nopasses againat the demote and no second line at the readmit, and keeps its start; an archive check it passes at the readmit passes again then, counted from its own start. Dropping either half of the fix turns it red.healthwith every ws url refused (TestHealth_EveryWebsocketURLRefusedServesNoSubscriptions): with the flag on and off, the admitted http row names the reason and the dead ws url is refused; a socket that answers leaves the reason out. It fails without the new case.missing-url. Each gets a socket, or its expectation names it.TestRetries_AreTheChecksThatFailedexpects the url it could not dial asconnect, not the whole pass.TestSkipWebsocketVerificationIsPerParserkeeps pinning that a parser's flag does not leak into another parser or into the process default; with the enforcement gone, both routers accept the http-only endpoint.64fe2574, rebased on main11878b1f:go build ./...,go vet ./...and themetricstests pass, and CI is green. The fullchainlib/...,metrics,common,holdoffandrpcsmartroutersuites, and-race -count=2on the retry, websocket-part, health, transition-log and hold-off tests, passed on the same code at7779bdb3;64fe2574changes only comments and the gauge's help string, which no test reads.a6e7a2b9, rebased on main36b0cd82:go build ./...(the wizard too), everychainlibpackage,rpcsmartrouterandmetricspass, and-raceon the websocket, health, transition-log, retry and per-path tests passes three runs in a row. Each of the thirteen commits builds and passesgo veton its own.36b0cd82, on k3d (smart-router-automationmag-4505, row A3, on eth-failover-real-sim-router, with the chart's flag and without it): with provider 1's ws port closed, its retries dialed the socket alone, and its http checks had the start pass alone, before and after the port opened. Subscriptions went to the other providers while it was closed, and reached it again after.36b0cd82,go test -raceonchainlib,metricsandcommon: passes.rpcsmartrouterpasses exceptTestHealthCommand_RejectsUnbootableCrossValidation, which fails the same way on main under-raceonce other tests have run (the health command writes the global log level), and passes alone.--new-from-revfeat!: retry each failing verification on its own backoff, log its transitions, serve GET /debug/verification (MAG-3950) #582's head) onchainlib,metricsandrpcsmartrouter: 0 issues.--epoch-duration 1m --verification-retry-delay 2s). Item 6 changes what those retries run, not what they decide: the two tests above cover the second strike and the re-admission. The three providers:https-onlywith one http url,with-wswith http and ws urls, andwrong-chainwith one http url answering chain id0x89.https-onlyandwrong-chainare both out onconfig: "subscriptions are applicable for this chain, but websocket is not provided". The refusal comes before any check, so it also hides thatwrong-chainis on the wrong chain.https-onlyis in the pairing, withservice=websocket,check=missing-url,PartialFail, no retry scheduled and one error line.wrong-chainis out onchain-id, verified over http. Onlywith-ws's socket is listed for subscriptions, and 10 of 10 relays were answered.with-ws's socket was killed right after boot, at 12:59:45. The epoch at 13:00:01 gave it the grace's first strike, and the retry at 13:00:03 refused it. It was published asscope=service,service=websocket,check=connect,PartialFail, with one error line, while the provider stayed in the pairing and relays kept being answered. Once the socket was back, the next retry cleared it at 13:00:07 (passes again,failedFor=4s).Release
Unreleased, as #580 and #582 are. #463 and #572 shipped in v1.6.5.
Jira ticket: MAG-3950
🤖 Generated with Claude Code