Repository navigation
fix(rpcsmartrouter): fail a new subscription over to the next endpoint until one takes it (MAG-4730) - #677
fix(rpcsmartrouter): fail a new subscription over to the next endpoint until one takes it (MAG-4730)#677avitenzer wants to merge 3 commits into
Conversation
…t until one takes it (MAG-4730) A new WebSocket or gRPC subscription selected one endpoint and returned any connect or subscribe error to the client. selectEndpoint was always called with no ignore set, so a dead sticky (affinity) endpoint was handed back on every later subscribe of the connection, and the cascade to the backup tier ran only when the primary tier was empty. A primary that refused connections but was still registered therefore failed new subscriptions, with a healthy primary or backup never tried, until re-verification demoted it about two epochs later; HTTP relays had failed over long before. subscribeWithFailover, in the shared subscription core, runs the attempt on the selected endpoint and, while it fails, on the next endpoint the cascade selects with every endpoint already tried ignored: past a dead sticky endpoint, whose affinity is dropped, through the rest of the primary tier, then the backup tier. The client gets an error only once every endpoint failed: the last one's, which is also the attempt its relay record names. The walk is bounded by the number of endpoints in both tiers, and it stops when the client's context ends, so no further endpoint is charged with a failure the client caused. It also stops, as HTTP retries do, at a node's refusal every endpoint would repeat. When the attempt's error carries the node's own answer (a JSON-RPC error object, a gRPC status) and the error registry rates it non-retryable (ClassifyNodeErrorForRetry with the transport's own rows, where the HTTP relay's IsNonRetryable comes from), the client gets that answer from the one endpoint asked: invalid params (-32602, gRPC InvalidArgument) end the walk. A rate limit, an unavailable node or a method this node does not offer moves on, as do dial, handshake and transport failures and answers the registry does not know. Failing over on every error would make one malformed subscribe ask every endpoint and, on WebSocket, charge each an optimizer failure. Each attempt still records its outcome against its own endpoint, as the single attempt did: the relay record, on WebSocket noteSubscribeFailure (optimizer failure or rate-limit hold-off), and on success RecordAnswer and the optimizer sample. selectFromTier now judges hold-off readiness among the endpoints not yet tried. Counting an endpoint that already failed as the ready one would narrow the tier to it, and a held-off primary would be passed over for the backup tier, or an error, without being tried. docs/RATE-LIMIT-HOLDOFF.md's two subscription rows say so. Moving a subscription from a backup back to a recovered primary is unchanged. Verified: subscription_failover_test.go reproduces the ticket's A2 and A3 on WebSocket (first-available and the router's optimizer) and on gRPC, A1 on WebSocket, and the hold-off case on both managers. On the unfixed code 10 of 10 (WS A2, first-available), 2 of 10 (WS A2, router optimizer), 10 of 10 (WS A3), 10 of 10 (gRPC A2) and 3 of 3 (gRPC A3) subscribes failed, A1's later subscribe failed with the dead endpoint's HTTP 503, and selection returned the backup. A WebSocket subscribe every node refuses with invalid params is asked of one endpoint and charged to it alone; without the stop it was asked of all three and charged three times. Refusals the registry rates retryable fail over, and a table pins the decision to the registry's JSON-RPC and gRPC rows. All pass, five times over under -race; go test ./protocol/rpcsmartrouter/... and go vet pass, and golangci-lint reports only two findings already on main. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Took 2 minutes Took 1 minute
Conflict in docs/RATE-LIMIT-HOLDOFF.md: MAG-4387 (628e1e5) rewrote the chain-tracker row and added the probe-loop row, and this branch edited the WS-subscriptions row beside them. Kept main's two rows and this branch's WS-subscriptions row. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Took 3 minutes
|
Updated the branch so it merges again ( MAG-4387 (#537, Tested on the merged tree: 🤖 Generated with Claude Code |
…tempt (MAG-4730) The subscription table says a new subscription's event has one attempt, the request it sent. A subscribe that now fails over sends several, and the event still holds one: the last, whose sent_ms includes the earlier attempts' time. The earlier failures are only in the failover warning. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
AnnaR-prog
left a comment
There was a problem hiding this comment.
Reviewed at 1188afa, tested merged with main c19b414 and again with current main ada2a55. The merge waits for an owner decision (677-1), so I'm not approving yet. The failover works as the body says, and a failed walk leaks nothing. One Low is fixed in 768dd3c, which I pushed.
1188afa is the review head 3ac1ef3 plus a merge of main at 628e1e5. The PR's code diff is unchanged. Its docs/RATE-LIMIT-HOLDOFF.md resolution (main's chain tracker and probe loop rows, then this PR's WS row) is the one I reached independently. It merges clean with ada2a55.
677-1 (owner decision, @Tomelia1999 and Omer): a new subscription now tries every endpoint, but an HTTP relay stops at the retry limit.
subscribeWithFailover(subscription_common.go:384) asks every primary, then every backup. Its only bound is the number of endpoints, and nothing reads--set-relay-retry-limit.- Tom ruled on MAG-3978 (comment 30709) that a relay stops at the configured retry limit, primaries and backups alike. The default is 2, so 3 calls. The docs#81 failover page says the same. MAG-4730's expected behaviour says the walk "matches HTTP failover". With 3 or fewer endpoints the two agree. With more, the walk asks nodes that a relay would not.
- The bound also sets the worst case. An endpoint that drops packets costs a full dial timeout, 30 s by default, on WebSocket as on gRPC. A WebSocket subscribe runs inside the socket's read loop (
consumer_websocket_manager.go:625), so that client's socket reads nothing else for N × 30 s. Identical subscribes wait on the creator's walk without watching their own context (direct_ws_subscription_manager.go:606). When it fails, each one becomes the creator in turn, so K of them wait up to K × N attempts. - Question: should a new subscription stop at the retry limit, as a relay does, or try every endpoint, as here? It is the same rule as the third open question on MAG-4283.
677-2 (Low): a refusal that every node gives, and that the registry rates retryable, is asked of every endpoint and charged to each.
- Probe with two primaries and a backup, all giving geth's answer to an unknown subscription name (
-32601,no "typo" subscription in eth namespace). All three were asked, backup included, and each was charged one optimizer failure. On main, one endpoint was asked and charged. The result is the same on ada2a55. - The registry rates that answer retryable and a node capability, and
-32001("notifications not supported") retryable and data scope. An HTTP relay does not fail the session on either:shouldFailSessionForResultexcuses node-capability and data-scope answers (rpcsmartrouter_server.go:2602). On WebSocket,noteSubscribeFailure(direct_ws_subscription_manager.go:447) charges a failure for every error except a 429. - The commit says the stop exists so that one malformed subscribe does not "ask every endpoint and, on WebSocket, charge each an optimizer failure". The stop covers only the non-retryable rows (-32602 and -32600 on JSON-RPC).
- Suggestion: whatever bound 677-1 settles on also bounds the fan-out. Scoring a refusal the way
shouldFailSessionForResultdoes would end the charging, but it also changes main's scoring of a single attempt, so it is the author's or the owners' call. Not fixed.
677-3 (Low): docs/RELAY-END-EVENT.md says a subscription request has one attempt, the request it sent.
- After a failover, the event still has one attempt: the last one.
SubscriptionUpstream.Sentkeeps only the last record (chainlib/subscription_relay_report.go:50). That attempt'ssent_msincludes the time the earlier attempts took, and the earlier attempts appear only in the failover warning. - Fixed in 768dd3c: a paragraph under the subscription table says so.
Checked and correct:
- The ticket's cases: A1 on WS, A2 and A3 on WS and gRPC. Cut to one attempt (main's behaviour), all of them fail. On the PR they pass. Readiness for the hold-off is now judged among untried endpoints: with main's whole-tier rule,
TestSelectEndpoint_HeldOffEndpointIsTriedOnceTheReadyOneFailedfails on both managers. A walk that never stops fails the invalid-params subtest and the registry table. - A failed walk leaves nothing behind. Probe: 40 failed WS walks (dial refused, or subscribe refused) and 40 failed gRPC walks that took a slot and released it. Afterwards: 0 slots, 0 live upstream subscriptions, no manager or sticky entry, and the same goroutine count. Run under -race on c19b414, and again without -race on ada2a55.
- Selection: primaries, then backups, on both managers. The router's optimizer honours the ignore set. A dead sticky endpoint drops the affinity, and the new endpoint is stored on success. On gRPC, a client that leaves ends the walk: the subscribe runs on the stream's context (
grpcproxy.go:151). On WebSocket, the subscribe's context ends only with the socket's handler, so that check never fires mid-walk, the same as main's single attempt. - With #664 on the same tree: a lost subscription is given up with "upstream subscription lost". The client resubscribes on a new connection, fails over past the endpoint that refused, and carries no end mark. Probe, 3 of 3 under -race. #678's edits to
docs/RATE-LIMIT-HOLDOFF.mdare in other paragraphs, and the two merge clean. - Disabled URLs are still offered to the walk. That is MAG-4161 (To Do, Tom), out of scope here.
Tested: On c19b414: build and vet on main+677 and main+677+664; full protocol/rpcsmartrouter on main+677+664 (304 s); the 261 subscription tests under -race on main+677+664, and again with #664's fixes; protocol/chainlib, protocol/chaintracker and protocol/endpointstate. Three mutants (one attempt, whole-tier hold-off, never stop), all caught. Leak and overlap probes under -race. On ada2a55, with both fixes: build and vet; this PR's tests and probes; full protocol/rpcsmartrouter; the 262 subscription tests under -race; chainlib, chaintracker, endpointstate and holdoff. golangci-lint --new-from-rev origin/main on rpcsmartrouter: 0 issues on both bases.
Description
A new WebSocket or gRPC subscription tried one upstream, so one refused connect failed the subscribe while a healthy primary or backup was never tried. It now moves on: past a dead sticky endpoint, through the other primaries, then the backups, and errors only when every endpoint failed.
A refusal the error registry rates non-retryable (invalid params) ends the walk at once, as it ends HTTP retries.
Split out of #666, which carried sixteen tickets in one PR, so each fix can be reviewed on its own. The commits are unchanged.
For reviewers:
ConnectionTimeoutbefore the walk moves on.backup.mdstill says "one upstream per subscribe".Jira ticket
Jira ticket: MAG-4730
Verification
mainand pass here, withTestWSSubscribe_FailsOverOnlyPastRefusalsAnotherEndpointCouldServeandTestEveryEndpointWouldRefuse_ReadsTheErrorRegistry.main:go build ./...andgo vetpass, and the touched test files pass. CI runs the full suite.🤖 Generated with Claude Code