Repository navigation
fix(rpcsmartrouter): tell a non-EVM node the unsubscribe its spec declares, and bound a subscription's restores (MAG-4278) - #590
Conversation
…ubscribe gets its own subscription (MAG-4345) Unsubscribe decided "last client" under the lock, called the node off the lock, and only then tore the subscription down. The entry stayed in the active map while the node call was in flight, so a client arriving for the same params joined it, a local step with no node call. The local cleanup then saw the joiner's mapping and tore nothing down, but the node had already dropped the subscription: the joiner got no pushes, and its channel was never closed, so it waited until it disconnected. The subscription now leaves the active map in the first critical section, before the node is called. A subscription the router has told the node to drop takes no new clients, and a client arriving meanwhile creates a subscription of its own on the node. A restore that reaches its commit point while the call is in flight finds nothing to commit onto and ends the subscription it re-opened (the MAG-2547 discard), so the restored-since-read branch goes. A node call that fails no longer returns early with the client still attached: the client asked to leave and the subscription is already retired, so it ends locally and the error goes back with the reply. A client that was not the last when its call started but is by the time the lock is retaken now tells the node, which that path never did. teardownSubscriptionLocked splits into retireSubscriptionLocked (the map entry, identity-guarded, and the counter) and endUpstreamSubscriptionLocked (the upstream half), so Unsubscribe can run the two halves apart. Tests: TestWS_JoinerDuringLastClientUnsubscribeGetsItsOwnSubscription, from Anna Rayer's attached repro (the arriving client subscribes in the background, because it now needs the node the test holds), and TestWS_LastClientUnsubscribeFailingUpstreamStillEndsTheSubscription (the node applies the unsubscribe but never answers). Both fail on main. TestUnsubscribeDuringRestore_TellsTheNodeTheRestoredID now asserts the outcome rather than the path: neither id is left on the node. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Took 20 minutes
…pec declares (MAG-4278 item 1)
The three router-initiated teardown paths — the last client disconnecting,
a failed restore's cleanup, and the discard of a subscription a restore
re-opened onto an entry that had gone — derived the unsubscribe method
from the subscribe method by string surgery whose fallback is the literal
"unsubscribe". Only Tendermint has that method, so on Solana and every
Substrate chain the node rejected the call, the failure was traced, and
the node kept publishing into the pooled connection: the MAG-3366 leak
shape. The same paths named the subscription with the decimal string the
router canonicalises to internally, which Solana also rejects. The
explicit-unsubscribe path was always right, because the client names the
method.
The method is now resolved from the spec once, when the subscription is
created, and stored on it. The spec gives no field linking a SUBSCRIBE
directive to its UNSUBSCRIBE one, so the pairing is recovered from the
names a collection declares, and only ever returns a name it declares:
1. the subscribe name with its first subscribe/Subscribe token replaced
by unsubscribe/Unsubscribe, when the collection declares that name;
2. for the names rule 1 cannot reach, namespace pairing: within one
namespace an unclaimed subscribe pairs with an unclaimed
unsubscribe when each is the only one of its kind there. Uniqueness
is required in both directions, so an ambiguous namespace resolves
to nothing rather than to a guess.
Measured over the whole lava-specs catalog (317 subscribe apis in 57
collections): rule 1 resolves 242, rule 2 resolves 63, 12 resolve to
nothing. Nine of those are gRPC or REST subscriptions this manager never
serves; the rest are Enjin's two archive methods, which share a namespace
with one unclaimed unsubscribe between them, and Filecoin.ChainNotify,
for which the FVM spec declares no unsubscribe at all. For those the node
is told nothing: a call the spec says it does not answer cannot stop it
publishing, and a warning naming the method is logged when the
subscription is created.
The id now goes in the chain's own shape on every path, through one
renderer shared with the explicit path. One upstreamSubscriptionTeardown
value carries what the node must be told, snapshotted under the lock, so
the three paths cannot pick different methods or id shapes from each
other again.
getUnsubscribeMethod becomes derivedUnsubscribeMethod and reports failure
instead of answering a name it cannot reach with the bare literal. It is
reached only for a collection that declares no unsubscribe at all.
Tests: TestWS_TeardownOnSolanaSendsTheSpecUnsubscribeMethod, from the
repro Anna Rayer attached to the ticket, strengthened to assert the method
and the unquoted id the node was actually asked for rather than only the
absence of "unsubscribe" — which a router that sends nothing also
satisfies; TestWS_ClientDisconnectOnSubstrateSendsTheSpecUnsubscribeMethod
for the commonest shape, a client closing its socket on chainHead_v1_follow;
TestWS_TeardownTellsTheNodeNothingWhenTheSpecDeclaresNoUnsubscribe;
TestWS_ExplicitUnsubscribeStillSendsTheClientsMethod; and
TestResolveTeardownUnsubscribeMethod over the directive shapes the catalog
declares. All four end-to-end tests fail on main.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Took 1 minute
…unt its pool slot on pick (MAG-4278 item 11) Two gaps between the WebSocket subscription manager and its gRPC counterpart, which has carried both since MAG-4177. A restore was unbounded. Each one spawns a fresh listener, and that listener's own failure spawns the next restore, so a node that accepts a subscribe and ends it immediately was re-subscribed for the life of the process: one round trip per cycle, on every subscription it does this to, with nothing counting the cycles. A subscription now carries the gRPC manager's accounting — restoresWithoutProgress against maxRestoresWithoutProgress, refilled by a restore that stayed up for restoreStableAfter — and is given up once it is spent. Uptime rather than a delivered push is the measure of progress, so a quiet subscription does not spend its budget on the reconnects a load balancer's connection age limit causes. A subscription's pool slot was counted after the node answered the subscribe. In that window the connection carries no counted subscription, and maybeScaleDown closes a connection past minConnections that carries none, so the subscription being created on it died on a socket the pool had just closed. The pool gains ReserveConnectionForSubscription, which counts the slot under the same lock as the pick, and every failure after the reservation releases it: a failed subscribe, a reply that cannot be built, and a failed re-subscribe on the restore path. GetConnection and GetConnectionForSubscription keep their behaviour for callers that do not open a subscription. Tests: TestRestoreBudget_GivesUpOnceTheBudgetIsSpent and TestSubscribeReservesItsPoolSlot fail on main. TestRestoreBudget_RefillsAfterASubscriptionStaysUp and TestSubscribeReleasesItsPoolSlotWhenTheSubscribeFails are guards for the refill and the release rather than repros, and pass either way. The mock node gained rejectSubscribes, which answers a subscribe with a JSON-RPC error. Still open on the ticket's item 11, and not in this change: the WS manager closes a given-up subscription's client channel with no status, where the gRPC manager ends the client's stream with a reason. Giving WebSocket clients one needs a new SubscriptionEndError reason and a close-code decision in the ingress, which is client-visible behaviour rather than a drift fix. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Took 2 minutes
…ol slot while the node answers (MAG-4278 item 11) TestSubscribeReservesItsPoolSlot drives the pool's ReserveConnectionForSubscription on its own. The manager calls it in two places, StartSubscription and the restore, and with both of them reverted to GetConnection plus an IncrementSubscriptions after the node's answer every test in the package still passed, so nothing pinned the fix the window was about. TestSubscriptionCountsItsPoolSlotWhileTheNodeAnswers holds the node's answer to a client's subscribe, then to the restore's re-subscribe after the socket drops, and requires the pool to count the subscription in both windows. It fails with either call site reverted. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… that ended under it (MAG-4345) A client that is not the last one releases the lock between Unsubscribe's two critical sections. If the subscription ends there by another path — the node ends it with an error, or a restore gives it up — the listener's cleanup releases every client, stops counting the subscription and tells the node. Unsubscribe then retook the lock, found no clients left, and tore the subscription down again: totalSubscriptions and the pooled connection's count each went to -1 and the node was told twice. On main the mapper's last-client answer, false once the cleanup had removed every mapping, kept it from doing so; the retire/end split dropped that guard. The teardown now runs only while the subscription is still the live entry under its params. beforeUnsubscribeRelease is a test hook, nil in production, in the shape of beforeRestoreCommit. TestWS_UnsubscribeDoesNotTearDownASubscriptionThatEndedUnderIt fails without the guard (totalSubscriptions -1, pool count -1, two eth_unsubscribe) and passes on main's manager with only the hook added. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…when the client attaches (MAG-4345)
The watcher a subscribe starts looked its subscription up when the
goroutine first ran, to read the client's detached signal. Unsubscribe now
takes the subscription out of the map in its first step, so a client that
unsubscribed before its watcher was scheduled left the watcher with a nil
channel: it stayed parked until the socket closed, one goroutine per
subscribe and unsubscribe on a long-lived socket. On main the entry stayed
mapped for the node's round trip, which hid the race.
The signal is now read where the client is attached — by the creator while
it still owns the subscription, by a joiner under the manager lock — and
handed to the watcher.
TestUnsubscribeThenDisconnect_LeavesNothingBehind ("the disconnect watcher
must not stay parked once the client unsubscribed") failed 6 of 20 runs
without this and passes 30 of 30 with it; it passes 20 of 20 on main.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… restore to commit (MAG-4278 item 11)
TestRestoreBudget_RefillsAfterASubscriptionStaysUp waited for
`upstreamID != "" && upstreamCalls("eth_subscribe") == 2`. The old upstream
id is still set during the restore, so the wait ended as soon as the node
had read the re-subscribe, before it answered and before the restore
committed. The test then returned, and its cleanup cancelled the
subscription's context with the re-subscribe possibly still in flight:
the window in which rpcclient's Subscribe can deadlock its dispatch loop
(an unbuffered reply channel the cancelled waiter no longer reads), after
which manager.Close never returns.
It now waits for the upstream id to change, which is the restore's commit.
The budget tests still fail with the budget removed.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
AnnaR-prog
left a comment
There was a problem hiding this comment.
Reviewed at e7281dc (the top two commits). I tested it merged with current main b875f86. I also reran the catalog measurement over lava-specs b813946 with the repo's own spec loader.
Item 1 is correct.
- The measurement reproduces exactly: 57 collections, 317 subscribe apis; 242 by rule 1, 63 by rule 2, 12 with nothing.
- Expanded per chain, which is what the router sees, the derived fallback never produces an undeclared name.
eth_subscriberesolves toeth_unsubscribein all 147 EVM collections, so EVM is unchanged. Tendermint, Solana and every Substrate pair resolve as described.- The three teardown tests fail with only the teardown reverted.
TestWS_ExplicitUnsubscribeStillSendsTheClientsMethodpasses either way, which is right for a guard, but the description lists it among the four that fail on main.
590-1 (Low): no test pins the manager's reserve-on-pick.
TestSubscribeReservesItsPoolSlotdrives the pool method only. I reverted both manager call sites (StartSubscription and the restore) toGetConnectionplusIncrementSubscriptionsafter the node answers. Every test still passed.- I pushed
TestSubscriptionCountsItsPoolSlotWhileTheNodeAnswers. It holds the node's answer to a client's subscribe, then to the restore's re-subscribe, and requires the pool to count the subscription in both windows. It fails with either call site reverted.
590-2 (Low, your call): a queue overflow counts against the restore budget.
- The listener sends
ErrSubscriptionQueueOverflowintohandleUpstreamDisconnectlike a lost socket. - A busy subscription that overflows its 20000-push queue six times in a row, each within 30 s of the last restore, is now given up: every client's channel closes with no status. Before, each overflow was restored.
- An overflow means the router fell behind, not that the node keeps ending the subscription. Count only restores after a lost socket, or keep this and say so in the description?
590-3 (Low): the "no unsubscribe" warning is logged on every subscribe request.
- The method is resolved and the warning logged at the top of
StartSubscription, before the rate limit and the join paths. On FVM and ENJ every subscribe request logs it, joins included, not once when the subscription is created. - Moving the two lines to the creation path, after the pending loop, would match the description.
590-4 (Low): TestRestoreBudget_RefillsAfterASubscriptionStaysUp can end before the restore commits.
upstreamID != ""is always true, because the old id stays set, so the wait ends when the node has read the re-subscribe. The test then returns, and its cleanup cancels the subscription with the re-subscribe possibly still in flight.- If that cancel lands while rpcclient delivers the answer,
Subscribedeadlocks the client's dispatch loop. This is a pre-existing rpcclient bug, not this PR's, andmanager.Closethen hangs the test binary. I hit that exact hang once in a stress run with a test of my own shaped like this one. - I pushed a version that waits for the upstream id to change, which happens at the commit.
Verified:
- Every failure after a reservation releases it, and the pool counts balance at the end of each test.
- The budget tests fail without the budget.
- The new tests pass under
-race. - This PR carries #589's commit, so it also needs #589's two fixes. I pushed them here as cherry-picks, so this PR's own CI run includes them.
| // Resolved once, here, while the spec collection this message matched is in hand: the | ||
| // teardown paths have no client message to read the method off, and deriving it there is | ||
| // what sent non-EVM nodes a method they do not have (MAG-4278 item 1). | ||
| unsubscribeMethod, hasUnsubscribeMethod := resolveTeardownUnsubscribeMethod(protocolMessage.GetApiCollection(), subscribeMethod) |
There was a problem hiding this comment.
Resolved, and warned about, before the rate limit and the join paths, so FVM and ENJ log this on every subscribe request, joins included (590-3). Moving it to just before selectEndpoint makes it once per created subscription, as the description says.
| if last := activeSub.lastRestoredAt.Load(); last != 0 && time.Since(time.Unix(0, last)) >= restoreStableAfter { | ||
| activeSub.restoresWithoutProgress.Store(0) | ||
| } | ||
| if attempt := activeSub.restoresWithoutProgress.Add(1); attempt > maxRestoresWithoutProgress { |
There was a problem hiding this comment.
A queue overflow reaches here too (the listener passes every non-node error), so a subscription that keeps overflowing is given up after five restores within 30 s windows (590-2). Intended?
The parent PR squash-merged into main; this keeps this PR's own commits on top of main. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Resolves this PR against #608, #609 and the WebSocket stack (#590, #591, #612, #614): - protocol/metrics: #609's positional vectors in this PR's register-as-built manager; the two mapped_labels wrapper files go, and so does their dupl rule in .golangci.yml. - direct_rpc_connection.go: #608's h.do, called with common.JSONContentType. - direct_ws_subscription_manager.go: the stack's unsubscribeParams, comparing the API interface with spectypes.APIInterfaceTendermintRPC. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Description
MAG-4278 items 1 and 11, the two the ticket deferred until PRs #508, #509, #511 and #519 had merged. One commit each.
Jira ticket: MAG-4278
Stacked on #589 (MAG-4345), which edits the same file. Review the top two commits; merge after it.
Item 1: the teardown sends a method the node does not have
The three router-initiated teardown paths — the last client disconnecting, a failed restore's cleanup, and the discard of a subscription a restore re-opened onto an entry that had gone — derived the unsubscribe method from the subscribe method by string surgery whose fallback is the literal
unsubscribe. Only Tendermint has that method, so on Solana and every Substrate chain the node rejected the call, the failure was traced, and the node kept publishing into the pooled connection. The same paths named the subscription with the decimal string the router canonicalises to internally, which Solana also rejects.The ticket's suggested fix shape does not work as written, and this is the finding worth reviewing. It asks to "resolve the unsubscribe method at subscribe time", but a SUBSCRIBE parse directive carries nothing except its tag and its api name. The spec has no field linking it to its UNSUBSCRIBE directive, and the directive order does not pair them either: 28 of the 57 collections in the catalog are not subscribe-then-unsubscribe alternating, and in every Substrate spec the two groups are each sorted by name, which misaligns them.
So the pairing is recovered from the names a collection declares, by two rules that only ever return a name it declares:
subscribe/Subscribetoken replaced byunsubscribe/Unsubscribe;_or.) an unclaimed subscribe pairs with an unclaimed unsubscribe when each is the only one of its kind there. Uniqueness is required in both directions, so an ambiguous namespace resolves to nothing rather than to a guess.Measured over every spec in lava-specs at
b813946— 317 subscribe apis in 57 collections:Rule 2's 63 are the three Substrate verbs no string rule reaches, on 21 chains each:
author_submitAndWatchExtrinsic→author_unwatchExtrinsic,chainHead_v1_follow→chainHead_v1_unfollow,transactionWatch_v1_submitAndWatch→transactionWatch_v1_unwatch. All three are hand-checked in the unit test.Nine of the twelve that resolve to nothing are gRPC or REST subscriptions, which this manager never serves. The other three are Enjin's two
archive_v1_*methods, which share a namespace with one unclaimed unsubscribe between them, andFilecoin.ChainNotify, for which the FVM spec declares no unsubscribe at all. For those the node is told nothing: a call the spec says it does not answer cannot stop it publishing, and a warning naming the chain and method is logged when the subscription is created. Adding the two Enjin pairs and Filecoin's is a lava-specs follow-up.The id now goes in the chain's own shape on every path, through one renderer shared with the explicit path. One
upstreamSubscriptionTeardownvalue carries what the node must be told, snapshotted under the lock, so the three paths cannot drift from each other again.Item 11: an unbounded restore, and a pool slot counted too late
A restore was unbounded. Each one spawns a fresh listener whose own failure spawns the next, so a node that accepts a subscribe and ends it immediately was re-subscribed for the life of the process. A subscription now carries the gRPC manager's accounting and is given up once it is spent; a restore that stayed up for
restoreStableAfterrefills it, so a quiet subscription does not spend its budget on the reconnects a load balancer's age limit causes.A subscription's pool slot was counted after the node answered the subscribe. In that window
maybeScaleDowncan close the connection as unused, and the subscription being created on it dies on a socket the pool just closed. The pool gainsReserveConnectionForSubscription, which counts the slot under the same lock as the pick, and every failure after the reservation releases it.Not in this change, and still open on item 11: the WS manager closes a given-up subscription's client channel with no status, where the gRPC manager ends the client's stream with a reason. Giving WebSocket clients one needs a new
SubscriptionEndErrorreason and a close-code decision in the ingress — client-visible behaviour rather than a drift fix.Test plan
TestWS_TeardownOnSolanaSendsTheSpecUnsubscribeMethod, from the repro Anna Rayer attached to the ticket. Strengthened to assert the method and the unquoted id the node was actually asked for: her assertion thatunsubscribeis absent is also satisfied by a router that sends nothing at all, which is what the first draft of this fix did.TestWS_ClientDisconnectOnSubstrateSendsTheSpecUnsubscribeMethod— the commonest shape, a client closing its socket onchainHead_v1_follow.TestWS_TeardownTellsTheNodeNothingWhenTheSpecDeclaresNoUnsubscribe.TestWS_ExplicitUnsubscribeStillSendsTheClientsMethod— the path that was always right.TestResolveTeardownUnsubscribeMethodover the directive shapes the catalog declares, including the two ambiguous ones it must refuse.TestRestoreBudget_GivesUpOnceTheBudgetIsSpent,TestSubscribeReservesItsPoolSlot.main, verified withgo test -overlayagainst a copy of the manager with only the fixed behaviours reverted.TestRestoreBudget_RefillsAfterASubscriptionStaysUpandTestSubscribeReleasesItsPoolSlotWhenTheSubscribeFailsare guards for the refill and the release rather than repros, and pass either way.go test -count=1 ./protocol/rpcsmartrouter/on the committed tree: ok, 245s.-raceon the committed tree: ok.go build -buildvcs=false ./...andgo veton the committed tree.CI runs no unit tests on this repo, so the local runs above are the only test evidence.
🤖 Generated with Claude Code