Repository navigation
fix(rpcsmartrouter): tell the node to drop the subscription a restore replaced, and keep a redialed connection (MAG-4396) - #591
Conversation
…ubscribe gets its own subscription (MAG-4345) Unsubscribe decided "last client" under the lock, called the node off the lock, and only then tore the subscription down. The entry stayed in the active map while the node call was in flight, so a client arriving for the same params joined it, a local step with no node call. The local cleanup then saw the joiner's mapping and tore nothing down, but the node had already dropped the subscription: the joiner got no pushes, and its channel was never closed, so it waited until it disconnected. The subscription now leaves the active map in the first critical section, before the node is called. A subscription the router has told the node to drop takes no new clients, and a client arriving meanwhile creates a subscription of its own on the node. A restore that reaches its commit point while the call is in flight finds nothing to commit onto and ends the subscription it re-opened (the MAG-2547 discard), so the restored-since-read branch goes. A node call that fails no longer returns early with the client still attached: the client asked to leave and the subscription is already retired, so it ends locally and the error goes back with the reply. A client that was not the last when its call started but is by the time the lock is retaken now tells the node, which that path never did. teardownSubscriptionLocked splits into retireSubscriptionLocked (the map entry, identity-guarded, and the counter) and endUpstreamSubscriptionLocked (the upstream half), so Unsubscribe can run the two halves apart. Tests: TestWS_JoinerDuringLastClientUnsubscribeGetsItsOwnSubscription, from Anna Rayer's attached repro (the arriving client subscribes in the background, because it now needs the node the test holds), and TestWS_LastClientUnsubscribeFailingUpstreamStillEndsTheSubscription (the node applies the unsubscribe but never answers). Both fail on main. TestUnsubscribeDuringRestore_TellsTheNodeTheRestoredID now asserts the outcome rather than the path: neither id is left on the node. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Took 20 minutes
…pec declares (MAG-4278 item 1)
The three router-initiated teardown paths — the last client disconnecting,
a failed restore's cleanup, and the discard of a subscription a restore
re-opened onto an entry that had gone — derived the unsubscribe method
from the subscribe method by string surgery whose fallback is the literal
"unsubscribe". Only Tendermint has that method, so on Solana and every
Substrate chain the node rejected the call, the failure was traced, and
the node kept publishing into the pooled connection: the MAG-3366 leak
shape. The same paths named the subscription with the decimal string the
router canonicalises to internally, which Solana also rejects. The
explicit-unsubscribe path was always right, because the client names the
method.
The method is now resolved from the spec once, when the subscription is
created, and stored on it. The spec gives no field linking a SUBSCRIBE
directive to its UNSUBSCRIBE one, so the pairing is recovered from the
names a collection declares, and only ever returns a name it declares:
1. the subscribe name with its first subscribe/Subscribe token replaced
by unsubscribe/Unsubscribe, when the collection declares that name;
2. for the names rule 1 cannot reach, namespace pairing: within one
namespace an unclaimed subscribe pairs with an unclaimed
unsubscribe when each is the only one of its kind there. Uniqueness
is required in both directions, so an ambiguous namespace resolves
to nothing rather than to a guess.
Measured over the whole lava-specs catalog (317 subscribe apis in 57
collections): rule 1 resolves 242, rule 2 resolves 63, 12 resolve to
nothing. Nine of those are gRPC or REST subscriptions this manager never
serves; the rest are Enjin's two archive methods, which share a namespace
with one unclaimed unsubscribe between them, and Filecoin.ChainNotify,
for which the FVM spec declares no unsubscribe at all. For those the node
is told nothing: a call the spec says it does not answer cannot stop it
publishing, and a warning naming the method is logged when the
subscription is created.
The id now goes in the chain's own shape on every path, through one
renderer shared with the explicit path. One upstreamSubscriptionTeardown
value carries what the node must be told, snapshotted under the lock, so
the three paths cannot pick different methods or id shapes from each
other again.
getUnsubscribeMethod becomes derivedUnsubscribeMethod and reports failure
instead of answering a name it cannot reach with the bare literal. It is
reached only for a collection that declares no unsubscribe at all.
Tests: TestWS_TeardownOnSolanaSendsTheSpecUnsubscribeMethod, from the
repro Anna Rayer attached to the ticket, strengthened to assert the method
and the unquoted id the node was actually asked for rather than only the
absence of "unsubscribe" — which a router that sends nothing also
satisfies; TestWS_ClientDisconnectOnSubstrateSendsTheSpecUnsubscribeMethod
for the commonest shape, a client closing its socket on chainHead_v1_follow;
TestWS_TeardownTellsTheNodeNothingWhenTheSpecDeclaresNoUnsubscribe;
TestWS_ExplicitUnsubscribeStillSendsTheClientsMethod; and
TestResolveTeardownUnsubscribeMethod over the directive shapes the catalog
declares. All four end-to-end tests fail on main.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Took 1 minute
…unt its pool slot on pick (MAG-4278 item 11) Two gaps between the WebSocket subscription manager and its gRPC counterpart, which has carried both since MAG-4177. A restore was unbounded. Each one spawns a fresh listener, and that listener's own failure spawns the next restore, so a node that accepts a subscribe and ends it immediately was re-subscribed for the life of the process: one round trip per cycle, on every subscription it does this to, with nothing counting the cycles. A subscription now carries the gRPC manager's accounting — restoresWithoutProgress against maxRestoresWithoutProgress, refilled by a restore that stayed up for restoreStableAfter — and is given up once it is spent. Uptime rather than a delivered push is the measure of progress, so a quiet subscription does not spend its budget on the reconnects a load balancer's connection age limit causes. A subscription's pool slot was counted after the node answered the subscribe. In that window the connection carries no counted subscription, and maybeScaleDown closes a connection past minConnections that carries none, so the subscription being created on it died on a socket the pool had just closed. The pool gains ReserveConnectionForSubscription, which counts the slot under the same lock as the pick, and every failure after the reservation releases it: a failed subscribe, a reply that cannot be built, and a failed re-subscribe on the restore path. GetConnection and GetConnectionForSubscription keep their behaviour for callers that do not open a subscription. Tests: TestRestoreBudget_GivesUpOnceTheBudgetIsSpent and TestSubscribeReservesItsPoolSlot fail on main. TestRestoreBudget_RefillsAfterASubscriptionStaysUp and TestSubscribeReleasesItsPoolSlotWhenTheSubscribeFails are guards for the refill and the release rather than repros, and pass either way. The mock node gained rejectSubscribes, which answers a subscribe with a JSON-RPC error. Still open on the ticket's item 11, and not in this change: the WS manager closes a given-up subscription's client channel with no status, where the gRPC manager ends the client's stream with a reason. Giving WebSocket clients one needs a new SubscriptionEndError reason and a close-code decision in the ingress, which is client-visible behaviour rather than a drift fix. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Took 2 minutes
… replaced, and keep a redialed connection (MAG-4396) The two leftovers PR #519's review left open. A restore that runs on a live connection never told the node to drop the subscription it replaced. rpcclient ends one subscription on a connection that stays up when its forward queue overflows, and ClientSubscription.run discards the unsubscribe-the-server flag forward returns, so rpcclient sends nothing; handleUpstreamDisconnect used the old upstream id only for logging and for dropping its id mappings. So the node kept publishing the replaced subscription on the same socket, rpcclient read those frames and dropped them, and a subscription that overflows repeatedly left one more of them behind each time. The restore now tells the node, through the same teardown the other paths use. Where the restore followed a lost socket the node lost the subscription with it, and teardownUpstreamSubscription skips a dead connection, so one call is right on both paths. The local rpcclient subscription is deliberately not unsubscribed with it. Unsubscribe closes its Err channel, and a listener still parked there reads the close as a nil error, ends, and runs its deferred cleanup — tearing down the subscription the restore just installed. The listener that handed off has already returned, so nothing is parked there on the restore path, but the margin is one scheduling decision wide and the only thing it buys is dropping rpcclient's handler entry for a replaced subscription, which closing the connection does anyway. The decision the ticket asks for: errClientReconnected leaves the pooled connection alone. PR #519 made every subscription error except a queue overflow or a node error retire the connection, and this is the one case where the client has already recovered — rpcclient replaced the socket itself and holds a working one. Retiring it there closes that client, so the pool dials again for nothing and every other subscription sharing the connection ends too, including one a restore has just re-created on the new socket. A real socket failure still retires it. errClientReconnected becomes ErrClientReconnected: the fact that the connection is healthy is not readable from anywhere else, and a caller that treats every subscription error as a dead socket has no way to tell. Tests: TestOverflowRestore_TellsTheNodeToDropTheOldSubscription, TestClientRedial_KeepsTheConnectionAndItsSiblingSubscriptions and TestRetireConnectionOnUpstreamError_LeavesAClientThatRedialed fail on main. TestRestoreAfterADroppedSocket_DoesNotCallADeadNode guards the other half of the same call. The redial is driven through the upstreamErrSource seam the listener documents for it: rpcclient mints the error internally, and the MAG-4037 trigger, a call whose deadline has already passed, fails the redial on that same expired deadline, so no reconnect lands. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ly after a queue overflow (MAG-4396)
The restore told the node to drop the subscription it replaced after every
restore on a healthy connection. Only a queue overflow leaves that
subscription on a live socket. When rpcclient redials on its own
(ErrClientReconnected), it closes the old socket and the connection is now
kept, so the goodbye went out over the NEW socket. On Tendermint the
subscription id is the query, so `unsubscribe {query}` ended the
subscription the restore had just opened: every client of every Tendermint
subscription on that connection was left on a subscription the node no
longer publishes, with its channel open. On other chains the goodbye named
an id the new socket does not know, or, on a node that numbers its
subscriptions per socket, a sibling the redial had just restored.
The restore now receives the error that ended the subscription and says
goodbye only after a queue overflow, and never when the old id names the
restored subscription on the same connection.
TestTendermintRedial_KeepsTheRestoredSubscriptionOnTheNode and
TestClientRedial_RestoresEverySubscriptionOnTheKeptConnection drive a real
redial (calls past their deadline, then a live call) and fail on b784643.
TestReplacedStillOnNode pins the decision.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… that ended under it (MAG-4345) A client that is not the last one releases the lock between Unsubscribe's two critical sections. If the subscription ends there by another path — the node ends it with an error, or a restore gives it up — the listener's cleanup releases every client, stops counting the subscription and tells the node. Unsubscribe then retook the lock, found no clients left, and tore the subscription down again: totalSubscriptions and the pooled connection's count each went to -1 and the node was told twice. On main the mapper's last-client answer, false once the cleanup had removed every mapping, kept it from doing so; the retire/end split dropped that guard. The teardown now runs only while the subscription is still the live entry under its params. beforeUnsubscribeRelease is a test hook, nil in production, in the shape of beforeRestoreCommit. TestWS_UnsubscribeDoesNotTearDownASubscriptionThatEndedUnderIt fails without the guard (totalSubscriptions -1, pool count -1, two eth_unsubscribe) and passes on main's manager with only the hook added. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…when the client attaches (MAG-4345)
The watcher a subscribe starts looked its subscription up when the
goroutine first ran, to read the client's detached signal. Unsubscribe now
takes the subscription out of the map in its first step, so a client that
unsubscribed before its watcher was scheduled left the watcher with a nil
channel: it stayed parked until the socket closed, one goroutine per
subscribe and unsubscribe on a long-lived socket. On main the entry stayed
mapped for the node's round trip, which hid the race.
The signal is now read where the client is attached — by the creator while
it still owns the subscription, by a joiner under the manager lock — and
handed to the watcher.
TestUnsubscribeThenDisconnect_LeavesNothingBehind ("the disconnect watcher
must not stay parked once the client unsubscribed") failed 6 of 20 runs
without this and passes 30 of 30 with it; it passes 20 of 20 on main.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ol slot while the node answers (MAG-4278 item 11) TestSubscribeReservesItsPoolSlot drives the pool's ReserveConnectionForSubscription on its own. The manager calls it in two places, StartSubscription and the restore, and with both of them reverted to GetConnection plus an IncrementSubscriptions after the node's answer every test in the package still passed, so nothing pinned the fix the window was about. TestSubscriptionCountsItsPoolSlotWhileTheNodeAnswers holds the node's answer to a client's subscribe, then to the restore's re-subscribe after the socket drops, and requires the pool to count the subscription in both windows. It fails with either call site reverted. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… restore to commit (MAG-4278 item 11)
TestRestoreBudget_RefillsAfterASubscriptionStaysUp waited for
`upstreamID != "" && upstreamCalls("eth_subscribe") == 2`. The old upstream
id is still set during the restore, so the wait ended as soon as the node
had read the re-subscribe, before it answered and before the restore
committed. The test then returned, and its cleanup cancelled the
subscription's context with the re-subscribe possibly still in flight:
the window in which rpcclient's Subscribe can deadlock its dispatch loop
(an unbuffered reply channel the cancelled waiter no longer reads), after
which manager.Close never returns.
It now waits for the upstream id to change, which is the restore's commit.
The budget tests still fail with the budget removed.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
AnnaR-prog
left a comment
There was a problem hiding this comment.
Reviewed at b784643 (the top commit). I tested it merged with current main b875f86, and drove a real rpcclient redial instead of the listener seam.
591-1 (Medium): after a redial, the restore's goodbye ends the restored Tendermint subscription.
- With
ErrClientReconnectednow leaving the connection alone, the restore re-subscribes over the socket rpcclient just opened, on the sameUpstreamWSConnection. - The goodbye for the replaced subscription then goes out through that same client, so over the new socket. The old subscription died with the old socket.
- A Tendermint subscription's id is its query, so
unsubscribe {"query": …}ends the subscription the restore just opened. Every Tendermint subscription on that connection goes silent, and its clients stay on it with their channels open. - With a CometBFT-shaped mock and a real redial, the node saw
subscribe Q, status, subscribe Q, unsubscribe Qand held no subscription afterwards. It fails 10 of 10 on this head and passes on #590. - On EVM the goodbye names an id the new socket does not know. On a node that numbers subscriptions per socket, it could name a sibling the redial just restored.
- Only a queue overflow leaves the old subscription on a live socket. I pushed a fix:
- the listener hands the error to
handleUpstreamDisconnect; replacedStillOnNodesays goodbye only after an overflow, and never when the old id names the restored subscription on the same connection;- tests:
TestTendermintRedial_KeepsTheRestoredSubscriptionOnTheNode,TestClientRedial_RestoresEverySubscriptionOnTheKeptConnectionandTestReplacedStillOnNode. The first two fail on this head.
- the listener hands the error to
591-2 (Low): the redial can be forced, and the redial test should run a real one.
- A call whose deadline has passed fails its write and its own redial. The next call with a live deadline redials, and every subscription on the old socket ends with
ErrClientReconnected. TestClientRedial_KeepsTheConnectionAndItsSiblingSubscriptionsfeeds the error through the seam while the old socket and subscription stay alive, which a real redial cannot produce. It checks only the sibling, not that the redialed subscription is restored and keeps delivering, which is MAG-4396's Done-when.- The new EVM test does both with a real redial. I'm also correcting the
endSubscriptionWithcomment.
Verified:
- The overflow goodbye and the
ErrClientReconnectedpredicate are both pinned: the PR's tests fail with either one reverted. - A dropped socket still sends nothing.
- The pool counts balance when the restore reuses the same connection.
- The whole
protocol/rpcsmartrouterpackage passes merged with main, with the fixes for #589 and #590 included.
| // pushes in the meantime. The pool notify goes after the node call, so the connection is | ||
| // still live while it is used. | ||
| go func() { | ||
| dwsm.teardownUpstreamSubscription(oldUpstream) |
There was a problem hiding this comment.
After a redial the connection is kept, and its client now writes on the new socket, so this goodbye travels over a socket the old subscription never lived on. For Tendermint the old id is the query, the same as the restored one, so the node ends the subscription this restore just opened (591-1). Only after ErrSubscriptionQueueOverflow is the old subscription still on a live socket.
|
|
||
| // endSubscriptionWith drives the real listener's error branch with err, through the | ||
| // upstreamErrSource seam listenForUpstreamMessages takes for exactly this. A client that | ||
| // redialed cannot be produced from the node side: rpcclient mints ErrClientReconnected |
There was a problem hiding this comment.
A redial can be produced: a call past its deadline fails its write and its own redial, and the next live call redials. The seam leaves the old socket and subscription alive, which a real redial does not (591-2).
The parent PR squash-merged into main; this keeps this PR's own commits on top of main. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Resolves this PR against #608, #609 and the WebSocket stack (#590, #591, #612, #614): - protocol/metrics: #609's positional vectors in this PR's register-as-built manager; the two mapped_labels wrapper files go, and so does their dupl rule in .golangci.yml. - direct_rpc_connection.go: #608's h.do, called with common.JSONContentType. - direct_ws_subscription_manager.go: the stack's unsubscribeParams, comparing the API interface with spectypes.APIInterfaceTendermintRPC. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Description
The two leftovers PR #519's review left open: the overflow-restore leak, and the decision about what an rpcclient redial should do to a pooled connection.
Jira ticket: MAG-4396
Stacked on #590 (MAG-4278), which is itself stacked on #589 (MAG-4345); all three edit the same file. Review the top commit; merge after them.
1. The overflow restore never told the node to drop the old subscription
When a subscription's forward queue overflows, rpcclient ends it on a connection that stays up and the restore re-subscribes on that same live connection. Nothing told the node to drop the old one:
ClientSubscription.rundiscards the unsubscribe-the-server flagforwardreturns, so rpcclient sends nothing, andhandleUpstreamDisconnectused the old upstream id only for logging and for dropping its id mappings. The node kept publishing the replaced subscription on the same socket, rpcclient read those frames and dropped them, and a subscription that overflows repeatedly left one more behind each time.The restore now tells the node, through the same teardown value the other paths use. Where the restore followed a lost socket the node lost the subscription with it, and
teardownUpstreamSubscriptionalready skips a dead connection — so one call is right on both paths, andTestRestoreAfterADroppedSocket_DoesNotCallADeadNodepins that it does not redial into an outage just to say goodbye (MAG-4242).One thing deliberately left out, and it bit during this change. The ticket's alternative was to also end the local
ClientSubscription. Doing that closes itsErrchannel, and a listener still parked there reads the close as a nil error, ends, and runs its deferred cleanup — tearing down the subscription the restore has just installed.TestSolanaRouterIDSurvivesUpstreamReconnectcaught it: the client's reply channel came back closed. On the real restore path the listener that handed off has already returned, so nothing is parked there, but the margin is one scheduling decision wide and the only thing it buys is dropping rpcclient's handler entry for a replaced subscription, which closing the connection does anyway. If we want that entry released sooner it needs a path that does not touch the subscription'sErr.2. The decision:
errClientReconnectedleaves the connection alonePR #519 made every subscription error except a queue overflow or a node error retire the pooled connection, so the pool prunes and closes it.
errClientReconnectedis one of those errors, and it is the one case where the client has already recovered: rpcclient replaced the socket itself when a write failed while the old read loop was alive, and by the time the error reaches the listener it holds a new, working socket.Retiring the connection there is the opposite of a recovery. It closes the client rpcclient just fixed, so the pool dials again for nothing, and every other subscription sharing the pooled connection is ended too — including one a restore has already re-created on the new socket, which is the case the ticket describes. That is the same harm
TestUpstreamError_RetiresTheSharedConnectionOnlyOnASocketFailurewas written to prevent; a client that redialed on its own is not a socket failure.So:
ErrClientReconnectedjoins the queue overflow as an error that leaves the connection alone. The restore still runs — it finds the pool healthy and re-subscribes over the socket rpcclient opened. A real socket failure still retires the connection. Say so if you read it differently; it is a one-line predicate either way.errClientReconnectedis exported asErrClientReconnected, next toErrSubscriptionQueueOverflowand for the same reason: it says something a caller cannot read anywhere else.Test plan
TestOverflowRestore_TellsTheNodeToDropTheOldSubscription— after an overflow restore the node no longer lists the old id, holds only the restored one, and the client keeps receiving.TestClientRedial_KeepsTheConnectionAndItsSiblingSubscriptions— the connection stays in the pool and healthy, and the sibling subscription on it keeps receiving.TestRetireConnectionOnUpstreamError_LeavesAClientThatRedialed— the classification on its own, so a change at the call site cannot pass by never reaching it.TestRestoreAfterADroppedSocket_DoesNotCallADeadNode— the other half of the same call.main, verified withgo test -overlayagainst a copy of the manager with only the fixed behaviours reverted.upstreamErrSourceseam the listener documents for exactly this. It cannot be produced from the node side: rpcclient mints the error internally, and the MAG-4037 trigger — a call whose deadline has already passed — fails the redial on that same expired deadline, so no reconnect lands. Worth knowing before anyone tries to reproduce MAG-4037 that way.go test -count=1onprotocol/rpcsmartrouter(251s),protocol/chainlibandprotocol/chainlib/chainproxy/rpcclient, on the committed tree: all ok.go build -buildvcs=false ./...andgo veton the committed tree.CI runs no unit tests on this repo, so the local runs above are the only test evidence.
🤖 Generated with Claude Code