Skip to content

fix(rpcsmartrouter): tell the node to drop the subscription a restore replaced, and keep a redialed connection (MAG-4396) - #591

Merged
nimrod-teich merged 10 commits into
mainfrom
fix/MAG-4396-overflow-restore-unsubscribe
Nov 3, 2026
Merged

nimrod-teich merged 10 commits into
mainfrom
fix/MAG-4396-overflow-restore-unsubscribe

Conversation

@avitenzer

Copy link
Copy Markdown
Contributor

Description

The two leftovers PR #519's review left open: the overflow-restore leak, and the decision about what an rpcclient redial should do to a pooled connection.

Jira ticket: MAG-4396

Stacked on #590 (MAG-4278), which is itself stacked on #589 (MAG-4345); all three edit the same file. Review the top commit; merge after them.

1. The overflow restore never told the node to drop the old subscription

When a subscription's forward queue overflows, rpcclient ends it on a connection that stays up and the restore re-subscribes on that same live connection. Nothing told the node to drop the old one: ClientSubscription.run discards the unsubscribe-the-server flag forward returns, so rpcclient sends nothing, and handleUpstreamDisconnect used the old upstream id only for logging and for dropping its id mappings. The node kept publishing the replaced subscription on the same socket, rpcclient read those frames and dropped them, and a subscription that overflows repeatedly left one more behind each time.

The restore now tells the node, through the same teardown value the other paths use. Where the restore followed a lost socket the node lost the subscription with it, and teardownUpstreamSubscription already skips a dead connection — so one call is right on both paths, and TestRestoreAfterADroppedSocket_DoesNotCallADeadNode pins that it does not redial into an outage just to say goodbye (MAG-4242).

One thing deliberately left out, and it bit during this change. The ticket's alternative was to also end the local ClientSubscription. Doing that closes its Err channel, and a listener still parked there reads the close as a nil error, ends, and runs its deferred cleanup — tearing down the subscription the restore has just installed. TestSolanaRouterIDSurvivesUpstreamReconnect caught it: the client's reply channel came back closed. On the real restore path the listener that handed off has already returned, so nothing is parked there, but the margin is one scheduling decision wide and the only thing it buys is dropping rpcclient's handler entry for a replaced subscription, which closing the connection does anyway. If we want that entry released sooner it needs a path that does not touch the subscription's Err.

2. The decision: errClientReconnected leaves the connection alone

PR #519 made every subscription error except a queue overflow or a node error retire the pooled connection, so the pool prunes and closes it. errClientReconnected is one of those errors, and it is the one case where the client has already recovered: rpcclient replaced the socket itself when a write failed while the old read loop was alive, and by the time the error reaches the listener it holds a new, working socket.

Retiring the connection there is the opposite of a recovery. It closes the client rpcclient just fixed, so the pool dials again for nothing, and every other subscription sharing the pooled connection is ended too — including one a restore has already re-created on the new socket, which is the case the ticket describes. That is the same harm TestUpstreamError_RetiresTheSharedConnectionOnlyOnASocketFailure was written to prevent; a client that redialed on its own is not a socket failure.

So: ErrClientReconnected joins the queue overflow as an error that leaves the connection alone. The restore still runs — it finds the pool healthy and re-subscribes over the socket rpcclient opened. A real socket failure still retires the connection. Say so if you read it differently; it is a one-line predicate either way.

errClientReconnected is exported as ErrClientReconnected, next to ErrSubscriptionQueueOverflow and for the same reason: it says something a caller cannot read anywhere else.

Test plan

  • TestOverflowRestore_TellsTheNodeToDropTheOldSubscription — after an overflow restore the node no longer lists the old id, holds only the restored one, and the client keeps receiving.
  • TestClientRedial_KeepsTheConnectionAndItsSiblingSubscriptions — the connection stays in the pool and healthy, and the sibling subscription on it keeps receiving.
  • TestRetireConnectionOnUpstreamError_LeavesAClientThatRedialed — the classification on its own, so a change at the call site cannot pass by never reaching it.
  • TestRestoreAfterADroppedSocket_DoesNotCallADeadNode — the other half of the same call.
  • The first three fail on main, verified with go test -overlay against a copy of the manager with only the fixed behaviours reverted.
  • The redial is driven through the upstreamErrSource seam the listener documents for exactly this. It cannot be produced from the node side: rpcclient mints the error internally, and the MAG-4037 trigger — a call whose deadline has already passed — fails the redial on that same expired deadline, so no reconnect lands. Worth knowing before anyone tries to reproduce MAG-4037 that way.
  • go test -count=1 on protocol/rpcsmartrouter (251s), protocol/chainlib and protocol/chainlib/chainproxy/rpcclient, on the committed tree: all ok.
  • go build -buildvcs=false ./... and go vet on the committed tree.

CI runs no unit tests on this repo, so the local runs above are the only test evidence.

🤖 Generated with Claude Code

avitenzer and others added 4 commits October 28, 2026 16:21
…ubscribe gets its own subscription (MAG-4345)

Unsubscribe decided "last client" under the lock, called the node off the
lock, and only then tore the subscription down. The entry stayed in the
active map while the node call was in flight, so a client arriving for the
same params joined it, a local step with no node call. The local cleanup
then saw the joiner's mapping and tore nothing down, but the node had
already dropped the subscription: the joiner got no pushes, and its
channel was never closed, so it waited until it disconnected.

The subscription now leaves the active map in the first critical section,
before the node is called. A subscription the router has told the node to
drop takes no new clients, and a client arriving meanwhile creates a
subscription of its own on the node. A restore that reaches its commit
point while the call is in flight finds nothing to commit onto and ends
the subscription it re-opened (the MAG-2547 discard), so the
restored-since-read branch goes. A node call that fails no longer returns
early with the client still attached: the client asked to leave and the
subscription is already retired, so it ends locally and the error goes
back with the reply. A client that was not the last when its call started
but is by the time the lock is retaken now tells the node, which that
path never did.

teardownSubscriptionLocked splits into retireSubscriptionLocked (the map
entry, identity-guarded, and the counter) and
endUpstreamSubscriptionLocked (the upstream half), so Unsubscribe can run
the two halves apart.

Tests: TestWS_JoinerDuringLastClientUnsubscribeGetsItsOwnSubscription,
from Anna Rayer's attached repro (the arriving client subscribes in the
background, because it now needs the node the test holds), and
TestWS_LastClientUnsubscribeFailingUpstreamStillEndsTheSubscription (the
node applies the unsubscribe but never answers). Both fail on main.
TestUnsubscribeDuringRestore_TellsTheNodeTheRestoredID now asserts the
outcome rather than the path: neither id is left on the node.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Took 20 minutes
…pec declares (MAG-4278 item 1)

The three router-initiated teardown paths — the last client disconnecting,
a failed restore's cleanup, and the discard of a subscription a restore
re-opened onto an entry that had gone — derived the unsubscribe method
from the subscribe method by string surgery whose fallback is the literal
"unsubscribe". Only Tendermint has that method, so on Solana and every
Substrate chain the node rejected the call, the failure was traced, and
the node kept publishing into the pooled connection: the MAG-3366 leak
shape. The same paths named the subscription with the decimal string the
router canonicalises to internally, which Solana also rejects. The
explicit-unsubscribe path was always right, because the client names the
method.

The method is now resolved from the spec once, when the subscription is
created, and stored on it. The spec gives no field linking a SUBSCRIBE
directive to its UNSUBSCRIBE one, so the pairing is recovered from the
names a collection declares, and only ever returns a name it declares:

  1. the subscribe name with its first subscribe/Subscribe token replaced
     by unsubscribe/Unsubscribe, when the collection declares that name;
  2. for the names rule 1 cannot reach, namespace pairing: within one
     namespace an unclaimed subscribe pairs with an unclaimed
     unsubscribe when each is the only one of its kind there. Uniqueness
     is required in both directions, so an ambiguous namespace resolves
     to nothing rather than to a guess.

Measured over the whole lava-specs catalog (317 subscribe apis in 57
collections): rule 1 resolves 242, rule 2 resolves 63, 12 resolve to
nothing. Nine of those are gRPC or REST subscriptions this manager never
serves; the rest are Enjin's two archive methods, which share a namespace
with one unclaimed unsubscribe between them, and Filecoin.ChainNotify,
for which the FVM spec declares no unsubscribe at all. For those the node
is told nothing: a call the spec says it does not answer cannot stop it
publishing, and a warning naming the method is logged when the
subscription is created.

The id now goes in the chain's own shape on every path, through one
renderer shared with the explicit path. One upstreamSubscriptionTeardown
value carries what the node must be told, snapshotted under the lock, so
the three paths cannot pick different methods or id shapes from each
other again.

getUnsubscribeMethod becomes derivedUnsubscribeMethod and reports failure
instead of answering a name it cannot reach with the bare literal. It is
reached only for a collection that declares no unsubscribe at all.

Tests: TestWS_TeardownOnSolanaSendsTheSpecUnsubscribeMethod, from the
repro Anna Rayer attached to the ticket, strengthened to assert the method
and the unquoted id the node was actually asked for rather than only the
absence of "unsubscribe" — which a router that sends nothing also
satisfies; TestWS_ClientDisconnectOnSubstrateSendsTheSpecUnsubscribeMethod
for the commonest shape, a client closing its socket on chainHead_v1_follow;
TestWS_TeardownTellsTheNodeNothingWhenTheSpecDeclaresNoUnsubscribe;
TestWS_ExplicitUnsubscribeStillSendsTheClientsMethod; and
TestResolveTeardownUnsubscribeMethod over the directive shapes the catalog
declares. All four end-to-end tests fail on main.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Took 1 minute
…unt its pool slot on pick (MAG-4278 item 11)

Two gaps between the WebSocket subscription manager and its gRPC
counterpart, which has carried both since MAG-4177.

A restore was unbounded. Each one spawns a fresh listener, and that
listener's own failure spawns the next restore, so a node that accepts a
subscribe and ends it immediately was re-subscribed for the life of the
process: one round trip per cycle, on every subscription it does this to,
with nothing counting the cycles. A subscription now carries the gRPC
manager's accounting — restoresWithoutProgress against
maxRestoresWithoutProgress, refilled by a restore that stayed up for
restoreStableAfter — and is given up once it is spent. Uptime rather than
a delivered push is the measure of progress, so a quiet subscription does
not spend its budget on the reconnects a load balancer's connection age
limit causes.

A subscription's pool slot was counted after the node answered the
subscribe. In that window the connection carries no counted subscription,
and maybeScaleDown closes a connection past minConnections that carries
none, so the subscription being created on it died on a socket the pool
had just closed. The pool gains ReserveConnectionForSubscription, which
counts the slot under the same lock as the pick, and every failure after
the reservation releases it: a failed subscribe, a reply that cannot be
built, and a failed re-subscribe on the restore path. GetConnection and
GetConnectionForSubscription keep their behaviour for callers that do not
open a subscription.

Tests: TestRestoreBudget_GivesUpOnceTheBudgetIsSpent and
TestSubscribeReservesItsPoolSlot fail on main.
TestRestoreBudget_RefillsAfterASubscriptionStaysUp and
TestSubscribeReleasesItsPoolSlotWhenTheSubscribeFails are guards for the
refill and the release rather than repros, and pass either way. The mock
node gained rejectSubscribes, which answers a subscribe with a JSON-RPC
error.

Still open on the ticket's item 11, and not in this change: the WS
manager closes a given-up subscription's client channel with no status,
where the gRPC manager ends the client's stream with a reason. Giving
WebSocket clients one needs a new SubscriptionEndError reason and a
close-code decision in the ingress, which is client-visible behaviour
rather than a drift fix.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Took 2 minutes
… replaced, and keep a redialed connection (MAG-4396)

The two leftovers PR #519's review left open.

A restore that runs on a live connection never told the node to drop the
subscription it replaced. rpcclient ends one subscription on a connection
that stays up when its forward queue overflows, and
ClientSubscription.run discards the unsubscribe-the-server flag forward
returns, so rpcclient sends nothing; handleUpstreamDisconnect used the
old upstream id only for logging and for dropping its id mappings. So the
node kept publishing the replaced subscription on the same socket,
rpcclient read those frames and dropped them, and a subscription that
overflows repeatedly left one more of them behind each time. The restore
now tells the node, through the same teardown the other paths use. Where
the restore followed a lost socket the node lost the subscription with
it, and teardownUpstreamSubscription skips a dead connection, so one call
is right on both paths.

The local rpcclient subscription is deliberately not unsubscribed with
it. Unsubscribe closes its Err channel, and a listener still parked there
reads the close as a nil error, ends, and runs its deferred cleanup —
tearing down the subscription the restore just installed. The listener
that handed off has already returned, so nothing is parked there on the
restore path, but the margin is one scheduling decision wide and the only
thing it buys is dropping rpcclient's handler entry for a replaced
subscription, which closing the connection does anyway.

The decision the ticket asks for: errClientReconnected leaves the pooled
connection alone. PR #519 made every subscription error except a queue
overflow or a node error retire the connection, and this is the one case
where the client has already recovered — rpcclient replaced the socket
itself and holds a working one. Retiring it there closes that client, so
the pool dials again for nothing and every other subscription sharing the
connection ends too, including one a restore has just re-created on the
new socket. A real socket failure still retires it.

errClientReconnected becomes ErrClientReconnected: the fact that the
connection is healthy is not readable from anywhere else, and a caller
that treats every subscription error as a dead socket has no way to tell.

Tests: TestOverflowRestore_TellsTheNodeToDropTheOldSubscription,
TestClientRedial_KeepsTheConnectionAndItsSiblingSubscriptions and
TestRetireConnectionOnUpstreamError_LeavesAClientThatRedialed fail on
main. TestRestoreAfterADroppedSocket_DoesNotCallADeadNode guards the
other half of the same call. The redial is driven through the
upstreamErrSource seam the listener documents for it: rpcclient mints the
error internally, and the MAG-4037 trigger, a call whose deadline has
already passed, fails the redial on that same expired deadline, so no
reconnect lands.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
AnnaR-prog and others added 5 commits November 3, 2026 13:02
…ly after a queue overflow (MAG-4396)

The restore told the node to drop the subscription it replaced after every
restore on a healthy connection. Only a queue overflow leaves that
subscription on a live socket. When rpcclient redials on its own
(ErrClientReconnected), it closes the old socket and the connection is now
kept, so the goodbye went out over the NEW socket. On Tendermint the
subscription id is the query, so `unsubscribe {query}` ended the
subscription the restore had just opened: every client of every Tendermint
subscription on that connection was left on a subscription the node no
longer publishes, with its channel open. On other chains the goodbye named
an id the new socket does not know, or, on a node that numbers its
subscriptions per socket, a sibling the redial had just restored.

The restore now receives the error that ended the subscription and says
goodbye only after a queue overflow, and never when the old id names the
restored subscription on the same connection.

TestTendermintRedial_KeepsTheRestoredSubscriptionOnTheNode and
TestClientRedial_RestoresEverySubscriptionOnTheKeptConnection drive a real
redial (calls past their deadline, then a live call) and fail on b784643.
TestReplacedStillOnNode pins the decision.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… that ended under it (MAG-4345)

A client that is not the last one releases the lock between Unsubscribe's
two critical sections. If the subscription ends there by another path —
the node ends it with an error, or a restore gives it up — the listener's
cleanup releases every client, stops counting the subscription and tells
the node. Unsubscribe then retook the lock, found no clients left, and tore
the subscription down again: totalSubscriptions and the pooled
connection's count each went to -1 and the node was told twice. On main
the mapper's last-client answer, false once the cleanup had removed every
mapping, kept it from doing so; the retire/end split dropped that guard.

The teardown now runs only while the subscription is still the live entry
under its params. beforeUnsubscribeRelease is a test hook, nil in
production, in the shape of beforeRestoreCommit.

TestWS_UnsubscribeDoesNotTearDownASubscriptionThatEndedUnderIt fails
without the guard (totalSubscriptions -1, pool count -1, two
eth_unsubscribe) and passes on main's manager with only the hook added.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…when the client attaches (MAG-4345)

The watcher a subscribe starts looked its subscription up when the
goroutine first ran, to read the client's detached signal. Unsubscribe now
takes the subscription out of the map in its first step, so a client that
unsubscribed before its watcher was scheduled left the watcher with a nil
channel: it stayed parked until the socket closed, one goroutine per
subscribe and unsubscribe on a long-lived socket. On main the entry stayed
mapped for the node's round trip, which hid the race.

The signal is now read where the client is attached — by the creator while
it still owns the subscription, by a joiner under the manager lock — and
handed to the watcher.

TestUnsubscribeThenDisconnect_LeavesNothingBehind ("the disconnect watcher
must not stay parked once the client unsubscribed") failed 6 of 20 runs
without this and passes 30 of 30 with it; it passes 20 of 20 on main.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ol slot while the node answers (MAG-4278 item 11)

TestSubscribeReservesItsPoolSlot drives the pool's
ReserveConnectionForSubscription on its own. The manager calls it in two
places, StartSubscription and the restore, and with both of them reverted to
GetConnection plus an IncrementSubscriptions after the node's answer every
test in the package still passed, so nothing pinned the fix the window was
about.

TestSubscriptionCountsItsPoolSlotWhileTheNodeAnswers holds the node's answer
to a client's subscribe, then to the restore's re-subscribe after the socket
drops, and requires the pool to count the subscription in both windows. It
fails with either call site reverted.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… restore to commit (MAG-4278 item 11)

TestRestoreBudget_RefillsAfterASubscriptionStaysUp waited for
`upstreamID != "" && upstreamCalls("eth_subscribe") == 2`. The old upstream
id is still set during the restore, so the wait ended as soon as the node
had read the re-subscribe, before it answered and before the restore
committed. The test then returned, and its cleanup cancelled the
subscription's context with the re-subscribe possibly still in flight:
the window in which rpcclient's Subscribe can deadlock its dispatch loop
(an unbuffered reply channel the cancelled waiter no longer reads), after
which manager.Close never returns.

It now waits for the upstream id to change, which is the restore's commit.
The budget tests still fail with the budget removed.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

@AnnaR-prog AnnaR-prog left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at b784643 (the top commit). I tested it merged with current main b875f86, and drove a real rpcclient redial instead of the listener seam.

591-1 (Medium): after a redial, the restore's goodbye ends the restored Tendermint subscription.

  • With ErrClientReconnected now leaving the connection alone, the restore re-subscribes over the socket rpcclient just opened, on the same UpstreamWSConnection.
  • The goodbye for the replaced subscription then goes out through that same client, so over the new socket. The old subscription died with the old socket.
  • A Tendermint subscription's id is its query, so unsubscribe {"query": …} ends the subscription the restore just opened. Every Tendermint subscription on that connection goes silent, and its clients stay on it with their channels open.
  • With a CometBFT-shaped mock and a real redial, the node saw subscribe Q, status, subscribe Q, unsubscribe Q and held no subscription afterwards. It fails 10 of 10 on this head and passes on #590.
  • On EVM the goodbye names an id the new socket does not know. On a node that numbers subscriptions per socket, it could name a sibling the redial just restored.
  • Only a queue overflow leaves the old subscription on a live socket. I pushed a fix:
    • the listener hands the error to handleUpstreamDisconnect;
    • replacedStillOnNode says goodbye only after an overflow, and never when the old id names the restored subscription on the same connection;
    • tests: TestTendermintRedial_KeepsTheRestoredSubscriptionOnTheNode, TestClientRedial_RestoresEverySubscriptionOnTheKeptConnection and TestReplacedStillOnNode. The first two fail on this head.

591-2 (Low): the redial can be forced, and the redial test should run a real one.

  • A call whose deadline has passed fails its write and its own redial. The next call with a live deadline redials, and every subscription on the old socket ends with ErrClientReconnected.
  • TestClientRedial_KeepsTheConnectionAndItsSiblingSubscriptions feeds the error through the seam while the old socket and subscription stay alive, which a real redial cannot produce. It checks only the sibling, not that the redialed subscription is restored and keeps delivering, which is MAG-4396's Done-when.
  • The new EVM test does both with a real redial. I'm also correcting the endSubscriptionWith comment.

Verified:

  • The overflow goodbye and the ErrClientReconnected predicate are both pinned: the PR's tests fail with either one reverted.
  • A dropped socket still sends nothing.
  • The pool counts balance when the restore reuses the same connection.
  • The whole protocol/rpcsmartrouter package passes merged with main, with the fixes for #589 and #590 included.

// pushes in the meantime. The pool notify goes after the node call, so the connection is
// still live while it is used.
go func() {
dwsm.teardownUpstreamSubscription(oldUpstream)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After a redial the connection is kept, and its client now writes on the new socket, so this goodbye travels over a socket the old subscription never lived on. For Tendermint the old id is the query, the same as the restored one, so the node ends the subscription this restore just opened (591-1). Only after ErrSubscriptionQueueOverflow is the old subscription still on a live socket.


// endSubscriptionWith drives the real listener's error branch with err, through the
// upstreamErrSource seam listenForUpstreamMessages takes for exactly this. A client that
// redialed cannot be produced from the node side: rpcclient mints ErrClientReconnected

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A redial can be produced: a call past its deadline fails its write and its own redial, and the next live call redials. The seam leaves the old socket and subscription alive, which a real redial does not (591-2).

The parent PR squash-merged into main; this keeps this PR's own commits on top of main.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

@AnnaR-prog AnnaR-prog left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving at a872516: main de08be4 (with #589 and #590) plus this PR's own commits and the review fix. The rpcsmartrouter tests pass on this exact tree.

@nimrod-teich
nimrod-teich merged commit 04811cf into main Nov 3, 2026
@nimrod-teich
nimrod-teich deleted the fix/MAG-4396-overflow-restore-unsubscribe branch November 3, 2026 18:33
AnnaR-prog added a commit that referenced this pull request Nov 3, 2026
Resolves this PR against #608, #609 and the WebSocket stack (#590, #591, #612, #614):
- protocol/metrics: #609's positional vectors in this PR's register-as-built manager; the two
  mapped_labels wrapper files go, and so does their dupl rule in .golangci.yml.
- direct_rpc_connection.go: #608's h.do, called with common.JSONContentType.
- direct_ws_subscription_manager.go: the stack's unsubscribeParams, comparing the API interface
  with spectypes.APIInterfaceTendermintRPC.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants