Skip to content

fix(rpcsmartrouter): tell a non-EVM node the unsubscribe its spec declares, and bound a subscription's restores (MAG-4278) - #590

Merged
AnnaR-prog merged 8 commits into
mainfrom
fix/MAG-4278-ws-teardown-unsubscribe
Nov 3, 2026
Merged

AnnaR-prog merged 8 commits into
mainfrom
fix/MAG-4278-ws-teardown-unsubscribe

Conversation

@avitenzer

Copy link
Copy Markdown
Contributor

Description

MAG-4278 items 1 and 11, the two the ticket deferred until PRs #508, #509, #511 and #519 had merged. One commit each.

Jira ticket: MAG-4278

Stacked on #589 (MAG-4345), which edits the same file. Review the top two commits; merge after it.

Item 1: the teardown sends a method the node does not have

The three router-initiated teardown paths — the last client disconnecting, a failed restore's cleanup, and the discard of a subscription a restore re-opened onto an entry that had gone — derived the unsubscribe method from the subscribe method by string surgery whose fallback is the literal unsubscribe. Only Tendermint has that method, so on Solana and every Substrate chain the node rejected the call, the failure was traced, and the node kept publishing into the pooled connection. The same paths named the subscription with the decimal string the router canonicalises to internally, which Solana also rejects.

The ticket's suggested fix shape does not work as written, and this is the finding worth reviewing. It asks to "resolve the unsubscribe method at subscribe time", but a SUBSCRIBE parse directive carries nothing except its tag and its api name. The spec has no field linking it to its UNSUBSCRIBE directive, and the directive order does not pair them either: 28 of the 57 collections in the catalog are not subscribe-then-unsubscribe alternating, and in every Substrate spec the two groups are each sorted by name, which misaligns them.

So the pairing is recovered from the names a collection declares, by two rules that only ever return a name it declares:

  1. the subscribe name with its first subscribe/Subscribe token replaced by unsubscribe/Unsubscribe;
  2. for the names rule 1 cannot reach, namespace pairing: within one namespace (the name up to its last _ or .) an unclaimed subscribe pairs with an unclaimed unsubscribe when each is the only one of its kind there. Uniqueness is required in both directions, so an ambiguous namespace resolves to nothing rather than to a guess.

Measured over every spec in lava-specs at b813946 — 317 subscribe apis in 57 collections:

Resolution Subscribe apis
rule 1, the declared token 242
rule 2, namespace pairing 63
nothing 12

Rule 2's 63 are the three Substrate verbs no string rule reaches, on 21 chains each: author_submitAndWatchExtrinsic → author_unwatchExtrinsic, chainHead_v1_follow → chainHead_v1_unfollow, transactionWatch_v1_submitAndWatch → transactionWatch_v1_unwatch. All three are hand-checked in the unit test.

Nine of the twelve that resolve to nothing are gRPC or REST subscriptions, which this manager never serves. The other three are Enjin's two archive_v1_* methods, which share a namespace with one unclaimed unsubscribe between them, and Filecoin.ChainNotify, for which the FVM spec declares no unsubscribe at all. For those the node is told nothing: a call the spec says it does not answer cannot stop it publishing, and a warning naming the chain and method is logged when the subscription is created. Adding the two Enjin pairs and Filecoin's is a lava-specs follow-up.

The id now goes in the chain's own shape on every path, through one renderer shared with the explicit path. One upstreamSubscriptionTeardown value carries what the node must be told, snapshotted under the lock, so the three paths cannot drift from each other again.

Item 11: an unbounded restore, and a pool slot counted too late

A restore was unbounded. Each one spawns a fresh listener whose own failure spawns the next, so a node that accepts a subscribe and ends it immediately was re-subscribed for the life of the process. A subscription now carries the gRPC manager's accounting and is given up once it is spent; a restore that stayed up for restoreStableAfter refills it, so a quiet subscription does not spend its budget on the reconnects a load balancer's age limit causes.

A subscription's pool slot was counted after the node answered the subscribe. In that window maybeScaleDown can close the connection as unused, and the subscription being created on it dies on a socket the pool just closed. The pool gains ReserveConnectionForSubscription, which counts the slot under the same lock as the pick, and every failure after the reservation releases it.

Not in this change, and still open on item 11: the WS manager closes a given-up subscription's client channel with no status, where the gRPC manager ends the client's stream with a reason. Giving WebSocket clients one needs a new SubscriptionEndError reason and a close-code decision in the ingress — client-visible behaviour rather than a drift fix.

Test plan

  • TestWS_TeardownOnSolanaSendsTheSpecUnsubscribeMethod, from the repro Anna Rayer attached to the ticket. Strengthened to assert the method and the unquoted id the node was actually asked for: her assertion that unsubscribe is absent is also satisfied by a router that sends nothing at all, which is what the first draft of this fix did.
  • TestWS_ClientDisconnectOnSubstrateSendsTheSpecUnsubscribeMethod — the commonest shape, a client closing its socket on chainHead_v1_follow.
  • TestWS_TeardownTellsTheNodeNothingWhenTheSpecDeclaresNoUnsubscribe.
  • TestWS_ExplicitUnsubscribeStillSendsTheClientsMethod — the path that was always right.
  • TestResolveTeardownUnsubscribeMethod over the directive shapes the catalog declares, including the two ambiguous ones it must refuse.
  • TestRestoreBudget_GivesUpOnceTheBudgetIsSpent, TestSubscribeReservesItsPoolSlot.
  • All four end-to-end item 1 tests and both of those item 11 tests fail on main, verified with go test -overlay against a copy of the manager with only the fixed behaviours reverted. TestRestoreBudget_RefillsAfterASubscriptionStaysUp and TestSubscribeReleasesItsPoolSlotWhenTheSubscribeFails are guards for the refill and the release rather than repros, and pass either way.
  • go test -count=1 ./protocol/rpcsmartrouter/ on the committed tree: ok, 245s.
  • The subscription tests under -race on the committed tree: ok.
  • go build -buildvcs=false ./... and go vet on the committed tree.

CI runs no unit tests on this repo, so the local runs above are the only test evidence.

🤖 Generated with Claude Code

…ubscribe gets its own subscription (MAG-4345)

Unsubscribe decided "last client" under the lock, called the node off the
lock, and only then tore the subscription down. The entry stayed in the
active map while the node call was in flight, so a client arriving for the
same params joined it, a local step with no node call. The local cleanup
then saw the joiner's mapping and tore nothing down, but the node had
already dropped the subscription: the joiner got no pushes, and its
channel was never closed, so it waited until it disconnected.

The subscription now leaves the active map in the first critical section,
before the node is called. A subscription the router has told the node to
drop takes no new clients, and a client arriving meanwhile creates a
subscription of its own on the node. A restore that reaches its commit
point while the call is in flight finds nothing to commit onto and ends
the subscription it re-opened (the MAG-2547 discard), so the
restored-since-read branch goes. A node call that fails no longer returns
early with the client still attached: the client asked to leave and the
subscription is already retired, so it ends locally and the error goes
back with the reply. A client that was not the last when its call started
but is by the time the lock is retaken now tells the node, which that
path never did.

teardownSubscriptionLocked splits into retireSubscriptionLocked (the map
entry, identity-guarded, and the counter) and
endUpstreamSubscriptionLocked (the upstream half), so Unsubscribe can run
the two halves apart.

Tests: TestWS_JoinerDuringLastClientUnsubscribeGetsItsOwnSubscription,
from Anna Rayer's attached repro (the arriving client subscribes in the
background, because it now needs the node the test holds), and
TestWS_LastClientUnsubscribeFailingUpstreamStillEndsTheSubscription (the
node applies the unsubscribe but never answers). Both fail on main.
TestUnsubscribeDuringRestore_TellsTheNodeTheRestoredID now asserts the
outcome rather than the path: neither id is left on the node.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Took 20 minutes
…pec declares (MAG-4278 item 1)

The three router-initiated teardown paths — the last client disconnecting,
a failed restore's cleanup, and the discard of a subscription a restore
re-opened onto an entry that had gone — derived the unsubscribe method
from the subscribe method by string surgery whose fallback is the literal
"unsubscribe". Only Tendermint has that method, so on Solana and every
Substrate chain the node rejected the call, the failure was traced, and
the node kept publishing into the pooled connection: the MAG-3366 leak
shape. The same paths named the subscription with the decimal string the
router canonicalises to internally, which Solana also rejects. The
explicit-unsubscribe path was always right, because the client names the
method.

The method is now resolved from the spec once, when the subscription is
created, and stored on it. The spec gives no field linking a SUBSCRIBE
directive to its UNSUBSCRIBE one, so the pairing is recovered from the
names a collection declares, and only ever returns a name it declares:

  1. the subscribe name with its first subscribe/Subscribe token replaced
     by unsubscribe/Unsubscribe, when the collection declares that name;
  2. for the names rule 1 cannot reach, namespace pairing: within one
     namespace an unclaimed subscribe pairs with an unclaimed
     unsubscribe when each is the only one of its kind there. Uniqueness
     is required in both directions, so an ambiguous namespace resolves
     to nothing rather than to a guess.

Measured over the whole lava-specs catalog (317 subscribe apis in 57
collections): rule 1 resolves 242, rule 2 resolves 63, 12 resolve to
nothing. Nine of those are gRPC or REST subscriptions this manager never
serves; the rest are Enjin's two archive methods, which share a namespace
with one unclaimed unsubscribe between them, and Filecoin.ChainNotify,
for which the FVM spec declares no unsubscribe at all. For those the node
is told nothing: a call the spec says it does not answer cannot stop it
publishing, and a warning naming the method is logged when the
subscription is created.

The id now goes in the chain's own shape on every path, through one
renderer shared with the explicit path. One upstreamSubscriptionTeardown
value carries what the node must be told, snapshotted under the lock, so
the three paths cannot pick different methods or id shapes from each
other again.

getUnsubscribeMethod becomes derivedUnsubscribeMethod and reports failure
instead of answering a name it cannot reach with the bare literal. It is
reached only for a collection that declares no unsubscribe at all.

Tests: TestWS_TeardownOnSolanaSendsTheSpecUnsubscribeMethod, from the
repro Anna Rayer attached to the ticket, strengthened to assert the method
and the unquoted id the node was actually asked for rather than only the
absence of "unsubscribe" — which a router that sends nothing also
satisfies; TestWS_ClientDisconnectOnSubstrateSendsTheSpecUnsubscribeMethod
for the commonest shape, a client closing its socket on chainHead_v1_follow;
TestWS_TeardownTellsTheNodeNothingWhenTheSpecDeclaresNoUnsubscribe;
TestWS_ExplicitUnsubscribeStillSendsTheClientsMethod; and
TestResolveTeardownUnsubscribeMethod over the directive shapes the catalog
declares. All four end-to-end tests fail on main.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Took 1 minute
…unt its pool slot on pick (MAG-4278 item 11)

Two gaps between the WebSocket subscription manager and its gRPC
counterpart, which has carried both since MAG-4177.

A restore was unbounded. Each one spawns a fresh listener, and that
listener's own failure spawns the next restore, so a node that accepts a
subscribe and ends it immediately was re-subscribed for the life of the
process: one round trip per cycle, on every subscription it does this to,
with nothing counting the cycles. A subscription now carries the gRPC
manager's accounting — restoresWithoutProgress against
maxRestoresWithoutProgress, refilled by a restore that stayed up for
restoreStableAfter — and is given up once it is spent. Uptime rather than
a delivered push is the measure of progress, so a quiet subscription does
not spend its budget on the reconnects a load balancer's connection age
limit causes.

A subscription's pool slot was counted after the node answered the
subscribe. In that window the connection carries no counted subscription,
and maybeScaleDown closes a connection past minConnections that carries
none, so the subscription being created on it died on a socket the pool
had just closed. The pool gains ReserveConnectionForSubscription, which
counts the slot under the same lock as the pick, and every failure after
the reservation releases it: a failed subscribe, a reply that cannot be
built, and a failed re-subscribe on the restore path. GetConnection and
GetConnectionForSubscription keep their behaviour for callers that do not
open a subscription.

Tests: TestRestoreBudget_GivesUpOnceTheBudgetIsSpent and
TestSubscribeReservesItsPoolSlot fail on main.
TestRestoreBudget_RefillsAfterASubscriptionStaysUp and
TestSubscribeReleasesItsPoolSlotWhenTheSubscribeFails are guards for the
refill and the release rather than repros, and pass either way. The mock
node gained rejectSubscribes, which answers a subscribe with a JSON-RPC
error.

Still open on the ticket's item 11, and not in this change: the WS
manager closes a given-up subscription's client channel with no status,
where the gRPC manager ends the client's stream with a reason. Giving
WebSocket clients one needs a new SubscriptionEndError reason and a
close-code decision in the ingress, which is client-visible behaviour
rather than a drift fix.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Took 2 minutes
AnnaR-prog and others added 4 commits November 3, 2026 17:13
…ol slot while the node answers (MAG-4278 item 11)

TestSubscribeReservesItsPoolSlot drives the pool's
ReserveConnectionForSubscription on its own. The manager calls it in two
places, StartSubscription and the restore, and with both of them reverted to
GetConnection plus an IncrementSubscriptions after the node's answer every
test in the package still passed, so nothing pinned the fix the window was
about.

TestSubscriptionCountsItsPoolSlotWhileTheNodeAnswers holds the node's answer
to a client's subscribe, then to the restore's re-subscribe after the socket
drops, and requires the pool to count the subscription in both windows. It
fails with either call site reverted.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… that ended under it (MAG-4345)

A client that is not the last one releases the lock between Unsubscribe's
two critical sections. If the subscription ends there by another path —
the node ends it with an error, or a restore gives it up — the listener's
cleanup releases every client, stops counting the subscription and tells
the node. Unsubscribe then retook the lock, found no clients left, and tore
the subscription down again: totalSubscriptions and the pooled
connection's count each went to -1 and the node was told twice. On main
the mapper's last-client answer, false once the cleanup had removed every
mapping, kept it from doing so; the retire/end split dropped that guard.

The teardown now runs only while the subscription is still the live entry
under its params. beforeUnsubscribeRelease is a test hook, nil in
production, in the shape of beforeRestoreCommit.

TestWS_UnsubscribeDoesNotTearDownASubscriptionThatEndedUnderIt fails
without the guard (totalSubscriptions -1, pool count -1, two
eth_unsubscribe) and passes on main's manager with only the hook added.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…when the client attaches (MAG-4345)

The watcher a subscribe starts looked its subscription up when the
goroutine first ran, to read the client's detached signal. Unsubscribe now
takes the subscription out of the map in its first step, so a client that
unsubscribed before its watcher was scheduled left the watcher with a nil
channel: it stayed parked until the socket closed, one goroutine per
subscribe and unsubscribe on a long-lived socket. On main the entry stayed
mapped for the node's round trip, which hid the race.

The signal is now read where the client is attached — by the creator while
it still owns the subscription, by a joiner under the manager lock — and
handed to the watcher.

TestUnsubscribeThenDisconnect_LeavesNothingBehind ("the disconnect watcher
must not stay parked once the client unsubscribed") failed 6 of 20 runs
without this and passes 30 of 30 with it; it passes 20 of 20 on main.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… restore to commit (MAG-4278 item 11)

TestRestoreBudget_RefillsAfterASubscriptionStaysUp waited for
`upstreamID != "" && upstreamCalls("eth_subscribe") == 2`. The old upstream
id is still set during the restore, so the wait ended as soon as the node
had read the re-subscribe, before it answered and before the restore
committed. The test then returned, and its cleanup cancelled the
subscription's context with the re-subscribe possibly still in flight:
the window in which rpcclient's Subscribe can deadlock its dispatch loop
(an unbuffered reply channel the cancelled waiter no longer reads), after
which manager.Close never returns.

It now waits for the upstream id to change, which is the restore's commit.
The budget tests still fail with the budget removed.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

@AnnaR-prog AnnaR-prog left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at e7281dc (the top two commits). I tested it merged with current main b875f86. I also reran the catalog measurement over lava-specs b813946 with the repo's own spec loader.

Item 1 is correct.

  • The measurement reproduces exactly: 57 collections, 317 subscribe apis; 242 by rule 1, 63 by rule 2, 12 with nothing.
  • Expanded per chain, which is what the router sees, the derived fallback never produces an undeclared name.
  • eth_subscribe resolves to eth_unsubscribe in all 147 EVM collections, so EVM is unchanged. Tendermint, Solana and every Substrate pair resolve as described.
  • The three teardown tests fail with only the teardown reverted. TestWS_ExplicitUnsubscribeStillSendsTheClientsMethod passes either way, which is right for a guard, but the description lists it among the four that fail on main.

590-1 (Low): no test pins the manager's reserve-on-pick.

  • TestSubscribeReservesItsPoolSlot drives the pool method only. I reverted both manager call sites (StartSubscription and the restore) to GetConnection plus IncrementSubscriptions after the node answers. Every test still passed.
  • I pushed TestSubscriptionCountsItsPoolSlotWhileTheNodeAnswers. It holds the node's answer to a client's subscribe, then to the restore's re-subscribe, and requires the pool to count the subscription in both windows. It fails with either call site reverted.

590-2 (Low, your call): a queue overflow counts against the restore budget.

  • The listener sends ErrSubscriptionQueueOverflow into handleUpstreamDisconnect like a lost socket.
  • A busy subscription that overflows its 20000-push queue six times in a row, each within 30 s of the last restore, is now given up: every client's channel closes with no status. Before, each overflow was restored.
  • An overflow means the router fell behind, not that the node keeps ending the subscription. Count only restores after a lost socket, or keep this and say so in the description?

590-3 (Low): the "no unsubscribe" warning is logged on every subscribe request.

  • The method is resolved and the warning logged at the top of StartSubscription, before the rate limit and the join paths. On FVM and ENJ every subscribe request logs it, joins included, not once when the subscription is created.
  • Moving the two lines to the creation path, after the pending loop, would match the description.

590-4 (Low): TestRestoreBudget_RefillsAfterASubscriptionStaysUp can end before the restore commits.

  • upstreamID != "" is always true, because the old id stays set, so the wait ends when the node has read the re-subscribe. The test then returns, and its cleanup cancels the subscription with the re-subscribe possibly still in flight.
  • If that cancel lands while rpcclient delivers the answer, Subscribe deadlocks the client's dispatch loop. This is a pre-existing rpcclient bug, not this PR's, and manager.Close then hangs the test binary. I hit that exact hang once in a stress run with a test of my own shaped like this one.
  • I pushed a version that waits for the upstream id to change, which happens at the commit.

Verified:

  • Every failure after a reservation releases it, and the pool counts balance at the end of each test.
  • The budget tests fail without the budget.
  • The new tests pass under -race.
  • This PR carries #589's commit, so it also needs #589's two fixes. I pushed them here as cherry-picks, so this PR's own CI run includes them.

// Resolved once, here, while the spec collection this message matched is in hand: the
// teardown paths have no client message to read the method off, and deriving it there is
// what sent non-EVM nodes a method they do not have (MAG-4278 item 1).
unsubscribeMethod, hasUnsubscribeMethod := resolveTeardownUnsubscribeMethod(protocolMessage.GetApiCollection(), subscribeMethod)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Resolved, and warned about, before the rate limit and the join paths, so FVM and ENJ log this on every subscribe request, joins included (590-3). Moving it to just before selectEndpoint makes it once per created subscription, as the description says.

if last := activeSub.lastRestoredAt.Load(); last != 0 && time.Since(time.Unix(0, last)) >= restoreStableAfter {
activeSub.restoresWithoutProgress.Store(0)
}
if attempt := activeSub.restoresWithoutProgress.Add(1); attempt > maxRestoresWithoutProgress {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A queue overflow reaches here too (the listener passes every non-node error), so a subscription that keeps overflowing is given up after five restores within 30 s windows (590-2). Intended?

The parent PR squash-merged into main; this keeps this PR's own commits on top of main.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

@AnnaR-prog AnnaR-prog left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving at 93c2148: main 322a967 (with #589) plus this PR's own commits and the two review fixes. The rpcsmartrouter tests pass on this exact tree.

@AnnaR-prog
AnnaR-prog merged commit de08be4 into main Nov 3, 2026
@AnnaR-prog
AnnaR-prog deleted the fix/MAG-4278-ws-teardown-unsubscribe branch November 3, 2026 18:26
AnnaR-prog added a commit that referenced this pull request Nov 3, 2026
Resolves this PR against #608, #609 and the WebSocket stack (#590, #591, #612, #614):
- protocol/metrics: #609's positional vectors in this PR's register-as-built manager; the two
  mapped_labels wrapper files go, and so does their dupl rule in .golangci.yml.
- direct_rpc_connection.go: #608's h.do, called with common.JSONContentType.
- direct_ws_subscription_manager.go: the stack's unsubscribeParams, comparing the API interface
  with spectypes.APIInterfaceTendermintRPC.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants