Skip to content

fix(rpcsmartrouter): let a WebSocket teardown win over a restore in flight (MAG-2547) - #509

Merged
AnnaR-prog merged 5 commits into
mainfrom
fix/MAG-2547-ws-reconnect-vs-client-teardown
Oct 14, 2026
Merged

AnnaR-prog merged 5 commits into
mainfrom
fix/MAG-2547-ws-reconnect-vs-client-teardown

Conversation

@avitenzer

Copy link
Copy Markdown
Contributor

Description

Jira ticket: MAG-2547

Summary

When an upstream WebSocket drops, handleUpstreamDisconnect re-subscribes on a fresh connection and then rewrites the subscription's upstreamSubscription, upstreamID and upstreamConnection under the manager lock. A client that disconnected during that window ran teardownSubscriptionLocked, whose goroutine read those same fields off the lock. That is the data race the ticket describes, and whichever half of the restore the goroutine saw, the other half leaked: the restored upstream subscription stayed live on the node with nobody reading it, and the new connection kept counting it, so the pool could never scale it down.

  • Teardown snapshots under the lock. teardownSubscriptionLocked and dropSubscriptionLocked read the upstream subscription, id, connection and method while they still hold dwsm.lock and hand the values to their goroutine; teardownUpstreamSubscription takes those values instead of the subscription. A nil upstream subscription is tolerated.
  • The restore commits only if it still owns the map entry. After the node answers, handleUpstreamDisconnect checks under the lock that activeSubscriptions[hashedParams] is still its subscription. If a teardown won in between, it ends the upstream subscription it just created (locally and on the node), uncounts it on the connection, and returns. Nothing is left to leak.
  • Cleanup is identity-guarded. cleanupSubscription now takes the owner and leaves the entry alone when it belongs to a newer subscription under the same key — the map is keyed by params alone, so a listener ending or a restore failing late could otherwise destroy a subscription a new client had just registered. The listener's deferred cleanup and the restore's three failure paths pass their owner.

The ticket's "Expected" was for client teardown to wait for a restore in flight. This does the complementary thing: teardown proceeds at once (it cannot wait under the lock, and a client's disconnect should not be held for a reconnect window) and the restore yields to it at its commit point. The outcome the ticket asks for holds: no leaked upstream subscription, correct connection counts, and every read and write of the upstream fields under the same lock.

Test plan

  • New TestRestoreAfterTeardown_LeavesNoUpstreamSubscription (direct_ws_subscription_restore_test.go), end to end against the mock upstream: subscribe, drop the server-side socket so the router re-subscribes, hold the restore just before its commit (a nil-outside-tests beforeRestoreCommit seam), disconnect the last client, release — then the node must hold no live subscription, no pooled connection may count one, the counter and active map must be empty, and no upstream listener may survive. The mock server gained dropConnections()/liveSubscriptions() and the harness pooledSubscriptionCount().
  • Fails before the fix: with a seam-only copy of main's direct_ws_subscription_manager.go overlaid (go test -overlay), the test fails on "the restored upstream subscription was left running on the node with nobody reading it", and -race reports the ticket's race: the restore's writes at :1638/:1639 (upstreamSubscription, upstreamID) against the teardown goroutine's reads at :1804 (Unsubscribe) and :1816 (teardownUpstreamSubscription).
  • Passes here under -race in 0.02 s.
  • go test -count=1 -race ./protocol/rpcsmartrouter/ (175 s): pass, 0 races.
  • golangci-lint run --new-from-rev=origin/main ./protocol/rpcsmartrouter/... (fresh cache): 0 issues.

Note: in this test ReconnectWithBackoff returns at once because the pool never marks a connection unhealthy (MAG-4242 item 6) and rpcclient redials inside Subscribe; that behaviour is unchanged here. Found by the 2026-10-12 code audit.

🤖 Generated with Claude Code

…light (MAG-2547)

When an upstream socket drops, handleUpstreamDisconnect re-subscribes on a fresh
connection and then rewrites the subscription's upstreamSubscription, upstreamID and
upstreamConnection under the manager lock. A client that disconnected during that window
ran teardownSubscriptionLocked, whose goroutine read those same fields off the lock — a
data race — and whichever half of the restore it saw, the other half leaked: the restored
upstream subscription stayed live on the node with nobody reading it, and the new
connection kept counting it, so the pool could never scale it down.

The teardown now snapshots everything its goroutine needs while it still holds the lock,
and the restore commits only if its subscription is still the one in the map; otherwise it
ends the upstream subscription it just created, node side too, and uncounts it. Cleanup
is identity-guarded for the same reason: the map is keyed by params alone, so a listener
ending or a restore failing late could otherwise destroy a newer subscription registered
under the same key.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Took 2 minutes

@AnnaR-prog AnnaR-prog left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at bd6cbbb and tested on current main + this PR, also together with #508.

What this PR changes is correct.

  • One lock for the shared fields. Every read and write of upstreamSubscription, upstreamID, upstreamConnection and messagesChan now happens under dwsm.lock.
  • No network call under it. The discard unlocks before Unsubscribe, teardownUpstreamSubscription and NotifySubscriptionRemoved.
  • No new lock nesting.
  • Counts balance when the restore loses, whether the pool hands back the same connection object or a new one.
  • No listener leaks.
  • Every cleanupSubscription caller passes its own subscription.
  • The test hook is safe. beforeRestoreCommit is nil in production. Tests set it before the goroutines that read it start.

The negative control reproduces as the body states: the failure message, plus the race on upstreamID under -race. All 75 WS subscription tests pass with -race -count=3 with #508 and #509 together.

Findings. I'm pushing fixes or tests for 509-1 to 509-3. 509-4 needs a design call.

509-2 (Low, exists on main): a successful restore on the same connection object leaves that connection counting one extra subscription forever.

  • Cause: the restore increments the connection it re-subscribed on. The commit uncounts the old one only when oldConnection != nil && oldConnection != conn (:1676). The pool never marks a connection unhealthy (MAG-4242 item 6), so GetConnection returns the same object.
  • Result: each restore adds one extra count per subscription. After one blip, a connection with N subscriptions counts 2N. After they all leave it still counts N, so the pool never scales it down. That is the ticket's "count is left wrong" symptom, on the commit path.
  • Probe: sameConnectionObject=true pooledAfterRestore=2 pooledAfterTeardown=1, 3/3 on main and here.
  • Fix: uncount the old connection whenever there is one. With the fix the probe gives 1 and 0.
  • #519 only partly covers this. Its retire-on-socket-error handles socket-level errors. A restore after a queue overflow or a node error push still reuses the same object.

Nit: if upstreamSub != nil (:1800) is redundant, because ClientSubscription.Unsubscribe is nil-safe.


// Update the ID mapping, message channel, and connection bookkeeping
dwsm.lock.Lock()
if dwsm.activeSubscriptions[hashedParams] != activeSub {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Medium · exists on main, same race family as this PR. An explicit eth_unsubscribe that overlaps the restore commit still leaves the restored subscription live on the node. So "Nothing is left to leak" doesn't hold on this path:

  1. Unsubscribe reads conn and upstreamID under the lock (:1002-1003), unlocks, and tells the node to drop the old id.
  2. Meanwhile the restore commits. The entry is still mapped, so this check passes, and the id, the connection and the client's mapping move to the new subscription.
  3. Unsubscribe re-locks. RemoveMapping finds the mapping, so this is the last client, and it tears down with tellNode=false (:1063). Nobody tells the node about the restored id.

This is the natural ordering in production.

  • The pool hands back the same connection, so the unsubscribe queues on the same rpcclient behind the restore's Subscribe.
  • A real node answers the stale id with a JSON-RPC error, and CallContext returns that as a message with err == nil.

Probe: hold the restore at the test hook, stall the node's writes, start Unsubscribe, then release. It prints methods=[eth_subscribe eth_subscribe eth_unsubscribe] nodeHasNew=true total=0, 3/3 here, on main and with #508.

Fix (pushing):

  • In Unsubscribe's second critical section, tell the node when a restore committed since the read (activeSub.upstreamID != upstreamID || activeSub.upstreamConnection != conn).
  • Add TestUnsubscribeDuringRestore_TellsTheNodeTheRestoredID.

if !found {
return
}
if activeSub != owner {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Low · test gap. This guard fixes a real main bug, but no test pins it: with the guard removed, all 73 WS subscription tests on this branch still pass, the new one included.

The main bug it prevents:

  • A torn-down subscription's listener exits only after the teardown's node call returns, which can take up to upstreamUnsubscribeTimeout (10 s).
  • Its deferred cleanup by key then destroys any successor registered under the same params in the meantime: it closes the successor client's channel, unsubscribes it on the node, and decrements the counter to 0.

Tests I'm adding:

  • TestListenerEndingLate_LeavesASuccessorAlone: fails on main, and here without the guard.
  • TestRestoreAfterTeardown_LeavesASuccessorAlone: fails on main.

utils.LogAttr("newUpstreamID", newUpstreamID),
)
newUpstreamSub.Unsubscribe()
dwsm.teardownUpstreamSubscription(conn, newUpstreamID, activeSub.subscribeMethod)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Low · exists on main. The MAG-3527 closing comment noted this, but no ticket was filed.

  • Cause: teardownUpstreamSubscription derives the method with getUnsubscribeMethod and always sends the id as a string. That is right only for eth_subscribe/subscribe with string ids.
  • Result on Solana: this discard sends unsubscribe, a method the node doesn't have, so the discarded subscription stays live there. With the Solana mock, the node saw [accountSubscribe accountSubscribe unsubscribe unsubscribe].
  • Why not here: the fix means storing the resolved unsubscribe method and the id encoding at subscribe time. That is a design call, so I'm not changing it in this PR; it is worth its own ticket.

AnnaR-prog and others added 3 commits October 14, 2026 11:53
…nsubscribe overlapped (MAG-2547)

Unsubscribe reads the upstream id and connection under the lock, unlocks, and asks the
node to drop that id. If a restore commits meanwhile, the entry is still mapped, so the
restore's identity check passes and the client's mapping moves to the new id; the
unsubscribe then tears down with tellNode=false and the restored subscription stays live
on the node with nobody reading it. Tell the node when a restore committed since the read.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…on the same one (MAG-2547)

The restore counts the subscription on the connection it re-subscribed on, but uncounted
the old connection only when it was a different object. The pool never marks a pooled
connection unhealthy, so it hands back the same one and every restore left one extra
count per subscription: the connection could never be scaled down.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…r the same key alone (MAG-2547)

With the identity guard in cleanupSubscription removed, every WebSocket subscription test
still passed. Two tests that fail on main: a listener that ends after its teardown's node
call returns, and a restore that loses to a teardown, both while a new subscription is
registered under the same params.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@AnnaR-prog

Copy link
Copy Markdown
Contributor

Review fixes pushed: 87b1a8d, d09e446, 56d842e

Finding Status
509-1 (Medium): an explicit eth_unsubscribe that overlaps the restore commit leaves the restored subscription on the node Fixed in 87b1a8d. Unsubscribe now tells the node when a restore committed since its read. Adds TestUnsubscribeDuringRestore_TellsTheNodeTheRestoredID, which fails on bd6cbbb.
509-2 (Low): a restore on the same connection object leaves one extra count Fixed in d09e446. The old connection is uncounted whenever there is one. Adds TestRestoreOnTheSameConnection_KeepsOneCount.
509-3 (Low): the identity guard is unpinned Fixed in 56d842e. Adds TestListenerEndingLate_LeavesASuccessorAlone, which fails on main and here without the guard, and TestRestoreAfterTeardown_LeavesASuccessorAlone, which fails on main.
509-4 (Low): the discard sends unsubscribe on non-EVM chains Not changed. It needs the resolved unsubscribe method and id encoding stored at subscribe time, which is a design call. Follow-up ticket.
Nit: redundant upstreamSub != nil Left as is.

Verified:

  • go test -race -run 'Restore|Unsubscribe|Successor|Teardown|DirectWS|Release' ./protocol/rpcsmartrouter/ passes on this branch.
  • On current main with all nine PRs in this batch and every review fix applied, the full go test ./... passes.

#519: 87b1a8d and d09e446 touch Unsubscribe and the restore commit. #519 (stacked on this branch) doesn't edit either, so it picks them up cleanly.

@AnnaR-prog AnnaR-prog left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving at 35f0e1a, which is main d966dbe5 (with #508) plus this PR and the review fixes 87b1a8d, d09e446 and 56d842e.

Findings: in my review above.

  • Fixed: 509-1, 509-2 and 509-3.
  • Follow-up: 509-4, the unsubscribe method on non-EVM chains, needs a design call.

Verified on this exact tree:

  • go build is clean, and so is go vet on rpcsmartrouter.
  • The WebSocket subscription, restore, unsubscribe, successor and teardown tests pass under -race, including with #508's join.

@AnnaR-prog
AnnaR-prog merged commit 30c7545 into main Oct 14, 2026
@AnnaR-prog
AnnaR-prog deleted the fix/MAG-2547-ws-reconnect-vs-client-teardown branch October 14, 2026 09:41
AnnaR-prog added a commit that referenced this pull request Oct 17, 2026
#508 (MAG-2546) and #509 (MAG-2547) landed on main as squashes with their
review fixes (d966dbe, 30c7545), so this merge takes their final
versions from main and keeps this PR's own three commits (1edee77,
22b9fe6, cfb4738); the tree equals `git rebase --onto origin/main
21fd439`. #511 (MAG-4244) removed routeMessageToClients' unused
hashedParams parameter, which 1edee77's dropSlowClient call needed: the
call reads it off the subscription instead (activeSub.hashedParams).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
AnnaR-prog added a commit that referenced this pull request Oct 17, 2026
…nection that still answers (MAG-4242)

TestUnsubscribeDuringRestore_TellsTheNodeTheRestoredID (#509) started its restore by
dropping the socket. Now the listener retires that connection and the pool closes it once
the reconnect lands, so the unsubscribe finds no client to call, skips the node, and the
test's wait for the node to receive it timed out every run.

The race it pins (an unsubscribe on the node while a restore commits a new upstream id) now
only happens when the restore runs on a connection that stays healthy: rpcclient giving up
on the subscription's queue. The test triggers exactly that. While the manager lock holds
the listener, the node pushes more than rpcclient queues (20000), the forwarder ends with
ErrSubscriptionQueueOverflow, and the restore re-subscribes on the same connection. The rest
is unchanged, and it still fails when the teardown is not told about the restored id.

The overflow also joins the retirement table: the connection stays pooled and healthy, and
the subscription is restored on it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants