Repository navigation
fix(rpcsmartrouter): let a WebSocket teardown win over a restore in flight (MAG-2547) - #509
Conversation
…light (MAG-2547) When an upstream socket drops, handleUpstreamDisconnect re-subscribes on a fresh connection and then rewrites the subscription's upstreamSubscription, upstreamID and upstreamConnection under the manager lock. A client that disconnected during that window ran teardownSubscriptionLocked, whose goroutine read those same fields off the lock — a data race — and whichever half of the restore it saw, the other half leaked: the restored upstream subscription stayed live on the node with nobody reading it, and the new connection kept counting it, so the pool could never scale it down. The teardown now snapshots everything its goroutine needs while it still holds the lock, and the restore commits only if its subscription is still the one in the map; otherwise it ends the upstream subscription it just created, node side too, and uncounts it. Cleanup is identity-guarded for the same reason: the map is keyed by params alone, so a listener ending or a restore failing late could otherwise destroy a newer subscription registered under the same key. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Took 2 minutes
AnnaR-prog
left a comment
There was a problem hiding this comment.
Reviewed at bd6cbbb and tested on current main + this PR, also together with #508.
What this PR changes is correct.
- One lock for the shared fields. Every read and write of
upstreamSubscription,upstreamID,upstreamConnectionandmessagesChannow happens underdwsm.lock. - No network call under it. The discard unlocks before
Unsubscribe,teardownUpstreamSubscriptionandNotifySubscriptionRemoved. - No new lock nesting.
- Counts balance when the restore loses, whether the pool hands back the same connection object or a new one.
- No listener leaks.
- Every
cleanupSubscriptioncaller passes its own subscription. - The test hook is safe.
beforeRestoreCommitis nil in production. Tests set it before the goroutines that read it start.
The negative control reproduces as the body states: the failure message, plus the race on upstreamID under -race. All 75 WS subscription tests pass with -race -count=3 with #508 and #509 together.
Findings. I'm pushing fixes or tests for 509-1 to 509-3. 509-4 needs a design call.
509-2 (Low, exists on main): a successful restore on the same connection object leaves that connection counting one extra subscription forever.
- Cause: the restore increments the connection it re-subscribed on. The commit uncounts the old one only when
oldConnection != nil && oldConnection != conn(:1676). The pool never marks a connection unhealthy (MAG-4242 item 6), soGetConnectionreturns the same object. - Result: each restore adds one extra count per subscription. After one blip, a connection with N subscriptions counts 2N. After they all leave it still counts N, so the pool never scales it down. That is the ticket's "count is left wrong" symptom, on the commit path.
- Probe:
sameConnectionObject=true pooledAfterRestore=2 pooledAfterTeardown=1, 3/3 on main and here. - Fix: uncount the old connection whenever there is one. With the fix the probe gives 1 and 0.
- #519 only partly covers this. Its retire-on-socket-error handles socket-level errors. A restore after a queue overflow or a node error push still reuses the same object.
Nit: if upstreamSub != nil (:1800) is redundant, because ClientSubscription.Unsubscribe is nil-safe.
|
|
||
| // Update the ID mapping, message channel, and connection bookkeeping | ||
| dwsm.lock.Lock() | ||
| if dwsm.activeSubscriptions[hashedParams] != activeSub { |
There was a problem hiding this comment.
Medium · exists on main, same race family as this PR. An explicit eth_unsubscribe that overlaps the restore commit still leaves the restored subscription live on the node. So "Nothing is left to leak" doesn't hold on this path:
UnsubscribereadsconnandupstreamIDunder the lock (:1002-1003), unlocks, and tells the node to drop the old id.- Meanwhile the restore commits. The entry is still mapped, so this check passes, and the id, the connection and the client's mapping move to the new subscription.
Unsubscribere-locks.RemoveMappingfinds the mapping, so this is the last client, and it tears down withtellNode=false(:1063). Nobody tells the node about the restored id.
This is the natural ordering in production.
- The pool hands back the same connection, so the unsubscribe queues on the same rpcclient behind the restore's
Subscribe. - A real node answers the stale id with a JSON-RPC error, and
CallContextreturns that as a message witherr == nil.
Probe: hold the restore at the test hook, stall the node's writes, start Unsubscribe, then release. It prints methods=[eth_subscribe eth_subscribe eth_unsubscribe] nodeHasNew=true total=0, 3/3 here, on main and with #508.
Fix (pushing):
- In
Unsubscribe's second critical section, tell the node when a restore committed since the read (activeSub.upstreamID != upstreamID || activeSub.upstreamConnection != conn). - Add
TestUnsubscribeDuringRestore_TellsTheNodeTheRestoredID.
| if !found { | ||
| return | ||
| } | ||
| if activeSub != owner { |
There was a problem hiding this comment.
Low · test gap. This guard fixes a real main bug, but no test pins it: with the guard removed, all 73 WS subscription tests on this branch still pass, the new one included.
The main bug it prevents:
- A torn-down subscription's listener exits only after the teardown's node call returns, which can take up to
upstreamUnsubscribeTimeout(10 s). - Its deferred cleanup by key then destroys any successor registered under the same params in the meantime: it closes the successor client's channel, unsubscribes it on the node, and decrements the counter to 0.
Tests I'm adding:
TestListenerEndingLate_LeavesASuccessorAlone: fails on main, and here without the guard.TestRestoreAfterTeardown_LeavesASuccessorAlone: fails on main.
| utils.LogAttr("newUpstreamID", newUpstreamID), | ||
| ) | ||
| newUpstreamSub.Unsubscribe() | ||
| dwsm.teardownUpstreamSubscription(conn, newUpstreamID, activeSub.subscribeMethod) |
There was a problem hiding this comment.
Low · exists on main. The MAG-3527 closing comment noted this, but no ticket was filed.
- Cause:
teardownUpstreamSubscriptionderives the method withgetUnsubscribeMethodand always sends the id as a string. That is right only foreth_subscribe/subscribewith string ids. - Result on Solana: this discard sends
unsubscribe, a method the node doesn't have, so the discarded subscription stays live there. With the Solana mock, the node saw[accountSubscribe accountSubscribe unsubscribe unsubscribe]. - Why not here: the fix means storing the resolved unsubscribe method and the id encoding at subscribe time. That is a design call, so I'm not changing it in this PR; it is worth its own ticket.
…nsubscribe overlapped (MAG-2547) Unsubscribe reads the upstream id and connection under the lock, unlocks, and asks the node to drop that id. If a restore commits meanwhile, the entry is still mapped, so the restore's identity check passes and the client's mapping moves to the new id; the unsubscribe then tears down with tellNode=false and the restored subscription stays live on the node with nobody reading it. Tell the node when a restore committed since the read. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…on the same one (MAG-2547) The restore counts the subscription on the connection it re-subscribed on, but uncounted the old connection only when it was a different object. The pool never marks a pooled connection unhealthy, so it hands back the same one and every restore left one extra count per subscription: the connection could never be scaled down. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…r the same key alone (MAG-2547) With the identity guard in cleanupSubscription removed, every WebSocket subscription test still passed. Two tests that fail on main: a listener that ends after its teardown's node call returns, and a restore that loses to a teardown, both while a new subscription is registered under the same params. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Review fixes pushed: 87b1a8d, d09e446, 56d842e
Verified:
#519: 87b1a8d and d09e446 touch |
AnnaR-prog
left a comment
There was a problem hiding this comment.
Approving at 35f0e1a, which is main d966dbe5 (with #508) plus this PR and the review fixes 87b1a8d, d09e446 and 56d842e.
Findings: in my review above.
- Fixed: 509-1, 509-2 and 509-3.
- Follow-up: 509-4, the unsubscribe method on non-EVM chains, needs a design call.
Verified on this exact tree:
go buildis clean, and so isgo veton rpcsmartrouter.- The WebSocket subscription, restore, unsubscribe, successor and teardown tests pass under
-race, including with #508's join.
#508 (MAG-2546) and #509 (MAG-2547) landed on main as squashes with their review fixes (d966dbe, 30c7545), so this merge takes their final versions from main and keeps this PR's own three commits (1edee77, 22b9fe6, cfb4738); the tree equals `git rebase --onto origin/main 21fd439`. #511 (MAG-4244) removed routeMessageToClients' unused hashedParams parameter, which 1edee77's dropSlowClient call needed: the call reads it off the subscription instead (activeSub.hashedParams). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…nection that still answers (MAG-4242) TestUnsubscribeDuringRestore_TellsTheNodeTheRestoredID (#509) started its restore by dropping the socket. Now the listener retires that connection and the pool closes it once the reconnect lands, so the unsubscribe finds no client to call, skips the node, and the test's wait for the node to receive it timed out every run. The race it pins (an unsubscribe on the node while a restore commits a new upstream id) now only happens when the restore runs on a connection that stays healthy: rpcclient giving up on the subscription's queue. The test triggers exactly that. While the manager lock holds the listener, the node pushes more than rpcclient queues (20000), the forwarder ends with ErrSubscriptionQueueOverflow, and the restore re-subscribes on the same connection. The rest is unchanged, and it still fails when the teardown is not told about the restored id. The overflow also joins the retirement table: the connection stays pooled and healthy, and the subscription is restored on it. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Description
Jira ticket: MAG-2547
Summary
When an upstream WebSocket drops,
handleUpstreamDisconnectre-subscribes on a fresh connection and then rewrites the subscription'supstreamSubscription,upstreamIDandupstreamConnectionunder the manager lock. A client that disconnected during that window ranteardownSubscriptionLocked, whose goroutine read those same fields off the lock. That is the data race the ticket describes, and whichever half of the restore the goroutine saw, the other half leaked: the restored upstream subscription stayed live on the node with nobody reading it, and the new connection kept counting it, so the pool could never scale it down.teardownSubscriptionLockedanddropSubscriptionLockedread the upstream subscription, id, connection and method while they still holddwsm.lockand hand the values to their goroutine;teardownUpstreamSubscriptiontakes those values instead of the subscription. A nil upstream subscription is tolerated.handleUpstreamDisconnectchecks under the lock thatactiveSubscriptions[hashedParams]is still its subscription. If a teardown won in between, it ends the upstream subscription it just created (locally and on the node), uncounts it on the connection, and returns. Nothing is left to leak.cleanupSubscriptionnow takes the owner and leaves the entry alone when it belongs to a newer subscription under the same key — the map is keyed by params alone, so a listener ending or a restore failing late could otherwise destroy a subscription a new client had just registered. The listener's deferred cleanup and the restore's three failure paths pass their owner.The ticket's "Expected" was for client teardown to wait for a restore in flight. This does the complementary thing: teardown proceeds at once (it cannot wait under the lock, and a client's disconnect should not be held for a reconnect window) and the restore yields to it at its commit point. The outcome the ticket asks for holds: no leaked upstream subscription, correct connection counts, and every read and write of the upstream fields under the same lock.
Test plan
TestRestoreAfterTeardown_LeavesNoUpstreamSubscription(direct_ws_subscription_restore_test.go), end to end against the mock upstream: subscribe, drop the server-side socket so the router re-subscribes, hold the restore just before its commit (a nil-outside-testsbeforeRestoreCommitseam), disconnect the last client, release — then the node must hold no live subscription, no pooled connection may count one, the counter and active map must be empty, and no upstream listener may survive. The mock server gaineddropConnections()/liveSubscriptions()and the harnesspooledSubscriptionCount().main'sdirect_ws_subscription_manager.gooverlaid (go test -overlay), the test fails on "the restored upstream subscription was left running on the node with nobody reading it", and-racereports the ticket's race: the restore's writes at:1638/:1639(upstreamSubscription,upstreamID) against the teardown goroutine's reads at:1804(Unsubscribe) and:1816(teardownUpstreamSubscription).-racein 0.02 s.go test -count=1 -race ./protocol/rpcsmartrouter/(175 s): pass, 0 races.golangci-lint run --new-from-rev=origin/main ./protocol/rpcsmartrouter/...(fresh cache): 0 issues.Note: in this test
ReconnectWithBackoffreturns at once because the pool never marks a connection unhealthy (MAG-4242 item 6) andrpcclientredials insideSubscribe; that behaviour is unchanged here. Found by the 2026-10-12 code audit.🤖 Generated with Claude Code