Repository navigation
fix(rpcsmartrouter): join a WebSocket subscription that became active between the two checks (MAG-2546) - #508
Conversation
… between the two checks (MAG-2546) StartSubscription looks for an active subscription and registers a pending one under two separate acquisitions of the manager lock. A creator that stored its subscription and cleared its pending entry in that gap left the second caller seeing neither, so it became a second creator: a second upstream subscription for the same params, the first map entry overwritten and orphaned (its listener goroutine and node-side subscription living on unowned), and totalSubscriptions incremented twice for an entry that is dropped once. The pending check now re-checks the active map under the same lock and hands such a caller a settled success so it takes the join path. The cleanup-ticker branch the ticket names (a cancelled context deleting without decrementing) was already routed through dropSubscriptionLocked by #386; the new test pins that so it stays fixed. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Took 47 minutes
24e9bfc to
3b13631
Compare
AnnaR-prog
left a comment
There was a problem hiding this comment.
Reviewed at 3b13631 and tested on current main + this PR.
The fix is correct. With the active map checked again under the same lock, a caller now does one of three things: sees the active entry, waits on a live creator, or becomes the only creator. I checked the hand-off in four cases:
- the found subscription is torn down before the join;
- it is restoring;
- its upstream is already gone;
- the caller would be handed its own subscription.
In every case the joiner either joins a live subscription or falls back to creating one. No client or pending entry is left behind, and totalSubscriptions stays balanced.
Reproduced on main end to end. I ran 256 identical concurrent eth_subscribes per round:
- main: 10 of 750 rounds opened a second upstream subscription. Each one left
totalSubscriptionsabove 0 after every client had left, and a subscription live on the node. - this PR: 0 of 600 rounds.
- with #509: 0 of 300 rounds.
The new pending-check test fails on main's manager at line 104 and passes here, also with -race -count=3.
One finding (Low, test gap), inline. I'm pushing a test for it.
A bug already on main, not caused or widened here. This is a follow-up, not a fix for this PR.
- Trigger: a client joins while the last client's
eth_unsubscribeis waiting on the node. - Why:
Unsubscribedecides "last client" from its snapshot and calls the node off-lock.RemoveMappingthen sees the joiner's mapping and skips the teardown. - Result: the joiner attaches to a subscription the node has already dropped. It gets no pushes, and its channel never closes.
- Reproduction: a probe that stalls the node's writes. It fails the same way on main and here.
- Fix shape: take the entry out of the map in
Unsubscribe's first critical section when the snapshot says last client, or release the clients that joined in the meantime.
| // global counter must come back down with it. That branch once deleted without | ||
| // decrementing, so under client churn the counter drifted up until the global limit | ||
| // refused every new eth_subscribe (reject mode) or warned forever (warn mode). | ||
| func TestDirectWS_CleanupOfCancelledSubscriptionDecrementsTotal(t *testing.T) { |
There was a problem hiding this comment.
Low · test gap. Production can't reach the branch this test pins:
subCtxderives fromcontext.WithoutCancel.- Its
cancelruns only in the teardown goroutine, incleanupSubscriptionand inClose, each after or together with the delete. - So a subscription that is still registered never has a cancelled context.
The branch production can reach is the janitor's no-clients branch (the second branch in the ticket's 2026-09-09 comment, MAG-3527).
- When it runs:
connectedClientsis empty while the id-mapper says "not last". For example, a restore re-registers router ids it collected before a client left. - Today: it routes through
dropSubscriptionLockedcorrectly, but no test pins it. With the branch put back to its pre-fix: stop the router heap growing for the life of the process #386 shape, the whole suite still passes.
I'm adding TestDirectWS_CleanupOfAClientlessSubscriptionDecrementsTotalAndTellsTheNode. It checks that the counter reaches 0, the node gets one eth_unsubscribe, and the pooled count reaches 0. It fails against the pre-#386 shape.
…oduction reaches (MAG-2546) The cancelled-context branch the ticket names cannot be reached in production: subCtx derives from context.WithoutCancel, and its cancel runs only together with the delete (the teardown goroutine, cleanupSubscription, Close). The no-clients branch can run (connectedClients empty while the id mapper says "not last"), and nothing pinned it: put back to its pre-#386 shape, the whole suite still passed. The new test checks that it decrements the counter, tells the node, and uncounts the pooled connection. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Review fix pushed: 4ac802c
Verified:
|
AnnaR-prog
left a comment
There was a problem hiding this comment.
Approving at 8525da1, which is main 803d4852 plus this PR and the review fix 4ac802c.
Findings: in my review above. The test gap is closed in 4ac802c, and the join-during-unsubscribe bug that is already on main is a follow-up.
Verified on this exact tree:
go buildis clean, and so isgo veton rpcsmartrouter.- The WebSocket subscription tests pass under
-race.
#508 (MAG-2546) and #509 (MAG-2547) landed on main as squashes with their review fixes (d966dbe, 30c7545), so this merge takes their final versions from main and keeps this PR's own three commits (1edee77, 22b9fe6, cfb4738); the tree equals `git rebase --onto origin/main 21fd439`. #511 (MAG-4244) removed routeMessageToClients' unused hashedParams parameter, which 1edee77's dropSlowClient call needed: the call reads it off the subscription instead (activeSub.hashedParams). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Description
Jira ticket: MAG-2546
Summary
MAG-2546 reports
totalSubscriptionsdrifting upward until the global limit refuses every neweth_subscribe(reject mode) or warns forever (warn mode). It names the cleanup ticker's cancelled-context branch, which once deleted a subscription without decrementing. That branch has been routed throughdropSubscriptionLockedsince #386 and decrements correctly, but nothing pinned it, and the same counter still drifted through a second path, which this PR closes:StartSubscriptionlooks for an active subscription and registers a pending one under two separate acquisitions of the manager lock. A creator that stored its subscription and cleared its pending entry in that gap left the second caller seeing neither, so it became a second creator: a second upstream subscription for the same params, the first map entry overwritten and orphaned (its listener goroutine and node-side subscription living on unowned), andtotalSubscriptionsincremented twice for an entry that is dropped once.checkAndAddPendingSubscriptionnow re-checks the active map under the same lock and hands such a caller an already-settled success, so it takes the join pathStartSubscriptionalready has for a true broadcast. No caller changes.Test plan
direct_ws_subscription_counter_test.go:TestDirectWS_CleanupOfCancelledSubscriptionDecrementsTotal— the check the ticket asks for: a subscription whose context ended is reaped byperformCleanupand the counter returns to 0. Passes onmaintoday (fixed by fix: stop the router heap growing for the life of the process #386); kept so it stays fixed.TestDirectWS_PendingCheckJoinsASubscriptionStoredSinceTheActiveCheck— the gap above, driven step by step: active check misses, creator stores and clears pending, pending check must return a settled success and register nothing, the join then shares the one subscription. Fails onmain(go test -overlaywith main'sdirect_ws_subscription_manager.go:require.True(found)at line 104), passes here.go test -raceon both new tests: pass.go test -count=1 ./protocol/rpcsmartrouter/(162 s): pass.go test -count=1 -race ./protocol/rpcsmartrouter/: every test passes except one pre-existing race unrelated to this change — a command test writing the global log level (health_cmd.go:118→lavalog.go:297) while anrpcclientgoroutine left by an earlier test reads it (lavalog.go:486). It reproduces on an untouchedorigin/main(there it landed onTestRealCommand_RespCacheFromYAML), so it is not introduced here; the failing test passes when run alone.golangci-lint run --new-from-rev=origin/main ./protocol/rpcsmartrouter/...(fresh cache): 0 issues.Found by the 2026-10-12 code audit (MAG-4242 item 9 is the gRPC manager's version of the same gap; it stays there).
🤖 Generated with Claude Code