Skip to content

fix(rpcsmartrouter): join a WebSocket subscription that became active between the two checks (MAG-2546) - #508

Merged
AnnaR-prog merged 3 commits into
mainfrom
fix/MAG-2546-ws-subscription-counter
Oct 14, 2026
Merged

AnnaR-prog merged 3 commits into
mainfrom
fix/MAG-2546-ws-subscription-counter

Conversation

@avitenzer

Copy link
Copy Markdown
Contributor

Description

Jira ticket: MAG-2546

Summary

MAG-2546 reports totalSubscriptions drifting upward until the global limit refuses every new eth_subscribe (reject mode) or warns forever (warn mode). It names the cleanup ticker's cancelled-context branch, which once deleted a subscription without decrementing. That branch has been routed through dropSubscriptionLocked since #386 and decrements correctly, but nothing pinned it, and the same counter still drifted through a second path, which this PR closes:

  • StartSubscription looks for an active subscription and registers a pending one under two separate acquisitions of the manager lock. A creator that stored its subscription and cleared its pending entry in that gap left the second caller seeing neither, so it became a second creator: a second upstream subscription for the same params, the first map entry overwritten and orphaned (its listener goroutine and node-side subscription living on unowned), and totalSubscriptions incremented twice for an entry that is dropped once.
  • checkAndAddPendingSubscription now re-checks the active map under the same lock and hands such a caller an already-settled success, so it takes the join path StartSubscription already has for a true broadcast. No caller changes.

Test plan

  • New direct_ws_subscription_counter_test.go:
    • TestDirectWS_CleanupOfCancelledSubscriptionDecrementsTotal — the check the ticket asks for: a subscription whose context ended is reaped by performCleanup and the counter returns to 0. Passes on main today (fixed by fix: stop the router heap growing for the life of the process #386); kept so it stays fixed.
    • TestDirectWS_PendingCheckJoinsASubscriptionStoredSinceTheActiveCheck — the gap above, driven step by step: active check misses, creator stores and clears pending, pending check must return a settled success and register nothing, the join then shares the one subscription. Fails on main (go test -overlay with main's direct_ws_subscription_manager.go: require.True(found) at line 104), passes here.
  • go test -race on both new tests: pass.
  • go test -count=1 ./protocol/rpcsmartrouter/ (162 s): pass.
  • go test -count=1 -race ./protocol/rpcsmartrouter/: every test passes except one pre-existing race unrelated to this change — a command test writing the global log level (health_cmd.go:118 → lavalog.go:297) while an rpcclient goroutine left by an earlier test reads it (lavalog.go:486). It reproduces on an untouched origin/main (there it landed on TestRealCommand_RespCacheFromYAML), so it is not introduced here; the failing test passes when run alone.
  • golangci-lint run --new-from-rev=origin/main ./protocol/rpcsmartrouter/... (fresh cache): 0 issues.

Found by the 2026-10-12 code audit (MAG-4242 item 9 is the gRPC manager's version of the same gap; it stays there).

🤖 Generated with Claude Code

@avitenzer
avitenzer requested a review from a team as a code owner October 12, 2026 12:41
… between the two checks (MAG-2546)

StartSubscription looks for an active subscription and registers a pending one under
two separate acquisitions of the manager lock. A creator that stored its subscription
and cleared its pending entry in that gap left the second caller seeing neither, so it
became a second creator: a second upstream subscription for the same params, the first
map entry overwritten and orphaned (its listener goroutine and node-side subscription
living on unowned), and totalSubscriptions incremented twice for an entry that is
dropped once. The pending check now re-checks the active map under the same lock and
hands such a caller a settled success so it takes the join path.

The cleanup-ticker branch the ticket names (a cancelled context deleting without
decrementing) was already routed through dropSubscriptionLocked by #386; the new test
pins that so it stays fixed.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Took 47 minutes

@AnnaR-prog AnnaR-prog left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at 3b13631 and tested on current main + this PR.

The fix is correct. With the active map checked again under the same lock, a caller now does one of three things: sees the active entry, waits on a live creator, or becomes the only creator. I checked the hand-off in four cases:

  • the found subscription is torn down before the join;
  • it is restoring;
  • its upstream is already gone;
  • the caller would be handed its own subscription.

In every case the joiner either joins a live subscription or falls back to creating one. No client or pending entry is left behind, and totalSubscriptions stays balanced.

Reproduced on main end to end. I ran 256 identical concurrent eth_subscribes per round:

  • main: 10 of 750 rounds opened a second upstream subscription. Each one left totalSubscriptions above 0 after every client had left, and a subscription live on the node.
  • this PR: 0 of 600 rounds.
  • with #509: 0 of 300 rounds.

The new pending-check test fails on main's manager at line 104 and passes here, also with -race -count=3.

One finding (Low, test gap), inline. I'm pushing a test for it.

A bug already on main, not caused or widened here. This is a follow-up, not a fix for this PR.

  • Trigger: a client joins while the last client's eth_unsubscribe is waiting on the node.
  • Why: Unsubscribe decides "last client" from its snapshot and calls the node off-lock. RemoveMapping then sees the joiner's mapping and skips the teardown.
  • Result: the joiner attaches to a subscription the node has already dropped. It gets no pushes, and its channel never closes.
  • Reproduction: a probe that stalls the node's writes. It fails the same way on main and here.
  • Fix shape: take the entry out of the map in Unsubscribe's first critical section when the snapshot says last client, or release the clients that joined in the meantime.

// global counter must come back down with it. That branch once deleted without
// decrementing, so under client churn the counter drifted up until the global limit
// refused every new eth_subscribe (reject mode) or warned forever (warn mode).
func TestDirectWS_CleanupOfCancelledSubscriptionDecrementsTotal(t *testing.T) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Low · test gap. Production can't reach the branch this test pins:

  • subCtx derives from context.WithoutCancel.
  • Its cancel runs only in the teardown goroutine, in cleanupSubscription and in Close, each after or together with the delete.
  • So a subscription that is still registered never has a cancelled context.

The branch production can reach is the janitor's no-clients branch (the second branch in the ticket's 2026-09-09 comment, MAG-3527).

  • When it runs: connectedClients is empty while the id-mapper says "not last". For example, a restore re-registers router ids it collected before a client left.
  • Today: it routes through dropSubscriptionLocked correctly, but no test pins it. With the branch put back to its pre-fix: stop the router heap growing for the life of the process #386 shape, the whole suite still passes.

I'm adding TestDirectWS_CleanupOfAClientlessSubscriptionDecrementsTotalAndTellsTheNode. It checks that the counter reaches 0, the node gets one eth_unsubscribe, and the pooled count reaches 0. It fails against the pre-#386 shape.

…oduction reaches (MAG-2546)

The cancelled-context branch the ticket names cannot be reached in production: subCtx
derives from context.WithoutCancel, and its cancel runs only together with the delete
(the teardown goroutine, cleanupSubscription, Close). The no-clients branch can run
(connectedClients empty while the id mapper says "not last"), and nothing pinned it:
put back to its pre-#386 shape, the whole suite still passed. The new test checks that
it decrements the counter, tells the node, and uncounts the pooled connection.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@AnnaR-prog

Copy link
Copy Markdown
Contributor

Review fix pushed: 4ac802c

Finding Status
508-1 (Low, test gap): nothing pinned the janitor's no-clients branch, the one production can reach Fixed in 4ac802c. Adds TestDirectWS_CleanupOfAClientlessSubscriptionDecrementsTotalAndTellsTheNode, which checks the counter, the eth_unsubscribe to the node and the pooled count. It fails with the branch in its pre-#386 shape.
A join while the last client's eth_unsubscribe is in flight lands on a dropped subscription Not changed here. It is already on main, and this PR neither causes nor widens it. Follow-up.

Verified:

  • go test -race -run 'TestDirectWS_' ./protocol/rpcsmartrouter/ passes on this branch.
  • On current main with all nine PRs in this batch and every review fix applied, go build/go vet are clean and the full go test ./... passes.

@AnnaR-prog AnnaR-prog left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving at 8525da1, which is main 803d4852 plus this PR and the review fix 4ac802c.

Findings: in my review above. The test gap is closed in 4ac802c, and the join-during-unsubscribe bug that is already on main is a follow-up.

Verified on this exact tree:

  • go build is clean, and so is go vet on rpcsmartrouter.
  • The WebSocket subscription tests pass under -race.

@AnnaR-prog
AnnaR-prog merged commit d966dbe into main Oct 14, 2026
@AnnaR-prog
AnnaR-prog deleted the fix/MAG-2546-ws-subscription-counter branch October 14, 2026 09:34
AnnaR-prog added a commit that referenced this pull request Oct 17, 2026
#508 (MAG-2546) and #509 (MAG-2547) landed on main as squashes with their
review fixes (d966dbe, 30c7545), so this merge takes their final
versions from main and keeps this PR's own three commits (1edee77,
22b9fe6, cfb4738); the tree equals `git rebase --onto origin/main
21fd439`. #511 (MAG-4244) removed routeMessageToClients' unused
hashedParams parameter, which 1edee77's dropSlowClient call needed: the
call reads it off the subscription instead (activeSub.hashedParams).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants