Repository navigation
fix(rpcsmartrouter): retire dead WebSocket connections, drop slow subscription clients, and share one gRPC stream per subscribe (MAG-4242) - #519
Conversation
…light (MAG-2547) When an upstream socket drops, handleUpstreamDisconnect re-subscribes on a fresh connection and then rewrites the subscription's upstreamSubscription, upstreamID and upstreamConnection under the manager lock. A client that disconnected during that window ran teardownSubscriptionLocked, whose goroutine read those same fields off the lock — a data race — and whichever half of the restore it saw, the other half leaked: the restored upstream subscription stayed live on the node with nobody reading it, and the new connection kept counting it, so the pool could never scale it down. The teardown now snapshots everything its goroutine needs while it still holds the lock, and the restore commits only if its subscription is still the one in the map; otherwise it ends the upstream subscription it just created, node side too, and uncounts it. Cleanup is identity-guarded for the same reason: the map is keyed by params alone, so a listener ending or a restore failing late could otherwise destroy a newer subscription registered under the same key. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Took 2 minutes
… between the two checks (MAG-2546) StartSubscription looks for an active subscription and registers a pending one under two separate acquisitions of the manager lock. A creator that stored its subscription and cleared its pending entry in that gap left the second caller seeing neither, so it became a second creator: a second upstream subscription for the same params, the first map entry overwritten and orphaned (its listener goroutine and node-side subscription living on unowned), and totalSubscriptions incremented twice for an entry that is dropped once. The pending check now re-checks the active map under the same lock and hands such a caller a settled success so it takes the join path. The cleanup-ticker branch the ticket names (a cancelled context deleting without decrementing) was already routed through dropSubscriptionLocked by #386; the new test pins that so it stays fixed. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Took 47 minutes
…nter' into fix/MAG-4242-subscriptions
…ent-teardown' into fix/MAG-4242-subscriptions
…g its pushes (MAG-4242) The per-client reply channel was unbuffered on WebSocket and the sender polled it ten times, one millisecond apart, then dropped the push with a trace log: a client whose previous frame took more than 10 ms to write lost pushes with no counter and no signal, and because the fan-out is serial every other client of the shared subscription paid the same stall. gRPC had a 100-deep buffer and the same drop after it. Both managers now give each client a buffer of subscriptionClientReplyBuffer pushes and send without waiting (SafeChannelSender.TrySend). A client that lets its buffer fill is dropped: its channel closes so its subscription visibly ends (a gRPC stream ends with Unavailable, "did not keep up"), the drop is counted in smartrouter_subscription_clients_dropped_total, and the other clients are never stalled. The WebSocket harness now closes its manager on cleanup, as a live subscription left behind keeps the pool busy. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Took 18 minutes
… restore reconnects with backoff (MAG-4242) Nothing ever called MarkUnhealthy on a pooled connection, so when the upstream socket dropped the pool still read it as healthy: ReconnectWithBackoff returned at once, GetConnection handed the dead socket back, the re-subscribe got rpcclient's single dial, and when that failed the subscription was cleaned up with every client released. The backoff ladder never ran, and the cleanup then redialled just to send an unsubscribe for a subscription that had died with the socket. The listener now retires the connection on a socket-level error (a subscription-level end — queue overflow, or the node answering that one subscription with an error — leaves the socket alone), the pool prunes connections marked dead instead of handing them out again or keeping them, and the upstream unsubscribe is skipped on a dead connection. The mock node refuses dials on demand and, like a real node, forgets the subscriptions of a socket that died. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… subscribes (MAG-4242) The gRPC manager allocated a pending-subscription map and never used it. Clients arriving with the same params while the first was dialling, reflecting and opening its stream each opened a stream of their own, and the last to finish overwrote the map entry: every earlier stream lived on upstream with a client that could never unsubscribe it. The WebSocket manager's pending broadcast is ported: the first client registers as the creator before it opens the stream, later ones wait for its outcome and join, and the creator settles the entry on every path. A subscription the join turned away (dead, still registered until its listener releases it) is never waited on or joined; it is replaced. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
@avitenzer heads-up for the rebase. #508, #509 and #511 are on main now, so this branch conflicts in 4 files. Here is what I found against main at 8ad0ff6, in a scratch tree; I haven't pushed anything to this branch.
|
AnnaR-prog
left a comment
There was a problem hiding this comment.
Reviewed this PR's three commits replayed onto current main (8ad0ff6), as in my earlier comment: git rebase --onto origin/main 21fd4392 plus the compile fix.
Items 6 and 9 are sound.
- Item 6: only a socket-level failure retires the shared pooled connection.
- I put two subscriptions on one connection and ended the first by each cause, using the real rpcclient, pool and manager.
- An explicit unsubscribe, a client disconnect, a node JSON-RPC error push and a queue overflow all leave the connection healthy, and the other subscription keeps receiving. Only a dropped socket retires it.
- All six subscription tests fail with main's managers and pool overlaid, and pass here.
- Item 9: I drove the pending entry by hand under
-race.- A creator that fails: the waiters retry and share one stream.
- A waiter whose context ends: it returns at once and leaves nothing behind.
- A dead subscription that is still registered: it's replaced, with one stream.
- The creator settles on every path, panics included. Your four-client test passes 30/30 under
-race.
519-1 (High): item 8 drops healthy clients that read every push, only slower than a burst arrives.
- The fan-out drops a client at the first push that finds its 100-deep channel full, without waiting.
- The router hands pushes over in about 13 µs each. A client that reads a little slower is 100 behind within a millisecond or two of a burst's start. An unfiltered
logssubscription gets hundreds of pushes per block, in one burst. - I measured this with the real manager and rpcclient and ~700-byte log frames:
- WebSocket, 20 µs per frame (50k frames/s): dropped 255 pushes into a 300-push burst.
- WebSocket, 50 µs per frame: dropped after 138 pushes.
- gRPC, 20–200 µs per frame: dropped after about 101–108 messages of a 300- or 1000-message burst.
- Main delivers every push in every case, because the unbuffered channel and
Send's 10 × 1 ms retries paced the fan-out to the client.
- Your test covers only a client that never reads.
- I'm pushing a bounded wait (
SafeChannelSender.SendWithin). When a client's buffer is full, the fan-out waits up to 10 ms for room before dropping it. That is main's per-push bound.- The manager-level sweeps are then clean: WebSocket 18/18, gRPC 8/8.
- A client that never reads is still dropped, after one 10 ms wait.
- Two new tests,
TestBurstyClientThatReadsIsNotDroppedandTestGRPCBurstyClientThatReadsIsNotDropped, fail with the wait set to 0.
- The wait does not cover socket-level stalls. Through the real WebSocket ingress on localhost:
- a 3000-push burst to a client reading at 100 µs per frame is still dropped with a 10 or 25 ms grace, and kept with 50 ms;
- a 500 ms pause is dropped with any grace up to 250 ms.
- A remote client sees longer stalls, from round-trip time and retransmission timeouts.
- The grace length and the buffer depth are a policy decision.
519-3 (Medium): a dropped WebSocket client is never told.
- On gRPC the drop is visible: the stream ends with
Unavailable. - On WebSocket, closing the client's channel sends nothing, neither an error frame nor a close.
- The socket stays open on keep-alives, and the pushes just stop.
- A later
eth_unsubscribegets "subscription not found". - Only the idle reaper eventually closes the socket.
- So on WebSocket this turns main's "a few lost pushes" into "every later push lost, silently". The rationale "a gap is worse than a stream that visibly ends" doesn't hold there.
- Probe: a client pauses 500 ms during a 3000-push burst, then reads. Here, with or without the bounded wait, it gets 885 of 3000 and then nothing, on an open socket. On main it gets all 3000.
- Options, each with a drawback or cost:
- close the client's socket with a close code such as 1013, which also ends its other subscriptions on that socket;
- push a JSON-RPC error for the subscription id;
- don't drop WebSocket clients at all.
- The
dropSlowClientdoc and the description should match whichever is chosen.
519-2 (High, merge blocker): the branch replayed onto main doesn't compile.
- This is K1 from my earlier comment:
undefined: hashedParams. - Fixed by passing
activeSub.hashedParams, which the subscription already carries.
519-4 (Medium): TestUnsubscribeDuringRestore_TellsTheNodeTheRestoredID fails 3 runs out of 3 on the merged tree.
- This is K3.
- I'm reworking it to start the restore with a real queue overflow: the mock node pushes 20,500 frames while the listener is held, so the restore runs on a connection that stays healthy.
- It passes 3/3, and 10/10 under
-race. It still fails if the restored id isn't sent to the node.
519-5 (Low): an explicit unsubscribe during an upstream outage fails and leaves the client attached.
- A subscription now lives through an outage, and its dead connection is closed only once a redial lands.
- So
Unsubscribestill calls the node, rpcclient redials into the outage, andUnsubscribereturns an error before it releases the client. - When the node recovers, the restore resumes the pushes the client asked to stop.
- I'm pushing a fix:
- it skips the node call on a closed or dead connection, as the teardown does;
- it logs the skip at debug;
- it releases the client.
519-6 (Low): a gRPC joiner that hung up is counted as a slow consumer.
TrySendreturns false both for a full channel and for a closed sender.- So a push that arrives between the client's hang-up and
UnsubscribeAllreachesdropSlowClient. The client is counted in the new metric, logged at WARN, and given the slow-consumer end status. - I'm pushing a three-way
TrySendresult, so the managers drop only on a full channel.
519-7 (Low): the JsonError branch in retireConnectionOnUpstreamError is unreachable after #511.
- This is K2.
- I'm dropping the branch and its doc clause. A table test pins which ends retire the shared connection.
519-8 (Low): smartrouter_subscription_clients_dropped_total isn't in docs/METRICS.md. I'm adding a row; it counts both WebSocket and gRPC drops.
519-9 (Nit): direct_ws_subscription_outage_test.go:44-46 says the mock node still lists the subscription that died with the socket. This PR makes the mock forget it.
Questions:
errClientReconnected:- If a write fails while the old read loop is still alive, rpcclient redials by itself and ends the old socket's subscriptions with
errClientReconnected. - The listener treats that as a socket-level failure and retires the connection, even though its client now has a working socket.
- Should that error leave the connection alone?
- If a write fails while the old read loop is still alive, rpcclient redials by itself and ends the old socket's subscriptions with
- Already on main:
- After an overflow restore, neither the old
ClientSubscriptionnor the node's old upstream id is unsubscribed. - So the node keeps pushing the old id until the socket closes.
- Worth a follow-up ticket.
- After an overflow restore, neither the old
#508 (MAG-2546) and #509 (MAG-2547) landed on main as squashes with their review fixes (d966dbe, 30c7545), so this merge takes their final versions from main and keeps this PR's own three commits (1edee77, 22b9fe6, cfb4738); the tree equals `git rebase --onto origin/main 21fd439`. #511 (MAG-4244) removed routeMessageToClients' unused hashedParams parameter, which 1edee77's dropSlowClient call needed: the call reads it off the subscription instead (activeSub.hashedParams). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… listener never reaches (MAG-4242) Since #511 (1979be8) the listener reads a *rpcclient.JsonError pushed on a subscription as the node ending it and cleans the subscription up before it ever calls retireConnectionOnUpstreamError. The branch that spared the connection on that error could not run, and the doc comment's "the node answering this one subscription with an error ... the restore re-subscribes on it" described a restore that no longer happens. The test puts two subscriptions on one pooled connection and ends the first by each cause that is not the socket's: unsubscribe, disconnect, a node error push. The connection stays healthy and pooled and the second keeps receiving; a dropped socket is retired. It fails if a node error push reaches the retirement. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…nection that still answers (MAG-4242) TestUnsubscribeDuringRestore_TellsTheNodeTheRestoredID (#509) started its restore by dropping the socket. Now the listener retires that connection and the pool closes it once the reconnect lands, so the unsubscribe finds no client to call, skips the node, and the test's wait for the node to receive it timed out every run. The race it pins (an unsubscribe on the node while a restore commits a new upstream id) now only happens when the restore runs on a connection that stays healthy: rpcclient giving up on the subscription's queue. The test triggers exactly that. While the manager lock holds the listener, the node pushes more than rpcclient queues (20000), the forwarder ends with ErrSubscriptionQueueOverflow, and the restore re-subscribes on the same connection. The rest is unchanged, and it still fails when the teardown is not told about the restored id. The overflow also joins the retirement table: the connection stays pooled and healthy, and the subscription is restored on it. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…a dead connection (MAG-4242) A subscription now lives through an upstream outage: the listener marks its connection dead and the pool reconnects with backoff, but closes the dead connection only once a reconnect lands. An eth_unsubscribe in that window still found a client on the connection and called the node. rpcclient redialled to send it, the node refused, and Unsubscribe returned before its local release: the client got an error, stayed attached, and the restore resumed the pushes it had asked to stop. The node lost the subscription with the socket, so there is nobody to tell. Unsubscribe now skips the call on a connection that is closed or marked dead, as the teardown already does, logs the skip at debug, and releases the client. Before this the skip on a closed connection was silent. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ull, not when it left (MAG-4242) TrySend answered false both for a full channel and for a sender that was closed or whose context had ended, and the fan-out dropped and counted both as slow consumers. A gRPC joiner's sender is bound to its stream's context: when the client hangs up, the stream's Close releases it through UnsubscribeAll, but a push can reach the fan-out first. That client was counted in smartrouter_subscription_clients_dropped_total, logged at WARN as not keeping up, and given the "did not keep up" status, though it had just left. TrySend now reports TrySendDelivered, TrySendFull or TrySendClosed. Both managers drop and count only on TrySendFull; a closed sender belongs to a client already leaving by another path, which releases it. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ng it, so bursts keep readers (MAG-4242) The slow-consumer drop ended a client's subscription at the first push that found its 100-deep channel full, without waiting. A node publishes a block's logs back to back (an unfiltered logs subscription gets hundreds of pushes per block in one burst), and the fan-out hands them over in about 13 us each, so a client reading every push only a little slower is a buffer behind within microseconds. Measured against main, through the real manager and mock node: - WebSocket fan-out: a client reading at 20 us a frame (50,000 frames/s) was dropped 255 pushes into a 300-push burst; at 50 us, 134 pushes into a 150-push burst. Main delivered every push in every case. - Through the real ingress and a real client socket, the kernel buffers absorb up to about 1000 log pushes. A 3000-push burst dropped a client reading at 20 us a frame after 2399 pushes, and a client that paused 500 ms after 885. Main delivered all 3000. - gRPC: a client reading at 20 us a frame was dropped about 100 messages into a 300 burst. Main delivered all of them. On WebSocket a dropped client is not told (its socket stays open and the pushes just stop), so a healthy client silently lost its subscription for good. A full channel now waits up to subscriptionClientSendGrace (10 ms) for room before the client is dropped (SafeChannelSender.SendWithin). A client that is reading frees a slot within microseconds and the fan-out paces itself to it, as Send's retries always did. A client that frees nothing for the whole grace is dropped and counted as before. The bound is the ~10 ms those retries allowed every push, and the wait holds the sender's lock as they did, so nothing waits longer than on main, and only while a client is a full buffer behind. Owner-tunable: zero restores drop-at-once. The new burst tests fail with the grace at zero (WebSocket: ended after about 120 of 1000 pushes; gRPC: after about 101 of 300). The tests for a client that never reads pass unchanged. The grace does not cover every socket stall. Through the real ingress, once the kernel buffers are full a slot can take tens of milliseconds to free: a 3000-push burst to a client reading at 100 us a frame still drops it with a 10 or 25 ms grace and keeps it with 50 ms. A client that pauses for longer than the grace while a full buffer behind is still dropped (a 500 ms pause is dropped with any grace up to 250 ms), where main lost pushes but kept the client. How long to wait, how deep to buffer, and whether a stuck client may stall its subscription's other clients that long are the owner's call. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…l (MAG-4242) The slow-consumer counter is registered and exported, but docs/METRICS.md, the metrics reference, has no row for it. It goes in the WebSocket table next to the subscription counters, and the row notes that it counts gRPC drops too and what a dropped client sees on each interface. The test pins what the row promises: the constructor registers the series under its documented name and labels, so it reaches /metrics. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…(MAG-4242) The comment said the mock node still lists the subscription that died with the socket. The same change taught the mock to forget it, as a real node does: its pusher closes the subscription on the first write to the dead socket. That forgetting is the reason the test counts subscribe calls instead of live subscriptions, and the comment now says so. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…013 so it reconnects (MAG-4242)
A WebSocket client the fan-out dropped as a slow consumer was never told. Its reply
channel closed, the ingress forwarder ended, and nothing reached the client: no error
frame, no close. Its socket stayed open on keep-alives, its router id was forgotten, and
the pushes just stopped, so a client could not tell a dropped subscription from a quiet
one. gRPC already ends the stream with Unavailable.
The manager now records the drop for the client's connection, under the same lock that
closes the channel, and reports it through chainlib.WSSubscriptionEndReporter, the
WebSocket counterpart of GRPCSubscriptionEndReporter. When a subscription's reply channel
closes, the ingress asks. If the answer is ErrSubscriptionClientTooSlow, it sends a close
frame, 1013 (try again later) with "subscription client did not keep up", through the
connection's writer, after the pushes the channel still held. The writer then tears the
connection down as it does for the idle close, and the deferred UnsubscribeAll ends the
client's other subscriptions on that socket. The drop is read once and is forgotten when
the connection ends.
Only the slow-consumer drop does this. A subscription that ends any other way (the
client's unsubscribe, a node error, a failed restore, Close) leaves the socket open as
before. gRPC is unchanged.
Tests: through the real ingress (JSON-RPC listener, ConsumerWebsocketManager, a real
client socket), TestSlowWebSocketClientIsToldToReconnect stops reading during pushes, then
gets the close with 1013, and both of its subscriptions end. On the previous commit it
times out with nothing ("after 4921 pushes ... i/o timeout instead of a close frame").
TestWebSocketSubscriptionTheNodeEndsKeepsTheSocket pins the scope: a subscription the node
ends with an error leaves the socket serving. In chainlib,
TestConsumerWebsocketManager_SlowSubscriberIsClosedWithTryAgainLater and
TestConsumerWebsocketManager_OrdinarySubscriptionEndKeepsTheSocket pin the ingress half
against a fake manager.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Review fixes pushed: 3b0c0efThe branch now merges main, in cc87221. The merge keeps #508 and #509 as they landed on main, plus this PR's three commits, so the tree equals
The two questions in the review ( Verified on 3b0c0ef:
|
…n (MAG-4242) The connection's rate-limit and idle checker read the WebSocketRateLimit and WebSocketBanDuration globals from its own goroutine, which nothing waits for. The globals are process-wide flags, set once at startup, so this never mattered in production, but it is a data race against any later write of them: under -race, the new ingress tests in rpcsmartrouter (the first there to serve real WebSocket connections) left that read unsynchronised with TestRealCommand_RespCacheFromYAML, which registers the root command's flags again and so writes both. ListenToMessages now reads the two settings once, in the connection's own goroutine, before the checker starts, and the checker and the read loop use that snapshot. Behaviour is unchanged: nothing writes the globals after startup. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
One more fix pushed: 01903fbFound in an integration run of this PR with #520 and #522–#525 on main. Under
On the integration tree (main plus all six PRs), all of these are clean:
|
AnnaR-prog
left a comment
There was a problem hiding this comment.
Approving at 01903fb: main 8ad0ff6 plus this PR and the review fixes, including the 1013 close for a dropped WebSocket client. I validated it together with #520, #522, #523, #524 and #525 on one tree: golangci-lint reports 0 issues, go test ./... passes, and -race is clean on the touched packages. The two questions in my review are still open for a follow-up.
… replaced, and keep a redialed connection (MAG-4396) (#591) * fix(rpcsmartrouter): a client that joins during the last client's unsubscribe gets its own subscription (MAG-4345) Unsubscribe decided "last client" under the lock, called the node off the lock, and only then tore the subscription down. The entry stayed in the active map while the node call was in flight, so a client arriving for the same params joined it, a local step with no node call. The local cleanup then saw the joiner's mapping and tore nothing down, but the node had already dropped the subscription: the joiner got no pushes, and its channel was never closed, so it waited until it disconnected. The subscription now leaves the active map in the first critical section, before the node is called. A subscription the router has told the node to drop takes no new clients, and a client arriving meanwhile creates a subscription of its own on the node. A restore that reaches its commit point while the call is in flight finds nothing to commit onto and ends the subscription it re-opened (the MAG-2547 discard), so the restored-since-read branch goes. A node call that fails no longer returns early with the client still attached: the client asked to leave and the subscription is already retired, so it ends locally and the error goes back with the reply. A client that was not the last when its call started but is by the time the lock is retaken now tells the node, which that path never did. teardownSubscriptionLocked splits into retireSubscriptionLocked (the map entry, identity-guarded, and the counter) and endUpstreamSubscriptionLocked (the upstream half), so Unsubscribe can run the two halves apart. Tests: TestWS_JoinerDuringLastClientUnsubscribeGetsItsOwnSubscription, from Anna Rayer's attached repro (the arriving client subscribes in the background, because it now needs the node the test holds), and TestWS_LastClientUnsubscribeFailingUpstreamStillEndsTheSubscription (the node applies the unsubscribe but never answers). Both fail on main. TestUnsubscribeDuringRestore_TellsTheNodeTheRestoredID now asserts the outcome rather than the path: neither id is left on the node. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Took 20 minutes * fix(rpcsmartrouter): tell a non-EVM node the unsubscribe method its spec declares (MAG-4278 item 1) The three router-initiated teardown paths — the last client disconnecting, a failed restore's cleanup, and the discard of a subscription a restore re-opened onto an entry that had gone — derived the unsubscribe method from the subscribe method by string surgery whose fallback is the literal "unsubscribe". Only Tendermint has that method, so on Solana and every Substrate chain the node rejected the call, the failure was traced, and the node kept publishing into the pooled connection: the MAG-3366 leak shape. The same paths named the subscription with the decimal string the router canonicalises to internally, which Solana also rejects. The explicit-unsubscribe path was always right, because the client names the method. The method is now resolved from the spec once, when the subscription is created, and stored on it. The spec gives no field linking a SUBSCRIBE directive to its UNSUBSCRIBE one, so the pairing is recovered from the names a collection declares, and only ever returns a name it declares: 1. the subscribe name with its first subscribe/Subscribe token replaced by unsubscribe/Unsubscribe, when the collection declares that name; 2. for the names rule 1 cannot reach, namespace pairing: within one namespace an unclaimed subscribe pairs with an unclaimed unsubscribe when each is the only one of its kind there. Uniqueness is required in both directions, so an ambiguous namespace resolves to nothing rather than to a guess. Measured over the whole lava-specs catalog (317 subscribe apis in 57 collections): rule 1 resolves 242, rule 2 resolves 63, 12 resolve to nothing. Nine of those are gRPC or REST subscriptions this manager never serves; the rest are Enjin's two archive methods, which share a namespace with one unclaimed unsubscribe between them, and Filecoin.ChainNotify, for which the FVM spec declares no unsubscribe at all. For those the node is told nothing: a call the spec says it does not answer cannot stop it publishing, and a warning naming the method is logged when the subscription is created. The id now goes in the chain's own shape on every path, through one renderer shared with the explicit path. One upstreamSubscriptionTeardown value carries what the node must be told, snapshotted under the lock, so the three paths cannot pick different methods or id shapes from each other again. getUnsubscribeMethod becomes derivedUnsubscribeMethod and reports failure instead of answering a name it cannot reach with the bare literal. It is reached only for a collection that declares no unsubscribe at all. Tests: TestWS_TeardownOnSolanaSendsTheSpecUnsubscribeMethod, from the repro Anna Rayer attached to the ticket, strengthened to assert the method and the unquoted id the node was actually asked for rather than only the absence of "unsubscribe" — which a router that sends nothing also satisfies; TestWS_ClientDisconnectOnSubstrateSendsTheSpecUnsubscribeMethod for the commonest shape, a client closing its socket on chainHead_v1_follow; TestWS_TeardownTellsTheNodeNothingWhenTheSpecDeclaresNoUnsubscribe; TestWS_ExplicitUnsubscribeStillSendsTheClientsMethod; and TestResolveTeardownUnsubscribeMethod over the directive shapes the catalog declares. All four end-to-end tests fail on main. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Took 1 minute * fix(rpcsmartrouter): bound a WebSocket subscription's restores and count its pool slot on pick (MAG-4278 item 11) Two gaps between the WebSocket subscription manager and its gRPC counterpart, which has carried both since MAG-4177. A restore was unbounded. Each one spawns a fresh listener, and that listener's own failure spawns the next restore, so a node that accepts a subscribe and ends it immediately was re-subscribed for the life of the process: one round trip per cycle, on every subscription it does this to, with nothing counting the cycles. A subscription now carries the gRPC manager's accounting — restoresWithoutProgress against maxRestoresWithoutProgress, refilled by a restore that stayed up for restoreStableAfter — and is given up once it is spent. Uptime rather than a delivered push is the measure of progress, so a quiet subscription does not spend its budget on the reconnects a load balancer's connection age limit causes. A subscription's pool slot was counted after the node answered the subscribe. In that window the connection carries no counted subscription, and maybeScaleDown closes a connection past minConnections that carries none, so the subscription being created on it died on a socket the pool had just closed. The pool gains ReserveConnectionForSubscription, which counts the slot under the same lock as the pick, and every failure after the reservation releases it: a failed subscribe, a reply that cannot be built, and a failed re-subscribe on the restore path. GetConnection and GetConnectionForSubscription keep their behaviour for callers that do not open a subscription. Tests: TestRestoreBudget_GivesUpOnceTheBudgetIsSpent and TestSubscribeReservesItsPoolSlot fail on main. TestRestoreBudget_RefillsAfterASubscriptionStaysUp and TestSubscribeReleasesItsPoolSlotWhenTheSubscribeFails are guards for the refill and the release rather than repros, and pass either way. The mock node gained rejectSubscribes, which answers a subscribe with a JSON-RPC error. Still open on the ticket's item 11, and not in this change: the WS manager closes a given-up subscription's client channel with no status, where the gRPC manager ends the client's stream with a reason. Giving WebSocket clients one needs a new SubscriptionEndError reason and a close-code decision in the ingress, which is client-visible behaviour rather than a drift fix. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Took 2 minutes * fix(rpcsmartrouter): tell the node to drop the subscription a restore replaced, and keep a redialed connection (MAG-4396) The two leftovers PR #519's review left open. A restore that runs on a live connection never told the node to drop the subscription it replaced. rpcclient ends one subscription on a connection that stays up when its forward queue overflows, and ClientSubscription.run discards the unsubscribe-the-server flag forward returns, so rpcclient sends nothing; handleUpstreamDisconnect used the old upstream id only for logging and for dropping its id mappings. So the node kept publishing the replaced subscription on the same socket, rpcclient read those frames and dropped them, and a subscription that overflows repeatedly left one more of them behind each time. The restore now tells the node, through the same teardown the other paths use. Where the restore followed a lost socket the node lost the subscription with it, and teardownUpstreamSubscription skips a dead connection, so one call is right on both paths. The local rpcclient subscription is deliberately not unsubscribed with it. Unsubscribe closes its Err channel, and a listener still parked there reads the close as a nil error, ends, and runs its deferred cleanup — tearing down the subscription the restore just installed. The listener that handed off has already returned, so nothing is parked there on the restore path, but the margin is one scheduling decision wide and the only thing it buys is dropping rpcclient's handler entry for a replaced subscription, which closing the connection does anyway. The decision the ticket asks for: errClientReconnected leaves the pooled connection alone. PR #519 made every subscription error except a queue overflow or a node error retire the connection, and this is the one case where the client has already recovered — rpcclient replaced the socket itself and holds a working one. Retiring it there closes that client, so the pool dials again for nothing and every other subscription sharing the connection ends too, including one a restore has just re-created on the new socket. A real socket failure still retires it. errClientReconnected becomes ErrClientReconnected: the fact that the connection is healthy is not readable from anywhere else, and a caller that treats every subscription error as a dead socket has no way to tell. Tests: TestOverflowRestore_TellsTheNodeToDropTheOldSubscription, TestClientRedial_KeepsTheConnectionAndItsSiblingSubscriptions and TestRetireConnectionOnUpstreamError_LeavesAClientThatRedialed fail on main. TestRestoreAfterADroppedSocket_DoesNotCallADeadNode guards the other half of the same call. The redial is driven through the upstreamErrSource seam the listener documents for it: rpcclient mints the error internally, and the MAG-4037 trigger, a call whose deadline has already passed, fails the redial on that same expired deadline, so no reconnect lands. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(rpcsmartrouter): tell the node to drop a replaced subscription only after a queue overflow (MAG-4396) The restore told the node to drop the subscription it replaced after every restore on a healthy connection. Only a queue overflow leaves that subscription on a live socket. When rpcclient redials on its own (ErrClientReconnected), it closes the old socket and the connection is now kept, so the goodbye went out over the NEW socket. On Tendermint the subscription id is the query, so `unsubscribe {query}` ended the subscription the restore had just opened: every client of every Tendermint subscription on that connection was left on a subscription the node no longer publishes, with its channel open. On other chains the goodbye named an id the new socket does not know, or, on a node that numbers its subscriptions per socket, a sibling the redial had just restored. The restore now receives the error that ended the subscription and says goodbye only after a queue overflow, and never when the old id names the restored subscription on the same connection. TestTendermintRedial_KeepsTheRestoredSubscriptionOnTheNode and TestClientRedial_RestoresEverySubscriptionOnTheKeptConnection drive a real redial (calls past their deadline, then a live call) and fail on b784643. TestReplacedStillOnNode pins the decision. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix(rpcsmartrouter): an unsubscribe does not tear down a subscription that ended under it (MAG-4345) A client that is not the last one releases the lock between Unsubscribe's two critical sections. If the subscription ends there by another path — the node ends it with an error, or a restore gives it up — the listener's cleanup releases every client, stops counting the subscription and tells the node. Unsubscribe then retook the lock, found no clients left, and tore the subscription down again: totalSubscriptions and the pooled connection's count each went to -1 and the node was told twice. On main the mapper's last-client answer, false once the cleanup had removed every mapping, kept it from doing so; the retire/end split dropped that guard. The teardown now runs only while the subscription is still the live entry under its params. beforeUnsubscribeRelease is a test hook, nil in production, in the shape of beforeRestoreCommit. TestWS_UnsubscribeDoesNotTearDownASubscriptionThatEndedUnderIt fails without the guard (totalSubscriptions -1, pool count -1, two eth_unsubscribe) and passes on main's manager with only the hook added. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix(rpcsmartrouter): hand the disconnect watcher its client's signal when the client attaches (MAG-4345) The watcher a subscribe starts looked its subscription up when the goroutine first ran, to read the client's detached signal. Unsubscribe now takes the subscription out of the map in its first step, so a client that unsubscribed before its watcher was scheduled left the watcher with a nil channel: it stayed parked until the socket closed, one goroutine per subscribe and unsubscribe on a long-lived socket. On main the entry stayed mapped for the node's round trip, which hid the race. The signal is now read where the client is attached — by the creator while it still owns the subscription, by a joiner under the manager lock — and handed to the watcher. TestUnsubscribeThenDisconnect_LeavesNothingBehind ("the disconnect watcher must not stay parked once the client unsubscribed") failed 6 of 20 runs without this and passes 30 of 30 with it; it passes 20 of 20 on main. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * test(rpcsmartrouter): pin that a WebSocket subscription counts its pool slot while the node answers (MAG-4278 item 11) TestSubscribeReservesItsPoolSlot drives the pool's ReserveConnectionForSubscription on its own. The manager calls it in two places, StartSubscription and the restore, and with both of them reverted to GetConnection plus an IncrementSubscriptions after the node's answer every test in the package still passed, so nothing pinned the fix the window was about. TestSubscriptionCountsItsPoolSlotWhileTheNodeAnswers holds the node's answer to a client's subscribe, then to the restore's re-subscribe after the socket drops, and requires the pool to count the subscription in both windows. It fails with either call site reverted. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * test(rpcsmartrouter): let the restore-budget refill test wait for the restore to commit (MAG-4278 item 11) TestRestoreBudget_RefillsAfterASubscriptionStaysUp waited for `upstreamID != "" && upstreamCalls("eth_subscribe") == 2`. The old upstream id is still set during the restore, so the wait ended as soon as the node had read the re-subscribe, before it answered and before the restore committed. The test then returned, and its cleanup cancelled the subscription's context with the re-subscribe possibly still in flight: the window in which rpcclient's Subscribe can deadlock its dispatch loop (an unbuffered reply channel the cancelled waiter no longer reads), after which manager.Close never returns. It now waits for the upstream id to change, which is the restore's commit. The budget tests still fail with the budget removed. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: Anna Rayer <anna@magmadevs.com>
Description
MAG-4242 items 6, 8 and 9: the subscription managers' resilience under upstream flaps, slow clients and concurrent identical subscribes. One commit per item. Stacked on #508 (MAG-2546) and #509 (MAG-2547), which edit the same WebSocket manager; the diff shows their commits until they merge, after which this branch is rebased onto
main.Jira ticket: MAG-4242
Summary
1edee77b). The WebSocket per-client reply channel was unbuffered and the sender polled it ten times a millisecond apart before dropping the push with a trace log, so a client whose previous frame took more than 10 ms to write lost pushes with no counter and no signal, and because the fan-out is serial every other client of the shared subscription paid the same stall; gRPC had a 100-deep buffer and the same drop after it. Both managers now give each client asubscriptionClientReplyBuffer(the depth gRPC always had, now shared) and send with the new non-waitingSafeChannelSender.TrySend. A client that lets its buffer fill is dropped: its channel closes so its subscription visibly ends (a gRPC stream ends withUnavailable, "did not keep up"), the drop is counted in the newsmartrouter_subscription_clients_dropped_total{spec, apiInterface}, and the other clients are never stalled.22b9fe63). Nothing ever calledMarkUnhealthyon a pooled connection, so after the upstream socket dropped the pool still read it as healthy:ReconnectWithBackoffreturned at once,GetConnectionhanded the dead socket back, the re-subscribe got rpcclient's single dial, and when that failed the subscription was cleaned up with every client released; the backoff ladder never ran, and the cleanup redialled just to unsubscribe a subscription that had died with the socket. The listener now retires the connection on a socket-level error (a queue overflow or a node error for that one subscription leaves the socket alone), the pool prunes connections marked dead, and the upstream unsubscribe is skipped on a dead connection. gRPC is untouched here: grpc-go reconnects its own transport.cfb4738a). The gRPC manager allocatedpendingSubscriptionsand never used it, so clients arriving while the first was dialling, reflecting and opening its stream each opened a stream of their own and the last overwrote the map entry, stranding the earlier streams upstream with clients that could never unsubscribe them. The WebSocket pending broadcast is ported: the first client registers as creator, later ones wait and join, the creator settles the entry on every path, and a subscription the join turned away (dead, still registered) is replaced rather than waited on.setRefusing) and, like a real node, forgets the subscriptions of a socket that died; the harness closes its manager on cleanup, because a subscription left live now keeps the pool reconnecting into the next tests.Test plan
TestRestoreAfterUpstreamOutage_ReconnectsWithBackoff: the node drops every socket and refuses dials; the router retries with backoff, re-subscribes once the node is back, and the client keeps receiving on the same channel.TestUpstreamWSPool_RetiresAConnectionMarkedDead,TestTeardownUpstreamSubscription_SkipsADeadConnection.TestSlowClientIsDroppedNotWaitedFor(WebSocket) andTestGRPCSlowClientStreamEndsWithStatus: the client that never reads is dropped and counted, its channel closes (the gRPC stream ends withUnavailable), the client that keeps up is unaffected.TestSafeChannelSender_TrySend.TestGRPCStartSubscription_ConcurrentIdenticalSubscribesShareOneStream: four concurrent identical subscribes open one upstream stream and all four receive the shared push.go test ./protocol/common/ ./protocol/metrics/ ./protocol/rpcsmartrouter/green on the committed tree;golangci-lint run --new-from-rev=origin/main: 0 issues.Unavailablestatus for a dropped gRPC client.🤖 Generated with Claude Code