Skip to content

fix(rpcsmartrouter): retire dead WebSocket connections, drop slow subscription clients, and share one gRPC stream per subscribe (MAG-4242) - #519

Merged
AnnaR-prog merged 17 commits into
mainfrom
fix/MAG-4242-subscriptions
Oct 18, 2026
Merged

AnnaR-prog merged 17 commits into
mainfrom
fix/MAG-4242-subscriptions

Conversation

@avitenzer

Copy link
Copy Markdown
Contributor

Description

MAG-4242 items 6, 8 and 9: the subscription managers' resilience under upstream flaps, slow clients and concurrent identical subscribes. One commit per item. Stacked on #508 (MAG-2546) and #509 (MAG-2547), which edit the same WebSocket manager; the diff shows their commits until they merge, after which this branch is rebased onto main.

Jira ticket: MAG-4242

Summary

  • Item 8, slow consumers (1edee77b). The WebSocket per-client reply channel was unbuffered and the sender polled it ten times a millisecond apart before dropping the push with a trace log, so a client whose previous frame took more than 10 ms to write lost pushes with no counter and no signal, and because the fan-out is serial every other client of the shared subscription paid the same stall; gRPC had a 100-deep buffer and the same drop after it. Both managers now give each client a subscriptionClientReplyBuffer (the depth gRPC always had, now shared) and send with the new non-waiting SafeChannelSender.TrySend. A client that lets its buffer fill is dropped: its channel closes so its subscription visibly ends (a gRPC stream ends with Unavailable, "did not keep up"), the drop is counted in the new smartrouter_subscription_clients_dropped_total{spec, apiInterface}, and the other clients are never stalled.
  • Item 6, dead pooled WebSocket connections (22b9fe63). Nothing ever called MarkUnhealthy on a pooled connection, so after the upstream socket dropped the pool still read it as healthy: ReconnectWithBackoff returned at once, GetConnection handed the dead socket back, the re-subscribe got rpcclient's single dial, and when that failed the subscription was cleaned up with every client released; the backoff ladder never ran, and the cleanup redialled just to unsubscribe a subscription that had died with the socket. The listener now retires the connection on a socket-level error (a queue overflow or a node error for that one subscription leaves the socket alone), the pool prunes connections marked dead, and the upstream unsubscribe is skipped on a dead connection. gRPC is untouched here: grpc-go reconnects its own transport.
  • Item 9, concurrent identical gRPC subscribes (cfb4738a). The gRPC manager allocated pendingSubscriptions and never used it, so clients arriving while the first was dialling, reflecting and opening its stream each opened a stream of their own and the last overwrote the map entry, stranding the earlier streams upstream with clients that could never unsubscribe them. The WebSocket pending broadcast is ported: the first client registers as creator, later ones wait and join, the creator settles the entry on every path, and a subscription the join turned away (dead, still registered) is replaced rather than waited on.
  • The mock node refuses dials on demand (setRefusing) and, like a real node, forgets the subscriptions of a socket that died; the harness closes its manager on cleanup, because a subscription left live now keeps the pool reconnecting into the next tests.

Test plan

  • TestRestoreAfterUpstreamOutage_ReconnectsWithBackoff: the node drops every socket and refuses dials; the router retries with backoff, re-subscribes once the node is back, and the client keeps receiving on the same channel. TestUpstreamWSPool_RetiresAConnectionMarkedDead, TestTeardownUpstreamSubscription_SkipsADeadConnection.
  • TestSlowClientIsDroppedNotWaitedFor (WebSocket) and TestGRPCSlowClientStreamEndsWithStatus: the client that never reads is dropped and counted, its channel closes (the gRPC stream ends with Unavailable), the client that keeps up is unaffected. TestSafeChannelSender_TrySend.
  • TestGRPCStartSubscription_ConcurrentIdenticalSubscribesShareOneStream: four concurrent identical subscribes open one upstream stream and all four receive the shared push.
  • With the pre-fix managers and pool overlaid into the build, all six subscription tests fail; on the branch they pass. go test ./protocol/common/ ./protocol/metrics/ ./protocol/rpcsmartrouter/ green on the committed tree; golangci-lint run --new-from-rev=origin/main: 0 issues.
  • Reviewer: the slow-consumer policy is "drop and count" (not "wait"); confirm the buffer depth of 100 pushes and the Unavailable status for a dropped gRPC client.

🤖 Generated with Claude Code

avitenzer and others added 7 commits October 12, 2026 15:39
…light (MAG-2547)

When an upstream socket drops, handleUpstreamDisconnect re-subscribes on a fresh
connection and then rewrites the subscription's upstreamSubscription, upstreamID and
upstreamConnection under the manager lock. A client that disconnected during that window
ran teardownSubscriptionLocked, whose goroutine read those same fields off the lock — a
data race — and whichever half of the restore it saw, the other half leaked: the restored
upstream subscription stayed live on the node with nobody reading it, and the new
connection kept counting it, so the pool could never scale it down.

The teardown now snapshots everything its goroutine needs while it still holds the lock,
and the restore commits only if its subscription is still the one in the map; otherwise it
ends the upstream subscription it just created, node side too, and uncounts it. Cleanup
is identity-guarded for the same reason: the map is keyed by params alone, so a listener
ending or a restore failing late could otherwise destroy a newer subscription registered
under the same key.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Took 2 minutes
… between the two checks (MAG-2546)

StartSubscription looks for an active subscription and registers a pending one under
two separate acquisitions of the manager lock. A creator that stored its subscription
and cleared its pending entry in that gap left the second caller seeing neither, so it
became a second creator: a second upstream subscription for the same params, the first
map entry overwritten and orphaned (its listener goroutine and node-side subscription
living on unowned), and totalSubscriptions incremented twice for an entry that is
dropped once. The pending check now re-checks the active map under the same lock and
hands such a caller a settled success so it takes the join path.

The cleanup-ticker branch the ticket names (a cancelled context deleting without
decrementing) was already routed through dropSubscriptionLocked by #386; the new test
pins that so it stays fixed.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Took 47 minutes
…ent-teardown' into fix/MAG-4242-subscriptions
…g its pushes (MAG-4242)

The per-client reply channel was unbuffered on WebSocket and the sender
polled it ten times, one millisecond apart, then dropped the push with a
trace log: a client whose previous frame took more than 10 ms to write
lost pushes with no counter and no signal, and because the fan-out is
serial every other client of the shared subscription paid the same
stall. gRPC had a 100-deep buffer and the same drop after it.

Both managers now give each client a buffer of
subscriptionClientReplyBuffer pushes and send without waiting
(SafeChannelSender.TrySend). A client that lets its buffer fill is
dropped: its channel closes so its subscription visibly ends (a gRPC
stream ends with Unavailable, "did not keep up"), the drop is counted in
smartrouter_subscription_clients_dropped_total, and the other clients
are never stalled. The WebSocket harness now closes its manager on
cleanup, as a live subscription left behind keeps the pool busy.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Took 18 minutes
… restore reconnects with backoff (MAG-4242)

Nothing ever called MarkUnhealthy on a pooled connection, so when the
upstream socket dropped the pool still read it as healthy:
ReconnectWithBackoff returned at once, GetConnection handed the dead
socket back, the re-subscribe got rpcclient's single dial, and when that
failed the subscription was cleaned up with every client released. The
backoff ladder never ran, and the cleanup then redialled just to send
an unsubscribe for a subscription that had died with the socket.

The listener now retires the connection on a socket-level error (a
subscription-level end — queue overflow, or the node answering that one
subscription with an error — leaves the socket alone), the pool prunes
connections marked dead instead of handing them out again or keeping
them, and the upstream unsubscribe is skipped on a dead connection. The
mock node refuses dials on demand and, like a real node, forgets the
subscriptions of a socket that died.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… subscribes (MAG-4242)

The gRPC manager allocated a pending-subscription map and never used
it. Clients arriving with the same params while the first was dialling,
reflecting and opening its stream each opened a stream of their own, and
the last to finish overwrote the map entry: every earlier stream lived on
upstream with a client that could never unsubscribe it.

The WebSocket manager's pending broadcast is ported: the first client
registers as the creator before it opens the stream, later ones wait for
its outcome and join, and the creator settles the entry on every path. A
subscription the join turned away (dead, still registered until its
listener releases it) is never waited on or joined; it is replaced.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@AnnaR-prog

Copy link
Copy Markdown
Contributor

@avitenzer heads-up for the rebase. #508, #509 and #511 are on main now, so this branch conflicts in 4 files. Here is what I found against main at 8ad0ff6, in a scratch tree; I haven't pushed anything to this branch.

  1. Only this PR's own three commits need to move. fix(rpcsmartrouter): join a WebSocket subscription that became active between the two checks (MAG-2546) #508 and fix(rpcsmartrouter): let a WebSocket teardown win over a restore in flight (MAG-2547) #509 landed as squashes with the review fixes (d966dbe and 30c7545), so bd6cbbb, 3b13631 and the two merges can go. git rebase --onto origin/main 21fd4392 replays 1edee77, 22b9fe6 and cfb4738 with no textual conflict.

  2. The result does not compile: undefined: hashedParams in routeMessageToClients. perf(protocol): enable six linters and act on the audit's lint-derived efficiency signals (MAG-4244) #511's unparam pass removed that function's unused hashedParams parameter, and 1edee77's dropSlowClient(hashedParams, activeSub, clientKey) needs it back. Three lines fix it: the parameter, its argument at the listener's call site, and the one test call in direct_ws_subscription_manager_test.go. With them, go build ./... and go vet are clean and protocol/chainlib/chainproxy/rpcclient passes. protocol/rpcsmartrouter has one failure, item 4.

  3. perf(protocol): enable six linters and act on the audit's lint-derived efficiency signals (MAG-4244) #511 changed the listener in a way that meets 22b9fe6. The listener now treats a JSON-RPC error pushed on a subscription (*rpcclient.JsonError) as the node ending it. It cleans up instead of restoring, because restoring re-subscribed at once on the same healthy connection with no backoff (review fix 4e4770a). That check returns before retireConnectionOnUpstreamError is called, so:

    • the JsonError branch inside that function is unreachable;
    • its doc comment's "or the node answering this one subscription with an error ... the restore re-subscribes on it" no longer holds.

    I suggest dropping that branch and that clause. The queue-overflow case is unaffected.

  4. TestUnsubscribeDuringRestore_TellsTheNodeTheRestoredID fails 3/3 after the rebase (it passes 3/3 on main). It is the test from fix(rpcsmartrouter): let a WebSocket teardown win over a restore in flight (MAG-2547) #509's review fix 87b1a8d.

    • What it does: drops the connection, holds the restore at its commit point, then waits for the explicit eth_unsubscribe to reach the node on the old connection.
    • Why it fails: with 22b9fe6 the dropped connection is marked dead and pruneDeadConnectionsLocked closes it. So GetClient() returns nil, Unsubscribe skips the node call without a log line, and the wait at direct_ws_subscription_restore_test.go:128 times out.
    • What still works: in that run, the restore's discard path still ends the restored subscription.
    • What to change: the race the test pins (a node call in flight while a restore commits) now happens only when a restore runs on a connection that stays healthy, i.e. after ErrSubscriptionQueueOverflow. Triggering the restore that way should keep the test meaningful.

@AnnaR-prog AnnaR-prog left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed this PR's three commits replayed onto current main (8ad0ff6), as in my earlier comment: git rebase --onto origin/main 21fd4392 plus the compile fix.

Items 6 and 9 are sound.

  • Item 6: only a socket-level failure retires the shared pooled connection.
    • I put two subscriptions on one connection and ended the first by each cause, using the real rpcclient, pool and manager.
    • An explicit unsubscribe, a client disconnect, a node JSON-RPC error push and a queue overflow all leave the connection healthy, and the other subscription keeps receiving. Only a dropped socket retires it.
    • All six subscription tests fail with main's managers and pool overlaid, and pass here.
  • Item 9: I drove the pending entry by hand under -race.
    • A creator that fails: the waiters retry and share one stream.
    • A waiter whose context ends: it returns at once and leaves nothing behind.
    • A dead subscription that is still registered: it's replaced, with one stream.
    • The creator settles on every path, panics included. Your four-client test passes 30/30 under -race.

519-1 (High): item 8 drops healthy clients that read every push, only slower than a burst arrives.

  • The fan-out drops a client at the first push that finds its 100-deep channel full, without waiting.
  • The router hands pushes over in about 13 µs each. A client that reads a little slower is 100 behind within a millisecond or two of a burst's start. An unfiltered logs subscription gets hundreds of pushes per block, in one burst.
  • I measured this with the real manager and rpcclient and ~700-byte log frames:
    • WebSocket, 20 µs per frame (50k frames/s): dropped 255 pushes into a 300-push burst.
    • WebSocket, 50 µs per frame: dropped after 138 pushes.
    • gRPC, 20–200 µs per frame: dropped after about 101–108 messages of a 300- or 1000-message burst.
    • Main delivers every push in every case, because the unbuffered channel and Send's 10 × 1 ms retries paced the fan-out to the client.
  • Your test covers only a client that never reads.
  • I'm pushing a bounded wait (SafeChannelSender.SendWithin). When a client's buffer is full, the fan-out waits up to 10 ms for room before dropping it. That is main's per-push bound.
    • The manager-level sweeps are then clean: WebSocket 18/18, gRPC 8/8.
    • A client that never reads is still dropped, after one 10 ms wait.
    • Two new tests, TestBurstyClientThatReadsIsNotDropped and TestGRPCBurstyClientThatReadsIsNotDropped, fail with the wait set to 0.
  • The wait does not cover socket-level stalls. Through the real WebSocket ingress on localhost:
    • a 3000-push burst to a client reading at 100 µs per frame is still dropped with a 10 or 25 ms grace, and kept with 50 ms;
    • a 500 ms pause is dropped with any grace up to 250 ms.
    • A remote client sees longer stalls, from round-trip time and retransmission timeouts.
  • The grace length and the buffer depth are a policy decision.

519-3 (Medium): a dropped WebSocket client is never told.

  • On gRPC the drop is visible: the stream ends with Unavailable.
  • On WebSocket, closing the client's channel sends nothing, neither an error frame nor a close.
    • The socket stays open on keep-alives, and the pushes just stop.
    • A later eth_unsubscribe gets "subscription not found".
    • Only the idle reaper eventually closes the socket.
  • So on WebSocket this turns main's "a few lost pushes" into "every later push lost, silently". The rationale "a gap is worse than a stream that visibly ends" doesn't hold there.
  • Probe: a client pauses 500 ms during a 3000-push burst, then reads. Here, with or without the bounded wait, it gets 885 of 3000 and then nothing, on an open socket. On main it gets all 3000.
  • Options, each with a drawback or cost:
    • close the client's socket with a close code such as 1013, which also ends its other subscriptions on that socket;
    • push a JSON-RPC error for the subscription id;
    • don't drop WebSocket clients at all.
  • The dropSlowClient doc and the description should match whichever is chosen.

519-2 (High, merge blocker): the branch replayed onto main doesn't compile.

  • This is K1 from my earlier comment: undefined: hashedParams.
  • Fixed by passing activeSub.hashedParams, which the subscription already carries.

519-4 (Medium): TestUnsubscribeDuringRestore_TellsTheNodeTheRestoredID fails 3 runs out of 3 on the merged tree.

  • This is K3.
  • I'm reworking it to start the restore with a real queue overflow: the mock node pushes 20,500 frames while the listener is held, so the restore runs on a connection that stays healthy.
  • It passes 3/3, and 10/10 under -race. It still fails if the restored id isn't sent to the node.

519-5 (Low): an explicit unsubscribe during an upstream outage fails and leaves the client attached.

  • A subscription now lives through an outage, and its dead connection is closed only once a redial lands.
  • So Unsubscribe still calls the node, rpcclient redials into the outage, and Unsubscribe returns an error before it releases the client.
  • When the node recovers, the restore resumes the pushes the client asked to stop.
  • I'm pushing a fix:
    • it skips the node call on a closed or dead connection, as the teardown does;
    • it logs the skip at debug;
    • it releases the client.

519-6 (Low): a gRPC joiner that hung up is counted as a slow consumer.

  • TrySend returns false both for a full channel and for a closed sender.
  • So a push that arrives between the client's hang-up and UnsubscribeAll reaches dropSlowClient. The client is counted in the new metric, logged at WARN, and given the slow-consumer end status.
  • I'm pushing a three-way TrySend result, so the managers drop only on a full channel.

519-7 (Low): the JsonError branch in retireConnectionOnUpstreamError is unreachable after #511.

  • This is K2.
  • I'm dropping the branch and its doc clause. A table test pins which ends retire the shared connection.

519-8 (Low): smartrouter_subscription_clients_dropped_total isn't in docs/METRICS.md. I'm adding a row; it counts both WebSocket and gRPC drops.

519-9 (Nit): direct_ws_subscription_outage_test.go:44-46 says the mock node still lists the subscription that died with the socket. This PR makes the mock forget it.

Questions:

  • errClientReconnected:
    • If a write fails while the old read loop is still alive, rpcclient redials by itself and ends the old socket's subscriptions with errClientReconnected.
    • The listener treats that as a socket-level failure and retires the connection, even though its client now has a working socket.
    • Should that error leave the connection alone?
  • Already on main:
    • After an overflow restore, neither the old ClientSubscription nor the node's old upstream id is unsubscribed.
    • So the node keeps pushing the old id until the socket closes.
    • Worth a follow-up ticket.

AnnaR-prog and others added 9 commits October 17, 2026 20:25
#508 (MAG-2546) and #509 (MAG-2547) landed on main as squashes with their
review fixes (d966dbe, 30c7545), so this merge takes their final
versions from main and keeps this PR's own three commits (1edee77,
22b9fe6, cfb4738); the tree equals `git rebase --onto origin/main
21fd439`. #511 (MAG-4244) removed routeMessageToClients' unused
hashedParams parameter, which 1edee77's dropSlowClient call needed: the
call reads it off the subscription instead (activeSub.hashedParams).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… listener never reaches (MAG-4242)

Since #511 (1979be8) the listener reads a *rpcclient.JsonError pushed on a subscription as
the node ending it and cleans the subscription up before it ever calls
retireConnectionOnUpstreamError. The branch that spared the connection on that error could
not run, and the doc comment's "the node answering this one subscription with an error ...
the restore re-subscribes on it" described a restore that no longer happens.

The test puts two subscriptions on one pooled connection and ends the first by each cause
that is not the socket's: unsubscribe, disconnect, a node error push. The connection stays
healthy and pooled and the second keeps receiving; a dropped socket is retired. It fails if
a node error push reaches the retirement.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…nection that still answers (MAG-4242)

TestUnsubscribeDuringRestore_TellsTheNodeTheRestoredID (#509) started its restore by
dropping the socket. Now the listener retires that connection and the pool closes it once
the reconnect lands, so the unsubscribe finds no client to call, skips the node, and the
test's wait for the node to receive it timed out every run.

The race it pins (an unsubscribe on the node while a restore commits a new upstream id) now
only happens when the restore runs on a connection that stays healthy: rpcclient giving up
on the subscription's queue. The test triggers exactly that. While the manager lock holds
the listener, the node pushes more than rpcclient queues (20000), the forwarder ends with
ErrSubscriptionQueueOverflow, and the restore re-subscribes on the same connection. The rest
is unchanged, and it still fails when the teardown is not told about the restored id.

The overflow also joins the retirement table: the connection stays pooled and healthy, and
the subscription is restored on it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…a dead connection (MAG-4242)

A subscription now lives through an upstream outage: the listener marks its connection
dead and the pool reconnects with backoff, but closes the dead connection only once a
reconnect lands. An eth_unsubscribe in that window still found a client on the connection
and called the node. rpcclient redialled to send it, the node refused, and Unsubscribe
returned before its local release: the client got an error, stayed attached, and the
restore resumed the pushes it had asked to stop.

The node lost the subscription with the socket, so there is nobody to tell. Unsubscribe
now skips the call on a connection that is closed or marked dead, as the teardown already
does, logs the skip at debug, and releases the client. Before this the skip on a closed
connection was silent.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ull, not when it left (MAG-4242)

TrySend answered false both for a full channel and for a sender that was closed or whose
context had ended, and the fan-out dropped and counted both as slow consumers. A gRPC
joiner's sender is bound to its stream's context: when the client hangs up, the stream's
Close releases it through UnsubscribeAll, but a push can reach the fan-out first. That
client was counted in smartrouter_subscription_clients_dropped_total, logged at WARN as
not keeping up, and given the "did not keep up" status, though it had just left.

TrySend now reports TrySendDelivered, TrySendFull or TrySendClosed. Both managers drop
and count only on TrySendFull; a closed sender belongs to a client already leaving by
another path, which releases it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ng it, so bursts keep readers (MAG-4242)

The slow-consumer drop ended a client's subscription at the first push that found its
100-deep channel full, without waiting. A node publishes a block's logs back to back
(an unfiltered logs subscription gets hundreds of pushes per block in one burst), and the
fan-out hands them over in about 13 us each, so a client reading every push only a little
slower is a buffer behind within microseconds. Measured against main, through the real
manager and mock node:

- WebSocket fan-out: a client reading at 20 us a frame (50,000 frames/s) was dropped
  255 pushes into a 300-push burst; at 50 us, 134 pushes into a 150-push burst. Main
  delivered every push in every case.
- Through the real ingress and a real client socket, the kernel buffers absorb up to
  about 1000 log pushes. A 3000-push burst dropped a client reading at 20 us a frame
  after 2399 pushes, and a client that paused 500 ms after 885. Main delivered all 3000.
- gRPC: a client reading at 20 us a frame was dropped about 100 messages into a 300 burst.
  Main delivered all of them.

On WebSocket a dropped client is not told (its socket stays open and the pushes just
stop), so a healthy client silently lost its subscription for good.

A full channel now waits up to subscriptionClientSendGrace (10 ms) for room before the
client is dropped (SafeChannelSender.SendWithin). A client that is reading frees a slot
within microseconds and the fan-out paces itself to it, as Send's retries always did. A
client that frees nothing for the whole grace is dropped and counted as before. The bound
is the ~10 ms those retries allowed every push, and the wait holds the sender's lock as
they did, so nothing waits longer than on main, and only while a client is a full buffer
behind. Owner-tunable: zero restores drop-at-once.

The new burst tests fail with the grace at zero (WebSocket: ended after about 120 of
1000 pushes; gRPC: after about 101 of 300). The tests for a client that never reads pass
unchanged.

The grace does not cover every socket stall. Through the real ingress, once the kernel
buffers are full a slot can take tens of milliseconds to free: a 3000-push burst to a
client reading at 100 us a frame still drops it with a 10 or 25 ms grace and keeps it
with 50 ms. A client that pauses for longer than the grace while a full buffer behind is
still dropped (a 500 ms pause is dropped with any grace up to 250 ms), where main lost
pushes but kept the client. How long to wait, how deep to buffer, and whether a stuck
client may stall its subscription's other clients that long are the owner's call.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…l (MAG-4242)

The slow-consumer counter is registered and exported, but docs/METRICS.md, the metrics
reference, has no row for it. It goes in the WebSocket table next to the subscription
counters, and the row notes that it counts gRPC drops too and what a dropped client sees
on each interface.

The test pins what the row promises: the constructor registers the series under its
documented name and labels, so it reaches /metrics.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…(MAG-4242)

The comment said the mock node still lists the subscription that died with the socket.
The same change taught the mock to forget it, as a real node does: its pusher closes the
subscription on the first write to the dead socket. That forgetting is the reason the test
counts subscribe calls instead of live subscriptions, and the comment now says so.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…013 so it reconnects (MAG-4242)

A WebSocket client the fan-out dropped as a slow consumer was never told. Its reply
channel closed, the ingress forwarder ended, and nothing reached the client: no error
frame, no close. Its socket stayed open on keep-alives, its router id was forgotten, and
the pushes just stopped, so a client could not tell a dropped subscription from a quiet
one. gRPC already ends the stream with Unavailable.

The manager now records the drop for the client's connection, under the same lock that
closes the channel, and reports it through chainlib.WSSubscriptionEndReporter, the
WebSocket counterpart of GRPCSubscriptionEndReporter. When a subscription's reply channel
closes, the ingress asks. If the answer is ErrSubscriptionClientTooSlow, it sends a close
frame, 1013 (try again later) with "subscription client did not keep up", through the
connection's writer, after the pushes the channel still held. The writer then tears the
connection down as it does for the idle close, and the deferred UnsubscribeAll ends the
client's other subscriptions on that socket. The drop is read once and is forgotten when
the connection ends.

Only the slow-consumer drop does this. A subscription that ends any other way (the
client's unsubscribe, a node error, a failed restore, Close) leaves the socket open as
before. gRPC is unchanged.

Tests: through the real ingress (JSON-RPC listener, ConsumerWebsocketManager, a real
client socket), TestSlowWebSocketClientIsToldToReconnect stops reading during pushes, then
gets the close with 1013, and both of its subscriptions end. On the previous commit it
times out with nothing ("after 4921 pushes ... i/o timeout instead of a close frame").
TestWebSocketSubscriptionTheNodeEndsKeepsTheSocket pins the scope: a subscription the node
ends with an error leaves the socket serving. In chainlib,
TestConsumerWebsocketManager_SlowSubscriberIsClosedWithTryAgainLater and
TestConsumerWebsocketManager_OrdinarySubscriptionEndKeepsTheSocket pin the ingress half
against a fake manager.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@AnnaR-prog

Copy link
Copy Markdown
Contributor

Review fixes pushed: 3b0c0ef

The branch now merges main, in cc87221. The merge keeps #508 and #509 as they landed on main, plus this PR's three commits, so the tree equals git rebase --onto origin/main 21fd4392. It also reads the slow client's subscription key off the subscription, because #511 removed the hashedParams parameter. That covers 519-2.

Finding Status
519-1 (High): healthy bursty clients dropped Fixed in cc15d50. SafeChannelSender.SendWithin: a full client gets up to 10 ms for room before it is dropped. That is the bound main's Send gave every push. TestBurstyClientThatReadsIsNotDropped (WebSocket) and TestGRPCBurstyClientThatReadsIsNotDropped fail with the wait set to 0. A client that never reads is still dropped, after one 10 ms wait.
519-3 (Medium): a dropped WebSocket client is never told Fixed in 3b0c0ef, as the reviewer decided. The manager marks the drop and reports it through chainlib.WSSubscriptionEndReporter, the WebSocket counterpart of GRPCSubscriptionEndReporter. When the channel closes, the ingress sends a close frame through its writer, after the pushes still buffered: code 1013 (try again later), reason "subscription client did not keep up". The deferred UnsubscribeAll then ends the client's other subscriptions on that socket. Any other end of a subscription leaves the socket open, as before, and gRPC is unchanged. Tests: TestSlowWebSocketClientIsToldToReconnect (through the real JSON-RPC listener, ConsumerWebsocketManager and a client socket) and TestWebSocketSubscriptionTheNodeEndsKeepsTheSocket, plus two ingress tests in chainlib. One limit: a client that never reads can't receive the close frame. Its connection still closes when the 10 s write timeout fires.
519-4 (Medium): #509's unsubscribe-during-restore test Fixed in 569e4d6. The restore is now started by a real queue overflow, on a connection that stays healthy. It passes 3/3 and 10/10 under -race, and still fails if the restored id isn't sent to the node.
519-5 (Low): unsubscribe during an outage Fixed in 624a956. On a closed or dead connection, the node call is skipped, the skip is logged at debug, and the client is released. Test: TestUnsubscribeDuringOutage_ReleasesTheClient.
519-6 (Low): a departed gRPC joiner counted as slow Fixed in 4776966. TrySend now returns delivered, full or closed, and the managers drop only on full. Test: TestGRPCDepartedJoinerIsNotCountedAsSlow.
519-7 (Low): unreachable JsonError branch Fixed in 0f0ad88. TestUpstreamError_RetiresTheSharedConnectionOnlyOnASocketFailure pins which ends retire the shared connection.
519-8 (Low): metric not documented Fixed in 0b84dc9, together with what a dropped client sees on each transport.
519-9 (Nit): outage test comment Fixed in 727179f.

The two questions in the review (errClientReconnected, and the old upstream id after an overflow restore) are still open for you.

Verified on 3b0c0ef:

  • go build ./... and go vet ./... are clean.
  • go test passes for ./protocol/common/, ./protocol/metrics/, ./protocol/chainlib/ and ./protocol/rpcsmartrouter/.
  • The subscription and ingress subset passes under -race. The four new close tests pass 20/20 each under -race.
  • golangci-lint reports 0 issues, both on the touched packages and against main.

…n (MAG-4242)

The connection's rate-limit and idle checker read the WebSocketRateLimit
and WebSocketBanDuration globals from its own goroutine, which nothing
waits for. The globals are process-wide flags, set once at startup, so
this never mattered in production, but it is a data race against any
later write of them: under -race, the new ingress tests in
rpcsmartrouter (the first there to serve real WebSocket connections)
left that read unsynchronised with TestRealCommand_RespCacheFromYAML,
which registers the root command's flags again and so writes both.

ListenToMessages now reads the two settings once, in the connection's
own goroutine, before the checker starts, and the checker and the read
loop use that snapshot. Behaviour is unchanged: nothing writes the
globals after startup.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@AnnaR-prog

Copy link
Copy Markdown
Contributor

One more fix pushed: 01903fb

Found in an integration run of this PR with #520 and #522–#525 on main. Under -race, this PR's new ingress tests expose a data race. They are the first tests in rpcsmartrouter to serve real WebSocket connections.

  • The race: the connection's rate-limit and idle checker read the WebSocketRateLimit and WebSocketBanDuration globals from a goroutine nothing waits for. Nothing orders those reads before TestRealCommand_RespCacheFromYAML, which registers the root command's flags again and so writes both globals.
  • The fix: ListenToMessages now reads the two settings once, in the connection's own goroutine, and the checker and the read loop use that snapshot.
  • No behaviour change: nothing writes these globals after startup.

On the integration tree (main plus all six PRs), all of these are clean:

  • go test ./...: all 38 packages pass;
  • -race on rpcsmartrouter, chainlib, lavasession and the other touched packages: passes, no races;
  • golangci-lint: 0 issues.

@AnnaR-prog AnnaR-prog left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving at 01903fb: main 8ad0ff6 plus this PR and the review fixes, including the 1013 close for a dropped WebSocket client. I validated it together with #520, #522, #523, #524 and #525 on one tree: golangci-lint reports 0 issues, go test ./... passes, and -race is clean on the touched packages. The two questions in my review are still open for a follow-up.

@AnnaR-prog
AnnaR-prog merged commit 9d1c7fc into main Oct 18, 2026
@AnnaR-prog
AnnaR-prog deleted the fix/MAG-4242-subscriptions branch October 18, 2026 08:59
AnnaR-prog added a commit that referenced this pull request Oct 18, 2026
…into refactor/MAG-4279-batch1a

Brings in #523's last two fixes and main after #519 and #520, so this
branch carries the exact #523 that is about to be squash-merged.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
AnnaR-prog added a commit that referenced this pull request Oct 18, 2026
… into refactor/MAG-4279-descriptor-resolver

Brings in #523's last two fixes and main after #519, #520 and #523, so
this branch carries the exact #524 that is about to be squash-merged.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
nimrod-teich pushed a commit that referenced this pull request Nov 3, 2026
… replaced, and keep a redialed connection (MAG-4396) (#591)

* fix(rpcsmartrouter): a client that joins during the last client's unsubscribe gets its own subscription (MAG-4345)

Unsubscribe decided "last client" under the lock, called the node off the
lock, and only then tore the subscription down. The entry stayed in the
active map while the node call was in flight, so a client arriving for the
same params joined it, a local step with no node call. The local cleanup
then saw the joiner's mapping and tore nothing down, but the node had
already dropped the subscription: the joiner got no pushes, and its
channel was never closed, so it waited until it disconnected.

The subscription now leaves the active map in the first critical section,
before the node is called. A subscription the router has told the node to
drop takes no new clients, and a client arriving meanwhile creates a
subscription of its own on the node. A restore that reaches its commit
point while the call is in flight finds nothing to commit onto and ends
the subscription it re-opened (the MAG-2547 discard), so the
restored-since-read branch goes. A node call that fails no longer returns
early with the client still attached: the client asked to leave and the
subscription is already retired, so it ends locally and the error goes
back with the reply. A client that was not the last when its call started
but is by the time the lock is retaken now tells the node, which that
path never did.

teardownSubscriptionLocked splits into retireSubscriptionLocked (the map
entry, identity-guarded, and the counter) and
endUpstreamSubscriptionLocked (the upstream half), so Unsubscribe can run
the two halves apart.

Tests: TestWS_JoinerDuringLastClientUnsubscribeGetsItsOwnSubscription,
from Anna Rayer's attached repro (the arriving client subscribes in the
background, because it now needs the node the test holds), and
TestWS_LastClientUnsubscribeFailingUpstreamStillEndsTheSubscription (the
node applies the unsubscribe but never answers). Both fail on main.
TestUnsubscribeDuringRestore_TellsTheNodeTheRestoredID now asserts the
outcome rather than the path: neither id is left on the node.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Took 20 minutes

* fix(rpcsmartrouter): tell a non-EVM node the unsubscribe method its spec declares (MAG-4278 item 1)

The three router-initiated teardown paths — the last client disconnecting,
a failed restore's cleanup, and the discard of a subscription a restore
re-opened onto an entry that had gone — derived the unsubscribe method
from the subscribe method by string surgery whose fallback is the literal
"unsubscribe". Only Tendermint has that method, so on Solana and every
Substrate chain the node rejected the call, the failure was traced, and
the node kept publishing into the pooled connection: the MAG-3366 leak
shape. The same paths named the subscription with the decimal string the
router canonicalises to internally, which Solana also rejects. The
explicit-unsubscribe path was always right, because the client names the
method.

The method is now resolved from the spec once, when the subscription is
created, and stored on it. The spec gives no field linking a SUBSCRIBE
directive to its UNSUBSCRIBE one, so the pairing is recovered from the
names a collection declares, and only ever returns a name it declares:

  1. the subscribe name with its first subscribe/Subscribe token replaced
     by unsubscribe/Unsubscribe, when the collection declares that name;
  2. for the names rule 1 cannot reach, namespace pairing: within one
     namespace an unclaimed subscribe pairs with an unclaimed
     unsubscribe when each is the only one of its kind there. Uniqueness
     is required in both directions, so an ambiguous namespace resolves
     to nothing rather than to a guess.

Measured over the whole lava-specs catalog (317 subscribe apis in 57
collections): rule 1 resolves 242, rule 2 resolves 63, 12 resolve to
nothing. Nine of those are gRPC or REST subscriptions this manager never
serves; the rest are Enjin's two archive methods, which share a namespace
with one unclaimed unsubscribe between them, and Filecoin.ChainNotify,
for which the FVM spec declares no unsubscribe at all. For those the node
is told nothing: a call the spec says it does not answer cannot stop it
publishing, and a warning naming the method is logged when the
subscription is created.

The id now goes in the chain's own shape on every path, through one
renderer shared with the explicit path. One upstreamSubscriptionTeardown
value carries what the node must be told, snapshotted under the lock, so
the three paths cannot pick different methods or id shapes from each
other again.

getUnsubscribeMethod becomes derivedUnsubscribeMethod and reports failure
instead of answering a name it cannot reach with the bare literal. It is
reached only for a collection that declares no unsubscribe at all.

Tests: TestWS_TeardownOnSolanaSendsTheSpecUnsubscribeMethod, from the
repro Anna Rayer attached to the ticket, strengthened to assert the method
and the unquoted id the node was actually asked for rather than only the
absence of "unsubscribe" — which a router that sends nothing also
satisfies; TestWS_ClientDisconnectOnSubstrateSendsTheSpecUnsubscribeMethod
for the commonest shape, a client closing its socket on chainHead_v1_follow;
TestWS_TeardownTellsTheNodeNothingWhenTheSpecDeclaresNoUnsubscribe;
TestWS_ExplicitUnsubscribeStillSendsTheClientsMethod; and
TestResolveTeardownUnsubscribeMethod over the directive shapes the catalog
declares. All four end-to-end tests fail on main.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Took 1 minute

* fix(rpcsmartrouter): bound a WebSocket subscription's restores and count its pool slot on pick (MAG-4278 item 11)

Two gaps between the WebSocket subscription manager and its gRPC
counterpart, which has carried both since MAG-4177.

A restore was unbounded. Each one spawns a fresh listener, and that
listener's own failure spawns the next restore, so a node that accepts a
subscribe and ends it immediately was re-subscribed for the life of the
process: one round trip per cycle, on every subscription it does this to,
with nothing counting the cycles. A subscription now carries the gRPC
manager's accounting — restoresWithoutProgress against
maxRestoresWithoutProgress, refilled by a restore that stayed up for
restoreStableAfter — and is given up once it is spent. Uptime rather than
a delivered push is the measure of progress, so a quiet subscription does
not spend its budget on the reconnects a load balancer's connection age
limit causes.

A subscription's pool slot was counted after the node answered the
subscribe. In that window the connection carries no counted subscription,
and maybeScaleDown closes a connection past minConnections that carries
none, so the subscription being created on it died on a socket the pool
had just closed. The pool gains ReserveConnectionForSubscription, which
counts the slot under the same lock as the pick, and every failure after
the reservation releases it: a failed subscribe, a reply that cannot be
built, and a failed re-subscribe on the restore path. GetConnection and
GetConnectionForSubscription keep their behaviour for callers that do not
open a subscription.

Tests: TestRestoreBudget_GivesUpOnceTheBudgetIsSpent and
TestSubscribeReservesItsPoolSlot fail on main.
TestRestoreBudget_RefillsAfterASubscriptionStaysUp and
TestSubscribeReleasesItsPoolSlotWhenTheSubscribeFails are guards for the
refill and the release rather than repros, and pass either way. The mock
node gained rejectSubscribes, which answers a subscribe with a JSON-RPC
error.

Still open on the ticket's item 11, and not in this change: the WS
manager closes a given-up subscription's client channel with no status,
where the gRPC manager ends the client's stream with a reason. Giving
WebSocket clients one needs a new SubscriptionEndError reason and a
close-code decision in the ingress, which is client-visible behaviour
rather than a drift fix.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Took 2 minutes

* fix(rpcsmartrouter): tell the node to drop the subscription a restore replaced, and keep a redialed connection (MAG-4396)

The two leftovers PR #519's review left open.

A restore that runs on a live connection never told the node to drop the
subscription it replaced. rpcclient ends one subscription on a connection
that stays up when its forward queue overflows, and
ClientSubscription.run discards the unsubscribe-the-server flag forward
returns, so rpcclient sends nothing; handleUpstreamDisconnect used the
old upstream id only for logging and for dropping its id mappings. So the
node kept publishing the replaced subscription on the same socket,
rpcclient read those frames and dropped them, and a subscription that
overflows repeatedly left one more of them behind each time. The restore
now tells the node, through the same teardown the other paths use. Where
the restore followed a lost socket the node lost the subscription with
it, and teardownUpstreamSubscription skips a dead connection, so one call
is right on both paths.

The local rpcclient subscription is deliberately not unsubscribed with
it. Unsubscribe closes its Err channel, and a listener still parked there
reads the close as a nil error, ends, and runs its deferred cleanup —
tearing down the subscription the restore just installed. The listener
that handed off has already returned, so nothing is parked there on the
restore path, but the margin is one scheduling decision wide and the only
thing it buys is dropping rpcclient's handler entry for a replaced
subscription, which closing the connection does anyway.

The decision the ticket asks for: errClientReconnected leaves the pooled
connection alone. PR #519 made every subscription error except a queue
overflow or a node error retire the connection, and this is the one case
where the client has already recovered — rpcclient replaced the socket
itself and holds a working one. Retiring it there closes that client, so
the pool dials again for nothing and every other subscription sharing the
connection ends too, including one a restore has just re-created on the
new socket. A real socket failure still retires it.

errClientReconnected becomes ErrClientReconnected: the fact that the
connection is healthy is not readable from anywhere else, and a caller
that treats every subscription error as a dead socket has no way to tell.

Tests: TestOverflowRestore_TellsTheNodeToDropTheOldSubscription,
TestClientRedial_KeepsTheConnectionAndItsSiblingSubscriptions and
TestRetireConnectionOnUpstreamError_LeavesAClientThatRedialed fail on
main. TestRestoreAfterADroppedSocket_DoesNotCallADeadNode guards the
other half of the same call. The redial is driven through the
upstreamErrSource seam the listener documents for it: rpcclient mints the
error internally, and the MAG-4037 trigger, a call whose deadline has
already passed, fails the redial on that same expired deadline, so no
reconnect lands.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(rpcsmartrouter): tell the node to drop a replaced subscription only after a queue overflow (MAG-4396)

The restore told the node to drop the subscription it replaced after every
restore on a healthy connection. Only a queue overflow leaves that
subscription on a live socket. When rpcclient redials on its own
(ErrClientReconnected), it closes the old socket and the connection is now
kept, so the goodbye went out over the NEW socket. On Tendermint the
subscription id is the query, so `unsubscribe {query}` ended the
subscription the restore had just opened: every client of every Tendermint
subscription on that connection was left on a subscription the node no
longer publishes, with its channel open. On other chains the goodbye named
an id the new socket does not know, or, on a node that numbers its
subscriptions per socket, a sibling the redial had just restored.

The restore now receives the error that ended the subscription and says
goodbye only after a queue overflow, and never when the old id names the
restored subscription on the same connection.

TestTendermintRedial_KeepsTheRestoredSubscriptionOnTheNode and
TestClientRedial_RestoresEverySubscriptionOnTheKeptConnection drive a real
redial (calls past their deadline, then a live call) and fail on b784643.
TestReplacedStillOnNode pins the decision.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix(rpcsmartrouter): an unsubscribe does not tear down a subscription that ended under it (MAG-4345)

A client that is not the last one releases the lock between Unsubscribe's
two critical sections. If the subscription ends there by another path —
the node ends it with an error, or a restore gives it up — the listener's
cleanup releases every client, stops counting the subscription and tells
the node. Unsubscribe then retook the lock, found no clients left, and tore
the subscription down again: totalSubscriptions and the pooled
connection's count each went to -1 and the node was told twice. On main
the mapper's last-client answer, false once the cleanup had removed every
mapping, kept it from doing so; the retire/end split dropped that guard.

The teardown now runs only while the subscription is still the live entry
under its params. beforeUnsubscribeRelease is a test hook, nil in
production, in the shape of beforeRestoreCommit.

TestWS_UnsubscribeDoesNotTearDownASubscriptionThatEndedUnderIt fails
without the guard (totalSubscriptions -1, pool count -1, two
eth_unsubscribe) and passes on main's manager with only the hook added.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix(rpcsmartrouter): hand the disconnect watcher its client's signal when the client attaches (MAG-4345)

The watcher a subscribe starts looked its subscription up when the
goroutine first ran, to read the client's detached signal. Unsubscribe now
takes the subscription out of the map in its first step, so a client that
unsubscribed before its watcher was scheduled left the watcher with a nil
channel: it stayed parked until the socket closed, one goroutine per
subscribe and unsubscribe on a long-lived socket. On main the entry stayed
mapped for the node's round trip, which hid the race.

The signal is now read where the client is attached — by the creator while
it still owns the subscription, by a joiner under the manager lock — and
handed to the watcher.

TestUnsubscribeThenDisconnect_LeavesNothingBehind ("the disconnect watcher
must not stay parked once the client unsubscribed") failed 6 of 20 runs
without this and passes 30 of 30 with it; it passes 20 of 20 on main.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* test(rpcsmartrouter): pin that a WebSocket subscription counts its pool slot while the node answers (MAG-4278 item 11)

TestSubscribeReservesItsPoolSlot drives the pool's
ReserveConnectionForSubscription on its own. The manager calls it in two
places, StartSubscription and the restore, and with both of them reverted to
GetConnection plus an IncrementSubscriptions after the node's answer every
test in the package still passed, so nothing pinned the fix the window was
about.

TestSubscriptionCountsItsPoolSlotWhileTheNodeAnswers holds the node's answer
to a client's subscribe, then to the restore's re-subscribe after the socket
drops, and requires the pool to count the subscription in both windows. It
fails with either call site reverted.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* test(rpcsmartrouter): let the restore-budget refill test wait for the restore to commit (MAG-4278 item 11)

TestRestoreBudget_RefillsAfterASubscriptionStaysUp waited for
`upstreamID != "" && upstreamCalls("eth_subscribe") == 2`. The old upstream
id is still set during the restore, so the wait ended as soon as the node
had read the re-subscribe, before it answered and before the restore
committed. The test then returned, and its cleanup cancelled the
subscription's context with the re-subscribe possibly still in flight:
the window in which rpcclient's Subscribe can deadlock its dispatch loop
(an unbuffered reply channel the cancelled waiter no longer reads), after
which manager.Close never returns.

It now waits for the upstream id to change, which is the restore's commit.
The budget tests still fail with the budget removed.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Anna Rayer <anna@magmadevs.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants