Skip to content

Commit cc92100

Browse files
authored
fix: stop the router heap growing for the life of the process (#386)
* fix(provideroptimizer): bound the per-provider relay time history updateRelayTime appended a time.Time to a per-provider slice on every relay, success or failure, and nothing ever truncated it: only the debug reset endpoints clear the cache. That is 24 bytes per relay for the life of the process, so a busy pod's heap climbed for weeks until the container limit killed it. Each reallocation also copied the whole history under the provider's stripe lock, and the slice's Location pointers made the GC walk all of it every cycle. calculateHalfTime reads exactly one element, the median, so the history is now a fixed 128-slot ring in arrival order. Its median means "the time this provider takes to serve half a window of relays", which is the input the half-life heuristic wants; the unbounded median drifted to half the uptime and the MaxHalfTime clamp discarded it anyway. The ring is mutated in place under the stripe lock, so the cached pointer never needs a replacement Set. Jira ticket: MAG-3722 * fix(ws): release a client's subscriptions and writer when its connection ends Three ways a websocket client's departure left state behind for the life of the process, all on the same path. The subscription manager never released a JOINED client. A second subscriber to a query joins the first one's upstream subscription, and only the creating client had a disconnect watcher: when a joiner closed its socket its reply channel stayed open, which parks the ingress forwarder goroutine ranging over it, it kept counting as connected, and the upstream subscription outlived every real subscriber. The creator's watcher had the mirror omission: it deleted the client's sender without closing it, so its forwarder parked too. Every client now attaches through one helper and leaves through one helper that closes the channel, drops the bookkeeping and wakes the watcher, so no path can skip a step again. An explicit unsubscribe wakes the watcher instead of leaving it parked until the socket closes, and UnsubscribeAll now tells the node when the last client leaves, which it used to skip. The shared subscription's context was derived from the creating client's connection, so the creator leaving ended the upstream listener underneath everyone who had joined. It is now owned by the manager, the rule the gRPC manager already applies. The ingress writer goroutine returned on a failed write without telling anyone. Every later frame sender then blocked forever, and the read loop's own error path is one of them, so the handler never returned and fasthttp kept the 128 KiB read buffer, the handler goroutine, the ticker goroutine and the per-IP limiter slot. The writer now closes a done channel on the way out, senders select on it, and a failed write sets an immediate read deadline so the read loop wakes. The handler also releases every subscription it holds through UnsubscribeAll on return, synchronously, rather than relying on a cancelled context being noticed. Smaller entries that never went away: the id mapper's per-client counter (one per connection that ever subscribed) is dropped with the client, and rate limiters idle for their refill time are swept, since such an entry decides exactly as a fresh one would. Jira ticket: MAG-3722 * fix(rpcclient): drop a subscription's dispatch entry on unsubscribe The handler's subscription table only shrank when the connection closed. Upstream websocket connections are pooled and live for the life of the process, so every subscription ever torn down on one left its entry behind. Unsubscribe now asks the dispatch loop, the table's only owner, to forget the entry, on a goroutine of its own because dispatch may be busy delivering to another subscription whose consumer waits on a lock the caller holds. Jira ticket: MAG-3722 * fix(relaycore): never block on the return-condition signal The state machine reads returnCondition at most once and may already have left on another branch, yet every validateReturnCondition goroutine sent into the buffer-1 channel unconditionally. A third stop decision inside one 15 ms burst parked its goroutine forever, pinning the whole state machine and the request it carries. The send is now non-blocking; only the first reason can matter. Jira ticket: MAG-3722 * fix(metrics): prune stale epochs from the optimizer stake map The stake map kept one entry per provider per epoch and nothing removed them. Reads only ever ask for the epoch being scored, so the current and previous epoch are all that is kept now. Jira ticket: MAG-3722 * fix(ws): release per-client state on every exit path, number router ids per mapper Review follow-ups on the client release. A client that unsubscribed everything explicitly and then closed its socket still leaked: the explicit unsubscribe emptied its subscription set, and the connection-close release returned early on the empty set before handing back the sticky endpoint and rate limiters. The early return now releases them. Unsubscribe's last-client branch and Close go through the same teardown and release helpers as every other path, so a subscription retires one way only, and Unsubscribe documents why it trusts the mapper's count. Router ids are numbered by one mapper-wide counter instead of one per client. The per-client counter map was the leak on both the ws and the gRPC manager, and it needed a release call on every exit path to stay bounded; a single counter needs nothing released and never reuses an id for the life of the process, so the reuse caveat on an explicit unsubscribe_all goes away too. The id format is unchanged; only the numbering is mapper-wide. Jira ticket: MAG-3722 * fix(provideroptimizer): keep the relay windows in a locked map, state the half-life change Review follow-ups on the relay-time window. The windows live in a plain mutex-guarded map rather than a ristretto cache: the set is bounded by the pairing list, nothing in it should be evicted, and ristretto's asynchronous Set let the relay right after a window's birth miss it and start a second window that dropped the samples in between. The window grows one sample at a time and wraps once full, so a provider that served a handful of relays costs a handful of entries. The comment now says what the change does to routing: with the unbounded history the median sat at half the uptime, so every provider on a pod up more than six hours decayed on the 3 h clamp; a busy provider now decays on the 1 h default, which is what the heuristic was written to do. A new test pins the values at three relay rates. Jira ticket: MAG-3722 * docs(rpcclient): describe the id-bearing case in forgetClientSub A late push without a request id is dropped as before; one with a request id now reaches the call path and is answered with an error upstream, where the stale entry used to swallow it. Jira ticket: MAG-3722
1 parent e2aee89 commit cc92100

18 files changed

Lines changed: 1279 additions & 251 deletions

‎protocol/chainlib/chainproxy/rpcclient/client.go‎

Lines changed: 30 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -104,14 +104,15 @@ type Client struct {
104104

105105
// for dispatch
106106
close chan struct{}
107-
closing chan struct{} // closed when client is quitting
108-
didClose chan struct{} // closed when client quits
109-
reconnected chan ServerCodec // where write/reconnect sends the new connection
110-
readOp chan readOp // read messages
111-
readErr chan error // errors from read
112-
reqInit chan *requestOp // register response IDs, takes write lock
113-
reqSent chan error // signals write completion, releases write lock
114-
reqTimeout chan *requestOp // removes response IDs when call timeout expires
107+
closing chan struct{} // closed when client is quitting
108+
didClose chan struct{} // closed when client quits
109+
reconnected chan ServerCodec // where write/reconnect sends the new connection
110+
readOp chan readOp // read messages
111+
readErr chan error // errors from read
112+
reqInit chan *requestOp // register response IDs, takes write lock
113+
reqSent chan error // signals write completion, releases write lock
114+
reqTimeout chan *requestOp // removes response IDs when call timeout expires
115+
subDone chan *ClientSubscription // drops a subscription's dispatch entry after Unsubscribe
115116
}
116117

117118
type reconnectFunc func(ctx context.Context) (ServerCodec, error)
@@ -214,6 +215,7 @@ func initClient(conn ServerCodec, idgen func() ID, services *serviceRegistry) *C
214215
reqInit: make(chan *requestOp),
215216
reqSent: make(chan error, 1),
216217
reqTimeout: make(chan *requestOp),
218+
subDone: make(chan *ClientSubscription),
217219
}
218220
if !isHTTP {
219221
go c.dispatch(conn)
@@ -724,10 +726,30 @@ func (c *Client) dispatch(codec ServerCodec) {
724726

725727
case op := <-c.reqTimeout:
726728
conn.handler.removeRequestOp(op)
729+
730+
case sub := <-c.subDone:
731+
conn.handler.forgetClientSub(sub)
727732
}
728733
}
729734
}
730735

736+
// forgetSubscription asks the dispatch loop to drop the handler's entry for sub. Until
737+
// now that entry lived until the connection closed, one per subscription ever torn
738+
// down on a pooled connection (MAG-3722). It runs on its own goroutine because
739+
// dispatch may be busy delivering to another subscription whose consumer is waiting
740+
// on a lock the Unsubscribe caller holds.
741+
func (c *Client) forgetSubscription(sub *ClientSubscription) {
742+
if c == nil || c.isHTTP {
743+
return
744+
}
745+
go func() {
746+
select {
747+
case c.subDone <- sub:
748+
case <-c.closing:
749+
}
750+
}()
751+
}
752+
731753
// drainRead drops read messages until an error occurs.
732754
func (c *Client) drainRead() {
733755
for {

‎protocol/chainlib/chainproxy/rpcclient/handler.go‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -529,6 +529,21 @@ func (h *handler) runMethod(ctx context.Context, msg *JsonrpcMessage, callb *cal
529529
return msg.response(result)
530530
}
531531

532+
// forgetClientSub drops sub from the dispatch table once its forwarding loop has been
533+
// told to quit. Runs on the dispatch goroutine, which is the only owner of clientSubs.
534+
// A push for the id that still arrives then reaches deliverSubscriptionPush as an
535+
// unmatched frame: without a request id it is dropped, the fate it met while the entry
536+
// existed; with one it goes to the call path, which answers the node with a JSON-RPC
537+
// error where the entry used to swallow it.
538+
func (h *handler) forgetClientSub(sub *ClientSubscription) {
539+
if sub == nil {
540+
return
541+
}
542+
if current, ok := h.clientSubs[sub.subid]; ok && current == sub {
543+
delete(h.clientSubs, sub.subid)
544+
}
545+
}
546+
532547
// unsubscribe is the callback function for all *_unsubscribe calls.
533548
func (h *handler) unsubscribe(ctx context.Context, id ID) (bool, error) {
534549
h.subLock.Lock()

‎protocol/chainlib/chainproxy/rpcclient/subscription.go‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -274,6 +274,7 @@ func (sub *ClientSubscription) Unsubscribe() {
274274
// Unsubscribe already requested (or subscription already closed).
275275
}
276276
close(sub.err)
277+
sub.client.forgetSubscription(sub)
277278
})
278279
}
279280

Lines changed: 141 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,141 @@
1+
package rpcclient
2+
3+
import (
4+
"context"
5+
"encoding/json"
6+
"reflect"
7+
"sync"
8+
"testing"
9+
"time"
10+
11+
"github.com/stretchr/testify/require"
12+
)
13+
14+
// TestForgetClientSub_DropsOnlyThatSubscription pins the identity check: a later
15+
// subscription registered under a reused id must not be removed by a stale unsubscribe.
16+
func TestForgetClientSub_DropsOnlyThatSubscription(t *testing.T) {
17+
h := newTestHandler()
18+
ch := make(chan *JsonrpcMessage)
19+
old := newClientSubscription(nil, "eth", reflect.ValueOf(ch))
20+
old.subid = "0xabc"
21+
h.clientSubs[old.subid] = old
22+
23+
h.forgetClientSub(old)
24+
require.NotContains(t, h.clientSubs, "0xabc")
25+
26+
replacement := newClientSubscription(nil, "eth", reflect.ValueOf(ch))
27+
replacement.subid = "0xabc"
28+
h.clientSubs[replacement.subid] = replacement
29+
h.forgetClientSub(old) // stale
30+
require.Same(t, replacement, h.clientSubs["0xabc"], "a stale unsubscribe must not evict the live subscription")
31+
32+
h.forgetClientSub(nil) // must not panic
33+
}
34+
35+
// scriptedCodec is an in-memory ServerCodec: the test feeds it frames to read and
36+
// records what the client writes.
37+
type scriptedCodec struct {
38+
incoming chan []*JsonrpcMessage
39+
closedCh chan interface{}
40+
once sync.Once
41+
42+
mu sync.Mutex
43+
written []*JsonrpcMessage
44+
}
45+
46+
func newScriptedCodec() *scriptedCodec {
47+
return &scriptedCodec{incoming: make(chan []*JsonrpcMessage, 16), closedCh: make(chan interface{})}
48+
}
49+
50+
func (c *scriptedCodec) peerInfo() PeerInfo { return PeerInfo{} }
51+
func (c *scriptedCodec) readBatch() ([]*JsonrpcMessage, bool, error) {
52+
select {
53+
case msgs := <-c.incoming:
54+
return msgs, false, nil
55+
case <-c.closedCh:
56+
return nil, false, ErrClientQuit
57+
}
58+
}
59+
func (c *scriptedCodec) close() { c.once.Do(func() { close(c.closedCh) }) }
60+
func (c *scriptedCodec) closed() <-chan interface{} { return c.closedCh }
61+
func (c *scriptedCodec) remoteAddr() string { return "scripted" }
62+
func (c *scriptedCodec) writeJSON(_ context.Context, v interface{}) error {
63+
c.mu.Lock()
64+
defer c.mu.Unlock()
65+
if msg, ok := v.(*JsonrpcMessage); ok {
66+
c.written = append(c.written, msg)
67+
}
68+
return nil
69+
}
70+
71+
func (c *scriptedCodec) writtenCount() int {
72+
c.mu.Lock()
73+
defer c.mu.Unlock()
74+
return len(c.written)
75+
}
76+
77+
func (c *scriptedCodec) lastWritten() *JsonrpcMessage {
78+
c.mu.Lock()
79+
defer c.mu.Unlock()
80+
if len(c.written) == 0 {
81+
return nil
82+
}
83+
return c.written[len(c.written)-1]
84+
}
85+
86+
// TestUnsubscribe_ForgetsDispatchEntry is the regression for MAG-3722's rpcclient leak:
87+
// the dispatch table kept an entry for every subscription ever unsubscribed until the
88+
// connection closed, and on a pooled upstream connection that is the life of the process.
89+
//
90+
// The entry is observed through behaviour rather than a hook. A push that names the
91+
// subscription and carries a request id is claimed silently while the entry exists, and
92+
// handed to the call path once it is gone, where the handler answers it. So the answer
93+
// appearing on the wire is the entry disappearing.
94+
func TestUnsubscribe_ForgetsDispatchEntry(t *testing.T) {
95+
codec := newScriptedCodec()
96+
client, err := newClient(context.Background(), func(context.Context) (ServerCodec, error) { return codec, nil })
97+
require.NoError(t, err)
98+
defer client.Close()
99+
100+
// Answer the subscribe request as the node would.
101+
go func() {
102+
require.Eventually(t, func() bool { return codec.writtenCount() == 1 }, 5*time.Second, 5*time.Millisecond)
103+
req := codec.lastWritten()
104+
codec.incoming <- []*JsonrpcMessage{{Version: Vsn, ID: req.ID, Result: json.RawMessage(`"0xabc"`)}}
105+
}()
106+
ch := make(chan *JsonrpcMessage, 8)
107+
sub, _, err := client.Subscribe(context.Background(), nil, "eth_subscribe", ch, []interface{}{"newHeads"})
108+
require.NoError(t, err)
109+
require.NotNil(t, sub)
110+
111+
// A Substrate-shaped push: the handler routes it by the params envelope, and that
112+
// route is the one that hands an unmatched id-bearing frame back to the call path.
113+
// (An eth_subscription frame is claimed by its method name whether or not the
114+
// table still has the entry, so it cannot show the entry going away.)
115+
push := func() {
116+
codec.incoming <- []*JsonrpcMessage{{
117+
Version: Vsn,
118+
ID: json.RawMessage(`99`),
119+
Method: "chain_newHead",
120+
Params: json.RawMessage(`{"subscription":"0xabc","result":{"number":"0x1"}}`),
121+
}}
122+
}
123+
124+
push()
125+
select {
126+
case <-ch:
127+
case <-time.After(5 * time.Second):
128+
t.Fatal("a push must reach the subscriber while it is live")
129+
}
130+
require.Equal(t, 1, codec.writtenCount(), "a claimed push is not answered")
131+
132+
sub.Unsubscribe()
133+
134+
require.Eventually(t, func() bool {
135+
push()
136+
return codec.writtenCount() > 1
137+
}, 5*time.Second, 10*time.Millisecond,
138+
"once the entry is forgotten an id-bearing push is handed to the call path and answered")
139+
answer := codec.lastWritten()
140+
require.NotNil(t, answer.Error, "the call path answers an unknown method with an error")
141+
}

‎protocol/chainlib/consumer_websocket_manager.go‎

Lines changed: 84 additions & 49 deletions
Original file line numberDiff line numberDiff line change
@@ -105,6 +105,75 @@ func (cwm *ConsumerWebsocketManager) handleRateLimitReached(inpData []byte) ([]b
105105
return bytesRateLimitError, nil
106106
}
107107

108+
// webSocketMsgWithType is one frame queued for the connection's writer goroutine.
109+
type webSocketMsgWithType struct {
110+
messageType int
111+
msg []byte
112+
}
113+
114+
// websocketFrameWriter is the slice of the connection the writer goroutine touches.
115+
// *websocket.Conn satisfies it; tests hand in a fake whose writes fail on demand.
116+
type websocketFrameWriter interface {
117+
WriteMessage(messageType int, data []byte) error
118+
SetReadDeadline(t time.Time) error
119+
}
120+
121+
// runWebsocketWriter is the connection's single writer. Single-writer discipline
122+
// guarantees WriteMessage is never called concurrently, which the underlying
123+
// gorilla/fasthttp websocket library does not allow.
124+
//
125+
// It serves regular frames and the server-shutdown close frame, and closes writerDone
126+
// on its way out. A failed write ends it, and that used to leave every later
127+
// enqueueWebsocketFrame caller blocked forever: the read loop's own error path sends a
128+
// frame before it can return, so the handler never returned, and fasthttp kept the
129+
// 128 KiB read buffer, the handler goroutine and the per-IP limiter slot for the life
130+
// of the process (MAG-3722). Now a failed write also sets an immediate read deadline so
131+
// the read loop wakes and the handler tears the connection down.
132+
func runWebsocketWriter(ctx, webSocketCtx context.Context, conn websocketFrameWriter, frames <-chan webSocketMsgWithType, writerDone chan<- struct{}) {
133+
defer close(writerDone)
134+
for {
135+
select {
136+
case <-ctx.Done():
137+
// Server-wide shutdown — send a CloseGoingAway (1001) frame so the client can
138+
// distinguish an intentional shutdown from a crash, then unblock the read loop by
139+
// setting an immediate read deadline rather than Close()-ing the conn here.
140+
//
141+
// Close() from this goroutine raced gofiber's handler cleanup: it unblocks ReadMessage,
142+
// the handler returns, and gofiber recycles the Conn (fasthttp pooling) WHILE this
143+
// goroutine is still inside Close() — a use-after-recycle data race. SetReadDeadline
144+
// returns immediately (the read loop breaks on the resulting error) and lets the handler,
145+
// the conn's sole owner, do the single Close on return.
146+
_ = conn.WriteMessage(websocket.CloseMessage, websocket.FormatCloseMessage(websocket.CloseGoingAway, "server shutting down"))
147+
_ = conn.SetReadDeadline(time.Now())
148+
return
149+
case <-webSocketCtx.Done():
150+
utils.LavaFormatTrace("websocket's context cancelled", utils.LogAttr("GUID", webSocketCtx))
151+
return
152+
case msg, ok := <-frames:
153+
if !ok {
154+
return
155+
}
156+
if err := conn.WriteMessage(msg.messageType, msg.msg); err != nil {
157+
utils.LavaFormatTrace("error writing msg to the websocket", utils.LogAttr("err", err))
158+
_ = conn.SetReadDeadline(time.Now())
159+
return
160+
}
161+
}
162+
}
163+
}
164+
165+
// enqueueWebsocketFrame hands a frame to the writer goroutine, but gives up as soon as
166+
// ctx (server shutdown) or webSocketCtx (per-connection cleanup) cancels or the writer
167+
// is gone: a plain send would park the caller for good.
168+
func enqueueWebsocketFrame(ctx, webSocketCtx context.Context, writerDone <-chan struct{}, frames chan<- webSocketMsgWithType, msg webSocketMsgWithType) {
169+
select {
170+
case frames <- msg:
171+
case <-ctx.Done():
172+
case <-webSocketCtx.Done():
173+
case <-writerDone:
174+
}
175+
}
176+
108177
func (cwm *ConsumerWebsocketManager) ListenToMessages(ctx context.Context) {
109178
// adding metrics for how many active connections we have.
110179
cwm.rpcConsumerLogs.SetWebSocketConnectionActive(cwm.chainId, cwm.apiInterface, true)
@@ -116,12 +185,9 @@ func (cwm *ConsumerWebsocketManager) ListenToMessages(ctx context.Context) {
116185
err error
117186
)
118187

119-
type webSocketMsgWithType struct {
120-
messageType int
121-
msg []byte
122-
}
123-
124188
websocketConnWriteChan := make(chan webSocketMsgWithType)
189+
// Closed by the writer goroutine on its way out.
190+
writerDone := make(chan struct{})
125191

126192
websocketConn := cwm.websocketConn
127193
logger := cwm.rpcConsumerLogs
@@ -132,57 +198,26 @@ func (cwm *ConsumerWebsocketManager) ListenToMessages(ctx context.Context) {
132198
webSocketCtx = utils.WithUniqueIdentifier(webSocketCtx, guid)
133199
utils.LavaFormatDebug("consumer websocket manager started", utils.LogAttr("GUID", webSocketCtx))
134200
defer func() {
201+
// The connection is over. Release every subscription this client still holds
202+
// right here, synchronously: reply channels close and the forwarder goroutines
203+
// below end now, not whenever a cancelled context happens to be noticed. dappID
204+
// and the remote address are per-connection, so this key matches every
205+
// subscribe the loop registered.
206+
if cwm.wsSubscriptionManager != nil {
207+
dappID, _ := websocketConn.Locals(ProjectIDHeader).(string)
208+
if err := cwm.wsSubscriptionManager.UnsubscribeAll(webSocketCtx, dappID, websocketConn.RemoteAddr().String(), cwm.WebsocketConnectionUID, nil); err != nil {
209+
utils.LavaFormatDebug("error releasing subscriptions on websocket close", utils.LogAttr("err", err), utils.LogAttr("GUID", webSocketCtx))
210+
}
211+
}
135212
cancelWebSocketCtx() // In case there's a problem make sure to cancel the connection
136213
utils.LavaFormatDebug("consumer websocket manager stopped", utils.LogAttr("GUID", webSocketCtx))
137214
}()
138215

139-
// sendWS enqueues a frame for the writer goroutine, but unblocks if either
140-
// ctx (server shutdown) or webSocketCtx (per-conn cleanup) cancels first —
141-
// the writer exits on those signals, so a plain send would deadlock the
142-
// caller and leave the connection goroutine stuck (preventing wsWG.Done()
143-
// from firing during graceful shutdown).
144216
sendWS := func(msg webSocketMsgWithType) {
145-
select {
146-
case websocketConnWriteChan <- msg:
147-
case <-ctx.Done():
148-
case <-webSocketCtx.Done():
149-
}
217+
enqueueWebsocketFrame(ctx, webSocketCtx, writerDone, websocketConnWriteChan, msg)
150218
}
151219

152-
// Single writer goroutine: serves both regular messages and the server-shutdown
153-
// close frame. Single-writer discipline guarantees we never call WriteMessage
154-
// concurrently — the underlying gorilla/fasthttp websocket library is not safe
155-
// for concurrent writes.
156-
go func() {
157-
for {
158-
select {
159-
case <-ctx.Done():
160-
// Server-wide shutdown — send a CloseGoingAway (1001) frame so the client can
161-
// distinguish an intentional shutdown from a crash, then unblock the read loop by
162-
// setting an immediate read deadline rather than Close()-ing the conn here.
163-
//
164-
// Close() from this goroutine raced gofiber's handler cleanup: it unblocks ReadMessage,
165-
// the handler returns, and gofiber recycles the Conn (fasthttp pooling) WHILE this
166-
// goroutine is still inside Close() — a use-after-recycle data race. SetReadDeadline
167-
// returns immediately (the read loop breaks on the resulting error) and lets the handler,
168-
// the conn's sole owner, do the single Close on return.
169-
_ = cwm.websocketConn.WriteMessage(websocket.CloseMessage, websocket.FormatCloseMessage(websocket.CloseGoingAway, "server shutting down"))
170-
_ = cwm.websocketConn.SetReadDeadline(time.Now())
171-
return
172-
case <-webSocketCtx.Done():
173-
utils.LavaFormatTrace("websocket's context cancelled", utils.LogAttr("GUID", webSocketCtx))
174-
return
175-
case msg, ok := <-websocketConnWriteChan:
176-
if !ok {
177-
return
178-
}
179-
if err := cwm.websocketConn.WriteMessage(msg.messageType, msg.msg); err != nil {
180-
utils.LavaFormatTrace("error writing msg to the websocket")
181-
return
182-
}
183-
}
184-
}
185-
}()
220+
go runWebsocketWriter(ctx, webSocketCtx, cwm.websocketConn, websocketConnWriteChan, writerDone)
186221

187222
// set up a routine to check for rate limits or idle time
188223
idleFor := atomic.Int64{}

0 commit comments

Comments
 (0)