Repository navigation
fix: stop the router heap growing for the life of the process - #386
Conversation
avitenzer
left a comment
There was a problem hiding this comment.
Review: verified locally, two leaks still open + one undisclosed behaviour change
Thanks — the structural fix here is the right shape (one attach funnel, one release funnel, one teardown), and the tests are genuinely behavioural rather than hook-driven.
What I verified myself, at 682e29d:
| Check | Result |
|---|---|
go test ./... -count=1 (whole tree) |
clean |
-race -count=1, all 6 touched packages |
clean (optimizer 531s, rpcsmartrouter 113s) |
go build ./protocol/... + go vet |
clean |
| PR's "How to verify" commands | 6/6 confirmed |
The 3 ws release tests run against the merge base a545be5 |
3/3 FAIL, for exactly the stated reasons |
| Helm chart repo: refs to any changed flag/key/metric | 0 (control passes) — no chart change, no CrashLoop risk |
That merge-base check is the important one: the "fails against the previous code" claim holds up.
Blocking
- Ticket item 5 (
clientCounters) is still open on the commonest ws path.UnsubscribeAllearly-returns beforeforgetClientLocked, so subscribe →eth_unsubscribe→ socket close leaks the id counter and the sticky entry, per connection, forever. Proven with a probe test on this branch:idMapper.ClientCount()=1,stickyStore.Len()=1after the client is fully gone. One-line fix, verified green — see inline. - Same item, entirely unfixed on the gRPC side.
DirectGRPCSubscriptionManagerowns its ownSubscriptionIDMapperand never calls the newRemoveClient. See inline. - The optimizer change alters routing, and the PR says it doesn't.
DefaultHalfLifeTime=1h,MaxHalfTime=3h. Before: median ≈ uptime/2, so any pod up >6h hadhalfTimepinned at 3h for every provider. After: 1h for any busy provider. That is a 3× faster QoS decay in production. See inline.
Unproven
- Heap actually plateaus — only observable on a deployed pod; nothing here or in CI measures memory.
- Routing impact of 3h→1h —
clock_injection_test.go:205deliberately clearsproviderRelayStats, so the existing suite is structurally blind to it.
Worth knowing
- CI runs no tests on this repo. 21 checks — build ×8, Docker, lint, govulncheck, GitGuardian, Jira — and zero
go test. Not this PR's fault, but it means all test evidence for 1,115 lines is local. - Six subsystems in one PR makes revert all-or-nothing, and the riskiest piece (the optimizer,
1713448) is the most separable. Worth considering splitting it. #355confirmed open withbase=mag-3722, exactly as the body says.
Nothing below is a style nit unless I've labelled it one.
| utils.LavaFormatTrace("DirectWS: cleared sticky session and rate limiter on unsubscribe all", | ||
| utils.LogAttr("clientKey", clientKey), | ||
| ) | ||
| dwsm.forgetClientLocked(clientKey) |
There was a problem hiding this comment.
Blocking — ticket item 5 is still open on the commonest client path.
This line is unreachable for a client that unsubscribed before disconnecting. UnsubscribeAll returns ~25 lines above:
clientSubs, ok := dwsm.connectedClients[clientKey]
if !ok {
return nil // No subscriptions for this client
}The sequence subscribe → eth_unsubscribe → close socket deletes connectedClients[clientKey] in forgetSubscriptionForClientLocked (with clientGone=false, so it skips cleanup by design), and then the connection-close path that was supposed to finish the job bails out here. The client's own watcher doesn't cover it either — releaseClientLocked already closed its detached channel at unsubscribe time, so it took the early-exit branch.
Verified on this branch with a probe test:
after unsubscribe+disconnect: idMapper.ClientCount()=1 rateLimiter.ClientCount()=2 stickyLen=1
Leaked permanently, per connection: idMapper.clientCounters (this is MAG-3722 item 5 verbatim) and stickyStore. No sweep reaches either — the orphan loop in performCleanup iterates connectedClients, whose key is already gone, and SweepIdle only covers rate limiters (those do get cleaned within ~2 min, so that part is fine).
One line fixes it, and I confirmed the probe goes green:
clientSubs, ok := dwsm.connectedClients[clientKey]
if !ok {
dwsm.forgetClientLocked(clientKey)
return nil
}None of the three new tests cover this shape — TestUnsubscribeAll_ReleasesWatchers... calls UnsubscribeAll while subs are still live, and the disconnect tests never send an explicit unsubscribe. Worth a fourth case asserting idMapper.ClientCount()==0 and stickyStore.Len()==0.
There was a problem hiding this comment.
Confirmed and fixed in 31de994: the empty-set branch now calls forgetClientLocked before returning, so the connection-close release hands back the sticky endpoint and rate limiters for a client that unsubscribed first.
TestUnsubscribeThenDisconnect_LeavesNothingBehind is the shape you described (subscribe, eth_unsubscribe, close the socket). It asserts stickyStore.Len() == 0, rateLimiter.ClientCount() == 0, no connected-client entry, and exactly one upstream eth_unsubscribe. Ran it with that one line removed: it fails on the sticky assertion, so it pins the fix.
The id-counter half of your probe no longer exists; see the mapper thread.
| // subscriptions: the key carries the per-connection GUID, so nothing reuses it, | ||
| // and without this call the map kept one entry per connection that ever | ||
| // subscribed (MAG-3722). | ||
| func (m *SubscriptionIDMapper) RemoveClient(clientKey string) { |
There was a problem hiding this comment.
Blocking — this new API has two callers and only one was wired.
DirectGRPCSubscriptionManager holds its own SubscriptionIDMapper (direct_grpc_subscription_manager.go:92, constructed at :171) and mints router IDs at :533 and :631-632. Its UnsubscribeAll cleanup block at :1077-1084 does:
delete(dgm.clientSubscriptions, clientKey)
dgm.stickyStore.Delete(clientKey)
dgm.rateLimiter.CleanupClient(clientKey)— no idMapper.RemoveClient. So one counter still leaks per gRPC client connection, which is the same ticket item this PR is closing on the ws side.
Either wire it there too, or say in the PR body that item 5 is scoped to ws only, so #Closes MAG-3722 isn't auto-resolving a ticket whose named leak is still live on the gRPC path.
Separately, on the doc comment just above: "the key carries the per-connection GUID, so nothing reuses it" is true for the disconnect path, but not for the other caller. forgetClientLocked now also runs on a client-initiated unsubscribe_all over a still-open socket (consumer_websocket_manager.go:374), where the connection — and therefore the GUID, and therefore clientKey — is unchanged. The next subscribe reuses rs_<hash>_00001; subscription_id_mapper_leak_test.go:26-28 asserts exactly that reuse.
Low severity (worst case is an in-flight push carrying the old id being mis-attributed to the new subscription), but the justification written into the code doesn't hold for that caller. Note the rate-limiter reset on that path is pre-existing, not introduced here — only the id-counter reset is new.
There was a problem hiding this comment.
Both points taken, and resolved by removing the per-client counters rather than wiring RemoveClient into the gRPC manager (31de994).
GenerateRouterID now numbers ids with one mapper-wide atomic.Uint64. The rs_<hash6>_<n> format is unchanged; only the numbering is global. That closes item 5 on both managers with nothing to release on any exit path, so the omission you caught in the other thread has nowhere to recur here. It also retires the reuse caveat you raised on the doc comment: an id is never handed out twice for the life of the process, including across an explicit unsubscribe_all on an open socket, so the mis-attribution case is gone rather than argued away.
RemoveClient and ClientCount are removed. TestSubscriptionIDMapper_GenerateRouterID_DifferentClients now expects _00003 and _00004 for the second client. Nothing in the tree parses the counter part of an id, and I'm not aware of anything downstream that relies on per-client numbering; if you are, say so and I'll revisit.
| // to be wide enough for that median to mean "the time it takes this provider to | ||
| // serve half a window of relays". Unbounded, the slice grew by 24 bytes on every | ||
| // relay for the life of the process and its median drifted to half the uptime, | ||
| // which the MaxHalfTime clamp then discarded anyway (MAG-3722). |
There was a problem hiding this comment.
Blocking — this sentence is the opposite of what the clamp does, and it's hiding a live routing change.
DefaultHalfLifeTime = 1h, MaxHalfTime = 3h.
- Before: median of all relay times ≈ uptime/2. Any pod up more than ~6h →
relaysHalfTime > 3h→ clamped. In productionhalfTimewas pinned at 3h for every provider, always. - After: median of the last 128 relays is seconds old for a busy provider → 1h. Only providers slower than ~1 relay/2.8min still reach 3h.
The clamp didn't discard the drift — it produced 3h. So this ships a 3× faster QoS score decay for every busy provider, which changes provider selection in production.
I think the new behaviour is right: "half the time to serve a window of relays" is clearly the original intent, and "half the pod's uptime" is plainly the bug. I'm not asking you to change the code. But it needs to be stated in the PR body and the release notes, because right now a reviewer reads this comment and correctly concludes there's nothing to look at.
Two things that make it worth being explicit:
- Nothing tests it.
clock_injection_test.go:205-206deliberately clearsproviderRelayStatssocalculateHalfTimereturns the default — the existing suite is structurally incapable of seeing this change, which is why it's green. - It's the one part of this PR with production behaviour risk, and it's in a self-contained commit (
1713448). If the heap work needs to ship fast, this is the piece that could go separately with its own rollout watch.
Suggested edit: drop "which the MaxHalfTime clamp then discarded anyway" and say the effective half-life goes 3h → 1h for established providers.
Minor, same block: [128]time.Time is 3 KB allocated up front per provider, and CacheMaxCost = 20000 → a 60 MB ceiling. Fine for a real pairing list, but it turns a growth curve into a fixed floor; make([]time.Time, 0, relayStatsWindowSize) would give the same bound pay-as-you-go.
There was a problem hiding this comment.
Agreed on all of it, and thanks for doing the arithmetic: that sentence had the clamp backwards. Changes in 603d7c6:
- The comment now says what happens. With the unbounded history the median sat at half the uptime, so every provider on a pod up more than six hours decayed on the 3 h clamp. A busy provider now decays on the 1 h default, and that is a 3x faster decay in production, not only a memory fix.
- The PR body has a "Behaviour change" section saying the same, so a reader of the description sees it without opening the diff.
TestCalculateHalfTime_StretchesOnlyForSparseProviderspins the values: 1 h at one relay a second, 2 h 8 m at one every two minutes, 3 h at one every five. The suite is no longer blind to it.- Your allocation nit: the window grows one sample at a time and wraps once full. Doing that surfaced a second problem. ristretto's
Setis asynchronous, so the relay right after a window's birth could miss it onGetand start a second window, dropping the samples in between; the two-minute case above came out at 3 h because of exactly that. The windows now live in a mutex-guarded map keyed by provider. The set is bounded by the pairing list and nothing in it should ever be evicted, so ristretto bought nothing here.
Whether the optimizer commits ship in this PR or on their own with a rollout watch is Nimrod's call; the body says so.
One note on make test, since you ran it: on my machine the optimizer package sits right at the Makefile's 300 s per-package limit with or without this PR (250 s to 300 s run to run on identical code). The changed path itself benchmarks about 2.5x faster than before. The body suggests -timeout 600s for that package if it trips.
| // told first when tellNode is set, while the connection is still pooled and live, then the | ||
| // local subscription ends and the pool may scale the connection down. The network call runs | ||
| // off the lock. Caller holds dwsm.lock. | ||
| func (dwsm *DirectWSSubscriptionManager) teardownSubscriptionLocked(hashedParams string, activeSub *directActiveSubscription, tellNode bool) { |
There was a problem hiding this comment.
Non-blocking, but worth doing while this is fresh: two paths still bypass this helper.
Unsubscribe's last-client branch open-codesUnsubscribe()/cancel()/delete/Add(-1)/NotifySubscriptionRemoved— and does the pool notify synchronously underdwsm.lock, where this function deliberately moves it into a goroutine. It also skipsstopUpstreamListener(); that one's harmless in practice sincecancel()ends the listener and the follow-oncleanupSubscriptionno-ops on the deleted key.Close()callssender.Close()directly and never closes theclientDetachedchannels, so watchers stay parked until their connection contexts cancel. Fine at process shutdown, but it's another path outside the funnel.
The stated point of this refactor is that there's exactly one way a client leaves and one way a subscription retires. Leaving two paths out is where the next omission lands — and blocking finding #1 on this PR is precisely that failure mode already.
(For what it's worth, Unsubscribe keying off lastClient from idMapper.RemoveMapping rather than len(connectedClients) is the safer choice there — it correctly accounts for a joiner arriving during the unlocked upstream call. Worth a comment saying so, since it reads as an inconsistency.)
There was a problem hiding this comment.
Done in 31de994. Unsubscribe's last-client branch now calls teardownSubscriptionLocked(hashedParams, activeSub, false); the node was already told synchronously above, so no second wire call, and the pool notify moves off the lock with it. Close releases every client through releaseClientLocked, ends each listener through stopUpstreamListener, and resets connectedClients.
The lastClient choice has its comment: it is the mapper's view after this client's mapping went, read under the lock, so a joiner that attached during the unlocked upstream call keeps the subscription.
| // forgetClientSub drops sub from the dispatch table once its forwarding loop has been | ||
| // told to quit. Runs on the dispatch goroutine, which is the only owner of clientSubs. | ||
| // A push for the id that still arrives is dropped by deliverSubscriptionPush as an | ||
| // unmatched notification, exactly as before the entry was removed. |
There was a problem hiding this comment.
Small accuracy point: this isn't quite what deliverSubscriptionPush does.
Per its own doc comment at :288-297, it splits on whether the frame carries a request id — an id-less push is claimed and dropped with a trace, but an id-bearing one is handed back to the call path, which answers it with a JSON-RPC error written upstream to the node.
In practice this is almost always harmless: real EVM/Tendermint pushes are notifications with no id, so "dropped" is correct for them. But it isn't universally true, and TestUnsubscribe_ForgetsDispatchEntry depends on the other branch — its whole mechanism is that the frame gets answered once the entry is gone (require.NotNil(t, answer.Error)).
So for Substrate-shaped id-bearing frames still in flight at unsubscribe time, this change means the router now writes an error frame back to the node where it previously consumed them silently. Worth one clause here noting the id-bearing case, so the next reader doesn't take the comment at face value.
There was a problem hiding this comment.
Right, the comment described the id-less case as if it were the only one. Reworded in 626cd65: without a request id the late push is dropped as before; with one it reaches the call path and the node gets a JSON-RPC error where the stale entry used to swallow it. The PR body says the same under the rpcclient bullet, since it is a wire-visible difference for Substrate-shaped frames.
| } | ||
|
|
||
| // refillDuration is how long a limiter with the given rate takes to refill burst tokens. | ||
| func refillDuration(limit rate.Limit, burst int) time.Duration { |
There was a problem hiding this comment.
Style nit, ranked last — this is algebraically a constant.
subscribeRate = SubscriptionsPerMinutePerClient / 60.0 and subscribeBurst = SubscriptionsPerMinutePerClient, so burst/limit is exactly 60s for every possible config. The three-way max(..., ..., time.Minute) then makes idleTTL unconditionally one minute — which is what websocket_config_sweep_test.go:19 asserts.
The derivation is correct and future-proofs a burst/rate split, so I wouldn't remove it. But a one-line comment saying "today this always evaluates to 1 minute" would save the next reader the algebra.
The reasoning behind the sweep itself is sound, and lock ordering is consistent (dwsm.lock → crl.mu everywhere, never inverted) — I checked.
There was a problem hiding this comment.
Added in 31de994: a comment above the max says burst/rate is one minute for every configuration today, so idleTTL always evaluates to a minute, and that the derivation stays so a future burst/rate split cannot silently shorten the sweep. Thanks for checking the lock ordering.
updateRelayTime appended a time.Time to a per-provider slice on every relay, success or failure, and nothing ever truncated it: only the debug reset endpoints clear the cache. That is 24 bytes per relay for the life of the process, so a busy pod's heap climbed for weeks until the container limit killed it. Each reallocation also copied the whole history under the provider's stripe lock, and the slice's Location pointers made the GC walk all of it every cycle. calculateHalfTime reads exactly one element, the median, so the history is now a fixed 128-slot ring in arrival order. Its median means "the time this provider takes to serve half a window of relays", which is the input the half-life heuristic wants; the unbounded median drifted to half the uptime and the MaxHalfTime clamp discarded it anyway. The ring is mutated in place under the stripe lock, so the cached pointer never needs a replacement Set. Jira ticket: MAG-3722
…ion ends Three ways a websocket client's departure left state behind for the life of the process, all on the same path. The subscription manager never released a JOINED client. A second subscriber to a query joins the first one's upstream subscription, and only the creating client had a disconnect watcher: when a joiner closed its socket its reply channel stayed open, which parks the ingress forwarder goroutine ranging over it, it kept counting as connected, and the upstream subscription outlived every real subscriber. The creator's watcher had the mirror omission: it deleted the client's sender without closing it, so its forwarder parked too. Every client now attaches through one helper and leaves through one helper that closes the channel, drops the bookkeeping and wakes the watcher, so no path can skip a step again. An explicit unsubscribe wakes the watcher instead of leaving it parked until the socket closes, and UnsubscribeAll now tells the node when the last client leaves, which it used to skip. The shared subscription's context was derived from the creating client's connection, so the creator leaving ended the upstream listener underneath everyone who had joined. It is now owned by the manager, the rule the gRPC manager already applies. The ingress writer goroutine returned on a failed write without telling anyone. Every later frame sender then blocked forever, and the read loop's own error path is one of them, so the handler never returned and fasthttp kept the 128 KiB read buffer, the handler goroutine, the ticker goroutine and the per-IP limiter slot. The writer now closes a done channel on the way out, senders select on it, and a failed write sets an immediate read deadline so the read loop wakes. The handler also releases every subscription it holds through UnsubscribeAll on return, synchronously, rather than relying on a cancelled context being noticed. Smaller entries that never went away: the id mapper's per-client counter (one per connection that ever subscribed) is dropped with the client, and rate limiters idle for their refill time are swept, since such an entry decides exactly as a fresh one would. Jira ticket: MAG-3722
The handler's subscription table only shrank when the connection closed. Upstream websocket connections are pooled and live for the life of the process, so every subscription ever torn down on one left its entry behind. Unsubscribe now asks the dispatch loop, the table's only owner, to forget the entry, on a goroutine of its own because dispatch may be busy delivering to another subscription whose consumer waits on a lock the caller holds. Jira ticket: MAG-3722
The state machine reads returnCondition at most once and may already have left on another branch, yet every validateReturnCondition goroutine sent into the buffer-1 channel unconditionally. A third stop decision inside one 15 ms burst parked its goroutine forever, pinning the whole state machine and the request it carries. The send is now non-blocking; only the first reason can matter. Jira ticket: MAG-3722
The stake map kept one entry per provider per epoch and nothing removed them. Reads only ever ask for the epoch being scored, so the current and previous epoch are all that is kept now. Jira ticket: MAG-3722
…ds per mapper Review follow-ups on the client release. A client that unsubscribed everything explicitly and then closed its socket still leaked: the explicit unsubscribe emptied its subscription set, and the connection-close release returned early on the empty set before handing back the sticky endpoint and rate limiters. The early return now releases them. Unsubscribe's last-client branch and Close go through the same teardown and release helpers as every other path, so a subscription retires one way only, and Unsubscribe documents why it trusts the mapper's count. Router ids are numbered by one mapper-wide counter instead of one per client. The per-client counter map was the leak on both the ws and the gRPC manager, and it needed a release call on every exit path to stay bounded; a single counter needs nothing released and never reuses an id for the life of the process, so the reuse caveat on an explicit unsubscribe_all goes away too. The id format is unchanged; only the numbering is mapper-wide. Jira ticket: MAG-3722
… the half-life change Review follow-ups on the relay-time window. The windows live in a plain mutex-guarded map rather than a ristretto cache: the set is bounded by the pairing list, nothing in it should be evicted, and ristretto's asynchronous Set let the relay right after a window's birth miss it and start a second window that dropped the samples in between. The window grows one sample at a time and wraps once full, so a provider that served a handful of relays costs a handful of entries. The comment now says what the change does to routing: with the unbounded history the median sat at half the uptime, so every provider on a pod up more than six hours decayed on the 3 h clamp; a busy provider now decays on the 1 h default, which is what the heuristic was written to do. A new test pins the values at three relay rates. Jira ticket: MAG-3722
A late push without a request id is dropped as before; one with a request id now reaches the call path and is answered with an error upstream, where the stale entry used to swallow it. Jira ticket: MAG-3722
… between the two checks (MAG-2546) StartSubscription looks for an active subscription and registers a pending one under two separate acquisitions of the manager lock. A creator that stored its subscription and cleared its pending entry in that gap left the second caller seeing neither, so it became a second creator: a second upstream subscription for the same params, the first map entry overwritten and orphaned (its listener goroutine and node-side subscription living on unowned), and totalSubscriptions incremented twice for an entry that is dropped once. The pending check now re-checks the active map under the same lock and hands such a caller a settled success so it takes the join path. The cleanup-ticker branch the ticket names (a cancelled context deleting without decrementing) was already routed through dropSubscriptionLocked by #386; the new test pins that so it stays fixed. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Took 47 minutes
…oduction reaches (MAG-2546) The cancelled-context branch the ticket names cannot be reached in production: subCtx derives from context.WithoutCancel, and its cancel runs only together with the delete (the teardown goroutine, cleanupSubscription, Close). The no-clients branch can run (connectedClients empty while the id mapper says "not last"), and nothing pinned it: put back to its pre-#386 shape, the whole suite still passed. The new test checks that it decrements the counter, tells the node, and uncounts the pooled connection. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… between the two checks (MAG-2546) (#508) StartSubscription looks for an active subscription and registers a pending one under two separate acquisitions of the manager lock. A creator that stored its subscription and cleared its pending entry in that gap left the second caller seeing neither, so it became a second creator: a second upstream subscription for the same params, the first map entry overwritten and orphaned (its listener goroutine and node-side subscription living on unowned), and totalSubscriptions incremented twice for an entry that is dropped once. The pending check now re-checks the active map under the same lock and hands such a caller a settled success so it takes the join path. The cleanup-ticker branch the ticket names (a cancelled context deleting without decrementing) was already routed through dropSubscriptionLocked by #386; the new test pins that so it stays fixed. Review fixes (AnnaR-prog): - test(rpcsmartrouter): pin the janitor's no-clients branch, the one production reaches (MAG-2546) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
#Closes MAG-3722
Jira ticket: MAG-3722
Why
A router pod's heap climbs for the life of the process. In production a v1.3.0 EVM pod reached the chart's 4Gi limit in three weeks and was OOM-killed, and every other chain pod on the same cluster was climbing in proportion to its request rate. A full-tree audit of v1.3.0, re-checked against main, found the growth comes from several independent retention bugs. The two largest are still on main today, so v1.5.1 does not stop it.
updateRelayTimeappended to a per-provider[]time.Timeon every relay and nothing truncated it. 24 bytes per relay, forever. The reader only ever used the median, and that median drifted to half the uptime, which has a routing consequence described under "Behaviour change" below.validateReturnConditionsent unconditionally into a buffer-1 channel the loop reads at most once.What changed
calculateHalfTimewants. See "Behaviour change" below.attachClientLockedand leaves throughreleaseClientLocked(closes the channel, drops bookkeeping, wakes the watcher). The join path registers a disconnect watcher. An explicit unsubscribe wakes the watcher instead of leaving it parked until the socket closes.UnsubscribeAlltells the node when the last client leaves. The subscription context is owned by the manager (context.WithoutCancelof the creator's), the rule the gRPC manager already applies. The periodic sweep goes through the same teardown and now also decrements the subscription counter, which it previously did not.websocketFrameWriterinterface. The writer closeswriterDoneon exit, senders select on it, and a failed write sets an immediate read deadline so the read loop wakes.ListenToMessagesreleases every subscription throughUnsubscribeAllon return.SweepIdledrops entries idle for their refill time, which decide exactly as a fresh one would. Called from the periodic sweep.rs_<clientHash6>_<n>); only the numbering is mapper-wide.Unsubscribeasks the dispatch loop, the table's only owner, to forget the entry. One consequence: a late push for that id that carries a request id (Substrate-shaped frames; EVM and Tendermint pushes have none) is now handed to the call path and answered with a JSON-RPC error upstream, where the stale entry used to swallow it.Behaviour change: QoS half-life for busy providers goes from 3 h to 1 h
calculateHalfTimeismax(DefaultHalfLifeTime = 1 h, age of the median relay time), capped atMaxHalfTime = 3 h. With the unbounded history the median was half the pod's uptime, so on any pod up more than six hours every provider decayed on the 3 h cap, always. With the window a provider serving more than 128 relays an hour decays on the 1 h default, and only a sparse one stretches towards the cap. That is what the heuristic was written to do, and the old value was an artefact of the bug, but it is a 3x faster score decay for every busy provider in production and it changes provider selection dynamics.TestCalculateHalfTime_StretchesOnlyForSparseProviderspins the new values (1 h at one relay a second, 2 h 8 m at one every two minutes, 3 h at one every five). The existing suite could not see this:clock_injection_test.goclears the relay history on purpose. The optimizer change is its own commit (1713448plus its review follow-up) and can ship separately with its own rollout watch if that is preferred.Not changed:
GRPCConnector.GetRpcwas on the audit list for a context-less spin, but main already exits on context cancellation and on close (MAG-2808) and splits the pool's lifetime context from the first relay's dial context; both pins indirect_rpc_close_race_test.gopass.How to verify
Every fix has a regression test that fails against the previous code:
What each proves:
TestUpdateRelayTime_DoesNotGrowWithRelays: 10,000 relays leave exactly 128 samples cached, and the half-life for a busy provider equals the default.TestJoinedClientDisconnect_IsReleased: a joiner's disconnect closes its reply channel, removes it from the subscription and drops its id counter while the creator keeps receiving; the creator's disconnect then sendseth_unsubscribeupstream and leaves no subscription, counter or limiter behind.TestCreatorDisconnect_ClosesItsReplyChannel: the creator's channel closes on disconnect and the joiner keeps receiving pushes.TestUnsubscribeAll_ReleasesWatchersAndTellsNode: counts livehandleClientDisconnectgoroutines going 2, 1, 0 as clients are released, and the node is told on the last one.TestWebsocketWriterFailure_UnblocksSenders: after a failed write, a later sender returns immediately and the read loop was woken.TestUnsubscribe_ForgetsDispatchEntry: observed through behaviour, not a hook. An id-bearing push naming the subscription is claimed silently while the dispatch entry exists and answered by the call path once it is gone.TestSignalReturnCondition_NeverBlocks: three signals into the buffer-1 channel return.TestUnsubscribeThenDisconnect_LeavesNothingBehind(review follow-up): subscribe, expliciteth_unsubscribe, socket close leaves no sticky endpoint, rate limiter or connected-client entry; fails on the code before the follow-up commit.Full run:
make test(the tree,-count=1), plusgo test -raceon every touched package. Note that CI on this repo builds, lints and scans but runs nogo test; all test evidence here is from local runs. The optimizer package sits at the Makefile's 300 s per-package limit on a slow machine with or without this PR (250 s to 300 s on the same box, run to run); the changed path itself benchmarks 2.5x faster than before. Use-timeout 600sfor that package if it trips. The optimizer package under-raceexceeds go test's 10-minute default on main as well (TestProviderOptimizerAvailabilityRelayDataalone takes about four minutes there); run it with-timeout 20m.On a live pod the tell for item 1 is
POST /debug/reset-scores: the heap drops by the accumulated history and resumes the same slope. After this change there is nothing to drop.Related open PR
PR #355 in smart-router —
fix(chainlib): upgrade websockets on any path and keep them alive— is stacked on this branch. Its keep-alive ping and write deadline are built into therunWebsocketWriterthis PR introduces, so there is no conflict between them; merge this one first, then retarget #355 tomain.