Skip to content

fix: stop the router heap growing for the life of the process - #386

Merged
nimrod-teich merged 8 commits into
mainfrom
mag-3722
Sep 17, 2026
Merged

nimrod-teich merged 8 commits into
mainfrom
mag-3722

Conversation

@nimrod-teich

@nimrod-teich nimrod-teich commented Sep 16, 2026 •

Copy link
Copy Markdown
Contributor

#Closes MAG-3722

Jira ticket: MAG-3722

Why

A router pod's heap climbs for the life of the process. In production a v1.3.0 EVM pod reached the chart's 4Gi limit in three weeks and was OOM-killed, and every other chain pod on the same cluster was climbing in proportion to its request rate. A full-tree audit of v1.3.0, re-checked against main, found the growth comes from several independent retention bugs. The two largest are still on main today, so v1.5.1 does not stop it.

  1. The optimizer kept a timestamp for every relay ever served. updateRelayTime appended to a per-provider []time.Time on every relay and nothing truncated it. 24 bytes per relay, forever. The reader only ever used the median, and that median drifted to half the uptime, which has a routing consequence described under "Behaviour change" below.
  2. A websocket client's departure left its subscription state behind. A second subscriber to a query joins the first one's upstream subscription, and only the creator had a disconnect watcher; a joiner closing its socket kept its reply channel open (which parks the ingress forwarder goroutine ranging over it), kept counting as connected, and kept the upstream subscription alive. The creator's watcher deleted its sender without closing it, so its forwarder parked too. And the shared subscription's context was derived from the creator's connection, so the creator leaving killed the stream under every joiner.
  3. A dead ingress writer deadlocked the whole connection handler. The writer goroutine returned on a failed write without telling anyone; every later frame sender blocked forever, including the read loop's own error path, so the handler never returned and fasthttp kept the 128 KiB read buffer, the handler goroutine, the ticker goroutine and the per-IP limiter slot. About 140 KB per stuck connection.
  4. The relay state machine could park a goroutine holding the whole request. validateReturnCondition sent unconditionally into a buffer-1 channel the loop reads at most once.
  5. Smaller, all unbounded: the rpcclient dispatch table kept an entry per unsubscribed upstream subscription until the pooled connection closed; the id mapper kept a counter per connection that ever subscribed; ws rate limiters leaked for clients whose subscribe failed; the QoS client kept a stake entry per provider per epoch.

What changed

  • provider_optimizer.go: the relay time history is a 128-slot window in arrival order that grows one sample at a time and then wraps, kept in a mutex-guarded map per provider rather than a ristretto cache (the set is bounded by the pairing list, and ristretto's asynchronous Set let the relay right after a window's birth miss it). Its median means "the time this provider takes to serve half a window of relays", which is the input calculateHalfTime wants. See "Behaviour change" below.
  • direct_ws_subscription_manager.go: every client attaches through attachClientLocked and leaves through releaseClientLocked (closes the channel, drops bookkeeping, wakes the watcher). The join path registers a disconnect watcher. An explicit unsubscribe wakes the watcher instead of leaving it parked until the socket closes. UnsubscribeAll tells the node when the last client leaves. The subscription context is owned by the manager (context.WithoutCancel of the creator's), the rule the gRPC manager already applies. The periodic sweep goes through the same teardown and now also decrements the subscription counter, which it previously did not.
  • consumer_websocket_manager.go: the writer loop and frame enqueue are functions over a websocketFrameWriter interface. The writer closes writerDone on exit, senders select on it, and a failed write sets an immediate read deadline so the read loop wakes. ListenToMessages releases every subscription through UnsubscribeAll on return.
  • websocket_config.go: rate limiters record last use; SweepIdle drops entries idle for their refill time, which decide exactly as a fresh one would. Called from the periodic sweep.
  • subscription_id_mapper.go: router ids are numbered by one mapper-wide counter instead of one counter per client. The per-client map was the leak, on both the ws and the gRPC manager, and a single counter needs nothing released and never reuses an id for the life of the process. Format is unchanged (rs_<clientHash6>_<n>); only the numbering is mapper-wide.
  • rpcclient/client.go, handler.go, subscription.go: Unsubscribe asks the dispatch loop, the table's only owner, to forget the entry. One consequence: a late push for that id that carries a request id (Substrate-shaped frames; EVM and Tendermint pushes have none) is now handed to the call path and answered with a JSON-RPC error upstream, where the stale entry used to swallow it.
  • unified_relay_state_machine.go: the return-condition signal is non-blocking.
  • consumer_optimizer_qos_client.go: the stake map keeps the current and previous epoch only.

Behaviour change: QoS half-life for busy providers goes from 3 h to 1 h

calculateHalfTime is max(DefaultHalfLifeTime = 1 h, age of the median relay time), capped at MaxHalfTime = 3 h. With the unbounded history the median was half the pod's uptime, so on any pod up more than six hours every provider decayed on the 3 h cap, always. With the window a provider serving more than 128 relays an hour decays on the 1 h default, and only a sparse one stretches towards the cap. That is what the heuristic was written to do, and the old value was an artefact of the bug, but it is a 3x faster score decay for every busy provider in production and it changes provider selection dynamics. TestCalculateHalfTime_StretchesOnlyForSparseProviders pins the new values (1 h at one relay a second, 2 h 8 m at one every two minutes, 3 h at one every five). The existing suite could not see this: clock_injection_test.go clears the relay history on purpose. The optimizer change is its own commit (1713448 plus its review follow-up) and can ship separately with its own rollout watch if that is preferred.

Not changed: GRPCConnector.GetRpc was on the audit list for a context-less spin, but main already exits on context cancellation and on close (MAG-2808) and splits the pool's lifetime context from the first relay's dial context; both pins in direct_rpc_close_race_test.go pass.

How to verify

Every fix has a regression test that fails against the previous code:

go test -race -count=1 ./protocol/provideroptimizer/ -run 'RelayTimeWindow|UpdateRelayTime|StretchesOnly'
go test -race -count=1 ./protocol/rpcsmartrouter/ -run 'JoinedClientDisconnect|CreatorDisconnect|UnsubscribeAll_ReleasesWatchers|UnsubscribeThenDisconnect|SweepIdle|SubscriptionIDMapper'
go test -race -count=1 ./protocol/chainlib/ -run WebsocketWriter
go test -race -count=1 ./protocol/chainlib/chainproxy/rpcclient/ -run 'ForgetClientSub|ForgetsDispatchEntry'
go test -race -count=1 ./protocol/relaycore/ -run SignalReturnCondition
go test -race -count=1 ./protocol/metrics/ -run PrunesOldEpochs

What each proves:

  • TestUpdateRelayTime_DoesNotGrowWithRelays: 10,000 relays leave exactly 128 samples cached, and the half-life for a busy provider equals the default.
  • TestJoinedClientDisconnect_IsReleased: a joiner's disconnect closes its reply channel, removes it from the subscription and drops its id counter while the creator keeps receiving; the creator's disconnect then sends eth_unsubscribe upstream and leaves no subscription, counter or limiter behind.
  • TestCreatorDisconnect_ClosesItsReplyChannel: the creator's channel closes on disconnect and the joiner keeps receiving pushes.
  • TestUnsubscribeAll_ReleasesWatchersAndTellsNode: counts live handleClientDisconnect goroutines going 2, 1, 0 as clients are released, and the node is told on the last one.
  • TestWebsocketWriterFailure_UnblocksSenders: after a failed write, a later sender returns immediately and the read loop was woken.
  • TestUnsubscribe_ForgetsDispatchEntry: observed through behaviour, not a hook. An id-bearing push naming the subscription is claimed silently while the dispatch entry exists and answered by the call path once it is gone.
  • TestSignalReturnCondition_NeverBlocks: three signals into the buffer-1 channel return.
  • TestUnsubscribeThenDisconnect_LeavesNothingBehind (review follow-up): subscribe, explicit eth_unsubscribe, socket close leaves no sticky endpoint, rate limiter or connected-client entry; fails on the code before the follow-up commit.

Full run: make test (the tree, -count=1), plus go test -race on every touched package. Note that CI on this repo builds, lints and scans but runs no go test; all test evidence here is from local runs. The optimizer package sits at the Makefile's 300 s per-package limit on a slow machine with or without this PR (250 s to 300 s on the same box, run to run); the changed path itself benchmarks 2.5x faster than before. Use -timeout 600s for that package if it trips. The optimizer package under -race exceeds go test's 10-minute default on main as well (TestProviderOptimizerAvailabilityRelayData alone takes about four minutes there); run it with -timeout 20m.

On a live pod the tell for item 1 is POST /debug/reset-scores: the heap drops by the accumulated history and resumes the same slope. After this change there is nothing to drop.

Related open PR

PR #355 in smart-router — fix(chainlib): upgrade websockets on any path and keep them alive — is stacked on this branch. Its keep-alive ping and write deadline are built into the runWebsocketWriter this PR introduces, so there is no conflict between them; merge this one first, then retarget #355 to main.

@avitenzer avitenzer left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review: verified locally, two leaks still open + one undisclosed behaviour change

Thanks — the structural fix here is the right shape (one attach funnel, one release funnel, one teardown), and the tests are genuinely behavioural rather than hook-driven.

What I verified myself, at 682e29d:

Check Result
go test ./... -count=1 (whole tree) clean
-race -count=1, all 6 touched packages clean (optimizer 531s, rpcsmartrouter 113s)
go build ./protocol/... + go vet clean
PR's "How to verify" commands 6/6 confirmed
The 3 ws release tests run against the merge base a545be5 3/3 FAIL, for exactly the stated reasons
Helm chart repo: refs to any changed flag/key/metric 0 (control passes) — no chart change, no CrashLoop risk

That merge-base check is the important one: the "fails against the previous code" claim holds up.

Blocking

  1. Ticket item 5 (clientCounters) is still open on the commonest ws path. UnsubscribeAll early-returns before forgetClientLocked, so subscribe → eth_unsubscribe → socket close leaks the id counter and the sticky entry, per connection, forever. Proven with a probe test on this branch: idMapper.ClientCount()=1, stickyStore.Len()=1 after the client is fully gone. One-line fix, verified green — see inline.
  2. Same item, entirely unfixed on the gRPC side. DirectGRPCSubscriptionManager owns its own SubscriptionIDMapper and never calls the new RemoveClient. See inline.
  3. The optimizer change alters routing, and the PR says it doesn't. DefaultHalfLifeTime=1h, MaxHalfTime=3h. Before: median ≈ uptime/2, so any pod up >6h had halfTime pinned at 3h for every provider. After: 1h for any busy provider. That is a 3× faster QoS decay in production. See inline.

Unproven

  • Heap actually plateaus — only observable on a deployed pod; nothing here or in CI measures memory.
  • Routing impact of 3h→1h — clock_injection_test.go:205 deliberately clears providerRelayStats, so the existing suite is structurally blind to it.

Worth knowing

  • CI runs no tests on this repo. 21 checks — build ×8, Docker, lint, govulncheck, GitGuardian, Jira — and zero go test. Not this PR's fault, but it means all test evidence for 1,115 lines is local.
  • Six subsystems in one PR makes revert all-or-nothing, and the riskiest piece (the optimizer, 1713448) is the most separable. Worth considering splitting it.
  • #355 confirmed open with base=mag-3722, exactly as the body says.

Nothing below is a style nit unless I've labelled it one.

utils.LavaFormatTrace("DirectWS: cleared sticky session and rate limiter on unsubscribe all",
utils.LogAttr("clientKey", clientKey),
)
dwsm.forgetClientLocked(clientKey)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking — ticket item 5 is still open on the commonest client path.

This line is unreachable for a client that unsubscribed before disconnecting. UnsubscribeAll returns ~25 lines above:

clientSubs, ok := dwsm.connectedClients[clientKey]
if !ok {
    return nil // No subscriptions for this client
}

The sequence subscribe → eth_unsubscribe → close socket deletes connectedClients[clientKey] in forgetSubscriptionForClientLocked (with clientGone=false, so it skips cleanup by design), and then the connection-close path that was supposed to finish the job bails out here. The client's own watcher doesn't cover it either — releaseClientLocked already closed its detached channel at unsubscribe time, so it took the early-exit branch.

Verified on this branch with a probe test:

after unsubscribe+disconnect: idMapper.ClientCount()=1 rateLimiter.ClientCount()=2 stickyLen=1

Leaked permanently, per connection: idMapper.clientCounters (this is MAG-3722 item 5 verbatim) and stickyStore. No sweep reaches either — the orphan loop in performCleanup iterates connectedClients, whose key is already gone, and SweepIdle only covers rate limiters (those do get cleaned within ~2 min, so that part is fine).

One line fixes it, and I confirmed the probe goes green:

clientSubs, ok := dwsm.connectedClients[clientKey]
if !ok {
    dwsm.forgetClientLocked(clientKey)
    return nil
}

None of the three new tests cover this shape — TestUnsubscribeAll_ReleasesWatchers... calls UnsubscribeAll while subs are still live, and the disconnect tests never send an explicit unsubscribe. Worth a fourth case asserting idMapper.ClientCount()==0 and stickyStore.Len()==0.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed and fixed in 31de994: the empty-set branch now calls forgetClientLocked before returning, so the connection-close release hands back the sticky endpoint and rate limiters for a client that unsubscribed first.

TestUnsubscribeThenDisconnect_LeavesNothingBehind is the shape you described (subscribe, eth_unsubscribe, close the socket). It asserts stickyStore.Len() == 0, rateLimiter.ClientCount() == 0, no connected-client entry, and exactly one upstream eth_unsubscribe. Ran it with that one line removed: it fails on the sticky assertion, so it pins the fix.

The id-counter half of your probe no longer exists; see the mapper thread.

// subscriptions: the key carries the per-connection GUID, so nothing reuses it,
// and without this call the map kept one entry per connection that ever
// subscribed (MAG-3722).
func (m *SubscriptionIDMapper) RemoveClient(clientKey string) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking — this new API has two callers and only one was wired.

DirectGRPCSubscriptionManager holds its own SubscriptionIDMapper (direct_grpc_subscription_manager.go:92, constructed at :171) and mints router IDs at :533 and :631-632. Its UnsubscribeAll cleanup block at :1077-1084 does:

delete(dgm.clientSubscriptions, clientKey)
dgm.stickyStore.Delete(clientKey)
dgm.rateLimiter.CleanupClient(clientKey)

— no idMapper.RemoveClient. So one counter still leaks per gRPC client connection, which is the same ticket item this PR is closing on the ws side.

Either wire it there too, or say in the PR body that item 5 is scoped to ws only, so #Closes MAG-3722 isn't auto-resolving a ticket whose named leak is still live on the gRPC path.


Separately, on the doc comment just above: "the key carries the per-connection GUID, so nothing reuses it" is true for the disconnect path, but not for the other caller. forgetClientLocked now also runs on a client-initiated unsubscribe_all over a still-open socket (consumer_websocket_manager.go:374), where the connection — and therefore the GUID, and therefore clientKey — is unchanged. The next subscribe reuses rs_<hash>_00001; subscription_id_mapper_leak_test.go:26-28 asserts exactly that reuse.

Low severity (worst case is an in-flight push carrying the old id being mis-attributed to the new subscription), but the justification written into the code doesn't hold for that caller. Note the rate-limiter reset on that path is pre-existing, not introduced here — only the id-counter reset is new.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both points taken, and resolved by removing the per-client counters rather than wiring RemoveClient into the gRPC manager (31de994).

GenerateRouterID now numbers ids with one mapper-wide atomic.Uint64. The rs_<hash6>_<n> format is unchanged; only the numbering is global. That closes item 5 on both managers with nothing to release on any exit path, so the omission you caught in the other thread has nowhere to recur here. It also retires the reuse caveat you raised on the doc comment: an id is never handed out twice for the life of the process, including across an explicit unsubscribe_all on an open socket, so the mis-attribution case is gone rather than argued away.

RemoveClient and ClientCount are removed. TestSubscriptionIDMapper_GenerateRouterID_DifferentClients now expects _00003 and _00004 for the second client. Nothing in the tree parses the counter part of an id, and I'm not aware of anything downstream that relies on per-client numbering; if you are, say so and I'll revisit.

// to be wide enough for that median to mean "the time it takes this provider to
// serve half a window of relays". Unbounded, the slice grew by 24 bytes on every
// relay for the life of the process and its median drifted to half the uptime,
// which the MaxHalfTime clamp then discarded anyway (MAG-3722).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking — this sentence is the opposite of what the clamp does, and it's hiding a live routing change.

DefaultHalfLifeTime = 1h, MaxHalfTime = 3h.

  • Before: median of all relay times ≈ uptime/2. Any pod up more than ~6h → relaysHalfTime > 3h → clamped. In production halfTime was pinned at 3h for every provider, always.
  • After: median of the last 128 relays is seconds old for a busy provider → 1h. Only providers slower than ~1 relay/2.8min still reach 3h.

The clamp didn't discard the drift — it produced 3h. So this ships a 3× faster QoS score decay for every busy provider, which changes provider selection in production.

I think the new behaviour is right: "half the time to serve a window of relays" is clearly the original intent, and "half the pod's uptime" is plainly the bug. I'm not asking you to change the code. But it needs to be stated in the PR body and the release notes, because right now a reviewer reads this comment and correctly concludes there's nothing to look at.

Two things that make it worth being explicit:

  1. Nothing tests it. clock_injection_test.go:205-206 deliberately clears providerRelayStats so calculateHalfTime returns the default — the existing suite is structurally incapable of seeing this change, which is why it's green.
  2. It's the one part of this PR with production behaviour risk, and it's in a self-contained commit (1713448). If the heap work needs to ship fast, this is the piece that could go separately with its own rollout watch.

Suggested edit: drop "which the MaxHalfTime clamp then discarded anyway" and say the effective half-life goes 3h → 1h for established providers.

Minor, same block: [128]time.Time is 3 KB allocated up front per provider, and CacheMaxCost = 20000 → a 60 MB ceiling. Fine for a real pairing list, but it turns a growth curve into a fixed floor; make([]time.Time, 0, relayStatsWindowSize) would give the same bound pay-as-you-go.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed on all of it, and thanks for doing the arithmetic: that sentence had the clamp backwards. Changes in 603d7c6:

  • The comment now says what happens. With the unbounded history the median sat at half the uptime, so every provider on a pod up more than six hours decayed on the 3 h clamp. A busy provider now decays on the 1 h default, and that is a 3x faster decay in production, not only a memory fix.
  • The PR body has a "Behaviour change" section saying the same, so a reader of the description sees it without opening the diff.
  • TestCalculateHalfTime_StretchesOnlyForSparseProviders pins the values: 1 h at one relay a second, 2 h 8 m at one every two minutes, 3 h at one every five. The suite is no longer blind to it.
  • Your allocation nit: the window grows one sample at a time and wraps once full. Doing that surfaced a second problem. ristretto's Set is asynchronous, so the relay right after a window's birth could miss it on Get and start a second window, dropping the samples in between; the two-minute case above came out at 3 h because of exactly that. The windows now live in a mutex-guarded map keyed by provider. The set is bounded by the pairing list and nothing in it should ever be evicted, so ristretto bought nothing here.

Whether the optimizer commits ship in this PR or on their own with a rollout watch is Nimrod's call; the body says so.

One note on make test, since you ran it: on my machine the optimizer package sits right at the Makefile's 300 s per-package limit with or without this PR (250 s to 300 s run to run on identical code). The changed path itself benchmarks about 2.5x faster than before. The body suggests -timeout 600s for that package if it trips.

// told first when tellNode is set, while the connection is still pooled and live, then the
// local subscription ends and the pool may scale the connection down. The network call runs
// off the lock. Caller holds dwsm.lock.
func (dwsm *DirectWSSubscriptionManager) teardownSubscriptionLocked(hashedParams string, activeSub *directActiveSubscription, tellNode bool) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking, but worth doing while this is fresh: two paths still bypass this helper.

  • Unsubscribe's last-client branch open-codes Unsubscribe() / cancel() / delete / Add(-1) / NotifySubscriptionRemoved — and does the pool notify synchronously under dwsm.lock, where this function deliberately moves it into a goroutine. It also skips stopUpstreamListener(); that one's harmless in practice since cancel() ends the listener and the follow-on cleanupSubscription no-ops on the deleted key.
  • Close() calls sender.Close() directly and never closes the clientDetached channels, so watchers stay parked until their connection contexts cancel. Fine at process shutdown, but it's another path outside the funnel.

The stated point of this refactor is that there's exactly one way a client leaves and one way a subscription retires. Leaving two paths out is where the next omission lands — and blocking finding #1 on this PR is precisely that failure mode already.

(For what it's worth, Unsubscribe keying off lastClient from idMapper.RemoveMapping rather than len(connectedClients) is the safer choice there — it correctly accounts for a joiner arriving during the unlocked upstream call. Worth a comment saying so, since it reads as an inconsistency.)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 31de994. Unsubscribe's last-client branch now calls teardownSubscriptionLocked(hashedParams, activeSub, false); the node was already told synchronously above, so no second wire call, and the pool notify moves off the lock with it. Close releases every client through releaseClientLocked, ends each listener through stopUpstreamListener, and resets connectedClients.

The lastClient choice has its comment: it is the mapper's view after this client's mapping went, read under the lock, so a joiner that attached during the unlocked upstream call keeps the subscription.

// forgetClientSub drops sub from the dispatch table once its forwarding loop has been
// told to quit. Runs on the dispatch goroutine, which is the only owner of clientSubs.
// A push for the id that still arrives is dropped by deliverSubscriptionPush as an
// unmatched notification, exactly as before the entry was removed.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small accuracy point: this isn't quite what deliverSubscriptionPush does.

Per its own doc comment at :288-297, it splits on whether the frame carries a request id — an id-less push is claimed and dropped with a trace, but an id-bearing one is handed back to the call path, which answers it with a JSON-RPC error written upstream to the node.

In practice this is almost always harmless: real EVM/Tendermint pushes are notifications with no id, so "dropped" is correct for them. But it isn't universally true, and TestUnsubscribe_ForgetsDispatchEntry depends on the other branch — its whole mechanism is that the frame gets answered once the entry is gone (require.NotNil(t, answer.Error)).

So for Substrate-shaped id-bearing frames still in flight at unsubscribe time, this change means the router now writes an error frame back to the node where it previously consumed them silently. Worth one clause here noting the id-bearing case, so the next reader doesn't take the comment at face value.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, the comment described the id-less case as if it were the only one. Reworded in 626cd65: without a request id the late push is dropped as before; with one it reaches the call path and the node gets a JSON-RPC error where the stale entry used to swallow it. The PR body says the same under the rpcclient bullet, since it is a wire-visible difference for Substrate-shaped frames.

}

// refillDuration is how long a limiter with the given rate takes to refill burst tokens.
func refillDuration(limit rate.Limit, burst int) time.Duration {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Style nit, ranked last — this is algebraically a constant.

subscribeRate = SubscriptionsPerMinutePerClient / 60.0 and subscribeBurst = SubscriptionsPerMinutePerClient, so burst/limit is exactly 60s for every possible config. The three-way max(..., ..., time.Minute) then makes idleTTL unconditionally one minute — which is what websocket_config_sweep_test.go:19 asserts.

The derivation is correct and future-proofs a burst/rate split, so I wouldn't remove it. But a one-line comment saying "today this always evaluates to 1 minute" would save the next reader the algebra.

The reasoning behind the sweep itself is sound, and lock ordering is consistent (dwsm.lock → crl.mu everywhere, never inverted) — I checked.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added in 31de994: a comment above the max says burst/rate is one minute for every configuration today, so idleTTL always evaluates to a minute, and that the derivation stays so a future burst/rate split cannot silently shorten the sweep. Thanks for checking the lock ordering.

updateRelayTime appended a time.Time to a per-provider slice on every relay,
success or failure, and nothing ever truncated it: only the debug reset
endpoints clear the cache. That is 24 bytes per relay for the life of the
process, so a busy pod's heap climbed for weeks until the container limit
killed it. Each reallocation also copied the whole history under the
provider's stripe lock, and the slice's Location pointers made the GC walk
all of it every cycle.

calculateHalfTime reads exactly one element, the median, so the history is
now a fixed 128-slot ring in arrival order. Its median means "the time this
provider takes to serve half a window of relays", which is the input the
half-life heuristic wants; the unbounded median drifted to half the uptime
and the MaxHalfTime clamp discarded it anyway. The ring is mutated in place
under the stripe lock, so the cached pointer never needs a replacement Set.

Jira ticket: MAG-3722
…ion ends

Three ways a websocket client's departure left state behind for the life of
the process, all on the same path.

The subscription manager never released a JOINED client. A second subscriber
to a query joins the first one's upstream subscription, and only the creating
client had a disconnect watcher: when a joiner closed its socket its reply
channel stayed open, which parks the ingress forwarder goroutine ranging over
it, it kept counting as connected, and the upstream subscription outlived
every real subscriber. The creator's watcher had the mirror omission: it
deleted the client's sender without closing it, so its forwarder parked too.
Every client now attaches through one helper and leaves through one helper
that closes the channel, drops the bookkeeping and wakes the watcher, so no
path can skip a step again. An explicit unsubscribe wakes the watcher instead
of leaving it parked until the socket closes, and UnsubscribeAll now tells the
node when the last client leaves, which it used to skip.

The shared subscription's context was derived from the creating client's
connection, so the creator leaving ended the upstream listener underneath
everyone who had joined. It is now owned by the manager, the rule the gRPC
manager already applies.

The ingress writer goroutine returned on a failed write without telling
anyone. Every later frame sender then blocked forever, and the read loop's
own error path is one of them, so the handler never returned and fasthttp
kept the 128 KiB read buffer, the handler goroutine, the ticker goroutine and
the per-IP limiter slot. The writer now closes a done channel on the way out,
senders select on it, and a failed write sets an immediate read deadline so
the read loop wakes. The handler also releases every subscription it holds
through UnsubscribeAll on return, synchronously, rather than relying on a
cancelled context being noticed.

Smaller entries that never went away: the id mapper's per-client counter
(one per connection that ever subscribed) is dropped with the client, and
rate limiters idle for their refill time are swept, since such an entry
decides exactly as a fresh one would.

Jira ticket: MAG-3722
The handler's subscription table only shrank when the connection closed.
Upstream websocket connections are pooled and live for the life of the
process, so every subscription ever torn down on one left its entry behind.
Unsubscribe now asks the dispatch loop, the table's only owner, to forget
the entry, on a goroutine of its own because dispatch may be busy delivering
to another subscription whose consumer waits on a lock the caller holds.

Jira ticket: MAG-3722
The state machine reads returnCondition at most once and may already have
left on another branch, yet every validateReturnCondition goroutine sent
into the buffer-1 channel unconditionally. A third stop decision inside one
15 ms burst parked its goroutine forever, pinning the whole state machine
and the request it carries. The send is now non-blocking; only the first
reason can matter.

Jira ticket: MAG-3722
The stake map kept one entry per provider per epoch and nothing removed
them. Reads only ever ask for the epoch being scored, so the current and
previous epoch are all that is kept now.

Jira ticket: MAG-3722
…ds per mapper

Review follow-ups on the client release.

A client that unsubscribed everything explicitly and then closed its socket
still leaked: the explicit unsubscribe emptied its subscription set, and the
connection-close release returned early on the empty set before handing back
the sticky endpoint and rate limiters. The early return now releases them.
Unsubscribe's last-client branch and Close go through the same teardown and
release helpers as every other path, so a subscription retires one way only,
and Unsubscribe documents why it trusts the mapper's count.

Router ids are numbered by one mapper-wide counter instead of one per client.
The per-client counter map was the leak on both the ws and the gRPC manager,
and it needed a release call on every exit path to stay bounded; a single
counter needs nothing released and never reuses an id for the life of the
process, so the reuse caveat on an explicit unsubscribe_all goes away too.
The id format is unchanged; only the numbering is mapper-wide.

Jira ticket: MAG-3722
… the half-life change

Review follow-ups on the relay-time window.

The windows live in a plain mutex-guarded map rather than a ristretto cache:
the set is bounded by the pairing list, nothing in it should be evicted, and
ristretto's asynchronous Set let the relay right after a window's birth miss
it and start a second window that dropped the samples in between. The window
grows one sample at a time and wraps once full, so a provider that served a
handful of relays costs a handful of entries.

The comment now says what the change does to routing: with the unbounded
history the median sat at half the uptime, so every provider on a pod up more
than six hours decayed on the 3 h clamp; a busy provider now decays on the
1 h default, which is what the heuristic was written to do. A new test pins
the values at three relay rates.

Jira ticket: MAG-3722
A late push without a request id is dropped as before; one with a request id
now reaches the call path and is answered with an error upstream, where the
stale entry used to swallow it.

Jira ticket: MAG-3722
@nimrod-teich
nimrod-teich merged commit cc92100 into main Sep 17, 2026
23 checks passed
@nimrod-teich
nimrod-teich deleted the mag-3722 branch September 17, 2026 09:58
avitenzer added a commit that referenced this pull request Oct 12, 2026
… between the two checks (MAG-2546)

StartSubscription looks for an active subscription and registers a pending one under
two separate acquisitions of the manager lock. A creator that stored its subscription
and cleared its pending entry in that gap left the second caller seeing neither, so it
became a second creator: a second upstream subscription for the same params, the first
map entry overwritten and orphaned (its listener goroutine and node-side subscription
living on unowned), and totalSubscriptions incremented twice for an entry that is
dropped once. The pending check now re-checks the active map under the same lock and
hands such a caller a settled success so it takes the join path.

The cleanup-ticker branch the ticket names (a cancelled context deleting without
decrementing) was already routed through dropSubscriptionLocked by #386; the new test
pins that so it stays fixed.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Took 47 minutes
AnnaR-prog added a commit that referenced this pull request Oct 14, 2026
…oduction reaches (MAG-2546)

The cancelled-context branch the ticket names cannot be reached in production: subCtx
derives from context.WithoutCancel, and its cancel runs only together with the delete
(the teardown goroutine, cleanupSubscription, Close). The no-clients branch can run
(connectedClients empty while the id mapper says "not last"), and nothing pinned it:
put back to its pre-#386 shape, the whole suite still passed. The new test checks that
it decrements the counter, tells the node, and uncounts the pooled connection.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
AnnaR-prog pushed a commit that referenced this pull request Oct 14, 2026
… between the two checks (MAG-2546) (#508)

StartSubscription looks for an active subscription and registers a pending one under
two separate acquisitions of the manager lock. A creator that stored its subscription
and cleared its pending entry in that gap left the second caller seeing neither, so it
became a second creator: a second upstream subscription for the same params, the first
map entry overwritten and orphaned (its listener goroutine and node-side subscription
living on unowned), and totalSubscriptions incremented twice for an entry that is
dropped once. The pending check now re-checks the active map under the same lock and
hands such a caller a settled success so it takes the join path.

The cleanup-ticker branch the ticket names (a cancelled context deleting without
decrementing) was already routed through dropSubscriptionLocked by #386; the new test
pins that so it stays fixed.

Review fixes (AnnaR-prog):
- test(rpcsmartrouter): pin the janitor's no-clients branch, the one production reaches (MAG-2546)

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants