You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit cc92100
Browse filesBrowse the repository at this point in the historyBrowse files
fix: stop the router heap growing for the life of the process (#386)
* fix(provideroptimizer): bound the per-provider relay time history
updateRelayTime appended a time.Time to a per-provider slice on every relay,
success or failure, and nothing ever truncated it: only the debug reset
endpoints clear the cache. That is 24 bytes per relay for the life of the
process, so a busy pod's heap climbed for weeks until the container limit
killed it. Each reallocation also copied the whole history under the
provider's stripe lock, and the slice's Location pointers made the GC walk
all of it every cycle.
calculateHalfTime reads exactly one element, the median, so the history is
now a fixed 128-slot ring in arrival order. Its median means "the time this
provider takes to serve half a window of relays", which is the input the
half-life heuristic wants; the unbounded median drifted to half the uptime
and the MaxHalfTime clamp discarded it anyway. The ring is mutated in place
under the stripe lock, so the cached pointer never needs a replacement Set.
Jira ticket: MAG-3722
* fix(ws): release a client's subscriptions and writer when its connection ends
Three ways a websocket client's departure left state behind for the life of
the process, all on the same path.
The subscription manager never released a JOINED client. A second subscriber
to a query joins the first one's upstream subscription, and only the creating
client had a disconnect watcher: when a joiner closed its socket its reply
channel stayed open, which parks the ingress forwarder goroutine ranging over
it, it kept counting as connected, and the upstream subscription outlived
every real subscriber. The creator's watcher had the mirror omission: it
deleted the client's sender without closing it, so its forwarder parked too.
Every client now attaches through one helper and leaves through one helper
that closes the channel, drops the bookkeeping and wakes the watcher, so no
path can skip a step again. An explicit unsubscribe wakes the watcher instead
of leaving it parked until the socket closes, and UnsubscribeAll now tells the
node when the last client leaves, which it used to skip.
The shared subscription's context was derived from the creating client's
connection, so the creator leaving ended the upstream listener underneath
everyone who had joined. It is now owned by the manager, the rule the gRPC
manager already applies.
The ingress writer goroutine returned on a failed write without telling
anyone. Every later frame sender then blocked forever, and the read loop's
own error path is one of them, so the handler never returned and fasthttp
kept the 128 KiB read buffer, the handler goroutine, the ticker goroutine and
the per-IP limiter slot. The writer now closes a done channel on the way out,
senders select on it, and a failed write sets an immediate read deadline so
the read loop wakes. The handler also releases every subscription it holds
through UnsubscribeAll on return, synchronously, rather than relying on a
cancelled context being noticed.
Smaller entries that never went away: the id mapper's per-client counter
(one per connection that ever subscribed) is dropped with the client, and
rate limiters idle for their refill time are swept, since such an entry
decides exactly as a fresh one would.
Jira ticket: MAG-3722
* fix(rpcclient): drop a subscription's dispatch entry on unsubscribe
The handler's subscription table only shrank when the connection closed.
Upstream websocket connections are pooled and live for the life of the
process, so every subscription ever torn down on one left its entry behind.
Unsubscribe now asks the dispatch loop, the table's only owner, to forget
the entry, on a goroutine of its own because dispatch may be busy delivering
to another subscription whose consumer waits on a lock the caller holds.
Jira ticket: MAG-3722
* fix(relaycore): never block on the return-condition signal
The state machine reads returnCondition at most once and may already have
left on another branch, yet every validateReturnCondition goroutine sent
into the buffer-1 channel unconditionally. A third stop decision inside one
15 ms burst parked its goroutine forever, pinning the whole state machine
and the request it carries. The send is now non-blocking; only the first
reason can matter.
Jira ticket: MAG-3722
* fix(metrics): prune stale epochs from the optimizer stake map
The stake map kept one entry per provider per epoch and nothing removed
them. Reads only ever ask for the epoch being scored, so the current and
previous epoch are all that is kept now.
Jira ticket: MAG-3722
* fix(ws): release per-client state on every exit path, number router ids per mapper
Review follow-ups on the client release.
A client that unsubscribed everything explicitly and then closed its socket
still leaked: the explicit unsubscribe emptied its subscription set, and the
connection-close release returned early on the empty set before handing back
the sticky endpoint and rate limiters. The early return now releases them.
Unsubscribe's last-client branch and Close go through the same teardown and
release helpers as every other path, so a subscription retires one way only,
and Unsubscribe documents why it trusts the mapper's count.
Router ids are numbered by one mapper-wide counter instead of one per client.
The per-client counter map was the leak on both the ws and the gRPC manager,
and it needed a release call on every exit path to stay bounded; a single
counter needs nothing released and never reuses an id for the life of the
process, so the reuse caveat on an explicit unsubscribe_all goes away too.
The id format is unchanged; only the numbering is mapper-wide.
Jira ticket: MAG-3722
* fix(provideroptimizer): keep the relay windows in a locked map, state the half-life change
Review follow-ups on the relay-time window.
The windows live in a plain mutex-guarded map rather than a ristretto cache:
the set is bounded by the pairing list, nothing in it should be evicted, and
ristretto's asynchronous Set let the relay right after a window's birth miss
it and start a second window that dropped the samples in between. The window
grows one sample at a time and wraps once full, so a provider that served a
handful of relays costs a handful of entries.
The comment now says what the change does to routing: with the unbounded
history the median sat at half the uptime, so every provider on a pod up more
than six hours decayed on the 3 h clamp; a busy provider now decays on the
1 h default, which is what the heuristic was written to do. A new test pins
the values at three relay rates.
Jira ticket: MAG-3722
* docs(rpcclient): describe the id-bearing case in forgetClientSub
A late push without a request id is dropped as before; one with a request id
now reaches the call path and is answered with an error upstream, where the
stale entry used to swallow it.
Jira ticket: MAG-3722
0 commit comments