Repository navigation
refactor(grpc): give one descriptor resolver to a whole endpoint, not to each channel (MAG-4397) - #601
Conversation
… to each channel (MAG-4397) PR #525 moved the unary connection's descriptor rules into GRPCDescriptorResolver and gave the subscription stream pool the same type. Three comments then said the instance was shared. It was not: every UpstreamGRPCStreamConnection built its own resolver over its own cache, so an endpoint re-paid a cold service's reflection lookup on every channel the pool added. The ticket offered either scope. Sharing is the right one because of what the cached data describes: a method descriptor describes the SERVER, so every channel of one endpoint resolves a service to the same descriptor. A per-channel cache is redundant by construction, not just by measurement. The pool was already the per-endpoint object, holding one nodeUrl and a slice of channels, so the correct owner already existed. So the resolver and its cache move to the pool, which hands the same grpcEndpointDescriptors to every channel it dials. The resolver's two pieces of per-channel state move with it. Its lifetime becomes the pool's, ended in the pool's Close rather than in each channel's. Its borrow becomes borrowAnyChannel, which lends any channel of the endpoint, healthy first: health is the relay path's verdict and a node that fails relays can still answer reflection. The connection slice is snapshotted under the pool lock and the channels are asked outside it, so a lookup can never hold the pool lock while a caller waits on that lookup. That makes the three paths the ticket names free on the second channel: scale-up past streamsPerConn, PoolMinConnections above 1, and reconnect, which dials a fresh channel. It also stops one channel closing aborting a lookup, which the per-channel lifetime did. The two resolvers that remain are the unary connection's and the pool's. They stay separate because their borrow sources are: a resolver can only run reflection on a channel its owner holds, and those two hold different channels across a package boundary. All five comments now say that, in place of three that claimed one shared instance, the resolver type comment that claimed to be "the one resolver behind both", and a test comment that repeated it. Cached is deleted. It had no caller anywhere, tests included. Tests, answering the ticket's acceptance criterion: TwoChannelsOfOneEndpointResolveAColdServiceOnce forces a second channel through StreamsPerConnection 1, resolves a cold service on each, and asserts one reflection stream, the same descriptor pointer from both, and one shared owner object. ALookupBorrowsAnotherChannelWhenItsOwnIsClosed pins the lifetime half: a closed channel no longer makes its endpoint unresolvable. Both fail against a mutant that gives each channel a private owner again. The third test, AClosedPoolRefusesALookupPromptly, does not discriminate and says so: it survives mutants that drop the lifetime cancel and the pool-closed guard, because the closing channels are what end the lookup. Both are kept as belt-and-braces and the comment records that the test does not measure them, rather than letting a reader credit it with more than it checks. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Took 19 minutes
…pool lock (MAG-4397) borrowAnyChannel read the pool's connection list under the pool lock. The pool holds that lock across a dial when it scales up, for up to connectTimeout (30 s by default), so a cold lookup asked by a subscriber that already held a working channel waited for another subscriber's dial. The caller hit its own deadline, and with the default 5 s reflection-timeout the lookup ran out of its own budget before it borrowed anything. Before the resolver moved to the endpoint, a lookup used its caller's channel and never touched the pool lock. The endpoint's descriptors now keep their own list of open channels, under a lock nothing holds across a dial: a channel joins once its dial succeeds and leaves as it closes, and borrowAnyChannel reads that list. The pool's connection list and the code that changes it are untouched. TestUpstreamGRPCPool_AColdLookupDoesNotWaitForAnotherSubscribersDial holds a scale-up dial open on a proxy and asks for a cold descriptor on the first subscriber's channel. It fails on d15130b with the caller's deadline and passes here. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
AnnaR-prog
left a comment
There was a problem hiding this comment.
Reviewed at d15130b. I tested it merged with current main cb736b3, alone and under the #614 stack.
601-1 (Medium): a cold descriptor lookup now waits for another subscriber's scale-up dial.
borrowAnyChannelreadsp.connectionsunderp.lock.pickConnectionLockedholds that lock while it dials a new channel, for up toconnectTimeout(30 s by default).- So a subscriber that already holds a working channel cannot resolve a cold service until another subscriber's scale-up dial ends.
RLockdoes not see the caller's deadline or the lookup's reflection-timeout. When the dial runs to its timeout, the defaults (30 s dial, 5 s reflection-timeout) mean the lookup's own budget is gone before it borrows, so it fails, caches nothing, and every subscriber that joined it fails too. - On main the lookup ran on the caller's own channel and never took the pool lock. Probe on a proxy that holds the second connection (dial timeout 3 s, caller deadline 1 s): main resolves in 4 ms; this head returns "descriptor lookup outlived the relay" after 1 s, with no reflection stream opened.
- The description is right that a lookup never holds the pool lock. It does wait on it.
- I pushed a fix (572b9b6): the endpoint's
grpcEndpointDescriptorskeeps its own list of open channels (a channel joins once its dial succeeds and leaves inClose), andborrowAnyChannelreads that list. The pool's connection list and the code that changes it are untouched, so #614 takes the fix without conflicts.TestUpstreamGRPCPool_AColdLookupDoesNotWaitForAnotherSubscribersDialfails on this head and passes with the fix.
Checked and correct:
- No deadlock: the only
GetMethodDescriptorcaller iscreateNewSubscription, afterReserveConnectionForStreamreleased the lock. The restore path reusesactiveSub.methodDescriptorand does no lookup. - The two discriminating tests fail against a mutant that gives each channel a private owner. The two-resolvers comments match the live paths: one unary connection per provider URL, one stream pool per URL per gRPC listener.
- A stale descriptor after a reconnect to another backend: the cache now lives as long as the pool, the same as the unary connection's cache. Per-channel caching never followed backends either. Replies are re-marshalled in binary, so new reply fields pass through; only a JSON request using a newly added request field is refused, until that service's entries are refreshed (in practice, a restart).
- The unhealthy fallback in
borrowAnyChannelis unreachable today: an open pooled channel is never marked unhealthy (MarkUnhealthyhas no production caller). Harmless.
On refactor vs perf: refactor does not keep this out of the changelog. scripts/changelog-bump.sh lists fix and refactor together under "Bug fixes" and excludes only test and chore (v1.6.4 lists "refactor(lavasession): name the last-chance condition lastChanceBorrow" under Bug fixes). I'd squash it as perf(grpc): …, which lands under "Other work" and says what changed. No ! needed.
| return nil, nil, lavasession.ErrGRPCConnectionClosed | ||
| } | ||
|
|
||
| p.lock.RLock() |
There was a problem hiding this comment.
This RLock waits for any scale-up in progress: pickConnectionLocked dials under p.lock for up to connectTimeout. A cold lookup from a subscriber that already holds a working channel then misses its deadline and, with the default 30 s dial vs 5 s reflection-timeout, its own budget too (601-1). The fix reads a channel list kept by the endpoint descriptors instead.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
AnnaR-prog
left a comment
There was a problem hiding this comment.
Approving at 115908c: main 1e9c85a plus this PR and the 601-1 fix (572b9b6). go build ./... and the TestUpstreamGRPC* tests pass on this exact tree; the full protocol/rpcsmartrouter and protocol/lavasession suites passed on this PR plus the fix merged with main cb736b3. Squashing as perf(grpc): … as discussed.
Jira ticket: MAG-4397
MAG-4397 asked to either reword three comments and drop an unused method, or share the resolver per endpoint. This shares it. The comments and the dead method are handled on the way.
Why share rather than reword
Because of what the cached data describes. A method descriptor describes the server, so every channel of one endpoint resolves a service to the same descriptor. A per-channel cache is redundant by construction, not merely by measurement, and rewording the comments would have left the redundancy in place with an accurate note attached to it.
The owner also already existed.
UpstreamGRPCPoolholds onenodeUrland a slice of channels, so it is the per-endpoint object, and it outlives the channels in it.The resolver had exactly two pieces of per-channel state, and both move up:
lifetimeCloseCloseborrowborrowAnyChannel, any channel of the endpointWhat that buys
The three paths the ticket names stop re-paying a cold service's reflection lookup: scale-up past
streamsPerConn,PoolMinConnectionsabove 1, and reconnect, which dials a fresh channel. The endpoint's second channel is free.It also stops one channel closing aborting a lookup. A lookup now borrows whichever channel is healthy when it runs, so a channel being scaled down, closed or replaced mid-lookup no longer makes its endpoint unresolvable.
borrowAnyChannelprefers a healthy channel but will ask an unhealthy one when it is all there is: health is the relay path's verdict, and a node failing relays can still answer reflection.The deadlock I checked for
borrowAnyChannelneeds the pool lock to read the connection slice, and a lookup is something callers wait on. If a caller heldp.lockwhile waiting, that would deadlock.It does not:
ReserveConnectionForStreamreturns and releases the lock beforeGetMethodDescriptoris called, on both the subscribe path and the restore path, and the locks around those call sites are the manager's, not the pool's. Belt-and-braces anyway,borrowAnyChannelsnapshots the slice under the lock and asks the channels outside it, so a lookup never holds the pool lock at all.Comments and the dead method
Five, not three. The ticket named the resolver field and
GetMethodDescriptorinupstream_grpc_pool.goand the resolver field indirect_rpc_connection.go. Two more said the same thing: theGRPCDescriptorResolvertype comment claimed to be "the one resolver behind both", and a comment in the resolver test repeated it.All five now say what is true: two resolver instances per endpoint, not one, and they do not share a cache. The unary connection owns one and the pool owns the other. They stay separate because their borrow sources are, and a resolver can only run reflection on a channel its owner holds. Making those one object would mean reaching across a package boundary from
rpcsmartrouterintolavasessionfor a handle the subscription manager does not have.Cachedis deleted. It had no caller anywhere, tests included.Tests
TwoChannelsOfOneEndpointResolveAColdServiceOnceis the ticket's acceptance criterion. It forces a second channel throughStreamsPerConnection: 1, resolves one cold service through each, and asserts one reflection stream, the same descriptor pointer from both channels, and one shared owner object.ALookupBorrowsAnotherChannelWhenItsOwnIsClosedpins the lifetime half.Both fail against a mutant that hands each channel a private owner again, which is the pre-fix behaviour expressed in the new structure. The ticket's own test could not be written against the old shape, since the fields it asserts on did not exist, so a mutant is the pre-fix evidence here.
One test does not discriminate, and says so.
AClosedPoolRefusesALookupPromptlysurvives mutants that drop both the lifetime cancel and thep.closedguard, because the closing channels are what actually end the lookup. I kept both guards, an uncancelledcontext.WithCancelleaks one per pool and the guard closes the window betweenborrowreturning and the reflection call starting, and recorded in the test's comment that it does not measure them. Better than leaving a comment that credits the test with more than it checks.Verification
Run from a detached worktree of the commit, not the working directory.
One call for a reviewer
I typed this
refactor, so it stays out of the changelog. There is a real behaviour improvement in it (fewer reflection lookups per endpoint, and lookups that survive a channel closing), soperfwould also be defensible if you would rather it showed. Happy to retype the commit.🤖 Generated with Claude Code