Skip to content

refactor(grpc): give one descriptor resolver to a whole endpoint, not to each channel (MAG-4397) - #601

Merged
AnnaR-prog merged 3 commits into
mainfrom
refactor/MAG-4397-grpc-resolver-scope
Nov 3, 2026
Merged

AnnaR-prog merged 3 commits into
mainfrom
refactor/MAG-4397-grpc-resolver-scope

Conversation

@avitenzer

Copy link
Copy Markdown
Contributor

Jira ticket: MAG-4397

MAG-4397 asked to either reword three comments and drop an unused method, or share the resolver per endpoint. This shares it. The comments and the dead method are handled on the way.

Why share rather than reword

Because of what the cached data describes. A method descriptor describes the server, so every channel of one endpoint resolves a service to the same descriptor. A per-channel cache is redundant by construction, not merely by measurement, and rewording the comments would have left the redundancy in place with an accurate note attached to it.

The owner also already existed. UpstreamGRPCPool holds one nodeUrl and a slice of channels, so it is the per-endpoint object, and it outlives the channels in it.

The resolver had exactly two pieces of per-channel state, and both move up:

Before After
lifetime each channel's, ended in its Close the pool's, ended in the pool's Close
borrow that channel's one channel borrowAnyChannel, any channel of the endpoint
cache one per channel one per endpoint

What that buys

The three paths the ticket names stop re-paying a cold service's reflection lookup: scale-up past streamsPerConn, PoolMinConnections above 1, and reconnect, which dials a fresh channel. The endpoint's second channel is free.

It also stops one channel closing aborting a lookup. A lookup now borrows whichever channel is healthy when it runs, so a channel being scaled down, closed or replaced mid-lookup no longer makes its endpoint unresolvable.

borrowAnyChannel prefers a healthy channel but will ask an unhealthy one when it is all there is: health is the relay path's verdict, and a node failing relays can still answer reflection.

The deadlock I checked for

borrowAnyChannel needs the pool lock to read the connection slice, and a lookup is something callers wait on. If a caller held p.lock while waiting, that would deadlock.

It does not: ReserveConnectionForStream returns and releases the lock before GetMethodDescriptor is called, on both the subscribe path and the restore path, and the locks around those call sites are the manager's, not the pool's. Belt-and-braces anyway, borrowAnyChannel snapshots the slice under the lock and asks the channels outside it, so a lookup never holds the pool lock at all.

Comments and the dead method

Five, not three. The ticket named the resolver field and GetMethodDescriptor in upstream_grpc_pool.go and the resolver field in direct_rpc_connection.go. Two more said the same thing: the GRPCDescriptorResolver type comment claimed to be "the one resolver behind both", and a comment in the resolver test repeated it.

All five now say what is true: two resolver instances per endpoint, not one, and they do not share a cache. The unary connection owns one and the pool owns the other. They stay separate because their borrow sources are, and a resolver can only run reflection on a channel its owner holds. Making those one object would mean reaching across a package boundary from rpcsmartrouter into lavasession for a handle the subscription manager does not have.

Cached is deleted. It had no caller anywhere, tests included.

Tests

TwoChannelsOfOneEndpointResolveAColdServiceOnce is the ticket's acceptance criterion. It forces a second channel through StreamsPerConnection: 1, resolves one cold service through each, and asserts one reflection stream, the same descriptor pointer from both channels, and one shared owner object.

ALookupBorrowsAnotherChannelWhenItsOwnIsClosed pins the lifetime half.

Both fail against a mutant that hands each channel a private owner again, which is the pre-fix behaviour expressed in the new structure. The ticket's own test could not be written against the old shape, since the fields it asserts on did not exist, so a mutant is the pre-fix evidence here.

One test does not discriminate, and says so. AClosedPoolRefusesALookupPromptly survives mutants that drop both the lifetime cancel and the p.closed guard, because the closing channels are what actually end the lookup. I kept both guards, an uncancelled context.WithCancel leaks one per pool and the guard closes the window between borrow returning and the reflection call starting, and recorded in the test's comment that it does not measure them. Better than leaving a comment that credits the test with more than it checks.

Verification

Run from a detached worktree of the commit, not the working directory.

go build -buildvcs=false ./...               clean
go vet ./protocol/...                        clean
go test ./protocol/rpcsmartrouter/           ok 246.525s
go test ./protocol/lavasession/              ok  73.357s
go test -race -run TestUpstreamGRPC ./protocol/rpcsmartrouter/   ok 24.578s

One call for a reviewer

I typed this refactor, so it stays out of the changelog. There is a real behaviour improvement in it (fewer reflection lookups per endpoint, and lookups that survive a channel closing), so perf would also be defensible if you would rather it showed. Happy to retype the commit.

🤖 Generated with Claude Code

… to each channel (MAG-4397)

PR #525 moved the unary connection's descriptor rules into
GRPCDescriptorResolver and gave the subscription stream pool the same
type. Three comments then said the instance was shared. It was not: every
UpstreamGRPCStreamConnection built its own resolver over its own cache, so
an endpoint re-paid a cold service's reflection lookup on every channel the
pool added.

The ticket offered either scope. Sharing is the right one because of what
the cached data describes: a method descriptor describes the SERVER, so
every channel of one endpoint resolves a service to the same descriptor. A
per-channel cache is redundant by construction, not just by measurement.
The pool was already the per-endpoint object, holding one nodeUrl and a
slice of channels, so the correct owner already existed.

So the resolver and its cache move to the pool, which hands the same
grpcEndpointDescriptors to every channel it dials. The resolver's two
pieces of per-channel state move with it. Its lifetime becomes the pool's,
ended in the pool's Close rather than in each channel's. Its borrow becomes
borrowAnyChannel, which lends any channel of the endpoint, healthy first:
health is the relay path's verdict and a node that fails relays can still
answer reflection. The connection slice is snapshotted under the pool lock
and the channels are asked outside it, so a lookup can never hold the pool
lock while a caller waits on that lookup.

That makes the three paths the ticket names free on the second channel:
scale-up past streamsPerConn, PoolMinConnections above 1, and reconnect,
which dials a fresh channel. It also stops one channel closing aborting a
lookup, which the per-channel lifetime did.

The two resolvers that remain are the unary connection's and the pool's.
They stay separate because their borrow sources are: a resolver can only
run reflection on a channel its owner holds, and those two hold different
channels across a package boundary. All five comments now say that, in
place of three that claimed one shared instance, the resolver type comment
that claimed to be "the one resolver behind both", and a test comment that
repeated it.

Cached is deleted. It had no caller anywhere, tests included.

Tests, answering the ticket's acceptance criterion:
TwoChannelsOfOneEndpointResolveAColdServiceOnce forces a second channel
through StreamsPerConnection 1, resolves a cold service on each, and
asserts one reflection stream, the same descriptor pointer from both, and
one shared owner object. ALookupBorrowsAnotherChannelWhenItsOwnIsClosed
pins the lifetime half: a closed channel no longer makes its endpoint
unresolvable.

Both fail against a mutant that gives each channel a private owner again.
The third test, AClosedPoolRefusesALookupPromptly, does not discriminate
and says so: it survives mutants that drop the lifetime cancel and the
pool-closed guard, because the closing channels are what end the lookup.
Both are kept as belt-and-braces and the comment records that the test
does not measure them, rather than letting a reader credit it with more
than it checks.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Took 19 minutes
…pool lock (MAG-4397)

borrowAnyChannel read the pool's connection list under the pool lock. The
pool holds that lock across a dial when it scales up, for up to
connectTimeout (30 s by default), so a cold lookup asked by a subscriber
that already held a working channel waited for another subscriber's dial.
The caller hit its own deadline, and with the default 5 s
reflection-timeout the lookup ran out of its own budget before it borrowed
anything. Before the resolver moved to the endpoint, a lookup used its
caller's channel and never touched the pool lock.

The endpoint's descriptors now keep their own list of open channels, under
a lock nothing holds across a dial: a channel joins once its dial succeeds
and leaves as it closes, and borrowAnyChannel reads that list. The pool's
connection list and the code that changes it are untouched.

TestUpstreamGRPCPool_AColdLookupDoesNotWaitForAnotherSubscribersDial holds
a scale-up dial open on a proxy and asks for a cold descriptor on the first
subscriber's channel. It fails on d15130b with the caller's deadline and
passes here.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

@AnnaR-prog AnnaR-prog left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at d15130b. I tested it merged with current main cb736b3, alone and under the #614 stack.

601-1 (Medium): a cold descriptor lookup now waits for another subscriber's scale-up dial.

  • borrowAnyChannel reads p.connections under p.lock. pickConnectionLocked holds that lock while it dials a new channel, for up to connectTimeout (30 s by default).
  • So a subscriber that already holds a working channel cannot resolve a cold service until another subscriber's scale-up dial ends. RLock does not see the caller's deadline or the lookup's reflection-timeout. When the dial runs to its timeout, the defaults (30 s dial, 5 s reflection-timeout) mean the lookup's own budget is gone before it borrows, so it fails, caches nothing, and every subscriber that joined it fails too.
  • On main the lookup ran on the caller's own channel and never took the pool lock. Probe on a proxy that holds the second connection (dial timeout 3 s, caller deadline 1 s): main resolves in 4 ms; this head returns "descriptor lookup outlived the relay" after 1 s, with no reflection stream opened.
  • The description is right that a lookup never holds the pool lock. It does wait on it.
  • I pushed a fix (572b9b6): the endpoint's grpcEndpointDescriptors keeps its own list of open channels (a channel joins once its dial succeeds and leaves in Close), and borrowAnyChannel reads that list. The pool's connection list and the code that changes it are untouched, so #614 takes the fix without conflicts. TestUpstreamGRPCPool_AColdLookupDoesNotWaitForAnotherSubscribersDial fails on this head and passes with the fix.

Checked and correct:

  • No deadlock: the only GetMethodDescriptor caller is createNewSubscription, after ReserveConnectionForStream released the lock. The restore path reuses activeSub.methodDescriptor and does no lookup.
  • The two discriminating tests fail against a mutant that gives each channel a private owner. The two-resolvers comments match the live paths: one unary connection per provider URL, one stream pool per URL per gRPC listener.
  • A stale descriptor after a reconnect to another backend: the cache now lives as long as the pool, the same as the unary connection's cache. Per-channel caching never followed backends either. Replies are re-marshalled in binary, so new reply fields pass through; only a JSON request using a newly added request field is refused, until that service's entries are refreshed (in practice, a restart).
  • The unhealthy fallback in borrowAnyChannel is unreachable today: an open pooled channel is never marked unhealthy (MarkUnhealthy has no production caller). Harmless.

On refactor vs perf: refactor does not keep this out of the changelog. scripts/changelog-bump.sh lists fix and refactor together under "Bug fixes" and excludes only test and chore (v1.6.4 lists "refactor(lavasession): name the last-chance condition lastChanceBorrow" under Bug fixes). I'd squash it as perf(grpc): …, which lands under "Other work" and says what changed. No ! needed.

return nil, nil, lavasession.ErrGRPCConnectionClosed
}

p.lock.RLock()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This RLock waits for any scale-up in progress: pickConnectionLocked dials under p.lock for up to connectTimeout. A cold lookup from a subscriber that already holds a working channel then misses its deadline and, with the default 30 s dial vs 5 s reflection-timeout, its own budget too (601-1). The fix reads a channel list kept by the endpoint descriptors instead.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

@AnnaR-prog AnnaR-prog left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving at 115908c: main 1e9c85a plus this PR and the 601-1 fix (572b9b6). go build ./... and the TestUpstreamGRPC* tests pass on this exact tree; the full protocol/rpcsmartrouter and protocol/lavasession suites passed on this PR plus the fix merged with main cb736b3. Squashing as perf(grpc): … as discussed.

@AnnaR-prog
AnnaR-prog merged commit 157e1ee into main Nov 3, 2026
@AnnaR-prog
AnnaR-prog deleted the refactor/MAG-4397-grpc-resolver-scope branch November 3, 2026 15:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants