Repository navigation
fix(chainlib,rpcsmartrouter)!: refuse the gRPC calls the router does not serve before any provider, and answer the gRPC health check (MAG-5133) - #764
Open
avitenzer wants to merge 1 commit into
Conversation
…not serve before any provider, and answer the gRPC health check (MAG-5133) The gRPC listener read the first message of every call and relayed it as a single-reply call. A client stream lost the rest of its messages without an error. A two-way client that waited for a reply hung until its deadline. A server stream the spec does not name was refused inside a provider attempt, or, when no definition was held, ended by grpc-go's "cardinality violation" on the node's second reply, which the router took for a call that never reached the node: retried on every provider and counted as unreachable, so enough such calls switched healthy providers off. A method the spec switched off came back as UNKNOWN. Now, before the listener reads any message of a call, it refuses with UNIMPLEMENTED: - a method the chain spec switched off; - a client stream or a two-way stream; - a server stream, unless the spec names it as a subscription and the listener serves subscriptions. The call's shape comes from a definition the router already holds: each gRPC endpoint's reflection snapshot, then the methods its connection has looked up (RPCSmartRouterServer.GRPCCallShape). Nothing is asked of a node and nothing is waited on. The refusal ends that call only; the connection keeps serving. With no definition held, a stream is caught by what it does: - a second request message, read beside the relay, refuses the call instead of answering the first message alone; - a node's second reply (grpc-go's cardinality violation) is the router's refusal, common.UnservedGRPCCallError: no response, never retried, and the endpoint is released with no score. The in-attempt refusals for a definition found during the attempt carry the same error. What cannot be known without a definition: a two-way client that sends one message and waits without closing its side looks like a single-reply call, and is relayed as one, as before. grpc.health.v1 was registered on the server that reads every message as raw bytes, so its request could not be decoded. It moves to the server that answers reflection, with the standard codec, and answers from the reporter the listener's HTTP health check reads: SERVING or NOT_SERVING for the empty service name, NOT_FOUND for any other. Watch sends the status, then each change. No health call reaches a provider. BREAKING CHANGE: a gRPC caller now gets UNIMPLEMENTED for a method the chain spec switched off (was UNKNOWN), for a server stream the spec does not name (was UNKNOWN), and for a client stream or a two-way stream (were relayed as a single-reply call, or hung). The grpc.health.v1 service now answers (it failed with INTERNAL). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Contributor
Author
|
Pairs with Magma-Devs/lava-specs#210, which switches off the eight Sui and Concordium stream methods; with this PR they are refused with Merge note: this PR and #762 touch the same spot.
I tested the two merged together with the other open gRPC PRs (#761, #765, #767, #768, #770). 🤖 Generated with Claude Code |
This was referenced Nov 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Two gaps in what the gRPC listener serves.
1. Call types the router does not serve are refused before any provider
The listener read the first message of every call and relayed it as a single-reply call:
UNKNOWNUNIMPLEMENTED, before reading the callOKUNIMPLEMENTED, no provider askedUNIMPLEMENTEDat once, no provider askedUNKNOWN, provider counted as failedUNIMPLEMENTED, no provider askedUNIMPLEMENTED, one provider asked, not retried, nobody blamedHow:
GrpcChainListener.refuseUnservedCall, passed to the proxy asgrpcproxy.WithCallGate). It reads the spec (GrpcChainParser.methodInSpec: switched off, or named by a SUBSCRIBE directive) and the call's shape from a definition the router already holds (RPCSmartRouterServer.GRPCCallShape: each endpoint's reflection snapshot, then its connection's looked-up methods). It asks no node and waits on nothing. Running beforeRecvMsgis what answers a two-way client that sends nothing until it hears back.clientSentMore), turns the reply into the refusal. A single-message call has closed its side, so that read ends at once; it waits up to 50 ms only when it has not.handleGRPCErrorreads grpc-go's cardinality text before its node-reached check (grpc-go ends this call without the node's address) and returnscommon.UnservedGRPCCallErrorwith no response.common.UnservedGRPCCallErroris never retried (ShouldRetryErrorWithContext), skips the endpoint-health block inrelayInnerDirect, and releases the session with no score (OnSessionCancelled). The two in-attempt refusals that were already there now carry it too. The listener maps it toUNIMPLEMENTED, with one fixed sentence and the request id inlava-guid.What cannot be known without a definition: a two-way client that sends one message and waits without closing its side looks like a single-reply call. It is relayed as one, as on main (
TestGRPCProxy_WaitingTwoWayClientWithoutADefinitionIsRelayedAsSingleReplypins it). A client stream of exactly one message is relayed too, and is answered correctly, since on the wire it is a single-reply call.2. The gRPC health check answers
grpc.health.v1was registered on the server that reads every message as raw bytes, so its request could not be decoded: every call failed withINTERNAL. It now sits on the server that answers reflection, with the standard codec, and answers from the reporter the listener's HTTP health check reads:"":SERVINGorNOT_SERVING, the HTTP check's answer. With no reporter,SERVING.NOT_FOUND.Watch: the status at once, then each change (checked every second). An unknown name getsSERVICE_UNKNOWNand the call stays open, as the protocol asks.List: the one entry,"".Compressed calls (asked on the ticket)
Answered on the ticket, not changed here: a gzip call works (grpc-go's gzip is registered by an OpenTelemetry exporter the binary links), the router answers in the caller's encoding, and a node's gzip reply is decompressed and re-sent.
Breaking (footer in the commit): the statuses in the table above change for a gRPC caller, and the health service starts answering.
Verification
grpc_unserved_call_test.go), overgrpc.testing.TestServicenodes that count their calls:--bench-afterof 50):UNIMPLEMENTED, the node gets no call, no endpoint counted or switched off, no session failure, and a single-reply call on the same connection is then served;UNIMPLEMENTED, no call;UNIMPLEMENTED, exactly one provider asked per call, nobody blamed;UNIMPLEMENTED.OK, two-way streamDeadlineExceededafter 10 s, the restUNKNOWN. A scratch run on main of the no-definition server stream switched both providers off asunreachableafter 51 calls.health_test.go,health_watch_test.go, through the proxy's HTTP handler): gRPC and HTTP answers agree in both states, unknown nameNOT_FOUND, no reporterSERVING, Watch sends each change, nothing relayed. On main every Check failsINTERNAL"cannot decode".unserved_grpc_call_test.go);handleGRPCErroron the cardinality text with and without the node's address, and a node's ownINTERNALunchanged (grpc_streamed_reply_test.go); the refusal is never retried.fullnode.mainnet.sui.io:443with the lava-specs SUI spec, branch and main built from the same base:LedgerService/ListCheckpoints(enabled in the spec)UNIMPLEMENTED, refused by the gate, no provider askedUNKNOWN"is server-streaming upstream ... refusing"ListCheckpoints, switched off in the specUNIMPLEMENTED"The router does not serve this method"UNKNOWNgrpc.health.v1.Health/CheckSERVING; other nameNOT_FOUNDINTERNAL"cannot decode"LedgerService/GetServiceInfoSubscribeCheckpointsreached the stream path on both and failed the same way on both (the stream's descriptor lookup on this node timed out); not changed here.go build ./...,go vet ./...clean.go test -count=1passes forchainlib/...,routersession/...,common/...,rpcsmartrouter/...,relaycore/...,relaypolicy/.... The new tests pass 5× under-race. golangci-lint: 0 new issues.Ticket tests
UNIMPLEMENTED, no provider askedTestUnservedGRPCCall_RefusedBeforeAnyProviderWithTheDefinition..._ServerStreamWithoutADefinitionIsRefusedWhenItStreams(one provider asked there)TestGRPCHealth_AnswersAsTheHTTPHealthCheck, Watch testJira ticket: MAG-5133
🤖 Generated with Claude Code