Skip to content

fix(chainlib,rpcsmartrouter)!: refuse the gRPC calls the router does not serve before any provider, and answer the gRPC health check (MAG-5133) - #764

Open
avitenzer wants to merge 1 commit into
mainfrom
fix/MAG-5133-refuse-unserved-grpc-calls
Open

avitenzer wants to merge 1 commit into
mainfrom
fix/MAG-5133-refuse-unserved-grpc-calls

Conversation

@avitenzer

Copy link
Copy Markdown
Contributor

Description

Two gaps in what the gRPC listener serves.

1. Call types the router does not serve are refused before any provider

The listener read the first message of every call and relayed it as a single-reply call:

Call main this PR
Method the spec switched off UNKNOWN UNIMPLEMENTED, before reading the call
Client stream first message relayed, the rest dropped, OK UNIMPLEMENTED, no provider asked
Two-way stream, client waits for a reply hangs until the client's deadline UNIMPLEMENTED at once, no provider asked
Server stream the spec does not name, definition held refused inside a provider attempt, UNKNOWN, provider counted as failed UNIMPLEMENTED, no provider asked
Same, no definition held node's 2nd reply → grpc-go "cardinality violation", taken for an undelivered call: retried on every provider, each counted as unreachable UNIMPLEMENTED, one provider asked, not retried, nobody blamed
Server stream the spec names as a subscription served served, unchanged
Single-reply call served served, unchanged

How:

  • Gate before the first read (GrpcChainListener.refuseUnservedCall, passed to the proxy as grpcproxy.WithCallGate). It reads the spec (GrpcChainParser.methodInSpec: switched off, or named by a SUBSCRIBE directive) and the call's shape from a definition the router already holds (RPCSmartRouterServer.GRPCCallShape: each endpoint's reflection snapshot, then its connection's looked-up methods). It asks no node and waits on nothing. Running before RecvMsg is what answers a two-way client that sends nothing until it hears back.
  • No definition held: a stream is caught by what it does.
    • A second request message, read beside the relay (clientSentMore), turns the reply into the refusal. A single-message call has closed its side, so that read ends at once; it waits up to 50 ms only when it has not.
    • A node's second reply: handleGRPCError reads grpc-go's cardinality text before its node-reached check (grpc-go ends this call without the node's address) and returns common.UnservedGRPCCallError with no response.
  • common.UnservedGRPCCallError is never retried (ShouldRetryErrorWithContext), skips the endpoint-health block in relayInnerDirect, and releases the session with no score (OnSessionCancelled). The two in-attempt refusals that were already there now carry it too. The listener maps it to UNIMPLEMENTED, with one fixed sentence and the request id in lava-guid.
  • A refusal ends that call only; the connection keeps serving.

What cannot be known without a definition: a two-way client that sends one message and waits without closing its side looks like a single-reply call. It is relayed as one, as on main (TestGRPCProxy_WaitingTwoWayClientWithoutADefinitionIsRelayedAsSingleReply pins it). A client stream of exactly one message is relayed too, and is answered correctly, since on the wire it is a single-reply call.

2. The gRPC health check answers

grpc.health.v1 was registered on the server that reads every message as raw bytes, so its request could not be decoded: every call failed with INTERNAL. It now sits on the server that answers reflection, with the standard codec, and answers from the reporter the listener's HTTP health check reads:

  • Service "": SERVING or NOT_SERVING, the HTTP check's answer. With no reporter, SERVING.
  • Any other service name: NOT_FOUND.
  • Watch: the status at once, then each change (checked every second). An unknown name gets SERVICE_UNKNOWN and the call stays open, as the protocol asks.
  • List: the one entry, "".
  • No health call reaches a provider.

Compressed calls (asked on the ticket)

Answered on the ticket, not changed here: a gzip call works (grpc-go's gzip is registered by an OpenTelemetry exporter the binary links), the router answers in the caller's encoding, and a node's gzip reply is decompressed and re-sent.

Breaking (footer in the commit): the statuses in the table above change for a gRPC caller, and the health service starts answering.


Verification

  • End to end, real listener in front of the real router (grpc_unserved_call_test.go), over grpc.testing.TestService nodes that count their calls:
    • definition held: a client stream, a waiting two-way stream and an unnamed server stream, each 51 times (past the test's --bench-after of 50): UNIMPLEMENTED, the node gets no call, no endpoint counted or switched off, no session failure, and a single-reply call on the same connection is then served;
    • a switched-off method: UNIMPLEMENTED, no call;
    • no definition, two providers: an unnamed server stream 51 times: UNIMPLEMENTED, exactly one provider asked per call, nobody blamed;
    • no definition: a client stream of two messages: UNIMPLEMENTED.
  • Pre-fix evidence: the same file run against main's production code fails every case: client stream OK, two-way stream DeadlineExceeded after 10 s, the rest UNKNOWN. A scratch run on main of the no-definition server stream switched both providers off as unreachable after 51 calls.
  • Health (health_test.go, health_watch_test.go, through the proxy's HTTP handler): gRPC and HTTP answers agree in both states, unknown name NOT_FOUND, no reporter SERVING, Watch sends each change, nothing relayed. On main every Check fails INTERNAL "cannot decode".
  • Unit: the gate's reasons for each shape and the subscription cases (unserved_grpc_call_test.go); handleGRPCError on the cardinality text with and without the node's address, and a node's own INTERNAL unchanged (grpc_streamed_reply_test.go); the refusal is never retried.
  • Live, fullnode.mainnet.sui.io:443 with the lava-specs SUI spec, branch and main built from the same base:
branch main
LedgerService/ListCheckpoints (enabled in the spec) UNIMPLEMENTED, refused by the gate, no provider asked UNKNOWN "is server-streaming upstream ... refusing"
ListCheckpoints, switched off in the spec UNIMPLEMENTED "The router does not serve this method" UNKNOWN
grpc.health.v1.Health/Check SERVING; other name NOT_FOUND INTERNAL "cannot decode"
LedgerService/GetServiceInfo served served

SubscribeCheckpoints reached the stream path on both and failed the same way on both (the stream's descriptor lookup on this node timed out); not changed here.

  • Suites: go build ./..., go vet ./... clean. go test -count=1 passes for chainlib/..., routersession/..., common/..., rpcsmartrouter/..., relaycore/..., relaypolicy/.... The new tests pass 5× under -race. golangci-lint: 0 new issues.

Ticket tests

# Test Where
1 A server stream the spec names: served gate unit test; the stream path is unchanged
2 A single-reply call: served every end-to-end test
3, 4 Client stream, two-way stream: UNIMPLEMENTED, no provider asked TestUnservedGRPCCall_RefusedBeforeAnyProviderWithTheDefinition
5 Unnamed server stream, with and without the definition the same test, and ..._ServerStreamWithoutADefinitionIsRefusedWhenItStreams (one provider asked there)
6 The connection serves after a refusal every refusal test
7 Router process unchanged automation
8–11 Health check: serving, not serving, back, same as HTTP TestGRPCHealth_AnswersAsTheHTTPHealthCheck, Watch test
12 Health answered by the router the same tests: nothing relayed

Jira ticket: MAG-5133

🤖 Generated with Claude Code

…not serve before any provider, and answer the gRPC health check (MAG-5133)

The gRPC listener read the first message of every call and relayed it as a
single-reply call. A client stream lost the rest of its messages without an
error. A two-way client that waited for a reply hung until its deadline. A
server stream the spec does not name was refused inside a provider attempt,
or, when no definition was held, ended by grpc-go's "cardinality violation"
on the node's second reply, which the router took for a call that never
reached the node: retried on every provider and counted as unreachable, so
enough such calls switched healthy providers off. A method the spec switched
off came back as UNKNOWN.

Now, before the listener reads any message of a call, it refuses with
UNIMPLEMENTED:

- a method the chain spec switched off;
- a client stream or a two-way stream;
- a server stream, unless the spec names it as a subscription and the
  listener serves subscriptions.

The call's shape comes from a definition the router already holds: each
gRPC endpoint's reflection snapshot, then the methods its connection has
looked up (RPCSmartRouterServer.GRPCCallShape). Nothing is asked of a node
and nothing is waited on. The refusal ends that call only; the connection
keeps serving.

With no definition held, a stream is caught by what it does:

- a second request message, read beside the relay, refuses the call
  instead of answering the first message alone;
- a node's second reply (grpc-go's cardinality violation) is the router's
  refusal, common.UnservedGRPCCallError: no response, never retried, and
  the endpoint is released with no score.

The in-attempt refusals for a definition found during the attempt carry the
same error. What cannot be known without a definition: a two-way client
that sends one message and waits without closing its side looks like a
single-reply call, and is relayed as one, as before.

grpc.health.v1 was registered on the server that reads every message as raw
bytes, so its request could not be decoded. It moves to the server that
answers reflection, with the standard codec, and answers from the reporter
the listener's HTTP health check reads: SERVING or NOT_SERVING for the
empty service name, NOT_FOUND for any other. Watch sends the status, then
each change. No health call reaches a provider.

BREAKING CHANGE: a gRPC caller now gets UNIMPLEMENTED for a method the chain
spec switched off (was UNKNOWN), for a server stream the spec does not name
(was UNKNOWN), and for a client stream or a two-way stream (were relayed
as a single-reply call, or hung). The grpc.health.v1 service now answers
(it failed with INTERNAL).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@avitenzer
avitenzer requested a review from a team as a code owner November 19, 2026 07:24
@avitenzer

Copy link
Copy Markdown
Contributor Author

Pairs with Magma-Devs/lava-specs#210, which switches off the eight Sui and Concordium stream methods; with this PR they are refused with UNIMPLEMENTED.

Merge note: this PR and #762 touch the same spot.

I tested the two merged together with the other open gRPC PRs (#761, #765, #767, #768, #770). go vet is clean and the chainlib, routersession, common, relaycore and rpcsmartrouter suites pass.

🤖 Generated with Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant