Repository navigation
fix(protocol)!: return the node's answer as is on every interface; 502/504 only when no node answered (MAG-4481) - #587
Conversation
avitenzer
left a comment
There was a problem hiding this comment.
@Tomelia1999 I reviewed this at 33ddf2ad. go vet ./protocol/... is clean, and the tests in chainlib, relaycore, common, lavasession and endpointstate pass, along with the PR's tests in rpcsmartrouter. The rest of the change looks good to me: the caller's id is already in the node's reply, the ranking order holds, and writes keep their 500 on the hung-upstream path. One regression needs fixing before merge, see the inline comment.
| // Every attempt was refused for rate: the chain is temporarily unservable, not broken. | ||
| // 503 is the honest status and what well-behaved clients back off on; no Retry-After | ||
| // is surfaced — the hold-off registry owns that, internally. | ||
| if onlyRateLimited(nodeErrors, protocolErrors) { |
There was a problem hiding this comment.
Regression: when every attempt is rate limited, some transports now get a 500 instead of a 503.
This block was the only thing that set a status for the all-429 case. The new code above only covers 429s that arrive as lavasession.HTTPStatusError. Two other paths still produce rate-limit errors but now get no status:
WebSocketDirectRPCConnection, a 429 when the connection is opened (wsHandshakeErrorunwraps tocommon.RateLimitedError). This is the relay path for WebSocket-only provider URLs.- Direct gRPC, where a confirmed rate limit arrives as
GRPCStatusErrorwithrateLimitedset.
For these, clientFacingFromProtocolError returns nil, so StatusCode stays 0 and the HTTP listeners send a 500. On main this was a 503.
Repro: in relaycore, send two common.RateLimited(errors.New("websocket: bad handshake"), 0) protocol errors through newTwoProviderProcessor, then call ProcessingResult(). You get OnlyRateLimited=true, StatusCode=0 and ClientFacing=nil.
The tests don't catch this because TestProcessingResult_AllRateLimitedIsTheNodes429 was switched from common.RateLimited(errors.New("HTTP 429")) to HTTPStatusError, so this input is no longer tested.
Suggested fix: keep a fallback for 429s that carry no node status, using the ticket's 429:
if onlyRateLimited(nodeErrors, protocolErrors) && returnedResult.GetClientFacing().GetUpstreamStatus() == 0 {
stamped := *returnedResult // copy: returnedResult aliases the stored result
stamped.StatusCode = http.StatusTooManyRequests
returnedResult = &stamped
}(Read ClientFacing however you prefer. The point is to set the status only when no node status was forwarded.) Please also add a test with the common.RateLimited(...) input next to the HTTPStatusError one, and update the PR description's line "All attempts rate limited: the node's 429" if the fallback ends up being something other than 429.
There was a problem hiding this comment.
Thanks, confirmed: your repro gave status 0 on this branch and 503 on main. Fixed in 1ae7df8, which is now rebased on main.
- When every attempt was rate limited and none of them has a node status to forward (a 429 on the WebSocket handshake, or a gRPC
RESOURCE_EXHAUSTED), the reply is now a plain 429. - This only runs in that case. If a node answer is forwarded, its status and body are never overwritten.
- New test
TestProcessingResult_AllRateLimitedWithoutNodeStatusIs429uses yourcommon.RateLimited(...)input. It fails without the fix. - New test
TestProcessingResult_ForwardedNode429BeatsStatuslessRateLimitcovers a node's own 429 next to a 429 with no status. The node's 429 and body win in either arrival order. - I kept the PR description's 429 line and added this case to it. I also added a sentence about it to
docs/RATE-LIMIT-HOLDOFF.md.
The rebase conflicted with #559 (MAG-4454) in the four HTTP listeners. The failure log lines now use the cut request body (loggedBody / logFormattedMsg), as main does, with this PR's reply. go vet is clean, and relaycore, chainlib, rpcsmartrouter, lavasession, endpointstate, common, metrics and utils all pass.
…when no node answered (MAG-4481) The router is a proxy: when a node answered, its status, body and Content-Type go back to the client untouched. The router speaks only when no node answered. - JSON-RPC, REST, Tendermint RPC and GraphQL forward the node's answer byte for byte: no router envelope, no added jsonrpc field, the GUID stays in the Lava-Guid header. One shared builder, chainlib/failed_relay_reply.go (was upstream_error_envelope.go). - httpStatusRelayError keeps the node's status, body and Content-Type, which is what lets REST failures and every 429 forward the answer. Classification is unchanged for 400-599 on all three transports. - All attempts rate limited: the node's 429, not 503. - No node answered: 502 "no answer from upstream: <cause>", or 504 "no answer from upstream within <timeout>" for a timeout, including a hung upstream (was 500 "context deadline exceeded"). 1009 joins the transport set, 1043 leaves it, router_detail is gone. - The unclear-write 500 is unchanged. - The request log records what the client received; the relay-failure log line gains error_code, error_name, upstream_status and upstream_body; the poller logs the node's status and body. - End-to-end test through a real listener, router and sender on all four interfaces. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ttempt fails (MAG-4481) Keeping the 429's body made it an upstream answer, the same rank as a 503, so two 429s outvoted one 503 and the client got the throttle instead of the fault. Before MAG-4481 the 503 won (the MAG-4638 QA test). A 429 now ranks just below the other upstream answers, and still above router text and transport failures, so all-429 stays 429 and 429 next to a refused connection stays 429. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…504; pin the new statuses in relay_end (MAG-4481) A refused connection and a timeout both ranked as "no upstream answered", so the majority pick broke the one-to-one tie at random: the same failure came back 502 on about 33 runs in 40 and 504 on the rest. A timeout now ranks above the other transport failures, and both still lose to any other error. relay_end gains golden cases for the statuses this PR introduced: a node's 503 forwarded, every attempt throttled (429) and a node that never answers (504). Notes brought up to date: relay_end's has_reply and no_answer rows (an upstream's error status is a no_answer with that status), the fully capped chain in RATE-LIMIT-HOLDOFF (the node's 429, no longer a router 503), the REST 5xx section of ERROR-REGISTRY-DESIGN, and the 504 timeout, which names the request's budget rather than the time the attempt took. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…o node status (MAG-4481) A 429 on a WebSocket handshake or a gRPC RESOURCE_EXHAUSTED carries no node status to forward, so the all-rate-limited reply had no status and the HTTP listeners sent 500 (main sent 503). Set 429 in that case only; a forwarded node answer keeps its own status and body. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
0a7f514 to
1ae7df8
Compare
avitenzer
left a comment
There was a problem hiding this comment.
Re-checked at 1ae7df8f. The regression I raised is fixed.
go test ./protocol/relaycore/passes. With1ae7df8f'srelay_processor.gochange reverted,TestProcessingResult_AllRateLimitedWithoutNodeStatusIs429fails withexpected: 429, actual: 0, which is the status-0 case. The new branch only runs when no node answer was forwarded, so a node's own 429 body is never overwritten.TestProcessingResult_ForwardedNode429BeatsStatuslessRateLimitcovers both arrival orders.
Not approving yet. The branch no longer merges into main (278a4a0e). #611 (one handler for every HTTP listener route) conflicts in chainlib/graphql.go, jsonRPC.go, rest.go, tendermintRPC.go, and in endpointstate/endpoint_poller.go. Those are the listener reply paths this PR rewrites, so the resolution needs its own review. Please rebase. I'll approve and merge once the rebased head is green and TestFailedRelayReply_EndToEnd_MAG4481 still passes on it.
Resolve the conflicts with #611 (one handler for every HTTP listener route) and #608 (one direct HTTP sender): - chainlib: the six listener routes now share relayHTTP. MAG-4481's failed-relay reply moves into it: buildFailedRelayReply first (the node's answer as is, or the router's 502/504), else the masked errorBody with failedRelayStatus, and the request log records what the client is sent. httpRelaySpec gains noAnswer, the shape of the router's own no-answer reply: JSON-RPC for jsonrpc and Tendermint POST, {"error": ...} for REST, GraphQL and Tendermint GET. The jsonrpc errorBody loses the MAG-4218 convertUpstreamErrorToJsonRpc branch, as on this branch before the merge. - lavasession/endpointstate: both producers use main's NewHTTPStatusError, which now also fills ContentType from the reply's headers. - rpcsmartrouter: relayFailureLogAttrs calls common.AsLavaError; #604 folded extractLavaError into it. - listener_handlers.golden.json: regenerated. Only the 8 "error with upstream answer" cases change: the node's body and status as is, instead of the router's envelope. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Took 47 minutes
avitenzer
left a comment
There was a problem hiding this comment.
Approving at 721c91e0.
I resolved the conflicts with main by merging main into the branch (721c91e0, a fast-forward on top of 1ae7df8f; no history was rewritten):
- chainlib: #611 folded the six listener routes into one
relayHTTP. MAG-4481's failed-relay reply now lives there once.buildFailedRelayReplyruns first; otherwise the maskederrorBodyis sent withfailedRelayStatus, and the request log records what the client receives.httpRelaySpec.noAnswercarries each route's no-answer shape, as the branch had it per route: JSON-RPC for jsonrpc and Tendermint POST,{"error": ...}for REST POST/other, GraphQL and Tendermint GET. The jsonrpcerrorBodydrops the MAG-4218convertUpstreamErrorToJsonRpcbranch, as your branch already did. - lavasession / endpointstate: both producers use main's
NewHTTPStatusError, which now also fillsContentTypefrom the reply headers. Those headers come from net/http, so the keys are canonical. - rpcsmartrouter: this was a semantic conflict that git merged without a marker.
relayFailureLogAttrscalledextractLavaError, which #604 folded intocommon.AsLavaError(same body). It now callscommon.AsLavaError. listener_handlers.golden.json(new on main) is regenerated. Exactly 8 cases change, all of themerror with upstream answer(one per route). The node's body and status now go back as is instead of the router's envelope, and the Content-Type is no longer forced to JSON when the upstream sent none. No other case moved.
Verified on the merged tree:
go build ./...,go vet ./protocol/...andgofmtare clean.go testpasses for chainlib/..., endpointstate, relaycore, common, lavasession, metrics and rpcsmartrouter.TestFailedRelayReply_EndToEnd_MAG4481passes 33/33: 4 interfaces × node answer, empty body, 520, 429, refused → 502, hung → 504, masked.- CI on this head is green, including lint (dupl guard) and unit-tests-race.
My earlier finding (all-rate-limited with no node status) is fixed by 1ae7df8f, as noted above.
…t the configured key, end to end (MAG-4863) Since #587 a node's 4xx/5xx body goes back as is, built from the client-facing error this PR redacts. The new test runs a real listener and router on every HTTP interface and checks the reply is the node's body with the key withheld, inside its url and on its own. Also: the comments in relaycore named the envelope and upstream_body snippet that #587 removed, and the new relaycore test sat between TestClientFacingFromProtocolError_TransportFailures and its doc comment. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…es and logs (MAG-4863) (#648) * fix(security): withhold configured upstream secrets from client replies and logs (MAG-4863) A node's own error reached the caller with the operator's key in it. A JSON-RPC error inside an HTTP 200 was relayed as the reply with no redaction at all, and the url-shape redaction that every other exit uses missed a key after a ";" in a query, a url with escaped slashes, and any key outside a url. - Redaction by value: ParseStaticProviderEndpoints registers every secret a node-url carries (url password or user, path segments and query values that look like a key, auth-query values, auth-header values and their tokens). RedactSecrets replaces each one, raw, percent-encoded and JSON-escaped, before its url pass, so every log line, client error reply, upstream_body and router_detail loses it. - The url pass now keeps a query's ";" inside the url and redacts urls with escaped slashes without breaking the JSON around them. - relayInnerDirect redacts a node-error reply before it is returned, cached or compared; the node's error is still the reply. The status >= 400 body is redacted once, where the client-facing error takes it. - The nine log attributes that printed a configured node-url raw use UrlStr(), and a failed provider-config parse no longer logs every setting. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix(utils): leave one mark for a registered key inside a url (MAG-4863) The value pass replaces a registered key first, and the url pass then stops at the "[" of that mark, so a configured node-url came out as "https://host/[redacted][redacted]" in every log line and node error that repeated it. RedactSecrets now folds the repeated marks into one. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix(rpcsmartrouter): log the config key, not the settings, when the listen endpoints fail to parse (MAG-4863) ParseEndpoints is handed the whole viper config, and its fatal line logged AllSettings(): every node-url and auth header in the file, before any of them is registered for redaction. ParseStaticProviderEndpoints got the same fix in this PR. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * test(rpcsmartrouter): a node's error status reaches the client without the configured key, end to end (MAG-4863) Since #587 a node's 4xx/5xx body goes back as is, built from the client-facing error this PR redacts. The new test runs a real listener and router on every HTTP interface and checks the reply is the node's body with the key withheld, inside its url and on its own. Also: the comments in relaycore named the envelope and upstream_body snippet that #587 removed, and the new relaycore test sat between TestClientFacingFromProtocolError_TransportFailures and its doc comment. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Description
Step 2 of MAG-4218 (step 1 was #551). The router is a proxy: when a node answered, the client gets that answer as is (same status, same bytes, the node's Content-Type). The router speaks only when no node answered.
Client-visible changes
jsonrpc, no GUID in the body (it stays in theLava-Guidheader). Before, only JSON-RPC rendered the node's answer, re-marshalled; REST and every 429 dropped the body.no answer from upstream: <cause>; 504no answer from upstream within <timeout>for a timeout, including a hung upstream (was 500context deadline exceeded). A timeout next to a refused connection is always the 504.<timeout>is the request's processing timeout, not how long the attempt took. Same envelope as before, minusrouter_detail.down→ the node's 503 (was 500), every primary 429 → 429 (was 503).How
httpStatusRelayError(rpcsmartrouter_server.go) returns a*lavasession.HTTPStatusErrorcarrying the node's status, body and Content-Type instead of a bareHTTP %d.TestHTTPStatusRelayError_ClassifiesAsThePlainStatusDidproves classification is identical for 400–599 on GRPC, REST and JsonRPC transports.chainlib/failed_relay_reply.go(renamed fromupstream_error_envelope.go).relaycore: the all-rate-limited → 503 stamp is gone;replyRankranks a node's 429 just below its other error answers (keeping the 429's body would otherwise let two 429s outvote one 503); 1009 joins the no-answer set, 1043 leaves it; timeouts (1001, 1007) get 504 and rank above the other no-answers, so the same failure always gets the same status (the majority pick would otherwise choose 502 or 504 at random).idrestore needed: the router forwards the caller's own request bytes, so the node already echoes the caller's id.Logs
direct RPC relay failed in goroutinegainserror_code,error_name,upstream_status,upstream_body(256-byte cap, redacted).docs/RELAY-END-EVENT.md:status,has_replyandno_answerrows updated: an upstream's forwarded error status is ano_answerwith thatstatus. No schema bump.RATE-LIMIT-HOLDOFF.mdandERROR-REGISTRY-DESIGN.md§9.2 updated to match.Files to review first:
protocol/chainlib/failed_relay_reply.go,protocol/rpcsmartrouter/rpcsmartrouter_server.go(httpStatusRelayError, hung-upstream 504),protocol/relaycore/client_facing_error.go,protocol/relaycore/relay_processor.go.Tests
TestFailedRelayReply_EndToEnd_MAG4481: real listener + router + direct sender againsthttptestupstreams on jsonrpc, rest, tendermintrpc and graphql. Covers node 503 HTML, empty 503, 520, 429, refused → 502, hung → 504, and masked mode.relay_endgolden cases:read_upstream_error_status(503),read_rate_limited(429),read_no_answer_in_time(504). NewTestProcessingResult_ATimeoutBeatsARefusedConnectionandTestReplyProtocolError_ATimeoutBeatsOtherTransportFailures(both fail without the rank change). NewTestProcessingResult_AllRateLimitedWithoutNodeStatusIs429(fails without the fallback: status 0, so the listener sent 500) andTestProcessingResult_ForwardedNode429BeatsStatuslessRateLimit(the node's own 429 body wins in either order).HTTP 503error text) inrate_limit_failure_test.go,stateful_rate_limit_broadcast_test.go,write_outcome_rest_test.go,http_status_relay_error_test.go,client_facing_error_test.go. The tests they replaced are renamed, not dropped.down500 → 503; write with every node 429 503 → 429; GraphQL write with every primary dropping the connection stays 500; a solodrop_connectionstill saysEOF; cross-validation replies (quiet group member,-32601member) unchanged.Not in this PR (decided)
Retry-Afteris not forwarded (same as main).Jira ticket
Jira ticket: MAG-4481
🤖 Generated with Claude Code