Skip to content

fix(protocol)!: return the node's answer as is on every interface; 502/504 only when no node answered (MAG-4481) - #587

Merged
avitenzer merged 6 commits into
mainfrom
fix/MAG-4481-node-answer-as-is
Nov 4, 2026
Merged

avitenzer merged 6 commits into
mainfrom
fix/MAG-4481-node-answer-as-is

Conversation

@Tomelia1999

@Tomelia1999 Tomelia1999 commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Description

Step 2 of MAG-4218 (step 1 was #551). The router is a proxy: when a node answered, the client gets that answer as is (same status, same bytes, the node's Content-Type). The router speaks only when no node answered.

Client-visible changes

  • Node answered: JSON-RPC, REST, Tendermint RPC and GraphQL return the node's status and body byte for byte. No router envelope, no added jsonrpc, no GUID in the body (it stays in the Lava-Guid header). Before, only JSON-RPC rendered the node's answer, re-marshalled; REST and every 429 dropped the body.
  • All attempts rate limited: the node's 429 and body (was 503). When no attempt carried a node status to forward (a 429 on a WebSocket handshake, a gRPC RESOURCE_EXHAUSTED), a plain 429.
  • Mixed answers: a node's fault still beats its peers' throttles: one 503 next to two 429s is the 503, as on main (the MAG-4638 QA test). A 429 next to a refused connection is the 429.
  • No node answered: 502 no answer from upstream: <cause>; 504 no answer from upstream within <timeout> for a timeout, including a hung upstream (was 500 context deadline exceeded). A timeout next to a refused connection is always the 504. <timeout> is the request's processing timeout, not how long the attempt took. Same envelope as before, minus router_detail.
  • Writes: the unclear-write 500 is unchanged and still wins whenever a node may hold the transaction (a silent node, a 502/504, a dropped connection). When every node answered, a write gets the node's answer like a read: every primary down → the node's 503 (was 500), every primary 429 → 429 (was 503).

How

  • httpStatusRelayError (rpcsmartrouter_server.go) returns a *lavasession.HTTPStatusError carrying the node's status, body and Content-Type instead of a bare HTTP %d. TestHTTPStatusRelayError_ClassifiesAsThePlainStatusDid proves classification is identical for 400–599 on GRPC, REST and JsonRPC transports.
  • One reply builder for all four listeners: chainlib/failed_relay_reply.go (renamed from upstream_error_envelope.go).
  • relaycore: the all-rate-limited → 503 stamp is gone; replyRank ranks a node's 429 just below its other error answers (keeping the 429's body would otherwise let two 429s outvote one 503); 1009 joins the no-answer set, 1043 leaves it; timeouts (1001, 1007) get 504 and rank above the other no-answers, so the same failure always gets the same status (the majority pick would otherwise choose 502 or 504 at random).
  • No id restore needed: the router forwards the caller's own request bytes, so the node already echoes the caller's id.

Logs

  • The listeners' request log records what the client received.
  • direct RPC relay failed in goroutine gains error_code, error_name, upstream_status, upstream_body (256-byte cap, redacted).
  • The endpoint poller logs the node's status line and body (debug).
  • docs/RELAY-END-EVENT.md: status, has_reply and no_answer rows updated: an upstream's forwarded error status is a no_answer with that status. No schema bump. RATE-LIMIT-HOLDOFF.md and ERROR-REGISTRY-DESIGN.md §9.2 updated to match.

Files to review first: protocol/chainlib/failed_relay_reply.go, protocol/rpcsmartrouter/rpcsmartrouter_server.go (httpStatusRelayError, hung-upstream 504), protocol/relaycore/client_facing_error.go, protocol/relaycore/relay_processor.go.

Tests

  • New TestFailedRelayReply_EndToEnd_MAG4481: real listener + router + direct sender against httptest upstreams on jsonrpc, rest, tendermintrpc and graphql. Covers node 503 HTML, empty 503, 520, 429, refused → 502, hung → 504, and masked mode.
  • New relay_end golden cases: read_upstream_error_status (503), read_rate_limited (429), read_no_answer_in_time (504). New TestProcessingResult_ATimeoutBeatsARefusedConnection and TestReplyProtocolError_ATimeoutBeatsOtherTransportFailures (both fail without the rank change). New TestProcessingResult_AllRateLimitedWithoutNodeStatusIs429 (fails without the fallback: status 0, so the listener sent 500) and TestProcessingResult_ForwardedNode429BeatsStatuslessRateLimit (the node's own 429 body wins in either order).
  • Existing tests that pinned the old behaviour were updated (all-429 → 503, REST read 502/504 → 500, the HTTP 503 error text) in rate_limit_failure_test.go, stateful_rate_limit_broadcast_test.go, write_outcome_rest_test.go, http_status_relay_error_test.go, client_facing_error_test.go. The tests they replaced are renamed, not dropped.
  • Local run with the fix(protocol)!: return the upstream's own error to the client instead of the router's text (MAG-4218) #551 QA harness (46 JSON-RPC cases, 2 mock nodes, plus 9 REST cases), main vs this branch: every case matched the ticket. Writes where a node may hold the transaction keep main's 500.
  • Local run against the provider simulator (MAG-4676's "run once" items): GraphQL write with every primary down 500 → 503; write with every node 429 503 → 429; GraphQL write with every primary dropping the connection stays 500; a solo drop_connection still says EOF; cross-validation replies (quiet group member, -32601 member) unchanged.

Not in this PR (decided)

  • Masked mode gets no work of its own; no build sets it. What it does now: a node's answer goes back as is (as in every build), and a no-answer is masked, status 500 included.
  • The node's body is not edited in any way, including for keys it may echo.
  • Retry-After is not forwarded (same as main).

Jira ticket

Jira ticket: MAG-4481

🤖 Generated with Claude Code

@avitenzer avitenzer left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Tomelia1999 I reviewed this at 33ddf2ad. go vet ./protocol/... is clean, and the tests in chainlib, relaycore, common, lavasession and endpointstate pass, along with the PR's tests in rpcsmartrouter. The rest of the change looks good to me: the caller's id is already in the node's reply, the ranking order holds, and writes keep their 500 on the hung-upstream path. One regression needs fixing before merge, see the inline comment.

// Every attempt was refused for rate: the chain is temporarily unservable, not broken.
// 503 is the honest status and what well-behaved clients back off on; no Retry-After
// is surfaced — the hold-off registry owns that, internally.
if onlyRateLimited(nodeErrors, protocolErrors) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Regression: when every attempt is rate limited, some transports now get a 500 instead of a 503.

This block was the only thing that set a status for the all-429 case. The new code above only covers 429s that arrive as lavasession.HTTPStatusError. Two other paths still produce rate-limit errors but now get no status:

  • WebSocketDirectRPCConnection, a 429 when the connection is opened (wsHandshakeError unwraps to common.RateLimitedError). This is the relay path for WebSocket-only provider URLs.
  • Direct gRPC, where a confirmed rate limit arrives as GRPCStatusError with rateLimited set.

For these, clientFacingFromProtocolError returns nil, so StatusCode stays 0 and the HTTP listeners send a 500. On main this was a 503.

Repro: in relaycore, send two common.RateLimited(errors.New("websocket: bad handshake"), 0) protocol errors through newTwoProviderProcessor, then call ProcessingResult(). You get OnlyRateLimited=true, StatusCode=0 and ClientFacing=nil.

The tests don't catch this because TestProcessingResult_AllRateLimitedIsTheNodes429 was switched from common.RateLimited(errors.New("HTTP 429")) to HTTPStatusError, so this input is no longer tested.

Suggested fix: keep a fallback for 429s that carry no node status, using the ticket's 429:

if onlyRateLimited(nodeErrors, protocolErrors) && returnedResult.GetClientFacing().GetUpstreamStatus() == 0 {
	stamped := *returnedResult // copy: returnedResult aliases the stored result
	stamped.StatusCode = http.StatusTooManyRequests
	returnedResult = &stamped
}

(Read ClientFacing however you prefer. The point is to set the status only when no node status was forwarded.) Please also add a test with the common.RateLimited(...) input next to the HTTPStatusError one, and update the PR description's line "All attempts rate limited: the node's 429" if the fallback ends up being something other than 429.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, confirmed: your repro gave status 0 on this branch and 503 on main. Fixed in 1ae7df8, which is now rebased on main.

  • When every attempt was rate limited and none of them has a node status to forward (a 429 on the WebSocket handshake, or a gRPC RESOURCE_EXHAUSTED), the reply is now a plain 429.
  • This only runs in that case. If a node answer is forwarded, its status and body are never overwritten.
  • New test TestProcessingResult_AllRateLimitedWithoutNodeStatusIs429 uses your common.RateLimited(...) input. It fails without the fix.
  • New test TestProcessingResult_ForwardedNode429BeatsStatuslessRateLimit covers a node's own 429 next to a 429 with no status. The node's 429 and body win in either arrival order.
  • I kept the PR description's 429 line and added this case to it. I also added a sentence about it to docs/RATE-LIMIT-HOLDOFF.md.

The rebase conflicted with #559 (MAG-4454) in the four HTTP listeners. The failure log lines now use the cut request body (loggedBody / logFormattedMsg), as main does, with this PR's reply. go vet is clean, and relaycore, chainlib, rpcsmartrouter, lavasession, endpointstate, common, metrics and utils all pass.

Tomelia1999 and others added 4 commits November 3, 2026 17:29
…when no node answered (MAG-4481)

The router is a proxy: when a node answered, its status, body and Content-Type go back to the
client untouched. The router speaks only when no node answered.

- JSON-RPC, REST, Tendermint RPC and GraphQL forward the node's answer byte for byte: no router
  envelope, no added jsonrpc field, the GUID stays in the Lava-Guid header. One shared builder,
  chainlib/failed_relay_reply.go (was upstream_error_envelope.go).
- httpStatusRelayError keeps the node's status, body and Content-Type, which is what lets REST
  failures and every 429 forward the answer. Classification is unchanged for 400-599 on all
  three transports.
- All attempts rate limited: the node's 429, not 503.
- No node answered: 502 "no answer from upstream: <cause>", or 504 "no answer from upstream
  within <timeout>" for a timeout, including a hung upstream (was 500 "context deadline
  exceeded"). 1009 joins the transport set, 1043 leaves it, router_detail is gone.
- The unclear-write 500 is unchanged.
- The request log records what the client received; the relay-failure log line gains error_code,
  error_name, upstream_status and upstream_body; the poller logs the node's status and body.
- End-to-end test through a real listener, router and sender on all four interfaces.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ttempt fails (MAG-4481)

Keeping the 429's body made it an upstream answer, the same rank as a 503, so two 429s outvoted
one 503 and the client got the throttle instead of the fault. Before MAG-4481 the 503 won (the
MAG-4638 QA test). A 429 now ranks just below the other upstream answers, and still above router
text and transport failures, so all-429 stays 429 and 429 next to a refused connection stays 429.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…504; pin the new statuses in relay_end (MAG-4481)

A refused connection and a timeout both ranked as "no upstream answered", so the majority pick
broke the one-to-one tie at random: the same failure came back 502 on about 33 runs in 40 and
504 on the rest. A timeout now ranks above the other transport failures, and both still lose to
any other error.

relay_end gains golden cases for the statuses this PR introduced: a node's 503 forwarded, every
attempt throttled (429) and a node that never answers (504).

Notes brought up to date: relay_end's has_reply and no_answer rows (an upstream's error status
is a no_answer with that status), the fully capped chain in RATE-LIMIT-HOLDOFF (the node's 429,
no longer a router 503), the REST 5xx section of ERROR-REGISTRY-DESIGN, and the 504 timeout,
which names the request's budget rather than the time the attempt took.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…o node status (MAG-4481)

A 429 on a WebSocket handshake or a gRPC RESOURCE_EXHAUSTED carries no node status to
forward, so the all-rate-limited reply had no status and the HTTP listeners sent 500 (main
sent 503). Set 429 in that case only; a forwarded node answer keeps its own status and body.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Tomelia1999
Tomelia1999 force-pushed the fix/MAG-4481-node-answer-as-is branch from 0a7f514 to 1ae7df8 Compare November 3, 2026 15:35

@avitenzer avitenzer left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-checked at 1ae7df8f. The regression I raised is fixed.

  • go test ./protocol/relaycore/ passes. With 1ae7df8f's relay_processor.go change reverted, TestProcessingResult_AllRateLimitedWithoutNodeStatusIs429 fails with expected: 429, actual: 0, which is the status-0 case. The new branch only runs when no node answer was forwarded, so a node's own 429 body is never overwritten. TestProcessingResult_ForwardedNode429BeatsStatuslessRateLimit covers both arrival orders.

Not approving yet. The branch no longer merges into main (278a4a0e). #611 (one handler for every HTTP listener route) conflicts in chainlib/graphql.go, jsonRPC.go, rest.go, tendermintRPC.go, and in endpointstate/endpoint_poller.go. Those are the listener reply paths this PR rewrites, so the resolution needs its own review. Please rebase. I'll approve and merge once the rebased head is green and TestFailedRelayReply_EndToEnd_MAG4481 still passes on it.

Resolve the conflicts with #611 (one handler for every HTTP listener route) and
#608 (one direct HTTP sender):

- chainlib: the six listener routes now share relayHTTP. MAG-4481's failed-relay
  reply moves into it: buildFailedRelayReply first (the node's answer as is, or
  the router's 502/504), else the masked errorBody with failedRelayStatus, and
  the request log records what the client is sent. httpRelaySpec gains noAnswer,
  the shape of the router's own no-answer reply: JSON-RPC for jsonrpc and
  Tendermint POST, {"error": ...} for REST, GraphQL and Tendermint GET. The
  jsonrpc errorBody loses the MAG-4218 convertUpstreamErrorToJsonRpc branch, as
  on this branch before the merge.
- lavasession/endpointstate: both producers use main's NewHTTPStatusError, which
  now also fills ContentType from the reply's headers.
- rpcsmartrouter: relayFailureLogAttrs calls common.AsLavaError; #604 folded
  extractLavaError into it.
- listener_handlers.golden.json: regenerated. Only the 8 "error with upstream
  answer" cases change: the node's body and status as is, instead of the
  router's envelope.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

Took 47 minutes

@avitenzer avitenzer left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving at 721c91e0.

I resolved the conflicts with main by merging main into the branch (721c91e0, a fast-forward on top of 1ae7df8f; no history was rewritten):

  • chainlib: #611 folded the six listener routes into one relayHTTP. MAG-4481's failed-relay reply now lives there once. buildFailedRelayReply runs first; otherwise the masked errorBody is sent with failedRelayStatus, and the request log records what the client receives. httpRelaySpec.noAnswer carries each route's no-answer shape, as the branch had it per route: JSON-RPC for jsonrpc and Tendermint POST, {"error": ...} for REST POST/other, GraphQL and Tendermint GET. The jsonrpc errorBody drops the MAG-4218 convertUpstreamErrorToJsonRpc branch, as your branch already did.
  • lavasession / endpointstate: both producers use main's NewHTTPStatusError, which now also fills ContentType from the reply headers. Those headers come from net/http, so the keys are canonical.
  • rpcsmartrouter: this was a semantic conflict that git merged without a marker. relayFailureLogAttrs called extractLavaError, which #604 folded into common.AsLavaError (same body). It now calls common.AsLavaError.
  • listener_handlers.golden.json (new on main) is regenerated. Exactly 8 cases change, all of them error with upstream answer (one per route). The node's body and status now go back as is instead of the router's envelope, and the Content-Type is no longer forced to JSON when the upstream sent none. No other case moved.

Verified on the merged tree:

  • go build ./..., go vet ./protocol/... and gofmt are clean.
  • go test passes for chainlib/..., endpointstate, relaycore, common, lavasession, metrics and rpcsmartrouter.
  • TestFailedRelayReply_EndToEnd_MAG4481 passes 33/33: 4 interfaces × node answer, empty body, 520, 429, refused → 502, hung → 504, masked.
  • CI on this head is green, including lint (dupl guard) and unit-tests-race.

My earlier finding (all-rate-limited with no node status) is fixed by 1ae7df8f, as noted above.

@avitenzer
avitenzer merged commit e44a7d2 into main Nov 4, 2026
@avitenzer
avitenzer deleted the fix/MAG-4481-node-answer-as-is branch November 4, 2026 11:41
AnnaR-prog added a commit that referenced this pull request Nov 4, 2026
…t the configured key, end to end (MAG-4863)

Since #587 a node's 4xx/5xx body goes back as is, built from the
client-facing error this PR redacts. The new test runs a real listener
and router on every HTTP interface and checks the reply is the node's
body with the key withheld, inside its url and on its own.

Also: the comments in relaycore named the envelope and upstream_body
snippet that #587 removed, and the new relaycore test sat between
TestClientFacingFromProtocolError_TransportFailures and its doc comment.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
AnnaR-prog pushed a commit that referenced this pull request Nov 4, 2026
…es and logs (MAG-4863) (#648)

* fix(security): withhold configured upstream secrets from client replies and logs (MAG-4863)

A node's own error reached the caller with the operator's key in it. A
JSON-RPC error inside an HTTP 200 was relayed as the reply with no
redaction at all, and the url-shape redaction that every other exit
uses missed a key after a ";" in a query, a url with escaped slashes,
and any key outside a url.

- Redaction by value: ParseStaticProviderEndpoints registers every
  secret a node-url carries (url password or user, path segments and
  query values that look like a key, auth-query values, auth-header
  values and their tokens). RedactSecrets replaces each one, raw,
  percent-encoded and JSON-escaped, before its url pass, so every log
  line, client error reply, upstream_body and router_detail loses it.
- The url pass now keeps a query's ";" inside the url and redacts urls
  with escaped slashes without breaking the JSON around them.
- relayInnerDirect redacts a node-error reply before it is returned,
  cached or compared; the node's error is still the reply. The status
  >= 400 body is redacted once, where the client-facing error takes it.
- The nine log attributes that printed a configured node-url raw use
  UrlStr(), and a failed provider-config parse no longer logs every
  setting.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix(utils): leave one mark for a registered key inside a url (MAG-4863)

The value pass replaces a registered key first, and the url pass then
stops at the "[" of that mark, so a configured node-url came out as
"https://host/[redacted][redacted]" in every log line and node error
that repeated it. RedactSecrets now folds the repeated marks into one.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix(rpcsmartrouter): log the config key, not the settings, when the listen endpoints fail to parse (MAG-4863)

ParseEndpoints is handed the whole viper config, and its fatal line
logged AllSettings(): every node-url and auth header in the file, before
any of them is registered for redaction. ParseStaticProviderEndpoints
got the same fix in this PR.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* test(rpcsmartrouter): a node's error status reaches the client without the configured key, end to end (MAG-4863)

Since #587 a node's 4xx/5xx body goes back as is, built from the
client-facing error this PR redacts. The new test runs a real listener
and router on every HTTP interface and checks the reply is the node's
body with the key withheld, inside its url and on its own.

Also: the comments in relaycore named the envelope and upstream_body
snippet that #587 removed, and the new relaycore test sat between
TestClientFacingFromProtocolError_TransportFailures and its doc comment.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants