Repository navigation
fix(rpcsmartrouter): send the lava-* headers when a request's budget runs out unanswered (MAG-3980) - #671
Conversation
…runs out unanswered (MAG-3980) When a request's processing budget ran out before any attempt it dispatched came back, the reply carried no lava-* header at all, not even Lava-Guid. A read got the router's 504 (a 500 before MAG-4481) and a write the 500 "transaction status unclear", and neither said which request it was, which upstreams were asked or how many attempts went out. A request that failed on upstream errors instead, connections reset say, got Lava-Guid, Lava-Provider-Address and Lava-Retries. SendParsedRelay builds those two replies in its no-results branch and returned them before appendHeadersToRelayResult ran. Both now go through it, as every other failed reply does. appendHeadersToRelayResult named a result with no provider "Cached", the name of a cache hit. A reply nothing answered has no provider either, but it has no reply, and a cache hit always has one. Such a reply now names the attempts that went out, in dispatch order and with nothing after them, even when there was only one. Lava-Retries counts every dispatched attempt (MAG-1818, MAG-3762), so two quiet upstreams read Lava-Retries: 1 and are both named. The same rule names the attempts, not "Cached", on the rare failure whose chosen protocol error is the merged fallback, which carries no provider. The retry incident metrics (smartrouter_retries_*) are recorded from the same count as the header (docs/METRICS.md), and the hedge incident metric beside it, so they now count these requests too, as failed. A cross-validated request that runs out of time with no answer now gets its cross-validation headers, every participant pending, and is counted as a failed round, which it skipped before. Verified: in TestFailedRelayReply_EndToEnd_MAG4481, "hung upstream is the router's 504" now requires Lava-Guid and the upstream it asked on all four HTTP interfaces, and the new TestFailedRelayReply_BudgetExpiryKeepsTheRequestHeaders_MAG3980 drives two quiet upstreams past a hedge and the budget. On the code before this change both fail with "carries no Lava-Guid". The relay_end case write_unknown_every_upstream_silent pinned the header-less reply (require.Nil(result)); it now requires the GUID and both upstreams. The relay_end goldens are unchanged. go test ./protocol/rpcsmartrouter/... passes, and the two timing-dependent tests pass 8 of 8 under -race. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Took 4 minutes
…attempt beat (MAG-3778) MAG-3778 reported Lava-Retries 0 on a plain request whose every upstream was slower than the hedge window: the ticker hedged, the first attempt answered, and the header summed only the attempts that had come back, so the hedge still running was not counted. The formula the ticket quotes, successes + node errors + protocol errors - 1, is the one 62bd7ee (MAG-3762, #440) replaced. Lava-Retries counts every attempt the router sent (MAG-1818), dispatched attempts with no result included, and docs/METRICS.md says so. That fix first shipped in v1.6.0. The nightly failures on the ticket, 2026-09-22 to 2026-10-05, all predate it, and the v1.5.x line (v1.5.9, v1.5.10) does not carry it. No code change is needed. The scenario was pinned only against a mocked processor (TestRetryCountHeader_CountsAttemptsWithoutResult, "the first attempt wins while two hedges are still running"), and the existing end-to-end test has the hedge win. This adds the ticket's own shape end to end, through the JSON-RPC listener, the state machine's ticker and the dispatcher: two upstreams slower than the window, the first attempt answers and the hedge is cut off. It requires Lava-Retries 1, the hedge named before the upstream that answered, and Lava-Hedge-Triggered. Verified: it passes, 10 of 10 under -race. With the header counted by the ticket's formula (an overlay that drops the attempts without a result from the count) it fails with Lava-Retries absent, the nightly's symptom. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… budget runs out unanswered (MAG-3980) A cross-validated request whose budget ran out before any attempt came back got the router's 504 with its cross-validation headers but no lava-cross-validation-failure-reason. Every other cross-validation failure names one, and cross_validation_failures_total, the by-reason breakdown of cross_validation_failed_total, never counted this one. Nothing came back, so fewer successful responses than the threshold did: insufficient-responses, the reason the relay processor gives when too few came back. The 504 now carries it. The test sends a cross-validated request to three quiet upstreams with a threshold of two. Three, because the 429 cases earlier in the file leave upstream-1 held off, and selection leaves a held-off upstream out while another is ready. The upstreams' node-url timeout keeps every attempt out past the request's budget, since a cross-validated attempt is not cut off with the request. Without the fix it fails with an empty reason. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
AnnaR-prog
left a comment
There was a problem hiding this comment.
Reviewed at d4835a7, tested merged with main c19b414 and with main 52a648e (#679 landed). Both findings below are handled: 671-1 in e130834 and 671-2 in the squash message. I merged main into the branch, resolving the conflict with #679 (1631099), and I'm approving at that head.
671-1 (Low): a cross-validated request that runs out of budget unanswered says failed with no reason.
- Probe:
lava-cross-validation-max-participants: 2and-agreement-threshold: 2to two quiet upstreams. The new 504 carriesLava-Cross-Validation-Status: failedand both upstreams as pending, but noLava-Cross-Validation-Failure-Reason. - The 504's result has no
CrossValidationFailureReason(rpcsmartrouter_server.go:1672). The header andcross_validation_failures_totalare written only when it is set (:6412,:6302), whilecross_validation_failed_totalstill counts the request. - The README promises the reason on every failure. CROSS-VALIDATION.md says a request nobody answered reads
insufficient-responses, and METRICS.md callsfailures_totalthe by-reason breakdown offailed_total. The processor already saysinsufficient-responseswhen fewer than the threshold succeeded (relay_processor.go:1436). That includes the same timeout whenever a quiet attempt is recorded before the reply. - Fixed in e130834: the 504 of a cross-validated request names
insufficient-responses. New testTestFailedRelayReply_BudgetExpiryOfACrossValidatedRequestNamesItsReason_MAG3980, which fails without the fix ("expected insufficient-responses, actual empty").
671-2 (Nit): the body names the retry and hedge incident metrics; two more counters now count these requests.
smartrouter_consistency_failed_total(requests with a seen block,:6616) andsmartrouter_extension_unavailable_total(with its header) also count a budget expiry now. Both are right per METRICS.md.- Handled: the squash message says so.
Checked and correct:
- A wire probe through each listener, with two quiet upstreams, on main and on this PR. The 504 now carries
Lava-Guid(the same guid as in the body),Lava-Provider-Addressnaming both attempts,Lava-Retries: 1andLava-Hedge-Triggeredon JSON-RPC, REST, Tendermint RPC and GraphQL. gRPC gets them as trailers (chainlib/grpc.go:418), with the guid of the error message. Main sends none of them, unless a quiet attempt happened to be recorded before the reply. - The unclear write keeps its 500 and its body, and now carries
Lava-Guid, both upstreams and the stateful headers. WebSocket still sends its error frame, because it does not read the result on an error. Nothing else reads the new results differently: not relay_end, the readiness signal, or the listeners' error bodies. - Mutants: with main's
rpcsmartrouter_server.go, the hung-upstream subtests fail (each interface 7–8 of 8), and so do the new budget test and relay_end's write case.TestJsonRpcListener_FirstAttemptBeatingItsHedgeCountsTheHedgepasses on main and fails with the pre-#440 count (Lava-Retries absent). - Conflict with #679 (now on main): two struct fields added on each side in
failed_relay_reply_e2e_test.go. I kept both fields; all tests of both PRs pass on the result, also under-racex10. - Not covered, and an owner question on MAG-3980 (comment of 2026-11-08): a request refused before dispatch, such as a pin on a provider in no pool, still gets no headers.
Tested: go build ./..., go vet; the e2e file, TestRelayEnd_ExitPaths and the MAG-3778 test, 30x for the timing tests and -race x10; the mutants above; full protocol/rpcsmartrouter package; golangci-lint --new-from-rev (0 issues).
Description
When a request's budget ran out before any attempt came back, the reply (a read's 504, a write's "transaction status unclear" 500) carried no
lava-*header, not evenLava-Guid. Both now go through the same header builder as every other failed reply:Lava-Guid,Lava-Provider-Addressnaming the attempts, andLava-Retries.A second commit adds a test for MAG-3778 (already fixed by #440 in v1.6.0, ticket closed):
Lava-Retriescounts a hedge the first attempt beat. It uses this commit's test harness, so it rides here.Split out of #666, which carried sixteen tickets in one PR, so each fix can be reviewed on its own. The commits are unchanged.
For reviewers:
failed_relay_reply_e2e_test.go; whichever lands second keeps both fields.Jira ticket
Jira ticket: MAG-3980
Verification
main, includingTestFailedRelayReply_BudgetExpiryKeepsTheRequestHeaders_MAG3980; they pass here.TestJsonRpcListener_FirstAttemptBeatingItsHedgeCountsTheHedgepasses onmainand fails when the pre-fix(rpcsmartrouter)!: count and name a hedge's losing attempt again (MAG-3762) #440 formula is overlaid.main:go build ./...andgo vetpass, and the touched test files pass. CI runs the full suite.After deploy: Two strict xfails on MAG-3980 will XPASS after deploy: scenario 15 in
failover/test_the_parallel_attempt_and_who_it_blames.py, andall-hang-terminalintest_hung_and_dropped_connections_are_retried_on_a_peer_and_end_when_every_provider_fails.py.🤖 Generated with Claude Code