Skip to content

fix(rpcsmartrouter): send the lava-* headers when a request's budget runs out unanswered (MAG-3980) - #671

Merged
nimrod-teich merged 4 commits into
mainfrom
fix/MAG-3980-budget-expiry-headers
Nov 9, 2026
Merged

nimrod-teich merged 4 commits into
mainfrom
fix/MAG-3980-budget-expiry-headers

Conversation

@avitenzer

@avitenzer avitenzer commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

Description

When a request's budget ran out before any attempt came back, the reply (a read's 504, a write's "transaction status unclear" 500) carried no lava-* header, not even Lava-Guid. Both now go through the same header builder as every other failed reply: Lava-Guid, Lava-Provider-Address naming the attempts, and Lava-Retries.

A second commit adds a test for MAG-3778 (already fixed by #440 in v1.6.0, ticket closed): Lava-Retries counts a hedge the first attempt beat. It uses this commit's test harness, so it rides here.

Split out of #666, which carried sixteen tickets in one PR, so each fix can be reviewed on its own. The commits are unchanged.

For reviewers:

Jira ticket

Jira ticket: MAG-3980


Verification

  • Three cases failed with "carries no Lava-Guid" on main, including TestFailedRelayReply_BudgetExpiryKeepsTheRequestHeaders_MAG3980; they pass here. TestJsonRpcListener_FirstAttemptBeatingItsHedgeCountsTheHedge passes on main and fails when the pre-fix(rpcsmartrouter)!: count and name a hedge's losing attempt again (MAG-3762) #440 formula is overlaid.
  • On this branch, based on current main: go build ./... and go vet pass, and the touched test files pass. CI runs the full suite.

After deploy: Two strict xfails on MAG-3980 will XPASS after deploy: scenario 15 in failover/test_the_parallel_attempt_and_who_it_blames.py, and all-hang-terminal in test_hung_and_dropped_connections_are_retried_on_a_peer_and_end_when_every_provider_fails.py.

🤖 Generated with Claude Code

avitenzer and others added 2 commits November 8, 2026 17:49
…runs out unanswered (MAG-3980)

When a request's processing budget ran out before any attempt it dispatched came back, the
reply carried no lava-* header at all, not even Lava-Guid. A read got the router's 504 (a 500
before MAG-4481) and a write the 500 "transaction status unclear", and neither said which
request it was, which upstreams were asked or how many attempts went out. A request that failed
on upstream errors instead, connections reset say, got Lava-Guid, Lava-Provider-Address and
Lava-Retries.

SendParsedRelay builds those two replies in its no-results branch and returned them before
appendHeadersToRelayResult ran. Both now go through it, as every other failed reply does.

appendHeadersToRelayResult named a result with no provider "Cached", the name of a cache hit. A
reply nothing answered has no provider either, but it has no reply, and a cache hit always has
one. Such a reply now names the attempts that went out, in dispatch order and with nothing after
them, even when there was only one. Lava-Retries counts every dispatched attempt (MAG-1818,
MAG-3762), so two quiet upstreams read Lava-Retries: 1 and are both named. The same rule names
the attempts, not "Cached", on the rare failure whose chosen protocol error is the merged
fallback, which carries no provider.

The retry incident metrics (smartrouter_retries_*) are recorded from the same count as the
header (docs/METRICS.md), and the hedge incident metric beside it, so they now count these
requests too, as failed. A cross-validated request that runs out of time with no answer now gets
its cross-validation headers, every participant pending, and is counted as a failed round, which
it skipped before.

Verified: in TestFailedRelayReply_EndToEnd_MAG4481, "hung upstream is the router's 504" now
requires Lava-Guid and the upstream it asked on all four HTTP interfaces, and the new
TestFailedRelayReply_BudgetExpiryKeepsTheRequestHeaders_MAG3980 drives two quiet upstreams past a
hedge and the budget. On the code before this change both fail with "carries no Lava-Guid". The
relay_end case write_unknown_every_upstream_silent pinned the header-less reply
(require.Nil(result)); it now requires the GUID and both upstreams. The relay_end goldens are
unchanged. go test ./protocol/rpcsmartrouter/... passes, and the two timing-dependent tests pass
8 of 8 under -race.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

Took 4 minutes
…attempt beat (MAG-3778)

MAG-3778 reported Lava-Retries 0 on a plain request whose every upstream was slower than the
hedge window: the ticker hedged, the first attempt answered, and the header summed only the
attempts that had come back, so the hedge still running was not counted. The formula the ticket
quotes, successes + node errors + protocol errors - 1, is the one 62bd7ee (MAG-3762, #440)
replaced. Lava-Retries counts every attempt the router sent (MAG-1818), dispatched attempts with
no result included, and docs/METRICS.md says so. That fix first shipped in v1.6.0. The nightly
failures on the ticket, 2026-09-22 to 2026-10-05, all predate it, and the v1.5.x line (v1.5.9,
v1.5.10) does not carry it.

No code change is needed. The scenario was pinned only against a mocked processor
(TestRetryCountHeader_CountsAttemptsWithoutResult, "the first attempt wins while two hedges are
still running"), and the existing end-to-end test has the hedge win. This adds the ticket's own
shape end to end, through the JSON-RPC listener, the state machine's ticker and the dispatcher:
two upstreams slower than the window, the first attempt answers and the hedge is cut off. It
requires Lava-Retries 1, the hedge named before the upstream that answered, and
Lava-Hedge-Triggered.

Verified: it passes, 10 of 10 under -race. With the header counted by the ticket's formula (an
overlay that drops the attempts without a result from the count) it fails with Lava-Retries
absent, the nightly's symptom.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
AnnaR-prog and others added 2 commits November 9, 2026 12:43
… budget runs out unanswered (MAG-3980)

A cross-validated request whose budget ran out before any attempt came back got the router's 504
with its cross-validation headers but no lava-cross-validation-failure-reason. Every other
cross-validation failure names one, and cross_validation_failures_total, the by-reason breakdown
of cross_validation_failed_total, never counted this one.

Nothing came back, so fewer successful responses than the threshold did: insufficient-responses,
the reason the relay processor gives when too few came back. The 504 now carries it.

The test sends a cross-validated request to three quiet upstreams with a threshold of two. Three,
because the 429 cases earlier in the file leave upstream-1 held off, and selection leaves a
held-off upstream out while another is ready. The upstreams' node-url timeout keeps every attempt
out past the request's budget, since a cross-validated attempt is not cut off with the request.
Without the fix it fails with an empty reason.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Resolves failed_relay_reply_e2e_test.go against #679: both PRs add a field to failedReplyClientRead and its literal; both are kept, #679's first.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

@AnnaR-prog AnnaR-prog left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at d4835a7, tested merged with main c19b414 and with main 52a648e (#679 landed). Both findings below are handled: 671-1 in e130834 and 671-2 in the squash message. I merged main into the branch, resolving the conflict with #679 (1631099), and I'm approving at that head.

671-1 (Low): a cross-validated request that runs out of budget unanswered says failed with no reason.

  • Probe: lava-cross-validation-max-participants: 2 and -agreement-threshold: 2 to two quiet upstreams. The new 504 carries Lava-Cross-Validation-Status: failed and both upstreams as pending, but no Lava-Cross-Validation-Failure-Reason.
  • The 504's result has no CrossValidationFailureReason (rpcsmartrouter_server.go:1672). The header and cross_validation_failures_total are written only when it is set (:6412, :6302), while cross_validation_failed_total still counts the request.
  • The README promises the reason on every failure. CROSS-VALIDATION.md says a request nobody answered reads insufficient-responses, and METRICS.md calls failures_total the by-reason breakdown of failed_total. The processor already says insufficient-responses when fewer than the threshold succeeded (relay_processor.go:1436). That includes the same timeout whenever a quiet attempt is recorded before the reply.
  • Fixed in e130834: the 504 of a cross-validated request names insufficient-responses. New test TestFailedRelayReply_BudgetExpiryOfACrossValidatedRequestNamesItsReason_MAG3980, which fails without the fix ("expected insufficient-responses, actual empty").

671-2 (Nit): the body names the retry and hedge incident metrics; two more counters now count these requests.

  • smartrouter_consistency_failed_total (requests with a seen block, :6616) and smartrouter_extension_unavailable_total (with its header) also count a budget expiry now. Both are right per METRICS.md.
  • Handled: the squash message says so.

Checked and correct:

  • A wire probe through each listener, with two quiet upstreams, on main and on this PR. The 504 now carries Lava-Guid (the same guid as in the body), Lava-Provider-Address naming both attempts, Lava-Retries: 1 and Lava-Hedge-Triggered on JSON-RPC, REST, Tendermint RPC and GraphQL. gRPC gets them as trailers (chainlib/grpc.go:418), with the guid of the error message. Main sends none of them, unless a quiet attempt happened to be recorded before the reply.
  • The unclear write keeps its 500 and its body, and now carries Lava-Guid, both upstreams and the stateful headers. WebSocket still sends its error frame, because it does not read the result on an error. Nothing else reads the new results differently: not relay_end, the readiness signal, or the listeners' error bodies.
  • Mutants: with main's rpcsmartrouter_server.go, the hung-upstream subtests fail (each interface 7–8 of 8), and so do the new budget test and relay_end's write case. TestJsonRpcListener_FirstAttemptBeatingItsHedgeCountsTheHedge passes on main and fails with the pre-#440 count (Lava-Retries absent).
  • Conflict with #679 (now on main): two struct fields added on each side in failed_relay_reply_e2e_test.go. I kept both fields; all tests of both PRs pass on the result, also under -race x10.
  • Not covered, and an owner question on MAG-3980 (comment of 2026-11-08): a request refused before dispatch, such as a pin on a provider in no pool, still gets no headers.

Tested: go build ./..., go vet; the e2e file, TestRelayEnd_ExitPaths and the MAG-3778 test, 30x for the timing tests and -race x10; the mutants above; full protocol/rpcsmartrouter package; golangci-lint --new-from-rev (0 issues).

@nimrod-teich
nimrod-teich merged commit 6afab23 into main Nov 9, 2026
@nimrod-teich
nimrod-teich deleted the fix/MAG-3980-budget-expiry-headers branch November 9, 2026 11:24
avitenzer added a commit that referenced this pull request Nov 9, 2026
Resolves the conflict with #671 (MAG-3980): the budget-ran-out unclear-write return keeps
#671's headed result and returns the write's reason error instead of errUnknownWriteOutcome.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants