Repository navigation
fix(rpcsmartrouter)!: count and name a hedge's losing attempt again (MAG-3762) - #440
Conversation
4fabcdc to
1fd49aa
Compare
Review: manual pass (high effort) on
|
Second pass on
|
|
Thanks for both passes. All four blocking findings are addressed at d69a7d3. The branch now has current 1. Dispatch order. Fixed in bbc1915. One correction to the premise, for the record. Your example (p1 node error, p2 hedged over, p3 wins) also read It also reorders a retry with no hedge whose failures are of different kinds.
2. The slot test that could not fail. Fixed in bbc1915. It is replaced by four cases:
Restoring the fixed slot fails exactly the first three. 3. Pre-existing header tests blind to the new term. Fixed in 7c81585. Every literal that reaches the retry count now carries the dispatch history its scenario implies. The 12 cross-validation literals return before the history is read, so they are untouched.
Measured with
4. Decision items.
Verification at d69a7d3, from a clean checkout:
|
Re-review on
|
| # | Finding | Status |
|---|---|---|
| Pass 2 #1 | Blocking — Lava-Provider-Address can now be out of dispatch order |
fixed |
| Pass 2 #2 | Blocking — the test that covers the slot cannot fail | fixed |
| Pass 2 #3 | Blocking — every pre-existing header test is blind to the new term | fixed |
| Pass 2 #4 | Blocking — SessionsDispatched() has a client-visible second consumer the body never mentions |
fixed |
| Pass 1 #1 | Low (coverage) — CrossValidation is absent from the new table |
still open |
#1 is fixed at the root rather than patched. The list no longer walks results by kind (protocol errors → still-out → node errors → successes); it walks the request's dispatch history, one entry per attempt in batch order. That removes the whole class, not just the arrangement I happened to find — the commit message names both instances it fixes (a retry named before the attempt it followed, and an attempt still out named after a protocol error from a later hedge).
#2 is fixed with a case that can actually go red. My finding was that the table's only unfinished-attempt case put the fixed slot exactly where dispatch order would also put it, so the expectation passed either way. The table now has:
name: "the first attempt wins while two hedges are still running",
usedProviders: dispatchedTo("p1", "p2", "p3"),
successResults: []common.RelayResult{from("p1")},
resolver: "p1",
wantProviders: "p2,p3,p1",p2,p3,p1 is not reachable by any by-kind walk, and it pins the MAG-1871 resolver-last rule on top of dispatch order — so it discriminates in both dimensions at once. The new dispatchedTogether helper is also the right addition: it distinguishes one fan-out batch from three sequential ones, which the old dispatchedTo-only harness could not express.
#3 is fixed across the suite, not just where it was noticed. Every case that reaches the retry count now carries the history its scenario implies, and the commit is explicit about why that mattered: the pre-existing tests recorded results over an empty history, so they passed both with the new term removed and with it counting every dispatched attempt. That is the definition of a term no test constrains.
#4 is the one I expected to be pushed back on, and it was taken seriously. writeOutcomeIsUnknown compares the endpoints that answered against SessionsDispatched(), so on the degraded path a consistency-rejected session left in the history made a write the filter had kept from every node read as one sent to nodes that never answered — surfacing to the client as a 500 "transaction status unclear" once the budget ran out. degraded_session_release_test.go now runs a write and asserts the verdict directly. That is a client-visible outcome, and it now has a test.
Still open: CrossValidation has no case in the table
My pass-1 finding stands unchanged — there is no selection: relaycore.CrossValidation case in hedge_loser_retry_count_test.go, and CV is the selection whose header a reader is most likely to misread, because a CV round legitimately puts several providers in one batch and so looks like retries that never happened.
It is cheaper to close now than it was before, because dispatchedTogether already exists for exactly this shape — the stateful fan-out case is one line away from a CV twin:
{
name: "a cross-validation round is one batch, not N retries",
rp: &MockRelayProcessorForHeaders{
selection: relaycore.CrossValidation,
usedProviders: dispatchedTogether("p1", "p2", "p3"),
successResults: []common.RelayResult{from("p1")},
},
resolver: "p1",
wantProviders: …, // whichever the quorum path should name
},Low, and not a merge blocker — but it is the last selection with no coverage in a table that now covers the other three well.
One stale paragraph, shared across this stack
The MAG-3859 / "does not compile on main" caveat is no longer true — origin/main renamed the duplicate helper apart, and the runs above are on this head with nothing uncommitted.
Still true from the first pass
The merge-order note stands: protocol/lavasession/used_providers.go is also touched by #418 (the maps.Clone fix on GetErroredProviders), and rpcsmartrouter_server_test.go by #423 and #450. All mechanical.
🤖 Generated with Claude Code
…tries (MAG-3762) The retry-count table covered every selection but cross-validation, the one a reader is most likely to misread. A round sends several providers in one batch and early-exits once the quorum is met, so its unanswered participants look like attempts without a result. They are not retries. The cross-validation branch of appendHeadersToRelayResult names the round in its own headers and returns before the retry count is built, so the reply carries neither Lava-Retries nor Lava-Provider-Address. That is what the PR describes and what the existing cross-validation header tests assume. The new case sends three providers in one batch, records two answers and leaves the third out. It asserts that both headers are absent, that all-providers names the whole batch, and that pending names the third. A provider header the table expects as "" must now be absent, the same convention the retry header already uses. Addresses the remaining review finding on #440. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Review fix pushed:
|
AnnaR-prog
left a comment
There was a problem hiding this comment.
Approving: every finding from my review is fixed (see my fix comment above).
… only the count (MAG-3762) UsedProviders counted the sessions a request dispatched across every batch, and writeOutcomeIsUnknown compares that count with the results that came back. The reply's attempt headers need more than the count. They have to name an attempt that was sent and has not reported back, such as a hedge's losing attempt, and nothing else records one. The in-flight set drops a provider when its session is freed, which is before the provider's result is posted. The count becomes a list of provider names, one entry per session, in the order the batches went out, and SessionsDispatched is its length. So the two cannot drift, and the count behaves as before. ReleaseFromLatestBatch removes the provider's latest entry, which is the one its batch added. DispatchedProviders returns a copy. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…MAG-3762) Since v1.5.2 a hedge's losing attempt is missing from Lava-Retries. A request pinned to a slow provider with lava-relay-timeout: 300ms is answered by a peer and carries Lava-Hedge-Triggered: true, but no Lava-Retries header at all, and Lava-Provider-Address names only the peer. The header sums the results recorded when the reply is built. Since 9730cf5 an attempt outlives the window that dispatches the hedge, so when the hedge wins, the first attempt is still running. The router cancels it once the relay loop ends, and the attempt posts its result to a channel that nothing reads again, so it is in none of the lists. Before 9730cf5 the window killed that attempt first, and it was recorded as a timeout. The header now counts every attempt the router sent, as MAG-1818 ruled. Each dispatched attempt with no recorded result is added to Lava-Retries and named in Lava-Provider-Address, with the protocol errors, where v1.5.1 listed the attempt a hedge overtook. The provider that answered stays last. A provider asked twice is still named once, as before, so on that path the names stay one short of the count. The protocol-error count now comes from the same snapshot as the lists the subtraction reads. On a failure path a reader can record a result between the caller's read and the header's, and that attempt used to be counted zero times. A sequential retry, a stateful fan-out and cross-validation are unchanged. A request that ends with no recorded result at all, because every attempt hung, still returns before any header is built. That is MAG-3764. The retry incident metrics (smartrouter_retries_total, _success_total, _failed_total and smartrouter_retry_attempts) are recorded from the same count, so they count hedged requests again as well. docs/METRICS.md now says the retry and hedging families overlap. BREAKING CHANGE: for a request where a hedge fires, Lava-Retries and Lava-Provider-Address change relative to v1.5.2 and v1.5.3. The reply again reports the hedge's losing attempt: Lava-Retries is at least 1, and the losing provider is named before the one that answered. This is the behaviour of v1.5.1 and earlier, and what MAG-1818 specifies. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…taken with (MAG-3762) sendRelayToDirectEndpoints released every session the consistency filter rejected, and every session a cross-validation guard gave back, under a key built from the request's extensions. UsedProviders files a provider under the key its session was taken with. The two differ when a request degrades to a regular provider: its sessions are taken under the plain key, so the release found nothing and did nothing. The provider stayed in the in-flight set until the session was freed, and stayed in the dispatch history for good, and sessionsLatestBatch kept counting it. The previous commit made that history visible: Lava-Retries now counts every dispatched attempt with no result, so a rejected provider on that path would show as a retry at a node that never received the request. Each release now names the session's own key, through a new SingleConsumerSession.RouterKey. Off the degraded path the two keys are the same, so nothing changes there. The new test takes two sessions under the plain key for an archive request and has the filter reject both. With the old key both providers stay in the history. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…mpts went out (MAG-3762) The list walked the results by kind: protocol errors, then the attempts still out, then node errors, then successes. That is the order the providers were asked only when every failure was of one kind. A request whose first attempt came back with a node error, and whose retry was then overtaken by a hedge, read p2,p1,p3: the retry named before the attempt it followed. An attempt still out was likewise named after a protocol error from a hedge sent later. The list now walks the request's dispatch history, one entry per attempt in the order the batches went out. Only a stateful fan-out or a cross-validation round puts several sessions in one batch, and neither reaches this list, so it follows the order the providers were asked whatever became of each attempt. The results by kind follow, for a provider the history does not hold; a live request records every provider it asks, so they add nothing there. The provider that answered stays last (MAG-1871). This also reorders a retry with no hedge in it whose failures were of different kinds, which used to name every protocol error before every node error. The automation already reads the list as the order the providers were asked: a pinned provider must come first, and the primaries before the backup that answered. The header-table case for an attempt still out could not fail, because dispatch order and the fixed slot agreed on it. It is replaced by three cases that tell them apart, one of them a retry with no hedge, and by one that pins the fallback for a result the history does not hold. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Took 5 minutes
…al request has (MAG-3762) The header counts and names the attempts in the request's dispatch history, and a real request's history holds the provider of every result it recorded. The header tests written before MAG-3762 recorded results over an empty history, so the new term had nothing to count in any of them. They all passed with it removed, and all passed with it counting every dispatched attempt on top of the results. Every case that reaches the retry count now carries the history its scenario implies. Three TestRetryCountHeader cases passed a protocol-error count over an empty error list, which a live request cannot produce because the count is read from that list, so they now record the error. The stateful-API cases record one batch under Stateful selection, as the router sends a fan-out; three of them ran as Stateless requests. The hedge case now also checks that the reply reports the hedge as a retry, which is the reply MAG-3723 caught without it. The mock panics when a test records results but sets no history, so a new test cannot reopen the gap. With every dispatched attempt counted on top of the results, 13 cases that predate MAG-3762 now fail, where none did. With the new term removed, the hedge case fails. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…eceived it (MAG-3762) writeOutcomeIsUnknown compares the endpoints that answered against SessionsDispatched, the length of the same dispatch history. On the degraded path a consistency-rejected session used to stay in that history, so a write the filter kept from every node read as one sent to nodes that never answered. Once the budget ran out it came back as "transaction status unclear", a 500 telling the client the transaction may already be on chain. Releasing under the session's own key fixed that as well, but nothing checked it. The degraded test now runs a write too and asserts the verdict: no node received it, so it is a failure, not an unclear write. The comment at the release names both readers of the history. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…tries (MAG-3762) The retry-count table covered every selection but cross-validation, the one a reader is most likely to misread. A round sends several providers in one batch and early-exits once the quorum is met, so its unanswered participants look like attempts without a result. They are not retries. The cross-validation branch of appendHeadersToRelayResult names the round in its own headers and returns before the retry count is built, so the reply carries neither Lava-Retries nor Lava-Provider-Address. That is what the PR describes and what the existing cross-validation header tests assume. The new case sends three providers in one batch, records two answers and leaves the third out. It asserts that both headers are absent, that all-providers names the whole batch, and that pending names the third. A provider header the table expects as "" must now be absent, the same convention the retry header already uses. Addresses the remaining review finding on #440. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
893b07b to
30a4936
Compare
…gainst MAG-2734 (MAG-4156) (#485) * test(rpcsmartrouter): repair the test build MAG-3746 and #440 broke against MAG-2734 The rpcsmartrouter test binary has not compiled on main since MAG-3746 landed (1e08bca, 13:58). CI never runs go test, so three merges stacked on it unnoticed. main compiled at f10367e, the commit before MAG-3746. Collisions: - health_check_crafted_relay_test.go (MAG-3746, 1e08bca) was written against the API before MAG-2734 (c7c605a, merged 12:06 the same day), which removed RelayRetriesManager and the relayRetriesManager server field, dropped the retries-manager argument from relaycore.NewRelayProcessor, and dropped ctx from GetEmptyRelayState. Then MAG-3923 (2d18323) added the backupReserve parameter to sendRelayToEndpoint, which hit the same call. - degraded_session_release_test.go (#440 / MAG-3762, 9178c5b and e9b9070) passes lavaprotocol.NewRelayRetriesManager() to NewRelayProcessor. - hedge_loser_retry_count_test.go (#440 / MAG-3762, 62bd7ee) sets the removed relayRetriesManager field on the server literal. Fixes, test code only: drop the retries manager from each constructor and server literal, call GetEmptyRelayState(message), and pass backupReserve=false to sendRelayToEndpoint, as the production health path (sendRelayToEndpoint call in the internal relay loop) does. None of these tests asserted anything about the retries manager; it only fed the archive-upgrade mutation MAG-2734 removed, so each test's intent is unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Took 1 minute * test(rpcsmartrouter): pass backupReserve in the threshold-guard test MAG-3286 merged after this fix (MAG-4156) #423 (MAG-3286, d8f2d2b) merged after this branch was cut and added cross_validation_threshold_guard_test.go, which calls sendRelayToEndpoint with the six arguments it took before MAG-3923 added backupReserve. Same collision as the three files fixed here already; pass false, as the production health path and the other repaired call do. origin/main is merged in so the branch compiles against the main it will land on: go test -run '^$' ./... builds every package's tests. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Took 3 minutes --------- Co-authored-by: AnnaR-prog <anna@magmadevs.com>
…runs out unanswered (MAG-3980) (#671) * fix(rpcsmartrouter): send the lava-* headers when a request's budget runs out unanswered (MAG-3980) When a request's processing budget ran out before any attempt it dispatched came back, the reply carried no lava-* header at all, not even Lava-Guid. A read got the router's 504 (a 500 before MAG-4481) and a write the 500 "transaction status unclear", and neither said which request it was, which upstreams were asked or how many attempts went out. A request that failed on upstream errors instead, connections reset say, got Lava-Guid, Lava-Provider-Address and Lava-Retries. SendParsedRelay builds those two replies in its no-results branch and returned them before appendHeadersToRelayResult ran. Both now go through it, as every other failed reply does. appendHeadersToRelayResult named a result with no provider "Cached", the name of a cache hit. A reply nothing answered has no provider either, but it has no reply, and a cache hit always has one. Such a reply now names the attempts that went out, in dispatch order and with nothing after them, even when there was only one. Lava-Retries counts every dispatched attempt (MAG-1818, MAG-3762), so two quiet upstreams read Lava-Retries: 1 and are both named. The same rule names the attempts, not "Cached", on the rare failure whose chosen protocol error is the merged fallback, which carries no provider. The retry incident metrics (smartrouter_retries_*) are recorded from the same count as the header (docs/METRICS.md), and the hedge incident metric beside it, so they now count these requests too, as failed. A cross-validated request that runs out of time with no answer now gets its cross-validation headers, every participant pending, and is counted as a failed round, which it skipped before. Verified: in TestFailedRelayReply_EndToEnd_MAG4481, "hung upstream is the router's 504" now requires Lava-Guid and the upstream it asked on all four HTTP interfaces, and the new TestFailedRelayReply_BudgetExpiryKeepsTheRequestHeaders_MAG3980 drives two quiet upstreams past a hedge and the budget. On the code before this change both fail with "carries no Lava-Guid". The relay_end case write_unknown_every_upstream_silent pinned the header-less reply (require.Nil(result)); it now requires the GUID and both upstreams. The relay_end goldens are unchanged. go test ./protocol/rpcsmartrouter/... passes, and the two timing-dependent tests pass 8 of 8 under -race. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Took 4 minutes * test(rpcsmartrouter): pin that Lava-Retries counts a hedge the first attempt beat (MAG-3778) MAG-3778 reported Lava-Retries 0 on a plain request whose every upstream was slower than the hedge window: the ticker hedged, the first attempt answered, and the header summed only the attempts that had come back, so the hedge still running was not counted. The formula the ticket quotes, successes + node errors + protocol errors - 1, is the one 62bd7ee (MAG-3762, #440) replaced. Lava-Retries counts every attempt the router sent (MAG-1818), dispatched attempts with no result included, and docs/METRICS.md says so. That fix first shipped in v1.6.0. The nightly failures on the ticket, 2026-09-22 to 2026-10-05, all predate it, and the v1.5.x line (v1.5.9, v1.5.10) does not carry it. No code change is needed. The scenario was pinned only against a mocked processor (TestRetryCountHeader_CountsAttemptsWithoutResult, "the first attempt wins while two hedges are still running"), and the existing end-to-end test has the hedge win. This adds the ticket's own shape end to end, through the JSON-RPC listener, the state machine's ticker and the dispatcher: two upstreams slower than the window, the first attempt answers and the hedge is cut off. It requires Lava-Retries 1, the hedge named before the upstream that answered, and Lava-Hedge-Triggered. Verified: it passes, 10 of 10 under -race. With the header counted by the ticket's formula (an overlay that drops the attempts without a result from the count) it fails with Lava-Retries absent, the nightly's symptom. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix(rpcsmartrouter): name the reason when a cross-validated request's budget runs out unanswered (MAG-3980) A cross-validated request whose budget ran out before any attempt came back got the router's 504 with its cross-validation headers but no lava-cross-validation-failure-reason. Every other cross-validation failure names one, and cross_validation_failures_total, the by-reason breakdown of cross_validation_failed_total, never counted this one. Nothing came back, so fewer successful responses than the threshold did: insufficient-responses, the reason the relay processor gives when too few came back. The 504 now carries it. The test sends a cross-validated request to three quiet upstreams with a threshold of two. Three, because the 429 cases earlier in the file leave upstream-1 held off, and selection leaves a held-off upstream out while another is ready. The upstreams' node-url timeout keeps every attempt out past the request's budget, since a cross-validated attempt is not cut off with the request. Without the fix it fails with an empty reason. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: Anna Rayer <anna@magmadevs.com>
Description
Bug (MAG-3762). Since v1.5.2 a hedge's losing attempt is missing from
Lava-Retries. A request pinned to a slow provider withlava-relay-timeout: 300msis answered by a peer. The reply carriesLava-Hedge-Triggered: truebut noLava-Retriesheader, andLava-Provider-Addressnames only the peer. MAG-3723 is the nightly ticket for the same failure (test_relay_timeout_boundary_slow_provider_retries_within_budget, red on every nightly since 2026-09-16), and MAG-3693 was closed onto it.Cause.
appendHeadersToRelayResultsums the results recorded when the reply is built. Since 9730cf5 (a correct fix for two production incidents), an attempt outlives the window that dispatches the hedge, so the first attempt is still running when the hedge wins. The router cancels it once the relay loop ends, and the attempt posts its result to a channel that nothing reads again, so it is in none of the three lists. Before 9730cf5 the window killed that attempt first, and it was recorded as a timeout. The sum itself has not changed since the initial implementation (22cbf32).Decision. The ticket leaves the choice between its two options to the router team. This PR takes option 1 and restores the count, for three reasons:
The ticket records the choice. Say so if you would rather take option 2.
Breaking, on purpose. Relative to v1.5.2 and v1.5.3, a hedged reply changes.
Lava-Retriesis at least 1 again, andLava-Provider-Addressnames the losing provider before the one that answered, as v1.5.1 did.Lava-Provider-Addressnow also lists the providers of every retried request in the order the router asked them. It used to name every protocol error before every node error, and v1.5.1 did the same. README's MAJOR rule covers HTTP metadata headers and namesLava-Provider-Address, so the fix commit isfix!with aBREAKING CHANGE:footer, andscripts/changelog-bump.shwill list it. See Merge note below for the footer text to keep. 9730cf5 was not marked, and that is why the v1.5.2 and v1.5.3 release notes never mentioned the change (Victoria's 2026-09-24 comment).Commits.
refactor(lavasession):UsedProviderskept a count of the sessions a request dispatched (SessionsDispatched, used bywriteOutcomeIsUnknown). It now keeps their names, in the order the batches went out, and the count is the list's length.DispatchedProviders()returns a copy. The header needs the names, not just the count. The in-flight set can't supply them, because it drops a provider when its session is freed, and that happens before the provider's result is posted.fix(rpcsmartrouter)!:attemptsWithoutResultreturns the dispatched attempts with no recorded result, subtracting one attempt per result. The header adds them toLava-Retriesand names them inLava-Provider-Address. The provider that answered stays last (MAG-1871). The protocol-error count now comes from the same snapshot as those lists. Before, a result recorded by a still-draining reader between the two reads was counted zero times.docs/METRICS.mdnow says the retry and hedging series overlap.fix(rpcsmartrouter): rejected sessions were released under a key built from the request's extensions. A request that degrades to a regular provider takes its sessions under the plain key, so the release did nothing, and the provider stayed in the dispatch history. Commit 2 would have turned that into a retry naming a node that never got the request. Each release now uses the session's own key (SingleConsumerSession.RouterKey()). Off the degraded path the two keys are the same.Merge branch 'main': brings in currentmain, including test(rpcsmartrouter): fix the duplicate solanaGetBlockReply helper #433, so therpcsmartroutertest package compiles on this branch. It is a merge rather than a rebase, so the three reviewed commits keep their SHAs.fix(rpcsmartrouter):Lava-Provider-Addresswalks the dispatch history, then the results by kind for any provider the history does not hold. It used to walk the results by kind with the attempts still out in a fixed slot. A batch that reaches this list holds one session: only a stateful fan-out or a cross-validation round sends several at once, and neither builds this list. So the list follows the order the providers were asked, whatever became of each attempt (review finding 1).test(rpcsmartrouter): every pre-existing header test that reaches the retry count now carries the dispatch history its scenario implies, and the test mock panics if a test records results without one (review finding 3).test(rpcsmartrouter): the degraded-release test also runs a write and assertswriteOutcomeIsUnknown(review finding 4).Behaviour to know about.
Lava-Provider-Addressfollows the order the router asked the providers, with the one that answered last. A sequential retry whose failures were of different kinds reorders too. For example, a node error from p1 and then a 503 from p2 used to readp2,p1,p3onmain, and now readsp1,p2,p3. The automation already reads the list this way: a pinned provider must come first, and the primaries before the backup that answered.Lava-Provider-Address, as before. On that path the names stay one short ofLava-Retries + 1. Otherwise every counted attempt is also named.Lava-Hedge-Triggeredis untouched.smartrouter_retries_total,_success_total,_failed_total,smartrouter_retry_attempts) are recorded from the same count, so they count hedged requests again. Expect them to step up for hedged traffic after rollout. That is a correction, not new retry pressure.Not fixed here: every attempt hangs (MAG-3764). When no attempt reports back at all,
SendParsedRelayreturns before any header is built. The 500 then carries noLava-Retries, noLava-Guidand noLava-Provider-Address. That istest_transport_failure_all_providers_terminate_bounded[all-hang-terminal]. Victoria's MAG-3764 comment traces it to the same 9730cf5 change, but the fix is a result synthesized on the no-results error path, which is its own change.Tests
TestAttemptsWithoutResultcovers the subtraction: an attempt still out, two still out, node errors counting as results, a provider asked twice, and a cache hit (never dispatched, so it settles no attempt).TestRetryCountHeader_CountsAttemptsWithoutResultchecks both headers for:Lava-Retries: 2);p2,p1,p3;TestRetryCountHeader,TestRetryCountHeaderStatefulFanoutAbsorption,TestHedgeTriggeredHeader,TestCacheServedResponseHeaders,TestResolverAlwaysLastInProviderHeader,TestStatefulRelayTargetsHeader, the exhausted-transport and cache-backend cases) now record the dispatch history a real request has.TestRetryCountHeadercases passed a protocol-error count over an empty error list, and now record the error.TestHedgeTriggeredHeaderalso assertsLava-Retries: 1when the hedge fired, which is MAG-3723 at unit level.TestRESTListener_HedgedLoserCountsAsARetryruns the whole path end to end: the REST listener, the real state machine's hedge ticker, session selection, the dispatcher and the header. The pin goes to an upstream that never answers, andlava-relay-timeout: 300ms. It asserts the hedge fired,Lava-Retries: 1,Lava-Provider-Address: slow-upstream,fast-upstream, and that the router cut the losing attempt off.TestSendRelayToDirectEndpoints_ReleasesADegradedSessionUnderItsOwnKeytakes two sessions under the plain key for an archive request and has the consistency filter reject both. Both must leave the dispatch history.TestWriteOutcomeIsUnknown_ADegradedWriteTheFilterKeptFromEveryNoderuns the same with a write and a spent budget: the verdict must be a failure, not an unclear write.TestUsedProviders_DispatchedProvidersNamesEveryAttemptcovers the list: batch order, a provider asked twice, a release before dispatch (only the latest entry goes), an answer (removes nothing), and that the caller gets a copy.Each fix was broken on purpose to check that its tests go red. The first four rows were measured at d69a7d3 (via
go test -overlay), the last two at 1fd49aa:TestHedgeTriggeredHeader/hedge firedfail. Before the back-fill, no pre-existing test did.AddUsedstops recordingUsedProviderstests, the end-to-end test, the degraded-release test and two existingwriteOutcomeIsUnknowntests failAll at d69a7d3, run from a clean checkout. Its tree is identical to the one tested, and each commit's tree was also checked on its own:
go build ./...,go vet ./protocol/...andgo test -count=1ofprotocol/rpcsmartrouter,protocol/lavasessionandprotocol/relaycorepass.-race.--new-from-rev=origin/main, no issue caps) reports 0 issues onprotocol/rpcsmartrouterandprotocol/lavasession, and gofumpt is clean.Live check
Router binaries built from
main(1ab93ab), from this PR as first pushed (1fd49aa) and from this head (d69a7d3). They ran against a private copy of the provider simulator on shifted ports, so the one already running locally was not touched. Every request iseth_getBalance, pinned withlava-select-provider: sim-1.mainlava-relay-timeout: 300mstrue, noLava-Retries, only the peer that answeredtrue,1,sim-1,<peer>true,1,sim-1,<peer>(5/5, ~370 ms)1,sim-1,sim-32,sim-2,sim-1,sim-32,sim-1,sim-2,sim-32,sim-2,sim-1,sim-32,sim-1,sim-2,sim-3The earlier live check at 1fd49aa (all three providers slow, p1 down with a sequential retry, all fast, and all hang for MAG-3764) is unchanged by the new commits: they do not touch the count.
The automation tests this should turn green
The v1.5.3 automation run the ticket cites (36027998730) has eleven hedge or retry failures. Ten have a result when the reply is built and should pass on the first fast nightly after merge, which deploys router
mainby default:test_router_retry_hedge_ticker.py× 3test_retry_failover_by_chain.py::test_hedge_fires_when_all_providers_slow[solana]test_cv_round_never_hedges.py::test_plain_request_under_same_latency_does_hedgetest_freshness_lag_and_relay_timeout.py::test_relay_timeout_boundary_slow_provider_retries_within_budget(MAG-3723)test_primary_backup_failover.py::test_failover_respects_per_relay_timeout_budgettest_phase1_1_transport_failures.pyhang-recover-pinnedand the twohang-and-dropcasesThe eleventh,
all-hang-terminal, is MAG-3764 and stays red.Merge note
Squash and merge only. The branch takes
mainthrough a merge commit (82c9323), and on this repo that makes "Rebase and merge" fail with "cannot be rebased".scripts/changelog-bump.shtakes the firstBREAKING CHANGE:paragraph of the squash commit. With the default message, that is commit 2's paragraph, which predates the ordering and write-verdict changes. So give the squash an explicit message, for examplegh pr merge 440 --squash -F <file> --match-head-commit <head sha>, whose footer is:Follow-ups
docs/api/directives.mdstill says only "Number of retry attempts made for this relay". Proposed: attempts beyond the first that the router sent, including a hedged attempt it cancelled because another node answered first. The same page describesLava-Provider-Addressas a single node. On a retry or hedge it is a list, in the order the router asked, with the answering provider last. Owner: @avitenzer.docs/METRICS.mdsays so). The dashboard repo has not been checked for a panel that adds them.Noticed during review, not changed here (both predate this PR):
Lava-Hedge-Triggereda count, so a client could compute ordinary retries asLava-Retriesminus that count. The router emits"true", and the automation asserts"true", so neither the router nor the tests do the subtraction MAG-1818 intended.Lava-Errored-Providersheader (lava-debug-relay) looks errored providers up under a key built from the request's extensions. On the degraded path the providers are filed under the plain key, so it misses them. This is the same kind of key mix-up the third commit fixes for releases.Jira ticket: MAG-3762
Author Checklist
I have...
!in the type prefix if API or client breaking change (added:Lava-RetriesandLava-Provider-Addresschange relative to v1.5.3)mainbranchdocs/METRICS.mdhere; the public docs line is a follow-up, see above)Reviewers Checklist
I have...
🤖 Generated with Claude Code