Skip to content

fix(rpcsmartrouter)!: count and name a hedge's losing attempt again (MAG-3762) - #440

Merged
avitenzer merged 7 commits into
mainfrom
fix/MAG-3762-hedge-loser-counts-in-lava-retries
Oct 5, 2026
Merged

avitenzer merged 7 commits into
mainfrom
fix/MAG-3762-hedge-loser-counts-in-lava-retries

Conversation

@avitenzer

@avitenzer avitenzer commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

Description

Bug (MAG-3762). Since v1.5.2 a hedge's losing attempt is missing from Lava-Retries. A request pinned to a slow provider with lava-relay-timeout: 300ms is answered by a peer. The reply carries Lava-Hedge-Triggered: true but no Lava-Retries header, and Lava-Provider-Address names only the peer. MAG-3723 is the nightly ticket for the same failure (test_relay_timeout_boundary_slow_provider_retries_within_budget, red on every nightly since 2026-09-16), and MAG-3693 was closed onto it.

Cause. appendHeadersToRelayResult sums the results recorded when the reply is built. Since 9730cf5 (a correct fix for two production incidents), an attempt outlives the window that dispatches the hedge, so the first attempt is still running when the hedge wins. The router cancels it once the relay loop ends, and the attempt posts its result to a channel that nothing reads again, so it is in none of the three lists. Before 9730cf5 the window killed that attempt first, and it was recorded as a timeout. The sum itself has not changed since the initial implementation (22cbf32).

Decision. The ticket leaves the choice between its two options to the router team. This PR takes option 1 and restores the count, for three reasons:

  • It is what MAG-1818 ruled.
  • The automation tests are written against that ruling.
  • Narrowing the header silently breaks clients who read it, and v1.5.2 and v1.5.3 already did that without saying so.

The ticket records the choice. Say so if you would rather take option 2.

Breaking, on purpose. Relative to v1.5.2 and v1.5.3, a hedged reply changes. Lava-Retries is at least 1 again, and Lava-Provider-Address names the losing provider before the one that answered, as v1.5.1 did. Lava-Provider-Address now also lists the providers of every retried request in the order the router asked them. It used to name every protocol error before every node error, and v1.5.1 did the same. README's MAJOR rule covers HTTP metadata headers and names Lava-Provider-Address, so the fix commit is fix! with a BREAKING CHANGE: footer, and scripts/changelog-bump.sh will list it. See Merge note below for the footer text to keep. 9730cf5 was not marked, and that is why the v1.5.2 and v1.5.3 release notes never mentioned the change (Victoria's 2026-09-24 comment).

Commits.

  1. refactor(lavasession): UsedProviders kept a count of the sessions a request dispatched (SessionsDispatched, used by writeOutcomeIsUnknown). It now keeps their names, in the order the batches went out, and the count is the list's length. DispatchedProviders() returns a copy. The header needs the names, not just the count. The in-flight set can't supply them, because it drops a provider when its session is freed, and that happens before the provider's result is posted.
  2. fix(rpcsmartrouter)!: attemptsWithoutResult returns the dispatched attempts with no recorded result, subtracting one attempt per result. The header adds them to Lava-Retries and names them in Lava-Provider-Address. The provider that answered stays last (MAG-1871). The protocol-error count now comes from the same snapshot as those lists. Before, a result recorded by a still-draining reader between the two reads was counted zero times. docs/METRICS.md now says the retry and hedging series overlap.
  3. fix(rpcsmartrouter): rejected sessions were released under a key built from the request's extensions. A request that degrades to a regular provider takes its sessions under the plain key, so the release did nothing, and the provider stayed in the dispatch history. Commit 2 would have turned that into a retry naming a node that never got the request. Each release now uses the session's own key (SingleConsumerSession.RouterKey()). Off the degraded path the two keys are the same.
  4. Merge branch 'main': brings in current main, including test(rpcsmartrouter): fix the duplicate solanaGetBlockReply helper #433, so the rpcsmartrouter test package compiles on this branch. It is a merge rather than a rebase, so the three reviewed commits keep their SHAs.
  5. fix(rpcsmartrouter): Lava-Provider-Address walks the dispatch history, then the results by kind for any provider the history does not hold. It used to walk the results by kind with the attempts still out in a fixed slot. A batch that reaches this list holds one session: only a stateful fan-out or a cross-validation round sends several at once, and neither builds this list. So the list follows the order the providers were asked, whatever became of each attempt (review finding 1).
  6. test(rpcsmartrouter): every pre-existing header test that reaches the retry count now carries the dispatch history its scenario implies, and the test mock panics if a test records results without one (review finding 3).
  7. test(rpcsmartrouter): the degraded-release test also runs a write and asserts writeOutcomeIsUnknown (review finding 4).

Behaviour to know about.

  • Lava-Provider-Address follows the order the router asked the providers, with the one that answered last. A sequential retry whose failures were of different kinds reorders too. For example, a node error from p1 and then a 503 from p2 used to read p2,p1,p3 on main, and now reads p1,p2,p3. The automation already reads the list this way: a pinned provider must come first, and the primaries before the backup that answered.
  • A provider asked twice in one request is still named once in Lava-Provider-Address, as before. On that path the names stay one short of Lava-Retries + 1. Otherwise every counted attempt is also named.
  • A failed request that had an attempt still out now reports it too. For example, one attempt errored and another hung until the budget ran out.
  • A stateful write that degraded to regular providers no longer counts a node the consistency filter kept it from as "asked and silent". So when every node the write actually reached came back, it gets the ordinary failure instead of the 500 "transaction status unclear". That verdict is correct: the rejected nodes never got the transaction. The path is narrow, because the write has to carry an extension that no provider serves.
  • A sequential retry (its earlier attempt already has a result), a stateful fan-out (never reports retries), cross-validation (its own headers) and a request with no hedge are unchanged in their counts. Lava-Hedge-Triggered is untouched.
  • The four retry incident series (smartrouter_retries_total, _success_total, _failed_total, smartrouter_retry_attempts) are recorded from the same count, so they count hedged requests again. Expect them to step up for hedged traffic after rollout. That is a correction, not new retry pressure.

Not fixed here: every attempt hangs (MAG-3764). When no attempt reports back at all, SendParsedRelay returns before any header is built. The 500 then carries no Lava-Retries, no Lava-Guid and no Lava-Provider-Address. That is test_transport_failure_all_providers_terminate_bounded[all-hang-terminal]. Victoria's MAG-3764 comment traces it to the same 9730cf5 change, but the fix is a result synthesized on the no-results error path, which is its own change.

Tests

  • TestAttemptsWithoutResult covers the subtraction: an attempt still out, two still out, node errors counting as results, a provider asked twice, and a cache hit (never dispatched, so it settles no attempt).
  • TestRetryCountHeader_CountsAttemptsWithoutResult checks both headers for:
    • the ticket's race;
    • MAG-1818's own three-attempt example (Lava-Retries: 2);
    • the all-slow case where the first attempt wins;
    • a node error asked before the attempt still out, and a protocol error asked after it. The fixed slot read both as p2,p1,p3;
    • a sequential retry with a node error and then a protocol error, which is listed in the order it went out;
    • a result whose provider the dispatch history does not hold, which is still named;
    • a cache hit winning while an attempt is out;
    • a stateful fan-out, sent as one batch (still no header);
    • a result recorded after the caller read its count (counted once).
  • The pre-existing header tests (TestRetryCountHeader, TestRetryCountHeaderStatefulFanoutAbsorption, TestHedgeTriggeredHeader, TestCacheServedResponseHeaders, TestResolverAlwaysLastInProviderHeader, TestStatefulRelayTargetsHeader, the exhausted-transport and cache-backend cases) now record the dispatch history a real request has.
    • Three TestRetryCountHeader cases passed a protocol-error count over an empty error list, and now record the error.
    • The stateful-API cases run under Stateful selection with a one-batch fan-out, where three of them ran as Stateless.
    • TestHedgeTriggeredHeader also asserts Lava-Retries: 1 when the hedge fired, which is MAG-3723 at unit level.
  • TestRESTListener_HedgedLoserCountsAsARetry runs the whole path end to end: the REST listener, the real state machine's hedge ticker, session selection, the dispatcher and the header. The pin goes to an upstream that never answers, and lava-relay-timeout: 300ms. It asserts the hedge fired, Lava-Retries: 1, Lava-Provider-Address: slow-upstream,fast-upstream, and that the router cut the losing attempt off.
  • TestSendRelayToDirectEndpoints_ReleasesADegradedSessionUnderItsOwnKey takes two sessions under the plain key for an archive request and has the consistency filter reject both. Both must leave the dispatch history. TestWriteOutcomeIsUnknown_ADegradedWriteTheFilterKeptFromEveryNode runs the same with a write and a spent budget: the verdict must be a failure, not an unclear write.
  • TestUsedProviders_DispatchedProvidersNamesEveryAttempt covers the list: batch order, a provider asked twice, a release before dispatch (only the latest entry goes), an answer (removes nothing), and that the caller gets a copy.

Each fix was broken on purpose to check that its tests go red. The first four rows were measured at d69a7d3 (via go test -overlay), the last two at 1fd49aa:

break result
attempts still out not counted 6 cases in the new table and TestHedgeTriggeredHeader/hedge fired fail. Before the back-fill, no pre-existing test did.
every dispatched attempt counted on top of the results 13 pre-existing header cases fail, plus the new table. Before the back-fill, none of the pre-existing cases did.
attempts still out in the fixed slot (this PR as first pushed) the 3 new ordering cases fail
release under the request's key the degraded read test and the degraded write-verdict test fail
AddUsed stops recording both UsedProviders tests, the end-to-end test, the degraded-release test and two existing writeOutcomeIsUnknown tests fail
protocol errors counted from the caller's read only the snapshot case fails

All at d69a7d3, run from a clean checkout. Its tree is identical to the one tested, and each commit's tree was also checked on its own:

  • go build ./..., go vet ./protocol/... and go test -count=1 of protocol/rpcsmartrouter, protocol/lavasession and protocol/relaycore pass.
  • The changed header, degraded and write-outcome tests pass under -race.
  • golangci-lint (--new-from-rev=origin/main, no issue caps) reports 0 issues on protocol/rpcsmartrouter and protocol/lavasession, and gofumpt is clean.
  • Commits 5 and 6 each build and pass the header tests on their own.

Live check

Router binaries built from main (1ab93ab), from this PR as first pushed (1fd49aa) and from this head (d69a7d3). They ran against a private copy of the provider simulator on shifted ports, so the one already running locally was not touched. Every request is eth_getBalance, pinned with lava-select-provider: sim-1.

scenario main 1fd49aa d69a7d3
MAG-3723: sim-1 at 2500 ms, lava-relay-timeout: 300ms hedge true, no Lava-Retries, only the peer that answered hedge true, 1, sim-1,<peer> hedge true, 1, sim-1,<peer> (5/5, ~370 ms)
sim-1 node error, retry to sim-2 at 2500 ms, hedge answered by sim-3 1, sim-1,sim-3 2, sim-2,sim-1,sim-3 2, sim-1,sim-2,sim-3
sequential: sim-1 node error, sim-2 503, sim-3 answers 2, sim-2,sim-1,sim-3 same 2, sim-1,sim-2,sim-3

The earlier live check at 1fd49aa (all three providers slow, p1 down with a sequential retry, all fast, and all hang for MAG-3764) is unchanged by the new commits: they do not touch the count.

The automation tests this should turn green

The v1.5.3 automation run the ticket cites (36027998730) has eleven hedge or retry failures. Ten have a result when the reply is built and should pass on the first fast nightly after merge, which deploys router main by default:

  • test_router_retry_hedge_ticker.py × 3
  • test_retry_failover_by_chain.py::test_hedge_fires_when_all_providers_slow[solana]
  • test_cv_round_never_hedges.py::test_plain_request_under_same_latency_does_hedge
  • test_freshness_lag_and_relay_timeout.py::test_relay_timeout_boundary_slow_provider_retries_within_budget (MAG-3723)
  • test_primary_backup_failover.py::test_failover_respects_per_relay_timeout_budget
  • test_phase1_1_transport_failures.py hang-recover-pinned and the two hang-and-drop cases

The eleventh, all-hang-terminal, is MAG-3764 and stays red.

Merge note

Squash and merge only. The branch takes main through a merge commit (82c9323), and on this repo that makes "Rebase and merge" fail with "cannot be rebased".

scripts/changelog-bump.sh takes the first BREAKING CHANGE: paragraph of the squash commit. With the default message, that is commit 2's paragraph, which predates the ordering and write-verdict changes. So give the squash an explicit message, for example gh pr merge 440 --squash -F <file> --match-head-commit <head sha>, whose footer is:

BREAKING CHANGE: Lava-Retries and Lava-Provider-Address change relative to v1.5.2 and v1.5.3. A request where a hedge fires again reports the hedge's losing attempt: Lava-Retries is at least 1, and the losing provider is named before the one that answered, as in v1.5.1 and earlier and as MAG-1818 specifies. Lava-Provider-Address lists the providers of a retried request in the order the router asked them, where it used to name every protocol error before every node error; the provider that answered stays last. A write that degraded to regular providers no longer reports an unclear transaction status because of a node the consistency filter kept it from.

Follow-ups

  • Option 1 needs an explicit ack. The review asks for one on the ticket before merge, and none is recorded yet.
  • Public docs line (not in this PR: it lives in Magma-Devs/docs). The ticket asks for the contract to be written down whichever option is taken. docs/api/directives.md still says only "Number of retry attempts made for this relay". Proposed: attempts beyond the first that the router sent, including a hedged attempt it cancelled because another node answered first. The same page describes Lava-Provider-Address as a single node. On a retry or hedge it is a list, in the order the router asked, with the answering provider last. Owner: @avitenzer.
  • Dashboards. The four retry series now overlap the hedging series for hedged traffic (docs/METRICS.md says so). The dashboard repo has not been checked for a panel that adds them.
  • Delivery. Merging updates the nightly. Customers get the fix with the next release and rollout, and those release notes will list it under breaking changes. Owner: @avitenzer.
  • MAG-3764. Open, and needs its own router change on the no-results path.

Noticed during review, not changed here (both predate this PR):

  • MAG-1818's 2026-05-17 decisions made Lava-Hedge-Triggered a count, so a client could compute ordinary retries as Lava-Retries minus that count. The router emits "true", and the automation asserts "true", so neither the router nor the tests do the subtraction MAG-1818 intended.
  • The debug-only Lava-Errored-Providers header (lava-debug-relay) looks errored providers up under a key built from the request's extensions. On the degraded path the providers are filed under the plain key, so it misses them. This is the same kind of key mix-up the third commit fixes for releases.

Jira ticket: MAG-3762


Author Checklist

I have...

Reviewers Checklist

I have...

  • confirmed the correct type prefix in the PR title
  • confirmed all author checklist items have been addressed
  • reviewed state machine logic, API design and naming, documentation is accurate, tests and test coverage

🤖 Generated with Claude Code

@avitenzer
avitenzer requested a review from a team as a code owner September 25, 2026 11:24
@avitenzer
avitenzer force-pushed the fix/MAG-3762-hedge-loser-counts-in-lava-retries branch from 4fabcdc to 1fd49aa Compare September 25, 2026 12:03
@avitenzer avitenzer changed the title fix(rpcsmartrouter): count and name a hedge's losing attempt again (MAG-3762) fix(rpcsmartrouter)!: count and name a hedge's losing attempt again (MAG-3762) Sep 25, 2026
@AnnaR-prog

Copy link
Copy Markdown
Contributor

Review: manual pass (high effort) on 1fd49aa

Reviewed directly against the PR base 887e2bc (no /code-review fan-out), with a probe run in a scratch worktree. No correctness defects found. I went after three specific ways this could over-count and each one is already closed; details below so the work is not repeated. One coverage gap and a base note.

1. Low (coverage) — CONFIRMED: CrossValidation is absent from the new table, and it is the selection most likely to look affected

TestRetryCountHeader_CountsAttemptsWithoutResult carries a selection field with rows for Stateless and Stateful. Cross-validation early-exits the moment a hash reaches quorum and cancels the rest — relay_processor.go says so in as many words ("providers whose responses may not have been received due to early exit when threshold met"), and crossValidationPendingProviders exists for it — so a reader of this diff will reasonably assume a cross-validated request now reports its cancelled participants as retries.

It does not, and I only established that by running it. appendHeadersToRelayResult branches on selection == relaycore.CrossValidation at its very top and returns from that branch, so unfinished, totalAttempts and the provider-chain rewrite are never reached:

CV 3/2, quorum on 2, p3 cancelled -> Lava-Retries="" (present=false) Lava-Provider-Address=""
same shape with selection=Stateful  -> retry header present=false

That is the right behaviour, and it is worth one table row rather than leaving it to be rediscovered — a future refactor that unifies the two branches would change cross-validation reply headers for every early-exiting request, and nothing in the suite would notice. Same reason the Stateful row is there.

It also means the BREAKING CHANGE: footer's scope ("for a request where a hedge fires") is accurate as written — worth stating, since the METRICS.md note added here ("A hedged request is counted here as well") does not say the cross-validation path is untouched.

Verified and not flagged

These are the over-count paths I tried to build and could not:

  • A double release cannot delete an earlier batch's entry. ReleaseFromLatestBatch returns early on if _, inBatch := uniqueUsedProviders.providers[provider]; !inBatch, so the backwards scan over dispatched only runs for a provider actually in the batch under that router key. This is also exactly what the second commit's fix makes true: with the request-derived key, inBatch was false, the function did nothing, and the provider stayed in the dispatch history — which is the bug described.
  • A retried provider is accounted per attempt, not per name. dispatched carries one entry per session and attemptsWithoutResult decrements recorded[addr] as it walks, so a provider asked twice and answering once leaves exactly one attempt out. A provider asked twice in two batches, answering both times, leaves none.
  • No session escapes both a release and a result. Every entry of sessions is dispatched in a goroutine whose defer always calls relayProcessor.SetResponse(...), and the three early-exit paths before that loop (consistency-rejected, post-filter session shortfall, group shortfall) each pair ReleaseFromLatestBatch with Session.Free. The failedSessions and validSessions maps are disjoint splits of sessions, so nothing is released twice.
  • A provider-less protocol error cannot double-count. I expected recorded[""] from an error recorded before a provider was assigned to leave a dispatched attempt unsettled and counted twice. results_manager.go:77 builds every RelayError from response.RelayResult.ProviderInfo, and the direct path seeds localRelayResult with ProviderAddress: endpointAddress before anything can fail, so protocol errors on this path always carry an address. The one address-less RelayError (relay_errors.go:114) is a derived merge for the user-facing message and is never added to the list.
  • Stateful is absorbed for the provider chain too, not just the count: the allProvidersList rewrite lives inside if totalRetries > 0, which stays 0 for Stateful.
  • protocolErrorCount := max(protocolErrors, uint64(len(protocolErrorResults))) is the right shape for the race the comment describes, and the "a result recorded after the caller read its count is counted once" sub-test pins it. The list only grows, so the max is the snapshot's own count.
  • SessionsDispatched() semantics are unchanged — len(up.dispatched) equals the old counter under every mutation path above.
  • RouterKey() is symmetric with AddUsed, which files the provider under the session's own key, so release and add now agree by construction rather than by coincidence.
  • The METRICS.md note is the right disclosure: the retry and hedging families genuinely overlap now, and saying "do not add them together" is more useful than trying to separate them.

Base

go test ./protocol/rpcsmartrouter/ does not build at 887e2bc: solanaGetBlockReply is declared twice (direct_rpc_integration_test.go:872 and solana_context_slot_bench_test.go:75). Main fixed it in f404669, whose message notes CI never compiles the test binary. So hedge_loser_retry_count_test.go, degraded_session_release_test.go and the rpcsmartrouter_server_test.go additions have not run anywhere — not on CI, not locally at this head. Rebase onto current main; I confirmed the test binary compiles cleanly once f404669 is applied (go vet ./protocol/rpcsmartrouter/ passes on this head with it cherry-picked), so a rebase is the whole fix. I cherry-picked f404669 into a worktree to run the probe above. (protocol/lavasession builds and tests fine at this base.)

Merge order

All nine of #436–#444 branch from 887e2bc and none contains another, so each "Files changed" is its own. This one shares protocol/rpcsmartrouter/rpcsmartrouter_server.go with #438, #439, #441 and #444, but git merge-tree finds no conflict against any of them — the hunks are far apart, and used_providers.go / single_consumer_session.go are touched by this PR alone. Nothing here needs a particular merge order.

@AnnaR-prog

Copy link
Copy Markdown
Contributor

Second pass on 1fd49aa: I was wrong to say "no correctness defects"

My review above reported none. An adversarial pass found four, and I verified each against the tree myself. One is a behavioural regression in the header this PR exists to fix. Withdraw the "no defects found" line in my earlier comment.

1. Blocking — Lava-Provider-Address can now be out of dispatch order

The new loop is in a fixed slot: after every protocol error, before every node error.

for _, addr := range unfinished {
    if addr == providerAddress { continue }
    addProvider(addr)
}

Take a dispatch order of p1 → p2 → p3 where p1 returns a node error, p2 is hedged over and still running, and p3 wins. unfinished = [p2], the walk is protocol errors (none) → unfinished → node errors → successes → resolver last, so the header is:

p2,p1,p3

p2 named before p1, although p1 was asked first. The comment eight lines above still calls the list "chronological-ish order", and MAG-1871's "last entry == response source" still holds — it is the middle that is now wrong.

The PR body's justification is what breaks: it says the unfinished attempt "sits with the protocol errors, which is where v1.5.1 listed the attempt a hedge overtook." In v1.5.1 the window killed that attempt and it was recorded as a timeout, so its position among the protocol errors was chronological by construction. A fixed slot ahead of all node errors is a different thing.

This PR adds the fix for it: DispatchedProviders() returns the true dispatch order, and the ordering loop ignores it. Walking dispatched and emitting each name at its own position (skipping the resolver, appending it last) preserves both MAG-1871 and chronology.

2. Blocking — the test that covers the slot cannot fail

hedge_loser_retry_count_test.go:189-203:

name: "an attempt still out sits between the protocol errors and the node errors",
usedProviders:  dispatchedTo("p1", "p2", "p3", "p4"),
protocolErrors: [p1], nodeErrors: [p3], successResults: [p4],
resolver: "p4", wantProviders: "p1,p2,p3,p4",

unfinished is p2, so the fixed slot lands exactly where dispatch order would put it. The expectation passes whether the slot is right or wrong. The arrangement in finding 1 (node error before the unfinished attempt) is the one the table does not have.

3. Blocking — every pre-existing header test is blind to the new term

rpcsmartrouter_server_test.go has 36 MockRelayProcessorForHeaders{ literals and 1 occurrence of usedProviders:. So essentially all of them hand appendHeadersToRelayResult recorded results with an empty dispatch history — attemptsWithoutResult returns nil and the new term is inert. In a real request the history always has at least as many entries as there are results.

That means the MAG-1653 Bug #2 guard, the MAG-1871 "resolver last" contract and the stateful-absorption case all pass with the new code removed. Coverage for it is the eight cases in the one new file. The field comment ("Unset reads as a request whose dispatch history is empty, which is what every test written before MAG-3762 assumed") states the problem and moves on; back-filling usedProviders on the existing cases is what makes them guard again.

4. Blocking — SessionsDispatched() has a client-visible second consumer the body never mentions

rpcsmartrouter_server.go:1122, inside writeOutcomeIsUnknown:

dispatched := relayProcessor.GetUsedProviders().SessionsDispatched()

compared against answered, and that verdict forces a stateful write's reply to HTTP 500 "unclear write status". Commit 3 makes the degraded-path release actually work, so on that path dispatched gets smaller, answered >= dispatched becomes more likely, and the verdict can flip from unknown-write to ordinary failure — a different status code for a write.

The new direction is probably correct (a consistency-rejected provider never got the request). But it is a client-visible change on the write path, it is not in the BREAKING CHANGE: footer, which describes headers only, and degraded_session_release_test.go:110-113 asserts DispatchedProviders, SessionsLatestBatch and CurrentlyUsed — never the write verdict. Given the recent work in exactly this area on this repo, that is worth an assertion.

Worth a decision, not a code change

  • The PR body asks the team to confirm option 1 ("Say so if you would rather take option 2"). Nothing in the PR or the branch records an answer. README.md:477 defines MAJOR as a breaking change to HTTP metadata headers and names Lava-Provider-Address, so merging schedules a major bump to restore v1.5.1 behaviour. Worth an explicit ack on the ticket before merge rather than after.
  • The same if totalRetries > 0 block drives smartrouter_retries_total, _success_total, _failed_total and smartrouter_retry_attempts, so four counters change meaning for every hedging deployment. docs/METRICS.md documents the overlap accurately; the dashboard repo gets no mention.

Still correct from my first pass, and worth keeping

The data-structure work is sound and I re-verified it: the inBatch guard makes a cross-batch release impossible, per-attempt accounting is right for a provider asked twice, every dispatched session posts a result carrying its own address, cache hits settle no attempt, and CrossValidation never reaches this code (its branch returns early — I proved that by running it). protocol/lavasession is green at this head (go test ./protocol/lavasession/ -run 'UsedProviders|Dispatched'), so the dispatch-history invariants are proven. And TestRESTListener_HedgedLoserCountsAsARetry is a genuine end-to-end test of the ticket's repro, with premise assertions so it cannot pass vacuously — it has just never executed, for the base reason in my first comment.

@avitenzer

Copy link
Copy Markdown
Contributor Author

Thanks for both passes. All four blocking findings are addressed at d69a7d3.

The branch now has current main merged in (82c9323), which brings in #433. So the rpcsmartrouter test package compiles, and everything below ran from the branch. I merged instead of rebasing, so the three commits you reviewed keep their SHAs. The squash-merge flattens the history either way.

1. Dispatch order. Fixed in bbc1915. Lava-Provider-Address now walks DispatchedProviders(), skipping the resolver and appending it last. It then walks the results by kind, but only for a provider the history does not hold.

One correction to the premise, for the record. Your example (p1 node error, p2 hedged over, p3 wins) also read p2,p1,p3 under v1.5.1's code. The killed attempt was recorded as a protocol error, and protocol errors were walked before node errors. The fixed slot did diverge from v1.5.1 in the other direction: with an attempt still out and then a protocol error from the hedge, v1.5.1 read p1,p2,p3 and the slot read p2,p1,p3. Dispatch order fixes both. It is also what the automation already assumes: test_pin_behavior_locked.py asserts chain[0] == pinned and set(chain[:3]) == primaries.

It also reorders a retry with no hedge whose failures are of different kinds. main itself reads p2,p1,p3 for "p1 node error, p2 503, p3 answers", so that change is in the breaking note too. Live, with one private simulator for all three builds:

scenario main 1fd49aa d69a7d3
p1 node error, retry to slow p2, hedge answered by p3 1, p1,p3 2, p2,p1,p3 2, p1,p2,p3
sequential: p1 node error, p2 503, p3 answers 2, p2,p1,p3 same 2, p1,p2,p3

2. The slot test that could not fail. Fixed in bbc1915. It is replaced by four cases:

  • a node error asked before the attempt still out;
  • a protocol error asked after it;
  • a sequential retry with failures of different kinds;
  • the fallback for a result the history does not hold.

Restoring the fixed slot fails exactly the first three.

3. Pre-existing header tests blind to the new term. Fixed in 7c81585. Every literal that reaches the retry count now carries the dispatch history its scenario implies. The 12 cross-validation literals return before the history is read, so they are untouched.

  • Three TestRetryCountHeader cases passed a protocol-error count over an empty list, which a live request cannot produce. They now record the error.
  • The stateful-API cases run under Stateful selection with a one-batch fan-out. Three of them ran as Stateless.
  • TestHedgeTriggeredHeader now asserts Lava-Retries: 1 when the hedge fired.
  • The mock panics when a test records results without setting a history.

Measured with go test -overlay:

mutation pre-existing cases that fail, before and after the back-fill
every dispatched attempt counted on top of the results 0, then 13
attempts still out not counted 0, then 1 (hedge fired)

4. writeOutcomeIsUnknown. Fixed in d69a7d3. The degraded test is now a helper that runs twice. The write variant spends the budget and asserts the verdict is a failure, not an unclear write, because no node received the write. Releasing under the request's key again fails both the read and the write test. The release comment now names both readers of the history, and the change is described in the PR body and in the breaking note.

Decision items.

  • The option-1 ack is still open. That is the team's call.
  • The footer: changelog-bump.sh takes the first BREAKING CHANGE: paragraph, which is commit 2's. It predates findings 1 and 4, and rewording it would mean force-pushing over reviewed commits. The PR body's Merge note has the full paragraph to paste at squash time.
  • Dashboards are listed as a follow-up.

Verification at d69a7d3, from a clean checkout:

  • go build ./... and go vet ./protocol/... pass.
  • The full go test of rpcsmartrouter, lavasession and relaycore passes.
  • The changed tests pass under -race.
  • golangci-lint reports 0 new issues, and gofumpt is clean.

@AnnaR-prog

Copy link
Copy Markdown
Contributor

Re-review on d69a7d3 — all four blockers fixed; the one Low is still open

Re-reviewed the three author commits since my 2026-09-27 pass (bbc19155, 7c815858, d69a7d32) plus the main merge. protocol/rpcsmartrouter (113.4s) and protocol/relaycore both pass; TestRetryCountHeader is 22 passing subtests.

Prior findings

# Finding Status
Pass 2 #1 Blocking — Lava-Provider-Address can now be out of dispatch order fixed
Pass 2 #2 Blocking — the test that covers the slot cannot fail fixed
Pass 2 #3 Blocking — every pre-existing header test is blind to the new term fixed
Pass 2 #4 Blocking — SessionsDispatched() has a client-visible second consumer the body never mentions fixed
Pass 1 #1 Low (coverage) — CrossValidation is absent from the new table still open

#1 is fixed at the root rather than patched. The list no longer walks results by kind (protocol errors → still-out → node errors → successes); it walks the request's dispatch history, one entry per attempt in batch order. That removes the whole class, not just the arrangement I happened to find — the commit message names both instances it fixes (a retry named before the attempt it followed, and an attempt still out named after a protocol error from a later hedge).

#2 is fixed with a case that can actually go red. My finding was that the table's only unfinished-attempt case put the fixed slot exactly where dispatch order would also put it, so the expectation passed either way. The table now has:

name: "the first attempt wins while two hedges are still running",
usedProviders:  dispatchedTo("p1", "p2", "p3"),
successResults: []common.RelayResult{from("p1")},
resolver:      "p1",
wantProviders: "p2,p3,p1",

p2,p3,p1 is not reachable by any by-kind walk, and it pins the MAG-1871 resolver-last rule on top of dispatch order — so it discriminates in both dimensions at once. The new dispatchedTogether helper is also the right addition: it distinguishes one fan-out batch from three sequential ones, which the old dispatchedTo-only harness could not express.

#3 is fixed across the suite, not just where it was noticed. Every case that reaches the retry count now carries the history its scenario implies, and the commit is explicit about why that mattered: the pre-existing tests recorded results over an empty history, so they passed both with the new term removed and with it counting every dispatched attempt. That is the definition of a term no test constrains.

#4 is the one I expected to be pushed back on, and it was taken seriously. writeOutcomeIsUnknown compares the endpoints that answered against SessionsDispatched(), so on the degraded path a consistency-rejected session left in the history made a write the filter had kept from every node read as one sent to nodes that never answered — surfacing to the client as a 500 "transaction status unclear" once the budget ran out. degraded_session_release_test.go now runs a write and asserts the verdict directly. That is a client-visible outcome, and it now has a test.

Still open: CrossValidation has no case in the table

My pass-1 finding stands unchanged — there is no selection: relaycore.CrossValidation case in hedge_loser_retry_count_test.go, and CV is the selection whose header a reader is most likely to misread, because a CV round legitimately puts several providers in one batch and so looks like retries that never happened.

It is cheaper to close now than it was before, because dispatchedTogether already exists for exactly this shape — the stateful fan-out case is one line away from a CV twin:

{
	name: "a cross-validation round is one batch, not N retries",
	rp: &MockRelayProcessorForHeaders{
		selection:      relaycore.CrossValidation,
		usedProviders:  dispatchedTogether("p1", "p2", "p3"),
		successResults: []common.RelayResult{from("p1")},
	},
	resolver:      "p1",
	wantProviders: …,   // whichever the quorum path should name
},

Low, and not a merge blocker — but it is the last selection with no coverage in a table that now covers the other three well.

One stale paragraph, shared across this stack

The MAG-3859 / "does not compile on main" caveat is no longer true — origin/main renamed the duplicate helper apart, and the runs above are on this head with nothing uncommitted.

Still true from the first pass

The merge-order note stands: protocol/lavasession/used_providers.go is also touched by #418 (the maps.Clone fix on GetErroredProviders), and rpcsmartrouter_server_test.go by #423 and #450. All mechanical.

🤖 Generated with Claude Code

https://claude.ai/code/session_0154pmYvf8T1kbvB9pYjihvq

AnnaR-prog added a commit that referenced this pull request Oct 5, 2026
…tries (MAG-3762)

The retry-count table covered every selection but cross-validation, the one a reader is
most likely to misread. A round sends several providers in one batch and early-exits once
the quorum is met, so its unanswered participants look like attempts without a result.
They are not retries. The cross-validation branch of appendHeadersToRelayResult names the
round in its own headers and returns before the retry count is built, so the reply
carries neither Lava-Retries nor Lava-Provider-Address. That is what the PR describes and
what the existing cross-validation header tests assume.

The new case sends three providers in one batch, records two answers and leaves the third
out. It asserts that both headers are absent, that all-providers names the whole batch,
and that pending names the third. A provider header the table expects as "" must now be
absent, the same convention the retry header already uses.

Addresses the remaining review finding on #440.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@AnnaR-prog

Copy link
Copy Markdown
Contributor

Review fix pushed: 893b07b

The one open Low from my re-review on d69a7d3 is fixed: hedge_loser_retry_count_test.go's table had no CrossValidation case.

The new row, "a cross-validation round is one batch, not N retries":

  • Setup: it sends p1–p3 in one batch (dispatchedTogether), records answers from p1 and p2, and leaves p3 out.
  • What it pins: the CV branch of appendHeadersToRelayResult returns before the retry count is built. That is what the PR describes and what the existing CV header tests assume. So the reply carries:
    • neither Lava-Retries nor Lava-Provider-Address;
    • lava-cross-validation-all-providers = p1,p2,p3. The queried list is given unsorted, so the sort is pinned too;
    • lava-cross-validation-pending-providers = p3.
  • The value I left blank in my suggestion: the right answer for the provider header is "absent", not an empty value. wantProviders: "" now means the header must be absent, the same convention wantRetries already uses.

Red checks — each of these mutations fails only this row:

  • The CV branch falls through to the retry path.
  • A CV round's participants are counted as retries ("no retry header expected, got 2").
  • The CV branch also emits Lava-Provider-Address ("no provider header expected, got p1").

Results: the table's 12 subtests pass, and so does the full protocol/rpcsmartrouter package (114s).

🤖 Generated with Claude Code

@AnnaR-prog AnnaR-prog left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving: every finding from my review is fixed (see my fix comment above).

avitenzer and others added 7 commits October 5, 2026 16:24
… only the count (MAG-3762)

UsedProviders counted the sessions a request dispatched across every batch, and
writeOutcomeIsUnknown compares that count with the results that came back. The reply's
attempt headers need more than the count. They have to name an attempt that was sent and has
not reported back, such as a hedge's losing attempt, and nothing else records one. The
in-flight set drops a provider when its session is freed, which is before the provider's
result is posted.

The count becomes a list of provider names, one entry per session, in the order the batches
went out, and SessionsDispatched is its length. So the two cannot drift, and the count behaves
as before. ReleaseFromLatestBatch removes the provider's latest entry, which is the one its
batch added. DispatchedProviders returns a copy.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…MAG-3762)

Since v1.5.2 a hedge's losing attempt is missing from Lava-Retries. A request pinned to a slow
provider with lava-relay-timeout: 300ms is answered by a peer and carries
Lava-Hedge-Triggered: true, but no Lava-Retries header at all, and Lava-Provider-Address names
only the peer.

The header sums the results recorded when the reply is built. Since 9730cf5 an attempt
outlives the window that dispatches the hedge, so when the hedge wins, the first attempt is
still running. The router cancels it once the relay loop ends, and the attempt posts its
result to a channel that nothing reads again, so it is in none of the lists. Before 9730cf5
the window killed that attempt first, and it was recorded as a timeout.

The header now counts every attempt the router sent, as MAG-1818 ruled. Each dispatched
attempt with no recorded result is added to Lava-Retries and named in Lava-Provider-Address,
with the protocol errors, where v1.5.1 listed the attempt a hedge overtook. The provider that
answered stays last. A provider asked twice is still named once, as before, so on that path
the names stay one short of the count.

The protocol-error count now comes from the same snapshot as the lists the subtraction reads.
On a failure path a reader can record a result between the caller's read and the header's,
and that attempt used to be counted zero times.

A sequential retry, a stateful fan-out and cross-validation are unchanged. A request that ends
with no recorded result at all, because every attempt hung, still returns before any header is
built. That is MAG-3764.

The retry incident metrics (smartrouter_retries_total, _success_total, _failed_total and
smartrouter_retry_attempts) are recorded from the same count, so they count hedged requests
again as well. docs/METRICS.md now says the retry and hedging families overlap.

BREAKING CHANGE: for a request where a hedge fires, Lava-Retries and Lava-Provider-Address change relative to v1.5.2 and v1.5.3. The reply again reports the hedge's losing attempt: Lava-Retries is at least 1, and the losing provider is named before the one that answered. This is the behaviour of v1.5.1 and earlier, and what MAG-1818 specifies.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…taken with (MAG-3762)

sendRelayToDirectEndpoints released every session the consistency filter rejected, and every
session a cross-validation guard gave back, under a key built from the request's extensions.
UsedProviders files a provider under the key its session was taken with. The two differ when a
request degrades to a regular provider: its sessions are taken under the plain key, so the
release found nothing and did nothing. The provider stayed in the in-flight set until the
session was freed, and stayed in the dispatch history for good, and sessionsLatestBatch kept
counting it.

The previous commit made that history visible: Lava-Retries now counts every dispatched
attempt with no result, so a rejected provider on that path would show as a retry at a node
that never received the request. Each release now names the session's own key, through a new
SingleConsumerSession.RouterKey. Off the degraded path the two keys are the same, so nothing
changes there.

The new test takes two sessions under the plain key for an archive request and has the filter
reject both. With the old key both providers stay in the history.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…mpts went out (MAG-3762)

The list walked the results by kind: protocol errors, then the attempts still out, then node
errors, then successes. That is the order the providers were asked only when every failure was
of one kind. A request whose first attempt came back with a node error, and whose retry was then
overtaken by a hedge, read p2,p1,p3: the retry named before the attempt it followed. An attempt
still out was likewise named after a protocol error from a hedge sent later.

The list now walks the request's dispatch history, one entry per attempt in the order the
batches went out. Only a stateful fan-out or a cross-validation round puts several sessions in
one batch, and neither reaches this list, so it follows the order the providers were asked
whatever became of each attempt. The results by kind follow, for a provider the history does
not hold; a live request records every provider it asks, so they add nothing there. The provider
that answered stays last (MAG-1871).

This also reorders a retry with no hedge in it whose failures were of different kinds, which
used to name every protocol error before every node error. The automation already reads the
list as the order the providers were asked: a pinned provider must come first, and the
primaries before the backup that answered.

The header-table case for an attempt still out could not fail, because dispatch order and the
fixed slot agreed on it. It is replaced by three cases that tell them apart, one of them a retry
with no hedge, and by one that pins the fallback for a result the history does not hold.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

Took 5 minutes
…al request has (MAG-3762)

The header counts and names the attempts in the request's dispatch history, and a real request's
history holds the provider of every result it recorded. The header tests written before MAG-3762
recorded results over an empty history, so the new term had nothing to count in any of them.
They all passed with it removed, and all passed with it counting every dispatched attempt on top
of the results.

Every case that reaches the retry count now carries the history its scenario implies. Three
TestRetryCountHeader cases passed a protocol-error count over an empty error list, which a live
request cannot produce because the count is read from that list, so they now record the error.
The stateful-API cases record one batch under Stateful selection, as the router sends a
fan-out; three of them ran as Stateless requests. The hedge case now also checks that the reply
reports the hedge as a retry, which is the reply MAG-3723 caught without it.

The mock panics when a test records results but sets no history, so a new test cannot reopen
the gap.

With every dispatched attempt counted on top of the results, 13 cases that predate MAG-3762 now
fail, where none did. With the new term removed, the hedge case fails.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…eceived it (MAG-3762)

writeOutcomeIsUnknown compares the endpoints that answered against SessionsDispatched, the length
of the same dispatch history. On the degraded path a consistency-rejected session used to stay
in that history, so a write the filter kept from every node read as one sent to nodes that never
answered. Once the budget ran out it came back as "transaction status unclear", a 500 telling
the client the transaction may already be on chain.

Releasing under the session's own key fixed that as well, but nothing checked it. The degraded
test now runs a write too and asserts the verdict: no node received it, so it is a failure, not
an unclear write. The comment at the release names both readers of the history.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…tries (MAG-3762)

The retry-count table covered every selection but cross-validation, the one a reader is
most likely to misread. A round sends several providers in one batch and early-exits once
the quorum is met, so its unanswered participants look like attempts without a result.
They are not retries. The cross-validation branch of appendHeadersToRelayResult names the
round in its own headers and returns before the retry count is built, so the reply
carries neither Lava-Retries nor Lava-Provider-Address. That is what the PR describes and
what the existing cross-validation header tests assume.

The new case sends three providers in one batch, records two answers and leaves the third
out. It asserts that both headers are absent, that all-providers names the whole batch,
and that pending names the third. A provider header the table expects as "" must now be
absent, the same convention the retry header already uses.

Addresses the remaining review finding on #440.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@avitenzer
avitenzer force-pushed the fix/MAG-3762-hedge-loser-counts-in-lava-retries branch from 893b07b to 30a4936 Compare October 5, 2026 13:24
@avitenzer
avitenzer merged commit f7621f9 into main Oct 5, 2026
@avitenzer
avitenzer deleted the fix/MAG-3762-hedge-loser-counts-in-lava-retries branch October 5, 2026 13:30
AnnaR-prog added a commit that referenced this pull request Oct 6, 2026
…gainst MAG-2734 (MAG-4156) (#485)

* test(rpcsmartrouter): repair the test build MAG-3746 and #440 broke against MAG-2734

The rpcsmartrouter test binary has not compiled on main since MAG-3746
landed (1e08bca, 13:58). CI never runs go test, so three merges stacked
on it unnoticed. main compiled at f10367e, the commit before MAG-3746.

Collisions:

- health_check_crafted_relay_test.go (MAG-3746, 1e08bca) was written
  against the API before MAG-2734 (c7c605a, merged 12:06 the same day),
  which removed RelayRetriesManager and the relayRetriesManager server
  field, dropped the retries-manager argument from
  relaycore.NewRelayProcessor, and dropped ctx from GetEmptyRelayState.
  Then MAG-3923 (2d18323) added the backupReserve parameter to
  sendRelayToEndpoint, which hit the same call.
- degraded_session_release_test.go (#440 / MAG-3762, 9178c5b and
  e9b9070) passes lavaprotocol.NewRelayRetriesManager() to
  NewRelayProcessor.
- hedge_loser_retry_count_test.go (#440 / MAG-3762, 62bd7ee) sets the
  removed relayRetriesManager field on the server literal.

Fixes, test code only: drop the retries manager from each constructor and
server literal, call GetEmptyRelayState(message), and pass
backupReserve=false to sendRelayToEndpoint, as the production health path
(sendRelayToEndpoint call in the internal relay loop) does. None of these
tests asserted anything about the retries manager; it only fed the
archive-upgrade mutation MAG-2734 removed, so each test's intent is
unchanged.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

Took 1 minute

* test(rpcsmartrouter): pass backupReserve in the threshold-guard test MAG-3286 merged after this fix (MAG-4156)

#423 (MAG-3286, d8f2d2b) merged after this branch was cut and added
cross_validation_threshold_guard_test.go, which calls
sendRelayToEndpoint with the six arguments it took before MAG-3923
added backupReserve. Same collision as the three files fixed here
already; pass false, as the production health path and the other
repaired call do.

origin/main is merged in so the branch compiles against the main it
will land on: go test -run '^$' ./... builds every package's tests.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

Took 3 minutes

---------

Co-authored-by: AnnaR-prog <anna@magmadevs.com>
nimrod-teich pushed a commit that referenced this pull request Nov 9, 2026
…runs out unanswered (MAG-3980) (#671)

* fix(rpcsmartrouter): send the lava-* headers when a request's budget runs out unanswered (MAG-3980)

When a request's processing budget ran out before any attempt it dispatched came back, the
reply carried no lava-* header at all, not even Lava-Guid. A read got the router's 504 (a 500
before MAG-4481) and a write the 500 "transaction status unclear", and neither said which
request it was, which upstreams were asked or how many attempts went out. A request that failed
on upstream errors instead, connections reset say, got Lava-Guid, Lava-Provider-Address and
Lava-Retries.

SendParsedRelay builds those two replies in its no-results branch and returned them before
appendHeadersToRelayResult ran. Both now go through it, as every other failed reply does.

appendHeadersToRelayResult named a result with no provider "Cached", the name of a cache hit. A
reply nothing answered has no provider either, but it has no reply, and a cache hit always has
one. Such a reply now names the attempts that went out, in dispatch order and with nothing after
them, even when there was only one. Lava-Retries counts every dispatched attempt (MAG-1818,
MAG-3762), so two quiet upstreams read Lava-Retries: 1 and are both named. The same rule names
the attempts, not "Cached", on the rare failure whose chosen protocol error is the merged
fallback, which carries no provider.

The retry incident metrics (smartrouter_retries_*) are recorded from the same count as the
header (docs/METRICS.md), and the hedge incident metric beside it, so they now count these
requests too, as failed. A cross-validated request that runs out of time with no answer now gets
its cross-validation headers, every participant pending, and is counted as a failed round, which
it skipped before.

Verified: in TestFailedRelayReply_EndToEnd_MAG4481, "hung upstream is the router's 504" now
requires Lava-Guid and the upstream it asked on all four HTTP interfaces, and the new
TestFailedRelayReply_BudgetExpiryKeepsTheRequestHeaders_MAG3980 drives two quiet upstreams past a
hedge and the budget. On the code before this change both fail with "carries no Lava-Guid". The
relay_end case write_unknown_every_upstream_silent pinned the header-less reply
(require.Nil(result)); it now requires the GUID and both upstreams. The relay_end goldens are
unchanged. go test ./protocol/rpcsmartrouter/... passes, and the two timing-dependent tests pass
8 of 8 under -race.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

Took 4 minutes

* test(rpcsmartrouter): pin that Lava-Retries counts a hedge the first attempt beat (MAG-3778)

MAG-3778 reported Lava-Retries 0 on a plain request whose every upstream was slower than the
hedge window: the ticker hedged, the first attempt answered, and the header summed only the
attempts that had come back, so the hedge still running was not counted. The formula the ticket
quotes, successes + node errors + protocol errors - 1, is the one 62bd7ee (MAG-3762, #440)
replaced. Lava-Retries counts every attempt the router sent (MAG-1818), dispatched attempts with
no result included, and docs/METRICS.md says so. That fix first shipped in v1.6.0. The nightly
failures on the ticket, 2026-09-22 to 2026-10-05, all predate it, and the v1.5.x line (v1.5.9,
v1.5.10) does not carry it.

No code change is needed. The scenario was pinned only against a mocked processor
(TestRetryCountHeader_CountsAttemptsWithoutResult, "the first attempt wins while two hedges are
still running"), and the existing end-to-end test has the hedge win. This adds the ticket's own
shape end to end, through the JSON-RPC listener, the state machine's ticker and the dispatcher:
two upstreams slower than the window, the first attempt answers and the hedge is cut off. It
requires Lava-Retries 1, the hedge named before the upstream that answered, and
Lava-Hedge-Triggered.

Verified: it passes, 10 of 10 under -race. With the header counted by the ticket's formula (an
overlay that drops the attempts without a result from the count) it fails with Lava-Retries
absent, the nightly's symptom.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix(rpcsmartrouter): name the reason when a cross-validated request's budget runs out unanswered (MAG-3980)

A cross-validated request whose budget ran out before any attempt came back got the router's 504
with its cross-validation headers but no lava-cross-validation-failure-reason. Every other
cross-validation failure names one, and cross_validation_failures_total, the by-reason breakdown
of cross_validation_failed_total, never counted this one.

Nothing came back, so fewer successful responses than the threshold did: insufficient-responses,
the reason the relay processor gives when too few came back. The 504 now carries it.

The test sends a cross-validated request to three quiet upstreams with a threshold of two. Three,
because the 429 cases earlier in the file leave upstream-1 held off, and selection leaves a
held-off upstream out while another is ready. The upstreams' node-url timeout keeps every attempt
out past the request's budget, since a cross-validated attempt is not cut off with the request.
Without the fix it fails with an empty reason.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Anna Rayer <anna@magmadevs.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants