Repository navigation
Conversation
…tead of failing one dispatch short of it (MAG-3978) Three primaries that fail fast are three errors, one past the default retry limit of 2, and the policy stopped the request there: one dispatch before GetSessions would have found the primary pool empty and handed the request to the backup tier. The caller got HTTP 500 in about 20 ms while a healthy backup sat unused. Every tenant runs the default limit. The relay sender now answers whether a backup this request has not been sent to could still serve it (not blocked this epoch, not used or unwanted under the request's router key, serving its addon and extensions), the state machine carries that answer in DecisionInput on the gotResults path, and the tolerance does not stop a request while it is true. Dispatch order is unchanged: primaries until the pool is exhausted, then backups. The session manager's backup fallback and the new question read one eligibility list, so the two cannot drift. Recorded on purpose, for the owners' call: while a backup is untried the limit does not bound the primaries either, so a pool larger than the limit is tried in full before the backup; and a limit of 0 still reaches a backup. Jira ticket: MAG-3978 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Took 1 hour 41 minutes
|
Parked, not for merge. Tom ruled on MAG-3978 on 2026-10-25 (comment 30709) that the stop at the retry limit is the expected behaviour: at the configured retries the router does not try the remaining nodes, primaries or backups alike, the same decision as MAG-3923. The automation suite already encodes that ruling (MAG-4612 and MAG-4604, merged 2026-10-25): 22 tests now assert the stop with no backup asked, and they would fail on this build. This PR therefore stays a draft as a concrete proposal for revisiting the rule, with its two open semantics decisions in the body. Close it if the owners keep the ruling. |
…der is exhausted (MAG-3978) The end-to-end test asserted an error on the failed paths and a success on the served ones, but never the status. The verify-bug-fix run on the local k3d cluster read HTTP 502 for a closed port on both builds, while the pull request described 500, which is what the ticket measured before the MAG-4218 status rule. Every case now pins the status: 200 when a backup serves, 502 when no node answered (a refused connection), and the node's own 503 when it answered 503. The exhaustive and control cases run for both failure shapes and read the dispatch list and the retry count from the failed reply, which carries them. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Took 15 minutes
|
Closed unmerged. The router owner's 2026-10-25 ruling on MAG-3978 stands: the stop at the retry limit is intended, and MAG-4612 put that rule into the automation suite. The ticket is cancelled as working as designed. The branch stays as the record of the alternative. The description holds the design, the unit and k3d evidence (a backup answers where main stops, measured twice on 2026-10-28), and the two open semantics questions, should the rule ever be revisited. |
Draft, parked. Not for merge. Tom ruled on MAG-3978 on 2026-10-25 (comment 30709) that the stop at the retry limit is the expected behaviour: at the configured retries the router does not try the remaining nodes, primaries or backups alike, the same decision as MAG-3923. The automation suite encodes that ruling (MAG-4612 and MAG-4604, merged 2026-10-25): 22 tests assert the stop with no backup asked, and they fail on this build, as measured below. This PR is the concrete proposal for revisiting the rule, with its two open semantics decisions at the end. Close it if the owners keep the ruling.
Jira ticket: MAG-3978
What this changes
Three primaries that fail fast are three errors, one past the default retry limit of 2, and the policy stops the request there: one dispatch before
GetSessionswould have found the primary pool empty and handed the request to the backup tier. The caller gets an error in about 20 ms while a healthy backup sits unused: HTTP 502 for a refused connection and the node's own 503 for a 503 answer under the MAG-4218 status rule, and HTTP 500 on builds before it, which is what the ticket measured on v1.5.6. Every tenant runs the default limit.RelaySenderInf.HasUntriedBackup: the server answers, throughConsumerSessionManager.HasEligibleBackup, whether a backup this request has not been sent to could still serve it (not blocked this epoch, not used or unwanted under the request's router key, serving its addon and extensions).DecisionInput.UntriedBackupRemainson the gotResults path only; a hedge never asks.Decidestep 5 stops on the error tolerance only when it is false.HasEligibleBackupread one list,eligibleBackupAddressesLocked, so the policy's view of the tier cannot drift from what selection does.--set-relay-retry-limithelp text anddocs/ERROR-REGISTRY-DESIGN.mdsay so.Evidence
TestBackupServesWhenEveryPrimaryFailsAtTheRetryLimit_MAG3978drivesSendParsedRelaythrough a real session manager over local upstreams at the default limit, for both failure shapes the ticket measured (closed port, 503 at once). Withmain'spolicy.gooverlaid viago test -overlay, every fix case fails and the control passes. Every case pins the caller's status.policy.go)ErrorToleranceExceeded(502 for a closed port, 503 for down)Lava-Provider-Address: primary,primary,primary,backup-1,Lava-Retries: 3Lava-Retries: 5Lava-Retries: 4Lava-Retries: 2, the fourth never askedThe first version of this description said 500 for the failed cases. That is the pre-MAG-4218 status; the test now asserts the real ones.
Unit tests pin the policy rule (
policy_backup_tier_test.go), the state machine's hand-off (backup_tier_decision_input_test.go) and the session manager's eligibility against its own fallback (has_eligible_backup_test.go).Measured on the local k3d cluster (verify-bug-fix run, 2026-10-28)
Both builds were read off the pods:
mainatbb43ae96(imagev1.6.4-9-gbb43ae9) and this branch at315e843d(imagev1.6.4-10-g315e843). The two new methods count 0 times in the first binary and 2 and 3 times in the second;runtime.maincounts 4 on both and a made-up name 0. Both routers reportRelayRetryLimit2. Three attempts per build, one request per attempt.test_the_router_stops_at_its_retry_limit_when_every_primary_fails_at_once_and_asks_no_backup, cases down, rate_limit, node_error, port_closed, oneth-failover-mixed-sim-routerLava-Retries: 2, no backup, error in 18 to 20 msLava-Retries: 3, the simulator holds a row for the backupeth-failover-sim-routerLava-Retries: 2Lava-Retries: 5No automation test belongs to MAG-3978 any more (0 xfail marks name it), so the run reached no verdict and changed no marker. The failing rows are the suite asserting the ruling, not a router defect. If the owners accept this PR, those tests have to be rewritten to expect the backup; an xfail marker would be the wrong change.
Checks
go build -buildvcs=false ./...,go vet ./...clean.go test -count=1onrelaypolicy,relaycore,lavasession(76 s) andrpcsmartrouter(251 s): all ok.golangci-lint runwith a fresh cache: no findings in the touched files. Two pre-existing findings onmain(health_check_crafted_relay_test.goforcetypeassert,relay_health_signal_test.gogofumpt) are not from this branch.Two decisions for the owners, recorded in the tests on purpose
Not in this PR
The nightly comments of 10-27 and 10-28 on the ticket describe a different defect: gRPC answers
UNKNOWNwhere the tests expectUNAVAILABLEwhen every port is closed (protocol/chainlib/grpc.go:436, after MAG-4625ec999d5). MAG-4612 lists it as out of scope; it needs its own ticket.🤖 Generated with Claude Code