Skip to content

fix(relaypolicy): the error tolerance waits for an untried backup instead of failing one dispatch short of it (MAG-3978) - #588

Closed
avitenzer wants to merge 2 commits into
mainfrom
fix/MAG-3978-backup-tier-before-error-limit
Closed

avitenzer wants to merge 2 commits into
mainfrom
fix/MAG-3978-backup-tier-before-error-limit

Conversation

@avitenzer

@avitenzer avitenzer commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Draft, parked. Not for merge. Tom ruled on MAG-3978 on 2026-10-25 (comment 30709) that the stop at the retry limit is the expected behaviour: at the configured retries the router does not try the remaining nodes, primaries or backups alike, the same decision as MAG-3923. The automation suite encodes that ruling (MAG-4612 and MAG-4604, merged 2026-10-25): 22 tests assert the stop with no backup asked, and they fail on this build, as measured below. This PR is the concrete proposal for revisiting the rule, with its two open semantics decisions at the end. Close it if the owners keep the ruling.

Jira ticket: MAG-3978

What this changes

Three primaries that fail fast are three errors, one past the default retry limit of 2, and the policy stops the request there: one dispatch before GetSessions would have found the primary pool empty and handed the request to the backup tier. The caller gets an error in about 20 ms while a healthy backup sits unused: HTTP 502 for a refused connection and the node's own 503 for a 503 answer under the MAG-4218 status rule, and HTTP 500 on builds before it, which is what the ticket measured on v1.5.6. Every tenant runs the default limit.

  • RelaySenderInf.HasUntriedBackup: the server answers, through ConsumerSessionManager.HasEligibleBackup, whether a backup this request has not been sent to could still serve it (not blocked this epoch, not used or unwanted under the request's router key, serving its addon and extensions).
  • The state machine carries that answer in DecisionInput.UntriedBackupRemains on the gotResults path only; a hedge never asks.
  • Decide step 5 stops on the error tolerance only when it is false.
  • The session manager's backup fallback and HasEligibleBackup read one list, eligibleBackupAddressesLocked, so the policy's view of the tier cannot drift from what selection does.
  • Dispatch order is unchanged: primaries until the pool is exhausted, then backups. The --set-relay-retry-limit help text and docs/ERROR-REGISTRY-DESIGN.md say so.

Evidence

TestBackupServesWhenEveryPrimaryFailsAtTheRetryLimit_MAG3978 drives SendParsedRelay through a real session manager over local upstreams at the default limit, for both failure shapes the ticket measured (closed port, 503 at once). With main's policy.go overlaid via go test -overlay, every fix case fails and the control passes. Every case pins the caller's status.

Case Before (main's policy.go) After
3 primaries fail, 1 backup error after 3 primaries, ErrorToleranceExceeded (502 for a closed port, 503 for down) 200 from the backup, Lava-Provider-Address: primary,primary,primary,backup-1, Lava-Retries: 3
3 primaries + 2 of 3 backups fail error after 3 primaries 200 from the healthy backup, every primary asked first
3 primaries + 3 backups fail error after 3 attempts the same error (502 or 503) after all 6 are asked once, Lava-Retries: 5
4 primaries fail, 1 backup error after 3 primaries 200: all 4 primaries, then the backup, Lava-Retries: 4
control: 4 primaries, no backup error after 3 unchanged: the same error after 3, Lava-Retries: 2, the fourth never asked

The first version of this description said 500 for the failed cases. That is the pre-MAG-4218 status; the test now asserts the real ones.

Unit tests pin the policy rule (policy_backup_tier_test.go), the state machine's hand-off (backup_tier_decision_input_test.go) and the session manager's eligibility against its own fallback (has_eligible_backup_test.go).

Measured on the local k3d cluster (verify-bug-fix run, 2026-10-28)

Both builds were read off the pods: main at bb43ae96 (image v1.6.4-9-gbb43ae9) and this branch at 315e843d (image v1.6.4-10-g315e843). The two new methods count 0 times in the first binary and 2 and 3 times in the second; runtime.main counts 4 on both and a made-up name 0. Both routers report RelayRetryLimit 2. Three attempts per build, one request per attempt.

Automation test main this PR
test_the_router_stops_at_its_retry_limit_when_every_primary_fails_at_once_and_asks_no_backup, cases down, rate_limit, node_error, port_closed, on eth-failover-mixed-sim-router passed 3/3: 3 primaries named, Lava-Retries: 2, no backup, error in 18 to 20 ms failed 3/3, each for one reason only: a backup answered after the 3 primaries, Lava-Retries: 3, the simulator holds a row for the backup
closed port, request pinned to one primary, on eth-failover-sim-router passed 3/3 passed 3/3, same reply
every port closed, 3 primaries and 3 backups passed 3/3: 3 asked, HTTP 502, Lava-Retries: 2 passed 3/3: all 6 asked once, HTTP 502, Lava-Retries: 5

No automation test belongs to MAG-3978 any more (0 xfail marks name it), so the run reached no verdict and changed no marker. The failing rows are the suite asserting the ruling, not a router defect. If the owners accept this PR, those tests have to be rewritten to expect the backup; an xfail marker would be the wrong change.

Checks

  • go build -buildvcs=false ./..., go vet ./... clean.
  • go test -count=1 on relaypolicy, relaycore, lavasession (76 s) and rpcsmartrouter (251 s): all ok.
  • golangci-lint run with a fresh cache: no findings in the touched files. Two pre-existing findings on main (health_check_crafted_relay_test.go forcetypeassert, relay_health_signal_test.go gofumpt) are not from this branch.

Two decisions for the owners, recorded in the tests on purpose

  1. While a backup is untried, the limit does not bound the primaries either: a pool larger than the limit is tried in full before the backup (the 4-primaries case). The alternative, skipping the remaining primaries once the primary budget is spent, needs a prefer-backup dispatch and changes the documented order (backups once the primary pool is exhausted), so it was not taken.
  2. A limit of 0 ("disables retries") still reaches a backup. A special case for 0 is a one-line change if that contract should stay literal.

Not in this PR

The nightly comments of 10-27 and 10-28 on the ticket describe a different defect: gRPC answers UNKNOWN where the tests expect UNAVAILABLE when every port is closed (protocol/chainlib/grpc.go:436, after MAG-4625 ec999d5). MAG-4612 lists it as out of scope; it needs its own ticket.

🤖 Generated with Claude Code

…tead of failing one dispatch short of it (MAG-3978)

Three primaries that fail fast are three errors, one past the default
retry limit of 2, and the policy stopped the request there: one dispatch
before GetSessions would have found the primary pool empty and handed
the request to the backup tier. The caller got HTTP 500 in about 20 ms
while a healthy backup sat unused. Every tenant runs the default limit.

The relay sender now answers whether a backup this request has not been
sent to could still serve it (not blocked this epoch, not used or
unwanted under the request's router key, serving its addon and
extensions), the state machine carries that answer in DecisionInput on
the gotResults path, and the tolerance does not stop a request while it
is true. Dispatch order is unchanged: primaries until the pool is
exhausted, then backups. The session manager's backup fallback and the
new question read one eligibility list, so the two cannot drift.

Recorded on purpose, for the owners' call: while a backup is untried the
limit does not bound the primaries either, so a pool larger than the
limit is tried in full before the backup; and a limit of 0 still reaches
a backup.

Jira ticket: MAG-3978

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Took 1 hour 41 minutes
@avitenzer

Copy link
Copy Markdown
Contributor Author

Parked, not for merge. Tom ruled on MAG-3978 on 2026-10-25 (comment 30709) that the stop at the retry limit is the expected behaviour: at the configured retries the router does not try the remaining nodes, primaries or backups alike, the same decision as MAG-3923. The automation suite already encodes that ruling (MAG-4612 and MAG-4604, merged 2026-10-25): 22 tests now assert the stop with no backup asked, and they would fail on this build.

This PR therefore stays a draft as a concrete proposal for revisiting the rule, with its two open semantics decisions in the body. Close it if the owners keep the ruling.

…der is exhausted (MAG-3978)

The end-to-end test asserted an error on the failed paths and a success
on the served ones, but never the status. The verify-bug-fix run on the
local k3d cluster read HTTP 502 for a closed port on both builds, while
the pull request described 500, which is what the ticket measured before
the MAG-4218 status rule. Every case now pins the status: 200 when a
backup serves, 502 when no node answered (a refused connection), and the
node's own 503 when it answered 503. The exhaustive and control cases run
for both failure shapes and read the dispatch list and the retry count
from the failed reply, which carries them.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Took 15 minutes
@avitenzer

Copy link
Copy Markdown
Contributor Author

Closed unmerged. The router owner's 2026-10-25 ruling on MAG-3978 stands: the stop at the retry limit is intended, and MAG-4612 put that rule into the automation suite. The ticket is cancelled as working as designed.

The branch stays as the record of the alternative. The description holds the design, the unit and k3d evidence (a backup answers where main stops, measured twice on 2026-10-28), and the two open semantics questions, should the rule ever be revisited.

@avitenzer avitenzer closed this Oct 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant