You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit e273986
Browse filesBrowse the repository at this point in the historyBrowse files
docs(failover): the default relay retry limit is 3 (MAG-5182) (#106)
With the new default, one to three primaries that all fail fast still
reach the backup; four or more stop after the fourth error.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Copy file name to clipboardExpand all lines: docs/configuration/failover/backup.md
+6-6Lines changed: 6 additions & 6 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -50,16 +50,16 @@ A URL is switched off after `--bench-after` consecutive failures (default `50`).
50
50
51
51
### Errors: the retry limit can end a request before any backup
52
52
53
-
A request stops when it has seen more than `--set-relay-retry-limit` errors (default `2`), counting errors from every node, rate limits included. The limit does not know a backup is still untried. So when every primary fails at the same time, a backup is reached only if the limit allows as many errors as there are primaries:
53
+
A request stops when it has seen more than `--set-relay-retry-limit` errors (default `3`), counting errors from every node, rate limits included. The limit does not know a backup is still untried. So when every primary fails at the same time, a backup is reached only if the limit allows as many errors as there are primaries:
54
54
55
-
| Primaries | All fail fast (5xx, refused connection, node error), default limit `2` |
55
+
| Primaries | All fail fast (5xx, refused connection, node error), default limit `3` |
56
56
|---|---|
57
-
| 1 or 2 | the backup is asked next and can answer |
58
-
| 3 or more | **the request stops after the third error.** No backup is asked, and the caller gets a node's error (see [What the caller gets](#what-the-caller-gets-when-neither-tier-can-serve)) |
57
+
| 1 to 3 | the backup is asked next and can answer |
58
+
| 4 or more | **the request stops after the fourth error.** No backup is asked, and the caller gets a node's error (see [What the caller gets](#what-the-caller-gets-when-neither-tier-can-serve)) |
59
59
60
60
This is how the router is designed: the limit bounds how much one request costs. It changes once the failing primaries are blocked. A blocked primary is not selected at all, so later requests go straight to the backups. That takes `--bench-after` failures per URL.
61
61
62
-
If you run three or more primaries and want an error burst to reach the backups right away, raise `--set-relay-retry-limit` to at least the number of primaries. Each extra retry is one more upstream call.
62
+
If you run four or more primaries and want an error burst to reach the backups right away, raise `--set-relay-retry-limit` to at least the number of primaries. Each extra retry is one more upstream call.
63
63
64
64
A non-retryable error, such as `execution reverted` or a malformed request, ends the request at once on any tier. See [Retry](retry.md#whats-retryable).
65
65
@@ -194,7 +194,7 @@ Subscriptions fail over differently from requests:
194
194
195
195
Put your own nodes under `direct-rpc` and the paid provider under `backup-direct-rpc`, as in the example at the top. Normal traffic, including writes, stays on your nodes. Then check:
196
196
197
-
- **The retry limit.** With three or more primaries, raise `--set-relay-retry-limit` to the number of primaries if an error burst should reach the paid provider at once. Otherwise it is reached only after the primaries are blocked.
197
+
- **The retry limit.** With four or more primaries, raise `--set-relay-retry-limit` to the number of primaries if an error burst should reach the paid provider at once. Otherwise it is reached only after the primaries are blocked.
198
198
- **The window.** Keep `N × --min-relay-timeout` below `--default-processing-timeout`, so quiet primaries leave time for a backup.
199
199
- **Writes.** Leave `--stateful-to-backup` off unless writes should also go to the paid provider while your nodes are up.
200
200
- **Archive and debug.** If the paid provider is also your only archive or debug node, those requests go to it on every call. See the next recipe.
Copy file name to clipboardExpand all lines: docs/configuration/failover/index.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -14,7 +14,7 @@ Each policy is independent, and several can act during the life of a single rela
14
14
| Policy | Trigger | What it does |
15
15
|---|---|---|
16
16
|[**Primary and backup**](backup.md)| no primary can take the request | sends the attempt to a `backup-direct-rpc` node; the next request starts with the primaries again |
17
-
|[**Retry**](retry.md)| the upstream returns a retryable error | moves to a different node; tolerates `--set-relay-retry-limit` errors (default 2) |
17
+
|[**Retry**](retry.md)| the upstream returns a retryable error | moves to a different node; tolerates `--set-relay-retry-limit` errors (default 3) |
18
18
|[**Hedge**](hedge.md)| the request has not returned within a tick | sends a parallel attempt to another node; the first response wins |
19
19
|[**Timeout**](timeout.md)| per-attempt window passes, or overall budget runs out | window: sends a hedge while the first attempt keeps running; budget: ends the relay |
20
20
|[**Integrity**](integrity.md)| a node is out of sync | a pre-request lag check skips lagging nodes before they are picked |
Copy file name to clipboardExpand all lines: docs/configuration/failover/retry.md
+3-3Lines changed: 3 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -30,18 +30,18 @@ The router decides archive routing once, before the first attempt, from the bloc
30
30
31
31
| Limit | Value | Notes |
32
32
|---|---|---|
33
-
| Error-retry limit |`--set-relay-retry-limit` (default `2`) | Errors tolerated before the relay gives up. `0` turns retries off completely. This is the setting you tune. |
33
+
| Error-retry limit |`--set-relay-retry-limit` (default `3`) | Errors tolerated before the relay gives up. `0` turns retries off completely. This is the setting you tune. |
34
34
| Hard attempt ceiling |`10`| Hardcoded constant `MaximumNumberOfTickerRelayRetries`. It is an upper bound on *total* attempts, including ticker-driven [hedges](hedge.md). It is separate from the error-retry limit and is not available as a flag. |
35
35
| Overall budget |`--default-processing-timeout` (default `30s`) | Ends retries even if attempts remain. |
36
36
| Per-attempt window |`--min-relay-timeout` floor, or `lava-relay-timeout` header | When the window passes, the next node is tried in parallel, and the attempt in flight keeps running. See [Timeout](timeout.md). |
37
37
38
38
The cap you control is `--set-relay-retry-limit`. The error-retry path stops after
39
-
that many errors (default 2). The hardcoded `10` is only the ceiling that the
39
+
that many errors (default 3). The hardcoded `10` is only the ceiling that the
40
40
ticker/hedge path can reach. Most relays stop much sooner.
41
41
42
42
## Turning retries down or off
43
43
44
-
Retries are on by default and tolerate `--set-relay-retry-limit` errors (default 2). To change this, set the flag:
44
+
Retries are on by default and tolerate `--set-relay-retry-limit` errors (default 3). To change this, set the flag:
45
45
46
46
-`--set-relay-retry-limit 0` - **disable retries**. The first error is returned immediately.
47
47
-`--set-relay-retry-limit 5` - tolerate more errors before giving up.
Copy file name to clipboardExpand all lines: docs/reference/cli.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -56,7 +56,7 @@ See [RPC Node selection](../configuration/projects/selection-policies.md).
56
56
57
57
| Flag | Default | Description |
58
58
| --- | --- | --- |
59
-
|`--set-relay-retry-limit`|`2`| Errors a request tolerates, across every node and error type (rate limits included), before it stops. `0` disables retries. The count does not know that a backup is still untried: with more primaries than this limit, a request whose primaries all fail stops before any backup. See [Primary and backup nodes](../configuration/failover/backup.md#errors-the-retry-limit-can-end-a-request-before-any-backup). |
59
+
|`--set-relay-retry-limit`|`3`| Errors a request tolerates, across every node and error type (rate limits included), before it stops. `0` disables retries. The count does not know that a backup is still untried: with more primaries than this limit, a request whose primaries all fail stops before any backup. See [Primary and backup nodes](../configuration/failover/backup.md#errors-the-retry-limit-can-end-a-request-before-any-backup). |
60
60
|`--stateful-to-backup`|`false`| Also broadcast writes (transaction submission) to the backup tier. A write goes to every node it selects, so this sends **every** write to the backups, including the ones the primaries serve fine. See [Writes and batches](../configuration/failover/backup.md#writes-and-batches). |
61
61
|`--bench-after`|`50`| Consecutive failed requests to one URL before the router takes it out of rotation. A success resets the count. A provider with every URL out is blocked, and requests go to the other primaries or the backups. Must be greater than 0. |
62
62
|`--read-your-writes-window`|`30s`| How long a written transaction's sender and hash stay tied to the node that accepted it, so the sender's follow-up reads go there first (a backup included). Pod-local, first attempt only. `0` turns it off. |
0 commit comments