Skip to content

Commit e273986

Browse files
Tomelia1999claude
andauthored
docs(failover): the default relay retry limit is 3 (MAG-5182) (#106)
With the new default, one to three primaries that all fail fast still reach the backup; four or more stop after the fourth error. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
1 parent 7b1a6ae commit e273986

4 files changed

Lines changed: 11 additions & 11 deletions

File tree

‎docs/configuration/failover/backup.md‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -50,16 +50,16 @@ A URL is switched off after `--bench-after` consecutive failures (default `50`).
5050

5151
### Errors: the retry limit can end a request before any backup
5252

53-
A request stops when it has seen more than `--set-relay-retry-limit` errors (default `2`), counting errors from every node, rate limits included. The limit does not know a backup is still untried. So when every primary fails at the same time, a backup is reached only if the limit allows as many errors as there are primaries:
53+
A request stops when it has seen more than `--set-relay-retry-limit` errors (default `3`), counting errors from every node, rate limits included. The limit does not know a backup is still untried. So when every primary fails at the same time, a backup is reached only if the limit allows as many errors as there are primaries:
5454

55-
| Primaries | All fail fast (5xx, refused connection, node error), default limit `2` |
55+
| Primaries | All fail fast (5xx, refused connection, node error), default limit `3` |
5656
|---|---|
57-
| 1 or 2 | the backup is asked next and can answer |
58-
| 3 or more | **the request stops after the third error.** No backup is asked, and the caller gets a node's error (see [What the caller gets](#what-the-caller-gets-when-neither-tier-can-serve)) |
57+
| 1 to 3 | the backup is asked next and can answer |
58+
| 4 or more | **the request stops after the fourth error.** No backup is asked, and the caller gets a node's error (see [What the caller gets](#what-the-caller-gets-when-neither-tier-can-serve)) |
5959

6060
This is how the router is designed: the limit bounds how much one request costs. It changes once the failing primaries are blocked. A blocked primary is not selected at all, so later requests go straight to the backups. That takes `--bench-after` failures per URL.
6161

62-
If you run three or more primaries and want an error burst to reach the backups right away, raise `--set-relay-retry-limit` to at least the number of primaries. Each extra retry is one more upstream call.
62+
If you run four or more primaries and want an error burst to reach the backups right away, raise `--set-relay-retry-limit` to at least the number of primaries. Each extra retry is one more upstream call.
6363

6464
A non-retryable error, such as `execution reverted` or a malformed request, ends the request at once on any tier. See [Retry](retry.md#whats-retryable).
6565

@@ -194,7 +194,7 @@ Subscriptions fail over differently from requests:
194194

195195
Put your own nodes under `direct-rpc` and the paid provider under `backup-direct-rpc`, as in the example at the top. Normal traffic, including writes, stays on your nodes. Then check:
196196

197-
- **The retry limit.** With three or more primaries, raise `--set-relay-retry-limit` to the number of primaries if an error burst should reach the paid provider at once. Otherwise it is reached only after the primaries are blocked.
197+
- **The retry limit.** With four or more primaries, raise `--set-relay-retry-limit` to the number of primaries if an error burst should reach the paid provider at once. Otherwise it is reached only after the primaries are blocked.
198198
- **The window.** Keep `N × --min-relay-timeout` below `--default-processing-timeout`, so quiet primaries leave time for a backup.
199199
- **Writes.** Leave `--stateful-to-backup` off unless writes should also go to the paid provider while your nodes are up.
200200
- **Archive and debug.** If the paid provider is also your only archive or debug node, those requests go to it on every call. See the next recipe.

‎docs/configuration/failover/index.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@ Each policy is independent, and several can act during the life of a single rela
1414
| Policy | Trigger | What it does |
1515
|---|---|---|
1616
| [**Primary and backup**](backup.md) | no primary can take the request | sends the attempt to a `backup-direct-rpc` node; the next request starts with the primaries again |
17-
| [**Retry**](retry.md) | the upstream returns a retryable error | moves to a different node; tolerates `--set-relay-retry-limit` errors (default 2) |
17+
| [**Retry**](retry.md) | the upstream returns a retryable error | moves to a different node; tolerates `--set-relay-retry-limit` errors (default 3) |
1818
| [**Hedge**](hedge.md) | the request has not returned within a tick | sends a parallel attempt to another node; the first response wins |
1919
| [**Timeout**](timeout.md) | per-attempt window passes, or overall budget runs out | window: sends a hedge while the first attempt keeps running; budget: ends the relay |
2020
| [**Integrity**](integrity.md) | a node is out of sync | a pre-request lag check skips lagging nodes before they are picked |

‎docs/configuration/failover/retry.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -30,18 +30,18 @@ The router decides archive routing once, before the first attempt, from the bloc
3030

3131
| Limit | Value | Notes |
3232
|---|---|---|
33-
| Error-retry limit | `--set-relay-retry-limit` (default `2`) | Errors tolerated before the relay gives up. `0` turns retries off completely. This is the setting you tune. |
33+
| Error-retry limit | `--set-relay-retry-limit` (default `3`) | Errors tolerated before the relay gives up. `0` turns retries off completely. This is the setting you tune. |
3434
| Hard attempt ceiling | `10` | Hardcoded constant `MaximumNumberOfTickerRelayRetries`. It is an upper bound on *total* attempts, including ticker-driven [hedges](hedge.md). It is separate from the error-retry limit and is not available as a flag. |
3535
| Overall budget | `--default-processing-timeout` (default `30s`) | Ends retries even if attempts remain. |
3636
| Per-attempt window | `--min-relay-timeout` floor, or `lava-relay-timeout` header | When the window passes, the next node is tried in parallel, and the attempt in flight keeps running. See [Timeout](timeout.md). |
3737

3838
The cap you control is `--set-relay-retry-limit`. The error-retry path stops after
39-
that many errors (default 2). The hardcoded `10` is only the ceiling that the
39+
that many errors (default 3). The hardcoded `10` is only the ceiling that the
4040
ticker/hedge path can reach. Most relays stop much sooner.
4141

4242
## Turning retries down or off
4343

44-
Retries are on by default and tolerate `--set-relay-retry-limit` errors (default 2). To change this, set the flag:
44+
Retries are on by default and tolerate `--set-relay-retry-limit` errors (default 3). To change this, set the flag:
4545

4646
- `--set-relay-retry-limit 0` - **disable retries**. The first error is returned immediately.
4747
- `--set-relay-retry-limit 5` - tolerate more errors before giving up.

‎docs/reference/cli.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -56,7 +56,7 @@ See [RPC Node selection](../configuration/projects/selection-policies.md).
5656

5757
| Flag | Default | Description |
5858
| --- | --- | --- |
59-
| `--set-relay-retry-limit` | `2` | Errors a request tolerates, across every node and error type (rate limits included), before it stops. `0` disables retries. The count does not know that a backup is still untried: with more primaries than this limit, a request whose primaries all fail stops before any backup. See [Primary and backup nodes](../configuration/failover/backup.md#errors-the-retry-limit-can-end-a-request-before-any-backup). |
59+
| `--set-relay-retry-limit` | `3` | Errors a request tolerates, across every node and error type (rate limits included), before it stops. `0` disables retries. The count does not know that a backup is still untried: with more primaries than this limit, a request whose primaries all fail stops before any backup. See [Primary and backup nodes](../configuration/failover/backup.md#errors-the-retry-limit-can-end-a-request-before-any-backup). |
6060
| `--stateful-to-backup` | `false` | Also broadcast writes (transaction submission) to the backup tier. A write goes to every node it selects, so this sends **every** write to the backups, including the ones the primaries serve fine. See [Writes and batches](../configuration/failover/backup.md#writes-and-batches). |
6161
| `--bench-after` | `50` | Consecutive failed requests to one URL before the router takes it out of rotation. A success resets the count. A provider with every URL out is blocked, and requests go to the other primaries or the backups. Must be greater than 0. |
6262
| `--read-your-writes-window` | `30s` | How long a written transaction's sender and hash stay tied to the node that accepted it, so the sender's follow-up reads go there first (a backup included). Pod-local, first attempt only. `0` turns it off. |

0 commit comments

Comments
 (0)