You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 59ffcda
Browse filesBrowse the repository at this point in the historyBrowse files
docs(failover): document the batch no-retry rule and the post-#374 attempt window (MAG-3994)
Batch: a JSON-RPC batch gets no retry, failover or hedge by default
(relaypolicy returns Stop "BatchDisabled"), because a batch can carry a
write. The rate-limit-only carve-out still moves it to another node.
Documents --disable-batch-request-retry, --batch-node-error-on-any and
--max-batch-request-size on the retry page; notes it on hedge, index
and the CLI reference.
Timeout: since smart-router PR #374 the per-attempt window only
triggers the next attempt; the attempt in flight runs until the request
ends. Rewrites the timeout page, the failover index row, the retry
limits row and the --min-relay-timeout CLI entry, and replaces the
pitfalls that assumed the window killed the attempt.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Copy file name to clipboardExpand all lines: docs/configuration/failover/hedge.md
+1Lines changed: 1 addition & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -35,6 +35,7 @@ Both increment the same retry counter. Both cap at 10 attempts per relay.
35
35
| One unresponsive node | yes — but [retry](retry.md) does this too once it errors |
36
36
| Tail latency on indexer-style heavy reads | yes — significant p99 improvement |
37
37
| Write traffic (`sendRawTransaction`) | covered by **stateful fan-out** instead — write paths are sent to all eligible nodes unconditionally, independent of hedging |
38
+
| JSON-RPC batch requests | no — batches aren't hedged by default, because a batch can carry a write. See [Batch requests](retry.md#batch-requests)|
|[**Timeout**](timeout.md)| per-attempt window passes, or overall budget runs out | window: sends a hedge while the first attempt keeps running; budget: ends the relay|
17
17
|[**Integrity**](integrity.md)| out-of-sync node | pre-request lag check skips lagging nodes before they're picked |
18
18
|[**Circuit breaker**](circuit-breaker.md)| the node pool is exhausted | trips the relay early instead of retrying forever |
19
19
@@ -23,7 +23,9 @@ For catching *wrong* data by comparing answers across nodes, see [Cross-validati
23
23
24
24

25
25
26
-
The overall timeout (`--default-processing-timeout`) wraps the whole pipeline; if it fires, whatever's most useful is returned. Per-attempt timeouts (`--min-relay-timeout` floor or `lava-relay-timeout` header) bound each individual try.
26
+
The overall timeout (`--default-processing-timeout`) wraps the whole pipeline; if it fires, whatever's most useful is returned. The per-attempt window (`--min-relay-timeout` floor or `lava-relay-timeout` header) sets when the next node is tried; it does not cancel the attempt already in flight.
27
+
28
+
**JSON-RPC batch requests are not retried or failed over by default** — see [Batch requests](retry.md#batch-requests).
27
29
28
30
The orchestrator is the relay state machine in [`protocol/relaycore/unified_relay_state_machine.go`](https://github.com/Magma-Devs/smart-router/blob/main/protocol/relaycore/unified_relay_state_machine.go). The retryable-vs-terminal classifier is in [`protocol/common/error_registry.go`](https://github.com/Magma-Devs/smart-router/blob/main/protocol/common/error_registry.go) — see [Error codes](../../reference/error-codes.md).
Copy file name to clipboardExpand all lines: docs/configuration/failover/retry.md
+19-1Lines changed: 19 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -29,7 +29,7 @@ Same-response retries are deduplicated: if two nodes return the identical respon
29
29
| Error-retry limit |`--set-relay-retry-limit` (default `2`) | Errors tolerated before the relay gives up. `0` disables retries entirely. This is the knob you tune. |
30
30
| Hard attempt ceiling |`10`| Hardcoded constant `MaximumNumberOfTickerRelayRetries` — an upper bound on *total* attempts including ticker-driven [hedges](hedge.md), separate from the error-retry limit and not exposed as a flag. |
31
31
| Overall budget |`--default-processing-timeout` (default `30s`) | Ends retries even if attempts remain. |
32
-
| Per-attempt budget|`--min-relay-timeout` floor, or `lava-relay-timeout` header |Retries get the same per-attempt timeout. |
32
+
| Per-attempt window|`--min-relay-timeout` floor, or `lava-relay-timeout` header |When the window passes, the next node is tried in parallel; the attempt in flight keeps running. See [Timeout](timeout.md). |
33
33
34
34
The cap you actually control is `--set-relay-retry-limit`: the error-retry path stops
35
35
after that many errors (default 2). The hardcoded `10` is only the ceiling the
@@ -44,6 +44,24 @@ Retries are on by default, tolerating `--set-relay-retry-limit` errors (default
44
44
45
45
This is a global startup flag, not a per-relay control — there's no per-request header to disable retry for a single call.
46
46
47
+
## Batch requests
48
+
49
+
**By default, a JSON-RPC batch request gets no automatic retry and no failover.** If the node that received the batch returns an error, that result goes back to the client. The router does not resend the batch to another node, either as a retry or as a [hedge](hedge.md).
50
+
51
+
This is deliberate. A batch can include a write, such as `eth_sendRawTransaction`, next to reads. Resending the batch would resend the write. To retry a failed batch, have the client resend only the sub-requests that are safe to repeat.
52
+
53
+
There is one exception. If every attempt so far failed only because the node was rate-limited (`429`), nothing ran, so the batch moves to another node like any other rate-limited request.
54
+
55
+
Three startup flags control batch handling:
56
+
57
+
| Flag | Default | Effect |
58
+
|---|---|---|
59
+
|`--disable-batch-request-retry`|`true`| Batches get no retry or failover. Set it to `false` to retry batches like single requests; only do this if your clients never put writes in a batch. |
60
+
|`--batch-node-error-on-any`|`false`| Decides when a batch response counts as a node error. By default, a batch is an error only if **no** sub-request succeeded; one success hides failed sub-requests. Set it to `true` to count a batch as an error if **any** sub-request failed. |
61
+
|`--max-batch-request-size`|`0` (unlimited) | Largest batch accepted. A larger batch is rejected before it reaches a node, with [`PROTOCOL_BATCH_SIZE_EXCEEDED`](../../reference/error-codes.md) (`1022`). |
62
+
63
+
With the default `--disable-batch-request-retry true`, `--batch-node-error-on-any` doesn't change whether a batch is retried. It still decides whether the batch response counts as a node error.
64
+
47
65
## When retries don't help
48
66
49
67
Some classes of failure look retryable on the wire but won't recover by switching nodes — for example:
Copy file name to clipboardExpand all lines: docs/configuration/failover/timeout.md
+20-9Lines changed: 20 additions & 9 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -5,19 +5,29 @@ description: "Configure per-request and per-provider timeouts."
5
5
6
6
# Timeout
7
7
8
-
Two timeouts apply to every relay: an **overall budget** for the whole request, and a **per-attempt budget**for each upstream try. Hedging and retries operate inside these budgets.
8
+
Two timeouts apply to every relay: an **overall budget** for the whole request, and a **per-attempt window**that sets when the next node is tried. Hedging and retries operate inside the budget.
9
9
10
10
## The two timeouts
11
11
12
12
| Timeout | Default | Set by |
13
13
|---|---|---|
14
14
| Overall (whole relay) |`30s`|`--default-processing-timeout` CLI flag |
- The **overall** timeout caps the entire relay including all retries and hedges. When it fires, the client gets whatever's most useful (the best partial response, or a timeout error).
20
-
- The **per-attempt** timeout aborts a single upstream try and lets [retry](retry.md) move on. It's the more important knob for tuning tail latency.
19
+
- The **overall** timeout caps the entire relay including all retries and hedges. When it fires, the client gets whatever's most useful (the best partial response, or a timeout error). It is never shorter than the per-attempt window.
20
+
- The **per-attempt window** is how long a node is expected to take. When it passes with no answer, Smart Router sends the same request to another node — a [hedge](hedge.md). The first attempt is **not** cancelled: it keeps running, and whichever attempt answers first wins. It's the more important knob for tuning tail latency.
21
+
22
+
## What happens when the window passes
23
+
24
+
The window starts the next attempt; it does not end the current one. An attempt ends only when the request ends: a node answers, the [retry policy](retry.md) stops, or the overall budget runs out.
25
+
26
+
So a method that is slower than its window still succeeds, as long as it answers within the overall budget. Until September 2026 the window also cancelled the attempt, which meant a method slower than the window could fail on every node even with most of the budget unused. That no longer happens.
27
+
28
+
A node that is still silent when the overall budget runs out counts as unresponsive and loses score. A node that was cancelled because another node answered first is not penalized.
29
+
30
+
If a write (for example `eth_sendRawTransaction`) ends with no node having answered, the client gets an HTTP 500 that says the transaction status is unclear, not that it failed. The write may already have been broadcast, so check on-chain before you resubmit it.
21
31
22
32
## The header override
23
33
@@ -35,15 +45,15 @@ See [Directives](../../api/directives.md).
35
45
36
46
| Symptom | Adjust |
37
47
|---|---|
38
-
| p99 latency dominated by one slow attempt | lower `--min-relay-timeout` (faster failover) |
39
-
| Heavy methods (`debug_*`) always timing out| raise `lava-relay-timeout` per-request, not the global floor |
48
+
| p99 latency dominated by one slow attempt | lower `--min-relay-timeout` (hedge sooner) |
49
+
| Heavy methods (`debug_*`) hedged on every call| raise `lava-relay-timeout` per-request, not the global floor |
40
50
| Whole-relay timeouts in logs | raise `--default-processing-timeout` or investigate why retries aren't succeeding |
41
51
| Lots of clients hitting timeouts on first attempt | raise `--min-relay-timeout`|
42
52
43
53
## Common pitfalls
44
54
45
-
-**Setting per-attempt > overall.** Doesn't crash, but means the first attempt's timeout is effectively the overall budget and retry never gets to run.
46
-
-**Setting per-attempt below upstream RTT.**Every attempt times out before a healthy node can respond. Watch the metrics; if your timeout-rate is near 100% even on healthy upstreams, raise the floor.
55
+
-**Setting the window ≥ overall.** Doesn't crash. The overall budget is raised to match the window, so the window never passes before the request ends and no hedge is ever sent.
56
+
-**Setting the window below upstream RTT.**Healthy nodes still answer, but every request gets hedged, so you pay for extra upstream calls on every request. If `smartrouter_hedge_total` tracks your request rate on healthy upstreams, raise the floor.
47
57
-**Header override below the floor.** The header is *not* clamped to the floor. If a client sends `lava-relay-timeout: 100ms`, the attempt gets `100ms` even when `--min-relay-timeout` is `1s` — the floor only governs the server-derived default, not an explicit override.
48
58
49
59
## Observability
@@ -54,7 +64,8 @@ There's no dedicated timeout counter — timeouts surface through the latency, r
54
64
|---|---|
55
65
|`smartrouter_end_to_end_latency_milliseconds`| router-level end-to-end relay latency (histogram; ms buckets to 30 000) |
56
66
|`rpc_endpoint_end_to_end_latency_milliseconds`| per-endpoint latency — spot the slow upstream |
57
-
|`smartrouter_retries_total`| retries triggered when an attempt aborts (a per-attempt timeout drives failover) |
67
+
|`smartrouter_hedge_total`| hedges sent because an attempt outlived its window |
68
+
|`smartrouter_retries_total`| retries triggered when an attempt returns an error |
58
69
|`smartrouter_total_errored`| relays that ultimately failed, including overall-budget exhaustion |
59
70
| Tracing | each attempt span has a duration and an outcome status |
Copy file name to clipboardExpand all lines: docs/reference/cli.md
+2-2Lines changed: 2 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -56,10 +56,10 @@ See [RPC Node selection](../configuration/projects/selection-policies.md).
56
56
| --- | --- | --- |
57
57
|`--set-relay-retry-limit`|`2`| Max total retry attempts across all error types (node + protocol). `0` disables retries. |
58
58
|`--default-processing-timeout`|`30s`| Overall budget for a whole relay (all retries + hedges). |
59
-
|`--min-relay-timeout`|`1s`| Per-attempt timeout floor. The `lava-relay-timeout` header can raise it. |
59
+
|`--min-relay-timeout`|`1s`| Per-attempt window floor: when to try the next node. It does not cancel the attempt in flight. The `lava-relay-timeout` header overrides it. |
60
60
|`--max-sessions-per-provider`|`1000`| Max concurrent sessions per node. |
61
61
|`--maximum-streams-per-connection`|`100`| Max parallel streams over a single gRPC connection. |
|`--disable-batch-request-retry`|`true`| Don't retry, fail over, or hedge JSON-RPC batch requests. A batch can carry a write. See [Batch requests](../configuration/failover/retry.md#batch-requests). |
63
63
|`--batch-node-error-on-any`|`false`| Treat a batch as a node error if **any** sub-request fails (vs. only if **all** do). |
64
64
|`--max-batch-request-size`|`0`| Max requests per JSON-RPC batch (`0` = unlimited). |
0 commit comments