Skip to content

Commit 59ffcda

Browse files
avitenzerclaude
authored andcommitted
docs(failover): document the batch no-retry rule and the post-#374 attempt window (MAG-3994)
Batch: a JSON-RPC batch gets no retry, failover or hedge by default (relaypolicy returns Stop "BatchDisabled"), because a batch can carry a write. The rate-limit-only carve-out still moves it to another node. Documents --disable-batch-request-retry, --batch-node-error-on-any and --max-batch-request-size on the retry page; notes it on hedge, index and the CLI reference. Timeout: since smart-router PR #374 the per-attempt window only triggers the next attempt; the attempt in flight runs until the request ends. Rewrites the timeout page, the failover index row, the retry limits row and the --min-relay-timeout CLI entry, and replaces the pitfalls that assumed the window killed the attempt. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
1 parent 70e4524 commit 59ffcda

5 files changed

Lines changed: 46 additions & 14 deletions

File tree

‎docs/configuration/failover/hedge.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,7 @@ Both increment the same retry counter. Both cap at 10 attempts per relay.
3535
| One unresponsive node | yes — but [retry](retry.md) does this too once it errors |
3636
| Tail latency on indexer-style heavy reads | yes — significant p99 improvement |
3737
| Write traffic (`sendRawTransaction`) | covered by **stateful fan-out** instead — write paths are sent to all eligible nodes unconditionally, independent of hedging |
38+
| JSON-RPC batch requests | no — batches aren't hedged by default, because a batch can carry a write. See [Batch requests](retry.md#batch-requests) |
3839

3940
## When to be cautious
4041

‎docs/configuration/failover/index.md‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@ The umbrella for everything Smart Router does when a relay misbehaves: an RPC no
1313
|---|---|---|
1414
| [**Retry**](retry.md) | upstream returns a retryable error | rotates to a different node; tolerates `--set-relay-retry-limit` errors (default 2) |
1515
| [**Hedge**](hedge.md) | request hasn't returned within a tick | fires a parallel attempt to another node; first response wins |
16-
| [**Timeout**](timeout.md) | per-attempt or overall budget exceeded | aborts; lets retry rotate |
16+
| [**Timeout**](timeout.md) | per-attempt window passes, or overall budget runs out | window: sends a hedge while the first attempt keeps running; budget: ends the relay |
1717
| [**Integrity**](integrity.md) | out-of-sync node | pre-request lag check skips lagging nodes before they're picked |
1818
| [**Circuit breaker**](circuit-breaker.md) | the node pool is exhausted | trips the relay early instead of retrying forever |
1919

@@ -23,7 +23,9 @@ For catching *wrong* data by comparing answers across nodes, see [Cross-validati
2323

2424
![Failover pipeline — request flows through integrity, selection, attempt, and branches to success, retry, hedge, or circuit-breaker trip](../../assets/diagrams/failsafe-flow.svg)
2525

26-
The overall timeout (`--default-processing-timeout`) wraps the whole pipeline; if it fires, whatever's most useful is returned. Per-attempt timeouts (`--min-relay-timeout` floor or `lava-relay-timeout` header) bound each individual try.
26+
The overall timeout (`--default-processing-timeout`) wraps the whole pipeline; if it fires, whatever's most useful is returned. The per-attempt window (`--min-relay-timeout` floor or `lava-relay-timeout` header) sets when the next node is tried; it does not cancel the attempt already in flight.
27+
28+
**JSON-RPC batch requests are not retried or failed over by default** — see [Batch requests](retry.md#batch-requests).
2729

2830
The orchestrator is the relay state machine in [`protocol/relaycore/unified_relay_state_machine.go`](https://github.com/Magma-Devs/smart-router/blob/main/protocol/relaycore/unified_relay_state_machine.go). The retryable-vs-terminal classifier is in [`protocol/common/error_registry.go`](https://github.com/Magma-Devs/smart-router/blob/main/protocol/common/error_registry.go) — see [Error codes](../../reference/error-codes.md).
2931

‎docs/configuration/failover/retry.md‎

Lines changed: 19 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,7 @@ Same-response retries are deduplicated: if two nodes return the identical respon
2929
| Error-retry limit | `--set-relay-retry-limit` (default `2`) | Errors tolerated before the relay gives up. `0` disables retries entirely. This is the knob you tune. |
3030
| Hard attempt ceiling | `10` | Hardcoded constant `MaximumNumberOfTickerRelayRetries` — an upper bound on *total* attempts including ticker-driven [hedges](hedge.md), separate from the error-retry limit and not exposed as a flag. |
3131
| Overall budget | `--default-processing-timeout` (default `30s`) | Ends retries even if attempts remain. |
32-
| Per-attempt budget | `--min-relay-timeout` floor, or `lava-relay-timeout` header | Retries get the same per-attempt timeout. |
32+
| Per-attempt window | `--min-relay-timeout` floor, or `lava-relay-timeout` header | When the window passes, the next node is tried in parallel; the attempt in flight keeps running. See [Timeout](timeout.md). |
3333

3434
The cap you actually control is `--set-relay-retry-limit`: the error-retry path stops
3535
after that many errors (default 2). The hardcoded `10` is only the ceiling the
@@ -44,6 +44,24 @@ Retries are on by default, tolerating `--set-relay-retry-limit` errors (default
4444

4545
This is a global startup flag, not a per-relay control — there's no per-request header to disable retry for a single call.
4646

47+
## Batch requests
48+
49+
**By default, a JSON-RPC batch request gets no automatic retry and no failover.** If the node that received the batch returns an error, that result goes back to the client. The router does not resend the batch to another node, either as a retry or as a [hedge](hedge.md).
50+
51+
This is deliberate. A batch can include a write, such as `eth_sendRawTransaction`, next to reads. Resending the batch would resend the write. To retry a failed batch, have the client resend only the sub-requests that are safe to repeat.
52+
53+
There is one exception. If every attempt so far failed only because the node was rate-limited (`429`), nothing ran, so the batch moves to another node like any other rate-limited request.
54+
55+
Three startup flags control batch handling:
56+
57+
| Flag | Default | Effect |
58+
|---|---|---|
59+
| `--disable-batch-request-retry` | `true` | Batches get no retry or failover. Set it to `false` to retry batches like single requests; only do this if your clients never put writes in a batch. |
60+
| `--batch-node-error-on-any` | `false` | Decides when a batch response counts as a node error. By default, a batch is an error only if **no** sub-request succeeded; one success hides failed sub-requests. Set it to `true` to count a batch as an error if **any** sub-request failed. |
61+
| `--max-batch-request-size` | `0` (unlimited) | Largest batch accepted. A larger batch is rejected before it reaches a node, with [`PROTOCOL_BATCH_SIZE_EXCEEDED`](../../reference/error-codes.md) (`1022`). |
62+
63+
With the default `--disable-batch-request-retry true`, `--batch-node-error-on-any` doesn't change whether a batch is retried. It still decides whether the batch response counts as a node error.
64+
4765
## When retries don't help
4866

4967
Some classes of failure look retryable on the wire but won't recover by switching nodes — for example:

‎docs/configuration/failover/timeout.md‎

Lines changed: 20 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -5,19 +5,29 @@ description: "Configure per-request and per-provider timeouts."
55

66
# Timeout
77

8-
Two timeouts apply to every relay: an **overall budget** for the whole request, and a **per-attempt budget** for each upstream try. Hedging and retries operate inside these budgets.
8+
Two timeouts apply to every relay: an **overall budget** for the whole request, and a **per-attempt window** that sets when the next node is tried. Hedging and retries operate inside the budget.
99

1010
## The two timeouts
1111

1212
| Timeout | Default | Set by |
1313
|---|---|---|
1414
| Overall (whole relay) | `30s` | `--default-processing-timeout` CLI flag |
15-
| Per-attempt (each upstream try) | `1s` floor | `--min-relay-timeout` CLI flag, or `lava-relay-timeout` header |
15+
| Per-attempt window | `1s` floor | `--min-relay-timeout` CLI flag, or `lava-relay-timeout` header |
1616

1717
If both are set:
1818

19-
- The **overall** timeout caps the entire relay including all retries and hedges. When it fires, the client gets whatever's most useful (the best partial response, or a timeout error).
20-
- The **per-attempt** timeout aborts a single upstream try and lets [retry](retry.md) move on. It's the more important knob for tuning tail latency.
19+
- The **overall** timeout caps the entire relay including all retries and hedges. When it fires, the client gets whatever's most useful (the best partial response, or a timeout error). It is never shorter than the per-attempt window.
20+
- The **per-attempt window** is how long a node is expected to take. When it passes with no answer, Smart Router sends the same request to another node — a [hedge](hedge.md). The first attempt is **not** cancelled: it keeps running, and whichever attempt answers first wins. It's the more important knob for tuning tail latency.
21+
22+
## What happens when the window passes
23+
24+
The window starts the next attempt; it does not end the current one. An attempt ends only when the request ends: a node answers, the [retry policy](retry.md) stops, or the overall budget runs out.
25+
26+
So a method that is slower than its window still succeeds, as long as it answers within the overall budget. Until September 2026 the window also cancelled the attempt, which meant a method slower than the window could fail on every node even with most of the budget unused. That no longer happens.
27+
28+
A node that is still silent when the overall budget runs out counts as unresponsive and loses score. A node that was cancelled because another node answered first is not penalized.
29+
30+
If a write (for example `eth_sendRawTransaction`) ends with no node having answered, the client gets an HTTP 500 that says the transaction status is unclear, not that it failed. The write may already have been broadcast, so check on-chain before you resubmit it.
2131

2232
## The header override
2333

@@ -35,15 +45,15 @@ See [Directives](../../api/directives.md).
3545

3646
| Symptom | Adjust |
3747
|---|---|
38-
| p99 latency dominated by one slow attempt | lower `--min-relay-timeout` (faster failover) |
39-
| Heavy methods (`debug_*`) always timing out | raise `lava-relay-timeout` per-request, not the global floor |
48+
| p99 latency dominated by one slow attempt | lower `--min-relay-timeout` (hedge sooner) |
49+
| Heavy methods (`debug_*`) hedged on every call | raise `lava-relay-timeout` per-request, not the global floor |
4050
| Whole-relay timeouts in logs | raise `--default-processing-timeout` or investigate why retries aren't succeeding |
4151
| Lots of clients hitting timeouts on first attempt | raise `--min-relay-timeout` |
4252

4353
## Common pitfalls
4454

45-
- **Setting per-attempt > overall.** Doesn't crash, but means the first attempt's timeout is effectively the overall budget and retry never gets to run.
46-
- **Setting per-attempt below upstream RTT.** Every attempt times out before a healthy node can respond. Watch the metrics; if your timeout-rate is near 100% even on healthy upstreams, raise the floor.
55+
- **Setting the window ≥ overall.** Doesn't crash. The overall budget is raised to match the window, so the window never passes before the request ends and no hedge is ever sent.
56+
- **Setting the window below upstream RTT.** Healthy nodes still answer, but every request gets hedged, so you pay for extra upstream calls on every request. If `smartrouter_hedge_total` tracks your request rate on healthy upstreams, raise the floor.
4757
- **Header override below the floor.** The header is *not* clamped to the floor. If a client sends `lava-relay-timeout: 100ms`, the attempt gets `100ms` even when `--min-relay-timeout` is `1s` — the floor only governs the server-derived default, not an explicit override.
4858

4959
## Observability
@@ -54,7 +64,8 @@ There's no dedicated timeout counter — timeouts surface through the latency, r
5464
|---|---|
5565
| `smartrouter_end_to_end_latency_milliseconds` | router-level end-to-end relay latency (histogram; ms buckets to 30 000) |
5666
| `rpc_endpoint_end_to_end_latency_milliseconds` | per-endpoint latency — spot the slow upstream |
57-
| `smartrouter_retries_total` | retries triggered when an attempt aborts (a per-attempt timeout drives failover) |
67+
| `smartrouter_hedge_total` | hedges sent because an attempt outlived its window |
68+
| `smartrouter_retries_total` | retries triggered when an attempt returns an error |
5869
| `smartrouter_total_errored` | relays that ultimately failed, including overall-budget exhaustion |
5970
| Tracing | each attempt span has a duration and an outcome status |
6071

‎docs/reference/cli.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -56,10 +56,10 @@ See [RPC Node selection](../configuration/projects/selection-policies.md).
5656
| --- | --- | --- |
5757
| `--set-relay-retry-limit` | `2` | Max total retry attempts across all error types (node + protocol). `0` disables retries. |
5858
| `--default-processing-timeout` | `30s` | Overall budget for a whole relay (all retries + hedges). |
59-
| `--min-relay-timeout` | `1s` | Per-attempt timeout floor. The `lava-relay-timeout` header can raise it. |
59+
| `--min-relay-timeout` | `1s` | Per-attempt window floor: when to try the next node. It does not cancel the attempt in flight. The `lava-relay-timeout` header overrides it. |
6060
| `--max-sessions-per-provider` | `1000` | Max concurrent sessions per node. |
6161
| `--maximum-streams-per-connection` | `100` | Max parallel streams over a single gRPC connection. |
62-
| `--disable-batch-request-retry` | `true` | Don't retry JSON-RPC batch requests. |
62+
| `--disable-batch-request-retry` | `true` | Don't retry, fail over, or hedge JSON-RPC batch requests. A batch can carry a write. See [Batch requests](../configuration/failover/retry.md#batch-requests). |
6363
| `--batch-node-error-on-any` | `false` | Treat a batch as a node error if **any** sub-request fails (vs. only if **all** do). |
6464
| `--max-batch-request-size` | `0` | Max requests per JSON-RPC batch (`0` = unlimited). |
6565

0 commit comments

Comments
 (0)