Skip to content

Commit 0f8512a

Browse files
avitenzerclaude
authored andcommitted
docs(api): lava-relay-timeout is bounded between 300ms and the call's own budget (MAG-3600)
smart-router#427 stops using the lava-relay-timeout header verbatim. A value under 300ms is raised to 300ms (still below --min-relay-timeout, so callers can hedge sooner than the floor), and a value above the request's own budget, the budget it gets with no header, is held to that budget unless the operator sets the new --max-caller-relay-timeout. A value at or above the own budget turns hedging off for that request. The reply carries Lava-Relay-Timeout-Applied with the window used. directives.md and failover/timeout.md promised "verbatim" and that 100ms yields 100ms; both now describe the bounds, the hedging trade-off and the reply header. cli.md gains the new flag. retry.md and failover/index.md mention the header only generically and stay correct. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
1 parent 9729f25 commit 0f8512a

3 files changed

Lines changed: 29 additions & 5 deletions

File tree

‎docs/api/directives.md‎

Lines changed: 11 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -71,9 +71,17 @@ curl -X POST http://127.0.0.1:3360 \
7171
lava-relay-timeout: 12s
7272
```
7373

74-
Sets the timeout for each upstream attempt of this request. Format: any Go duration string (`500ms`, `5s`, `1m30s`). The value is used **verbatim** — unlike the server-side default it is *not* clamped to the `--min-relay-timeout` floor, so a client can ask for less. A zero or negative value, like one that does not parse, is ignored: the request keeps the router's own timeout.
74+
Sets the timeout for each upstream attempt of this request: how long the router waits on one node before it also tries the next. Format: any Go duration string (`500ms`, `5s`, `1m30s`). The router bounds the value:
7575

76-
**When to use:** known-slow methods (`debug_traceTransaction` on a deep block), or known-fast methods you don't want to wait for.
76+
- A zero or negative value, like one that does not parse, is ignored: the request keeps the router's own timeout.
77+
- A value under `300ms` is raised to `300ms`. It is *not* clamped to the `--min-relay-timeout` floor, so a client can still ask for less than that floor.
78+
- A value above the request's own budget is held to that budget. The own budget is what the request gets with no header: `--default-processing-timeout`, doubled for methods of 50 CU or more, and six times it for hanging, stateful or 100+ CU methods; a method whose own timeout is longer still keeps it (Bitcoin's `sendrawtransaction` gets about 20 minutes). An operator can let clients go further, up to `--max-caller-relay-timeout` (default `0`: no extension).
79+
80+
A value at or above the request's own budget turns hedging off for that request: the attempt timeout is also the hedge interval, so the two become equal. Retrying after a failed attempt still works, but a node that goes quiet holds the request for the whole budget.
81+
82+
The response carries [`Lava-Relay-Timeout-Applied`](#response-headers) with the timeout the router actually used.
83+
84+
**When to use:** known-fast methods you don't want to wait for (hedge sooner), or known-slow methods (`debug_traceTransaction` on a deep block) within their own budget.
7785

7886
## Enable debug logging for one request
7987

@@ -183,6 +191,7 @@ Smart Router annotates every response with metadata about how the relay was serv
183191
| `Lava-Reported-Providers` | Nodes reported as misbehaving. |
184192
| `Smart-Router-Version` | Router build serving the request. |
185193
| `Lava-Extension-Unavailable` | Comma-separated extensions requested with [`lava-extension`](#override-the-extension) that no node on this router offers. The response was served **without** them. Absent when every requested extension was honoured. |
194+
| `Lava-Relay-Timeout-Applied` | The per-attempt timeout the router used, as a Go duration, after [bounding `lava-relay-timeout`](#override-the-per-attempt-timeout). Only on a request that sent that header; it carries the router's own timeout when the value was ignored. Absent when the request failed without any reply. |
186195
| `lava-selection-stats` | Node-selection debug stats — only when the router runs with `--enable-selection-stats`. |
187196

188197
When cross-validation runs, the router also returns `lava-cross-validation-status`, `lava-cross-validation-agreeing-providers`, `lava-cross-validation-disagreeing-providers`, and — on failure — `lava-cross-validation-failure-reason` (`no-agreement`, `insufficient-responses`, `diversity-unmet`, `group-quorum-unmet`, or a structural `insufficient-capacity` / `insufficient-groups`).

‎docs/configuration/failover/timeout.md‎

Lines changed: 17 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -37,7 +37,20 @@ Clients can override the per-attempt timeout for a specific request:
3737
lava-relay-timeout: 12s
3838
```
3939

40-
Format: any Go duration (`500ms`, `5s`, `1m30s`). The override is used **verbatim** — unlike the server-derived default it is *not* clamped to the `--min-relay-timeout` floor, so a client can request less than the floor. Use this for known-slow methods (`debug_traceTransaction` on a deep block) without raising the global default.
40+
Format: any Go duration (`500ms`, `5s`, `1m30s`). The router bounds the override:
41+
42+
| Header value | Per-attempt timeout used |
43+
|---|---|
44+
| zero, negative, or not a duration | ignored; the server-derived default applies |
45+
| under `300ms` | `300ms` (*not* the `--min-relay-timeout` floor, so a client can still go below that floor) |
46+
| `300ms` up to the request's own budget | the value as sent |
47+
| above the request's own budget | the own budget, or up to `--max-caller-relay-timeout` if the operator set it higher |
48+
49+
The request's own budget is the overall timeout it gets with no header: `--default-processing-timeout`, doubled for methods of 50 CU or more, six times it for hanging, stateful or 100+ CU methods, and longer for a method whose own timeout is longer. `--max-caller-relay-timeout` defaults to `0`, so by default the header cannot make the router hold a request any longer than it would without it.
50+
51+
A value at or above the own budget turns [hedging](hedge.md) off for that request, since the per-attempt timeout is also the hedge interval. [Retry](retry.md) after a failed attempt is unaffected, but a node that goes quiet holds the request for the whole budget.
52+
53+
The response carries `Lava-Relay-Timeout-Applied` with the per-attempt timeout actually used.
4154

4255
See [Directives](../../api/directives.md).
4356

@@ -46,15 +59,16 @@ See [Directives](../../api/directives.md).
4659
| Symptom | Adjust |
4760
|---|---|
4861
| p99 latency dominated by one slow attempt | lower `--min-relay-timeout` (hedge sooner) |
49-
| Heavy methods (`debug_*`) hedged on every call | raise `lava-relay-timeout` per-request, not the global floor |
62+
| Heavy methods (`debug_*`) hedged on every call | raise `lava-relay-timeout` per-request (up to the method's own budget), not the global floor; beyond that budget, set `--max-caller-relay-timeout` |
5063
| Whole-relay timeouts in logs | raise `--default-processing-timeout` or investigate why retries aren't succeeding |
5164
| Lots of clients hitting timeouts on first attempt | raise `--min-relay-timeout` |
5265

5366
## Common pitfalls
5467

5568
- **Setting the window ≥ overall.** Doesn't crash. The overall budget is raised to match the window, so the window never passes before the request ends and no hedge is ever sent.
5669
- **Setting the window below upstream RTT.** Healthy nodes still answer, but every request gets hedged, so you pay for extra upstream calls on every request. If `smartrouter_hedge_total` tracks your request rate on healthy upstreams, raise the floor.
57-
- **Header override below the floor.** The header is *not* clamped to the floor. If a client sends `lava-relay-timeout: 100ms`, the attempt gets `100ms` even when `--min-relay-timeout` is `1s` — the floor only governs the server-derived default, not an explicit override.
70+
- **Header override below the floor.** The header is *not* clamped to `--min-relay-timeout`, only to `300ms`. If a client sends `lava-relay-timeout: 100ms`, the attempt gets `300ms` even when `--min-relay-timeout` is `1s` — the flag only governs the server-derived default, not an explicit override.
71+
- **Header override above the budget.** A client asking for more than the request's own budget gets that budget, unless the operator set `--max-caller-relay-timeout`. Check `Lava-Relay-Timeout-Applied` on the response to see what was used.
5872

5973
## Observability
6074

‎docs/reference/cli.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -57,6 +57,7 @@ See [RPC Node selection](../configuration/projects/selection-policies.md).
5757
| `--set-relay-retry-limit` | `2` | Max total retry attempts across all error types (node + protocol). `0` disables retries. |
5858
| `--default-processing-timeout` | `30s` | Overall budget for a whole relay (all retries + hedges). |
5959
| `--min-relay-timeout` | `1s` | Per-attempt window floor: when to try the next node. It does not cancel the attempt in flight. The `lava-relay-timeout` header overrides it. |
60+
| `--max-caller-relay-timeout` | `0` | Longest a caller's `lava-relay-timeout` header may make the router hold one request. It only lets the header extend a request past the budget it gets with no header, and never shortens one; `0` allows no extension. An extended request does not hedge. |
6061
| `--max-sessions-per-provider` | `1000` | Max concurrent sessions per node. |
6162
| `--maximum-streams-per-connection` | `100` | Max parallel streams over a single gRPC connection. |
6263
| `--disable-batch-request-retry` | `true` | Don't retry, fail over, or hedge JSON-RPC batch requests. A batch can carry a write. See [Batch requests](../configuration/failover/retry.md#batch-requests). |

0 commit comments

Comments
 (0)