You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 0f8512a
Browse filesBrowse the repository at this point in the historyBrowse files
docs(api): lava-relay-timeout is bounded between 300ms and the call's own budget (MAG-3600)
smart-router#427 stops using the lava-relay-timeout header verbatim. A
value under 300ms is raised to 300ms (still below --min-relay-timeout, so
callers can hedge sooner than the floor), and a value above the
request's own budget, the budget it gets with no header, is held to that
budget unless the operator sets the new --max-caller-relay-timeout. A
value at or above the own budget turns hedging off for that request.
The reply carries Lava-Relay-Timeout-Applied with the window used.
directives.md and failover/timeout.md promised "verbatim" and that 100ms
yields 100ms; both now describe the bounds, the hedging trade-off and the
reply header. cli.md gains the new flag. retry.md and failover/index.md
mention the header only generically and stay correct.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Copy file name to clipboardExpand all lines: docs/api/directives.md
+11-2Lines changed: 11 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -71,9 +71,17 @@ curl -X POST http://127.0.0.1:3360 \
71
71
lava-relay-timeout: 12s
72
72
```
73
73
74
-
Sets the timeout for each upstream attempt of this request. Format: any Go duration string (`500ms`, `5s`, `1m30s`). The value is used **verbatim** — unlike the server-side default it is *not* clamped to the `--min-relay-timeout` floor, so a client can ask for less. A zero or negative value, like one that does not parse, is ignored: the request keeps the router's own timeout.
74
+
Sets the timeout for each upstream attempt of this request: how long the router waits on one node before it also tries the next. Format: any Go duration string (`500ms`, `5s`, `1m30s`). The router bounds the value:
75
75
76
-
**When to use:** known-slow methods (`debug_traceTransaction` on a deep block), or known-fast methods you don't want to wait for.
76
+
- A zero or negative value, like one that does not parse, is ignored: the request keeps the router's own timeout.
77
+
- A value under `300ms` is raised to `300ms`. It is *not* clamped to the `--min-relay-timeout` floor, so a client can still ask for less than that floor.
78
+
- A value above the request's own budget is held to that budget. The own budget is what the request gets with no header: `--default-processing-timeout`, doubled for methods of 50 CU or more, and six times it for hanging, stateful or 100+ CU methods; a method whose own timeout is longer still keeps it (Bitcoin's `sendrawtransaction` gets about 20 minutes). An operator can let clients go further, up to `--max-caller-relay-timeout` (default `0`: no extension).
79
+
80
+
A value at or above the request's own budget turns hedging off for that request: the attempt timeout is also the hedge interval, so the two become equal. Retrying after a failed attempt still works, but a node that goes quiet holds the request for the whole budget.
81
+
82
+
The response carries [`Lava-Relay-Timeout-Applied`](#response-headers) with the timeout the router actually used.
83
+
84
+
**When to use:** known-fast methods you don't want to wait for (hedge sooner), or known-slow methods (`debug_traceTransaction` on a deep block) within their own budget.
77
85
78
86
## Enable debug logging for one request
79
87
@@ -183,6 +191,7 @@ Smart Router annotates every response with metadata about how the relay was serv
183
191
|`Lava-Reported-Providers`| Nodes reported as misbehaving. |
184
192
|`Smart-Router-Version`| Router build serving the request. |
185
193
|`Lava-Extension-Unavailable`| Comma-separated extensions requested with [`lava-extension`](#override-the-extension) that no node on this router offers. The response was served **without** them. Absent when every requested extension was honoured. |
194
+
|`Lava-Relay-Timeout-Applied`| The per-attempt timeout the router used, as a Go duration, after [bounding `lava-relay-timeout`](#override-the-per-attempt-timeout). Only on a request that sent that header; it carries the router's own timeout when the value was ignored. Absent when the request failed without any reply. |
186
195
|`lava-selection-stats`| Node-selection debug stats — only when the router runs with `--enable-selection-stats`. |
187
196
188
197
When cross-validation runs, the router also returns `lava-cross-validation-status`, `lava-cross-validation-agreeing-providers`, `lava-cross-validation-disagreeing-providers`, and — on failure — `lava-cross-validation-failure-reason` (`no-agreement`, `insufficient-responses`, `diversity-unmet`, `group-quorum-unmet`, or a structural `insufficient-capacity` / `insufficient-groups`).
Copy file name to clipboardExpand all lines: docs/configuration/failover/timeout.md
+17-3Lines changed: 17 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -37,7 +37,20 @@ Clients can override the per-attempt timeout for a specific request:
37
37
lava-relay-timeout: 12s
38
38
```
39
39
40
-
Format: any Go duration (`500ms`, `5s`, `1m30s`). The override is used **verbatim** — unlike the server-derived default it is *not* clamped to the `--min-relay-timeout` floor, so a client can request less than the floor. Use this for known-slow methods (`debug_traceTransaction` on a deep block) without raising the global default.
40
+
Format: any Go duration (`500ms`, `5s`, `1m30s`). The router bounds the override:
41
+
42
+
| Header value | Per-attempt timeout used |
43
+
|---|---|
44
+
| zero, negative, or not a duration | ignored; the server-derived default applies |
45
+
| under `300ms`|`300ms` (*not* the `--min-relay-timeout` floor, so a client can still go below that floor) |
46
+
|`300ms` up to the request's own budget | the value as sent |
47
+
| above the request's own budget | the own budget, or up to `--max-caller-relay-timeout` if the operator set it higher |
48
+
49
+
The request's own budget is the overall timeout it gets with no header: `--default-processing-timeout`, doubled for methods of 50 CU or more, six times it for hanging, stateful or 100+ CU methods, and longer for a method whose own timeout is longer. `--max-caller-relay-timeout` defaults to `0`, so by default the header cannot make the router hold a request any longer than it would without it.
50
+
51
+
A value at or above the own budget turns [hedging](hedge.md) off for that request, since the per-attempt timeout is also the hedge interval. [Retry](retry.md) after a failed attempt is unaffected, but a node that goes quiet holds the request for the whole budget.
52
+
53
+
The response carries `Lava-Relay-Timeout-Applied` with the per-attempt timeout actually used.
41
54
42
55
See [Directives](../../api/directives.md).
43
56
@@ -46,15 +59,16 @@ See [Directives](../../api/directives.md).
46
59
| Symptom | Adjust |
47
60
|---|---|
48
61
| p99 latency dominated by one slow attempt | lower `--min-relay-timeout` (hedge sooner) |
49
-
| Heavy methods (`debug_*`) hedged on every call | raise `lava-relay-timeout` per-request, not the global floor |
62
+
| Heavy methods (`debug_*`) hedged on every call | raise `lava-relay-timeout` per-request (up to the method's own budget), not the global floor; beyond that budget, set `--max-caller-relay-timeout`|
50
63
| Whole-relay timeouts in logs | raise `--default-processing-timeout` or investigate why retries aren't succeeding |
51
64
| Lots of clients hitting timeouts on first attempt | raise `--min-relay-timeout`|
52
65
53
66
## Common pitfalls
54
67
55
68
-**Setting the window ≥ overall.** Doesn't crash. The overall budget is raised to match the window, so the window never passes before the request ends and no hedge is ever sent.
56
69
-**Setting the window below upstream RTT.** Healthy nodes still answer, but every request gets hedged, so you pay for extra upstream calls on every request. If `smartrouter_hedge_total` tracks your request rate on healthy upstreams, raise the floor.
57
-
-**Header override below the floor.** The header is *not* clamped to the floor. If a client sends `lava-relay-timeout: 100ms`, the attempt gets `100ms` even when `--min-relay-timeout` is `1s` — the floor only governs the server-derived default, not an explicit override.
70
+
-**Header override below the floor.** The header is *not* clamped to `--min-relay-timeout`, only to `300ms`. If a client sends `lava-relay-timeout: 100ms`, the attempt gets `300ms` even when `--min-relay-timeout` is `1s` — the flag only governs the server-derived default, not an explicit override.
71
+
-**Header override above the budget.** A client asking for more than the request's own budget gets that budget, unless the operator set `--max-caller-relay-timeout`. Check `Lava-Relay-Timeout-Applied` on the response to see what was used.
Copy file name to clipboardExpand all lines: docs/reference/cli.md
+1Lines changed: 1 addition & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -57,6 +57,7 @@ See [RPC Node selection](../configuration/projects/selection-policies.md).
57
57
|`--set-relay-retry-limit`|`2`| Max total retry attempts across all error types (node + protocol). `0` disables retries. |
58
58
|`--default-processing-timeout`|`30s`| Overall budget for a whole relay (all retries + hedges). |
59
59
|`--min-relay-timeout`|`1s`| Per-attempt window floor: when to try the next node. It does not cancel the attempt in flight. The `lava-relay-timeout` header overrides it. |
60
+
|`--max-caller-relay-timeout`|`0`| Longest a caller's `lava-relay-timeout` header may make the router hold one request. It only lets the header extend a request past the budget it gets with no header, and never shortens one; `0` allows no extension. An extended request does not hedge. |
60
61
|`--max-sessions-per-provider`|`1000`| Max concurrent sessions per node. |
61
62
|`--maximum-streams-per-connection`|`100`| Max parallel streams over a single gRPC connection. |
62
63
|`--disable-batch-request-retry`|`true`| Don't retry, fail over, or hedge JSON-RPC batch requests. A batch can carry a write. See [Batch requests](../configuration/failover/retry.md#batch-requests). |
0 commit comments