Skip to content

Commit 6ea0c89

Browse files
committed
docs: document the fleet tracker gate behind --shared-state (MAG-2983)
smart-router PR #322 (MAG-2981) makes --shared-state share per-endpoint chain-tracker poll observations across router replicas through the cache, so an upstream is polled about once per interval fleet-wide instead of once per replica, and adds rpc_endpoint_tracker_gate_skips_total{source}. - cli.md: --shared-state row now names both things it shares; a short paragraph explains the borrowing rule and its three safety floors; the Polling relief section gains the flag as the fleet-level lever and points at the gate-skips counter. - metrics.md: new counter row, the quick-answer pointer, and a one-line recipe for confirming the gate is working across replicas. - cache.md: "Sharing state across routers" lists both shared things.
1 parent d409013 commit 6ea0c89

3 files changed

Lines changed: 61 additions & 14 deletions

File tree

‎docs/deployment/cache/index.md‎

Lines changed: 18 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -113,13 +113,24 @@ flags as the router.
113113
## Sharing state across routers
114114

115115
When several router instances share one cache, add `--shared-state` to the routers so
116-
they also share consumer-consistency state through it — keeping their "seen block" views
117-
aligned. See the [CLI reference](../../reference/cli.md#cache-shared-state).
118-
119-
The seen-block state travels through either backend. The separate per-endpoint
120-
chain-tracker gate — which lets replicas borrow each other's successful upstream polls
121-
instead of each polling independently — is a sidecar RPC, so it requires `cache-be:`; a
122-
router on the [RESP backend](redis.md#caveats) polls locally and logs a warning.
116+
they also share three things through it:
117+
118+
- **consumer-consistency state** — their "seen block" views stay aligned, so a client
119+
that just read block N from one replica is not served N-1 by another;
120+
- **chain-tracker poll observations** — each replica publishes the upstream polls it
121+
makes and borrows fresh ones from the others, so every upstream is polled about
122+
once per interval fleet-wide instead of once per replica. This is the lever that
123+
stops the router's own polling load from growing with the replica count;
124+
- **sticky sessions** — a `lava-stickiness` id resolves to the same upstream on every
125+
replica. Without the flag stickiness stays pod-local.
126+
127+
The seen-block state and sticky sessions travel through either backend. The chain-tracker
128+
gate is a sidecar RPC, so it requires `cache-be:`; a router on the
129+
[RESP backend](redis.md#caveats) polls locally and logs a warning.
130+
131+
See the [CLI reference](../../reference/cli.md#cache-shared-state) for the safety floors and
132+
[`rpc_endpoint_tracker_gate_skips_total`](../../reference/metrics.md#endpoint-scoped-rpc_endpoint_)
133+
for how to confirm it is working.
123134

124135
## Observability
125136

‎docs/reference/cli.md‎

Lines changed: 30 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -68,15 +68,22 @@ See [Failover & retry](../configuration/failover/index.md).
6868
## Polling relief
6969

7070
Each configured upstream gets its own chain tracker, which polls it for the latest
71-
block independently of the relays you send. These two flags lower that load; both
72-
are process-wide. Watch the effect on
73-
[`rpc_endpoint_tracker_requests_total`](metrics.md#endpoint-scoped-rpc_endpoint_),
74-
and confirm what is live via `HashPolling` in `GET /debug/endpoint-state`.
71+
block independently of the relays you send. These flags lower that load; all are
72+
process-wide. Watch the effect on
73+
[`rpc_endpoint_tracker_requests_total`](metrics.md#endpoint-scoped-rpc_endpoint_).
7574

7675
| Flag | Default | Description |
7776
| --- | --- | --- |
78-
| `--enable-fork-detection` | `false` | Turn on block-hash polling (reorg detection on upstreams). Off by default — the larger of the two savings. |
77+
| `--enable-fork-detection` | `false` | Turn on block-hash polling (reorg detection on upstreams). Off by default — the larger of the per-pod savings. |
7978
| `--chain-tracker-poll-divisor` | `2` | The tracker polls every `avgBlockTime ÷ divisor`. `1` halves the polling rate. Allowed `[1,8]`; out-of-range reverts to the default. |
79+
| `--shared-state` (with `--cache-be`) | `false` | Share poll observations across router replicas through the cache, so an upstream is polled about once per interval **fleet-wide** rather than once per pod. See [Cache & shared state](#cache-shared-state). |
80+
81+
The tracker also skips a poll whenever something else has already kept the upstream's
82+
tip fresh — the relays it served in the last block time, or (with `--shared-state`) a
83+
poll another replica made. Skipped ticks show up on
84+
[`rpc_endpoint_tracker_gate_skips_total`](metrics.md#endpoint-scoped-rpc_endpoint_) by
85+
`source`, so `requests_total` falling while `gate_skips_total` rises is the relief
86+
working, not the tracker stalling.
8087

8188
## Consistency tuning
8289

@@ -89,7 +96,7 @@ and confirm what is live via `HashPolling` in `GET /debug/endpoint-state`.
8996
| Flag | Default | Description |
9097
| --- | --- | --- |
9198
| `--cache-be` | — | Address of the cache server (e.g. `127.0.0.1:20100`). In Compose the cache address usually comes from `cache-be:` in the config instead. |
92-
| `--shared-state` | `false` | Share consistency state across router instances via the cache backend (`cache-be` or `resp-cache`). The per-endpoint chain-tracker poll sharing additionally requires `--cache-be` — see the [RESP backend caveats](../deployment/cache/redis.md#caveats). |
99+
| `--shared-state` | `false` | Share state across router replicas through the cache backend (`cache-be` or `resp-cache`): the consumer-consistency "seen block", per-endpoint chain-tracker poll observations (so the fleet polls each upstream about once per interval instead of once per replica), and cross-pod sticky sessions. Poll sharing additionally requires `--cache-be` — see the [RESP backend caveats](../deployment/cache/redis.md#caveats). |
93100
| `--resp-cache-addresses` | — | Comma-separated address(es) of a [RESP-compatible backend](../deployment/cache/redis.md) (Redis/Valkey). Enables the RESP backend, which takes precedence over `cache-be`. Standalone: the node address; sentinel: the sentinel addresses; cluster: the configuration endpoint. |
94101
| `--resp-cache-topology` | `standalone` | `standalone`, `sentinel`, or `cluster`. |
95102
| `--secondary-cache-be` | — | Address of an optional read-only [secondary cache](../deployment/cache/secondary.md), queried when the primary produces no hit. |
@@ -101,6 +108,23 @@ and confirm what is live via `HashPolling` in `GET /debug/endpoint-state`.
101108
the `resp-cache:` YAML block — only addresses and topology have flags. See
102109
[Redis / Valkey backend](../deployment/cache/redis.md#configuration-reference).
103110

111+
With `--shared-state`, every replica publishes each successful poll of an upstream to
112+
the cache and, before polling, checks whether **another** replica polled that upstream
113+
within the last block time. If one did, the tick is skipped and the peer's block is
114+
adopted as this replica's view of the upstream's tip. Four floors keep it safe:
115+
116+
- a replica never borrows its own observation, so a single replica polls exactly as
117+
without the flag;
118+
- a replica whose own last poll of an upstream failed does not borrow at all — it
119+
returns to full-cadence local polling until it succeeds itself;
120+
- every replica still polls each upstream itself every few ticks (peer skips count
121+
against the same skip budget as relay skips), so a broken path from one pod stays
122+
detectable;
123+
- an upstream that was disabled is only re-enabled by that pod's own successful poll —
124+
a peer observation never touches poll health.
125+
126+
Latency is never shared — a peer's round-trip says nothing about this pod's path.
127+
104128
## WebSocket
105129

106130
| Flag | Default | Description |

‎docs/reference/metrics.md‎

Lines changed: 13 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,7 @@ If you only graph a handful of things, graph these:
2929
| Are requests failing? | `smartrouter_total_errored`, `smartrouter_errors_total` (by `error_category` / `retryable`) |
3030
| How fast is it? | `smartrouter_end_to_end_latency_milliseconds` (histogram → p50/p95/p99) |
3131
| Is a node degrading? | `rpc_endpoint_overall_health`, `rpc_endpoint_end_to_end_latency_milliseconds`, `rpc_endpoint_latest_block` |
32-
| How much load does the router itself put on my node? | `rpc_endpoint_tracker_requests_total` (by `kind`) — the router's own polling, separate from the relays you sent it |
32+
| How much load does the router itself put on my node? | `rpc_endpoint_tracker_requests_total` (by `kind`) — the router's own polling, separate from the relays you sent it; `rpc_endpoint_tracker_gate_skips_total` (by `source`) — the polls it chose not to send |
3333
| Is failover working hard? | `smartrouter_retries_total`, `smartrouter_hedge_total` |
3434
| Is an upstream rate-limiting us? | `smartrouter_rate_limit_holdoffs_total` (by `provider` / `event`), `smartrouter_rate_limit_holdoff_seconds` |
3535
| Is the cache earning its keep? | `smartrouter_cache_success_total` / `smartrouter_cache_requests_total` |
@@ -208,6 +208,18 @@ They split into **endpoint-scoped** (`rpc_endpoint_*`) and **router-scoped**
208208
| `rpc_endpoint_fetch_latest_success` | Counter | `spec`, `apiInterface`, `endpoint_id` | New-block **detections** by the chain tracker, not successful requests. |
209209
| `rpc_endpoint_fetch_block_success` | Counter | `spec`, `apiInterface`, `endpoint_id` | Successful specific-block fetches. |
210210
| `rpc_endpoint_tracker_requests_total` | Counter | `spec`, `apiInterface`, `endpoint_id`, `kind` | Requests the chain tracker actually sent upstream, by `kind` — `latest_block`, or `block_hash` (zero unless [`--enable-fork-detection`](cli.md#polling-relief)). The only metric that measures tracker **request volume**. |
211+
| `rpc_endpoint_tracker_gate_skips_total` | Counter | `spec`, `apiInterface`, `endpoint_id`, `source` | Poll ticks the tracker skipped because the upstream's tip was already fresh, by `source` — `relay` (served traffic in the last block time) or `peer` (another replica's poll, shared via [`--shared-state`](cli.md#cache-shared-state)). The complement of `requests_total`: together they account for every tick. |
212+
| `rpc_endpoint_tracker_gate_errors_total` | Counter | `spec`, `apiInterface`, `endpoint_id`, `op` | Failed calls the fleet gate made to the cache's observation store, by `op` — `fetch` or `publish`. A failed fetch falls back to a local poll; a failed publish costs a peer one real poll. Rises when the sidecar predates the observation RPC. |
213+
214+
On a multi-replica deployment with `--shared-state`, the fleet-wide
215+
`sum(rate(rpc_endpoint_tracker_requests_total{kind="latest_block"}[5m]))` for an upstream
216+
should settle near a single replica's worth, and `source="peer"` skips should be non-zero on
217+
every replica. `peer` stuck at zero with the flag on means one of three things: the router is
218+
on the [RESP backend](../deployment/cache/redis.md#caveats), which has no peer gate at all; the
219+
replicas are not reaching the same cache; or the cache is reached but the calls fail, for
220+
example a sidecar older than the router. `rpc_endpoint_tracker_gate_errors_total` tells the
221+
last apart from the other two — a failing store raises it, a fleet with nothing to share
222+
leaves it flat.
211223

212224
Two caveats when reading the chain-tracker counters:
213225

0 commit comments

Comments
 (0)