You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 6ea0c89
Browse filesBrowse the repository at this point in the historyBrowse files
docs: document the fleet tracker gate behind --shared-state (MAG-2983)
smart-router PR #322 (MAG-2981) makes --shared-state share per-endpoint
chain-tracker poll observations across router replicas through the cache,
so an upstream is polled about once per interval fleet-wide instead of
once per replica, and adds rpc_endpoint_tracker_gate_skips_total{source}.
- cli.md: --shared-state row now names both things it shares; a short
paragraph explains the borrowing rule and its three safety floors; the
Polling relief section gains the flag as the fleet-level lever and
points at the gate-skips counter.
- metrics.md: new counter row, the quick-answer pointer, and a one-line
recipe for confirming the gate is working across replicas.
- cache.md: "Sharing state across routers" lists both shared things.
|`--enable-fork-detection`|`false`| Turn on block-hash polling (reorg detection on upstreams). Off by default — the larger of the two savings. |
77
+
|`--enable-fork-detection`|`false`| Turn on block-hash polling (reorg detection on upstreams). Off by default — the larger of the per-pod savings. |
79
78
|`--chain-tracker-poll-divisor`|`2`| The tracker polls every `avgBlockTime ÷ divisor`. `1` halves the polling rate. Allowed `[1,8]`; out-of-range reverts to the default. |
79
+
|`--shared-state` (with `--cache-be`) |`false`| Share poll observations across router replicas through the cache, so an upstream is polled about once per interval **fleet-wide** rather than once per pod. See [Cache & shared state](#cache-shared-state). |
80
+
81
+
The tracker also skips a poll whenever something else has already kept the upstream's
82
+
tip fresh — the relays it served in the last block time, or (with `--shared-state`) a
83
+
poll another replica made. Skipped ticks show up on
84
+
[`rpc_endpoint_tracker_gate_skips_total`](metrics.md#endpoint-scoped-rpc_endpoint_) by
85
+
`source`, so `requests_total` falling while `gate_skips_total` rises is the relief
86
+
working, not the tracker stalling.
80
87
81
88
## Consistency tuning
82
89
@@ -89,7 +96,7 @@ and confirm what is live via `HashPolling` in `GET /debug/endpoint-state`.
89
96
| Flag | Default | Description |
90
97
| --- | --- | --- |
91
98
|`--cache-be`| — | Address of the cache server (e.g. `127.0.0.1:20100`). In Compose the cache address usually comes from `cache-be:` in the config instead. |
92
-
|`--shared-state`|`false`| Share consistency state across router instances via the cache backend (`cache-be` or `resp-cache`). The per-endpoint chain-tracker poll sharing additionally requires `--cache-be` — see the [RESP backend caveats](../deployment/cache/redis.md#caveats). |
99
+
|`--shared-state`|`false`| Share state across router replicas through the cache backend (`cache-be` or `resp-cache`): the consumer-consistency "seen block", per-endpoint chain-tracker poll observations (so the fleet polls each upstream about once per interval instead of once per replica), and cross-pod sticky sessions. Poll sharing additionally requires `--cache-be` — see the [RESP backend caveats](../deployment/cache/redis.md#caveats). |
93
100
|`--resp-cache-addresses`| — | Comma-separated address(es) of a [RESP-compatible backend](../deployment/cache/redis.md) (Redis/Valkey). Enables the RESP backend, which takes precedence over `cache-be`. Standalone: the node address; sentinel: the sentinel addresses; cluster: the configuration endpoint. |
94
101
|`--resp-cache-topology`|`standalone`|`standalone`, `sentinel`, or `cluster`. |
95
102
|`--secondary-cache-be`| — | Address of an optional read-only [secondary cache](../deployment/cache/secondary.md), queried when the primary produces no hit. |
@@ -101,6 +108,23 @@ and confirm what is live via `HashPolling` in `GET /debug/endpoint-state`.
101
108
the `resp-cache:` YAML block — only addresses and topology have flags. See
| How fast is it? |`smartrouter_end_to_end_latency_milliseconds` (histogram → p50/p95/p99) |
31
31
| Is a node degrading? |`rpc_endpoint_overall_health`, `rpc_endpoint_end_to_end_latency_milliseconds`, `rpc_endpoint_latest_block`|
32
-
| How much load does the router itself put on my node? |`rpc_endpoint_tracker_requests_total` (by `kind`) — the router's own polling, separate from the relays you sent it |
32
+
| How much load does the router itself put on my node? |`rpc_endpoint_tracker_requests_total` (by `kind`) — the router's own polling, separate from the relays you sent it; `rpc_endpoint_tracker_gate_skips_total` (by `source`) — the polls it chose not to send|
33
33
| Is failover working hard? |`smartrouter_retries_total`, `smartrouter_hedge_total`|
34
34
| Is an upstream rate-limiting us? |`smartrouter_rate_limit_holdoffs_total` (by `provider` / `event`), `smartrouter_rate_limit_holdoff_seconds`|
35
35
| Is the cache earning its keep? |`smartrouter_cache_success_total` / `smartrouter_cache_requests_total`|
@@ -208,6 +208,18 @@ They split into **endpoint-scoped** (`rpc_endpoint_*`) and **router-scoped**
208
208
|`rpc_endpoint_fetch_latest_success`| Counter |`spec`, `apiInterface`, `endpoint_id`| New-block **detections** by the chain tracker, not successful requests. |
|`rpc_endpoint_tracker_requests_total`| Counter |`spec`, `apiInterface`, `endpoint_id`, `kind`| Requests the chain tracker actually sent upstream, by `kind` — `latest_block`, or `block_hash` (zero unless [`--enable-fork-detection`](cli.md#polling-relief)). The only metric that measures tracker **request volume**. |
211
+
|`rpc_endpoint_tracker_gate_skips_total`| Counter |`spec`, `apiInterface`, `endpoint_id`, `source`| Poll ticks the tracker skipped because the upstream's tip was already fresh, by `source` — `relay` (served traffic in the last block time) or `peer` (another replica's poll, shared via [`--shared-state`](cli.md#cache-shared-state)). The complement of `requests_total`: together they account for every tick. |
212
+
|`rpc_endpoint_tracker_gate_errors_total`| Counter |`spec`, `apiInterface`, `endpoint_id`, `op`| Failed calls the fleet gate made to the cache's observation store, by `op` — `fetch` or `publish`. A failed fetch falls back to a local poll; a failed publish costs a peer one real poll. Rises when the sidecar predates the observation RPC. |
213
+
214
+
On a multi-replica deployment with `--shared-state`, the fleet-wide
215
+
`sum(rate(rpc_endpoint_tracker_requests_total{kind="latest_block"}[5m]))` for an upstream
216
+
should settle near a single replica's worth, and `source="peer"` skips should be non-zero on
217
+
every replica. `peer` stuck at zero with the flag on means one of three things: the router is
218
+
on the [RESP backend](../deployment/cache/redis.md#caveats), which has no peer gate at all; the
219
+
replicas are not reaching the same cache; or the cache is reached but the calls fail, for
220
+
example a sidecar older than the router. `rpc_endpoint_tracker_gate_errors_total` tells the
221
+
last apart from the other two — a failing store raises it, a fleet with nothing to share
222
+
leaves it flat.
211
223
212
224
Two caveats when reading the chain-tracker counters:
0 commit comments