Skip to content

Commit 6a795a9

Browse files
committed
docs(metrics): read upstream state from smartrouter_csm_provider_blocked, not rpc_endpoint_overall_health (MAG-4782)
rpc_endpoint_overall_health dropped to 0 on one failed request while the url stayed in rotation, and came back only when that same url answered. Every reader asked "does the router send this upstream traffic", which smartrouter_csm_provider_blocked answers for primary and backup upstreams and republishes every second. smart-router #625 deletes the health gauge; this page follows: the What to watch row, the PromQL recipe, the suggested alert and the metric table read the blocked gauge, with an "Upstream rotation" subsection that says what takes an upstream out and what brings it back.
1 parent 625b501 commit 6a795a9

1 file changed

Lines changed: 25 additions & 5 deletions

File tree

‎docs/reference/metrics.md‎

Lines changed: 25 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -30,7 +30,7 @@ If you only graph a few things, graph these:
3030
| Is the router up and serving? | `smartrouter_overall_health`, `smartrouter_total_relays_serviced` |
3131
| Are requests failing? | `smartrouter_total_errored`, `smartrouter_errors_total` (by `error_category` / `retryable`) |
3232
| How fast is it? | `smartrouter_end_to_end_latency_milliseconds` (histogram → p50/p95/p99) |
33-
| Is a node degrading? | `rpc_endpoint_overall_health`, `rpc_endpoint_end_to_end_latency_milliseconds`, `rpc_endpoint_latest_block` |
33+
| Is an upstream out of rotation, or degrading? | `smartrouter_csm_provider_blocked` (1 = the router sends it no traffic), `rpc_endpoint_end_to_end_latency_milliseconds`, `rpc_endpoint_latest_block` |
3434
| How much load does the router itself put on your node? | `rpc_endpoint_tracker_requests_total` (by `kind`) - the router's own polling, separate from the relays you sent it; `rpc_endpoint_tracker_gate_skips_total` (by `source`) - the polls it chose not to send |
3535
| How often does the router retry or hedge? | `smartrouter_retries_total`, `smartrouter_hedge_total` |
3636
| Is an upstream rate-limiting you? | `smartrouter_rate_limit_holdoffs_total` (by `provider` / `event`), `smartrouter_rate_limit_holdoff_seconds` |
@@ -64,8 +64,11 @@ sum(rate(smartrouter_retries_total[5m]))
6464
# Non-retryable (terminal) errors only - usually client/chain problems, not infra
6565
sum by (error_name) (rate(smartrouter_errors_total{retryable="false"}[5m]))
6666
67-
# Nodes currently unhealthy
68-
rpc_endpoint_overall_health == 0
67+
# Upstreams the router is sending no traffic to right now
68+
smartrouter_csm_provider_blocked == 1
69+
70+
# The same, for any moment in the last 5 minutes (see "Upstream out of rotation" below)
71+
max_over_time(smartrouter_csm_provider_blocked[5m]) == 1
6972
7073
# Per-node request share (is selection lopsided?)
7174
sum by (provider_address) (rate(smartrouter_requests_total[5m]))
@@ -92,7 +95,7 @@ Starting points. Tune the thresholds to your traffic.
9295
| Router unhealthy | `smartrouter_overall_health == 0 for 1m` | The router reaches no upstream of any chain (see [When readiness changes](#when-readiness-changes)). Upstreams that answer with errors show on the error ratio instead. |
9396
| High error ratio | error-ratio recipe `> 0.05 for 5m` | Something broke upstream or in config. |
9497
| Latency regression | p99 recipe `> 2000 for 10m` | Tail latency degraded. |
95-
| Node down | `rpc_endpoint_overall_health == 0 for 5m` | A configured upstream is out. |
98+
| Upstream out of rotation | `max_over_time(smartrouter_csm_provider_blocked[5m]) == 1` | The router blocked a configured upstream, or none of its urls can take a request right now. Use the window rather than `for`: when every upstream of a chain is out, the router releases them all so it has something to serve from, and the instant value can read `0` in the middle of an outage. |
9699
| Cache cold | hit-ratio recipe `< 0.2 for 30m` | The cache is misconfigured or bypassed. This is a cost and latency risk. |
97100
| RESP cache backend unreachable | `smartrouter_resp_cache_connected == 0 for 5m` | Redis/Valkey is down or rejecting auth. Relays still succeed through the upstreams, so nothing else alerts, but you pay for every read. |
98101
| Post-finality divergence | `increase(smartrouter_cross_validation_mismatch_total{finality="finalized"}[15m]) > 0` | A node disagreed on *finalized* data. This is a high-signal correctness alert. |
@@ -183,6 +186,8 @@ The flag name and the `disabled` sentinel value are defined in
183186
`sum by (...)` over it gives the aggregate. Requests to methods outside the
184187
spec appear as `Default-<method>`.
185188
- **Boolean gauges** encode `1 = true / healthy / present`, `0 = false / unhealthy / absent`.
189+
The exception is `smartrouter_csm_provider_blocked`, where `1` means the upstream is out of
190+
rotation.
186191
- **Protocol version** gauges encode `major*1e6 + minor*1e3 + patch`.
187192
- Metrics are registered with `registerOrReuse`, which returns the already-registered
188193
collector instead of panicking on a duplicate. So re-registration (e.g. across test
@@ -231,7 +236,6 @@ and split into **endpoint-scoped** (`rpc_endpoint_*`) and **router-scoped**
231236
| `rpc_endpoint_total_errored` | Counter | `spec`, `apiInterface`, `endpoint_id`, `function` | Errored relays for this endpoint. |
232237
| `rpc_endpoint_requests_in_flight` | Gauge | `spec`, `apiInterface`, `endpoint_id`, `function` | Relays currently in flight to this endpoint. |
233238
| `rpc_endpoint_end_to_end_latency_milliseconds` | Histogram | `spec`, `apiInterface`, `endpoint_id`, `function` | End-to-end latency per function for this endpoint. |
234-
| `rpc_endpoint_overall_health` | Gauge | `spec`, `apiInterface`, `endpoint_id` | Endpoint health (1 healthy / 0 unhealthy). |
235239
| `rpc_endpoint_overall_health_breakdown` | Gauge | `spec`, `apiInterface` | Aggregate health per chain/interface. |
236240
| `rpc_endpoint_selection_score` | Gauge | `spec`, `apiInterface`, `endpoint_id`, `score_type` | Selection scores by `score_type` (availability / latency / sync / stake / composite). |
237241
| `rpc_endpoint_latest_block` | Gauge | `spec`, `apiInterface`, `endpoint_id` | Latest block reported by the endpoint. |
@@ -286,6 +290,22 @@ Keep two things in mind when you read the chain-tracker counters:
286290
| `smartrouter_latest_block` | Gauge | `spec`, `apiInterface` | Latest block known to the router. |
287291
| `smartrouter_protocol_version` | Gauge | `version` | Encoded protocol version. |
288292

293+
#### Upstream rotation
294+
295+
An upstream is in rotation while the router can send it a request. It leaves rotation when
296+
the router blocks it (every request it was picked for found no url to use) or when none of its
297+
urls can take a request: a url is disabled after too many straight failures and held out while
298+
its latest block stays behind the chain. One url back in rotation brings the upstream back.
299+
300+
| Metric | Type | Labels | Description |
301+
| --- | --- | --- | --- |
302+
| `smartrouter_csm_provider_blocked` | Gauge | `spec`, `apiInterface`, `provider_address` | Whether the router is sending this upstream traffic: `1` = out of rotation (blocked, or no url can take a request right now), `0` = at least one url in rotation. Covers primary and backup upstreams, and is republished every second from the router's own state, so it never sticks. The label is the upstream's configured name, never its url. A rate-limit hold-off does not show here; see [rate-limit hold-off](#rate-limit-hold-off). |
303+
| `smartrouter_csm_blocked_providers` | Gauge | `spec`, `apiInterface` | How many primary upstreams the router has blocked. Non-zero means the chain is degraded. |
304+
| `smartrouter_csm_blocked_backup_providers` | Gauge | `spec`, `apiInterface` | How many backup upstreams the router has blocked. |
305+
306+
An upstream whose urls alone take it out reads `1` on the per-upstream gauge without being
307+
counted by the two totals, so the count of upstreams at `1` can exceed their sum.
308+
289309
#### WebSocket
290310

291311
| Metric | Type | Labels | Description |

0 commit comments

Comments
 (0)