You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 6a795a9
Browse filesBrowse the repository at this point in the historyBrowse files
docs(metrics): read upstream state from smartrouter_csm_provider_blocked, not rpc_endpoint_overall_health (MAG-4782)
rpc_endpoint_overall_health dropped to 0 on one failed request while the
url stayed in rotation, and came back only when that same url answered.
Every reader asked "does the router send this upstream traffic", which
smartrouter_csm_provider_blocked answers for primary and backup upstreams
and republishes every second. smart-router #625 deletes the health gauge;
this page follows: the What to watch row, the PromQL recipe, the
suggested alert and the metric table read the blocked gauge, with an
"Upstream rotation" subsection that says what takes an upstream out and
what brings it back.
| How fast is it? |`smartrouter_end_to_end_latency_milliseconds` (histogram → p50/p95/p99) |
33
-
| Is a node degrading? |`rpc_endpoint_overall_health`, `rpc_endpoint_end_to_end_latency_milliseconds`, `rpc_endpoint_latest_block`|
33
+
| Is an upstream out of rotation, or degrading? |`smartrouter_csm_provider_blocked` (1 = the router sends it no traffic), `rpc_endpoint_end_to_end_latency_milliseconds`, `rpc_endpoint_latest_block`|
34
34
| How much load does the router itself put on your node? |`rpc_endpoint_tracker_requests_total` (by `kind`) - the router's own polling, separate from the relays you sent it; `rpc_endpoint_tracker_gate_skips_total` (by `source`) - the polls it chose not to send |
35
35
| How often does the router retry or hedge? |`smartrouter_retries_total`, `smartrouter_hedge_total`|
36
36
| Is an upstream rate-limiting you? |`smartrouter_rate_limit_holdoffs_total` (by `provider` / `event`), `smartrouter_rate_limit_holdoff_seconds`|
sum by (provider_address) (rate(smartrouter_requests_total[5m]))
@@ -92,7 +95,7 @@ Starting points. Tune the thresholds to your traffic.
92
95
| Router unhealthy |`smartrouter_overall_health == 0 for 1m`| The router reaches no upstream of any chain (see [When readiness changes](#when-readiness-changes)). Upstreams that answer with errors show on the error ratio instead. |
93
96
| High error ratio | error-ratio recipe `> 0.05 for 5m`| Something broke upstream or in config. |
|Node down |`rpc_endpoint_overall_health == 0 for 5m`|A configured upstreamis out. |
98
+
|Upstream out of rotation |`max_over_time(smartrouter_csm_provider_blocked[5m]) == 1`|The router blocked a configured upstream, or none of its urls can take a request right now. Use the window rather than `for`: when every upstream of a chain is out, the router releases them all so it has something to serve from, and the instant value can read `0` in the middle of an outage. |
96
99
| Cache cold | hit-ratio recipe `< 0.2 for 30m`| The cache is misconfigured or bypassed. This is a cost and latency risk. |
97
100
| RESP cache backend unreachable |`smartrouter_resp_cache_connected == 0 for 5m`| Redis/Valkey is down or rejecting auth. Relays still succeed through the upstreams, so nothing else alerts, but you pay for every read. |
98
101
| Post-finality divergence |`increase(smartrouter_cross_validation_mismatch_total{finality="finalized"}[15m]) > 0`| A node disagreed on *finalized* data. This is a high-signal correctness alert. |
@@ -183,6 +186,8 @@ The flag name and the `disabled` sentinel value are defined in
183
186
`sum by (...)` over it gives the aggregate. Requests to methods outside the
- Metrics are registered with `registerOrReuse`, which returns the already-registered
188
193
collector instead of panicking on a duplicate. So re-registration (e.g. across test
@@ -231,7 +236,6 @@ and split into **endpoint-scoped** (`rpc_endpoint_*`) and **router-scoped**
231
236
|`rpc_endpoint_total_errored`| Counter |`spec`, `apiInterface`, `endpoint_id`, `function`| Errored relays for this endpoint. |
232
237
|`rpc_endpoint_requests_in_flight`| Gauge |`spec`, `apiInterface`, `endpoint_id`, `function`| Relays currently in flight to this endpoint. |
233
238
|`rpc_endpoint_end_to_end_latency_milliseconds`| Histogram |`spec`, `apiInterface`, `endpoint_id`, `function`| End-to-end latency per function for this endpoint. |
An upstream is in rotation while the router can send it a request. It leaves rotation when
296
+
the router blocks it (every request it was picked for found no url to use) or when none of its
297
+
urls can take a request: a url is disabled after too many straight failures and held out while
298
+
its latest block stays behind the chain. One url back in rotation brings the upstream back.
299
+
300
+
| Metric | Type | Labels | Description |
301
+
| --- | --- | --- | --- |
302
+
|`smartrouter_csm_provider_blocked`| Gauge |`spec`, `apiInterface`, `provider_address`| Whether the router is sending this upstream traffic: `1` = out of rotation (blocked, or no url can take a request right now), `0` = at least one url in rotation. Covers primary and backup upstreams, and is republished every second from the router's own state, so it never sticks. The label is the upstream's configured name, never its url. A rate-limit hold-off does not show here; see [rate-limit hold-off](#rate-limit-hold-off). |
303
+
|`smartrouter_csm_blocked_providers`| Gauge |`spec`, `apiInterface`| How many primary upstreams the router has blocked. Non-zero means the chain is degraded. |
304
+
|`smartrouter_csm_blocked_backup_providers`| Gauge |`spec`, `apiInterface`| How many backup upstreams the router has blocked. |
305
+
306
+
An upstream whose urls alone take it out reads `1` on the per-upstream gauge without being
307
+
counted by the two totals, so the count of upstreams at `1` can exceed their sum.
0 commit comments