Repository navigation
fix(metrics): report the providers that are actually blocked - #332
Conversation
smartrouter_csm_blocked_providers read 0 while providers were blocked and
serving no traffic. An all-providers-down outage looked identical to a healthy
router on every dashboard and alert built on it.
Observed on a live router with two primaries blocked:
/debug/provider-routing
ValidAddresses: []
CurrentlyBlockedProviderAddresses: ["primary-a","primary-b"]
/metrics
smartrouter_csm_blocked_providers{...} 0
The gauge published len(previousEpochBlockedProviders) - the cross-epoch
carry-over set, which UpdateAllProviders fills at an epoch boundary and
checkAndUnblockHealthyReBlockedProviders empties moments later, so it is
near-always zero. The standing block lives in currentlyBlockedProviderAddresses,
which is what /debug/provider-routing correctly reports.
The gauge now publishes that. The previous-epoch size keeps its own signal under
smartrouter_csm_previous_epoch_blocked_providers, so the /debug/reset-all
post-condition it was built for (MAG-1762) loses nothing. That post-condition
still holds on the corrected gauge too: ResetTransientFailureState deliberately
preserves the standing block, but /debug/reset-all also calls
ResetBlockedProviders, which clears it and republishes.
Also implements SetBlockedProvider on SmartRouterMetricsManager. It was an empty
function body, so the three call sites in blockProvider and the restore paths
published nothing and the smart router had no per-provider blocked signal at
all - only the (broken) aggregate. It now writes
smartrouter_csm_provider_blocked{provider,provider_endpoint}, so a dashboard can
show which provider went out rather than only how many.
Tests block a provider through blockProvider rather than by writing the field,
and fail on the old code with expected 1 / actual 0 - the production symptom.
Found while integration-testing MAG-2599 (#328); pre-existing, unrelated to it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
avitenzer
left a comment
There was a problem hiding this comment.
Overview
smartrouter_csm_blocked_providers published len(previousEpochBlockedProviders) — a set that UpdateAllProviders fills at an epoch boundary and the re-block pass drains moments later — so it read 0 through outages. This PR repoints it at currentlyBlockedProviderAddresses, moves the old value to smartrouter_csm_previous_epoch_blocked_providers, and implements the previously-empty SetBlockedProvider as a per-provider smartrouter_csm_provider_blocked gauge.
The diagnosis is correct and the core fix is right. Verified locally: build clean, go vet clean, gofmt clean on changed files, protocol/metrics green, and the four new tests pass. The split preserves the MAG-1762 signal, both reads happen under one RLock, nil-guards match the sibling setters, and the interface addition is covered on NoOpConsumerMetrics.
Two things I'd fix before merge.
1. The new per-provider gauge goes stale — and contradicts the aggregate it complements ⚠️ blocking
setValidAddressesToDefaultValue (protocol/lavasession/consumer_session_manager.go:791-794) empties currentlyBlockedProviderAddresses wholesale and emits no SetBlockedProvider(…, false). It's reached from:
resetValidAddresses(:860) ←releaseBlockedProvidersIfPoolEmpty(:966) — theNO VALID PROVIDERS - TRIGGERING RESETpath, i.e. exactly the all-providers-down scenario this PR exists to make visibleUpdateAllProviders(:405), where the compensatingResetBlockedProvidersMetrics(:433) is still an empty stub onSmartRouterMetricsManager(protocol/metrics/smartrouter_metrics_manager.go:1539)
Verified with a throwaway test (since removed): block all 10 providers, then resetValidAddresses("", nil) →
all blocked: aggregate=10 per-provider={provider0..provider9: true}
after reset: aggregate=0 providers still reported blocked=1: 10
Those ten series stay at 1 until that provider is individually blocked-then-restored or /debug/reset-all runs. A provider that trips once, gets cleared by the pool-empty reset, and then serves fine forever reads blocked=1 permanently. Worse at an epoch flip: a blocked provider absent from the new pairing list leaves a phantom series that nothing can ever clear.
So sum(smartrouter_csm_provider_blocked) and smartrouter_csm_blocked_providers will disagree on the dashboard — the same class of bug this PR is fixing, in the metric it adds.
Fix: implement ResetBlockedProvidersMetrics (it already takes providerAddressToEndpoint — it was built for precisely this) and emit false for the addresses drained in setValidAddressesToDefaultValue.
2. TestSetBlockedProvider_IsNotAStub passes on the old code ⚠️
require.NotPanics around a call whose old body was {} — an empty body doesn't panic either. Same signature, so it compiles and passes unchanged on main. The test asserts nothing about the value it exists to prove.
It also sits in protocol/lavasession while testing protocol/metrics, and calls the real constructor — registering the entire production metric set on the global default registry as a side effect of the lavasession test binary. The house pattern (protocol/metrics/request_group_metrics_test.go:14, used by five sibling test files) builds a manager literal with test-scoped names and asserts with testutil.ToFloat64:
// protocol/metrics/blocked_provider_metric_test.go
func TestSetBlockedProvider_PublishesBothDirections(t *testing.T) {
m := &SmartRouterMetricsManager{csmProviderBlocked: prometheus.NewGaugeVec(
prometheus.GaugeOpts{Name: "t_sr_provider_blocked"},
[]string{"spec", "apiInterface", "provider", "provider_endpoint"})}
g := m.csmProviderBlocked.WithLabelValues("LAVA", "tendermintrpc", "p1", "http://127.0.0.1:1")
m.SetBlockedProvider("LAVA", "tendermintrpc", "p1", "http://127.0.0.1:1", true)
require.Equal(t, float64(1), testutil.ToFloat64(g))
m.SetBlockedProvider("LAVA", "tendermintrpc", "p1", "http://127.0.0.1:1", false)
require.Equal(t, float64(0), testutil.ToFloat64(g))
}That version fails on main. The t.Skip guard is also dead — NewSmartRouterMetricsManager returns nil only for NetworkAddress == DisabledFlagOption; {} gives register-only mode.
The other three tests are good — they drive blockProvider rather than writing the field, and they do fail on old code.
Smaller points
- "Non-zero means degraded" holds; "zero means healthy" does not. During a sustained outage the pool-empty reset drains the blocked list, so the corrected gauge sawtooths 0↔N against a 1s publisher tick.
> 0at instant is fine, but afor: 30salert can miss. Since METRICS.md now tells operators to build on this, worth adding: alert onmax_over_time(smartrouter_csm_blocked_providers[5m]) > 0. - Label naming.
providerandprovider_endpointare one-offs. The repo usesprovider_address(3 metrics, and throughout METRICS.md) andendpoint_id(21 uses). Renaming now is free; after release it isn't. provider_endpointusesEndpoints[0].NetworkAddresson both the block and unblock paths. If that changes between the two, thefalselands on a new series and the old one stays at1. Cheap insurance: drop the label, or key the reset off the provider address alone.docs/METRICS.mdhardcodes "Before v1.4.1" twice. CHANGELOG/versioning is auto-generated on tag push (.github/workflows/release.yml), so the number isn't knowable here — if this ships in 1.5.0 the doc is wrong. Suggest "previously" instead.- Test readability nit.
blockOneProvider(t, csm, csm.validAddresses[0])twice reads as blocking the same provider twice; it works only because the slice shifts underneath (it actually blocksprovider6thenprovider9). Capture two addresses up front.
On the two open questions
Repointing the name rather than renaming — I'd keep your choice. I checked the consumers: no dashboard panel or alert reads it. The only external reference is a recording rule at smart-router-standalone/deploy/customer-dashboard/deploy/kubernetes/smart-router-metric-compatibility.yaml:45, smartrouter_csm_blocked_providers = lava_rpcsmartrouter_csm_blocked_providers. That's a name-alias shim, not a semantics dependency — but note it will keep writing old-semantics data into the corrected series name for any pre-fix router still being scraped. Worth deleting that line or gating it in the same change.
The per-provider gauge — implementing it beats deleting it, and the cardinality (one series per provider per chain) is not a concern for static direct-RPC configs. But it only pays off if #1 is fixed; a gauge that can be permanently wrong in the "provider is down" direction is a worse trap than the empty stub was.
Verdict: approve after #1 and #2. The bug is real, the analysis is right, and the fix is the correct one.
🤖 Generated with Claude Code
Review follow-up on MAG-3106. smartrouter_csm_provider_blocked was published
only on the block/unblock edges, so any path that drains the standing block
wholesale left every series it emptied stuck at 1.
setValidAddressesToDefaultValue empties currentlyBlockedProviderAddresses in one
move and publishes nothing per provider. It runs on the pool-empty release in
releaseBlockedProvidersIfPoolEmpty - the last resort of the failover cascade,
i.e. exactly the all-providers-down case this metric exists for - and on every
epoch transition. The unblock path only publishes for addresses it finds in the
block list, so after a drain nothing could ever clear those series:
all blocked: aggregate=10 per-provider={provider0..provider9: 1}
after release: aggregate=0 per-provider={provider0..provider9: 1}
The aggregate and the per-provider gauge then disagreed on the dashboard, and a
provider that tripped once and served fine forever after read blocked=1
permanently - the same class of bug this branch set out to fix.
publishStateSizes now republishes the state of every paired provider on each
tick instead. A level-triggered gauge cannot go stale, cannot race the re-block
pass, and converges within one tick whichever path drained the list.
Also implements ResetBlockedProvidersMetrics, which was the compensating stub:
UpdateAllProviders already calls it with the new epoch's provider set, and
without it a blocked provider dropped from the pairing left a phantom series
that nothing could clear - a slow cardinality leak that reads as a permanent
outage. It now drops the chain's series and seeds 0 for the new pairing.
Drops provider_endpoint from the labels. Callers pass Endpoints[0].NetworkAddress,
the raw configured node URL, and node URLs embed API keys in their path or query
- docs/METRICS.md states the rule outright and the relay path already labels by
provider name for exactly this reason. It was also unstable: blocking is a
per-provider decision but Endpoints[0] is one arbitrary endpoint of several, so
a change between the block and the unblock orphaned the old series at 1. Renames
the remaining label to provider_address to match the documented glossary.
Replaces TestSetBlockedProvider_IsNotAStub, which asserted only NotPanics and so
passed against the empty body it was named after. The new tests in
protocol/metrics assert values via testutil.ToFloat64 on a bare manager, per the
house pattern, and cover both directions, the reset rebase, chain scoping and
nil safety. Each fails against the code it guards, as does the new
TestPerProviderGauge_ClearedByThePoolEmptyRelease against the drain path.
METRICS.md documents the sawtooth: the release drains the count to 0 during a
sustained total outage, so alerts must use max_over_time rather than the instant
value, and endpoint_serving_tier remains the unambiguous "is it dark" signal.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SpY8wY9rZgmgjufBpy2569
|
@avitenzer thanks — both blocking findings were right, and I reproduced each one before fixing. Addressed in d4217b9. 1. The stale per-provider gaugeFixed, but one level up from where you pointed. Rather than emitting The reason is that the drain sites are not the real problem — edge-triggered publishing is. Any future path that mutates the blocked list without emitting reintroduces exactly this bug, and I also implemented New 2. The vacuous testDeleted. Replaced with On the endpoint label — it is worse than stalenessYou flagged Callers pass
Dropping it also happens to be the cleaner answer to your staleness point: blocking is a per-provider decision, and One correction on the consumer search
There are six references across three repos:
The circuit-breaker page is the one that matters: it is published guidance telling operators to compare this gauge to the pool size, and "equals the pool size" is precisely the state the pool-empty release destroys. That makes your sawtooth point sharper than a dashboard nit — there is already customer-facing documentation built on a property the metric does not have. I have taken your Also taken
Still open
Verification: build, vet, gofmt clean; |
Jira ticket: MAG-3106
The bug
smartrouter_csm_blocked_providersreads 0 while providers are blocked and serving no traffic. An all-providers-down outage looks identical to a healthy router on every dashboard and alert built on this metric.Observed on a live router with two primaries deliberately failed until blocked:
Two providers blocked, gauge reads 0. It never moved off 0 at any point during the session.
Root cause
The gauge published
len(previousEpochBlockedProviders)— the cross-epoch carry-over set, whichUpdateAllProvidersfills at an epoch boundary andcheckAndUnblockHealthyReBlockedProvidersempties moments later. It is near-always zero.The standing block lives in
currentlyBlockedProviderAddresses— the field/debug/provider-routingcorrectly reports.The gauge's
Helpstring was honest about this ("Size of ConsumerSessionManager.previousEpochBlockedProviders"); the name was not. It was built to verify a/debug/reset-allpost-condition (MAG-1762) but is named as the blocked-provider count, so that is how it gets used.The change
1. The gauge reports what its name promises.
smartrouter_csm_blocked_providersnow publisheslen(currentlyBlockedProviderAddresses).2. Nothing is lost. The previous-epoch size moves to its own
smartrouter_csm_previous_epoch_blocked_providers, so the MAG-1762 signal survives under a name that says what it is.The MAG-1762 post-condition also still holds on the corrected gauge:
ResetTransientFailureStatedeliberately preserves the standing block (it is an epoch-boundary operation in lava-pairing mode), but/debug/reset-allalso callsResetBlockedProviders, which clears it and republishes. Verified by test.3.
SetBlockedProvideris no longer an empty stub.blockProviderand the two restore paths call this on every block and unblock, and in the smart router all three published nothing — so there was no per-provider blocked signal at all, only the broken aggregate. It now writessmartrouter_csm_provider_blocked{provider, provider_endpoint}, so a dashboard can show which provider went out, not just how many.Tests
Four, in
protocol/lavasession/blocked_providers_metric_test.go:blockProvider, not by writing the field — shows in the gauge, and scales with the listResetBlockedProviderszeroes the gauge;ResetTransientFailureStatedeliberately does notSetBlockedProviderwrites both directions without panickingThey fail on the old code with
expected: 1, actual: 0— the exact production symptom. There was previously no coverage that could have caught this, because the only test setpreviousEpochBlockedProvidersdirectly and asserted on it.TestPublishStateSizes_PopulateThenResetis updated to the corrected semantics and now asserts on both gauges separately.Verification
go build ./...,go vet ./...clean ·gofmtclean on changed files · fullgo test ./...green.Two decisions worth a reviewer's opinion
I repointed the existing metric name rather than renaming it.
smartrouter_csm_blocked_providerskeeps its name and starts meaning what it says. The alternative — rename it and give the correct value a new name — avoids changing an existing series but leaves the misleading name in place. In-repo the only reference wasdocs/METRICS.md(updated), but if any external dashboard or automation depends on the old semantics, this is the line to push back on.The per-provider gauge adds cardinality — one series per provider per chain. Small for static direct-RPC configs, worth a glance if you run many providers. The alternative was deleting
SetBlockedProviderfrom this manager's interface; I chose to implement it because three call sites already expect it to do something, and a silently-empty interface method is a trap for the next reader.Provenance
Found while integration-testing #328 (MAG-2599). Pre-existing and unrelated to that PR — reproduced identically on
origin/main. That PR's description originally claimed this metric would start reflecting reality during an outage; it has since been corrected to point here.🤖 Generated with Claude Code
https://claude.ai/code/session_01HsVw3GRjz59spS7kWYjcaX