Skip to content

fix(metrics): report the providers that are actually blocked - #332

Merged
Tomelia1999 merged 3 commits into
mainfrom
fix/mag-blocked-providers-metric
Aug 25, 2026
Merged

Tomelia1999 merged 3 commits into
mainfrom
fix/mag-blocked-providers-metric

Conversation

@Tomelia1999

@Tomelia1999 Tomelia1999 commented Aug 25, 2026 •

Copy link
Copy Markdown
Contributor

Jira ticket: MAG-3106

The bug

smartrouter_csm_blocked_providers reads 0 while providers are blocked and serving no traffic. An all-providers-down outage looks identical to a healthy router on every dashboard and alert built on this metric.

Observed on a live router with two primaries deliberately failed until blocked:

$ curl -s http://127.0.0.1:7910/debug/provider-routing
[{"ValidAddresses":[],
  "CurrentlyBlockedProviderAddresses":["primary-a","primary-b"],
  "BlockedBackupProviders":[]}]

$ curl -s http://127.0.0.1:7810/metrics | grep csm_blocked
smartrouter_csm_blocked_providers{apiInterface="tendermintrpc",spec="LAVA"} 0

Two providers blocked, gauge reads 0. It never moved off 0 at any point during the session.

Root cause

The gauge published len(previousEpochBlockedProviders) — the cross-epoch carry-over set, which UpdateAllProviders fills at an epoch boundary and checkAndUnblockHealthyReBlockedProviders empties moments later. It is near-always zero.

The standing block lives in currentlyBlockedProviderAddresses — the field /debug/provider-routing correctly reports.

The gauge's Help string was honest about this ("Size of ConsumerSessionManager.previousEpochBlockedProviders"); the name was not. It was built to verify a /debug/reset-all post-condition (MAG-1762) but is named as the blocked-provider count, so that is how it gets used.

The change

1. The gauge reports what its name promises. smartrouter_csm_blocked_providers now publishes len(currentlyBlockedProviderAddresses).

2. Nothing is lost. The previous-epoch size moves to its own smartrouter_csm_previous_epoch_blocked_providers, so the MAG-1762 signal survives under a name that says what it is.

The MAG-1762 post-condition also still holds on the corrected gauge: ResetTransientFailureState deliberately preserves the standing block (it is an epoch-boundary operation in lava-pairing mode), but /debug/reset-all also calls ResetBlockedProviders, which clears it and republishes. Verified by test.

3. SetBlockedProvider is no longer an empty stub.

func (m *SmartRouterMetricsManager) SetBlockedProvider(string, string, string, string, bool) {}

blockProvider and the two restore paths call this on every block and unblock, and in the smart router all three published nothing — so there was no per-provider blocked signal at all, only the broken aggregate. It now writes smartrouter_csm_provider_blocked{provider, provider_endpoint}, so a dashboard can show which provider went out, not just how many.

Tests

Four, in protocol/lavasession/blocked_providers_metric_test.go:

  • a real block — driven through blockProvider, not by writing the field — shows in the gauge, and scales with the list
  • blocking does not populate the cross-epoch set (the two can't be confused again)
  • ResetBlockedProviders zeroes the gauge; ResetTransientFailureState deliberately does not
  • SetBlockedProvider writes both directions without panicking

They fail on the old code with expected: 1, actual: 0 — the exact production symptom. There was previously no coverage that could have caught this, because the only test set previousEpochBlockedProviders directly and asserted on it.

TestPublishStateSizes_PopulateThenReset is updated to the corrected semantics and now asserts on both gauges separately.

Verification

go build ./..., go vet ./... clean · gofmt clean on changed files · full go test ./... green.

Two decisions worth a reviewer's opinion

I repointed the existing metric name rather than renaming it. smartrouter_csm_blocked_providers keeps its name and starts meaning what it says. The alternative — rename it and give the correct value a new name — avoids changing an existing series but leaves the misleading name in place. In-repo the only reference was docs/METRICS.md (updated), but if any external dashboard or automation depends on the old semantics, this is the line to push back on.

The per-provider gauge adds cardinality — one series per provider per chain. Small for static direct-RPC configs, worth a glance if you run many providers. The alternative was deleting SetBlockedProvider from this manager's interface; I chose to implement it because three call sites already expect it to do something, and a silently-empty interface method is a trap for the next reader.

Provenance

Found while integration-testing #328 (MAG-2599). Pre-existing and unrelated to that PR — reproduced identically on origin/main. That PR's description originally claimed this metric would start reflecting reality during an outage; it has since been corrected to point here.

🤖 Generated with Claude Code

https://claude.ai/code/session_01HsVw3GRjz59spS7kWYjcaX

smartrouter_csm_blocked_providers read 0 while providers were blocked and
serving no traffic. An all-providers-down outage looked identical to a healthy
router on every dashboard and alert built on it.

Observed on a live router with two primaries blocked:

  /debug/provider-routing
    ValidAddresses: []
    CurrentlyBlockedProviderAddresses: ["primary-a","primary-b"]
  /metrics
    smartrouter_csm_blocked_providers{...} 0

The gauge published len(previousEpochBlockedProviders) - the cross-epoch
carry-over set, which UpdateAllProviders fills at an epoch boundary and
checkAndUnblockHealthyReBlockedProviders empties moments later, so it is
near-always zero. The standing block lives in currentlyBlockedProviderAddresses,
which is what /debug/provider-routing correctly reports.

The gauge now publishes that. The previous-epoch size keeps its own signal under
smartrouter_csm_previous_epoch_blocked_providers, so the /debug/reset-all
post-condition it was built for (MAG-1762) loses nothing. That post-condition
still holds on the corrected gauge too: ResetTransientFailureState deliberately
preserves the standing block, but /debug/reset-all also calls
ResetBlockedProviders, which clears it and republishes.

Also implements SetBlockedProvider on SmartRouterMetricsManager. It was an empty
function body, so the three call sites in blockProvider and the restore paths
published nothing and the smart router had no per-provider blocked signal at
all - only the (broken) aggregate. It now writes
smartrouter_csm_provider_blocked{provider,provider_endpoint}, so a dashboard can
show which provider went out rather than only how many.

Tests block a provider through blockProvider rather than by writing the field,
and fail on the old code with expected 1 / actual 0 - the production symptom.

Found while integration-testing MAG-2599 (#328); pre-existing, unrelated to it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@Tomelia1999
Tomelia1999 requested a review from a team as a code owner August 25, 2026 08:23

@avitenzer avitenzer left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overview

smartrouter_csm_blocked_providers published len(previousEpochBlockedProviders) — a set that UpdateAllProviders fills at an epoch boundary and the re-block pass drains moments later — so it read 0 through outages. This PR repoints it at currentlyBlockedProviderAddresses, moves the old value to smartrouter_csm_previous_epoch_blocked_providers, and implements the previously-empty SetBlockedProvider as a per-provider smartrouter_csm_provider_blocked gauge.

The diagnosis is correct and the core fix is right. Verified locally: build clean, go vet clean, gofmt clean on changed files, protocol/metrics green, and the four new tests pass. The split preserves the MAG-1762 signal, both reads happen under one RLock, nil-guards match the sibling setters, and the interface addition is covered on NoOpConsumerMetrics.

Two things I'd fix before merge.


1. The new per-provider gauge goes stale — and contradicts the aggregate it complements ⚠️ blocking

setValidAddressesToDefaultValue (protocol/lavasession/consumer_session_manager.go:791-794) empties currentlyBlockedProviderAddresses wholesale and emits no SetBlockedProvider(…, false). It's reached from:

  • resetValidAddresses (:860) ← releaseBlockedProvidersIfPoolEmpty (:966) — the NO VALID PROVIDERS - TRIGGERING RESET path, i.e. exactly the all-providers-down scenario this PR exists to make visible
  • UpdateAllProviders (:405), where the compensating ResetBlockedProvidersMetrics (:433) is still an empty stub on SmartRouterMetricsManager (protocol/metrics/smartrouter_metrics_manager.go:1539)

Verified with a throwaway test (since removed): block all 10 providers, then resetValidAddresses("", nil) →

all blocked:   aggregate=10  per-provider={provider0..provider9: true}
after reset:   aggregate=0   providers still reported blocked=1: 10

Those ten series stay at 1 until that provider is individually blocked-then-restored or /debug/reset-all runs. A provider that trips once, gets cleared by the pool-empty reset, and then serves fine forever reads blocked=1 permanently. Worse at an epoch flip: a blocked provider absent from the new pairing list leaves a phantom series that nothing can ever clear.

So sum(smartrouter_csm_provider_blocked) and smartrouter_csm_blocked_providers will disagree on the dashboard — the same class of bug this PR is fixing, in the metric it adds.

Fix: implement ResetBlockedProvidersMetrics (it already takes providerAddressToEndpoint — it was built for precisely this) and emit false for the addresses drained in setValidAddressesToDefaultValue.

2. TestSetBlockedProvider_IsNotAStub passes on the old code ⚠️

require.NotPanics around a call whose old body was {} — an empty body doesn't panic either. Same signature, so it compiles and passes unchanged on main. The test asserts nothing about the value it exists to prove.

It also sits in protocol/lavasession while testing protocol/metrics, and calls the real constructor — registering the entire production metric set on the global default registry as a side effect of the lavasession test binary. The house pattern (protocol/metrics/request_group_metrics_test.go:14, used by five sibling test files) builds a manager literal with test-scoped names and asserts with testutil.ToFloat64:

// protocol/metrics/blocked_provider_metric_test.go
func TestSetBlockedProvider_PublishesBothDirections(t *testing.T) {
	m := &SmartRouterMetricsManager{csmProviderBlocked: prometheus.NewGaugeVec(
		prometheus.GaugeOpts{Name: "t_sr_provider_blocked"},
		[]string{"spec", "apiInterface", "provider", "provider_endpoint"})}
	g := m.csmProviderBlocked.WithLabelValues("LAVA", "tendermintrpc", "p1", "http://127.0.0.1:1")

	m.SetBlockedProvider("LAVA", "tendermintrpc", "p1", "http://127.0.0.1:1", true)
	require.Equal(t, float64(1), testutil.ToFloat64(g))
	m.SetBlockedProvider("LAVA", "tendermintrpc", "p1", "http://127.0.0.1:1", false)
	require.Equal(t, float64(0), testutil.ToFloat64(g))
}

That version fails on main. The t.Skip guard is also dead — NewSmartRouterMetricsManager returns nil only for NetworkAddress == DisabledFlagOption; {} gives register-only mode.

The other three tests are good — they drive blockProvider rather than writing the field, and they do fail on old code.


Smaller points

  • "Non-zero means degraded" holds; "zero means healthy" does not. During a sustained outage the pool-empty reset drains the blocked list, so the corrected gauge sawtooths 0↔N against a 1s publisher tick. > 0 at instant is fine, but a for: 30s alert can miss. Since METRICS.md now tells operators to build on this, worth adding: alert on max_over_time(smartrouter_csm_blocked_providers[5m]) > 0.
  • Label naming. provider and provider_endpoint are one-offs. The repo uses provider_address (3 metrics, and throughout METRICS.md) and endpoint_id (21 uses). Renaming now is free; after release it isn't.
  • provider_endpoint uses Endpoints[0].NetworkAddress on both the block and unblock paths. If that changes between the two, the false lands on a new series and the old one stays at 1. Cheap insurance: drop the label, or key the reset off the provider address alone.
  • docs/METRICS.md hardcodes "Before v1.4.1" twice. CHANGELOG/versioning is auto-generated on tag push (.github/workflows/release.yml), so the number isn't knowable here — if this ships in 1.5.0 the doc is wrong. Suggest "previously" instead.
  • Test readability nit. blockOneProvider(t, csm, csm.validAddresses[0]) twice reads as blocking the same provider twice; it works only because the slice shifts underneath (it actually blocks provider6 then provider9). Capture two addresses up front.

On the two open questions

Repointing the name rather than renaming — I'd keep your choice. I checked the consumers: no dashboard panel or alert reads it. The only external reference is a recording rule at smart-router-standalone/deploy/customer-dashboard/deploy/kubernetes/smart-router-metric-compatibility.yaml:45, smartrouter_csm_blocked_providers = lava_rpcsmartrouter_csm_blocked_providers. That's a name-alias shim, not a semantics dependency — but note it will keep writing old-semantics data into the corrected series name for any pre-fix router still being scraped. Worth deleting that line or gating it in the same change.

The per-provider gauge — implementing it beats deleting it, and the cardinality (one series per provider per chain) is not a concern for static direct-RPC configs. But it only pays off if #1 is fixed; a gauge that can be permanently wrong in the "provider is down" direction is a worse trap than the empty stub was.


Verdict: approve after #1 and #2. The bug is real, the analysis is right, and the fix is the correct one.

🤖 Generated with Claude Code

Review follow-up on MAG-3106. smartrouter_csm_provider_blocked was published
only on the block/unblock edges, so any path that drains the standing block
wholesale left every series it emptied stuck at 1.

setValidAddressesToDefaultValue empties currentlyBlockedProviderAddresses in one
move and publishes nothing per provider. It runs on the pool-empty release in
releaseBlockedProvidersIfPoolEmpty - the last resort of the failover cascade,
i.e. exactly the all-providers-down case this metric exists for - and on every
epoch transition. The unblock path only publishes for addresses it finds in the
block list, so after a drain nothing could ever clear those series:

  all blocked:   aggregate=10  per-provider={provider0..provider9: 1}
  after release: aggregate=0   per-provider={provider0..provider9: 1}

The aggregate and the per-provider gauge then disagreed on the dashboard, and a
provider that tripped once and served fine forever after read blocked=1
permanently - the same class of bug this branch set out to fix.

publishStateSizes now republishes the state of every paired provider on each
tick instead. A level-triggered gauge cannot go stale, cannot race the re-block
pass, and converges within one tick whichever path drained the list.

Also implements ResetBlockedProvidersMetrics, which was the compensating stub:
UpdateAllProviders already calls it with the new epoch's provider set, and
without it a blocked provider dropped from the pairing left a phantom series
that nothing could clear - a slow cardinality leak that reads as a permanent
outage. It now drops the chain's series and seeds 0 for the new pairing.

Drops provider_endpoint from the labels. Callers pass Endpoints[0].NetworkAddress,
the raw configured node URL, and node URLs embed API keys in their path or query
- docs/METRICS.md states the rule outright and the relay path already labels by
provider name for exactly this reason. It was also unstable: blocking is a
per-provider decision but Endpoints[0] is one arbitrary endpoint of several, so
a change between the block and the unblock orphaned the old series at 1. Renames
the remaining label to provider_address to match the documented glossary.

Replaces TestSetBlockedProvider_IsNotAStub, which asserted only NotPanics and so
passed against the empty body it was named after. The new tests in
protocol/metrics assert values via testutil.ToFloat64 on a bare manager, per the
house pattern, and cover both directions, the reset rebase, chain scoping and
nil safety. Each fails against the code it guards, as does the new
TestPerProviderGauge_ClearedByThePoolEmptyRelease against the drain path.

METRICS.md documents the sawtooth: the release drains the count to 0 during a
sustained total outage, so alerts must use max_over_time rather than the instant
value, and endpoint_serving_tier remains the unambiguous "is it dark" signal.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SpY8wY9rZgmgjufBpy2569
@Tomelia1999

Copy link
Copy Markdown
Contributor Author

@avitenzer thanks — both blocking findings were right, and I reproduced each one before fixing. Addressed in d4217b9.

1. The stale per-provider gauge

Fixed, but one level up from where you pointed. Rather than emitting false from each drain site, I made the gauge level-triggered: publishStateSizes now republishes the state of every paired provider on each tick.

The reason is that the drain sites are not the real problem — edge-triggered publishing is. Any future path that mutates the blocked list without emitting reintroduces exactly this bug, and setValidAddressesToDefaultValue is reached from more than one place already. A level-triggered gauge cannot go stale, cannot race the re-block pass on an epoch flip, and converges within one tick regardless of which path drained the list.

I also implemented ResetBlockedProvidersMetrics as you suggested. It is doing the piece the reconcile loop genuinely cannot: dropping series for providers that have left the pairing entirely, which would otherwise accumulate as phantoms. It drops the chain's series and seeds 0 for the new pairing set.

New TestPerProviderGauge_ClearedByThePoolEmptyRelease fails against the previous behaviour with provider7 is back in rotation and serving, so its per-provider gauge must not still read blocked.

2. The vacuous test

Deleted. Replaced with protocol/metrics/blocked_provider_metric_test.go following the pattern you pointed at — bare manager literal, test-scoped metric name, testutil.ToFloat64 — covering both directions, the reset rebase, chain scoping and nil safety. I confirmed each new test fails against the stub it guards, which the old one did not.

On the endpoint label — it is worse than staleness

You flagged provider_endpoint as a drift risk. Checking it, it is also a credential leak, so I dropped the label rather than trying to keep it stable.

Callers pass Endpoints[0].NetworkAddress, which is the raw configured node URL, and direct-RPC URLs routinely carry API keys in the path. The repo already states this rule in two places:

  • rpcsmartrouter_server.go:3943 — "Use provider name (configured name) instead of raw URL to avoid leaking API keys"
  • docs/METRICS.md:298 — "node URLs can embed API keys in their path or query and a credential must never reach a Prometheus series"

Dropping it also happens to be the cleaner answer to your staleness point: blocking is a per-provider decision, and Endpoints[0] was an arbitrary pick among several endpoints to begin with. The parameter stays in the signature since the interface fixes it, with a comment on why it is ignored, and TestSetBlockedProvider_DoesNotLabelByNodeURL pins it. Remaining label renamed to provider_address per the glossary, as you suggested.

One correction on the consumer search

no dashboard panel or alert reads it. The only external reference is a recording rule at smart-router-metric-compatibility.yaml:45

There are six references across three repos:

Repo File What it does
smart-router-dashboard promql/builders.ts — qCsm() queries it as one of the four CSM gauges (plus the constant and two tests)
Magma-Devs/docs reference/metrics.md:393 still carries the old description — goes stale on merge
Magma-Devs/docs configuration/failover/circuit-breaker.md:52 "when this gauge equals the pool size, every node is excluded — the condition that trips the breaker"
smart-router-standalone smart-router-metric-compatibility.yaml:45 the alias shim you found

The circuit-breaker page is the one that matters: it is published guidance telling operators to compare this gauge to the pool size, and "equals the pool size" is precisely the state the pool-empty release destroys. That makes your sawtooth point sharper than a dashboard nit — there is already customer-facing documentation built on a property the metric does not have.

I have taken your max_over_time suggestion into docs/METRICS.md along with an explanation of why the sawtooth happens, and pointed operators at smartrouter_endpoint_serving_tier for the unambiguous "is it dark" signal. The two Magma-Devs/docs pages need a companion PR — tracking separately.

Also taken

t.Skip guard removed, "Before v1.4.1" → "previously" (you are right that the tag is not knowable here), and the validAddresses[0]-twice readability trap fixed by capturing both addresses up front. Minor: the specific providers vary per run rather than being provider6/provider9 — the point stands either way.

Still open

  • ResetBlockedProviders emits false for every provider in the pairing rather than only the released ones, and spawns a goroutine each. Not harmful now that the reconcile loop is authoritative, but worth tidying.
  • Backups never reach the per-provider gauge at all — blockProvider's backup branch has no SetBlockedProvider call. Documented as a caveat for now.
  • Unrelated: TestHappyFlow in protocol/metrics/relays_monitor_test.go fails under -race on a test-local bool. Confirmed pre-existing — reproduces with these changes stashed. Needs its own ticket.

Verification: build, vet, gofmt clean; protocol/lavasession and protocol/metrics green; new tests race-clean at -count=3.

@Tomelia1999
Tomelia1999 merged commit d395ec2 into main Aug 25, 2026
22 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants