Skip to content

Commit 60154d2

Browse files
AnnaR-progclaude
andcommitted
fix(rpcsmartrouter): a provider out of the pairing passes none of the checks it was failing (MAG-3950)
A provider out of the pairing is listed by the check that took it out alone, so everything else it failed left the list at the demote and logged "verification check passes again", though nothing had checked it. A provider with no ws url logged its missing url passing at each demote, and failing again at each readmit (review of #602). trackFailures now holds those failures while the provider is out: no line at the demote, and at the readmit each goes on with its start, or passes again then, counted from its start. The gauge publishes the start the logs name, so a part failure that outlasts a demote comes back with its own start, not the readmit's: the PartialFail alert of the alert-by-severity branch posts at the readmit when that start is older than 30 minutes. sr-verify-failed reads scope=provider alone and is unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
1 parent 39e02e5 commit 60154d2

4 files changed

Lines changed: 82 additions & 4 deletions

File tree

‎docs/METRICS.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -655,7 +655,7 @@ self-heals in seconds. Alert windows should assume the 3m ceiling.
655655

656656
| Metric | Type | Labels | Description |
657657
| --- | --- | --- | --- |
658-
| `smartrouter_provider_verification_failed` | Gauge | `spec`, `apiInterface`, `provider`, `tier`, `scope`, `service`, `check`, `severity` | While a configured provider is failing a verification: the unix time in seconds the failure started. The series exists only while the failure lasts, and a provider that moves to another check in the same `scope` and `service` keeps its start (a `scope=none` warning never borrows another's). `tier` is `static` or `backup`. |
658+
| `smartrouter_provider_verification_failed` | Gauge | `spec`, `apiInterface`, `provider`, `tier`, `scope`, `service`, `check`, `severity` | While a configured provider is failing a verification: the unix time in seconds the failure started. The series exists only while the failure lasts, and a provider that moves to another check in the same `scope` and `service` keeps its start (a `scope=none` warning never borrows another's). While a provider is out of the pairing it has only its `scope=provider` series; a failure it had before it went out comes back with its own start when it is readmitted and still fails, and no "passes again" line is logged for it at the demote. `tier` is `static` or `backup`. |
659659
| `smartrouter_provider_verification_check_failures_total` | Counter | same as above | One per failed check run: at boot, each verification retry, and each 15m epoch pass (a grace run before a demotion included). A rate-limited run is inconclusive and not counted, on any path, and a missing websocket url (`missing-url`) ran nothing, so it is not counted either. `increase(...[$__interval]) > 0` draws one mark per failed run. |
660660

661661
The serving tier only drops when a whole tier is gone, so a single provider that fails

‎protocol/rpcsmartrouter/spec_reverifier.go‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -128,6 +128,11 @@ type chainReverifyInputs struct {
128128
// "<tier>|<name>" (trackFailures). The gauge, the transition logs and
129129
// /debug/verification all read it. Guarded by RPCSmartRouter.mu; lazily allocated.
130130
failing map[string][]metrics.TimedVerificationFailure
131+
// heldWhileOut is what each provider out of the pairing failed besides the check that
132+
// took it out, keyed like failing (trackFailures). Nothing checked it while the
133+
// provider was out, so it did not pass: it comes back with its start when the provider
134+
// is readmitted, or passes again then. Guarded by RPCSmartRouter.mu; lazily allocated.
135+
heldWhileOut map[string][]metrics.TimedVerificationFailure
131136
// lastPass is the pass that refused each provider out of the pairing, keyed
132137
// "<tier>|<name>" (noteFailure). Its retry re-runs only the checks that failed there
133138
// and keeps the other verdicts. Dropped when the provider passes. Guarded by

‎protocol/rpcsmartrouter/verification_state.go‎

Lines changed: 34 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -15,12 +15,26 @@ import (
1515
// started (metrics.VerificationFailuresSince), and logs every change (MAG-3950): one line
1616
// when a check starts failing, an error, or a warning for a Warning check and for a
1717
// missing websocket url, which the config sets and no retry can fix (decided on #602),
18-
// and one when it passes again. A check that takes another's place in the same scope and service, a
19-
// provider out of the pairing that now fails chain-id where it failed connect, is one
20-
// line, naming the check it replaced. Callers hold RPCSmartRouter.mu.
18+
// and one when it passes again. A check that takes another's place in the same scope and
19+
// service, a provider out of the pairing that now fails chain-id where it failed connect,
20+
// is one line, naming the check it replaced.
21+
//
22+
// A provider out of the pairing fails at scope provider alone: its caller lists only the
23+
// check that took it out. Nothing checks what else it failed while it is out, so that
24+
// does not pass: it is held, with no line, and when the provider is back each held
25+
// failure goes on with its start, or passes again then (review of #602: a provider with
26+
// no ws url logged its missing url passing at each demote). Callers hold
27+
// RPCSmartRouter.mu.
2128
func (inputs *chainReverifyInputs) trackFailures(tier reverifyTier, name string, failures []metrics.ProviderVerificationFailure, now time.Time) []metrics.TimedVerificationFailure {
2229
key := retryKey(tier, name)
2330
previous := inputs.failing[key]
31+
out := slices.ContainsFunc(failures, func(f metrics.ProviderVerificationFailure) bool {
32+
return f.Scope == metrics.VerificationScopeProvider
33+
})
34+
if held := inputs.heldWhileOut[key]; !out && len(held) > 0 {
35+
previous = append(slices.Clone(previous), held...)
36+
delete(inputs.heldWhileOut, key)
37+
}
2438
next := metrics.VerificationFailuresSince(previous, failures, now)
2539
if inputs.failing == nil {
2640
inputs.failing = map[string][]metrics.TimedVerificationFailure{}
@@ -30,6 +44,20 @@ func (inputs *chainReverifyInputs) trackFailures(tier reverifyTier, name string,
3044
} else {
3145
inputs.failing[key] = next
3246
}
47+
if out {
48+
held := inputs.heldWhileOut[key]
49+
for _, f := range previous {
50+
if f.Scope != metrics.VerificationScopeProvider && !failingIn(held, f) {
51+
held = append(held, f)
52+
}
53+
}
54+
if len(held) > 0 {
55+
if inputs.heldWhileOut == nil {
56+
inputs.heldWhileOut = map[string][]metrics.TimedVerificationFailure{}
57+
}
58+
inputs.heldWhileOut[key] = held
59+
}
60+
}
3361

3462
attrs := func(f metrics.TimedVerificationFailure) []utils.Attribute {
3563
return []utils.Attribute{
@@ -62,6 +90,9 @@ func (inputs *chainReverifyInputs) trackFailures(tier reverifyTier, name string,
6290
if failingIn(next, f) {
6391
continue
6492
}
93+
if out && f.Scope != metrics.VerificationScopeProvider {
94+
continue // held until the provider is back
95+
}
6596
if _, ok := takesThePlaceOf(f, next, previous); ok {
6697
continue // logged as the check that replaced it
6798
}

‎protocol/rpcsmartrouter/verification_state_test.go‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -77,6 +77,48 @@ func TestTrackFailures_LogsEachTransitionOnce(t *testing.T) {
7777
require.Empty(t, inputs.failing["static|"+provider][0].Service)
7878
}
7979

80+
// A provider out of the pairing is listed by the check that took it out alone, but it
81+
// did not pass what else it failed (review of #602). Its missing websocket url logs no
82+
// "passes again" when it goes out and no second line when it is back, and keeps its
83+
// start; an archive check that it passes when it is back passes again then, counted
84+
// from its own start.
85+
func TestTrackFailures_AProviderOutOfThePairingPassesNothingItFailed(t *testing.T) {
86+
utils.EnableDebugLogBuffer(1000)
87+
defer utils.ClearDebugLogBuffer()
88+
utils.ClearDebugLogBuffer()
89+
const provider = "track-held-provider"
90+
inputs := &chainReverifyInputs{rpcEndpoint: bootTestEndpoint()}
91+
start := time.Date(2026, 11, 12, 10, 0, 0, 0, time.UTC)
92+
missing := metrics.ProviderVerificationFailure{Scope: metrics.VerificationScopeService, Service: chainlib.WebSocketExtension, Check: chainlib.CheckMissingURL}
93+
archive := metrics.ProviderVerificationFailure{Scope: metrics.VerificationScopeService, Service: "archive", Check: "pruning"}
94+
out := []metrics.ProviderVerificationFailure{{Scope: metrics.VerificationScopeProvider, Check: "chain-id"}}
95+
96+
inputs.trackFailures(reverifyTierStatic, provider, []metrics.ProviderVerificationFailure{missing, archive}, start)
97+
tracked := inputs.trackFailures(reverifyTierStatic, provider, out, start.Add(20*time.Minute))
98+
require.Len(t, tracked, 1, "while it is out, the provider is published by the check that took it out")
99+
inputs.trackFailures(reverifyTierStatic, provider, out, start.Add(21*time.Minute))
100+
tracked = inputs.trackFailures(reverifyTierStatic, provider, []metrics.ProviderVerificationFailure{missing}, start.Add(25*time.Minute))
101+
102+
require.Len(t, tracked, 1)
103+
require.Equal(t, chainlib.CheckMissingURL, tracked[0].Check)
104+
require.Equal(t, start, tracked[0].Since, "the url is missing since the start, not since the readmit")
105+
require.Empty(t, inputs.heldWhileOut, "nothing is held once the provider is back")
106+
107+
lines := verificationLogLines(t, provider)
108+
checks := make([]any, 0, len(lines))
109+
for _, line := range lines {
110+
checks = append(checks, line["check"])
111+
}
112+
require.Equal(t, []any{chainlib.CheckMissingURL, "pruning", "chain-id", "chain-id", "pruning"}, checks,
113+
"the start fails missing-url and pruning, the demote fails chain-id alone, the readmit passes chain-id and pruning")
114+
require.Equal(t, "verification check failing", lines[2]["message"])
115+
require.Nil(t, lines[2]["previousCheck"], "chain-id takes no part's place")
116+
require.Equal(t, "verification check passes again", lines[3]["message"])
117+
require.Equal(t, "5m0s", lines[3]["failedFor"])
118+
require.Equal(t, "verification check passes again", lines[4]["message"])
119+
require.Equal(t, "25m0s", lines[4]["failedFor"], "pruning failed from the start until the readmit")
120+
}
121+
80122
// GET /debug/verification shows, per configured provider, what it is failing with the
81123
// start the gauge publishes, whether it is in the pairing, and each check's next retry.
82124
func TestDebugVerification_ShowsFailingChecksAndRetries(t *testing.T) {

0 commit comments

Comments
 (0)