Repository navigation
fix(routersession,holdoff)!: hold a provider off only while every url that can serve the request is held (MAG-4732, MAG-4733, MAG-4734) - #618
Conversation
avitenzer
left a comment
There was a problem hiding this comment.
@Tomelia1999 I reviewed this at 3b1ba813. I don't think it can be approved yet. Details are in the inline comments.
The url pick inside a provider looks right, and I checked that the tests guard it. With the free-urls-first ordering disabled, TestHoldoff_OneHeldURLLeavesTheProviderToItsSibling and TestHoldoff_ExpiredSiblingIsPickedBeforeAHeldURL both go red. The relay path, the session manager and the chain tracker all build the provider key from the same endpoint's chain and api-interface, so the keys match. On the head, go build and go vet ./... are clean, and the tests pass for holdoff, chainlib, chaintracker, endpointstate, lavasession and rpcsmartrouter. Merged locally into current main (#624 touches consumer_types.go and consumer_session_manager.go too), it merges cleanly, and lavasession and holdoff pass.
Blocking:
- A single re-verify pass can hold a healthy url, depending on config order (inline on
spec_reverifier.go:358). - The re-verify skip rule goes against MAG-4733 and the PR gives no reason (inline on
:325). - Limiting escalation to one chain and api-interface changes publicly documented behaviour, but nothing marks it as breaking (inline on
holdoff.go:90). - CI never ran on
3b1ba813. The last runs are onf5a1bd93. The branch is also behind main, so updating it will trigger CI.
Non-blocking:
- The commit message of d3f866b says a rate limit not tied to a url "holds them all, as it held the provider before". With 2+ urls it now holds the whole entry, which is more than before. Only the
validateFntest hook reaches that path, sinceservingErroralways ties a throttle to a url. - Main's #624 added
inRotation, which is another copy of "which url can take a relay", next toservesRequest. Worth deciding whether they should be one rule.
| utils.LogAttr("url_index", index), | ||
| utils.LogAttr("holdoff", delay.String()), | ||
| ) | ||
| case answered: |
There was a problem hiding this comment.
Blocking: the outcome of one re-verify pass depends on the order of the urls in the config.
This loop calls RecordRateLimit / RecordAnswer per url in config order. RecordRateLimit holds the whole entry once 2 urls are held. RecordAnswer clears that hold (holdoff.go, clearLocked resets escalatedUntil). So a pass where one url answers and two return 429 gives different results depending on the order:
- order
[A answers, B 429, C 429]: the entry is held, soHeldOff(key, A)is true andProviderReadyAt(key, [A])reports held, even though A answered in this same pass. - order
[B 429, C 429, A answers]: A's answer clears the hold, and A is free.
I reproduced this with a throwaway test that calls the registry in both orders (NewRegistry, the same ProviderKey). Main recorded only one url per pass, so a single pass could never hold the whole entry. Once it is held, the skip at :325 stops the next pass from clearing it. The hold also applies to tracker polls and to the recovery probe.
Related: holdoffURLKeys returns every configured url, wss included, and each one is recorded under the provider key. The relay path never records a ws url there, and relayURLs leaves them out. So an https url and a wss url both returning 429 in one pass can hold the whole entry, including the https url relays use.
Suggested fix: apply the pass's answers after its rate limits, or don't hold the whole entry from a single pass. Also add a test with 3 urls (one answering) in both orders, and one with an https url and a wss url both rate limited.
There was a problem hiding this comment.
Fixed in b38b5e04. The pass now merges its results per url, records every 429 first, and only then records the answers, so the result no longer depends on config order (recordPassHoldoffs). A wss url is held under its own key, the one the WS path reads, so it no longer counts toward the entry escalation. The same url configured twice also keeps its 429 now. Tests: TestRecordPassHoldoffs_AnAnsweringURLIsFreeInEveryConfigOrder (3 urls, both orders), _AWebsocketRateLimitDoesNotHoldTheEntry, _TheSameURLTwiceKeepsItsRateLimit.
There was a problem hiding this comment.
Verified at edbd8d49. With the answer loop moved before the rate-limit loop in recordPassHoldoffs, TestRecordPassHoldoffs_AnAnsweringURLIsFreeInEveryConfigOrder goes red. With the per-url merge disabled, _TheSameURLTwiceKeepsItsRateLimit goes red. With the wss url keyed under the provider, _AWebsocketRateLimitDoesNotHoldTheEntry goes red. Resolved from my side.
| // The provider on this chain and api-interface, as relay selection keys it. | ||
| holdoffKey := holdoff.ProviderKey(inputs.rpcEndpoint.ChainID, inputs.rpcEndpoint.ApiInterface, p.Name) | ||
| keys := holdoffURLKeys(p) | ||
| for _, key := range keys { |
There was a problem hiding this comment.
Blocking: this goes against MAG-4733 and gives no reason.
The ticket's fix direction says: "Skip the re-verify only when every url is held." This loop skips the whole pass while any url is held under the provider key. The PR body and RATE-LIMIT-HOLDOFF.md say that it does this, but not why.
With relay 429s now recorded per url (the point of this PR), one busy url can keep the provider from being re-verified for as long as its hold lasts. During that time a sibling url that was refused for a capability problem can't be re-admitted. Please either follow the ticket (skip only when every url is held, or probe only the urls that aren't held), or put the reason in the PR body and the ticket.
There was a problem hiding this comment.
Fixed in b38b5e04, following the ticket. Held urls are not asked, the free urls are, and the pass is skipped only when every url is held. A url that was not asked keeps its last verdict and its hold-off. Tests: TestApplyReverification_AHeldURLIsNotAskedAndItsSiblingIs, _EveryURLHeldSkipsThePass, and _ARecoveredURLComesBackWhileItsSiblingIsHeld (the MAG-4733 case: a refused url is re-admitted while its sibling is held).
There was a problem hiding this comment.
Verified at edbd8d49. Skipping the pass when any url is held turns AHeldURLIsNotAskedAndItsSiblingIs and ARecoveredURLComesBackWhileItsSiblingIsHeld red, and so does passing nil as the skip set. Resolved from my side, but skipping a url introduced a new issue with per-service refusals (see my inline comment on chain_fetcher.go:917).
| // only, so keyed by the name alone two held urls of one entry escalated every entry that | ||
| // shares the name — a 429 storm on jsonrpc held REST too (MAG-4732). An empty name stays | ||
| // empty, so a caller's "no provider" still reads as one. | ||
| func ProviderKey(chainID, apiInterface, name string) string { |
There was a problem hiding this comment.
Blocking: this is a change to publicly documented behaviour.
Limiting the escalation to one chain and api-interface contradicts the public docs:
- Magma-Devs/docs
docs/configuration/failover/retry.md:91-94: "the whole provider is held off on every chain". - Magma-Devs/docs
docs/reference/metrics.md:102: theescalatedalert means the router "stopped calling that provider on every chain".
Under our rule for the documented router surface, that makes this a breaking change. It needs ! in the PR title and squash message, a BREAKING CHANGE: footer, and a docs follow-up for both pages. The tickets don't ask for this scoping either, so it would be good to add a line about it to MAG-4732.
In this repo the wording is also out of date: the metric help text at holdoff/metrics.go:29 ("widened to the provider name") and docs/METRICS.md:409 / :423 still describe a name-wide escalation.
There was a problem hiding this comment.
Done. The title has !, the body has a BREAKING CHANGE: footer, and the squash message will carry both. The metric help text (holdoff/metrics.go) and docs/METRICS.md now describe the per-entry escalation. The public docs follow-up is Magma-Devs/docs#86 (both pages). It stays a draft until the release that ships this.
There was a problem hiding this comment.
Marking it as breaking is done (title, footer, metric help, METRICS.md, docs#86). Thanks. The open part is the decision itself: this reverses MAG-2947's account-wide rationale, so it needs a product ruling before merge. I've noted that in the description and in the review summary.
There was a problem hiding this comment.
Ruling: we keep the per-entry scope.
The "other chains" case in MAG-2947 doesn't happen in the fleet. The chart creates one Deployment per chain and network (<id>-router), so every chain runs in its own process with its own hold-off registry, and each replica pod has its own too. A 429 on a vendor in the ETH router was never seen by the BTC router, before or after this PR. Name-wide escalation across chains only happens when several chains run in one process, like the local multichain example.
So in production the only thing this changes is across api-interfaces of one chain. Before, two held rest urls of a vendor also held its tendermintrpc and grpc entries. Now they don't. Each url that answers 429 is still held on its own, so another interface loses traffic only when it gets 429s itself. The ruling is recorded in MAG-4732.
| // the url itself, the key the relay path records a 429 under for the endpoint it dialed. A 429 | ||
| // is held on the url that answered it, so relay selection, which counts a provider's urls one by | ||
| // one, keeps off that url and not a sibling (MAG-4733). | ||
| func holdoffURLKeys(p *lavasession.RPCStaticProviderEndpoint) []string { |
There was a problem hiding this comment.
Non-blocking: re-verify holds and clears the configured urls (p.NodeUrls), but relays use the urls expandInternalPaths generates from them (rpcsmartrouter.go:2480). On a chain with internal paths, a root url with no internal path of its own produces base + path endpoints. So a 429 that re-verify sees on the base url holds only the root endpoint, and the per-path urls on the same host stay free. And a relay 429 on a generated url is never seen by this skip or clear. The TON row of the PR's table is tested only with hand-built endpoints in holdoff_url_pick_test.go, so this path isn't covered. Probably fine, since 429s are per host in practice, but worth either a sentence in the PR body or a test that goes through the expansion.
There was a problem hiding this comment.
Agreed, no change in this PR. Vendor rate limits are per host and account, so in practice a 429 on the base url and on its per-path urls arrives together, and each relay 429 is still held on the url that got it. The gap is that re-verify does not see the generated per-path urls, and that the TON row has no test through expandInternalPaths. I am keeping it as a known limitation and not widening this PR.
There was a problem hiding this comment.
Agreed. I added it to the description as a known limitation, along with the missing TON test through expandInternalPaths.
avitenzer
left a comment
There was a problem hiding this comment.
@Tomelia1999 I re-reviewed at edbd8d49. Thanks for the quick turnaround: the four blockers from my last review are fixed, and I've replied on each thread. It still can't be approved, for the reasons below.
What I checked
- From a clean worktree of
edbd8d49:go build ./...andgo vetare clean, andgo testpasses forholdoff,lavasession,rpcsmartrouter,chainlib,chaintrackerandendpointstate. - The new tests guard the fixes. Each of these mutants turns at least one of them red: skip the pass when any url is held, ask the held urls anyway, key a wss 429 under the provider, record answers before 429s, and let a later duplicate url overwrite an earlier 429.
Blocking
- New in edbd8d4: a url the pass skips loses its per-service refusals, so a pruned node gets archive re-admitted (inline on
chain_fetcher.go:917, with a repro). - The escalation scope needs a product ruling before merge. Marking it as breaking was right, but this reverses the design MAG-2947 settled on purpose: "a vendor cap is account-wide, and per-URL keying alone would let the process keep hammering the account through its other chains." MAG-4732 and MAG-4733 record no ruling, and the description said "by decision on this PR". I've updated the description to say a ruling is pending.
- Conflicts with main (78 commits behind;
holdoff/metrics.go,consumer_session_manager.go, and threelavasessiontest files). Most of it is mechanical, since main threadstriedByThisRequestwhere this PR threadsinternalPath. One hunk isn't:getValidConsumerSessionsWithProvideron main now carries the MAG-4829 last-chance retry block (#637) where this branch returns directly. The resolution has to keep that block.metrics.goalso has to keep main'sKeepUnregisteredOnFailure/RegisterOrReuseregistration. - CI hasn't run on
edbd8d49("no checks reported", because of the conflict). It will run once the branch is updated.
Description: I edited the description to match edbd8d49. The re-verify paragraph still described the old skip-on-any rule. I added chain_fetcher.go (ValidateCollectionsExcept, ErrNotAskedHeldOff) and recordPassHoldoffs to the code list, the per-path-url known limitation from our thread, and that MAG-4733 is fixed in the second and fifth commits. Your replies point to b38b5e04, but that commit doesn't exist on the branch; the work is in edbd8d49.
#622 is stacked on this PR and keeps waiting for it.
| // notAsked is the share of a pass of a url the pass leaves out: rate-limited, with nothing | ||
| // sent (ErrNotAskedHeldOff). | ||
| func (g *connectionGroup) notAsked(url common.NodeUrl) nodeUrlOutcome { | ||
| outcome := nodeUrlOutcome{url: url, served: url, err: ErrNotAskedHeldOff, head: spectypes.NOT_APPLICABLE} |
There was a problem hiding this comment.
Blocking (new in edbd8d4): a url the pass skips loses the add-ons it was refused, so a pruned node gets archive back.
notAsked makes the url throttled at the url level (throttledURLs[index]) and adds no checks. KeepingThrottledFrom (chain_fetcher.go:439) then keeps the url's previous url refusal (previous.refusedURLs[index]), but copies a previous service refusal (previous.refused[url+path+service]) only when the same service is in this pass's throttledServices. A skipped url has none, so every per-service refusal on it is dropped.
Repro: url A is a full node, url B is a pruned node with Addons: [archive]. At boot archive is refused on B and B is kept for the base collection. B then gets a relay 429. The next pass checks A, skips B, and logs re-verify: rebuilding static provider, admitted services changed: archive is admitted on B again. Archive requests can then reach the pruned node until B's hold ends and a later pass asks it and refuses archive again. Since B is the only archive url, the hold doesn't keep archive traffic off it either, because a provider whose every capable url is held still hands one out.
Test (fails at edbd8d4, passes at 3b1ba81)
func TestReview_HeldURLKeepsItsServiceRefusal(t *testing.T) {
up := newFakeUpstreams(t)
provider := staticProvider("ETH1",
common.NodeUrl{Url: up.httpURL("/a/" + upArchive)},
common.NodeUrl{Url: up.httpURL("/b/" + upPruned), Addons: []string{"archive"}},
)
parser := liveParser(t, "ETH1", true)
admissions := newAdmissionRegistry()
boot, err := validateProviderCollections(t.Context(), provider, parser, 20*time.Second)
require.NoError(t, err)
services, _ := boot.AdmittedServices(provider.NodeUrls[1])
require.Empty(t, services, "precondition: archive refused on B at boot")
admissions.record(provider, boot)
key := holdoff.ProviderKey("ETH1", "jsonrpc", provider.Name)
holdoffs := holdoff.NewRegistry()
holdoffs.RecordRateLimit(key, provider.NodeUrls[1].Url, 0) // B busy with relay 429s
inputs := &chainReverifyInputs{
chainParser: parser,
rpcEndpoint: &lavasession.RPCEndpoint{ChainID: "ETH1", ApiInterface: "jsonrpc"},
convertProvidersToSessions: fakeConvert,
configuredStatic: []*lavasession.RPCStaticProviderEndpoint{provider},
recordAdmission: admissions.record,
recordReverifiedAdmission: admissions.recordReverified,
admissionFor: admissions.admissionFor,
rateLimitHoldoff: holdoffs,
}
active := map[uint64]*lavasession.ConsumerSessionsWithProvider{0: makeSession(provider.Name)}
askedB := len(up.methods("http", "/b/"+upPruned))
applyReverification(t.Context(), inputs, active, reverifyTierStatic, 100)
require.Len(t, up.methods("http", "/b/"+upPruned), askedB, "B was not asked")
after, _ := admissions.admissionFor(provider).AdmittedServices(provider.NodeUrls[1])
require.Empty(t, after, "archive must stay refused on B: the pass never re-checked it")
}At edbd8d4: Should be empty, but was [archive].
One possible fix: when KeepingThrottledFrom keeps a url that was throttled at the url level, also keep every previous.refused key for that url. Whether the url was skipped or answered 429 to everything, nothing in this pass re-checked its services. The url-level 429 case had this gap before this PR too, but only on a fresh 429. With skipping it now happens on every pass where a sibling is free.
There was a problem hiding this comment.
Confirmed, thanks for the repro. Fixed in 00db21e: when a pass does not check a url again (held and skipped, or every check came back 429), the url keeps every add-on refusal and warning it had, not only its url refusal. Your test is in as TestApplyReverification_AHeldURLKeepsItsAddonRefusals, plus TestKeepingThrottledFrom_AURLNotCheckedKeepsItsServiceRefusals in chainlib. Both go red with the rule switched off.
#618 moved its branch from 3b1ba81 to edbd8d4. The only conflict is docs/RATE-LIMIT-HOLDOFF.md, on two adjacent table rows: keep #618's "Spec re-verification" row and this PR's "Hot relay path" row (MAG-4734). No code conflict. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Took 47 minutes
…on refusals (MAG-4733) A url the pass skipped because it is held off, or saw only rate-limited at the url level, kept its url verdict but lost every per-service refusal: those were kept only for a service throttled in this pass, and a skipped url has no checks. A pruned node refused archive at boot got archive admitted again on the next pass after a relay 429, without being asked. ProviderAdmission now remembers the url at each throttled position, and KeepingThrottledFrom keeps every service refusal the previous verdict gave that url. Found in review on #618. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2f9f2a0 to
ccadd20
Compare
|
Rebased on main at ccadd20. #622 was merged into this branch, so its commit is carried over (code identical to 2f9f2a0). Main reworked the re-verifier for per-check retries (MAG-3950), so I re-applied the re-verify part on that shape instead of rebasing it line by line (9fd687c):
One more thing came out of the rebase (061ca58): the probe loop from MAG-4387 read the relay hold-off under the plain provider name, so it would never see a relay hold on this branch. It now reads the scoped key, carried on Every new rule has a test, and each test goes red with its rule switched off. Note: the intermediate commits from the rebase don't each build on their own (the re-verifier work sits in 9fd687c). The squash merge makes that moot. |
avitenzer
left a comment
There was a problem hiding this comment.
@Tomelia1999 re-reviewed at ccadd203. Not ready yet, for one reason that is new since this morning: main moved under the branch when #659 merged (17dbfff2), and the two PRs disagree on which url list the re-verifier indexes. Everything I asked for last time is in and holds.
What I checked
- CI at
ccadd203: unit-tests, unit-tests-race, lint and the builds are green. govulncheck is red for GO-2026-6859, which #659 fixed on main with thego.modbump to 1.26.11; a rebase picks it up. - Trial merge with main
17dbfff2: one textual conflict (consumer_types.go, resolution inline), two of main's tests that need this PR's new arguments (inline), and one semantic conflict (inline onholdoffEntries, with a failing throwaway test). With the conflict resolved and the two tests patched the merged tree builds and vets, andgo test -count=1is green on holdoff, chainlib, lavasession, rpcsmartrouter, chaintracker and endpointstate. 00db21e9: dropping the service-refusal keep loop inKeepingThrottledFromturns bothTestKeepingThrottledFrom_AURLNotCheckedKeepsItsServiceRefusalsandTestApplyReverification_AHeldURLKeepsItsAddonRefusalsred. Resolved.- The skip-on-every-url rule and the 429-before-answer ordering are as verified at
edbd8d49; the re-apply on the MAG-3950 shape keeps them, and the new tests forURLAsked,heldOffUntiland the retry pass are there.
Blocking
- Configured-vs-dialed index mismatch after #659 (inline on
holdoffEntries). On chains with internal paths the hold-off entries, the skip set andurlPassResultsare positions intop.NodeUrls, while the pass now verifiesdialedNodeUrls(...). A 429 from a generated url is never held, a relay hold on one is not honoured, and a held pinned url is asked while a free sibling is skipped. Reproduced on the trial merge in all three shapes. The fix is to build the entries from the dialed list inholdoffEntriesandheldOffUntil; that also retires the "known limitation" paragraph, since the relay path already keys 429s by the dialed url.
For the rebase
- Fold
UnservedInternalPathsintorestrictsToPathLocked(inline), and give main's two tests the new arguments. - Main requires verified commit signatures since 2026-11-10 17:11 (ruleset "Main branch default"). Every commit on this branch is unsigned, so the squash is refused until they are rewritten signed. You have an SSH signing key registered;
git rebase --gpg-signwithgpg.format=sshdoes it. - The description's Jira section says "Also fixes MAG-4733". #622's commit now rides here, so add MAG-4734, and the squash message should name all three.
The scoping ruling. The ruling on MAG-4732 (comment of 2026-11-11) is yours, not product's. I checked its premise: the chart renders one Deployment per routers[] entry and each entry has one network, so a hold-off registry never saw two chains in the fleet, and the change is confined to api-interfaces of one entry. Accepted; recorded on MAG-4732.
| url string | ||
| } | ||
|
|
||
| func holdoffEntries(providerKey string, p *lavasession.RPCStaticProviderEndpoint) []holdoffEntry { |
There was a problem hiding this comment.
Blocking once rebased on current main: the hold-off entries and the skip set index the configured urls, but the pass now verifies the dialed list.
#659 (merged today, 17dbfff) makes revalidateProviderCollections verify dialedNodeUrls(provider.NodeUrls, …): on a chain with internal paths a bare url becomes one url per path, so the pass's positions are into [A/v2, A/v3, C/v2, C/v3] while holdoffEntries, held and urlPassResults(…, len(entries), held) are built from p.NodeUrls = [A, C]. The text merge is clean, so nothing flags it. On chains without internal paths the lists are equal and the PR behaves as described; on TON, STRK, AVAX and Monero it does not:
- a 429 from
C/v2(dialed index 2) is never read, sinceurlPassResultsstops atlen(entries), so the url that answered 429 is never held, which is MAG-4733's bug again; - a relay hold on
C/v2is not in the skip set, becauseHeldOff(key, C)is asked about the bare url, so the held url is sent the request it was told to stop sending; - with
[A bare, B pinned /v2]and B held,held={1}skips dialed index 1, which isA/v3, the free url, while B is asked. On TONT that also makes the pass inconclusive ("no node url verified serving internal path /v3: not asked: held off"), so the provider is never re-verified while B is held.
Reproduced on a trial merge of ccadd20 with main 17dbfff (conflict in consumer_types.go resolved as in my other comment):
Throwaway test, three subtests, all fail
// newLimitedTon: an httptest server answering 429 to every path, counting hits.
// tontInputs: chainReverifyInputs with tontParser(t), rpcEndpoint TONT/rest, fakeConvert, an admission registry.
func TestReview_ReverifyHoldoffIndexesConfiguredUrlsButPassIsDialed(t *testing.T) {
t.Run("a 429 from a generated url is never held", func(t *testing.T) {
a := newFakeTon(t, tontRootHash)
c, _ := newLimitedTon(t)
provider := tontProvider(commonlib.NodeUrl{Url: a.srv.URL}, commonlib.NodeUrl{Url: c.URL})
key := holdoff.ProviderKey("TONT", "rest", provider.Name)
holdoffs := holdoff.NewRegistry()
inputs, admissions := tontInputs(t, provider, holdoffs)
active := map[uint64]*lavasession.ConsumerSessionsWithProvider{0: makeSession(provider.Name)}
applyReverification(t.Context(), inputs, active, reverifyTierStatic, 100)
require.NotNil(t, admissions.admissionFor(provider).URLRateLimit(2), "the pass saw the 429 on dialed index 2 (C/v2)")
require.True(t, holdoffs.HeldOff(key, c.URL+"/v2"), "the url that answered 429 must be held") // FAILS
})
t.Run("a relay hold on a generated url is not honoured", func(t *testing.T) {
a := newFakeTon(t, tontRootHash)
c, hits := newLimitedTon(t)
provider := tontProvider(commonlib.NodeUrl{Url: a.srv.URL}, commonlib.NodeUrl{Url: c.URL})
key := holdoff.ProviderKey("TONT", "rest", provider.Name)
holdoffs := holdoff.NewRegistry()
holdoffs.RecordRateLimit(key, c.URL+"/v2", 0) // what the relay path records for C/v2
inputs, _ := tontInputs(t, provider, holdoffs)
active := map[uint64]*lavasession.ConsumerSessionsWithProvider{0: makeSession(provider.Name)}
applyReverification(t.Context(), inputs, active, reverifyTierStatic, 100)
require.Equal(t, 0, hits("/v2/getMasterchainInfo"), "a held url is not asked") // FAILS: 1
})
t.Run("a held pinned url is asked and a free sibling is skipped instead", func(t *testing.T) {
a := newFakeTon(t, tontRootHash)
b, hits := newLimitedTon(t)
pinned := commonlib.NodeUrl{Url: b.URL + "/v2", InternalPath: "/v2"}
provider := tontProvider(commonlib.NodeUrl{Url: a.srv.URL}, pinned)
key := holdoff.ProviderKey("TONT", "rest", provider.Name)
holdoffs := holdoff.NewRegistry()
holdoffs.RecordRateLimit(key, pinned.Url, 0)
inputs, _ := tontInputs(t, provider, holdoffs)
active := map[uint64]*lavasession.ConsumerSessionsWithProvider{0: makeSession(provider.Name)}
applyReverification(t.Context(), inputs, active, reverifyTierStatic, 100)
require.Equal(t, 0, hits("/v2/getMasterchainInfo"), "the held url B/v2 is not asked") // FAILS: 1
})
}Suggested fix: build the entries from the list the pass verifies, dialedNodeUrls(p.NodeUrls, inputs.chainParser, p.ApiInterface), in holdoffEntries and in heldOffUntil (every call site has inputs.chainParser or chainParser in scope). The relay path already records a 429 under the dialed url (targetEndpoint.NetworkAddress is base + path for a generated url), so the keys line up as well, and the "known limitation" paragraph in the description goes away. The three subtests above, plus the TON row of holdoff_url_pick_test.go through the expansion, would pin it.
There was a problem hiding this comment.
Confirmed, and fixed in 07dc208. holdoffEntries now builds from reverifyNodeUrls, which is dialedNodeUrls(p.NodeUrls, chainParser, p.ApiInterface), the same list the pass verifies. So the entries, the skip set and urlPassResults all index one list. With no chain parser (the validateFn test hook) it falls back to the configured list.
Your three subtests are in reverify_dialed_holdoff_test.go (TestReverifyHoldoff_UsesTheDialedURLs), plus a fourth that pins the entries heldOffUntil reads. With reverifyNodeUrls switched back to p.NodeUrls, all four go red.
I didn't add the TON row through the expansion to holdoff_url_pick_test.go. That test is in routersession, which gets its endpoints already expanded, and the expansion itself is in rpcsmartrouter. The second subtest covers that link: a relay hold recorded under C/v2 is honoured by the pass.
The "known limitation" paragraph is gone from the description. docs/RATE-LIMIT-HOLDOFF.md's re-verification row now says the urls are the dialed ones.
| } | ||
| return registry.ReadyAt(p.Name, holdoffURLKey(p)) | ||
| var first time.Time | ||
| for _, entry := range holdoffEntries(holdoff.ProviderKey(p.ChainID, p.ApiInterface, p.Name), p) { |
There was a problem hiding this comment.
Same list as in holdoffEntries: once the entries come from the dialed urls, heldOffUntil has to read the same ones, or a retry waits on the bare url's entry, which nothing records under on TON. All four call sites (rpcsmartrouter.go:3040, :4563, spec_reverifier.go:636, :748) have a chain parser in scope.
There was a problem hiding this comment.
Done in 07dc208: heldOffUntil takes the chain parser and reads the same holdoffEntries. All four call sites pass inputs.chainParser.
| // generated. | ||
| // | ||
| // cswp.Lock must be held. | ||
| func (cswp *ConsumerSessionsWithProvider) restrictsToPathLocked(internalPath *string) bool { |
There was a problem hiding this comment.
Rebase note: this is the one textual conflict with main 17dbfff. #659 added UnservedInternalPaths (a path the provider was configured for but lost at admission) to the old in-line check. The resolution that keeps both is to fold it in here, so the walk and relayURLs pass such a provider over together:
if _, lost := cswp.UnservedInternalPaths[*internalPath]; lost {
return true
}before the endpoint loop. With that, the merged tree builds and vets. Two of main's tests also need the new arguments: direct_rpc_session_selection_test.go:371 (the heldOff func, nil) and node_url_verification_test.go:63 and :70 (throttleURL now takes the url).
There was a problem hiding this comment.
Resolved as you suggested: restrictsToPathLocked returns true for a path in UnservedInternalPaths before the endpoint loop, so the walk and relayURLs pass such a provider over together. Main's two tests got the new arguments in 265d9ed.
The branch is now on main 36b0cd8, which includes #695's rename, so this file is protocol/routersession/consumer_types.go.
|
@Tomelia1999 over to you for the rebase onto main
The scoping ruling is accepted (MAG-4732). With the four points above and green CI, this is ready to approve. |
…at can serve the request is held (MAG-4732) A 429 is recorded per (provider, url), but selection asked only whether the provider name was held, and the registry only knows urls that answered 429. So one url's 429 held a provider whose other urls were free, and the same name's entry on another api-interface with it. Inside a provider the url walk never read the hold-off and handed out a held url listed first. ProviderReadyAt now takes the urls that could serve the request (usable, able to answer a relay, serving its add-on, extensions and internal path); a url with no entry is free, and escalation is unchanged. The request's internal path is passed down the provider selection chain for that. The url walk puts held urls after free ones and drops none, so a request still goes out when every url is held. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…red it (MAG-4733) The re-verifier filed every 429 under the provider's first node url. With relay selection now counting a provider's urls one by one (MAG-4732), a 429 from url B held url A, so relays went to B, the url that had just refused; with a websocket url first, the 429 was held where relay selection never looks, and held nothing. The pass already knows which urls met a rate limit. Each is now held under its own url, with its own Retry-After; a rate limit the pass does not tie to a url holds them all, as it held the provider before. Each url that answered clears its own entry. A pass is still skipped while any url of the provider is held. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…terface (MAG-4732) The registry keyed a provider by its name alone, and a name is unique per chain and api-interface only. Two held urls of one entry escalated the name, and ProviderReadyAt then held every entry sharing it: a 429 storm on the jsonrpc entry held the REST entry and the same name on other chains. holdoff.ProviderKey scopes the name to the chain and api-interface. Every caller that keys by provider uses it: relay path, provider selection and the url walk, re-verifier, recovery probe, and the chain tracker's relay read. Escalation semantics are unchanged within an entry. The url-only keys (chain tracker, WS and gRPC subscriptions) are untouched. The metric `provider` label stays the plain name. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… the hold-off lock-order comment (MAG-4732) relayURLs and the url walk each spelled out which url may serve a request (internal path, relay-capable, add-on and extensions). servesRequest is now the one rule both call, so the rate-limit read and the pick cannot drift apart. Usability stays with each caller: the walk relaxes it for the reconnect probe and the last chance. No behaviour change. The lock-order comment on heldOffProviderInfoLocked now lists the real chain: csm.lock, cswp.Lock, endpoint.mu, with the registry lock as a leaf. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ds 429s before answers (MAG-4732, MAG-4733) A re-verify pass skipped a provider while any of its urls was held, so one url busy with relay 429s kept a recovered sibling refused for as long as its hold lasted. The pass is now skipped only when every url is held. Otherwise it asks the free urls and leaves the held ones out: each reads as only rate-limited (chainlib.ErrNotAskedHeldOff), so it keeps its last verdict, and nothing is sent to it. The pass wrote its results one url at a time in config order. Two held urls escalate the entry and an answer drops the escalation, so a url that answered in the pass was held or free depending on where it sat in the config, and the same url configured twice (base and a standalone add-on) had its 429 cleared by the second entry. Results are now merged per url and every 429 is recorded before any answer. A websocket url's 429 is held under the url alone, the WS path's key, so it never counts toward the entry's escalation. The metric help text and docs/METRICS.md now describe escalation per entry. BREAKING CHANGE: a rate-limit escalation (two urls of one provider held at once) now holds that provider entry only (its name on one chain and api-interface), not the same name on every chain and interface. The `escalated` event of smartrouter_rate_limit_holdoffs_total keeps its plain-name `provider` label, but no longer means every chain stopped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…on refusals (MAG-4733) A url the pass skipped because it is held off, or saw only rate-limited at the url level, kept its url verdict but lost every per-service refusal: those were kept only for a service throttled in this pass, and a skipped url has no checks. A pruned node refused archive at boot got archive admitted again on the next pass after a relay 429, without being asked. ProviderAdmission now remembers the url at each throttled position, and KeepingThrottledFrom keeps every service refusal the previous verdict gave that url. Found in review on #618. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…etry pass too (MAG-4733) Main reworked the re-verifier into probeReverification / decideReverification with per-check retries (MAG-3950), so the per-url hold-off of this PR is re-applied on that shape instead of rebased line by line: - A probe skips the urls held off after a rate limit and asks the rest, on the epoch pass and on a retry (ValidateCollectionsAgain takes the skip set). Only a provider whose every url is held skips the probe. - Each url's result is recorded on its own entry, rate limits before answers. A url the pass sent no request to (held, or a retry kept its verdicts without dialing it: ProviderAdmission.URLAsked) changes nothing, and a pass that never ran (a config error) clears nothing. - A skipped url keeps its warnings as well as its refusals. - A retry waits until the first of the provider's urls is free, not until the first url is (heldOffUntil). The gRPC rate-limit relay test from main reads the scoped provider key. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…scoped provider key (MAG-4732) The probe loop from MAG-4387 leaves a held-off, stale endpoint out of its provider's QoS sample, reading the relay path's (provider, url) entry under the plain provider name. The relay path records under holdoff.ProviderKey, so the probe never saw a relay hold and scored the busy url availability 0 again. EndpointWithDirectConnection carries the provider's HoldoffKey, set by the session manager, and the probe reads it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…a dead one (MAG-4734) (#622) * fix(lavasession): the last chance lends a url that can answer before a dead one (MAG-4734) The last chance lends one url of the blocked provider it hands out, and it took the first url in config order that fits the request. A switched-off url listed first was lent while its sibling would have answered: a head-stalled url (alive, a block behind), or a url already back in rotation while the provider's block has not lifted yet (the epoch switch). The request's one chance was spent on the dead url. The walk now orders the urls for the last chance by the kind of state they are in: in rotation, then held by a state reason, then switched off (lendRankLocked). It reads the kind, not a reason's name, so a reason added later takes the rank of its kind. The sort is stable, so the rate-limit order (MAG-4732) and config order hold within a rank. Normal selection and the reconnect probe are unchanged, and the provider's url list is never reordered. The once-per-request limit is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(lavasession): build the last chance's url order in one slice (MAG-4734) gocritic flagged `ordered = append(free, held...)` (appendAssign): the result of appending to one slice was assigned to another. The urls that are not held off now go straight into ordered and the held ones are appended after them. Same order, no behaviour change; golangci-lint is clean on protocol/lavasession again. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Took 52 minutes * docs(lavasession): what the last chance's url rank does to the once-per-request limit (MAG-4734) The first commit says the once-per-request limit is unchanged. The rule is unchanged: the request's one chance is taken only when the lent url is switched off, so a lent head-stalled url takes none (kept by decision on MAG-4734). How often a head-stalled url is lent does change. Before, a blocked provider whose switched-off url is listed first was lent that url, and the request's one chance was spent. Now its head-stalled url is lent first and takes no chance; when it fails, the retry can be lent the head-stalled url of the next blocked provider. One request can now be lent one head-stalled url per blocked provider, where before it was lent one switched-off url. The walk's comment said the request is lent one url; it is the provider that is lent one url per request. RATE-LIMIT-HOLDOFF.md said a url that is not held off always goes before one that is; in the last chance the url's state comes first and the hold-off orders the urls within a rank. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: avitenzer <tenzer@clara.co.uk>
…rguments (MAG-4732) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ials (MAG-4733) Since MAG-4214 (#659) a re-verify pass verifies the dialed urls: on a chain with internal paths a bare url becomes one url per path. The hold-off entries, the skip set and the per-url pass results were still positions into the configured list, so on TON, STRK, AVAX and Monero a 429 from a generated url was never held, a relay hold on one was not honoured, and a held pinned url was asked while a free sibling was skipped. holdoffEntries and heldOffUntil now build from reverifyNodeUrls, the list the pass verifies (dialedNodeUrls). The relay path already records a 429 under the dialed url, so the keys line up and the per-path known limitation goes away. Found in review (PR #618). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ccadd20 to
07dc208
Compare
avitenzer
left a comment
There was a problem hiding this comment.
Re-reviewed at 07dc2080, rebased on main 36b0cd82, all 11 commits signed.
07dc2080does what the review asked:holdoffEntries, the skip set,urlPassResultsandheldOffUntilnow indexreverifyNodeUrls, the dialed list the pass verifies, with the configured list only when there is no parser. The wss detection reads the dialed url too. The three TON cases from my comment are inTestReverifyHoldoff_UsesTheDialedURLswith a fourth forheldOffUntil; withreverifyNodeUrlsreturning the configured list, all four go red.restrictsToPathLockedcarries main's lost-path rule, so the walk andrelayURLspass such a provider over together. Main's two tests have the new arguments (265d9ed0).- From a clean worktree of
07dc2080: build, vet and gofmt clean;go test -count=1is green on holdoff, chainlib, routersession, rpcsmartrouter, chaintracker and endpointstate. - Description: Jira line names MAG-4732, MAG-4733 and MAG-4734, the breaking footer is there, the known-limitation paragraph is gone, and the docs row says the urls are the dialed ones. The scoping ruling is accepted on MAG-4732.
Approving. One thing outside the code: no workflow ran on this head, as on every PR pushed between the 12th and this morning. The merge needs govulncheck and unit-tests on 07dc2080, so CI has to be re-triggered before the squash.
Bring in the hold-off change (#618, MAG-4732/4733/4734), which landed after the previous merge and touches the same four files. No conflicts; IsUsable and inRotation keep their meaning, so the two guards this branch adds read the same state they did. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Resolves the conflicts with #695 (MAG-2972 rename) and #618 (MAG-4732): - recovery_probe.go, recovery_probe_test.go, endpoint_relay_probe_test.go: deleted, as this PR does. Main's edits to them were the rename plus #618's two-line hold-off key scoping inside the replay, which goes with the replay. - consumer_types.go, rpcsmartrouter_server.go, relay_rate_limit_test.go: the replay blocks main renamed stay deleted. - direct_http_status_body_test.go: the relay-path half of the 5xx-body test, on the routersession package name. - error_codes.go: main's RouterError names with this PR's comment, which no longer mentions replay evidence. - docs/RATE-LIMIT-HOLDOFF.md: main's #618 rows without the recovery-probe row. - direct_conn_fake_test.go, relay_inner_direct_endpoint_health_test.go: this PR's own test code moved to the routersession package name. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Description
A 429 is recorded per (provider, url). Two things went wrong with how selection read it:
The rule after this PR
A provider is held only while every url of it that could serve this request is held. "Could serve" means usable (not disabled, not head-stalled), able to answer a relay (not a ws url), and serving the request's add-on, extensions and internal path. A url with no entry is free. Inside the provider, urls that are not held are handed out before held ones. Nothing is dropped: when every url is held, one is still used, as before.
/v2and/v3urls/v2url/v2requests onlyEscalation is per entry now. The registry keyed a provider by name alone, but a name is unique per chain and api-interface only, so two held urls of one entry escalated every entry sharing the name.
holdoff.ProviderKey(chain, interface, name)scopes the key; every caller that keys by provider uses it. Inside one entry escalation is unchanged (two held urls hold its other urls for every reader, probes and tracker polls included). This drops the cross-chain reach MAG-2947 gave escalation. MAG-2947's settled design widened escalation to the provider name on purpose ("a vendor cap is account-wide, and per-URL keying alone would let the process keep hammering the account through its other chains"), and this reverses that. Ruled 2026-11-10: keep the per-entry scope (recorded on MAG-4732). The fleet runs one router process per chain and network, each with its own hold-off state, so a vendor's 429s on one chain were never seen by another chain; in production the only change is across api-interfaces of one chain.The re-verifier holds the url that answered 429 (MAG-4733). It filed every 429 under the provider's first url. With selection counting urls one by one, that held the wrong url, or nothing when the first url is a websocket. The pass already knows which urls met a rate limit; each is held on its own url with its own Retry-After, and each url that answered clears its own entry. A url that is held is not asked: it reads as only rate-limited and keeps its last verdict, including its add-on refusals and warnings, while the provider's other urls are still checked. The per-check retry pass skips held urls too, and a retry waits until the first url is free. A url the pass sent no request to changes nothing in the hold-off, and a pass that never ran (a config error) clears nothing. The pass is skipped only when every url of the provider is held. Results are merged per url and every 429 is recorded before any answer, so the outcome does not depend on config order. A websocket url's 429 is held under the url alone (the WS path's key) and does not count toward the entry's escalation.
Unchanged: hold durations, the soonest-to-expire fallback between providers, the write broadcast ignoring the hold-off (MAG-4388), the chain tracker / WS / gRPC url-only keys, and the hold-off metrics'
providerlabel (still the plain name).Code
protocol/holdoff/holdoff.go:ProviderReadyAt(provider, urls)counts only the listed urls.protocol/routersession/consumer_types.go:relayURLs(the urls that can serve a request, the walk's own filter),endpointsByHoldoffLocked(free urls first),restrictsToPathLocked(the walk's existing path rule, pulled out so both use it). The walk takes the registry; the reconnect probe passes nil and is unchanged.protocol/routersession/consumer_session_manager.go:providerHeldOffreplaces the fourProviderReadyAt(address)calls. The request'sinternalPathis passed down the provider selection chain to reach them (one extra argument, no new logic)./debug/provider-routingreports holds as a base-collection request sees them.protocol/holdoff/holdoff.go,metrics.go:ProviderKey; the metric label strips the scope.rpcsmartrouter_server.go,holdoffProviderKey), recovery probe, re-verifier, session manager (holdoffKey,urlHeldOff), and the chain tracker's relay read (endpointstate/endpoint_monitor.go).protocol/rpcsmartrouter/spec_reverifier.go: per-url hold and clear (holdoffURLKeys,holdoffEntries), results written byrecordPassHoldoffs(merged per url, 429s before answers), held urls passed as a skip set torevalidateProviderCollections(first pass and per-check retry),urlPassResultsdecides which urls the pass actually asked;chainlib.ProviderAdmission.URLRateLimit(index)gives each url its own rate limit andURLAsked(index)says whether the url got a request.protocol/rpcsmartrouter/verification_retry.go:heldOffUntilwaits for the first free url, not url 0.protocol/chainlib/chain_fetcher.go:ValidateCollectionsExcept(ctx, skip)andValidateCollectionsAgain(ctx, previous, only, skip)leave the skipped urls unasked; each reads as rate-limited withErrNotAskedHeldOff, and a connection whose every url is skipped is not dialed.KeepingThrottledFromkeeps a skipped url's earlier add-on refusals and warnings, not only its url refusal.rpcsmartrouter_server.go): reads the relay hold-off under the scoped key, carried onroutersession.EndpointWithDirectConnection.HoldoffKey.docs/RATE-LIMIT-HOLDOFF.md: the query, tiered keying, the hot-path and re-verification rows, the metric label.docs/METRICS.mdand theholdoff/metrics.gohelp text: escalation per entry.The held urls are not sorted by expiry: two held urls of one entry escalate it, which holds both until the longer ends, so they always tie.
Tests
protocol/routersession/holdoff_url_pick_test.go: one test per table row, plus "every url held still hands one out" and "no registry keeps config order".protocol/holdoff/holdoff_test.go: a listed url with no entry frees the provider; a held url that is not listed does not count.http://primary-b/url), which the old rule accepted. They now record under the provider's real url (holdOffProviderhelper). The other test edits only add the new argument.TestProviderKey_EscalationStaysOnOneEntry(registry) andTestHoldoff_EscalationStaysOnTheEntryThatWasLimited(selection: same name on rest and on another chain stays free). Both fail with a name-only key.TestApplyReverification_ARateLimitedUrlBacksOffAndChangesNothingnow asserts the 429 is held on the url that answered it, andTestApplyReverification_ARateLimitBehindAWebsocketFirstUrlHoldsTheRelayUrlcovers a wss url listed first. Both fail on the old re-verifier.go build ./...,go vet, andgo testforholdoff,chainlib,chaintracker,endpointstate,routersessionandrpcsmartroutergreen locally, exceptTestBlockReason_SurvivesTheEpochTransition, which is flaky on main too (3 of 40 runs on origin/main, same message). Not run against real nodes.Context: found while mapping provider vs url state for MAG-4219 (closed PR #576). Under the product rule from docs#78 (urls of one entry share one state), the hold-off was still wrong in both directions.
Review follow-up (edbd8d4)
!in the title, the footer below, metric help text anddocs/METRICS.mdupdated. Public docs: docs(failover): a rate-limit escalation holds one provider entry, not every chain (MAG-4732) docs#86 (draft), to merge with the release that ships this.reverify_url_holdoff_test.go(7). The two re-verify tests for the skip rule fail on 3b1ba81.Rebase on main (36b0cd8)
lavasessionis nowroutersession). fix(lavasession): the last chance lends a url that can answer before a dead one (MAG-4734) #622 (MAG-4734) was merged into this branch and is carried over as fde07e5. Every commit is signed.dialedNodeUrls, so on a chain with internal paths a bare url is one url per path.holdoffEntriesandheldOffUntilnow build from that same list (reverifyNodeUrls), so the entries, the skip set and the per-url results share its positions, and the keys match what the relay path records (base + path). This also removes the earlier known limitation, where generated urls were not held or cleared by re-verify.TestReverifyHoldoff_UsesTheDialedURLs(4 subtests, all red with the configured list).restrictsToPathLockedalso passes over a provider for a path inUnservedInternalPaths, so the walk andrelayURLsagree. Main's two MAG-4214 tests got the new arguments (265d9ed).TestApplyReverification_AHeldURLKeepsItsAddonRefusals,TestKeepingThrottledFrom_AURLNotCheckedKeepsItsServiceRefusals.ValidateCollectionsAgain,URLAsked,heldOffUntil= first free url.TestURLAsked,TestHeldOffUntil_OneFreeURLIsEnough,TestProbeReverification_ARetryLeavesAURLItDidNotAskAsItWas,TestURLPassResults_APassThatNeverRanAnswersNoURL.HoldoffKey.go build ./...,go vet ./..., andgo test -count=1forholdoff,chainlib,routersession,rpcsmartrouter,chaintracker,endpointstateandcommonpass at 07dc208. The intermediate commits don't each build on their own, so squash-merge.Jira ticket
Jira ticket: MAG-4732
Also fixes MAG-4733 (second and fifth commits, and 07dc208) and MAG-4734 (fde07e5, from #622).
Squash message should name all three: MAG-4732, MAG-4733, MAG-4734.
BREAKING CHANGE: a rate-limit escalation (two urls of one provider held at once) now holds that provider entry only (its name on one chain and api-interface), not the same name on every chain and interface. The
escalatedevent ofsmartrouter_rate_limit_holdoffs_totalkeeps its plain-nameproviderlabel, but no longer means every chain stopped.🤖 Generated with Claude Code