Skip to content

fix(routersession,holdoff)!: hold a provider off only while every url that can serve the request is held (MAG-4732, MAG-4733, MAG-4734) - #618

Merged
avitenzer merged 11 commits into
mainfrom
fix/holdoff-url-pick-mag-4732
Nov 15, 2026
Merged

avitenzer merged 11 commits into
mainfrom
fix/holdoff-url-pick-mag-4732

Conversation

@Tomelia1999

@Tomelia1999 Tomelia1999 commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Description

A 429 is recorded per (provider, url). Two things went wrong with how selection read it:

  • Holds too much. Selection asked only whether the provider name was held. The registry only knows urls that answered 429, so when url A got a 429 and url B never had one, the provider read as held and B was skipped too. A provider name is unique per chain and api-interface, not across them, so a 429 on the jsonrpc entry's url also held the same-named REST entry.
  • Holds too little. Inside a provider the url walk never read the hold-off. When B had an old, expired 429 and A was held now, the provider read as free and the walk handed out A because it is listed first.

The rule after this PR

A provider is held only while every url of it that could serve this request is held. "Could serve" means usable (not disabled, not head-stalled), able to answer a relay (not a ws url), and serving the request's add-on, extensions and internal path. A url with no entry is free. Inside the provider, urls that are not held are handed out before held ones. Nothing is dropped: when every url is held, one is still used, as before.

setup 429 on before after
one jsonrpc entry, urls A and B A provider skipped, B unused B serves
same, B had an old expired 429 A A used while held B serves
jsonrpc entry (https + wss) and a rest entry, same name jsonrpc https url jsonrpc and REST skipped only jsonrpc held, REST serves
TON /v2 and /v3 urls /v2 url provider held for every path held for /v2 requests only
jsonrpc entry with 3 urls, rest entry and another chain, same name 2 jsonrpc urls escalation held the name everywhere jsonrpc entry held (escalation), REST and other chain serve
re-verifier probe, urls A and B B filed under A: provider held (before this PR) or B used (with the first commit alone) B held, relays go to A

Escalation is per entry now. The registry keyed a provider by name alone, but a name is unique per chain and api-interface only, so two held urls of one entry escalated every entry sharing the name. holdoff.ProviderKey(chain, interface, name) scopes the key; every caller that keys by provider uses it. Inside one entry escalation is unchanged (two held urls hold its other urls for every reader, probes and tracker polls included). This drops the cross-chain reach MAG-2947 gave escalation. MAG-2947's settled design widened escalation to the provider name on purpose ("a vendor cap is account-wide, and per-URL keying alone would let the process keep hammering the account through its other chains"), and this reverses that. Ruled 2026-11-10: keep the per-entry scope (recorded on MAG-4732). The fleet runs one router process per chain and network, each with its own hold-off state, so a vendor's 429s on one chain were never seen by another chain; in production the only change is across api-interfaces of one chain.

The re-verifier holds the url that answered 429 (MAG-4733). It filed every 429 under the provider's first url. With selection counting urls one by one, that held the wrong url, or nothing when the first url is a websocket. The pass already knows which urls met a rate limit; each is held on its own url with its own Retry-After, and each url that answered clears its own entry. A url that is held is not asked: it reads as only rate-limited and keeps its last verdict, including its add-on refusals and warnings, while the provider's other urls are still checked. The per-check retry pass skips held urls too, and a retry waits until the first url is free. A url the pass sent no request to changes nothing in the hold-off, and a pass that never ran (a config error) clears nothing. The pass is skipped only when every url of the provider is held. Results are merged per url and every 429 is recorded before any answer, so the outcome does not depend on config order. A websocket url's 429 is held under the url alone (the WS path's key) and does not count toward the entry's escalation.

Unchanged: hold durations, the soonest-to-expire fallback between providers, the write broadcast ignoring the hold-off (MAG-4388), the chain tracker / WS / gRPC url-only keys, and the hold-off metrics' provider label (still the plain name).

Code

  • protocol/holdoff/holdoff.go: ProviderReadyAt(provider, urls) counts only the listed urls.
  • protocol/routersession/consumer_types.go: relayURLs (the urls that can serve a request, the walk's own filter), endpointsByHoldoffLocked (free urls first), restrictsToPathLocked (the walk's existing path rule, pulled out so both use it). The walk takes the registry; the reconnect probe passes nil and is unchanged.
  • protocol/routersession/consumer_session_manager.go: providerHeldOff replaces the four ProviderReadyAt(address) calls. The request's internalPath is passed down the provider selection chain to reach them (one extra argument, no new logic). /debug/provider-routing reports holds as a base-collection request sees them.
  • protocol/holdoff/holdoff.go, metrics.go: ProviderKey; the metric label strips the scope.
  • Scoped key at the relay path (rpcsmartrouter_server.go, holdoffProviderKey), recovery probe, re-verifier, session manager (holdoffKey, urlHeldOff), and the chain tracker's relay read (endpointstate/endpoint_monitor.go).
  • protocol/rpcsmartrouter/spec_reverifier.go: per-url hold and clear (holdoffURLKeys, holdoffEntries), results written by recordPassHoldoffs (merged per url, 429s before answers), held urls passed as a skip set to revalidateProviderCollections (first pass and per-check retry), urlPassResults decides which urls the pass actually asked; chainlib.ProviderAdmission.URLRateLimit(index) gives each url its own rate limit and URLAsked(index) says whether the url got a request.
  • protocol/rpcsmartrouter/verification_retry.go: heldOffUntil waits for the first free url, not url 0.
  • protocol/chainlib/chain_fetcher.go: ValidateCollectionsExcept(ctx, skip) and ValidateCollectionsAgain(ctx, previous, only, skip) leave the skipped urls unasked; each reads as rate-limited with ErrNotAskedHeldOff, and a connection whose every url is skipped is not dialed. KeepingThrottledFrom keeps a skipped url's earlier add-on refusals and warnings, not only its url refusal.
  • Probe loop (MAG-4387, rpcsmartrouter_server.go): reads the relay hold-off under the scoped key, carried on routersession.EndpointWithDirectConnection.HoldoffKey.
  • docs/RATE-LIMIT-HOLDOFF.md: the query, tiered keying, the hot-path and re-verification rows, the metric label. docs/METRICS.md and the holdoff/metrics.go help text: escalation per entry.

The held urls are not sorted by expiry: two held urls of one entry escalate it, which holds both until the longer ends, so they always tie.

Tests

  • New protocol/routersession/holdoff_url_pick_test.go: one test per table row, plus "every url held still hands one out" and "no registry keeps config order".
  • protocol/holdoff/holdoff_test.go: a listed url with no entry frees the provider; a held url that is not listed does not count.
  • Existing tests recorded 429s under made-up urls (http://primary-b/url), which the old rule accepted. They now record under the provider's real url (holdOffProvider helper). The other test edits only add the new argument.
  • Escalation per entry: TestProviderKey_EscalationStaysOnOneEntry (registry) and TestHoldoff_EscalationStaysOnTheEntryThatWasLimited (selection: same name on rest and on another chain stays free). Both fail with a name-only key.
  • Re-verifier: TestApplyReverification_ARateLimitedUrlBacksOffAndChangesNothing now asserts the 429 is held on the url that answered it, and TestApplyReverification_ARateLimitBehindAWebsocketFirstUrlHoldsTheRelayUrl covers a wss url listed first. Both fail on the old re-verifier.
  • Tests that read the registry by plain name now use the scoped key.
  • go build ./..., go vet, and go test for holdoff, chainlib, chaintracker, endpointstate, routersession and rpcsmartrouter green locally, except TestBlockReason_SurvivesTheEpochTransition, which is flaky on main too (3 of 40 runs on origin/main, same message). Not run against real nodes.

Context: found while mapping provider vs url state for MAG-4219 (closed PR #576). Under the product rule from docs#78 (urls of one entry share one state), the hold-off was still wrong in both directions.

Review follow-up (edbd8d4)

  • Re-verify asks only free urls (MAG-4733). A pass is skipped only when every url is held. Otherwise the held urls are not asked: each reads as only rate-limited, keeps its last verdict, and gets no request. The other urls are checked, so a url refused for a capability problem comes back while a sibling is busy.
  • Re-verify results do not depend on config order. Results are merged per url (one url configured twice is one url), then every 429 is recorded before any answer. A url that answered in the pass is free in every order, and a second entry of the same url no longer clears the first one's 429.
  • A websocket url's 429 is held under the url alone (the WS path's key), so it never counts toward the entry's escalation.
  • Escalation per entry is marked as breaking: ! in the title, the footer below, metric help text and docs/METRICS.md updated. Public docs: docs(failover): a rate-limit escalation holds one provider entry, not every chain (MAG-4732) docs#86 (draft), to merge with the release that ships this.
  • Tests: reverify_url_holdoff_test.go (7). The two re-verify tests for the skip rule fail on 3b1ba81.

Rebase on main (36b0cd8)

Jira ticket

Jira ticket: MAG-4732

Also fixes MAG-4733 (second and fifth commits, and 07dc208) and MAG-4734 (fde07e5, from #622).

Squash message should name all three: MAG-4732, MAG-4733, MAG-4734.

BREAKING CHANGE: a rate-limit escalation (two urls of one provider held at once) now holds that provider entry only (its name on one chain and api-interface), not the same name on every chain and interface. The escalated event of smartrouter_rate_limit_holdoffs_total keeps its plain-name provider label, but no longer means every chain stopped.

🤖 Generated with Claude Code

@avitenzer avitenzer left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Tomelia1999 I reviewed this at 3b1ba813. I don't think it can be approved yet. Details are in the inline comments.

The url pick inside a provider looks right, and I checked that the tests guard it. With the free-urls-first ordering disabled, TestHoldoff_OneHeldURLLeavesTheProviderToItsSibling and TestHoldoff_ExpiredSiblingIsPickedBeforeAHeldURL both go red. The relay path, the session manager and the chain tracker all build the provider key from the same endpoint's chain and api-interface, so the keys match. On the head, go build and go vet ./... are clean, and the tests pass for holdoff, chainlib, chaintracker, endpointstate, lavasession and rpcsmartrouter. Merged locally into current main (#624 touches consumer_types.go and consumer_session_manager.go too), it merges cleanly, and lavasession and holdoff pass.

Blocking:

  1. A single re-verify pass can hold a healthy url, depending on config order (inline on spec_reverifier.go:358).
  2. The re-verify skip rule goes against MAG-4733 and the PR gives no reason (inline on :325).
  3. Limiting escalation to one chain and api-interface changes publicly documented behaviour, but nothing marks it as breaking (inline on holdoff.go:90).
  4. CI never ran on 3b1ba813. The last runs are on f5a1bd93. The branch is also behind main, so updating it will trigger CI.

Non-blocking:

  • The commit message of d3f866b says a rate limit not tied to a url "holds them all, as it held the provider before". With 2+ urls it now holds the whole entry, which is more than before. Only the validateFn test hook reaches that path, since servingError always ties a throttle to a url.
  • Main's #624 added inRotation, which is another copy of "which url can take a relay", next to servesRequest. Worth deciding whether they should be one rule.

utils.LogAttr("url_index", index),
utils.LogAttr("holdoff", delay.String()),
)
case answered:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: the outcome of one re-verify pass depends on the order of the urls in the config.

This loop calls RecordRateLimit / RecordAnswer per url in config order. RecordRateLimit holds the whole entry once 2 urls are held. RecordAnswer clears that hold (holdoff.go, clearLocked resets escalatedUntil). So a pass where one url answers and two return 429 gives different results depending on the order:

  • order [A answers, B 429, C 429]: the entry is held, so HeldOff(key, A) is true and ProviderReadyAt(key, [A]) reports held, even though A answered in this same pass.
  • order [B 429, C 429, A answers]: A's answer clears the hold, and A is free.

I reproduced this with a throwaway test that calls the registry in both orders (NewRegistry, the same ProviderKey). Main recorded only one url per pass, so a single pass could never hold the whole entry. Once it is held, the skip at :325 stops the next pass from clearing it. The hold also applies to tracker polls and to the recovery probe.

Related: holdoffURLKeys returns every configured url, wss included, and each one is recorded under the provider key. The relay path never records a ws url there, and relayURLs leaves them out. So an https url and a wss url both returning 429 in one pass can hold the whole entry, including the https url relays use.

Suggested fix: apply the pass's answers after its rate limits, or don't hold the whole entry from a single pass. Also add a test with 3 urls (one answering) in both orders, and one with an https url and a wss url both rate limited.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in b38b5e04. The pass now merges its results per url, records every 429 first, and only then records the answers, so the result no longer depends on config order (recordPassHoldoffs). A wss url is held under its own key, the one the WS path reads, so it no longer counts toward the entry escalation. The same url configured twice also keeps its 429 now. Tests: TestRecordPassHoldoffs_AnAnsweringURLIsFreeInEveryConfigOrder (3 urls, both orders), _AWebsocketRateLimitDoesNotHoldTheEntry, _TheSameURLTwiceKeepsItsRateLimit.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified at edbd8d49. With the answer loop moved before the rate-limit loop in recordPassHoldoffs, TestRecordPassHoldoffs_AnAnsweringURLIsFreeInEveryConfigOrder goes red. With the per-url merge disabled, _TheSameURLTwiceKeepsItsRateLimit goes red. With the wss url keyed under the provider, _AWebsocketRateLimitDoesNotHoldTheEntry goes red. Resolved from my side.

// The provider on this chain and api-interface, as relay selection keys it.
holdoffKey := holdoff.ProviderKey(inputs.rpcEndpoint.ChainID, inputs.rpcEndpoint.ApiInterface, p.Name)
keys := holdoffURLKeys(p)
for _, key := range keys {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: this goes against MAG-4733 and gives no reason.

The ticket's fix direction says: "Skip the re-verify only when every url is held." This loop skips the whole pass while any url is held under the provider key. The PR body and RATE-LIMIT-HOLDOFF.md say that it does this, but not why.

With relay 429s now recorded per url (the point of this PR), one busy url can keep the provider from being re-verified for as long as its hold lasts. During that time a sibling url that was refused for a capability problem can't be re-admitted. Please either follow the ticket (skip only when every url is held, or probe only the urls that aren't held), or put the reason in the PR body and the ticket.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in b38b5e04, following the ticket. Held urls are not asked, the free urls are, and the pass is skipped only when every url is held. A url that was not asked keeps its last verdict and its hold-off. Tests: TestApplyReverification_AHeldURLIsNotAskedAndItsSiblingIs, _EveryURLHeldSkipsThePass, and _ARecoveredURLComesBackWhileItsSiblingIsHeld (the MAG-4733 case: a refused url is re-admitted while its sibling is held).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified at edbd8d49. Skipping the pass when any url is held turns AHeldURLIsNotAskedAndItsSiblingIs and ARecoveredURLComesBackWhileItsSiblingIsHeld red, and so does passing nil as the skip set. Resolved from my side, but skipping a url introduced a new issue with per-service refusals (see my inline comment on chain_fetcher.go:917).

// only, so keyed by the name alone two held urls of one entry escalated every entry that
// shares the name — a 429 storm on jsonrpc held REST too (MAG-4732). An empty name stays
// empty, so a caller's "no provider" still reads as one.
func ProviderKey(chainID, apiInterface, name string) string {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: this is a change to publicly documented behaviour.

Limiting the escalation to one chain and api-interface contradicts the public docs:

  • Magma-Devs/docs docs/configuration/failover/retry.md:91-94: "the whole provider is held off on every chain".
  • Magma-Devs/docs docs/reference/metrics.md:102: the escalated alert means the router "stopped calling that provider on every chain".

Under our rule for the documented router surface, that makes this a breaking change. It needs ! in the PR title and squash message, a BREAKING CHANGE: footer, and a docs follow-up for both pages. The tickets don't ask for this scoping either, so it would be good to add a line about it to MAG-4732.

In this repo the wording is also out of date: the metric help text at holdoff/metrics.go:29 ("widened to the provider name") and docs/METRICS.md:409 / :423 still describe a name-wide escalation.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. The title has !, the body has a BREAKING CHANGE: footer, and the squash message will carry both. The metric help text (holdoff/metrics.go) and docs/METRICS.md now describe the per-entry escalation. The public docs follow-up is Magma-Devs/docs#86 (both pages). It stays a draft until the release that ships this.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Marking it as breaking is done (title, footer, metric help, METRICS.md, docs#86). Thanks. The open part is the decision itself: this reverses MAG-2947's account-wide rationale, so it needs a product ruling before merge. I've noted that in the description and in the review summary.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ruling: we keep the per-entry scope.

The "other chains" case in MAG-2947 doesn't happen in the fleet. The chart creates one Deployment per chain and network (<id>-router), so every chain runs in its own process with its own hold-off registry, and each replica pod has its own too. A 429 on a vendor in the ETH router was never seen by the BTC router, before or after this PR. Name-wide escalation across chains only happens when several chains run in one process, like the local multichain example.

So in production the only thing this changes is across api-interfaces of one chain. Before, two held rest urls of a vendor also held its tendermintrpc and grpc entries. Now they don't. Each url that answers 429 is still held on its own, so another interface loses traffic only when it gets 429s itself. The ruling is recorded in MAG-4732.

// the url itself, the key the relay path records a 429 under for the endpoint it dialed. A 429
// is held on the url that answered it, so relay selection, which counts a provider's urls one by
// one, keeps off that url and not a sibling (MAG-4733).
func holdoffURLKeys(p *lavasession.RPCStaticProviderEndpoint) []string {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: re-verify holds and clears the configured urls (p.NodeUrls), but relays use the urls expandInternalPaths generates from them (rpcsmartrouter.go:2480). On a chain with internal paths, a root url with no internal path of its own produces base + path endpoints. So a 429 that re-verify sees on the base url holds only the root endpoint, and the per-path urls on the same host stay free. And a relay 429 on a generated url is never seen by this skip or clear. The TON row of the PR's table is tested only with hand-built endpoints in holdoff_url_pick_test.go, so this path isn't covered. Probably fine, since 429s are per host in practice, but worth either a sentence in the PR body or a test that goes through the expansion.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, no change in this PR. Vendor rate limits are per host and account, so in practice a 429 on the base url and on its per-path urls arrives together, and each relay 429 is still held on the url that got it. The gap is that re-verify does not see the generated per-path urls, and that the TON row has no test through expandInternalPaths. I am keeping it as a known limitation and not widening this PR.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. I added it to the description as a known limitation, along with the missing TON test through expandInternalPaths.

@Tomelia1999 Tomelia1999 changed the title fix(lavasession,holdoff): hold a provider off only while every url that can serve the request is held (MAG-4732) fix(lavasession,holdoff)!: hold a provider off only while every url that can serve the request is held (MAG-4732) Nov 8, 2026

@avitenzer avitenzer left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Tomelia1999 I re-reviewed at edbd8d49. Thanks for the quick turnaround: the four blockers from my last review are fixed, and I've replied on each thread. It still can't be approved, for the reasons below.

What I checked

  • From a clean worktree of edbd8d49: go build ./... and go vet are clean, and go test passes for holdoff, lavasession, rpcsmartrouter, chainlib, chaintracker and endpointstate.
  • The new tests guard the fixes. Each of these mutants turns at least one of them red: skip the pass when any url is held, ask the held urls anyway, key a wss 429 under the provider, record answers before 429s, and let a later duplicate url overwrite an earlier 429.

Blocking

  1. New in edbd8d4: a url the pass skips loses its per-service refusals, so a pruned node gets archive re-admitted (inline on chain_fetcher.go:917, with a repro).
  2. The escalation scope needs a product ruling before merge. Marking it as breaking was right, but this reverses the design MAG-2947 settled on purpose: "a vendor cap is account-wide, and per-URL keying alone would let the process keep hammering the account through its other chains." MAG-4732 and MAG-4733 record no ruling, and the description said "by decision on this PR". I've updated the description to say a ruling is pending.
  3. Conflicts with main (78 commits behind; holdoff/metrics.go, consumer_session_manager.go, and three lavasession test files). Most of it is mechanical, since main threads triedByThisRequest where this PR threads internalPath. One hunk isn't: getValidConsumerSessionsWithProvider on main now carries the MAG-4829 last-chance retry block (#637) where this branch returns directly. The resolution has to keep that block. metrics.go also has to keep main's KeepUnregisteredOnFailure / RegisterOrReuse registration.
  4. CI hasn't run on edbd8d49 ("no checks reported", because of the conflict). It will run once the branch is updated.

Description: I edited the description to match edbd8d49. The re-verify paragraph still described the old skip-on-any rule. I added chain_fetcher.go (ValidateCollectionsExcept, ErrNotAskedHeldOff) and recordPassHoldoffs to the code list, the per-path-url known limitation from our thread, and that MAG-4733 is fixed in the second and fifth commits. Your replies point to b38b5e04, but that commit doesn't exist on the branch; the work is in edbd8d49.

#622 is stacked on this PR and keeps waiting for it.

// notAsked is the share of a pass of a url the pass leaves out: rate-limited, with nothing
// sent (ErrNotAskedHeldOff).
func (g *connectionGroup) notAsked(url common.NodeUrl) nodeUrlOutcome {
outcome := nodeUrlOutcome{url: url, served: url, err: ErrNotAskedHeldOff, head: spectypes.NOT_APPLICABLE}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking (new in edbd8d4): a url the pass skips loses the add-ons it was refused, so a pruned node gets archive back.

notAsked makes the url throttled at the url level (throttledURLs[index]) and adds no checks. KeepingThrottledFrom (chain_fetcher.go:439) then keeps the url's previous url refusal (previous.refusedURLs[index]), but copies a previous service refusal (previous.refused[url+path+service]) only when the same service is in this pass's throttledServices. A skipped url has none, so every per-service refusal on it is dropped.

Repro: url A is a full node, url B is a pruned node with Addons: [archive]. At boot archive is refused on B and B is kept for the base collection. B then gets a relay 429. The next pass checks A, skips B, and logs re-verify: rebuilding static provider, admitted services changed: archive is admitted on B again. Archive requests can then reach the pruned node until B's hold ends and a later pass asks it and refuses archive again. Since B is the only archive url, the hold doesn't keep archive traffic off it either, because a provider whose every capable url is held still hands one out.

Test (fails at edbd8d4, passes at 3b1ba81)
func TestReview_HeldURLKeepsItsServiceRefusal(t *testing.T) {
	up := newFakeUpstreams(t)
	provider := staticProvider("ETH1",
		common.NodeUrl{Url: up.httpURL("/a/" + upArchive)},
		common.NodeUrl{Url: up.httpURL("/b/" + upPruned), Addons: []string{"archive"}},
	)
	parser := liveParser(t, "ETH1", true)
	admissions := newAdmissionRegistry()
	boot, err := validateProviderCollections(t.Context(), provider, parser, 20*time.Second)
	require.NoError(t, err)
	services, _ := boot.AdmittedServices(provider.NodeUrls[1])
	require.Empty(t, services, "precondition: archive refused on B at boot")
	admissions.record(provider, boot)

	key := holdoff.ProviderKey("ETH1", "jsonrpc", provider.Name)
	holdoffs := holdoff.NewRegistry()
	holdoffs.RecordRateLimit(key, provider.NodeUrls[1].Url, 0) // B busy with relay 429s
	inputs := &chainReverifyInputs{
		chainParser:                parser,
		rpcEndpoint:                &lavasession.RPCEndpoint{ChainID: "ETH1", ApiInterface: "jsonrpc"},
		convertProvidersToSessions: fakeConvert,
		configuredStatic:           []*lavasession.RPCStaticProviderEndpoint{provider},
		recordAdmission:            admissions.record,
		recordReverifiedAdmission:  admissions.recordReverified,
		admissionFor:               admissions.admissionFor,
		rateLimitHoldoff:           holdoffs,
	}
	active := map[uint64]*lavasession.ConsumerSessionsWithProvider{0: makeSession(provider.Name)}
	askedB := len(up.methods("http", "/b/"+upPruned))
	applyReverification(t.Context(), inputs, active, reverifyTierStatic, 100)
	require.Len(t, up.methods("http", "/b/"+upPruned), askedB, "B was not asked")

	after, _ := admissions.admissionFor(provider).AdmittedServices(provider.NodeUrls[1])
	require.Empty(t, after, "archive must stay refused on B: the pass never re-checked it")
}

At edbd8d4: Should be empty, but was [archive].

One possible fix: when KeepingThrottledFrom keeps a url that was throttled at the url level, also keep every previous.refused key for that url. Whether the url was skipped or answered 429 to everything, nothing in this pass re-checked its services. The url-level 429 case had this gap before this PR too, but only on a fresh 429. With skipping it now happens on every pass where a sibling is free.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed, thanks for the repro. Fixed in 00db21e: when a pass does not check a url again (held and skipped, or every check came back 429), the url keeps every add-on refusal and warning it had, not only its url refusal. Your test is in as TestApplyReverification_AHeldURLKeepsItsAddonRefusals, plus TestKeepingThrottledFrom_AURLNotCheckedKeepsItsServiceRefusals in chainlib. Both go red with the rule switched off.

avitenzer added a commit that referenced this pull request Nov 10, 2026
#618 moved its branch from 3b1ba81 to edbd8d4. The only conflict is
docs/RATE-LIMIT-HOLDOFF.md, on two adjacent table rows: keep #618's
"Spec re-verification" row and this PR's "Hot relay path" row
(MAG-4734). No code conflict.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

Took 47 minutes
Tomelia1999 added a commit that referenced this pull request Nov 11, 2026
…on refusals (MAG-4733)

A url the pass skipped because it is held off, or saw only rate-limited at the
url level, kept its url verdict but lost every per-service refusal: those were
kept only for a service throttled in this pass, and a skipped url has no checks.
A pruned node refused archive at boot got archive admitted again on the next
pass after a relay 429, without being asked.

ProviderAdmission now remembers the url at each throttled position, and
KeepingThrottledFrom keeps every service refusal the previous verdict gave that
url. Found in review on #618.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Tomelia1999
Tomelia1999 force-pushed the fix/holdoff-url-pick-mag-4732 branch from 2f9f2a0 to ccadd20 Compare November 11, 2026 13:24
@Tomelia1999

Copy link
Copy Markdown
Contributor Author

Rebased on main at ccadd20. #622 was merged into this branch, so its commit is carried over (code identical to 2f9f2a0).

Main reworked the re-verifier for per-check retries (MAG-3950), so I re-applied the re-verify part on that shape instead of rebasing it line by line (9fd687c):

  • The retry pass skips held urls too (ValidateCollectionsAgain takes the skip set), and a retry waits until the first url is free, not only url 0 (heldOffUntil).
  • A url the pass sent no request to changes nothing in the hold-off. That covers a held url, and a url a retry kept whole without dialing (ProviderAdmission.URLAsked). A pass that never ran (a config error) clears nothing. That also fixes the old nit where a shape error cleared every url.

One more thing came out of the rebase (061ca58): the probe loop from MAG-4387 read the relay hold-off under the plain provider name, so it would never see a relay hold on this branch. It now reads the scoped key, carried on EndpointWithDirectConnection.HoldoffKey. The new gRPC rate-limit relay test from main reads the scoped key too.

Every new rule has a test, and each test goes red with its rule switched off. go build, go vet ./..., and go test for chainlib, rpcsmartrouter, holdoff, lavasession, chaintracker, endpointstate and common all pass at ccadd20.

Note: the intermediate commits from the rebase don't each build on their own (the re-verifier work sits in 9fd687c). The squash merge makes that moot.

@avitenzer avitenzer left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Tomelia1999 re-reviewed at ccadd203. Not ready yet, for one reason that is new since this morning: main moved under the branch when #659 merged (17dbfff2), and the two PRs disagree on which url list the re-verifier indexes. Everything I asked for last time is in and holds.

What I checked

  • CI at ccadd203: unit-tests, unit-tests-race, lint and the builds are green. govulncheck is red for GO-2026-6859, which #659 fixed on main with the go.mod bump to 1.26.11; a rebase picks it up.
  • Trial merge with main 17dbfff2: one textual conflict (consumer_types.go, resolution inline), two of main's tests that need this PR's new arguments (inline), and one semantic conflict (inline on holdoffEntries, with a failing throwaway test). With the conflict resolved and the two tests patched the merged tree builds and vets, and go test -count=1 is green on holdoff, chainlib, lavasession, rpcsmartrouter, chaintracker and endpointstate.
  • 00db21e9: dropping the service-refusal keep loop in KeepingThrottledFrom turns both TestKeepingThrottledFrom_AURLNotCheckedKeepsItsServiceRefusals and TestApplyReverification_AHeldURLKeepsItsAddonRefusals red. Resolved.
  • The skip-on-every-url rule and the 429-before-answer ordering are as verified at edbd8d49; the re-apply on the MAG-3950 shape keeps them, and the new tests for URLAsked, heldOffUntil and the retry pass are there.

Blocking

  1. Configured-vs-dialed index mismatch after #659 (inline on holdoffEntries). On chains with internal paths the hold-off entries, the skip set and urlPassResults are positions into p.NodeUrls, while the pass now verifies dialedNodeUrls(...). A 429 from a generated url is never held, a relay hold on one is not honoured, and a held pinned url is asked while a free sibling is skipped. Reproduced on the trial merge in all three shapes. The fix is to build the entries from the dialed list in holdoffEntries and heldOffUntil; that also retires the "known limitation" paragraph, since the relay path already keys 429s by the dialed url.

For the rebase

  • Fold UnservedInternalPaths into restrictsToPathLocked (inline), and give main's two tests the new arguments.
  • Main requires verified commit signatures since 2026-11-10 17:11 (ruleset "Main branch default"). Every commit on this branch is unsigned, so the squash is refused until they are rewritten signed. You have an SSH signing key registered; git rebase --gpg-sign with gpg.format=ssh does it.
  • The description's Jira section says "Also fixes MAG-4733". #622's commit now rides here, so add MAG-4734, and the squash message should name all three.

The scoping ruling. The ruling on MAG-4732 (comment of 2026-11-11) is yours, not product's. I checked its premise: the chart renders one Deployment per routers[] entry and each entry has one network, so a hold-off registry never saw two chains in the fleet, and the change is confined to api-interfaces of one entry. Accepted; recorded on MAG-4732.

url string
}

func holdoffEntries(providerKey string, p *lavasession.RPCStaticProviderEndpoint) []holdoffEntry {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking once rebased on current main: the hold-off entries and the skip set index the configured urls, but the pass now verifies the dialed list.

#659 (merged today, 17dbfff) makes revalidateProviderCollections verify dialedNodeUrls(provider.NodeUrls, …): on a chain with internal paths a bare url becomes one url per path, so the pass's positions are into [A/v2, A/v3, C/v2, C/v3] while holdoffEntries, held and urlPassResults(…, len(entries), held) are built from p.NodeUrls = [A, C]. The text merge is clean, so nothing flags it. On chains without internal paths the lists are equal and the PR behaves as described; on TON, STRK, AVAX and Monero it does not:

  • a 429 from C/v2 (dialed index 2) is never read, since urlPassResults stops at len(entries), so the url that answered 429 is never held, which is MAG-4733's bug again;
  • a relay hold on C/v2 is not in the skip set, because HeldOff(key, C) is asked about the bare url, so the held url is sent the request it was told to stop sending;
  • with [A bare, B pinned /v2] and B held, held={1} skips dialed index 1, which is A/v3, the free url, while B is asked. On TONT that also makes the pass inconclusive ("no node url verified serving internal path /v3: not asked: held off"), so the provider is never re-verified while B is held.

Reproduced on a trial merge of ccadd20 with main 17dbfff (conflict in consumer_types.go resolved as in my other comment):

Throwaway test, three subtests, all fail
// newLimitedTon: an httptest server answering 429 to every path, counting hits.
// tontInputs: chainReverifyInputs with tontParser(t), rpcEndpoint TONT/rest, fakeConvert, an admission registry.
func TestReview_ReverifyHoldoffIndexesConfiguredUrlsButPassIsDialed(t *testing.T) {
	t.Run("a 429 from a generated url is never held", func(t *testing.T) {
		a := newFakeTon(t, tontRootHash)
		c, _ := newLimitedTon(t)
		provider := tontProvider(commonlib.NodeUrl{Url: a.srv.URL}, commonlib.NodeUrl{Url: c.URL})
		key := holdoff.ProviderKey("TONT", "rest", provider.Name)
		holdoffs := holdoff.NewRegistry()
		inputs, admissions := tontInputs(t, provider, holdoffs)
		active := map[uint64]*lavasession.ConsumerSessionsWithProvider{0: makeSession(provider.Name)}
		applyReverification(t.Context(), inputs, active, reverifyTierStatic, 100)
		require.NotNil(t, admissions.admissionFor(provider).URLRateLimit(2), "the pass saw the 429 on dialed index 2 (C/v2)")
		require.True(t, holdoffs.HeldOff(key, c.URL+"/v2"), "the url that answered 429 must be held") // FAILS
	})
	t.Run("a relay hold on a generated url is not honoured", func(t *testing.T) {
		a := newFakeTon(t, tontRootHash)
		c, hits := newLimitedTon(t)
		provider := tontProvider(commonlib.NodeUrl{Url: a.srv.URL}, commonlib.NodeUrl{Url: c.URL})
		key := holdoff.ProviderKey("TONT", "rest", provider.Name)
		holdoffs := holdoff.NewRegistry()
		holdoffs.RecordRateLimit(key, c.URL+"/v2", 0) // what the relay path records for C/v2
		inputs, _ := tontInputs(t, provider, holdoffs)
		active := map[uint64]*lavasession.ConsumerSessionsWithProvider{0: makeSession(provider.Name)}
		applyReverification(t.Context(), inputs, active, reverifyTierStatic, 100)
		require.Equal(t, 0, hits("/v2/getMasterchainInfo"), "a held url is not asked") // FAILS: 1
	})
	t.Run("a held pinned url is asked and a free sibling is skipped instead", func(t *testing.T) {
		a := newFakeTon(t, tontRootHash)
		b, hits := newLimitedTon(t)
		pinned := commonlib.NodeUrl{Url: b.URL + "/v2", InternalPath: "/v2"}
		provider := tontProvider(commonlib.NodeUrl{Url: a.srv.URL}, pinned)
		key := holdoff.ProviderKey("TONT", "rest", provider.Name)
		holdoffs := holdoff.NewRegistry()
		holdoffs.RecordRateLimit(key, pinned.Url, 0)
		inputs, _ := tontInputs(t, provider, holdoffs)
		active := map[uint64]*lavasession.ConsumerSessionsWithProvider{0: makeSession(provider.Name)}
		applyReverification(t.Context(), inputs, active, reverifyTierStatic, 100)
		require.Equal(t, 0, hits("/v2/getMasterchainInfo"), "the held url B/v2 is not asked") // FAILS: 1
	})
}

Suggested fix: build the entries from the list the pass verifies, dialedNodeUrls(p.NodeUrls, inputs.chainParser, p.ApiInterface), in holdoffEntries and in heldOffUntil (every call site has inputs.chainParser or chainParser in scope). The relay path already records a 429 under the dialed url (targetEndpoint.NetworkAddress is base + path for a generated url), so the keys line up as well, and the "known limitation" paragraph in the description goes away. The three subtests above, plus the TON row of holdoff_url_pick_test.go through the expansion, would pin it.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed, and fixed in 07dc208. holdoffEntries now builds from reverifyNodeUrls, which is dialedNodeUrls(p.NodeUrls, chainParser, p.ApiInterface), the same list the pass verifies. So the entries, the skip set and urlPassResults all index one list. With no chain parser (the validateFn test hook) it falls back to the configured list.

Your three subtests are in reverify_dialed_holdoff_test.go (TestReverifyHoldoff_UsesTheDialedURLs), plus a fourth that pins the entries heldOffUntil reads. With reverifyNodeUrls switched back to p.NodeUrls, all four go red.

I didn't add the TON row through the expansion to holdoff_url_pick_test.go. That test is in routersession, which gets its endpoints already expanded, and the expansion itself is in rpcsmartrouter. The second subtest covers that link: a relay hold recorded under C/v2 is honoured by the pass.

The "known limitation" paragraph is gone from the description. docs/RATE-LIMIT-HOLDOFF.md's re-verification row now says the urls are the dialed ones.

}
return registry.ReadyAt(p.Name, holdoffURLKey(p))
var first time.Time
for _, entry := range holdoffEntries(holdoff.ProviderKey(p.ChainID, p.ApiInterface, p.Name), p) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same list as in holdoffEntries: once the entries come from the dialed urls, heldOffUntil has to read the same ones, or a retry waits on the bare url's entry, which nothing records under on TON. All four call sites (rpcsmartrouter.go:3040, :4563, spec_reverifier.go:636, :748) have a chain parser in scope.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 07dc208: heldOffUntil takes the chain parser and reads the same holdoffEntries. All four call sites pass inputs.chainParser.

// generated.
//
// cswp.Lock must be held.
func (cswp *ConsumerSessionsWithProvider) restrictsToPathLocked(internalPath *string) bool {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rebase note: this is the one textual conflict with main 17dbfff. #659 added UnservedInternalPaths (a path the provider was configured for but lost at admission) to the old in-line check. The resolution that keeps both is to fold it in here, so the walk and relayURLs pass such a provider over together:

if _, lost := cswp.UnservedInternalPaths[*internalPath]; lost {
	return true
}

before the endpoint loop. With that, the merged tree builds and vets. Two of main's tests also need the new arguments: direct_rpc_session_selection_test.go:371 (the heldOff func, nil) and node_url_verification_test.go:63 and :70 (throttleURL now takes the url).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Resolved as you suggested: restrictsToPathLocked returns true for a path in UnservedInternalPaths before the endpoint loop, so the walk and relayURLs pass such a provider over together. Main's two tests got the new arguments in 265d9ed.

The branch is now on main 36b0cd8, which includes #695's rename, so this file is protocol/routersession/consumer_types.go.

@avitenzer

Copy link
Copy Markdown
Contributor

@Tomelia1999 over to you for the rebase onto main 17dbfff2 (#659 is in). The review above has the details; the short list:

  1. Hold-off entries from the dialed list. holdoffEntries and heldOffUntil should build from dialedNodeUrls(p.NodeUrls, chainParser, p.ApiInterface), the list the pass now verifies, so the entries, the skip set and urlPassResults share its positions. The three TONT subtests in my inline comment on holdoffEntries fail on the trial merge today and should pass after; they are worth keeping as the regression tests. That also closes the "known limitation" paragraph.
  2. Conflict in consumer_types.go: fold UnservedInternalPaths into restrictsToPathLocked (inline comment). Main's direct_rpc_session_selection_test.go:371 and node_url_verification_test.go:63/:70 need the new arguments.
  3. Signed commits. Main has required verified signatures since 2026-11-10; every commit on this branch is unsigned, so the squash is refused until they are rewritten signed (git rebase --gpg-sign with your registered SSH key, gpg.format=ssh).
  4. Add MAG-4734 to the "Also fixes" line, since fix(lavasession): the last chance lends a url that can answer before a dead one (MAG-4734) #622 rides here.

The scoping ruling is accepted (MAG-4732). With the four points above and green CI, this is ready to approve.

Tomelia1999 and others added 11 commits November 12, 2026 18:20
…at can serve the request is held (MAG-4732)

A 429 is recorded per (provider, url), but selection asked only whether the
provider name was held, and the registry only knows urls that answered 429.
So one url's 429 held a provider whose other urls were free, and the same
name's entry on another api-interface with it. Inside a provider the url walk
never read the hold-off and handed out a held url listed first.

ProviderReadyAt now takes the urls that could serve the request (usable, able
to answer a relay, serving its add-on, extensions and internal path); a url
with no entry is free, and escalation is unchanged. The request's internal path
is passed down the provider selection chain for that. The url walk puts held
urls after free ones and drops none, so a request still goes out when every url
is held.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…red it (MAG-4733)

The re-verifier filed every 429 under the provider's first node url. With
relay selection now counting a provider's urls one by one (MAG-4732), a 429
from url B held url A, so relays went to B, the url that had just refused;
with a websocket url first, the 429 was held where relay selection never
looks, and held nothing.

The pass already knows which urls met a rate limit. Each is now held under its
own url, with its own Retry-After; a rate limit the pass does not tie to a url
holds them all, as it held the provider before. Each url that answered clears
its own entry. A pass is still skipped while any url of the provider is held.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…terface (MAG-4732)

The registry keyed a provider by its name alone, and a name is unique per
chain and api-interface only. Two held urls of one entry escalated the name,
and ProviderReadyAt then held every entry sharing it: a 429 storm on the
jsonrpc entry held the REST entry and the same name on other chains.

holdoff.ProviderKey scopes the name to the chain and api-interface. Every
caller that keys by provider uses it: relay path, provider selection and the
url walk, re-verifier, recovery probe, and the chain tracker's relay read.
Escalation semantics are unchanged within an entry. The url-only keys
(chain tracker, WS and gRPC subscriptions) are untouched. The metric
`provider` label stays the plain name.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… the hold-off lock-order comment (MAG-4732)

relayURLs and the url walk each spelled out which url may serve a request
(internal path, relay-capable, add-on and extensions). servesRequest is now
the one rule both call, so the rate-limit read and the pick cannot drift
apart. Usability stays with each caller: the walk relaxes it for the
reconnect probe and the last chance. No behaviour change.

The lock-order comment on heldOffProviderInfoLocked now lists the real
chain: csm.lock, cswp.Lock, endpoint.mu, with the registry lock as a leaf.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ds 429s before answers (MAG-4732, MAG-4733)

A re-verify pass skipped a provider while any of its urls was held, so one
url busy with relay 429s kept a recovered sibling refused for as long as its
hold lasted. The pass is now skipped only when every url is held. Otherwise
it asks the free urls and leaves the held ones out: each reads as only
rate-limited (chainlib.ErrNotAskedHeldOff), so it keeps its last verdict,
and nothing is sent to it.

The pass wrote its results one url at a time in config order. Two held urls
escalate the entry and an answer drops the escalation, so a url that
answered in the pass was held or free depending on where it sat in the
config, and the same url configured twice (base and a standalone add-on)
had its 429 cleared by the second entry. Results are now merged per url and
every 429 is recorded before any answer. A websocket url's 429 is held
under the url alone, the WS path's key, so it never counts toward the
entry's escalation.

The metric help text and docs/METRICS.md now describe escalation per entry.

BREAKING CHANGE: a rate-limit escalation (two urls of one provider held at
once) now holds that provider entry only (its name on one chain and
api-interface), not the same name on every chain and interface. The
`escalated` event of smartrouter_rate_limit_holdoffs_total keeps its
plain-name `provider` label, but no longer means every chain stopped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…on refusals (MAG-4733)

A url the pass skipped because it is held off, or saw only rate-limited at the
url level, kept its url verdict but lost every per-service refusal: those were
kept only for a service throttled in this pass, and a skipped url has no checks.
A pruned node refused archive at boot got archive admitted again on the next
pass after a relay 429, without being asked.

ProviderAdmission now remembers the url at each throttled position, and
KeepingThrottledFrom keeps every service refusal the previous verdict gave that
url. Found in review on #618.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…etry pass too (MAG-4733)

Main reworked the re-verifier into probeReverification / decideReverification
with per-check retries (MAG-3950), so the per-url hold-off of this PR is
re-applied on that shape instead of rebased line by line:

- A probe skips the urls held off after a rate limit and asks the rest, on the
  epoch pass and on a retry (ValidateCollectionsAgain takes the skip set). Only
  a provider whose every url is held skips the probe.
- Each url's result is recorded on its own entry, rate limits before answers.
  A url the pass sent no request to (held, or a retry kept its verdicts without
  dialing it: ProviderAdmission.URLAsked) changes nothing, and a pass that
  never ran (a config error) clears nothing.
- A skipped url keeps its warnings as well as its refusals.
- A retry waits until the first of the provider's urls is free, not until the
  first url is (heldOffUntil).

The gRPC rate-limit relay test from main reads the scoped provider key.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…scoped provider key (MAG-4732)

The probe loop from MAG-4387 leaves a held-off, stale endpoint out of its
provider's QoS sample, reading the relay path's (provider, url) entry under
the plain provider name. The relay path records under holdoff.ProviderKey, so
the probe never saw a relay hold and scored the busy url availability 0 again.

EndpointWithDirectConnection carries the provider's HoldoffKey, set by the
session manager, and the probe reads it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…a dead one (MAG-4734) (#622)

* fix(lavasession): the last chance lends a url that can answer before a dead one (MAG-4734)

The last chance lends one url of the blocked provider it hands out, and it
took the first url in config order that fits the request. A switched-off url
listed first was lent while its sibling would have answered: a head-stalled
url (alive, a block behind), or a url already back in rotation while the
provider's block has not lifted yet (the epoch switch). The request's one
chance was spent on the dead url.

The walk now orders the urls for the last chance by the kind of state they
are in: in rotation, then held by a state reason, then switched off
(lendRankLocked). It reads the kind, not a reason's name, so a reason added
later takes the rank of its kind. The sort is stable, so the rate-limit order
(MAG-4732) and config order hold within a rank. Normal selection and the
reconnect probe are unchanged, and the provider's url list is never
reordered. The once-per-request limit is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(lavasession): build the last chance's url order in one slice (MAG-4734)

gocritic flagged `ordered = append(free, held...)` (appendAssign): the
result of appending to one slice was assigned to another. The urls that are
not held off now go straight into ordered and the held ones are appended
after them. Same order, no behaviour change; golangci-lint is clean on
protocol/lavasession again.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

Took 52 minutes

* docs(lavasession): what the last chance's url rank does to the once-per-request limit (MAG-4734)

The first commit says the once-per-request limit is unchanged. The rule is
unchanged: the request's one chance is taken only when the lent url is
switched off, so a lent head-stalled url takes none (kept by decision on
MAG-4734). How often a head-stalled url is lent does change. Before, a
blocked provider whose switched-off url is listed first was lent that url,
and the request's one chance was spent. Now its head-stalled url is lent
first and takes no chance; when it fails, the retry can be lent the
head-stalled url of the next blocked provider. One request can now be lent
one head-stalled url per blocked provider, where before it was lent one
switched-off url.

The walk's comment said the request is lent one url; it is the provider
that is lent one url per request. RATE-LIMIT-HOLDOFF.md said a url that is
not held off always goes before one that is; in the last chance the url's
state comes first and the hold-off orders the urls within a rank.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: avitenzer <tenzer@clara.co.uk>
…rguments (MAG-4732)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ials (MAG-4733)

Since MAG-4214 (#659) a re-verify pass verifies the dialed urls: on a chain with
internal paths a bare url becomes one url per path. The hold-off entries, the skip
set and the per-url pass results were still positions into the configured list, so
on TON, STRK, AVAX and Monero a 429 from a generated url was never held, a relay
hold on one was not honoured, and a held pinned url was asked while a free sibling
was skipped.

holdoffEntries and heldOffUntil now build from reverifyNodeUrls, the list the pass
verifies (dialedNodeUrls). The relay path already records a 429 under the dialed
url, so the keys line up and the per-path known limitation goes away.

Found in review (PR #618).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Tomelia1999
Tomelia1999 force-pushed the fix/holdoff-url-pick-mag-4732 branch from ccadd20 to 07dc208 Compare November 12, 2026 16:27
@Tomelia1999 Tomelia1999 changed the title fix(lavasession,holdoff)!: hold a provider off only while every url that can serve the request is held (MAG-4732) fix(routersession,holdoff)!: hold a provider off only while every url that can serve the request is held (MAG-4732, MAG-4733, MAG-4734) Nov 12, 2026

@avitenzer avitenzer left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed at 07dc2080, rebased on main 36b0cd82, all 11 commits signed.

  • 07dc2080 does what the review asked: holdoffEntries, the skip set, urlPassResults and heldOffUntil now index reverifyNodeUrls, the dialed list the pass verifies, with the configured list only when there is no parser. The wss detection reads the dialed url too. The three TON cases from my comment are in TestReverifyHoldoff_UsesTheDialedURLs with a fourth for heldOffUntil; with reverifyNodeUrls returning the configured list, all four go red.
  • restrictsToPathLocked carries main's lost-path rule, so the walk and relayURLs pass such a provider over together. Main's two tests have the new arguments (265d9ed0).
  • From a clean worktree of 07dc2080: build, vet and gofmt clean; go test -count=1 is green on holdoff, chainlib, routersession, rpcsmartrouter, chaintracker and endpointstate.
  • Description: Jira line names MAG-4732, MAG-4733 and MAG-4734, the breaking footer is there, the known-limitation paragraph is gone, and the docs row says the urls are the dialed ones. The scoping ruling is accepted on MAG-4732.

Approving. One thing outside the code: no workflow ran on this head, as on every PR pushed between the 12th and this morning. The merge needs govulncheck and unit-tests on 07dc2080, so CI has to be re-triggered before the squash.

@avitenzer avitenzer closed this Nov 15, 2026
@avitenzer avitenzer reopened this Nov 15, 2026
@avitenzer
avitenzer merged commit 78fecde into main Nov 15, 2026
@avitenzer
avitenzer deleted the fix/holdoff-url-pick-mag-4732 branch November 15, 2026 08:41
avitenzer added a commit that referenced this pull request Nov 15, 2026
Bring in the hold-off change (#618, MAG-4732/4733/4734), which landed after
the previous merge and touches the same four files. No conflicts; IsUsable
and inRotation keep their meaning, so the two guards this branch adds read
the same state they did.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
avitenzer added a commit that referenced this pull request Nov 15, 2026
Resolves the conflicts with #695 (MAG-2972 rename) and #618 (MAG-4732):

- recovery_probe.go, recovery_probe_test.go, endpoint_relay_probe_test.go:
  deleted, as this PR does. Main's edits to them were the rename plus #618's
  two-line hold-off key scoping inside the replay, which goes with the replay.
- consumer_types.go, rpcsmartrouter_server.go, relay_rate_limit_test.go:
  the replay blocks main renamed stay deleted.
- direct_http_status_body_test.go: the relay-path half of the 5xx-body test,
  on the routersession package name.
- error_codes.go: main's RouterError names with this PR's comment, which no
  longer mentions replay evidence.
- docs/RATE-LIMIT-HOLDOFF.md: main's #618 rows without the recovery-probe row.
- direct_conn_fake_test.go, relay_inner_direct_endpoint_health_test.go:
  this PR's own test code moved to the routersession package name.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants