Repository navigation
fix(holdoff): count a 429 burst as one strike (MAG-4386) - #536
Merged
Merged
Conversation
This was referenced Oct 18, 2026
Merged
Tomelia1999
approved these changes
Oct 18, 2026
Part 1 of MAG-4370. RecordRateLimit added a strike for every 429, including the ones that land while the url is already held off. A vendor's per-second quota answers a burst with one 429 per request in flight, so seven 429s inside one second took a 30 s hold-off to the 30-minute cap: 31 min with jitter on main, and QuickNode's 28 real 429s in about 2 s gave 31-35 min in the 2026-10-15 real-node run. A strike is now a hold-off that ended in another 429. A 429 inside a standing hold-off adds no strike and leaves the hold-off where it was, and RecordRateLimit returns what is left of it. That return is what ChainTracker.noteFetchOutcome stores as its own wait and hands to the start retry; every other caller only logs it. A Retry-After longer than what is left extends the hold-off to the ask, still without a strike. The provider-name escalation is checked on every 429, as before. The recorded counter and the duration histogram now count hold-offs applied, so a burst counts once. x-ratelimit-reset and RateLimit-Reset are deliberately not read. Retry-After floors the hold-off under the 30 s first strike, so QuickNode's x-ratelimit-reset: 1 would change nothing. The header's unit differs by vendor (delta-seconds, epoch seconds, milliseconds), and under the floor rule a misread can only lengthen a hold-off. The relay path never sees the header anyway: the reply-header filter (MAG-3104) keeps only Retry-After and Date before httpStatusRelayError types the 429. Tests: TestBurstWithinOneHoldoffIsOneStrike, TestLongerRetryAfterInsideTheWindowExtendsWithoutAStrike and TestMetrics_BurstRecordsOneHoldoff fail on main. Seven existing tests recorded back-to-back 429s on a fixed clock as consecutive strikes. They now let each hold-off run out first, or assert the one-strike floor: TestConsecutiveStrikesDoubleUpToCap, TestRetryAfterIsTheFloor, TestAnswerClearsStrikesAndHoldoff, TestStrikeMemoryAgesOut, TestNoteFetchOutcome_NoAnswerKeepsTheHoldoff, TestNoteFetchOutcome_OwnRateLimitDoesNotHoldTheProviderForRelays and TestReplayFailingRelay_RecoveredClearsHoldoff. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
AnnaR-prog
force-pushed
the
fix/MAG-4386-holdoff-one-strike-per-window
branch
from
October 19, 2026 08:52
e520773 to
f246aec
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Part 1 of MAG-4370. A short burst of 429s from one vendor became a hold-off of about half an hour, because every 429 in the burst added a doubling strike. @nimrod-teich, please review: this is the registry half of the interaction with #488 (MAG-4165) that MAG-4370 describes.
holdoff.Registry.RecordRateLimitdide.strikes++and recomputedreadyAton every 429, including the ones that arrive while the url is still held off. A per-second quota answers a burst with one 429 per request already in flight:The 2026-10-15 real-node run on Sui mainnet measured the same thing live: 28 real QuickNode 429s in about 2 s held the url off for 31-35 min.
The rule now: a strike is a hold-off that ended in another 429, not a 429.
RecordRateLimitreturns what is left of it.Who reads the return value (it changed from "the hold-off just applied" to "how long the url is now held off"):
chaintracker.ChainTracker.noteFetchOutcomerateLimitedUntiland hands it to the start retry as Retry-After; the remaining time is the right wait thererpcsmartrouter_server.go), spec re-verify, WS and gRPC subscriptionsMetrics:
smartrouter_rate_limit_holdoffs_total{event="recorded"}andsmartrouter_rate_limit_holdoff_secondscount hold-offs applied (new or extended), so a burst counts once instead of once per 429. The "upstreams currently tripping their caps" PromQL in METRICS.md still works. A dashboard that readrecordedas a count of 429 replies will show lower numbers.x-ratelimit-reset/RateLimit-Resetare deliberately not read. MAG-4386 asks for them, and they are left out on purpose:x-ratelimit-reset: 1would change nothing. The header could only ever lengthen a hold-off.SendDirectRelay's reply-header filter (MAG-3104) keeps onlyRetry-AfterandDateof the upstream's headers beforehttpStatusRelayErrortypes the 429. Reading it there would mean leaking vendor quota headers to clients, or new plumbing onRelayResult.If a vendor with long windows shows up, the follow-up would be to parse the header in the senders from the raw headers and carry it on the result. Strike-once is what removes the 35-minute escalation.
Not in this PR (the other parts of MAG-4370): the tracker re-arming its poll when the hold-off clears and the probe not scoring a held-off endpoint as dead (MAG-4387, #537), and stateful broadcasts ignoring the hold-off filter (MAG-4388, #538).
Merge order
Merge this before, or together with, MAG-4388 (#538, writes broadcast during a hold-off). That PR sends writes to providers that are still held off; on today's registry each 429 they answer adds a strike, and this PR is what stops a 429 inside a standing hold-off from adding one.
Tests
TestBurstWithinOneHoldoffIsOneStrike,TestLongerRetryAfterInsideTheWindowExtendsWithoutAStrikeandTestMetrics_BurstRecordsOneHoldoff(protocol/holdoff/burst_test.go).TestConsecutiveStrikesDoubleUpToCapTestRetryAfterIsTheFloorTestAnswerClearsStrikesAndHoldoff: now also discriminates; before, its final assertion passed even with no clearTestStrikeMemoryAgesOutTestNoteFetchOutcome_NoAnswerKeepsTheHoldoffTestReplayFailingRelay_RecoveredClearsHoldoffTestNoteFetchOutcome_OwnRateLimitDoesNotHoldTheProviderForRelays: asserts the one-strike floor; three refusals in one window are one strikego test -race -count=1 ./protocol/holdoff/...passes. Targeted-raceruns also pass:go test -race -count=1on main 1b4548f:holdoff,rpcsmartrouter,lavasession,endpointstateandcommonpass, 0 races.-race,TestChainTrackerFetchSpreadAcrossPollingTimeandTestChainTrackerCallbacksfail on unmodified main 1b4548f too (data races in the test's mock fetcher). With those two skipped the package passes with 0 races, and without-racethe whole package passes.-race,TestHealthCommand_RejectsUnbootableCrossValidationcan fail with "race detected during execution of test". The health command sets the global log level while a WebSocket goroutine left running byTestEndToEnd_SubstratePushFrameReachesClientstill logs. It reproduces on unmodified main 1b4548f (-run 'TestEndToEnd_SubstratePushFrameReachesClient|TestHealthCommand_RejectsUnbootableCrossValidation' -count=3).--new-from-rev=origin/mainreports 0 issues.holdoff,probing,rpcInterfaceMessages,chaintrackerand the lavasession/relaycore stateful and hold-off tests pass on it with-race.Jira ticket
Jira ticket: MAG-4386
Author Checklist
fix)!in the type prefix if API or client breaking change — not breaking: internal scheduling only, nothing reaches the client; therecordedmetric now counts once per burst (noted above)mainbranchdocs/RATE-LIMIT-HOLDOFF.md(one strike per hold-off; only Retry-After is read),docs/METRICS.md(recorded)🤖 Generated with Claude Code