Skip to content

fix(holdoff): count a 429 burst as one strike (MAG-4386) - #536

Merged
AnnaR-prog merged 1 commit into
mainfrom
fix/MAG-4386-holdoff-one-strike-per-window
Oct 19, 2026
Merged

AnnaR-prog merged 1 commit into
mainfrom
fix/MAG-4386-holdoff-one-strike-per-window

Conversation

@AnnaR-prog

@AnnaR-prog AnnaR-prog commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

Description

Part 1 of MAG-4370. A short burst of 429s from one vendor became a hold-off of about half an hour, because every 429 in the burst added a doubling strike. @nimrod-teich, please review: this is the registry half of the interaction with #488 (MAG-4165) that MAG-4370 describes.

holdoff.Registry.RecordRateLimit did e.strikes++ and recomputed readyAt on every 429, including the ones that arrive while the url is still held off. A per-second quota answers a burst with one 429 per request already in flight:

429 in one burst Hold-off on main (jitter 0.17)
#1 at +0 s 31 s
#2 at +140 ms 1 m 2 s
#3 at +280 ms 2 m 4 s
#4 at +420 ms 4 m 8 s
#5 at +560 ms 8 m 16 s
#6 at +700 ms 16 m 33 s
#7 at +840 ms 31 m 1 s

The 2026-10-15 real-node run on Sui mainnet measured the same thing live: 28 real QuickNode 429s in about 2 s held the url off for 31-35 min.

The rule now: a strike is a hold-off that ended in another 429, not a 429.

  • A 429 inside a standing hold-off adds no strike and leaves the hold-off where it was. RecordRateLimit returns what is left of it.
  • A Retry-After longer than what is left extends the hold-off to the ask, still without a strike.
  • A 429 after the hold-off ran out is the next strike, exactly as before. Persistent overload still escalates 30 s → 1 m → 2 m → … → 30 m.
  • The provider-name escalation is checked on every 429, as before.

Who reads the return value (it changed from "the hold-off just applied" to "how long the url is now held off"):

Caller Use
chaintracker.ChainTracker.noteFetchOutcome stores it as rateLimitedUntil and hands it to the start retry as Retry-After; the remaining time is the right wait there
relay path (rpcsmartrouter_server.go), spec re-verify, WS and gRPC subscriptions logged only
recovery probe ignored

Metrics: smartrouter_rate_limit_holdoffs_total{event="recorded"} and smartrouter_rate_limit_holdoff_seconds count hold-offs applied (new or extended), so a burst counts once instead of once per 429. The "upstreams currently tripping their caps" PromQL in METRICS.md still works. A dashboard that read recorded as a count of 429 replies will show lower numbers.

x-ratelimit-reset / RateLimit-Reset are deliberately not read. MAG-4386 asks for them, and they are left out on purpose:

  • No effect for QuickNode. Retry-After only floors the hold-off, and the first strike is already 30 s, so QuickNode's x-ratelimit-reset: 1 would change nothing. The header could only ever lengthen a hold-off.
  • The unit varies by vendor. It is delta-seconds at QuickNode, epoch seconds at GitHub-style APIs and milliseconds at others. Under the floor rule a misread parks a healthy provider for up to the 1 h bound, the same over-hold this ticket fixes.
  • The relay path never sees it. SendDirectRelay's reply-header filter (MAG-3104) keeps only Retry-After and Date of the upstream's headers before httpStatusRelayError types the 429. Reading it there would mean leaking vendor quota headers to clients, or new plumbing on RelayResult.

If a vendor with long windows shows up, the follow-up would be to parse the header in the senders from the raw headers and carry it on the result. Strike-once is what removes the 35-minute escalation.

Not in this PR (the other parts of MAG-4370): the tracker re-arming its poll when the hold-off clears and the probe not scoring a held-off endpoint as dead (MAG-4387, #537), and stateful broadcasts ignoring the hold-off filter (MAG-4388, #538).

Merge order

Merge this before, or together with, MAG-4388 (#538, writes broadcast during a hold-off). That PR sends writes to providers that are still held off; on today's registry each 429 they answer adds a strike, and this PR is what stops a 429 inside a standing hold-off from adding one.

Tests

  • New, each failing on main: TestBurstWithinOneHoldoffIsOneStrike, TestLongerRetryAfterInsideTheWindowExtendsWithoutAStrike and TestMetrics_BurstRecordsOneHoldoff (protocol/holdoff/burst_test.go).
  • Seven existing tests recorded back-to-back 429s on a fixed clock and counted them as consecutive strikes. Each now lets the hold-off run out before the next 429, so the escalation they pin is unchanged:
    • TestConsecutiveStrikesDoubleUpToCap
    • TestRetryAfterIsTheFloor
    • TestAnswerClearsStrikesAndHoldoff: now also discriminates; before, its final assertion passed even with no clear
    • TestStrikeMemoryAgesOut
    • TestNoteFetchOutcome_NoAnswerKeepsTheHoldoff
    • TestReplayFailingRelay_RecoveredClearsHoldoff
    • TestNoteFetchOutcome_OwnRateLimitDoesNotHoldTheProviderForRelays: asserts the one-strike floor; three refusals in one window are one strike
  • go test -race -count=1 ./protocol/holdoff/... passes. Targeted -race runs also pass:
    • every test in the rpcsmartrouter rate-limit and hold-off test files (re-verify, recovery probe, WS/gRPC selection, relay rate-limit classification);
    • the chaintracker hold-off, poll-now and init tests;
    • the lavasession rate-limit filter, stateful-to-backup, routing-snapshot and preferred-provider tests;
    • the endpointstate poll-now and rate-limit tests;
    • the common retry-after tests.
  • Full packages, go test -race -count=1 on main 1b4548f: holdoff, rpcsmartrouter, lavasession, endpointstate and common pass, 0 races.
  • Known failures on main, unrelated: under -race, TestChainTrackerFetchSpreadAcrossPollingTime and TestChainTrackerCallbacks fail on unmodified main 1b4548f too (data races in the test's mock fetcher). With those two skipped the package passes with 0 races, and without -race the whole package passes.
  • Known failure on main, unrelated: under -race, TestHealthCommand_RejectsUnbootableCrossValidation can fail with "race detected during execution of test". The health command sets the global log level while a WebSocket goroutine left running by TestEndToEnd_SubstratePushFrameReachesClient still logs. It reproduces on unmodified main 1b4548f (-run 'TestEndToEnd_SubstratePushFrameReachesClient|TestHealthCommand_RejectsUnbootableCrossValidation' -count=3).
  • gofmt and goimports are clean, and golangci-lint (latest) --new-from-rev=origin/main reports 0 issues.
  • The five MAG-4368 fix branches (MAG-4369 fix(chainlib)!: a GraphQL 2xx with neither data nor errors is a node error, not the answer (MAG-4369) #534, MAG-4373 fix(rpcsmartrouter): read no tip out of a GraphQL user relay's reply (MAG-4373) #535, MAG-4386 fix(holdoff): count a 429 burst as one strike (MAG-4386) #536, MAG-4387 fix(chaintracker): resume polling when a hold-off clears and don't probe-score a held-off endpoint (MAG-4387) #537, MAG-4388 fix(lavasession)!: broadcast a write to every provider during a rate-limit hold-off (MAG-4388) #538) merge with each other cleanly in every pairing. The combined tree builds, and holdoff, probing, rpcInterfaceMessages, chaintracker and the lavasession/relaycore stateful and hold-off tests pass on it with -race.

Jira ticket

Jira ticket: MAG-4386


Author Checklist

  • read the contribution guide
  • included the correct type prefix in the PR title (fix)
  • confirmed ! in the type prefix if API or client breaking change — not breaking: internal scheduling only, nothing reaches the client; the recorded metric now counts once per burst (noted above)
  • targeted the main branch
  • provided a link to the relevant issue or specification — MAG-4386, part of MAG-4370
  • reviewed "Files changed" and left comments if necessary
  • included the necessary unit and integration tests
  • updated the relevant documentation or specification — docs/RATE-LIMIT-HOLDOFF.md (one strike per hold-off; only Retry-After is read), docs/METRICS.md (recorded)
  • confirmed all CI checks have passed — pending

🤖 Generated with Claude Code

Part 1 of MAG-4370. RecordRateLimit added a strike for every 429,
including the ones that land while the url is already held off. A
vendor's per-second quota answers a burst with one 429 per request in
flight, so seven 429s inside one second took a 30 s hold-off to the
30-minute cap: 31 min with jitter on main, and QuickNode's 28 real 429s
in about 2 s gave 31-35 min in the 2026-10-15 real-node run.

A strike is now a hold-off that ended in another 429. A 429 inside a
standing hold-off adds no strike and leaves the hold-off where it was,
and RecordRateLimit returns what is left of it. That return is what
ChainTracker.noteFetchOutcome stores as its own wait and hands to the
start retry; every other caller only logs it. A Retry-After longer than
what is left extends the hold-off to the ask, still without a strike.
The provider-name escalation is checked on every 429, as before. The
recorded counter and the duration histogram now count hold-offs
applied, so a burst counts once.

x-ratelimit-reset and RateLimit-Reset are deliberately not read.
Retry-After floors the hold-off under the 30 s first strike, so
QuickNode's x-ratelimit-reset: 1 would change nothing. The header's
unit differs by vendor (delta-seconds, epoch seconds, milliseconds), and
under the floor rule a misread can only lengthen a hold-off. The relay
path never sees the header anyway: the reply-header filter (MAG-3104)
keeps only Retry-After and Date before httpStatusRelayError types the
429.

Tests: TestBurstWithinOneHoldoffIsOneStrike,
TestLongerRetryAfterInsideTheWindowExtendsWithoutAStrike and
TestMetrics_BurstRecordsOneHoldoff fail on main. Seven existing tests
recorded back-to-back 429s on a fixed clock as consecutive strikes. They
now let each hold-off run out first, or assert the one-strike floor:
TestConsecutiveStrikesDoubleUpToCap, TestRetryAfterIsTheFloor,
TestAnswerClearsStrikesAndHoldoff, TestStrikeMemoryAgesOut,
TestNoteFetchOutcome_NoAnswerKeepsTheHoldoff,
TestNoteFetchOutcome_OwnRateLimitDoesNotHoldTheProviderForRelays and
TestReplayFailingRelay_RecoveredClearsHoldoff.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@AnnaR-prog
AnnaR-prog force-pushed the fix/MAG-4386-holdoff-one-strike-per-window branch from e520773 to f246aec Compare October 19, 2026 08:52
@AnnaR-prog
AnnaR-prog merged commit d3413bb into main Oct 19, 2026
@AnnaR-prog
AnnaR-prog deleted the fix/MAG-4386-holdoff-one-strike-per-window branch October 19, 2026 08:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants