Repository navigation
fix(rpcsmartrouter): ignore a non-positive lava-relay-timeout instead of crashing (MAG-3988) - #452
Conversation
Review: manual pass (high effort) on
|
| this PR (#452) | #427 | |
|---|---|---|
HandleDirectiveHeadersForMessage drops a non-positive value |
yes, switch on err != nil / timeout <= 0 / default |
yes, same switch, same three arms |
GetRelayTimeout honours only > 0 |
yes | yes (via BoundCallerRelayTimeout returning ok == false) |
time.NewTicker guarded at the call site |
no | yes (relayTimeout <= 0 → MaxInt64) |
| upper bound on a huge value | explicitly out of scope | yes (--max-caller-relay-timeout) |
| lower bound on a tiny positive value | explicitly out of scope | yes (MinCallerRelayTimeout) |
Three concrete collisions:
-
Add/add conflict. Both PRs create
protocol/rpcsmartrouter/relay_timeout_header_test.goas a new file. Git cannot merge that; one has to be deleted by hand. -
Same-hunk conflict. Both rewrite the same
HandleDirectiveHeadersForMessageblock (including the identical_, ok =→_, ok :=fixup) and the sameGetRelayTimeoutguard. -
The behaviours contradict each other, and both PRs ship a test pinning their own. This PR's
relay_timeout_header_test.goasserts250msis honoured verbatim — it passes here:--- PASS: TestRelayTimeoutHeader_WindowHandedToTheStateMachine/honoured_250msfix(rpcsmartrouter)!: bound lava-relay-timeout so a caller cannot crash the router or outlast its budget (MAG-3600, MAG-3992) #427 floors every positive value at
MinCallerRelayTimeout, which iscommon.AverageWorldLatency= 300ms, soBoundCallerRelayTimeout(250ms, …)returnsmax(min(250ms, ceiling), 300ms)=300ms. fix(rpcsmartrouter)!: bound lava-relay-timeout so a caller cannot crash the router or outlast its budget (MAG-3600, MAG-3992) #427's own table makes this explicit: its smallest honoured case is300ms, and1nsis asserted to come back asMinCallerRelayTimeout. So after fix(rpcsmartrouter)!: bound lava-relay-timeout so a caller cannot crash the router or outlast its budget (MAG-3600, MAG-3992) #427, this PR'shonoured_250mscase fails — and this PR's body names that clamp as the thing it deliberately refused to do.
This is a decision for you and the router owner, not a code defect, so I'm not prescribing one. The two shapes I can see:
- Land this one first (it is 116 lines and fixes a live process-ending bug now), then rebase fix(rpcsmartrouter)!: bound lava-relay-timeout so a caller cannot crash the router or outlast its budget (MAG-3600, MAG-3992) #427 and drop its duplicated parse-site and
GetRelayTimeouthunks — and accept that fix(rpcsmartrouter)!: bound lava-relay-timeout so a caller cannot crash the router or outlast its budget (MAG-3600, MAG-3992) #427 then reverses the250mscontract this PR just pinned, which is a!on fix(rpcsmartrouter)!: bound lava-relay-timeout so a caller cannot crash the router or outlast its budget (MAG-3600, MAG-3992) #427. - Close this one in favour of fix(rpcsmartrouter)!: bound lava-relay-timeout so a caller cannot crash the router or outlast its budget (MAG-3600, MAG-3992) #427 and accept the larger review surface and the new flag plus reply header before the crash is fixed.
Either way the sub-floor question has to be settled once, in one place, because right now two open PRs pin opposite answers to it.
2. Medium — the body's central justification cites a file that is not in this repo, and nothing in-repo records the contract
Why not clamp to
--min-relay-timeout, as the ticket suggests:docs/api/directives.mdpromises the override is used verbatim and "not clamped to the--min-relay-timeoutfloor, so a client can ask for less".
There is no docs/api/ directory in smart-router, and:
$ grep -rn "lava-relay-timeout" docs/ protocol/rpcsmartrouter/README.md
(no output)
I take it the quote is from Magma-Devs/docs (the PR links docs#54 there), which is fine as a source — but the citation reads as in-repo, and the practical consequence is that the "verbatim" contract this PR exists to protect is invisible to anyone working in this repo. That is very plausibly how finding #1 happened: #427, by the same author, clamps the same value to a different floor. #427 does add a protocol/rpcsmartrouter/README.md section; this PR should carry that one sentence too, so whichever lands first leaves the rule written down next to the code that enforces it.
3. Low — preserveRelayTimeout is the other TimeoutOverride consumer and still tests == 0
protocol/relaycore/relay_state.go:270:
func preserveRelayTimeout(src, dst chainlib.ProtocolMessage) {
timeout := src.TimeoutOverride()
if timeout == 0 {
return
}
dst.TimeoutOverride(timeout)
if hdrs := dst.GetDirectiveHeaders(); hdrs != nil {
hdrs[common.RELAY_TIMEOUT_HEADER_NAME] = timeout.String()
}
}The body says:
Defence in depth (
GetRelayTimeout): honour only an override> 0(was!= 0), so no other path that sets an override can reachNewTickerwith a value it panics on.
That holds — but it holds because GetRelayTimeout is now the chokepoint, not because this site is safe. This is the one place that would copy a non-positive override onto a rebuilt message and re-materialise it as a lava-relay-timeout header string on the downstream request. With this PR the header path can no longer store such a value, so it is unreachable today; it is the kind of unreachable that stops being unreachable when someone adds a third setter. A timeout <= 0 here makes the invariant local to each site instead of resting on a single reader three packages away. Your call whether that is worth a line.
Verified and not flagged
- The parse-site and
GetRelayTimeoutchanges are both correct, and the two are genuinely independent — theGetRelayTimeoutguard holds for any future setter, and the parse-site drop keeps a bad value out of the stored state and out ofpreserveRelayTimeout's copy. - The ignore-rather-than-clamp choice is right for
0, which!= 0already treated as "no override" by accident of sign; making it explicit costs nothing. TimeoutOverridehas exactly two readers (GetRelayTimeout,preserveRelayTimeout) and two setters (the parse site,preserveRelayTimeout). I checked both; there is no third path.- The
computed := 2 * time.Secondcomment inrelay_timeout_override_test.gois accurate.GetRelayTimeout→common.GetTimePerCu(10)→LocalNodeTimePerCu(10)= 1s, below the mutatedMinimumTimePerRelayDelayof 2s, soGetTimePerCureturns the 2s floor. Thet.Cleanuprestore of the package var is there, so the mutation does not leak into other tests in the package. time.Duration(-1 << 63)as the "most negative" case is the right boundary —-xon it overflows back to itself, so a-timeoutstyle fix would have missed exactly that value. Good case to have.- The tests discriminate. I did not repeat the
-overlaypre-fix run, but the positive cases (12s,250ms, and the unparseable baseline) are present and would fail a fix that over-corrected into clamping or into ignoring everything.
🤖 Generated with Claude Code
Review fix pushed:
|
| # | Finding from my review on 40cf4bf |
Status |
|---|---|---|
| 1 | High: #427 supersedes this PR, and the two cannot both merge as written | not changed: your and the router owner's decision |
| 2 | Medium: the "verbatim" contract is not recorded in-repo | not changed, on purpose: writing "used verbatim" into the README would lock in the contract #427 changes while that decision is open |
| 3 | Low: preserveRelayTimeout still tests == 0 |
fixed |
3. preserveRelayTimeout now treats a non-positive override as no override (timeout <= 0). The invariant now holds at each reader of TimeoutOverride, instead of resting on GetRelayTimeout.
- New test:
TestArchiveAddDropsNonPositiveRelayTimeoutcovers −1ns, −1s and the most negative duration. It goes throughaddArchiveExtension, the production caller, and requires that neither the override nor thelava-relay-timeoutheader survives the rebuild. - Red/green: with
== 0restored, all three subtests fail on both assertions. With the fix,go test ./protocol/relaycore/passes.
If this PR is closed in favour of #427: #427 does not touch relay_state.go, so its preserveRelayTimeout still tests == 0. This one-line guard would need porting.
🤖 Generated with Claude Code
PR #452 lands first: it stops a non-positive lava-relay-timeout from crashing the router (time.NewTicker panic). This merge reconciles PR #427's full bound on top of it instead of a force-pushed rebase. Resolution: - chainlib.GetRelayTimeout and HandleDirectiveHeadersForMessage keep #427's version (common.BoundCallerRelayTimeout; non-positive and unparseable values dropped at parse time). #452's guards are subsumed. - relaycore.preserveRelayTimeout's `<= 0` guard and its test are kept from #452 unchanged. - relay_timeout_header_test.go (add/add): #427's table-driven test is kept; #452's non-redundant header cases ("0", "-2562047h", "250ms") are folded into it and #452's duplicate test function dropped. - chainlib/relay_timeout_override_test.go from #452 is kept, with its comment updated and a 250ms case added for the caller floor. Behaviour change: #452 honoured a sub-floor window such as 250ms verbatim; with this PR a caller's window is raised to MinCallerRelayTimeout (300ms). That rule change lives here, under #427's `!`. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Took 1 hour 8 minutes
|
@AnnaR-prog option 1: this one lands first. #427 merged this branch in ( No changes to this PR. Your finding 2 stays as you left it: once #427 lands, the README section it adds is where the rule is written down. 🤖 Generated with Claude Code |
…in/main) into MAG-3600 #452's branch already merged origin/main (b3feebe), so this brings main too. The relaycore side resolved cleanly through #452's merge: MAG-2734 (c7c605a) deleted the archive retry rebuild, so b033501's `<= 0` guard in preserveRelayTimeout and archive_extension_test.go go with the code they guarded; no site on main copies a TimeoutOverride any more. README conflict: kept both #427's lava-relay-timeout section and main's new "Request tracing headers" section, in that order. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
|
@AnnaR-prog Your
The parse-site drop, the Tests: 🤖 Generated with Claude Code |
… of crashing (MAG-3988) time.ParseDuration accepts "-1s", and the router passed a caller's lava-relay-timeout through unchecked: GetRelayTimeout returned any non-zero override, and the relay state machine handed it to time.NewTicker, which panics on a non-positive interval. The ticker is created in the per-request relay goroutine, which has no recover, so one unauthenticated request header ended the whole router process and every request it was serving. A non-positive value is now ignored at the parse site, exactly like an unparseable one, so the request keeps the router's own timeout. GetRelayTimeout also honours only a positive override, so no other path that sets one can reach NewTicker with a value it panics on. A positive value is still used verbatim, including one below --min-relay-timeout: the directives docs promise the override is not clamped to that floor, so clamping, as the ticket suggested, would break a documented contract. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Took 6 minutes
…sh the router or outlast its budget (MAG-3600, MAG-3992) (#427) * fix(rpcsmartrouter): bound lava-relay-timeout so a caller cannot crash the router or outlast its budget (MAG-3600) Any caller can send lava-relay-timeout, unauthenticated, and the router used the value unchecked as the attempt window: - A negative or zero value reached time.NewTicker in the relay state machine's goroutine. NewTicker panics on a non-positive interval and nothing recovers there, so one request ended the process for every request on the pod. - A large value became the processing budget, because the budget is never shorter than the window. 45m held a light call up to ~50 minutes against unresponsive nodes, instead of its 30s. - A tiny positive value (1ns) made the ticker fire continuously, fanning one request out to every endpoint the retry limits allow. The value is now bounded in one place, common.BoundCallerRelayTimeout, which chainlib.GetRelayTimeout applies for every consumer of the window. Non-positive values are ignored, values under 300ms are raised to it, and values above the request's own budget are held to it. The new --max-relay-timeout flag lets an operator allow callers to extend the budget up to its value; the default 0 allows no extension. Also: - HandleDirectiveHeadersForMessage drops non-positive values at parse time, so they are never stored or copied onto a rebuilt message. - The state machine refuses a non-positive window before NewTicker and does not hedge instead, whatever the relay sender returns. - A reply to a request that carried the header includes Lava-Relay-Timeout-Applied with the window actually used. The floor is deliberately not --min-relay-timeout. Deployments run that at 7-10s, and the automation suite sends 300ms, 1s and 5s to hedge sooner; all three keep working. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix(rpcsmartrouter)!: bound lava-relay-timeout by the call's own budget, and name the flag for the caller (MAG-3600) Review of the first commit found the bound measured from the call's category alone: GetTimeoutForProcessing(0, info). A call whose own window is longer than its category budget lost time it had with no header. Bitcoin's sendrawtransaction is a hanging write on a 10-minute-block chain: with no header it gets ~20 minutes, and a caller asking for 25m or 45m got 3m. The bound now takes the router's own window (chainlib.routerRelayTimeout) and holds the header to max(the call's own budget, --max-caller-relay-timeout). Also: - Rename --max-relay-timeout to --max-caller-relay-timeout. Next to --min-relay-timeout the old name read like a cap on every request. It is not in any release yet, so the rename costs nothing now. - Test that an ignored value is never stored on the message, so the parse-time check is covered on its own. - The property test now checks that a caller asking for at least the router's own window never gets a smaller budget than no header gives. - Correct the README and doc comments: the flag never shortens a budget, but a header below the router's own window still replaces it, which on a window-driven call also shortens the budget, as before. BREAKING CHANGE: lava-relay-timeout is no longer used verbatim. A value that is not a positive duration is ignored, a value under 300ms is raised to 300ms, and a value above the request's own budget is held to it unless the operator raises --max-caller-relay-timeout (default 0: no extension). Callers who raised it per request for heavy methods, as the public docs recommend, now get the method's own budget. To keep letting callers extend budgets, run with --max-caller-relay-timeout, e.g. 5m, the outbound client's per-attempt limit. The public docs (api/directives.md, configuration/failover/timeout.md, reference/cli.md) still describe the verbatim behaviour. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * test(rpcsmartrouter): a huge lava-relay-timeout gives up at the router's budget on a quiet node (MAG-3992) Drives the real state machine with the router's own timeout schedule, a 10m header and a node that never answers. The request must end at the call's own budget on the clock. Removing the upper bound fails both the budget check and, on its own, the elapsed-time check. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * docs(rpcsmartrouter): say a lava-relay-timeout at or above the budget turns hedging off (MAG-3600) Addresses review. The attempt window is also the hedge interval, and GetTimeoutForProcessing never makes the budget shorter than the window, so a caller value at or above the request's own budget leaves the two equal. That holds with --max-caller-relay-timeout at its default, where such a value is held to the own budget, and for every request the flag extends, whose window is its whole extended budget. The README section now says so next to the header rules, and says that an extended request is one of those, so one quiet endpoint can hold it for as long as the caller asked. The flag help gains one clause. Retrying after a failed attempt is unaffected: the state machine decides whether to retry a completed failure without consulting the window. A throwaway probe against the state machine, not committed, showed both: with the window equal to the budget a refused attempt was retried at once, while a quiet one got its second dispatch only as the budget expired. Docs and help text only. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * docs(rpcsmartrouter): a relay-timeout at or above the budget also leaves the backup reserve unarmed (MAG-3600) backupReserveDelay(budget, window) returns ok == false when max(budget-window, window) >= budget, i.e. whenever window >= budget. GetTimeoutForProcessing never returns a budget shorter than the window, so a caller lava-relay-timeout at or above the request's own budget (including one extended by --max-caller-relay-timeout) makes the two equal and the MAG-3923 backup-reserve hedge is never armed. Say so in the README and in the flag's help text. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * test(rpcsmartrouter): build the relay-timeout tests against main's NewRelayProcessor (MAG-3600) MAG-2734 (c7c605a) removed RelayRetriesManager, dropping the retries- manager argument from relaycore.NewRelayProcessor and relaycoretest.RelayRetriesManagerInstance. Merging main left the two NewRelayProcessor calls in relay_timeout_header_test.go on the old signature; drop the argument. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: Anna Rayer <anna@magmadevs.com>
d206b20 to
6ded365
Compare
Magma-Devs/smart-router#452 stops a negative lava-relay-timeout from crashing the router by ignoring it, the same way an unparseable value already was. Say so next to the "used verbatim" promise, which still holds for every positive value. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This reverts commit 2d18323. This reverts commit 334fda5. This reverts commit ea5ad0a. This reverts commit a90ef62. #449 added a second timer to the relay state machine, keyed to the request's processing budget. One attempt window before the budget ran out, it sent one attempt that preferred the backup tier, whether or not any primary was still unused on the request. That changed when the backup tier is used. Before, a backup served only once every primary had been used on the request. With #449 a backup could serve while primaries were still untried. And when the budget is a whole number of windows (the chart's 10s window and 30s budget), the timer fired at the same moment as the hedge to the third primary. In the MAG-3945 test, with two quiet primaries and a healthy third, a backup answered instead of the healthy primary in 2 of 3 runs. The router owner had confirmed before the merge that the quiet-primary behaviour MAG-3923 reports is intended, and a change to it needs a design discussion first. This restores the behaviour of v1.5.8 and every earlier release: the budget only ends a request, attempts go out one window apart, and the backup tier is reached once the primary pool is exhausted. No release tag contains #449. Changes beyond the plain revert: - consumer_session_manager.go keeps the gofumpt form of ProbeLoopInterval that #449 introduced. Restoring the old var block would fail lint on a file this commit touches. - cross_validation_threshold_guard_test.go and health_check_crafted_relay_test.go were written after #449. Their calls to sendRelayToEndpoint drop the trailing backupReserve argument. - The rpcsmartrouter README drops the two sentences #452 added about the hedge not being armed when the window equals the budget. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…527) * docs(changelog): rewrite the v1.6.0 highlights and breaking changes (MAG-4330) The v1.6.0 section was generated on the tag push. The Highlights was one 300-word paragraph; the breaking changes were commit subjects with their BREAKING CHANGE footers pasted under them; two of those footers were superseded within their own squash (#492 lists NODE_LIMIT_EXCEEDED as newly retryable, while the merged code keeps 2011 non-retryable; #452 says the public docs still describe the verbatim lava-relay-timeout, while they describe the bounded one); and the archive-routing change of #389 had no conventional-commit prefix, so it sat under "Other work" with no note. Highlights now carries the archive routing rule as it works at v1.6.0, what is new, the notable fixes by area, the #449 revert, the two removals and the Go bump. Breaking changes are seven numbered entries, each with what changed, who notices and what to do; #389 is the seventh. The generated Changes lists are untouched. Link definitions for #471 and #472 are added; every other reference was already defined. * docs(changelog): say what the v1.5.x pre-releases brought since v1.4.0 (MAG-4330) v1.4.0 is the only graduated release; v1.5.0 through v1.5.8 are pre-releases, so a reader upgrading to v1.6.0 gets their changes too. Repeat the three largest in the v1.6.0 Highlights, with the version each shipped in: REST replies classified like JSON-RPC errors (#435, #448), Solana replies read by path instead of decoded (#428, #430, #431), and the heavy-response work (gzip per node url #432, presized body reads #429, the 1 MiB extraction cap #430, --cache-max-entry-bytes #434), plus what v1.6.0 adds on each.
Jira ticket: MAG-3988
Problem
One unauthenticated request header could end the router process:
time.ParseDuration("-1s")succeeds, and the value flows through unchecked:HandleDirectiveHeadersForMessagestores it as the override.GetRelayTimeoutreturns any non-zero override.time.NewTicker, which panics on a non-positive interval.The ticker lives in the per-request relay goroutine, which has no
recover, so the panic kills the whole process and every request the pod is serving.Fix
HandleDirectiveHeadersForMessage): a non-positive value is ignored, exactly like an unparseable one already is, so the request keeps the router's own timeout. It's logged at debug.GetRelayTimeout): honour only an override> 0(was!= 0), so no other path that sets an override can reachNewTickerwith a value it panics on.Why not clamp to
--min-relay-timeout, as the ticket suggests:docs/api/directives.mdpromises the override is used verbatim and "not clamped to the--min-relay-timeoutfloor, so a client can ask for less". Clamping would break that contract for valid sub-floor values like250ms. The fix only removes values that aren't a timeout at all. Zero was already harmless, as the ticket noted, and is now explicitly covered.Evidence
Live, real binary, private local router (the ticket asks for a disposable router only):
lava-relay-timeout: -1s12s)mainpanic: non-positive interval for NewTickerTests (the ticket's definition of done):
chainlib/relay_timeout_override_test.go,GetRelayTimeout: negative (including the most negative duration) falls back to the computed window; unset is unchanged; positive above and below the floor is verbatim; the result is always positive.rpcsmartrouter/relay_timeout_header_test.go, the real path (header →HandleDirectiveHeadersForMessage→GetProcessingTimeout):-1s,-1ns,-2562047h,0sand0are all ignored like an unparseable value;12sand250msare honoured.main's two files overlaid (go test -overlay), every negative case fails. Zero and positive values pass both before and after, so the tests discriminate rather than pass on everything.chainlib/...,relaycoreandrpcsmartrouterpass from a clean worktree of the commit. CI doesn't rungo test, so these local runs are the only test evidence.Docs
Magma-Devs/docs#54 adds one sentence to the
lava-relay-timeoutsection: a zero or negative value is ignored.Not in this PR (noted, not changed)
1nsis valid and documented as verbatim, so it's still honoured. The 10-hedge retry ceiling bounds the resulting fan-out, but it could still be worth a lower bound.1000h) raises the processing budget with it (GetTimeoutForProcessingtakes the larger of the two). That lets a caller hold a request open for as long as they stay connected.Both are product decisions about the documented "verbatim" contract, not crashes, so I left them out of this fix.
🤖 Generated with Claude Code