Repository navigation
fix(rpcsmartrouter): tell the caller when an archive request degrades to a non-archive node (MAG-4729) - #675
Conversation
… to a non-archive node (MAG-4729) When the only node-url tagged archive was a backup refused archive at boot (its archive check failed, or it was unreachable), a request with lava-extension: archive was answered by the pruned primary with 200 and no Lava-Extension-Unavailable header. The caller could not tell it got non-archive data, which is what MAG-3935's header exists to say. The extension policy is built from the add-ons both tiers declare, before verification, and verification never narrows it. So parsing finds archive offered and records nothing unavailable, GetSessions with archive finds no node and returns PairingListEmptyError, and the degrade path in sendRelayToEndpoint resends without extensions. The header was built only from what parsing recorded, so the degraded reply went out unmarked. The degrade path now records on the request's RelayProcessor the extensions it dropped that the caller named in lava-extension, and appendHeadersToRelayResult names them alongside the ones parsing recorded, with the same WARN and smartrouter_extension_unavailable_total count. They live on the processor rather than the message because the earliest-block re-parse can swap the message an attempt sees, while the header is built from the request's original one. Extensions the router adds on its own, such as archive for a deep block, are still never reported. Narrowing the policy to what verification admitted is the larger alternative: it would have to be re-derived whenever admission changes, and would not cover a node admitted with the extension that cannot take a given request. The rpcsmartrouter README now says a node offers an extension only while it serves it. A reply served after the request degraded is no longer written to the cache, neither from the relay goroutine nor as a cross-validation winner. The entry was keyed by the lava-extension directive, so every repeat archive request was answered from it with pruned-node data and no header, since a cache hit never reaches the degrade path, and went on being answered so after the archive node recovered. Skipping the write costs only the hits a degraded request would have had; the cache read path is unchanged. TestArchiveOnUnadmittedBackupReportsExtensionUnavailable boots the endpoint through CreateSmartRouterEndpoint over fakeUpstreams and sends the request through the listener. Against main (go test -overlay) the refused-check and unreachable-backup cases fail with an empty header where "archive" is expected; they pass now. Three guards pass on both: an admitted archive backup serves with no header, a router with no archive node-url still reports archive once, and a router-added archive that degrades stays unreported (removing the caller filter fails that case). TestArchiveOnUnadmittedBackupIsNotCached wires a real loopback cache and sends the degraded request twice. Without the cache skip the first reply is written and the repeat is served from the cache with no header; now both reach the primary and carry the header, and its control shows a reply served with archive is still written. go build, go vet, golangci-lint (no new issues) and the rpcsmartrouter and relaycore suites pass. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…inner is not cached (MAG-4729) A reply served after the request degraded is kept out of the cache at two sites: the relay goroutine, and SendParsedRelay, which writes a cross-validation winner once, after the quorum. Only the first was tested. With the second skip removed, every test still passed. The new test boots two archive and two pruned primaries. It sends a cross-validated archive request that blocks both archive nodes with lava-providers-block, so the request degrades to the pruned pair. Their agreed answer carries Lava-Extension-Unavailable and is not written; without the skip it is written on every such request. The control sends the same request with the archive nodes free, and the winner is written. bootArchiveBackupEndpoint now wraps bootETH1Endpoint, which takes any primaries and backups, and relayGetCode wraps relayGetCodeWithHeaders. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
AnnaR-prog
left a comment
There was a problem hiding this comment.
Reviewed at 18b8058, tested merged with main ada2a55 and 0e8b9bc. 675-1 is fixed in 95de2d5 (a test), and I'm approving at that head. 675-2 is for the metric's owner.
675-1 (Low): nothing tests that a cross-validation winner is kept out of the cache; with the skip removed every test passes.
- Mutants: removing
AddDegradedExtensions, the caller filter, theslices.Concator the relay goroutine's skip (rpcsmartrouter_server.go:3170) each fails a test. Removing the winner's skip inSendParsedRelay(:1829) fails none. - The path is live. A cross-validated
lava-extension: archiverequest whoselava-providers-blocknames both archive nodes degrades to two pruned primaries. The reply is 200 withLava-Extension-Unavailable: archiveand their consensus. Without the skip, that winner is written, and the next plain archive request is answered from the cache (Lava-Provider-Address: Cached) with no header, while the archive nodes are healthy. With the skip, it goes toarchive-1. - Fixed in 95de2d5, test only:
TestArchiveDegradedCrossValidationWinnerIsNotCached. Its control has the archive nodes serve, and the winner is written. The test fails without the skip.
675-2 (Nit): the operator signals now also fire on the caller's own exclusion.
- When
lava-providers-blocknames every archive node,smartrouter_extension_unavailable_totalcounts the request and the once-per-extension WARN says no node offers archive (:6223), although archive nodes are serving. METRICS.md still reads "this deployment has no node for" the extension. - The header is right for the caller. Whether the operator counter should count a degrade the caller chose is for the metric's owner to decide; either way METRICS.md could say so.
Checked and correct:
- MAG-4729's cases, through the real boot and listener. A refused archive check and an archive backup unreachable at boot now get
Lava-Extension-Unavailable: archive. An admitted archive backup serves with no header, a router with no archive node-url still reports archive once, and an archive the router adds itself stays unreported (MAG-3935). Probe: a primary that declares archive and fails its check gets the header too, as the ticket comment says. - The semantics match MAG-3935 and the README. It is the same header, WARN and counter as for an extension parsing finds unavailable. Only extensions the caller named are reported, with the spec's names, so the label stays bounded. The new README paragraph (
README.md:478) describes what the code does. - A cross-validated archive request with no archive node admitted is refused before it can degrade (500,
insufficient-capacity), not served from a pruned node. The ticket accepts a refusal. - The two skips cover every write of an upstream reply. The only other
tryCacheWriteResolvedcaller backfills a secondary-tier hit (secondary_cache.go:233), and the chain fetcher writes only the router's own polls. - Overlaps: on main with #671, a degraded request that runs out of budget gets the 504 with
Lava-Extension-Unavailable,Lava-Guidand the provider. It also merges cleanly with #626, #663, #668, #669, #676 and #678 together. #676 rewrites the same degrade branch: the record stays after the successful degradedGetSessions, and #676's capacity reason stays on the failures. No other PR adds aRelayProcessorForHeadersimplementer.
Tested: go build ./...; go vet ./protocol/... on main with #675 and the overlapping PRs; the #675 tests and the new one, also -race x3, and the new test 20x alone and 5x with its file; the five mutants above; full protocol/rpcsmartrouter, plus relaycore, lavasession, chainstate and provideroptimizer, on that combined tree. The only failure was TestNodeErrorProviderIsDemoted_MAG2156, a load-sensitive optimizer test that never reaches this code; it passes 30/30 alone on the same tree and on main. golangci-lint --new-from-rev (0 issues).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Description
When the only node-url tagged
archivewas refused it at boot (or unreachable then), alava-extension: archiverequest was served by a pruned primary with HTTP 200 and noLava-Extension-Unavailable.The degrade branch now records the caller-requested extensions it drops, so the reply carries
Lava-Extension-Unavailable: archive, and such a reply is not cached. Router-added extensions are still never reported (MAG-3935).Split out of #666, which carried sixteen tickets in one PR, so each fix can be reviewed on its own. The commits are unchanged.
For reviewers:
lava-providers-blockexcludes them.backup.md.Jira ticket
Jira ticket: MAG-4729
Verification
TestArchiveOnUnadmittedBackupReportsExtensionUnavailable(5 cases) andTestArchiveOnUnadmittedBackupIsNotCachedfailed onmainwhere they should, and pass here.main:go build ./...andgo vetpass, and the touched test files pass. CI runs the full suite.🤖 Generated with Claude Code