Skip to content

fix(rpcsmartrouter): tell the caller when an archive request degrades to a non-archive node (MAG-4729) - #675

Merged
AnnaR-prog merged 3 commits into
mainfrom
fix/MAG-4729-archive-degrade-header
Nov 9, 2026
Merged

AnnaR-prog merged 3 commits into
mainfrom
fix/MAG-4729-archive-degrade-header

Conversation

@avitenzer

Copy link
Copy Markdown
Contributor

Description

When the only node-url tagged archive was refused it at boot (or unreachable then), a lava-extension: archive request was served by a pruned primary with HTTP 200 and no Lava-Extension-Unavailable.

The degrade branch now records the caller-requested extensions it drops, so the reply carries Lava-Extension-Unavailable: archive, and such a reply is not cached. Router-added extensions are still never reported (MAG-3935).

Split out of #666, which carried sixteen tickets in one PR, so each fix can be reviewed on its own. The commits are unchanged.

For reviewers:

  • The header also appears when archive nodes are admitted but none could take this request, for example when lava-providers-block excludes them.
  • Docs owed: drop the known-issue warning in docs#81 backup.md.

Jira ticket

Jira ticket: MAG-4729


Verification

  • TestArchiveOnUnadmittedBackupReportsExtensionUnavailable (5 cases) and TestArchiveOnUnadmittedBackupIsNotCached failed on main where they should, and pass here.
  • On this branch, based on current main: go build ./... and go vet pass, and the touched test files pass. CI runs the full suite.

🤖 Generated with Claude Code

… to a non-archive node (MAG-4729)

When the only node-url tagged archive was a backup refused archive at
boot (its archive check failed, or it was unreachable), a request with
lava-extension: archive was answered by the pruned primary with 200 and
no Lava-Extension-Unavailable header. The caller could not tell it got
non-archive data, which is what MAG-3935's header exists to say.

The extension policy is built from the add-ons both tiers declare,
before verification, and verification never narrows it. So parsing
finds archive offered and records nothing unavailable, GetSessions with
archive finds no node and returns PairingListEmptyError, and the degrade
path in sendRelayToEndpoint resends without extensions. The header was
built only from what parsing recorded, so the degraded reply went out
unmarked.

The degrade path now records on the request's RelayProcessor the
extensions it dropped that the caller named in lava-extension, and
appendHeadersToRelayResult names them alongside the ones parsing
recorded, with the same WARN and smartrouter_extension_unavailable_total
count. They live on the processor rather than the message because the
earliest-block re-parse can swap the message an attempt sees, while the
header is built from the request's original one. Extensions the router
adds on its own, such as archive for a deep block, are still never
reported. Narrowing the policy to what verification admitted is the
larger alternative: it would have to be re-derived whenever admission
changes, and would not cover a node admitted with the extension that
cannot take a given request. The rpcsmartrouter README now says a node
offers an extension only while it serves it.

A reply served after the request degraded is no longer written to the
cache, neither from the relay goroutine nor as a cross-validation
winner. The entry was keyed by the lava-extension directive, so every
repeat archive request was answered from it with pruned-node data and
no header, since a cache hit never reaches the degrade path, and went on
being answered so after the archive node recovered. Skipping the write
costs only the hits a degraded request would have had; the cache read
path is unchanged.

TestArchiveOnUnadmittedBackupReportsExtensionUnavailable boots the
endpoint through CreateSmartRouterEndpoint over fakeUpstreams and sends
the request through the listener. Against main (go test -overlay) the
refused-check and unreachable-backup cases fail with an empty header
where "archive" is expected; they pass now. Three guards pass on both:
an admitted archive backup serves with no header, a router with no
archive node-url still reports archive once, and a router-added archive
that degrades stays unreported (removing the caller filter fails that
case). TestArchiveOnUnadmittedBackupIsNotCached wires a real loopback
cache and sends the degraded request twice. Without the cache skip the
first reply is written and the repeat is served from the cache with no
header; now both reach the primary and carry the header, and its
control shows a reply served with archive is still written. go build,
go vet, golangci-lint (no new issues) and the rpcsmartrouter and
relaycore suites pass.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…inner is not cached (MAG-4729)

A reply served after the request degraded is kept out of the cache at two sites: the relay
goroutine, and SendParsedRelay, which writes a cross-validation winner once, after the quorum.
Only the first was tested. With the second skip removed, every test still passed.

The new test boots two archive and two pruned primaries. It sends a cross-validated archive
request that blocks both archive nodes with lava-providers-block, so the request degrades to the
pruned pair. Their agreed answer carries Lava-Extension-Unavailable and is not written; without
the skip it is written on every such request. The control sends the same request with the
archive nodes free, and the winner is written.

bootArchiveBackupEndpoint now wraps bootETH1Endpoint, which takes any primaries and backups, and
relayGetCode wraps relayGetCodeWithHeaders.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

@AnnaR-prog AnnaR-prog left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at 18b8058, tested merged with main ada2a55 and 0e8b9bc. 675-1 is fixed in 95de2d5 (a test), and I'm approving at that head. 675-2 is for the metric's owner.

675-1 (Low): nothing tests that a cross-validation winner is kept out of the cache; with the skip removed every test passes.

  • Mutants: removing AddDegradedExtensions, the caller filter, the slices.Concat or the relay goroutine's skip (rpcsmartrouter_server.go:3170) each fails a test. Removing the winner's skip in SendParsedRelay (:1829) fails none.
  • The path is live. A cross-validated lava-extension: archive request whose lava-providers-block names both archive nodes degrades to two pruned primaries. The reply is 200 with Lava-Extension-Unavailable: archive and their consensus. Without the skip, that winner is written, and the next plain archive request is answered from the cache (Lava-Provider-Address: Cached) with no header, while the archive nodes are healthy. With the skip, it goes to archive-1.
  • Fixed in 95de2d5, test only: TestArchiveDegradedCrossValidationWinnerIsNotCached. Its control has the archive nodes serve, and the winner is written. The test fails without the skip.

675-2 (Nit): the operator signals now also fire on the caller's own exclusion.

  • When lava-providers-block names every archive node, smartrouter_extension_unavailable_total counts the request and the once-per-extension WARN says no node offers archive (:6223), although archive nodes are serving. METRICS.md still reads "this deployment has no node for" the extension.
  • The header is right for the caller. Whether the operator counter should count a degrade the caller chose is for the metric's owner to decide; either way METRICS.md could say so.

Checked and correct:

  • MAG-4729's cases, through the real boot and listener. A refused archive check and an archive backup unreachable at boot now get Lava-Extension-Unavailable: archive. An admitted archive backup serves with no header, a router with no archive node-url still reports archive once, and an archive the router adds itself stays unreported (MAG-3935). Probe: a primary that declares archive and fails its check gets the header too, as the ticket comment says.
  • The semantics match MAG-3935 and the README. It is the same header, WARN and counter as for an extension parsing finds unavailable. Only extensions the caller named are reported, with the spec's names, so the label stays bounded. The new README paragraph (README.md:478) describes what the code does.
  • A cross-validated archive request with no archive node admitted is refused before it can degrade (500, insufficient-capacity), not served from a pruned node. The ticket accepts a refusal.
  • The two skips cover every write of an upstream reply. The only other tryCacheWriteResolved caller backfills a secondary-tier hit (secondary_cache.go:233), and the chain fetcher writes only the router's own polls.
  • Overlaps: on main with #671, a degraded request that runs out of budget gets the 504 with Lava-Extension-Unavailable, Lava-Guid and the provider. It also merges cleanly with #626, #663, #668, #669, #676 and #678 together. #676 rewrites the same degrade branch: the record stays after the successful degraded GetSessions, and #676's capacity reason stays on the failures. No other PR adds a RelayProcessorForHeaders implementer.

Tested: go build ./...; go vet ./protocol/... on main with #675 and the overlapping PRs; the #675 tests and the new one, also -race x3, and the new test 20x alone and 5x with its file; the five mutants above; full protocol/rpcsmartrouter, plus relaycore, lavasession, chainstate and provideroptimizer, on that combined tree. The only failure was TestNodeErrorProviderIsDemoted_MAG2156, a load-sensitive optimizer test that never reaches this code; it passes 30/30 alone on the same tree and on main. golangci-lint --new-from-rev (0 issues).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@AnnaR-prog
AnnaR-prog merged commit cc118a2 into main Nov 9, 2026
@AnnaR-prog
AnnaR-prog deleted the fix/MAG-4729-archive-degrade-header branch November 9, 2026 17:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants