Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
42380fb
fix(chainlib): hand the graphql listener detached request headers (MA…
avitenzer Oct 13, 2026
dd75a90
fix(chainlib): classify tendermint batch elements with the tendermint…
avitenzer Oct 13, 2026
79ea255
fix(chainlib): answer an undeclared rest verb with 501 on the post ha…
avitenzer Oct 13, 2026
14bcfab
fix(chainlib): classify tendermint uri transport errors like the othe…
avitenzer Oct 13, 2026
b0626ea
fix(chainlib): fail the url in ValidateCollect when its head probe fa…
avitenzer Oct 13, 2026
5de550c
fix(chainlib): echo the caller's id on the websocket rate-limit reply…
avitenzer Oct 13, 2026
531e229
fix(lavasession): clear the redemption flag on the pool-empty release…
avitenzer Oct 13, 2026
2b273f8
fix(keeper): warn about spec files that fail to load beside good ones…
avitenzer Oct 13, 2026
d4ea442
fix(rpcsmartrouter): replay the spec's reply headers on a secondary c…
avitenzer Oct 13, 2026
1f1a547
fix(rpcsmartrouter): reset chain trackers and probe back-off on /debu…
avitenzer Oct 13, 2026
c276cc0
fix(rpcsmartrouter): read the test and health commands' own flags (MA…
avitenzer Oct 13, 2026
c258f7f
Merge main into fix/MAG-4278-drift-fixes
AnnaR-prog Oct 15, 2026
302932b
fix(chainlib): answer a rate-limited websocket batch per member and a…
AnnaR-prog Oct 15, 2026
9ef1fc5
fix(lavasession): keep the redemption flag of a provider the epoch re…
AnnaR-prog Oct 15, 2026
5fb8bc8
test(rpcsmartrouter): pin that reset-all drops the endpoint tip; upda…
AnnaR-prog Oct 15, 2026
7c54ab7
test(chainlib): pin the tendermint batch classifier through the parse…
AnnaR-prog Oct 15, 2026
a1de465
docs(secondary-cache): a secondary hit now replays the headers a prim…
AnnaR-prog Oct 15, 2026
2e7a0c7
docs(chainlib): the tendermint uri path classifies transport errors, …
AnnaR-prog Oct 15, 2026
604361f
fix(rpcsmartrouter): keep plaintext-first gRPC as the test command's …
AnnaR-prog Oct 17, 2026
b9a1443
fix(chainlib): keep reporting the checks that need no head when Valid…
AnnaR-prog Oct 17, 2026
59bd166
test(lavasession): drive the epoch-rebuild drain directly in the rede…
AnnaR-prog Oct 17, 2026
a40be3f
fix(chaintracker): clear latestChangeTime under the lock when a track…
AnnaR-prog Oct 17, 2026
6b298fe
Merge branch 'main' into fix/MAG-4278-drift-fixes
AnnaR-prog Oct 18, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -134,6 +134,11 @@ An upstream with multiple node URLs (e.g. an `https://` and a `wss://` endpoint)
URL**, distinguished by `url`/`transport`. An endpoint's `ok` is `true` only when the spec loaded
(`specValid`) **and** every verification passed. Top-level `ok` is the AND of all rows.

When a URL's latest-block call fails, its row starts with a failed `fetch-latest-block` verification
and reports `latestBlock: -1`. Checks that measure a distance from the head are then reported failed
with `latest block unknown` instead of being sent; every other check still runs, so the row still
shows what the node does answer.

| Flag | Default | Purpose |
| --- | --- | --- |
| `--use-static-spec` | (required) | Spec source(s) — file, directory, or remote URL (same paths as the main command) |
Expand Down
45 changes: 24 additions & 21 deletions docs/SECONDARY-CACHE.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,18 +53,23 @@ request ──► primary cache ──hit──► served ("Cached")
is down, and is even valid with no primary configured at all (reads work;
nothing backfills — the router logs an advisory warning for this topology).
- **No identity leakage.** Cache entries *do* carry data that identifies where
they came from — the upstream's response headers plus the writer's
signatures. Every entry crossing in from the secondary is copied and stripped
before it is served or backfilled: `Sig` and `SigBlocks` go entirely, and
`Metadata` is reduced to an **allowlist** of the headers that describe how to
decode the payload (`Content-Type`, `Content-Encoding`). Everything else in it
goes, including header names this router has never heard of — an allowlist
rather than a denylist because upstream response headers are an open set that
no denylist can be proven to cover. The caller therefore sees the response
body, its content type, and the router's own locally minted headers
they came from — the upstream's response headers plus the writer's signatures.
Every entry crossing in from the secondary is copied and stripped before it is
served or backfilled: `Sig` and `SigBlocks` go entirely, and `Metadata` is
reduced to an **allowlist**, the same filter a primary-cache hit and the live
relay path apply. It keeps the headers that describe how to decode the payload
(`Content-Type`, `Content-Encoding`), `Retry-After` and `Date`, and the
headers the chain spec declares in the reply direction (`pass_reply`,
`pass_both`, `pass_ignore`; the Aptos ledger-state headers, for example),
minus any name the router mints itself. Everything else in it goes, including
header names this router has never heard of — an allowlist rather than a
denylist because upstream response headers are an open set that no denylist
can be proven to cover. The caller therefore sees the response body, those
headers, and the router's own locally minted headers
(`Lava-Provider-Address: Cached`, the GUID, `Provider-Latest-Block`) — the
same header set a primary-cache hit produces. The stripped copy is the only
copy used, so nothing unsanitized can reach your primary cache either.
same header set a primary-cache hit of the same entry produces. The stripped
copy is the only copy used, so nothing unsanitized can reach your primary
cache either.
- **No foreign block heights.** The entry's `LatestBlock` is dropped in the same
step, and the router re-stamps its *own* tracked tip in its place so
`Provider-Latest-Block` stays truthful and locally sourced. Left as-is, the
Expand Down Expand Up @@ -566,15 +571,13 @@ Lava-Cache-Outcome: primary=miss,secondary=hit

`Lava-Provider-Address` alone cannot close this step — a primary hit and a
secondary hit both report `Cached`, both carry the same locally minted header
set, and the body is the same bytes. (The one difference is invisible here and
by design: an entry that came from the secondary replays only its
`Content-Type` / `Content-Encoding`, not whatever other upstream headers the
writer's node happened to send. Since MAG-3104 the live path is nearly as
strict — `filterUpstreamReplyMetadata` keeps those two, `Retry-After`, `Date`
and the spec's own reply headers, and drops the rest before the reply leaves
the sender — and a primary hit is passed through the same filter when it is
served, so the primary replays that reduced set whichever router wrote the
entry.)
set, and the body is the same bytes. (The upstream headers they replay are the
same too: since MAG-3104 the live path keeps only `Content-Type`,
`Content-Encoding`, `Retry-After`, `Date` and the spec's own reply headers —
`filterUpstreamReplyMetadata` drops the rest before the reply leaves the
sender — and since MAG-4278 a hit from either tier is passed through that same
filter when it is served, so both replay that reduced set whichever router
wrote the entry.)

`Lava-Cache-Tier` names the tier that answered, and `Lava-Cache-Outcome` says
what each tier did on the way. Both are part of the reply, so there is nothing
Expand Down Expand Up @@ -703,7 +706,7 @@ walkthrough](#manual-demo-walkthrough); this section records where each one is
| Configurable lookup timeout; exceeded = miss | Met | `secondary-cache-timeout`, default `50ms` |
| Failure never affects serving; primary-grade resilience | Met | Same reconnect loop and non-blocking failure semantics as the primary |
| Backwards compatible when unconfigured | Met | No secondary series in metrics, no added step in the request path |
| No provider-identifying metadata exposed | Met — and **not** a no-op | The format *does* carry it (`Sig`, `SigBlocks`, `Metadata` holding upstream response headers). `Sig`/`SigBlocks` are dropped entirely and `Metadata` is reduced to the `Content-Type`/`Content-Encoding` allowlist, on a private copy that becomes the only copy used. Allowlist, not denylist: upstream headers are an open set. `TestSanitizeForeignCacheReplyDropsIdentityMetadataAndSignatures`, `TestSanitizeForeignCacheReplyKeepsTransportHeaders`, `TestSecondaryPoisonedEntrySanitizedForCallerAndBackfill` |
| No provider-identifying metadata exposed | Met — and **not** a no-op | The format *does* carry it (`Sig`, `SigBlocks`, `Metadata` holding upstream response headers). `Sig`/`SigBlocks` are dropped entirely and `Metadata` is reduced to the allowlist a primary hit is filtered with (the `Content-Type`/`Content-Encoding` pair, `Retry-After`, `Date` and the spec's reply-direction headers, never a router-owned name), on a private copy that becomes the only copy used. Allowlist, not denylist: upstream headers are an open set. `TestSanitizeForeignCacheReplyDropsIdentityMetadataAndSignatures`, `TestSanitizeForeignCacheReplyKeepsTransportHeaders`, `TestSecondaryPoisonedEntrySanitizedForCallerAndBackfill`, `TestSecondaryCacheHitReplaysTheSameFilteredHeadersAsAPrimaryHit` |
| No foreign chain state adopted | Met on all four channels | `SeenBlock` is never fed to `adoptSharedStateTip` (`TestSecondarySeenBlockNeverAdoptedIntoChainState`); `Reply.LatestBlock` is zeroed by the sanitizer and re-stamped from the local tip, so the backfill cannot publish a foreign head as the primary's chain-level tip, where it would shift `LATEST`/`SAFE`/`FINALIZED`/`PENDING` resolution chain-wide (`TestSanitizeForeignCacheReplyDropsLatestBlock`, `TestSecondaryHitRestampsLatestBlockFromLocalTip`, `TestSecondaryPoisonedEntrySanitizedForCallerAndBackfill`); `BlocksHashesToHeights` — which sits on the outer `CacheRelayReply`, beyond the sanitizer's reach — is never requested, so foreign heights cannot raise the effective requested block or force archive routing (`TestSecondaryLookupNeverRequestsForeignBlockHashHeights`); and the backfill's validity-floor lift is clamped to the local gated tip, so the lookup **key** itself — the one foreign-vouched scalar the sanitizer cannot reach, since for an explicit-block request it is the raw user-chosen height — cannot be published as the primary's chain-level tip either (`TestSecondaryFutureKeyBackfillNeverRaisesPrimaryChainTip`) |

### Nice-to-have requirements
Expand Down
51 changes: 43 additions & 8 deletions protocol/chainlib/chain_fetcher.go
Original file line number Diff line number Diff line change
Expand Up @@ -154,13 +154,20 @@ func needsLatestBlock(url common.NodeUrl, verifications []VerificationContainer)
if skipVerification(url, v.Name) {
continue
}
if v.Value == "" && v.LatestDistance != 0 {
if verificationNeedsHead(v) {
return true
}
}
return false
}

// verificationNeedsHead reports whether a verification measures a distance from the chain head:
// it has no expected value and a LatestDistance. needsLatestBlock asks it of the verifications that
// will run; ValidateCollect asks it of each one when the head probe failed.
func verificationNeedsHead(v VerificationContainer) bool {
return v.Value == "" && v.LatestDistance != 0
}

// stopValidateRetries reports whether a failed attempt should end Validate's zero-delay
// retry budget early. The budget exists for a cold-booting node; a rate-limited attempt
// is different in kind — the extra requests land inside the same limiter window and only
Expand Down Expand Up @@ -394,6 +401,13 @@ type NodeURLValidation struct {
// decides the health rollup (an endpoint is healthy iff the spec loaded and every verification
// passed, regardless of severity). Severity is intentionally not surfaced: for a health probe a
// failed verification is a failed verification.
//
// A failed head probe (some verification needs the latest block and every attempt to fetch it
// failed) is reported as a failed "fetch-latest-block" row, the verdict Validate gives, and
// LatestBlock stays NOT_APPLICABLE. Each verification that needs the head is then reported failed
// with "latest block unknown" instead of being run: against an unknown head a latest-distance check
// passes vacuously. Every other verification still runs, with 0 (Verify's "unknown") as the latest
// block, so the report still shows what the node does answer (MAG-4278).
func (cf *ChainFetcher) ValidateCollect(ctx context.Context) []NodeURLValidation {
results := make([]NodeURLValidation, 0, len(cf.endpoint.NodeUrls))
defer cf.invalidateVerificationsCache()
Expand All @@ -413,6 +427,7 @@ func (cf *ChainFetcher) ValidateCollect(ctx context.Context) []NodeURLValidation
verifications = verificationsForNodeUrl(url, verifications)

var latestBlock int64
var headErr error
if needsLatestBlock(url, verifications) {
for attempts := 0; attempts < 3; attempts++ {
// Same collection-aware probe as Validate, so `smartrouter health`
Expand All @@ -422,26 +437,46 @@ func (cf *ChainFetcher) ValidateCollect(ctx context.Context) []NodeURLValidation
break
}
}
if err != nil {
// Same verdict as validate: a head the node cannot answer fails the url. Verifying
// against NOT_APPLICABLE instead made every latest-distance check pass vacuously
// against 2^64-1 and reported the url green. Unlike validate this path keeps going:
// the checks that need no head still run below.
headErr = err
latestBlock = 0
nodeResult.Verifications = append(nodeResult.Verifications, VerificationResult{
Name: "fetch-latest-block",
Ok: false,
Error: err.Error(),
})
}
}
if headErr == nil {
nodeResult.LatestBlock = latestBlock
}
nodeResult.LatestBlock = latestBlock

for _, verification := range verifications {
if skipVerification(url, verification.Name) {
continue
}
res := VerificationResult{
Name: verification.Name,
Addon: verification.Addon,
Extension: verification.Extension,
}
if headErr != nil && verificationNeedsHead(verification) {
res.Error = fmt.Errorf("latest block unknown, the head probe failed: %w", headErr).Error()
nodeResult.Verifications = append(nodeResult.Verifications, res)
continue
}
var verifyErr error
for attempts := 0; attempts < 3; attempts++ {
verifyErr = cf.Verify(ctx, verification, uint64(latestBlock))
if stopValidateRetries(verifyErr) {
break
}
}
res := VerificationResult{
Name: verification.Name,
Addon: verification.Addon,
Extension: verification.Extension,
Ok: verifyErr == nil,
}
res.Ok = verifyErr == nil
if verifyErr != nil {
res.Error = verifyErr.Error()
}
Expand Down
101 changes: 101 additions & 0 deletions protocol/chainlib/chain_fetcher_retry_after_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,13 @@ import (
"context"
"errors"
"net/http"
"sync"
"testing"
"time"

"github.com/magma-Devs/smart-router/protocol/common"
"github.com/magma-Devs/smart-router/protocol/lavasession"
pairingtypes "github.com/magma-Devs/smart-router/types/relay"
spectypes "github.com/magma-Devs/smart-router/types/spec"
"github.com/magma-Devs/smart-router/utils"
specutils "github.com/magma-Devs/smart-router/utils/keeper"
Expand Down Expand Up @@ -88,3 +90,102 @@ func TestChainFetcher_FetchLatestBlockNum_PlainErrorIsNotRateLimit(t *testing.T)
_, ok := common.RetryAfterFrom(err)
require.False(t, ok)
}

// validate aborts a url whose head probe fails; ValidateCollect used to carry on and verify every
// latest-distance check against NOT_APPLICABLE (2^64-1 once cast), reporting the url green to
// `smartrouter health`. The failed probe is now a failed fetch-latest-block row, each check that
// needs the head is failed as "latest block unknown", and the rest still run (MAG-4278).
func TestValidateCollect_FailedHeadProbeFailsTheURL(t *testing.T) {
cf := newEthChainFetcherWithRouterErr(t, errors.New("node down"))
cf.endpoint.NodeUrls = []common.NodeUrl{{Url: "http://127.0.0.1:1"}}
verifications, err := cf.chainParser.GetVerifications(nil, "", cf.endpoint.ApiInterface)
require.NoError(t, err)

results := cf.ValidateCollect(context.Background())
require.Len(t, results, 1)
row := results[0]
require.Equal(t, spectypes.NOT_APPLICABLE, row.LatestBlock)
require.Len(t, row.Verifications, 1+len(verifications), "the head row, then one row per verification")
require.Equal(t, "fetch-latest-block", row.Verifications[0].Name)
require.False(t, row.Verifications[0].Ok)
require.Contains(t, row.Verifications[0].Error, "node down")

headDependent, headFree := 0, 0
for i, verification := range verifications {
got := row.Verifications[i+1]
require.Equal(t, verification.Name, got.Name)
require.False(t, got.Ok, "every relay fails, so no verification passes: %s", got.Name)
if verificationNeedsHead(verification) {
headDependent++
require.Contains(t, got.Error, "latest block unknown", got.Name)
require.Contains(t, got.Error, "node down", "the head error is carried: %s", got.Name)
} else {
headFree++
require.NotContains(t, got.Error, "latest block unknown", "a check that needs no head is run: %s", got.Name)
}
}
require.Positive(t, headDependent, "ETH1 has a latest-distance check, or this test proves nothing")
require.Positive(t, headFree, "ETH1 has a check that needs no head, or this test proves nothing")
}

// headOnlyRefusingRouter answers every relay except the head probe, the way a node behind a proxy
// that refuses or rate-limits eth_blockNumber does, and counts the methods it is sent.
type headOnlyRefusingRouter struct {
mu sync.Mutex
sent map[string]int
}

func (r *headOnlyRefusingRouter) SendNodeMsg(ctx context.Context, chainMessage ChainMessageForSend, extensions []string) (*RelayReplyWrapper, common.NodeUrl, string, error) {
method := chainMessage.GetApi().Name
r.mu.Lock()
r.sent[method]++
r.mu.Unlock()
if method == "eth_blockNumber" {
return nil, common.NodeUrl{}, "", errors.New("head probe refused")
}
return &RelayReplyWrapper{StatusCode: http.StatusOK, RelayReply: &pairingtypes.RelayReply{Data: []byte(`{"jsonrpc":"2.0","id":1,"result":"0x1"}`)}}, common.NodeUrl{}, "", nil
}

func (r *headOnlyRefusingRouter) ExtensionsSupported(internalPath string, extensions []string) bool {
return true
}

// A node that answers everything but the head probe keeps the checks it passes in the health
// report: chain-id needs no head and is reported ok, the latest-distance check (pruning) is failed
// as "latest block unknown" without being sent, and the url still does not pass (MAG-4278).
func TestValidateCollect_HeadFailureStillReportsHeadIndependentChecks(t *testing.T) {
router := &headOnlyRefusingRouter{sent: map[string]int{}}
cf := newEthChainFetcherWithRouterErr(t, nil)
cf.chainRouter = router
cf.endpoint.NodeUrls = []common.NodeUrl{{Url: "http://127.0.0.1:1"}}

results := cf.ValidateCollect(context.Background())
require.Len(t, results, 1)
row := results[0]
require.Equal(t, spectypes.NOT_APPLICABLE, row.LatestBlock, "no head was measured")

byName := map[string]VerificationResult{}
allOk := true
for _, v := range row.Verifications {
byName[v.Name] = v
allOk = allOk && v.Ok
}
require.False(t, allOk, "a url whose head probe failed does not pass")
require.False(t, byName["fetch-latest-block"].Ok)
require.Contains(t, byName["fetch-latest-block"].Error, "head probe refused")

chainID, reported := byName["chain-id"]
require.True(t, reported, "chain-id needs no head; it must still be reported")
require.True(t, chainID.Ok, chainID.Error)

pruning, reported := byName["pruning"]
require.True(t, reported, "a check that needs the head is reported failed, not dropped")
require.False(t, pruning.Ok)
require.Contains(t, pruning.Error, "latest block unknown")
require.Contains(t, pruning.Error, "head probe refused", "the head error says why")

router.mu.Lock()
defer router.mu.Unlock()
require.Positive(t, router.sent["eth_chainId"], "the head-independent check was sent")
require.Zero(t, router.sent["eth_getBlockByNumber"], "the latest-distance check is not sent against an unknown head")
}
Original file line number Diff line number Diff line change
Expand Up @@ -361,6 +361,24 @@ func NewBatchMessage(msgs []JsonrpcMessage) (JsonrpcBatchMessage, error) {
// checkJsonrpcEnvelope so the rules stay in lockstep with the single path,
// including the error:null edge case.
func CheckResponseErrorForJsonRpcBatch(data []byte, httpStatusCode int) (hasError bool, errorMessage string) {
return checkResponseErrorForBatch(data, httpStatusCode, func(element []byte, _ int) (bool, string) {
elemHasErr, elemMsg, _ := checkJsonrpcEnvelope(element, "JSON-RPC batch element")
return elemHasErr, elemMsg
})
}

// CheckResponseErrorForTendermintBatch classifies each element with the single-reply Tendermint
// rule (the JSON-RPC envelope, then the embedded result.response.{code,log}), so an application
// error inside a batched abci_query is a node error in a batch exactly as it is on its own. The
// JSON-RPC walker above only reads the envelope, which is right for EVM-shaped replies and wrong
// for Tendermint's 200-with-app-error shape.
func CheckResponseErrorForTendermintBatch(data []byte, httpStatusCode int) (hasError bool, errorMessage string) {
return checkResponseErrorForBatch(data, httpStatusCode, TendermintrpcMessage{}.CheckResponseError)
}

// checkResponseErrorForBatch walks a batch reply and classifies every element with elementCheck.
// Default mode: a single success masks any sibling faults; BatchNodeErrorOnAny flips that.
func checkResponseErrorForBatch(data []byte, httpStatusCode int, elementCheck func(element []byte, httpStatusCode int) (bool, string)) (hasError bool, errorMessage string) {
var (
hasAnySuccess bool
aggregated strings.Builder
Expand All @@ -373,7 +391,7 @@ func CheckResponseErrorForJsonRpcBatch(data []byte, httpStatusCode int) (hasErro
}

walkErr := scanJsonrpcBatchElements(data, func(element []byte) bool {
elemHasErr, elemMsg, _ := checkJsonrpcEnvelope(element, "JSON-RPC batch element")
elemHasErr, elemMsg := elementCheck(element, httpStatusCode)
if elemHasErr {
appendFault(elemMsg)
return true
Expand Down
Loading