Repository navigation
fix: eleven behaviour drifts between duplicated code paths (MAG-4278) - #523
Conversation
…G-4278) The GraphQL listener still read fiberCtx.GetReqHeaders(), the pre-MAG-3881 shape: fiber's header strings alias the per-connection buffer, and the metrics goroutine reads them after the reply. Every other listener uses detachedReqHeaders. The shared keep-no-header-past-the-request test now covers the GraphQL listener too. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Took 14 minutes
… rule (MAG-4278)
A single Tendermint reply is checked by TendermintrpcMessage.CheckResponseError,
which reads the JSON-RPC envelope and then result.response.{code,log}; a batch
went through the JSON-RPC walker, which reads only the envelope. An
application error inside a batched abci_query therefore counted as a success.
The batch walker now takes the per-element rule as a parameter; the JSON-RPC
batch keeps the envelope-only rule, the Tendermint batch uses the single-reply
one.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ndler too (MAG-4278) The catch-all REST handler maps APINotSupportedError to 501; the POST handler masked the same refusal as a GUID 500. Both handlers now give the same answer, pinned by a listener test that posts and gets an undeclared path. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…r proxies (MAG-4278) The REST and JSON-RPC proxies run an httpClient.Do failure through HandleNodeError, so the router sees a coded, retryable LavaError and the node's address is handled by the classifier. The Tendermint URI proxy returned the raw *url.Error. A refused dial on the URI path now resolves to LavaErrorConnectionRefused like the REST path. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ils (MAG-4278) validate aborts a url whose three head probes fail; ValidateCollect carried on with NOT_APPLICABLE, which casts to 2^64-1, so every latest-distance check passed vacuously and smartrouter health reported the url green. The collect path now records a failed fetch-latest-block verification and skips the rest, the verdict validate already gives. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… (MAG-4278) Every other locally generated JSON-RPC reply echoes the request id verbatim through MarshalJsonRPCErrorWithRequestID; the rate-limit reply coerced it to an int, answering string ids (and numeric strings, through Atoi on the quoted token) with id 0. It now uses the shared marshaller. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… (MAG-4278) The per-provider release and the operator reset both write BlockedProviderSessionUnusedStatus when a blocked provider returns to rotation. The pool-empty bulk release (setValidAddressesToDefaultValue) did not, so a provider that came back through it kept BlockedProviderSessionUsedStatus and every later failure read as a second-chance attempt that never re-blocked it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… (MAG-4278) The remote loaders warn about every file they could not decode; the local directory loader recorded the error and dropped it as long as one spec loaded, so a corrupt file vanished silently. The walk is now its own helper that returns what it recorded, and the loader warns when specs loaded with errors. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ache hit (MAG-4278) A primary cache hit passes the spec's pass_reply headers through filterCachedReplyMetadata; a secondary hit of the same entry went through the foreign-reply sanitiser's own, narrower allow-list and kept only the transport pair. The secondary path now runs the same filter over the foreign entry's headers, so both tiers answer with the same header set. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…g/reset-all (MAG-4278) /debug/reset-all claims to be the superset of /debug/reset-scores, but it never reset the per-endpoint chain trackers (so the tip re-seeded from the pre-reset trackers and the process-wide endpoint-tip store) nor the probe back-off. Both endpoints now reset the same stores, and reset-scores reports the probe back-off count it already reset. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…G-4278) The test command read --use-tls and --allow-insecure-connection through viper, but only the root command binds its flag set to viper, so both were always false there. Neither the test nor the health command accepted the --github-token/--gitlab-token flags the root command uses for private spec repositories, and both loaded specs with empty tokens. The TLS flags are read from the command's own flag set, and both subcommands register and pass the spec tokens. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
AnnaR-prog
left a comment
There was a problem hiding this comment.
Reviewed at c276cc0. I tested it as it will land: this branch merged with current main (8ad0ff6). The merge is clean apart from one test call, because #511 made resetValidAddresses return a single value.
All eleven fixes do what they claim on current main.
- For items 2, 6, 8, 9, 10, 12 and 19, reverting the fix hunk makes its test fail.
- Item 7's walker composes with #514's batch fold:
TestTendermintRpcBatchCallstill sees requested 100 and earliest 99. JSON-RPC batches keep the envelope-only rule. - The
TestBlockReason_SurvivesTheEpochTransitionfailure is the pre-existing race you describe. It failed 3 of 200 runs here and 6 of 200 on main, with the same message.
523-1 (Medium, item 14): smartrouter test rpcsmartrouter now dials gRPC with TLS only.
- Before: the command read
--use-tlsthrough viper, which has no binding on that command, soUseTLSwasfalse. The connector dialled plaintext first and fell back to TLS, so both kinds of listener worked. - Now: the flag's registered default,
true, takes effect for the first time. The connector sets up TLS up front and never tries plaintext. - Impact: a router's gRPC listener is plaintext unless the listen endpoint enables TLS. So the command's own example,
127.0.0.1:3334 LAV1 grpc, no longer connects without--use-tls=false. - Probe: against a plaintext gRPC server,
AuthConfig{}connects, and the command's defaultAuthConfigfails withcontext deadline exceeded. - Not stated: nothing in the PR marks this as a behaviour change.
- Proposed fix: register
--use-tlswith a default offalseon the test command. The flag then works, the default behaviour stays the same, and--allow-insecure-connectionstill reaches the TLS retry.
523-2 (Low, item 10): a rate-limited batch or notification gets "id":1.
MarshalJsonRPCErrorWithRequestIDechoes a top-levelid. A batch frame and a notification have none, so the reply keeps the template's hardcoded1. Before this PR it was0.- A client whose ids start at 1 can settle the wrong request with that reply, while every batch member waits for its timeout.
- I'm pushing a fix that goes through
buildLocalReply, as the unsubscribe replies do: one error per batch member that has an id, and nothing for a notification.
523-3 (Low, item 8): the redemption-flag reset also runs on the epoch rebuild.
- The write sits in
setValidAddressesToDefaultValue, which also servesReleaseEpochRebuild. - On a real epoch tick the sessions are fresh objects, so the write does nothing.
- The failed-provider readmit and
/debug/reset-pairingrebuild within the same epoch and reuse the same session objects. There, the write clears the flag of a redemption that is still in flight, so when that redemption succeeds, the provider is no longer returned to rotation. - I'm pushing a fix that skips the write for
ReleaseEpochRebuild, with a test.
523-4 (Low, item 15): the reset test checks only strings, and the change to reset-scores isn't stated.
TestDebugResetAll_SmartRouter_ResetsTrackersAndProbeBackoffLikeResetScoresbuilds the mux without a router, so it still passes with the two new reset calls removed.- On main, neither reset-scores nor reset-all reset the probe back-off; only
/debug/reset-probe-backoffdid. So "the probe back-off count it already reset" isn't right: this PR adds the reset to both. - Two comments still say no other reset reaches the back-off: the one on
resetAllProbeBackoffand the one on/debug/reset-probe-backoff. - I'm pushing a test with a real tracker, which checks that reset-all drops the tip, plus the comment fixes.
523-5 (Low, item 6): docs/SECONDARY-CACHE.md is out of date.
- It still says a secondary hit replays only Content-Type and Content-Encoding "by design" (
:59-67,:568-576,:706). - Since this item, a secondary hit also replays
Retry-After,Dateand the spec's reply headers, like a primary hit. - I'm pushing the doc update.
523-6 (Low, item 9): ValidateCollect stops reporting a URL once its head probe fails.
- Its doc and README line 83 promise "every verification for every node URL".
- With this change, a URL whose head probe fails gets one
fetch-latest-blockrow and nothing else. - So the checks that need no head (
chain-id,trustless-rpc, the archivepruningvalue) aren't reported. A rate-limitedeth_blockNumberalone now turns a URL red with no other diagnostics. - The verdict itself is right. The options:
- keep reporting the head-independent checks after the failed row; or
- update the doc and the README.
- Either way, the test's loop over
row.Verifications[1:]runs zero times, so it asserts nothing.
523-7 (Low, item 7): the test doesn't pin the parser wiring.
- It calls
CheckResponseErrorForTendermintBatchdirectly. IfnewBatchChainMessageis pointed back at the JSON-RPC checker, the test stays green. - I'm adding a parser-level test that parses a batched
abci_querywith the LAVA spec.
523-8 (Nit, item 19): a comment overstates the change.
- The new comment says the URI path now masks the transport error.
- The coded error still carries the node URL (
NewLavaError(classified, err.Error())), as it does on the REST and JSON-RPC paths. - I'm rewording it.
Description:
- The lint numbers are stale. Main is at 0 findings, and this PR adds none.
- Item 12 reaches fewer specs than described. The bundled cosmoshub and lava specs inherit a POST collection, so only specs with no POST collection answer 501.
The merge is clean apart from one test call: #511 (MAG-4244) made resetValidAddresses return only didReset, so the item-8 test reads the single result. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… notification not at all (MAG-4278) MarshalJsonRPCErrorWithRequestID echoes a top-level id and otherwise keeps the template's own id, which for the rate-limit error is 1. A batch frame and a notification have no top-level id, so a rate-limited batch was answered with one object carrying "id":1 (before the id fix it was 0). Clients often start their ids at 1, so that reply can settle a different request, and every other member of the batch waits for its timeout. The rate-limit reply now goes through buildLocalReply like the other local replies in this file: a batch gets an array with one error per member that has an id, and a notification gets no reply (JSON-RPC 2.0 sections 4.1 and 6). The read loop sends nothing when the reply is empty. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…build re-blocks (MAG-4278) The redemption-flag reset added to setValidAddressesToDefaultValue runs for all three of its callers, not only the pool-empty release. On ReleaseEpochRebuild, UpdateAllProviders re-blocks every carried-over provider right after the drain, so none of them is back in rotation. The failed-provider readmit and /debug/reset-pairing call UpdateAllProviders in the same epoch with the same session objects, so the reset erased the flag of a redemption session still in flight. OnSessionDone reads that flag to return a provider whose redemption succeeded to rotation, so that success no longer did. The reset now skips ReleaseEpochRebuild, as the release log line in the same function already does. Every path that later returns such a provider to rotation clears the flag itself. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…te the back-off comments (MAG-4278) TestDebugResetAll_SmartRouter_ResetsTrackersAndProbeBackoffLikeResetScores builds the debug mux with no router, so it checks only the strings in the reply: it stays green with the two new reset calls removed. The new test seeds a real per-endpoint tip through an EndpointMonitor and checks that reset-all drops it. Before this change neither reset-scores nor reset-all touched the probe back-off; only /debug/reset-probe-backoff did. Now both reset it, but the comments on resetAllProbeBackoff and /debug/reset-probe-backoff still said no other reset reaches it, and the reset-scores comment did not list it. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…r (MAG-4278) TestCheckResponseErrorForTendermintBatch_SeesTheAppErrorASingleReplySees calls CheckResponseErrorForTendermintBatch directly. The fix it covers is the one-line wiring in TendermintChainParser.newBatchChainMessage, and the test stays green when that line points back at the JSON-RPC walker. The new test parses a batched abci_query with the Tendermint parser and classifies the reply through the chain message, so it fails if the wiring is reverted. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ary hit does (MAG-4278) The secondary path now runs the primary tier's reply-header filter over the foreign entry's headers, so a secondary hit replays Content-Type, Content-Encoding, Retry-After, Date and the spec's reply-direction headers, minus any router-owned name. docs/SECONDARY-CACHE.md still said the secondary keeps only Content-Type and Content-Encoding "by design", in the "No identity leakage" bullet, in the demo walkthrough and in the requirements table. Neither PR stacked on this one (#524, #525) touches the document. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…it does not mask them (MAG-4278) The comment on SendURI's transport-error branch said the error is classified "and masked" and that the raw *url.Error "used to leave here with the node's address in it", which reads as if the address no longer does. handleAndClassify builds the coded error with NewLavaError(classified, err.Error()), so the text still carries the full node URL, as on the REST and JSON-RPC paths. A node URL that embeds an API key in its path is no more hidden here than before, and the comment should not suggest otherwise. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…default (MAG-4278) Reading --use-tls from the test command's own flag set made its registered default (true) take effect for the first time. Before, viper had no binding on that command and read false, and the gRPC connector dialled plaintext first and TLS only if that failed. With true it dials TLS only, so a router's gRPC listener, which is plaintext unless its endpoint enables TLS, became unreachable without --use-tls=false. That includes the command's own Example (127.0.0.1:3334 LAV1 grpc). --use-tls now defaults to false, so the command dials as it did before; --use-tls forces TLS, and --allow-insecure-connection accepts a self-signed certificate on either TLS path. The flag help says what the default does, and a test dials a plaintext gRPC listener with the default flags. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ateCollect's head probe fails (MAG-4278) Failing the url when its head probe fails is right: verifying against NOT_APPLICABLE made every latest-distance check pass vacuously. But ending the url there dropped every other check from the smartrouter health report, against ValidateCollect's documented contract to run every verification. A node that answers chain-id and refuses or rate-limits only eth_blockNumber (a 429 ends the head probe's retries at once) then read the same as a dead one. After the failed fetch-latest-block row, ValidateCollect now goes through every verification. One that needs the head (the predicate needsLatestBlock applies, now verificationNeedsHead) is reported failed with "latest block unknown" and the head error, without being sent. Every other one runs, with 0 as the latest block, which Verify reads as unknown. LatestBlock stays NOT_APPLICABLE, so the row keeps no height and the health rollup still fails it. The PR test's loop over the remaining rows ran zero times. It now checks each row against the spec's verifications. A new test uses a router that refuses only eth_blockNumber: chain-id is reported ok, pruning fails as head-unknown without being sent, and the url does not pass. A health_cmd test pins the rollup: the row is not ok, it is not backfilled, and the top-level ok is false. The README describes the row. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Review fixes pushed: b9a1443The branch now merges main (c258f7f). #511 made
Verified on b9a1443:
|
… into refactor/MAG-4279-batch1a One conflict, protocol/chainlib/chainproxy/connector.go: #511 (MAG-4244) folded the connector's three dial sites into a grpcDialOptions helper on main; this branch replaces the same three sites with dialOptions over the shared common.NodeUrl.GRPCDialOptions, which carries the same MAG-2218 auth-headers and receive-size cap. Keep this branch's side. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…MAG-4279-batch1a Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ew fixes) into refactor/MAG-4279-descriptor-resolver Clean merge. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…s) into refactor/MAG-4279-descriptor-resolver Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…mption-flag test (MAG-4278) TestSameEpochRebuild_KeepsTheRedemptionFlagOfAReblockedProvider went through UpdateAllProviders twice, and so raced its deferred checkAndUnblockHealthyReBlockedProviders: that runs on a goroutine after a random sleep of up to 500 ms and may release the re-blocked provider, clearing its flag as every release does, before the assertions run. It failed that way in a full-suite run (the same race the older TestBlockReason_SurvivesTheEpochTransition is known for). The test now runs setValidAddressesToDefaultValue with ReleaseEpochRebuild under the lock, the way UpdateAllProviders does, and checks the flag the in-flight redemption reads. 30/30 green; with the drain's route guard removed it fails. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…er starts (MAG-4278) start() reset latestChangeTime with a plain write, outside blockQueueMu, although ResetLatestBlock writes the same field under that lock and the file's own rule is that every access outside the lock goes through the locked helpers. A reset that lands while a tracker is still starting is a data race: /debug/reset-scores could already hit it, and with this branch /debug/reset-all resets the trackers too. Under -race, TestDebugResetAll_SmartRouter_DropsTheEndpointTip, which resets a tracker it has just created, reported it (ResetLatestBlock against start). start() now goes through setLatestChangeTime. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Two more fixes pushed: a40be3fBoth were found in an integration run of this PR with #519, #520, #522, #524 and #525 on main.
Already failing on main: On the integration tree (main plus all six PRs):
|
There was a problem hiding this comment.
Approving at 6b298fe: main 027cc1c (after #519 and #520) plus this PR and the review fixes, including the two follow-up fixes from the integration run (the redemption-flag test and the latestChangeTime race).
- On this exact tree,
go build ./...,go vet ./protocol/... ./utils/...and the tests of./protocol/lavasession/,./protocol/rpcsmartrouter/,./protocol/chainlib/...,./protocol/common/...,./protocol/chaintracker/and./utils/...pass. - I also validated this PR together with #519, #520, #522, #524 and #525 on one integration tree: golangci-lint reports 0 issues,
go test ./...passes, and-raceis clean on the touched packages. go test -race ./protocol/chaintracker/fails on main too (test-level races that predate this PR); it is not counted above.
Jira ticket: MAG-4278
What this fixes
MAG-4278 lists twenty places where two copies of the same logic drifted apart and the drift changes behaviour. This PR takes the eleven that are self-contained: no open PR edits the same code and none touches admission or failover semantics. Each item is its own commit, with a test that fails on the old copy and pins the copy that was right.
detachedReqHeaderslike the other three listeners (it was created from the pre-MAG-3881 shape; fiber's header strings alias the per-connection buffer and the metrics goroutine reads them after the reply)TestListeners_KeepNoHeaderPastTheRequestfilterCachedReplyMetadataa primary hit does, so the spec'spass_replyheaders (the Aptos ledger-state set, for one) are replayed from both tiers instead of only the transport pairTestSecondaryCacheHitReplaysTheSameFilteredHeadersAsAPrimaryHitresult.response.{code,log}); the batch walker takes the element rule as a parameter and the JSON-RPC batch keeps its envelope-only ruleTestCheckResponseErrorForTendermintBatch_SeesTheAppErrorASingleReplySeesBlockedProviderSessionUsedStatuslike the per-provider release and the operator reset, so a provider it returns to rotation can be re-blockedTestResetValidAddresses_ClearsTheRedemptionFlagLikeTheOtherReleasesValidateCollectfails a url whose head probe failed (afetch-latest-blockrow, no vacuous latest-distance passes against 2^64-1) likevalidatedoes, sosmartrouter healthstops reporting such a node greenTestValidateCollect_FailedHeadProbeFailsTheURLMarshalJsonRPCErrorWithRequestIDinstead of coercing it to an int (string ids, and numeric strings throughAtoion the quoted token, came back as"id":0)TestHandleRateLimitReached_EchoesTheCallersIdTestRestListener_UndeclaredVerbIs501OnEveryHandlertest rpcsmartrouterreads--use-tlsand--allow-insecure-connectionfrom its own flag set (only the root command binds its flags to viper, so both readfalsebefore);testandhealthaccept and pass--github-token/--gitlab-tokenfor private spec repositoriesTestTestAndHealthCommandsReadTheirOwnFlags/debug/reset-allresets the chain trackers (which also drops the process-wide endpoint-tip entries, the "tip re-seeds after reset" behaviour) and the probe back-off likereset-scores;reset-scoresreports the probe back-off count it already resetTestDebugResetAll_SmartRouter_ResetsTrackersAndProbeBackoffLikeResetScores--use-static-specfile made its chain vanish with no log lineTestGetAllSpecsFromLocalDir_ReportsACorruptSiblingSendURIruns a transport failure throughHandleNodeErrorlike the REST and JSON-RPC proxies, so the router gets a coded LavaError (LavaErrorConnectionRefusedfor a refused dial) instead of the raw*url.ErrorTestTendermintSendURI_ClassifiesTransportErrorsLikeTheOtherProxiesScope notes:
healthcommand registers--use-tls/--allow-insecure-connectionbut never reads them, before and after this PR. That is the "never-read CLI flags" follow-up on MAG-4243, not a drift between copies, so it stays there.reset-allnow listschain-trackersandprobe-backoffin itsclearedarray;reset-scoresgains aprobe_backoff_resetcount in its reply.Not in this PR (the ticket's remaining items)
direct_ws_subscription_manager.gois edited by fix(rpcsmartrouter): join a WebSocket subscription that became active between the two checks (MAG-2546) #508, fix(rpcsmartrouter): let a WebSocket teardown win over a restore in flight (MAG-2547) #509, perf(protocol): enable six linters and act on the audit's lint-derived efficiency signals (MAG-4244) #511 and fix(rpcsmartrouter): retire dead WebSocket connections, drop slow subscription clients, and share one gRPC stream per subscribe (MAG-4242) #519; they land after those.429and "too many requests" to the registry's rate-limit rows, which is half of the gap; the remainder waits for it.Verification
go build ./...,go vet ./...clean on the committed tree.go test -count=1 ./...on the committed tree: 37 packages ok.protocol/lavasessionfailed once onTestBlockReason_SurvivesTheEpochTransition: the epoch health probe released the provider before the assertion ran, a known timing race in that test; the probe never reads the flag this PR writes andblockProvidernever sets it, so the drain-loop change is a no-op on that path. 13/13 in isolation and the whole package green on rerun.golangci-lint runwith the repo config: exactly the 34 findingsorigin/mainreports with this linter version, none new.🤖 Generated with Claude Code