Repository navigation
fix: one init retry loop in the chain tracker, and a close frame for a WebSocket subscription the router could not restore (MAG-5022) - #664
Conversation
…op logging a retried start as ERROR (MAG-5022) The head fetch and the block fetch each had their own retry loop, and the two had drifted: the head fetch got five attempts and the block fetch four, under a comment claiming the same spacing. A node that answered the block fetch on its fifth try failed the start. Both steps now go through retryInitFetch: the same attempts, the same flatPollInterval spacing between failures, and the same hand-off of a rate limit to the start loop. A start that ran out of attempts logged "critical -- failed fetching data from the node, chain tracker creation error" at ERROR. Its only production caller, endpointstate.startTrackerWithRetry, retries the start for as long as the endpoint is down and warns once per attempt with this error, so a dead endpoint wrote an ERROR every 30 s or less for the whole outage. The failure is now returned at debug level, naming the step, the attempts and the endpoint, and the context-deadline suggestion rides it as an attribute. No alert rule, dashboard or test reads the old line. The head-only "no usable latest block" error is unchanged. MAG-4278 item 20. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Took 2 minutes
…subscription cannot be restored (MAG-5022) When the router lost an upstream WebSocket subscription and could not restore it, it closed the client's reply channel and said nothing. The client kept an open socket that nothing would push on again, and had no reason to resubscribe. The gRPC manager ends such a stream with Unavailable "upstream subscription lost". A failed restore now gives the subscription up through giveUpSubscription on all four exits: the restores without progress ran out, the reconnect failed, the pool had no connection after it, or the node refused the re-subscribe. Each client's connection is marked with the new chainlib.ErrSubscriptionUpstreamLost under the lock that closes its channel, so the ingress cannot miss it. The ingress then closes the socket with 1013 (try again later) and that reason, as it already does for a client too slow to keep up. The manager's per-connection mark, endErrors, replaces slowClients and holds either reason. A subscription the node itself ends with an error still leaves the socket open, as #519 decided and TestWebSocketSubscriptionTheNodeEndsKeepsTheSocket pins: the node answered, a resubscribe would be ended again, and the close would end the client's other subscriptions on the socket too. A client that unsubscribed is told nothing. The rest of MAG-4278 item 11. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… at every log level (MAG-5022)
retryInitFetch returned its spent-attempts failure through LavaFormatDebugErr.
A LavaFormat call whose level is off returns the bare cause, so at the
router's default --log-level (info) the start loop's WARN and the tracker's
retrying_start state carried only the cause ("head fetch refused"): no
step, no attempt count, and the context-deadline suggestion, an ERROR line
before, appeared nowhere.
The error is now built with fmt.Errorf at every level, wraps the cause,
adds the suggestion on a timeout, and is logged at debug. The endpoint
stays out of the text; the WARN names it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…iled (MAG-5022) Two of the four failed-restore exits had a test. Reverting the reconnect-failed exit to a silent cleanup left the whole subscription test set green. TestRestoreGivesUpWhenTheReconnectFails shortens the pool's ladder, keeps the node down, and requires the upstream-lost mark. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
AnnaR-prog
left a comment
There was a problem hiding this comment.
Reviewed at 6ae85f6, tested merged with main c19b414 and again with current main ada2a55. The merge waits for an owner decision on the 1013 close (664-1), so I'm not approving yet. The two Lows are fixed in be45f9d and 4cccc89, which I pushed. The branch merges clean with current main, including #537's changes to chain_tracker.go.
664-1 (owner decision, Omer or @Tomelia1999): the 1013 close for a lost subscription was decided in the PR, not by an owner.
- The body records the decision itself ("Decision recorded here"). MAG-5022's description says this needs "a close-code decision in the ingress" and is "client-visible behaviour, not a drift fix". Its done-when asks for a recorded decision. Neither MAG-5022 nor MAG-4278 has a comment from Omer or Tom.
- What changes for a client: a socket that used to stay open now closes with 1013 "upstream subscription lost". The close also ends the client's other subscriptions on that socket, including ones on healthy endpoints, so the client has to resubscribe all of them.
- The code does what the body says (see Checked below). Item 20 does not depend on this answer.
- Question: when the router cannot restore a WebSocket subscription, should the client's socket close with 1013 and end its other subscriptions too, or should the client get something else?
664-2 (Low): at the default log level, the spent init failure comes back as the bare cause.
retryInitFetchreturned throughutils.LavaFormatDebugErr. When debug is off,LavaFormatLogreturns the cause unformatted (utils/lavalog.go:512-514).- Probe at
--log-level info, the default (rpcsmartrouter.go:3869): the returned error washead fetch refused, and on a timeoutcontext deadline exceeded. Only at debug did it carry the step, the attempts and the suggestion. - So the start loop's WARN and the tracker's
retrying_starterror name neither the step nor the attempts, although the body says they do. The "context deadline exceeded" suggestion, an ERROR line on main, now appears nowhere. - Fixed in be45f9d: the error is built with
fmt.Errorfat every level, wraps the cause, and adds the suggestion on a timeout. It is logged at debug. The endpoint stays out of the text because the WARN names it. Atinfoit now reads "chain tracker init: failed fetching data from the node: head fetch failed after 5 attempts: head fetch refused". A new test runs atinfoand fails when theLavaFormatDebugErrreturn is put back.
664-3 (Low): two of the four give-up exits had no test.
- I reverted each
giveUpSubscriptioncall tocleanupSubscription, one at a time, and ran the whole subscription test set. The budget exit and the refused re-subscribe exit fail their tests. The reconnect-failed exit and the no-connection-after-reconnect exit stay green. - Fixed in 4cccc89:
TestRestoreGivesUpWhenTheReconnectFailsgives the pool a one-rung ladder, keeps the node down, and requires the upstream-lost mark. It fails with that exit reverted. - The no-connection-after-reconnect exit is still untested. Reaching it needs a pool that reconnects and then hands out no connection.
Checked and correct:
- Item 20: both steps get initRetriesCount+1 attempts, the same spacing and the same 429 hand-off. With main's
chain_tracker.gooverlaid (it includes #579), the first three tests fail and the cancel test passes, as the body says. The block fetch now records every outcome and waits out an existing hold-off, as the head fetch did.fetchAllPreviousBlocksrecords nothing itself, so a 429 is still one strike. With #537 merged, the init wait reads the hold-off throughURLReadyAt, and the init tests and #537's hold-off tests pass together. Nothing in this repo or its docs reads the old ERROR line. - Item 11: on all four restore exits, the clients are marked under the lock that closes their channels, before the close. A node-ended subscription and an unsubscribed client stay quiet.
- The body's two mutants reproduce. With no marking, both restore tests fail. With the ingress closing for slow clients only, the ingress test fails. The node-ended and outage tests stay green.
- A given-up client's mark, sticky entry and limiter are released when its socket closes:
UnsubscribeAllcallsforgetClientLockedeven when nothing is left to unsubscribe. - With #677 on the same tree, a given-up client resubscribes on a new connection. The subscribe fails over past the endpoint that refused it, and the new connection has no end mark (probe, 3 of 3 under -race).
Tested: On c19b414: build and vet on main+664, main+fix, main+664+677 and main+fix+677; protocol/chaintracker, protocol/chainlib and protocol/endpointstate; full protocol/rpcsmartrouter on main+664+677 (304 s); the 261 subscription tests under -race on main+664+677 and on main+fix+677; -race ×3 on the new and changed tests with the fix. Mutants: main's tracker, no marking, slow-client-only ingress, each exit reverted, and the fix reverted. On ada2a55 with the fix: build and vet; this PR's tests and the log-level probe; full protocol/rpcsmartrouter and the 262 subscription tests under -race, both with #677; chainlib, chaintracker, endpointstate and holdoff. golangci-lint --new-from-rev origin/main on chaintracker, chainlib and rpcsmartrouter: 0 issues on both bases. TestChainTrackerMaintainMemory/one_long_test failed once in 12 runs under heavy load. It is a timing test in the poll path and passes when run alone.
Jira ticket: MAG-5022
The code half of MAG-5022, which holds the four MAG-4278 items that had no carrier. Two commits, one per item. Items 3 and 4 wait for Omer (below).
Item 20: the tracker's two init retry loops
The head fetch and the block fetch had their own retry loops, and they had drifted: five attempts for the head, four for the blocks, under a comment claiming the same spacing. A node that answered the block fetch on its fifth try failed the start. Both steps now go through one
retryInitFetch, with the same attempts, the sameflatPollIntervalspacing and the same rate-limit hand-off to the start loop.A start that ran out of attempts logged
critical -- failed fetching data from the node, chain tracker creation errorat ERROR. The only production caller,endpointstate.startTrackerWithRetry, retries the start for as long as the endpoint is down and warns once per attempt with this error. So a dead endpoint wrote an ERROR every 30 s or less for the whole outage. The failure is now returned at debug level, naming the step, the attempts and the endpoint, and the per-attempt WARN still carries it. No alert rule, dashboard or test in smart-router-infrastructure, the Helm chart, smart-router-automation or the docs reads the old line.Item 11: a given-up WebSocket subscription's end status
When the router lost an upstream WebSocket subscription and could not restore it, it closed the client's reply channel and said nothing. The client kept a socket nothing would push on again, with no reason to resubscribe. The gRPC manager ends such a stream with
Unavailable"upstream subscription lost".Decision recorded here: the socket closes with 1013 (try again later) and the reason
upstream subscription lost. The ingress already sends that close code to a client too slow to keep up (#519), so a client handles one router close code for "reconnect and resubscribe", and the reason matches the gRPC status. The close also ends the client's other subscriptions on the socket, as the slow-client close does, because WebSocket has no per-subscription end frame.All four failed-restore exits take it: the restores without progress ran out, the reconnect failed, the pool had no connection after it, or the node refused the re-subscribe. The mark is set under the lock that closes the client's channel, so the ingress cannot miss it.
endErrorsreplacesslowClientsand holds either reason.What still keeps the socket open: a subscription the node itself ends with a JSON-RPC error. #519 decided that, and
TestWebSocketSubscriptionTheNodeEndsKeepsTheSocketpins it. The node answered, a resubscribe would be ended again, and the close would end the client's other subscriptions. A client that unsubscribed is told nothing either.Items 3 and 4: waiting for Omer
Both change admission or availability semantics, so this PR leaves them alone. They were asked on MAG-4278 on 2026-10-28, and again on MAG-5022 this morning.
Tests
Item 20,
protocol/chaintracker/init_retry_test.go:The first three fail with main's
chain_tracker.gooverlaid. The cancel test passes on both, as it should.Item 11:
TestRestoreBudget_GivesUpOnceTheBudgetIsSpentand the newTestRestoreGivesUpWhenTheNodeRefusesTheResubscriberequire the reason, read once.TestRestoreBudget_RefillsAfterASubscriptionStaysUpandTestUnsubscribeDuringOutage_ReleasesTheClientrequire none.TestConsumerWebsocketManager_GivenUpSubscriptionIsClosedWithTryAgainLaterrequires the 1013 close with the reason.Main's files do not compile against the new tests, so the evidence is two single-change mutants of this PR. With the give-up marking removed, both restore tests fail. With the ingress closing only for slow clients, the ingress test fails. The node-ended and outage tests stay green under both.
go build -buildvcs=false ./...andgo test ./protocol/... ./utils/...pass on each commit, each from its own detached worktree.-racepasses on the new and changed tests.golangci-lint --new-from-rev origin/mainonchaintracker,chainlibandrpcsmartrouter: 0 issues.🤖 Generated with Claude Code