Skip to content

fix: one init retry loop in the chain tracker, and a close frame for a WebSocket subscription the router could not restore (MAG-5022) - #664

Open
avitenzer wants to merge 4 commits into
mainfrom
fix/MAG-5022-tracker-init-retry
Open

avitenzer wants to merge 4 commits into
mainfrom
fix/MAG-5022-tracker-init-retry

Conversation

@avitenzer

Copy link
Copy Markdown
Contributor

Jira ticket: MAG-5022

The code half of MAG-5022, which holds the four MAG-4278 items that had no carrier. Two commits, one per item. Items 3 and 4 wait for Omer (below).

Item 20: the tracker's two init retry loops

The head fetch and the block fetch had their own retry loops, and they had drifted: five attempts for the head, four for the blocks, under a comment claiming the same spacing. A node that answered the block fetch on its fifth try failed the start. Both steps now go through one retryInitFetch, with the same attempts, the same flatPollInterval spacing and the same rate-limit hand-off to the start loop.

A start that ran out of attempts logged critical -- failed fetching data from the node, chain tracker creation error at ERROR. The only production caller, endpointstate.startTrackerWithRetry, retries the start for as long as the endpoint is down and warns once per attempt with this error. So a dead endpoint wrote an ERROR every 30 s or less for the whole outage. The failure is now returned at debug level, naming the step, the attempts and the endpoint, and the per-attempt WARN still carries it. No alert rule, dashboard or test in smart-router-infrastructure, the Helm chart, smart-router-automation or the docs reads the old line.

Item 11: a given-up WebSocket subscription's end status

When the router lost an upstream WebSocket subscription and could not restore it, it closed the client's reply channel and said nothing. The client kept a socket nothing would push on again, with no reason to resubscribe. The gRPC manager ends such a stream with Unavailable "upstream subscription lost".

Decision recorded here: the socket closes with 1013 (try again later) and the reason upstream subscription lost. The ingress already sends that close code to a client too slow to keep up (#519), so a client handles one router close code for "reconnect and resubscribe", and the reason matches the gRPC status. The close also ends the client's other subscriptions on the socket, as the slow-client close does, because WebSocket has no per-subscription end frame.

All four failed-restore exits take it: the restores without progress ran out, the reconnect failed, the pool had no connection after it, or the node refused the re-subscribe. The mark is set under the lock that closes the client's channel, so the ingress cannot miss it. endErrors replaces slowClients and holds either reason.

What still keeps the socket open: a subscription the node itself ends with a JSON-RPC error. #519 decided that, and TestWebSocketSubscriptionTheNodeEndsKeepsTheSocket pins it. The node answered, a resubscribe would be ended again, and the close would end the client's other subscriptions. A client that unsubscribed is told nothing either.

Items 3 and 4: waiting for Omer

Both change admission or availability semantics, so this PR leaves them alone. They were asked on MAG-4278 on 2026-10-28, and again on MAG-5022 this morning.

Tests

  • Item 20, protocol/chaintracker/init_retry_test.go:

    • both steps get the same attempts, so a node that answers each step on its last try starts;
    • a spent step fails with its own cause, after exactly its attempts;
    • the spent failure logs no ERROR;
    • a cancel between attempts returns at once.

    The first three fail with main's chain_tracker.go overlaid. The cancel test passes on both, as it should.

  • Item 11:

    • TestRestoreBudget_GivesUpOnceTheBudgetIsSpent and the new TestRestoreGivesUpWhenTheNodeRefusesTheResubscribe require the reason, read once.
    • TestRestoreBudget_RefillsAfterASubscriptionStaysUp and TestUnsubscribeDuringOutage_ReleasesTheClient require none.
    • At the ingress, TestConsumerWebsocketManager_GivenUpSubscriptionIsClosedWithTryAgainLater requires the 1013 close with the reason.

    Main's files do not compile against the new tests, so the evidence is two single-change mutants of this PR. With the give-up marking removed, both restore tests fail. With the ingress closing only for slow clients, the ingress test fails. The node-ended and outage tests stay green under both.

  • go build -buildvcs=false ./... and go test ./protocol/... ./utils/... pass on each commit, each from its own detached worktree. -race passes on the new and changed tests.

  • golangci-lint --new-from-rev origin/main on chaintracker, chainlib and rpcsmartrouter: 0 issues.

🤖 Generated with Claude Code

avitenzer and others added 2 commits November 8, 2026 16:45
…op logging a retried start as ERROR (MAG-5022)

The head fetch and the block fetch each had their own retry loop, and the
two had drifted: the head fetch got five attempts and the block fetch
four, under a comment claiming the same spacing. A node that answered the
block fetch on its fifth try failed the start.

Both steps now go through retryInitFetch: the same attempts, the same
flatPollInterval spacing between failures, and the same hand-off of a
rate limit to the start loop.

A start that ran out of attempts logged "critical -- failed fetching data
from the node, chain tracker creation error" at ERROR. Its only production
caller, endpointstate.startTrackerWithRetry, retries the start for as long
as the endpoint is down and warns once per attempt with this error, so a
dead endpoint wrote an ERROR every 30 s or less for the whole outage. The
failure is now returned at debug level, naming the step, the attempts and
the endpoint, and the context-deadline suggestion rides it as an
attribute. No alert rule, dashboard or test reads the old line. The
head-only "no usable latest block" error is unchanged.

MAG-4278 item 20.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

Took 2 minutes
…subscription cannot be restored (MAG-5022)

When the router lost an upstream WebSocket subscription and could not
restore it, it closed the client's reply channel and said nothing. The
client kept an open socket that nothing would push on again, and had no
reason to resubscribe. The gRPC manager ends such a stream with
Unavailable "upstream subscription lost".

A failed restore now gives the subscription up through giveUpSubscription
on all four exits: the restores without progress ran out, the reconnect
failed, the pool had no connection after it, or the node refused the
re-subscribe. Each client's connection is marked with the new
chainlib.ErrSubscriptionUpstreamLost under the lock that closes its
channel, so the ingress cannot miss it. The ingress then closes the
socket with 1013 (try again later) and that reason, as it already does
for a client too slow to keep up. The manager's per-connection mark,
endErrors, replaces slowClients and holds either reason.

A subscription the node itself ends with an error still leaves the socket
open, as #519 decided and TestWebSocketSubscriptionTheNodeEndsKeepsTheSocket
pins: the node answered, a resubscribe would be ended again, and the
close would end the client's other subscriptions on the socket too. A
client that unsubscribed is told nothing.

The rest of MAG-4278 item 11.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@avitenzer
avitenzer requested a review from a team as a code owner November 8, 2026 14:57
AnnaR-prog and others added 2 commits November 9, 2026 12:36
… at every log level (MAG-5022)

retryInitFetch returned its spent-attempts failure through LavaFormatDebugErr.
A LavaFormat call whose level is off returns the bare cause, so at the
router's default --log-level (info) the start loop's WARN and the tracker's
retrying_start state carried only the cause ("head fetch refused"): no
step, no attempt count, and the context-deadline suggestion, an ERROR line
before, appeared nowhere.

The error is now built with fmt.Errorf at every level, wraps the cause,
adds the suggestion on a timeout, and is logged at debug. The endpoint
stays out of the text; the WARN names it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…iled (MAG-5022)

Two of the four failed-restore exits had a test. Reverting the
reconnect-failed exit to a silent cleanup left the whole subscription test
set green. TestRestoreGivesUpWhenTheReconnectFails shortens the pool's
ladder, keeps the node down, and requires the upstream-lost mark.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

@AnnaR-prog AnnaR-prog left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at 6ae85f6, tested merged with main c19b414 and again with current main ada2a55. The merge waits for an owner decision on the 1013 close (664-1), so I'm not approving yet. The two Lows are fixed in be45f9d and 4cccc89, which I pushed. The branch merges clean with current main, including #537's changes to chain_tracker.go.

664-1 (owner decision, Omer or @Tomelia1999): the 1013 close for a lost subscription was decided in the PR, not by an owner.

  • The body records the decision itself ("Decision recorded here"). MAG-5022's description says this needs "a close-code decision in the ingress" and is "client-visible behaviour, not a drift fix". Its done-when asks for a recorded decision. Neither MAG-5022 nor MAG-4278 has a comment from Omer or Tom.
  • What changes for a client: a socket that used to stay open now closes with 1013 "upstream subscription lost". The close also ends the client's other subscriptions on that socket, including ones on healthy endpoints, so the client has to resubscribe all of them.
  • The code does what the body says (see Checked below). Item 20 does not depend on this answer.
  • Question: when the router cannot restore a WebSocket subscription, should the client's socket close with 1013 and end its other subscriptions too, or should the client get something else?

664-2 (Low): at the default log level, the spent init failure comes back as the bare cause.

  • retryInitFetch returned through utils.LavaFormatDebugErr. When debug is off, LavaFormatLog returns the cause unformatted (utils/lavalog.go:512-514).
  • Probe at --log-level info, the default (rpcsmartrouter.go:3869): the returned error was head fetch refused, and on a timeout context deadline exceeded. Only at debug did it carry the step, the attempts and the suggestion.
  • So the start loop's WARN and the tracker's retrying_start error name neither the step nor the attempts, although the body says they do. The "context deadline exceeded" suggestion, an ERROR line on main, now appears nowhere.
  • Fixed in be45f9d: the error is built with fmt.Errorf at every level, wraps the cause, and adds the suggestion on a timeout. It is logged at debug. The endpoint stays out of the text because the WARN names it. At info it now reads "chain tracker init: failed fetching data from the node: head fetch failed after 5 attempts: head fetch refused". A new test runs at info and fails when the LavaFormatDebugErr return is put back.

664-3 (Low): two of the four give-up exits had no test.

  • I reverted each giveUpSubscription call to cleanupSubscription, one at a time, and ran the whole subscription test set. The budget exit and the refused re-subscribe exit fail their tests. The reconnect-failed exit and the no-connection-after-reconnect exit stay green.
  • Fixed in 4cccc89: TestRestoreGivesUpWhenTheReconnectFails gives the pool a one-rung ladder, keeps the node down, and requires the upstream-lost mark. It fails with that exit reverted.
  • The no-connection-after-reconnect exit is still untested. Reaching it needs a pool that reconnects and then hands out no connection.

Checked and correct:

  • Item 20: both steps get initRetriesCount+1 attempts, the same spacing and the same 429 hand-off. With main's chain_tracker.go overlaid (it includes #579), the first three tests fail and the cancel test passes, as the body says. The block fetch now records every outcome and waits out an existing hold-off, as the head fetch did. fetchAllPreviousBlocks records nothing itself, so a 429 is still one strike. With #537 merged, the init wait reads the hold-off through URLReadyAt, and the init tests and #537's hold-off tests pass together. Nothing in this repo or its docs reads the old ERROR line.
  • Item 11: on all four restore exits, the clients are marked under the lock that closes their channels, before the close. A node-ended subscription and an unsubscribed client stay quiet.
  • The body's two mutants reproduce. With no marking, both restore tests fail. With the ingress closing for slow clients only, the ingress test fails. The node-ended and outage tests stay green.
  • A given-up client's mark, sticky entry and limiter are released when its socket closes: UnsubscribeAll calls forgetClientLocked even when nothing is left to unsubscribe.
  • With #677 on the same tree, a given-up client resubscribes on a new connection. The subscribe fails over past the endpoint that refused it, and the new connection has no end mark (probe, 3 of 3 under -race).

Tested: On c19b414: build and vet on main+664, main+fix, main+664+677 and main+fix+677; protocol/chaintracker, protocol/chainlib and protocol/endpointstate; full protocol/rpcsmartrouter on main+664+677 (304 s); the 261 subscription tests under -race on main+664+677 and on main+fix+677; -race ×3 on the new and changed tests with the fix. Mutants: main's tracker, no marking, slow-client-only ingress, each exit reverted, and the fix reverted. On ada2a55 with the fix: build and vet; this PR's tests and the log-level probe; full protocol/rpcsmartrouter and the 262 subscription tests under -race, both with #677; chainlib, chaintracker, endpointstate and holdoff. golangci-lint --new-from-rev origin/main on chaintracker, chainlib and rpcsmartrouter: 0 issues on both bases. TestChainTrackerMaintainMemory/one_long_test failed once in 12 runs under heavy load. It is a timing test in the poll path and passes when run alone.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants