Skip to content

fix(endpointstate): report every answered poll on the url's series, not only one the tip store kept (MAG-4347) - #592

Merged
AnnaR-prog merged 2 commits into
mainfrom
fix/MAG-4347-answered-until-refresh
Nov 3, 2026
Merged

AnnaR-prog merged 2 commits into
mainfrom
fix/MAG-4347-answered-until-refresh

Conversation

@avitenzer

Copy link
Copy Markdown
Contributor

Description

A url that answers every poll could read as down on its own per-url series, because the write behind that series only happened when the router's tip store kept the poll's block.

Jira ticket: MAG-4347

This must land before smart-router-infrastructure PR 840 applies the per-url stuck alerts, or Solana pods will false-alert on urls that are answering. It is independent of the three WebSocket PRs (#589, #590, #591) and can merge in any order relative to them.

Summary

EndpointMonitor.recordPollObservation called onPollBlock only when the per-url tip store accepted the poll's block. That hook is the write behind rpc_endpoint_url_latest_block and the answered-until time the per-url stuck alerts read. A block below a still-fresh stored tip is a straggler the store drops, so an answered poll wrote nothing for the url.

The fleet gate makes that a steady state rather than an occasional event: each peer skip adopts the peer's block again, which keeps the stored tip fresh, so every forced poll that answers lower than the peer is a straggler. Two shapes reach it without anything being wrong:

  • Solana, by design. The tracker polls getLatestBlockhash at finalized commitment; relays harvest the slot at the client's commitment, which is newer, and relay observations are published to the fleet store. A pod that serves no relays for a url adopts a peer's newer slot on each skip, and its own finalized polls are stragglers inside the stored tip's freshness horizon.
  • A url two providers share, on a single pod: the non-relayed provider's polls sit below the tip the relayed one keeps fresh.

The hook now fires on every answered poll, with the block the url answered rather than the stored tip. Keeping the answered block is what preserves the alert's signal: a url stuck on an old block still reads as stuck, because its series stands still while its answered-until time keeps moving. A failed poll still reports nothing, so a url that stops answering stops both and reads as down within one window of its last answer.

The ticket suggested recording the answered-until time separately from the block value. One write does both here: SetEndpointURLLatestBlock sets the value and the answered-until stamp together, so reporting the answered block is the smaller change and keeps one source for the series.

Test plan

  • TestAnsweredPollBelowAFreshPeerTipStillReportsTheAnswer — from the repro Anna Rayer attached to the ticket. A peer reports one block above what this pod's polls answer; four polls are answered and all four must be reported, each with the answered block and not the peer's.
  • TestStuckUrlKeepsAnsweringWithItsOldBlock — a url frozen far below the fleet reports its own frozen block every poll.
  • TestAnsweredUntilStopsWhenTheUrlStopsAnswering — the other half of the acceptance: the fix must not make a dead url look alive.
  • TestOnPollBlock_ReportsEveryAnsweredPoll pinned the defect. Its straggler row asserted that a block below a fresh tip is not reported, which is the behaviour this ticket calls wrong; it now asserts the answer. Worth a look, since it is the one place a reviewer can see the contract change.
  • The first two and the updated table test fail on main, verified with go test -overlay against origin/main's copy of observation.go. On main the repro reports 1 write for 4 answered polls.
  • go test -count=1 on protocol/endpointstate (24s) and protocol/rpcsmartrouter (243s) on the committed tree: ok. protocol/endpointstate also passes under -race on the committed tree.
  • go build -buildvcs=false ./... and go vet on the committed tree.

CI runs no unit tests on this repo, so the local runs above are the only test evidence.

🤖 Generated with Claude Code

…ot only one the tip store kept (MAG-4347)

The poll-block hook fired only when the per-url tip store accepted the
poll's block, and that hook is the write behind rpc_endpoint_url_latest_block
and the answered-until time the per-url stuck alerts read. A block below a
still-fresh stored tip is a straggler the store drops, so a url that
answered every poll wrote nothing for itself and went silent on its own
series.

The fleet gate makes that a steady state rather than an occasional event:
each peer skip adopts the peer's block again, which keeps the stored tip
fresh, so every forced poll that answers lower than the peer is a
straggler. Solana reaches it by design, because the tracker polls
getLatestBlockhash at finalized commitment while relays harvest the slot
at the client's, which is newer. So does the non-relayed provider of a
url two providers share, on a single pod.

The hook now fires on every ANSWERED poll, with the block the url
answered rather than the stored tip. A url stuck on an old block
therefore still reads as stuck: its series stands still while its
answered-until time keeps moving, which is the shape the alerts look for.
A failed poll still reports nothing, so a url that stops answering stops
both and reads as down within one window of its last answer.

This must land before smart-router-infrastructure PR 840 applies the
per-url stuck alerts, or Solana pods false-alert on urls that are
answering.

Tests: TestAnsweredPollBelowAFreshPeerTipStillReportsTheAnswer, from the
repro Anna Rayer attached to the ticket, and
TestStuckUrlKeepsAnsweringWithItsOldBlock both fail on main.
TestAnsweredUntilStopsWhenTheUrlStopsAnswering is the other half of the
acceptance: the fix must not make a dead url look alive.
TestOnPollBlock_ReportsEveryAnsweredPoll pinned the defect — its
straggler row asserted that a block below a fresh tip is not reported —
and now asserts the answer.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@avitenzer
avitenzer requested a review from a team as a code owner October 28, 2026 15:37

@AnnaR-prog AnnaR-prog left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at 13c2437. I tested it merged with current main 6d585d4.

The change is correct.

  • answeredBlock is set only for an answered poll with a block, after the generation and attempt-order guards, so a stale attempt, a replaced tracker and a stopped monitor still report nothing, and a failed poll still stops both series.
  • I read it against the per-url stuck rules on smart-router-infrastructure PR 840. A url frozen at an old block now writes that block on every answered poll, so its series stands still while its answered-until time moves, and sr-provider-stuck fires for it. On main, the same url's polls were stragglers below the fresh tip, so it read as down instead of stuck. The rules only use changes() and offset, so the change cannot make them misfire.
  • Solana's GET_BLOCKNUM in lava-specs polls getLatestBlockhash at finalized commitment, so the straggler case is the normal case there, as the description says. One side effect: a url whose relays harvest a newer slot than its polls now has a non-monotonic per-url series (it dips to the finalized slot after each poll). No rule or dashboard depends on it being monotonic.
  • With main's observation.go overlaid, the repro reports 1 of 4 answered polls and three tests fail. They pass here, also with -race, and with #600 merged on top.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

@AnnaR-prog AnnaR-prog left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving at 42334ef: main b875f86 plus this PR. go build ./... and the full protocol/endpointstate suite pass on this exact tree.

@AnnaR-prog
AnnaR-prog merged commit 1e9c85a into main Nov 3, 2026
@AnnaR-prog
AnnaR-prog deleted the fix/MAG-4347-answered-until-refresh branch November 3, 2026 15:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants