Skip to content

feat: initial smart router implementation - #1

Merged
nimrod-teich merged 7 commits into
mainfrom
feat/initial-smart-router
Apr 23, 2026
Merged

nimrod-teich merged 7 commits into
mainfrom
feat/initial-smart-router

Conversation

@avitenzer

Copy link
Copy Markdown
Contributor

Multi-protocol RPC gateway that routes JSON-RPC, REST, gRPC, and Tendermint RPC requests to statically configured upstream endpoints with QoS-based provider selection, caching, and automatic failover.

Components:

  • cmd/smartrouter, cmd/lavap: Cobra entrypoints for rpcsmartrouter and cache subcommands (lavap kept as a compat alias).
  • protocol/chainlib: per-interface parsers and proxies (JSON-RPC, REST, Tendermint RPC, gRPC) with chainproxy clients and rpcInterfaceMessages encoders.
  • protocol/rpcsmartrouter: router server, direct-RPC relay path, retry state machine, endpoint chain fetcher/tracker, WS/gRPC subscription managers.
  • protocol/relaycore, provideroptimizer, lavasession, chaintracker, qos: relay orchestration, QoS scoring and weighted provider selection, session/connection pooling, per-chain latest-block tracking.
  • protocol/metrics: Prometheus metrics and optional Kafka analytics.
  • ecosystem/cache: standalone cache server process the router connects to via --cache-be.
  • specs/: offline JSON specs for ethereum, lava, cosmossdk, tendermint, ibc, grpctest — loaded via --use-static-spec.
  • utils/, types/: shared logger (lavalog), scoring primitives, spec and relay types, spec fetcher.
  • scripts/pre_setups/: end-to-end bootstrap scripts for LAVA and ETH1.

Provenance: hard fork of github.com/lavanet/lava/v5, rebranded to github.com/magma-Devs/smart-router. Scope is trimmed to the router and its transitive dependencies — no chain/node/keeper code — and no upstream merges are expected. "lavanet/lava" strings remaining in specs/lava.json and protocol/chainlib/rest_test.go are Cosmos REST paths served by the Lava node, not GitHub refs, and must not be rewritten.

Took 2 minutes

Multi-protocol RPC gateway that routes JSON-RPC, REST, gRPC, and Tendermint RPC requests to statically configured upstream endpoints with QoS-based provider selection, caching, and automatic failover.

Components:
- cmd/smartrouter, cmd/lavap: Cobra entrypoints for rpcsmartrouter and cache subcommands (lavap kept as a compat alias).
- protocol/chainlib: per-interface parsers and proxies (JSON-RPC, REST, Tendermint RPC, gRPC) with chainproxy clients and rpcInterfaceMessages encoders.
- protocol/rpcsmartrouter: router server, direct-RPC relay path, retry state machine, endpoint chain fetcher/tracker, WS/gRPC subscription managers.
- protocol/relaycore, provideroptimizer, lavasession, chaintracker, qos: relay orchestration, QoS scoring and weighted provider selection, session/connection pooling, per-chain latest-block tracking.
- protocol/metrics: Prometheus metrics and optional Kafka analytics.
- ecosystem/cache: standalone cache server process the router connects to via --cache-be.
- specs/: offline JSON specs for ethereum, lava, cosmossdk, tendermint, ibc, grpctest — loaded via --use-static-spec.
- utils/, types/: shared logger (lavalog), scoring primitives, spec and relay types, spec fetcher.
- scripts/pre_setups/: end-to-end bootstrap scripts for LAVA and ETH1.

Provenance: hard fork of github.com/lavanet/lava/v5, rebranded to github.com/magma-Devs/smart-router. Scope is trimmed to the router and its transitive dependencies — no chain/node/keeper code — and no upstream merges are expected. "lavanet/lava" strings remaining in specs/lava.json and protocol/chainlib/rest_test.go are Cosmos REST paths served by the Lava node, not GitHub refs, and must not be rewritten.

Took 2 minutes
@github-actions

github-actions Bot commented Apr 20, 2026 •

Copy link
Copy Markdown
Contributor

Test Results

0 tests   0 ✅  0s ⏱️
0 suites  0 💤
1 files    0 ❌

Results for commit ecbe90a.

♻️ This comment has been updated with latest results.

The initial import carried over CI workflows verbatim from
lavanet/lava/v5. Several of them reference directories and scripts
that were stripped out of this fork (x/, testutil/e2e/,
testutil/common/, scripts/protocgen.sh, cmd/lavad, cmd/lavavisor),
causing every PR to fail 6+ checks.

Removed (upstream-only, non-operational in this fork):
- cli_tests.yml — references /home/runner/work/lava/lava paths; the
  only real test step was commented out.
- proto.yml — runs scripts/protocgen.sh, which does not exist.
- network-upgrades-tag-populate.yml — writes to the external
  lavanet/devops repo on every version tag.
- lava_helm_charts_tags_populate.yml — targets
  magma-Devs/smart-router-helm-charts, flagged in-file as unverified.

Rewrote lava.yml (now "Build and Test"):
- Single go test ./... run instead of consensus/protocol/e2e splits
  that referenced non-existent packages.
- Build matrix covers smartrouter + lavap across linux/darwin and
  amd64/arm64; removed lavad and lavavisor targets.
- Docker build uses the top-level Dockerfile (which actually exists)
  instead of cmd/*/Dockerfile (which don't), and doesn't push on PRs.
- Keeps junit reporting, Codecov upload, and EnricoMi's publish
  action for test result comments.

Follow-up: release.yml still references cmd/lavad/Dockerfile and a
cosmovisor image; it won't fail this PR (tag-triggered only) but
should be addressed before cutting the first release.

Took 35 minutes
- Rename .github/workflows/lava.yml -> smartrouter.yml so the filename
  reflects the fork's actual product.
- Drop lavap from the build matrix. CI now produces only smartrouter
  binaries across linux/darwin x amd64/arm64 (4 artifacts instead of 8).

Workflow display name remains "Build and Test".

Took 6 minutes
Replaces remaining "Lava" branding that referred to the upstream
codebase's identity with "Smart Router" — this fork is the centralised
routing solution, distinct from the Lava decentralised network which
remains a legitimate routing target.

Kept intentionally (Lava-as-upstream-chain):
- specs/lava.json and Lava REST paths in tests/specs
- config/smartrouter_examples/smartrouter_lava.yml (target-chain name)
- "Lava blockchain via PublicNode" wording in docs
- cmd/lavap/ alias binary for legacy ops tooling
- LICENSE.md "Copyright 2022 Lava Network" (required by Apache 2.0)

Changes:
- .goreleaser.yaml: project_name lava -> smart-router; drop lavavisor
  and lavad builds/archives; add smartrouter build and archive; keep
  lavap build for legacy distribution.
- .github/workflows/release.yml: retitle to "Publish Smart Router
  Release"; replace lavap/lavad docker matrix with a single
  smart-router image built from the root Dockerfile.
- .github/pr_labeler.yml: drop dead path labels (cmd/lavad,
  cmd/lavavisor, x/**, cookbook/**, proto/**, docs/**, Team:Consensus);
  add labels for cmd/smartrouter, ecosystem, utils, types, scripts.
- .github/ISSUE_TEMPLATE/bug_report.yml: rename "Lava Version" field to
  "Smart Router Version"; update placeholder.
- README.md: retitle to "Smart Router"; reframe intro as a centralised
  gateway; update renamed script paths.
- protocol/rpcsmartrouter/README.md: "Lava's RPC gateway" ->
  "centralised RPC gateway".
- CONTRIBUTING.md, CODING_GUIDELINES.md, CODE_OF_CONDUCT.md: retitle
  and reword product identity references.
- scripts/pre_setups/init_lava_smartrouter_{eth,lava}.sh renamed to
  init_smartrouter_{eth,lava}.sh; "killall lavap lavad" replaced with
  "killall smartrouter lavap"; self-referencing comment updated.

Out of scope (Phase 2, separate PR):
- Go package renames (lavasession, lavaprotocol, lavalog, lavaslices).
  Three collide with Go stdlib if renamed to the obvious targets; needs
  a design decision.
- scripts/useful_commands.sh lavad helper functions (unused by current
  scripts; would only matter if a developer calls them directly).
- /lava/health HTTP endpoint path (requires code changes).

Took 13 minutes
Docker build was failing because Dockerfile pinned Go 1.23 while go.mod
declares 1.26.2; the builder image could not compile Go 1.26 syntax.
Bumped ARG GO_VERSION to 1.26.

Lint was failing because golangci-lint v1.62.0 (built with Go 1.23)
refuses to lint a newer toolchain than itself. Migrated to:
- golangci-lint-action v6 with version: latest (picks up v2.x).
- .golangci.yml v2 schema: version: "2" header, linters moved under
  linters.default: none + enable list, settings nested in
  linters.settings, exclusions under linters.exclusions, and formatters
  (gofmt/gofumpt/goimports) hoisted into a dedicated formatters block
  per the v2 convention.

Lint is temporarily advisory (continue-on-error on every retry) while
the v2 migration settles. Once a pinned v2.x version is known to
produce a clean run against this codebase, drop the continue-on-error
flags to re-enforce the gate.

Took 4 minutes
golang:1.26-alpine3.21 does not exist on Docker Hub — Alpine 3.21 only
has golang:1.23 images. Go 1.26 was only published against alpine3.22
and alpine3.23. Confirmed via docker manifest inspect:
- golang:1.26-alpine3.21 -> no such manifest
- golang:1.26-alpine3.22 -> exists

Took 12 minutes
Took 12 minutes
@nimrod-teich
nimrod-teich merged commit 10a4506 into main Apr 23, 2026
9 checks passed
avitenzer added a commit that referenced this pull request Apr 30, 2026
…Sprint 3)

Routes runtime config through the SmartRouterConfig dispatcher introduced in
Sprint 2 — community now refuses non-EVM specs / non-jsonrpc interfaces / WS
and gRPC transports at startup with the pinned error messages, and enterprise
unlocks them after a valid embedded license is verified.

Engine gates (§3.3.6 audit rows in protocol/rpcsmartrouter/rpcsmartrouter.go):

- Rows #1, #2 — ValidateAPIInterface + ValidateSpec before NewChainParser
- Row  #3   — ValidateTransport on WS endpoints; HasPrefix stays as the
              discriminator, ValidateTransport is the gate
- Row  #3a  — bare-host gRPC detection in convertProvidersToSessions;
              closure signature widened to return error so the gate can
              propagate via the existing errCh path
- Rows #4, #7 — NewDirect*SubscriptionManager → ActiveConfig().Create*();
              Start(ctx) lifecycle stays outside the factory via an inline
              interface assert (only Direct impls need starting; noops don't)
- Row  #5   — unchanged (existing NewNoOpWSSubscriptionManager fallback
              matches what the community factory would return)
- Row  #6   — defensive ValidateAPIInterface(grpc) precondition; documented
              as defense-in-depth against future row-#1 refactors

Startup (cmd/smartrouter/):

- main.go calls installLicenseCheck(rootCmd) — license validation fires only
  on the router command's PreRunE, not on version / cache / test, matching
  CLAUDE.md's "cache is community-available" rule.
- startup_community.go (//go:build !enterprise) — INFO log line only.
- startup_enterprise.go (//go:build enterprise) — validateAndActivateLicense
  reads the embedded license, dispatches on LicenseStatus, calls
  ActivateConfig, and starts expiryWatcher(ctx, license). Watcher cadence:
  hourly during grace, hourly in last 7 days, daily in last 30 days, daily
  poll outside the warning window so long-running processes detect entry.
- validateAndActivateLicense takes the license key as a parameter so tests
  can mint per-state licenses with ephemeral keys (license_test.go pattern).

CI guard:

- scripts/check_gated_symbols.sh + make check-gates fail if any new caller
  reaches the gated constructors (NewDirect*SubscriptionManager,
  lavasession.NewDirectRPCConnection, chainlib.NewChainParser,
  RegisterEnterpriseConfig, the ws://-prefix discriminator) outside an
  explicit per-symbol allowlist. Test files are exempt.

Verified: go build ./... and go build -tags enterprise ./... both clean;
go vet both modes clean; full test suites pass both modes (rpcsmartrouter,
licensing, cmd/smartrouter); make check-gates reports 0 violations across
6 symbols; go tool nm confirms community binary contains 0 enterpriseConfig
/ EmbeddedLicense / expiryWatcher symbols (enterprise binary has all 9
enterpriseConfig methods plus the license-flow functions).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Took 1 hour 1 minute
nimrod-teich pushed a commit that referenced this pull request Jun 18, 2026
…Sprint 3)

Routes runtime config through the SmartRouterConfig dispatcher introduced in
Sprint 2 — community now refuses non-EVM specs / non-jsonrpc interfaces / WS
and gRPC transports at startup with the pinned error messages, and enterprise
unlocks them after a valid embedded license is verified.

Engine gates (§3.3.6 audit rows in protocol/rpcsmartrouter/rpcsmartrouter.go):

- Rows #1, #2 — ValidateAPIInterface + ValidateSpec before NewChainParser
- Row  #3   — ValidateTransport on WS endpoints; HasPrefix stays as the
              discriminator, ValidateTransport is the gate
- Row  #3a  — bare-host gRPC detection in convertProvidersToSessions;
              closure signature widened to return error so the gate can
              propagate via the existing errCh path
- Rows #4, #7 — NewDirect*SubscriptionManager → ActiveConfig().Create*();
              Start(ctx) lifecycle stays outside the factory via an inline
              interface assert (only Direct impls need starting; noops don't)
- Row  #5   — unchanged (existing NewNoOpWSSubscriptionManager fallback
              matches what the community factory would return)
- Row  #6   — defensive ValidateAPIInterface(grpc) precondition; documented
              as defense-in-depth against future row-#1 refactors

Startup (cmd/smartrouter/):

- main.go calls installLicenseCheck(rootCmd) — license validation fires only
  on the router command's PreRunE, not on version / cache / test, matching
  CLAUDE.md's "cache is community-available" rule.
- startup_community.go (//go:build !enterprise) — INFO log line only.
- startup_enterprise.go (//go:build enterprise) — validateAndActivateLicense
  reads the embedded license, dispatches on LicenseStatus, calls
  ActivateConfig, and starts expiryWatcher(ctx, license). Watcher cadence:
  hourly during grace, hourly in last 7 days, daily in last 30 days, daily
  poll outside the warning window so long-running processes detect entry.
- validateAndActivateLicense takes the license key as a parameter so tests
  can mint per-state licenses with ephemeral keys (license_test.go pattern).

CI guard:

- scripts/check_gated_symbols.sh + make check-gates fail if any new caller
  reaches the gated constructors (NewDirect*SubscriptionManager,
  lavasession.NewDirectRPCConnection, chainlib.NewChainParser,
  RegisterEnterpriseConfig, the ws://-prefix discriminator) outside an
  explicit per-symbol allowlist. Test files are exempt.

Verified: go build ./... and go build -tags enterprise ./... both clean;
go vet both modes clean; full test suites pass both modes (rpcsmartrouter,
licensing, cmd/smartrouter); make check-gates reports 0 violations across
6 symbols; go tool nm confirms community binary contains 0 enterpriseConfig
/ EmbeddedLicense / expiryWatcher symbols (enterprise binary has all 9
enterpriseConfig methods plus the license-flow functions).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Took 1 hour 1 minute
avitenzer added a commit that referenced this pull request Jun 23, 2026
…c gen, SyncBlock floor

P1-#1 dyncodec: keep reflection callback/parser paths slash-less; the leading
slash is added only at the grpc.ClientConn.Invoke boundary. Regression test
asserts both the slash-less callback and the slashed invoke.

P1-#2 tip eligibility: harvest a relay's block as the current tip only when the
method's spec function tag is GET_BLOCKNUM, or GET_BLOCK_BY_NUM requesting LATEST.
RequestedBlock==LATEST and GetUsedDefaultValue (non-deterministic for
eth_blockNumber) cannot discriminate receipt/by-hash historical responses. All
downstream tip writes stay gated behind the single harvest chokepoint.

P2-#3 chainstate baseline: split the overloaded baselineAt into baselineAt (TTL
freshness, refreshed every confirming Recompute) and baselineSince (establishment
time, preserved while the block is unchanged, reset on forward/downward/regained
change). GetConsensusBaselineWithTime returns baselineSince for sync-lag so a
stuck baseline accrues real age while a reconfirmed one never expires.

P2-#4 tracker generation: register the tracker and allocate its observation
generation synchronously before the relay dispatch captures it. GetOrCreateTracker
does no network I/O (the poll loop stays async), so an early relay no longer
captures generation 0 and silently drops its harvested tip.

P2-#5 optimizer SyncBlock: serialize each provider's QoS read-modify-write with a
striped lock and hold the authoritative monotonic SyncBlock floor under it. The
ristretto cache is async/eventually-consistent and cannot enforce monotonicity, so
a stale read (incl. the legacy availability-only writer) could regress the block;
the floor guarantees SyncBlock never decreases. ResetState clears the floors.

Verification: gofmt, go build ./..., go vet ./..., make test (all green), and
targeted -race for dyncodec, chainstate, endpointstate, provideroptimizer,
rpcsmartrouter.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
avitenzer added a commit that referenced this pull request Jun 30, 2026
Fixes from the PR #143 review (docs/PR-143-REVIEW.md). The top findings share
one root: getLatestBlock() now intentionally returns 0 on a TTL-stale tip, and
several consumers mishandle the 0.

- #1 archive routing: add getLatestBlockBestEffort() (strict tip, else last-known
  atomic, 0 only at cold-start) for the three getExtensionsFromDirectiveHeaders
  returns, so a stale window no longer forces every concrete-block request to the
  archive pool. Cache finalization stays on the strict getLatestBlock().
- #2 jsonRPC eth_call: guard the archive-depth subtraction (parsedBlock>=0 &&
  LatestBlock>ethCallArchiveBlockDepth) so uint64(0)-126 can't underflow and force
  archive for every eth_call (also covers chains younger than the depth).
- #3 averageBlockTime: floor the spec block time once (DefaultAverageBlockTime) and
  feed the same value to both ChainState and the EndpointMonitor, so the consensus
  window can't end up shorter than the poll cadence on no-average_block_time specs.
- #5 endpoint flap: dampen the poll-healthy/relay-failing flap with a capped,
  decaying re-enable hysteresis (reEnableAfterK << reenableProbeFlaps, cap 2; a
  successful relay decays it). Race-safe under e.mu; bounded well below the epoch.
- #7 hot path: cache the two tip-defining ApiNames once (sync.Once) so
  tipBlockFromRelay no longer takes the parser RWLock twice per relay.
- #8 cleanup: make endpointstate.ObservationSource a type alias of
  endpointtip.Source (drop the duplicate enum/String/converter).

#4 (cache finalization) and #6 (sub-majority sync) resolved as by-design with
hardened comments/rationale; two separate latent items recorded as follow-ups.

go build/vet clean; lavasession, endpointstate, chainstate, rpcsmartrouter,
chainlib suites green (incl. -race where applicable).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
avitenzer added a commit that referenced this pull request Jul 8, 2026
…ations → ChainState consensus → proactive prober (#143)

* refactor(endpointstate): extract per-endpoint ChainTracker into its own package

Prep for the ChainTracker & Probing Redesign (MAG-2171). Splits the
per-endpoint block-telemetry "data plane" out of rpcsmartrouter into a
dedicated package so the upcoming probing "decision plane" has a clean,
one-way boundary to build against (probing may read endpointstate; never
the reverse).

- Move endpoint_chain_tracker_manager.go -> endpointstate/endpoint_monitor.go
  (EndpointChainTrackerManager -> EndpointMonitor)
- Move endpoint_chain_fetcher.go -> endpointstate/endpoint_poller.go
  (EndpointChainFetcher -> EndpointPoller)
- Keep EndpointChainTrackerConfig / EndpointChainTrackerState names: they
  describe the underlying chaintracker concept, which is still accurate.
- Add protocol/probing/doc.go: empty decision-plane shell staking out the
  two-plane boundary for a later extraction ticket.
- Promote the in-package joinURLPath helper (surfaced as a cross-package
  dependency by the move) to common.JoinURLPath; relocate its test.
- Relocate the cancelFuncs lifecycle + ResetAllLatestBlocks count-correctness
  tests into endpointstate (they poke unexported maps, now in-package).
- Rewrite the rpcsmartrouter debug router-walk test to exported behavior
  only (real empty monitors + nil-safety); per-tracker count correctness is
  covered in endpointstate.

Verified: go build ./..., go vet ./..., endpointstate (+ -race), full
rpcsmartrouter suite, and common JoinURLPath all pass. One-way import edge
confirmed (rpcsmartrouter -> endpointstate); no cycle.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(endpointstate): add per-endpoint observation contract (MAG-2158, Topic A)

Topic A of the ChainTracker & Probing redesign: make the per-endpoint monitor
the reliable, side-effect-free observation source that Topics B/C/D read.

- EndpointObservation (observation.go): the locked contract — LatestBlock,
  ObservedAt, Source (Poll|Relay), LastPollAttempt, LastSuccessfulPoll,
  LastPollLatency, LastPollError, ConsecutivePollFailures. The block triple is
  written by whichever source produced the most recent block (ObservedAt
  monotonic); poll-health fields are written only by the poll path.
- EndpointMonitor: RecordPollObservation / RecordRelayObservation /
  GetObservation (consistent-snapshot read). Backed by a dedicated obsMu so the
  hot observation path doesn't contend with tracker-lifecycle (mu); documented
  lock ordering. Records dropped on RemoveTracker/Stop to stay bounded.
- EndpointPoller: an onPollObservation callback fires on every
  FetchLatestBlockNum round-trip (success or failure), with transport-only
  latency measured around sendRawRequest. Wired by the manager in
  GetOrCreateTracker.

Additive and behavior-neutral: recording never touches QoS or endpoint.Enabled;
Endpoint.LatestBlock/LastBlockUpdate are unchanged. RecordRelayObservation's
call site is wired by Topic B (MAG-2159).

Verified: go build ./..., go vet ./..., endpointstate (+ -race, incl. a
concurrent poll/relay/read snapshot test), and the full rpcsmartrouter suite
all pass.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(endpointstate): record poll observations on the SVM path; harden lifecycle

The per-endpoint observation contract (MAG-2158) attached its poll
instrumentation only to EndpointPoller.FetchLatestBlockNum, which
Solana-family ChainTrackers never call -- SVMChainTracker fetches the
latest slot via CustomMessage. So SOLANA/SOLANAT/KOII endpoints recorded
zero poll observations (the main fast-chain/quota case). The direct
Record* unit tests passed because nothing drove a real poll.

- Add chaintracker.PollObserver (ObserveLatestBlockPoll). EndpointPoller
  implements it; SVMChainTracker.FetchLatestBlockNum calls it exactly once
  per poll with transport-only latency measured around CustomMessage. The
  default path keeps recording via FetchLatestBlockNum's defer; the two
  are mutually exclusive by chain family, so no double-recording.
- RecordPollObservation -> internal recordPollObservation(url, gen, ...).
  A per-tracker generation gate rejects late callbacks from removed or
  replaced trackers (no recreate-after-remove, no overwrite of a new
  same-URL tracker); a sticky stopped flag refuses writes after Stop.
- Monotonic poll-health: a stale poll (older than the last attempt) is
  dropped wholesale; equal timestamps apply (last-writer-wins).
- ObservationSourceUnknown = iota, so a zero/failure-only record reports
  Unknown instead of masquerading as poll-sourced.

RecordRelayObservation stays exported; its production call site remains
deferred to MAG-2159.

Tests: real-poll e2e through GetOrCreateTracker for non-SVM and SVM
(success + failure); chaintracker-package tests asserting the SVM wrapper
records exactly one observation per poll on every path; lifecycle/
generation/monotonic/source unit tests.

Verified: gofmt; go test -race ./protocol/endpointstate; go test
./protocol/chaintracker ./protocol/lavasession ./protocol/rpcsmartrouter;
go vet ./protocol/...; go build ./...

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(rpcsmartrouter, chaintracker): add relay-block harvest logic and floored poll cadence (MAG-2159)

Implements relay-block harvesting into per-endpoint observation stores, keyed on NetworkAddress with Source=Relay, and ensures no impact on poll-health fields. Introduces a flat, floored poll cadence (avgBlockTime/2) for per-endpoint trackers, replacing adaptive tiers while keeping the global tracker unchanged. Adds comprehensive tests to validate behavior under both floored and legacy cadences.

* fix(mag-2159): harden relay tip-harvest — eligibility, Solana slot, flat cadence, lifecycle

Post-review hardening of the per-endpoint chain-tracker harvest path
(MAG-2159 review findings 1-6):

1. Tip eligibility: harvest a relay block only when it is the current tip
   — EVM/generic requires RequestedBlock()==LATEST_BLOCK and reply.LatestBlock>0,
   so historical getBlockByNumber(N)/getBlockByHash/receipt/logs no longer
   poison the tip. New tipBlockFromRelay() gate.
2. Solana context.slot: chain-gated extractSolanaContextSlot() harvests
   result.context.slot from successful, non-batch JSON-RPC replies; rejects
   error/null/batch/coincidental-slot/non-positive. No global slot interpretation.
3. Startup scheduling: flat trackers arm the periodic timer only after init
   completes (first poll >= FlatPollInterval after the final attempt); init
   retries are ctx-cancellable spaced so failures cannot burst quota.
4. Flat cadence: FlatPollInterval (renamed from PollIntervalFloor) is the mode
   switch — >0 selects a fixed flat scheduler that ignores adaptive tiers and
   block-gap recalibration; 0 keeps legacy adaptive (global tracker only).
5. Lifecycle safety: RecordRelayObservation takes an endpoint generation and
   records only when the live generation exists and matches, under obsMu —
   safe across RemoveTracker, same-URL replacement, and Stop.
6. Operator config: --chain-tracker-polling-multiplier help notes it no longer
   affects per-endpoint trackers, with a runtime warning when set.

Tests: real-parser EVM eligibility (eth_blockNumber -> LATEST_BLOCK harvested,
concrete block rejected), Solana slot table, generation-safety suite, and
real-StartAndServe startup-cadence timing tests.

Note: full `go test -race ./protocol/chaintracker` still fails on pre-existing
legacy block-gap races (smallestBlockGap/updatePollingTimeBasedOnBlockGap vs
AddBlockGap); the flat path does not touch those and does not widen the race.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(mag-2159): traffic-gate the per-endpoint poll on fresh relay observations (Pass 2 B-D2)

A per-endpoint dedicated poll now borrows a fresh relay-harvested tip instead
of spending an upstream call — the core quota win Topic B exists to deliver.

- EndpointPoller gains a nil-safe relayGate closure. FetchLatestBlockNum
  consults it BEFORE any upstream work: on a hit it returns the borrowed tip
  with no upstream round-trip and no poll observation (no poll happened, so
  poll-health stays owned by the poll path). It still returns the block so the
  ChainTracker advances its atomic latest — which GetSyncGap / consistency read
  (borrow-and-return, not skip-entirely; those readers are fed only by the poll
  loop, so skipping would stale the sync gap).
- EndpointMonitor.freshRelayTip is the predicate: borrowable only when the
  latest observation is Source == Relay (anti-circularity — a poll-sourced obs
  must never suppress the next poll, or one poll throttles all polls) AND
  younger than relayGateFreshness (the gate freshness threshold, default
  averageBlockTime ~ one block stale, 2x the flat poll interval). Wired per
  endpoint in GetOrCreateTracker.

Honest quota claim: eliminates the per-tick FetchLatestBlockNum; on a genuine
tip advance the borrowed block still trips one FetchBlockHashByNum upstream
(fork tracking, chain_tracker.go:435) — correct, not waste.

Deferred (documented in freshRelayTip): suppression is unbounded, safe today
because no decision-plane consumer reads poll-health (the gate is GetObservation's
first production reader). A busy-endpoint poll ceiling (forced real poll every N
intervals to catch a cached/lying upstream) lands with the Topic-D probe.

Tests (protocol/endpointstate/traffic_gate_test.go): anti-circularity (relay
suppresses, poll does not), freshness boundary, borrow-without-upstream-call via
a counting connection (+ tracker atomic advances, no observation recorded), and
nil-gate-always-polls. Full endpointstate suite green under -race.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* refactor(mag-2159): consolidate chain-tracker polling knobs into one floored control (Pass 2 B4)

Collapse the multiplier soup into a single relief control and retire the dead
knobs the per-endpoint flat cadence + Pass-1 hardening left behind.

Removed:
- config.PollingTimeMultiplier (per-tracker override) — set by NO caller in
  prod or test, so it was dead config. Its branch in newCustomChainTracker is
  gone; the multiplier now resolves solely via EffectivePollingMultiplier
  (flag override, else the built-in 16).
- clampPollingMultiplier (the in-tracker divide-by-zero clamp) — provably
  redundant once the dead field is gone: the only surviving feeders of the
  adaptive tiers (base/(multiplier/4)) are the default 16 and the flag, which is
  range-validated to [4,16] before assignment, so every value is >= 4 by
  construction. Its direct unit test goes with it.
- endpointstate.MinPollingInterval — defined, never read.

Kept (re-documented): MinPollingTimeMultiplier survives as the single named
[4,16] lower bound. The flag validation in rpcsmartrouter is now the SOLE
divide-by-zero guard (commented as such) rather than a redundant first layer
ahead of an in-tracker clamp.

Scope: zero runtime cadence change. The global tracker still uses the
multiplier-adaptive path; flattening it to a fixed interval is deferred to
Topic C (its readers — fallback latest-block, syncGap, consistency — are not
harvest-fed yet, so changing its cadence now would risk starving them). Runtime
log + flag help refreshed to "affects only the global tracker" (the interim
"until knob consolidation" wording is now stale).

Verify: gofmt; go vet; go build ./...; go test ./protocol/chaintracker
./protocol/endpointstate ./protocol/rpcsmartrouter — all green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(mag-2159): relocate traffic gate to ChainTracker level — gate Solana + the fork-hash fetch (review F1/F2)

The Pass-2 gate sat on EndpointPoller.FetchLatestBlockNum, which two things
bypass: SVMChainTracker (it polls via CustomMessage, never through
EndpointPoller — so Solana, the primary quota-sensitive chain, was never gated),
and forkChanged's per-tick FetchBlockHashByNum (which runs even when the block is
unchanged, so the old gate still paid one hash fetch per tick).

Move the gate up to ChainTracker.fetchAllPreviousBlocksIfNecessary, ABOVE the
generic/SVM iChainFetcherWrapper split, so it suppresses both families and the
whole cycle:

- ChainTrackerConfig.RelayTipFresh (set by EndpointMonitor from freshRelayTip)
  reports whether a fresh Relay-sourced observation covers the endpoint.
- When it does, the cycle is SKIPPED entirely — no FetchLatestBlockNum and no
  fork-check hash fetch: genuinely zero upstream (F2). Skip-only, not
  borrow-and-return: a skip touches nothing, so SVM seenBlock/slot/hash caches
  stay correct when a relay supplies a slot but no hash (F1).
- Bounded by MaxRelaySkipsBeforePoll (default 4): after N consecutive skips one
  real poll is forced for independent fork/liveness verification (the
  busy-endpoint ceiling). Tracker-atomic staleness is capped at ~2 blocks, well
  inside the 10-block EndpointLagThreshold consistency pre-validation uses.
- The gate fires on relay-observation freshness alone (Source==Relay
  anti-circularity); the global tracker leaves RelayTipFresh nil and never skips.

The old EndpointPoller.relayGate field + its borrow-and-return logic are removed.

Tests: white-box full-cycle upstream-call counts for fresh-unchanged /
fresh-advanced / stale / no-relay across BOTH the EVM and SVM wrappers
(chain_tracker_gate_test.go, internal package, deterministic — no timer); an
end-to-end SOLANA monitor test counting real getLatestBlockhash upstream calls;
bounded-verification and nil-gate (global tracker) guards.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(mag-2159): gate endpoint tip-state on eligibility + capture generation before dispatch (review F3/F4)

F4 — historical responses poisoned the endpoint tip. The relay success path
unconditionally wrote every positive Reply.LatestBlock to targetEndpoint.LatestBlock,
LastBlockUpdate, the global estimator, and the latest-block metric — including
historical eth_getBlockByNumber(N) responses, whose block is NOT the current tip.
Extract harvestAndUpdateTipFromRelay, which gates ALL of those tip-state writes on
tipBlockFromRelay eligibility (the same rule the observation store uses). The block
this response serviced (latestServicedBlock, for OnSessionDone) is kept separate and
ungated. A historical response now moves none of the tip state; the relay path
carries the tip for busy endpoints (tip-eligible only), backing the skip-only gate.

F3 — generation captured after the relay. endpointObservationGeneration was read at
harvest time, so an old in-flight relay could complete after a same-URL tracker
replacement, read the replacement's live generation, and be accepted as the new
tracker's. Ensure the tracker and capture the generation BEFORE relayInnerDirect,
then thread the captured value to the harvest — so a relay that began against one
incarnation carries that incarnation's generation and is rejected after replacement.

Tests: real-parser integration through harvestAndUpdateTipFromRelay — a historical
eth_getBlockByNumber(N) moves no tip state / no Relay observation, while eth_blockNumber
moves all of it; a real-incarnation remove/recreate proves the captured-generation
harvest is rejected after same-URL replacement and the live generation is accepted.
(The harvest/eligibility tests switch to the lightweight real parser to avoid the
CreateChainLibMocks SetGlobalLoggingLevel data race under -race.)

Note: no full live-goroutine concurrency test for F3 — there is no successful-relay
harness for sendRelayToDirectEndpoints (only a failure-path driver); the
capture-before-dispatch ordering is a straight-line property verified by reading the
call site plus the real-incarnation gate test.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(mag-2160): add per-chain ChainState tip with internal consensus (Topic C, phase 1)

First phase of Topic C: the per-chain "current chain head" that will replace the
three unreconciled latest-block sources (global ChainTracker, relay estimator,
monotonic atomic). This commit adds the type + consensus in isolation — no wiring,
no read-site migration, no deletions yet (those are the high-blast-radius later
phases).

protocol/chainstate.ChainState (built in-tree per T1, referencing lava's
protocol/chainstate, not importing it):
- SetLatestBlock: cheap monotonic + outlier-guarded write the relay-harvest and
  poll paths will call. Raises the tip monotonically; when a consensus baseline
  exists, rejects a block > baseline+OutlierThreshold as an anti-lie outlier. The
  tip only goes DOWN via Recompute, never a write.
- Recompute: pulls a snapshot of per-endpoint BlockObservations and computes the
  strict-majority consensus baseline ITSELF (T2 inverted — data plane owns
  consensus, no external SetMajorityBaseline), then realigns the tip down when a
  fresh majority sits > OutlierThreshold below it. Consensus rules (locked): >50%,
  min-2 agreeing, dedup by URL, fresh-only, bucketed by BucketWidth. A strict
  majority is unique, so map-iteration order is deterministic.
- GetLatestBlock / GetLatestBlockWithTime: TTL-fresh reads (T4); a stale tip
  reports not-found rather than freezing.
- Decoupled via the BlockObservation DTO (no endpointstate import → no cycle);
  consensus is a pure function; clock injectable for deterministic TTL tests.
- Proposed-default constants (tunable): BucketWidth=2, OutlierThreshold=100,
  StalenessWindow/TTL = DefaultStalenessMultiplier×avgBlockTime (floored), via
  DefaultConfig.

Tests: monotonic + outlier guard (only with a baseline), TTL expiry + paired
read, consensus cardinality (1/2/3-endpoint), freshness + dedup-by-URL, downward
realignment + recovery, single-endpoint guard-off. Green under -race.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(mag-2160): wire observations into ChainState tip + consensus tick (Topic C, phase 2)

Additive wiring — the legacy global tracker / estimator / atomic still stand and
still serve all reads; this phase only feeds the new ChainState so phase 3 can
migrate the reads onto it.

- EndpointMonitor gains an OnTipObservation(block) hook, fired for every ACCEPTED
  positive block from the poll path (recordPollObservation) and the relay-harvest
  path (RecordRelayObservation). It is fired AFTER obsMu is released (registered
  before the unlock defer, so LIFO runs it last) so the ChainState lock never
  nests inside the observation lock. Rejected writes — failed poll, stale
  generation, non-positive or stale block — do not fire it.
- EndpointMonitor.SnapshotObservations() returns a copy of all records under one
  read lock, for the consensus pull.
- rpcsmartrouter constructs a per-chain chainstate.ChainState (DefaultConfig from
  the spec avgBlockTime), wires OnTipObservation -> ChainState.SetLatestBlock (the
  cheap monotonic tip), and runs runChainStateConsensusLoop: a ticker (avgBlockTime,
  floored at 1s) that pulls SnapshotObservations and calls ChainState.Recompute.
  The snapshot is taken under the monitor lock and released before Recompute takes
  the ChainState lock — the two locks never nest, no deadlock.

Tests: OnTipObservation fires only for accepted positive poll/relay blocks (not
failed/stale/rejected); SnapshotObservations returns an independent copy; and an
end-to-end integration (real monitor + real ChainState) proving agreeing relay
observations feed the monotonic tip, a recompute establishes the strict-majority
baseline, and an anti-lie outlier is rejected. Green under -race.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* refactor(mag-2160): migrate the 3 latest-block read sites onto ChainState (Topic C, phase 3)

Point every "what block is the chain at" read at the new per-chain ChainState
tip instead of the global tracker. The global tracker is now constructed but
UNREAD — phase 4 deletes it.

- Site A getLatestBlock(): the router-wide tip (archive routing, cache
  finalization) is ChainState.GetLatestBlock() (TTL-fresh, consensus-guarded),
  replacing the global-tracker → estimator → atomic ladder. The monotonic
  latestBlockHeight atomic stays as the cold-start fallback so getLatestBlock
  never returns 0 in the bootstrap window before the first observation.
- Site B (Reply.LatestBlock fallback): when the response carried no block, stamp
  THIS endpoint's own observed latest (endpointChainTrackerManager.GetLatestBlockNum
  for the endpoint URL) instead of the global tracker's provider[0] value — fixes
  the cross-attribution (S6) where one node's tip was stamped onto another
  endpoint's QoS/seenBlock.
- Site C (syncGap): the chain tip is ChainState.GetLatestBlockWithTime(), so every
  endpoint is judged against the real consensus chain head rather than a single
  node (fixes the syncGap bias S5). The this-endpoint half stays the endpoint's
  own latest.

The estimator now has no readers (Site A was its only one); it and the global
tracker are removed in phase 4. Full rpcsmartrouter suite green (happy-path
parity).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* refactor(mag-2160): delete the global ChainTracker + retire the estimator (Topic C, phase 4)

With all three reads migrated to ChainState (phase 3), the global tracker and the
estimator have no readers — remove them.

Global ChainTracker (S1/S2/S5/S6/S10 — single point of trust bound to
healthyStaticProviders[0], no failover/recovery/anti-lie):
- Delete the construction + fire-and-forget StartAndServe goroutine in
  rpcsmartrouter.go (CreateSmartRouterEndpoint).
- Drop the chainTracker field on RPCSmartRouterServer, the ServeRPCRequests
  ctor param, and the field assignment; remove the now-unused chaintracker import.
- One fewer poller per pod.

Estimator (relaycore/latest_block_estimator.go) — its only reader was Site A,
now ChainState:
- Delete the file + its test, the latestBlockEstimator field, and the Record call
  in updateLatestBlockHeight.

The monotonic latestBlockHeight atomic is KEPT, but only as the cold-start
fallback for getLatestBlock (so it never returns 0 in the bootstrap window before
the first observation feeds ChainState). The chaintracker PACKAGE stays — the
per-endpoint trackers in endpointstate use it as the generic poll engine.

go build ./... green; relaycore + rpcsmartrouter suites green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* refactor(mag-2160): delete the dead gRPC IChainTracker service (Topic C, phase 5)

The chaintracker gRPC service (chain_tracker_grpc.go + chain_tracker_service.go)
exposed IChainTracker over gRPC. Nothing in-tree consumed ChainTrackerService /
RegisterChainTrackerServiceServer / NewChainTrackerService outside the package,
and the only thing that registered it was ChainTracker.serve() — which only ran
when a tracker set a server address. No tracker ever did (the global tracker, now
removed in phase 4, didn't set one either), so the whole path was dead.

- Delete both gRPC files.
- Remove ChainTracker.serve() entirely (it existed solely to host that service)
  and simplify StartAndServe to just start the poll loop. Drop the now-dead
  serverAddress field, the ServerAddress config knob, and the 7 imports that only
  serve() used (net/http, os, os/signal, grpcweb, http2, h2c, grpc).

Zero behavior change — serve() was always a no-op (server address always empty).
The chaintracker PACKAGE stays as the generic per-endpoint poll engine. go build
./... green; chaintracker suite green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(chainstate): resolve MAG-2160 review findings (tip/consensus/glue)

Address the 9-finding review of Topic C (per-chain ChainState tip +
global-tracker removal). The algorithm was sound; the bugs were in the
rpcsmartrouter glue wiring old read sites onto the new APIs.

ChainState internals (F2/F3/F4/F8):
- F2: split GetLatestBlock (optimistic observed tip) from GetConsensusBaseline
  (strict-majority, TTL-gated via baselineAt) so sync scoring no longer
  measures endpoints against a lone optimistic reporter's tip.
- F3: replace fixed block/BucketWidth buckets with distance-aware sliding-window
  clustering so two endpoints within BucketWidth always agree (1001/1002 no
  longer split across a boundary).
- F4: track lastObservedAt and refresh it on equal-block confirmations so a
  live-but-stable tip does not expire by TTL while endpoints keep reporting it.
- F8: floor a zero StalenessWindow at minStalenessWindow in New/NewWithClock so
  expiry can never be disabled.
- Add sticky Initialized() and a NewWithClock clock seam.

rpcsmartrouter glue (F1/F5/F6 + Site C):
- F1: getLatestBlock falls back to the latestBlockHeight atomic only on genuine
  cold start (!Initialized); a TTL-expired tip returns unknown rather than
  reviving a frozen value. Remove the non-tip-eligible cache-hit atomic writer.
- F5: recomputeChainStateConsensus no longer early-returns on an empty snapshot,
  so Recompute(empty) clears a stale baseline.
- F6: Site B reads the freshest per-endpoint observation (GetObservation) instead
  of the MAG-2159-gated tracker atomic (GetLatestBlockNum).
- Site C: syncGap scores against GetConsensusBaseline and stays 0 when no fresh
  majority exists.

Dead polling-multiplier flag (F7):
- Remove --chain-tracker-polling-multiplier, PollingTimeMultiplierOverride,
  EffectivePollingMultiplier, MinPollingTimeMultiplier, and startup logging. The
  adaptive engine branch stays, using the fixed built-in MostFrequentPollingMultiplier.

Tests: focused integration tests for every corrected rpcss seam
(chainstate_glue_test.go) plus ChainState unit tests for clustering boundary,
equal-block freshness, empty-snapshot clear, observed-vs-consensus, and config
defaults. Clean up stale "global estimator" comments.

Verification: gofmt clean; go build ./... green; go test + go vet across
chainstate/endpointstate/chaintracker/rpcsmartrouter pass; go test -race on
chainstate/endpointstate pass.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(qos): Topic E — QoS + endpoint.Enabled contract (precedes MAG-2161/D)

Build the contract both the prober (Topic D) and the relay path honor, ahead of
D so the prober targets a settled API. Unit-testable without a running prober.

QoS sync reference → consensus baseline (MAG-2160 Finding 2):
- ProviderOptimizer gains a syncReference seam (SetSyncReferenceGetter): BOTH
  the relay and probe sync dimensions measure lag against ChainState's
  consensus baseline instead of max-block-across-providers (which one fast/lying
  reporter could inflate, dinging the whole pod). Falls back to the legacy
  reference when there is no fresh majority (cold start / single-endpoint).
- chainstate.GetConsensusBaselineWithTime exposes the baseline + its timestamp
  for the lag calc. Wired live per chain interface in ServeRPCRequests.

Probe QoS feed + fraction-healthy availability:
- New ProviderOptimizer.AppendProbeData(provider, availability, latency,
  syncBlock, healthy) feeds all three dimensions at ProbeUpdateWeight.
- AvailabilityScoreStore relaxed from strict {0,1} to [0,1] so a provider's
  fraction-of-endpoints-healthy sample decays the score (partial degradation).

Provider aggregation (rule E2):
- probing.AggregateProviderSample collapses per-endpoint verdicts into ONE
  sample/provider/cycle: availability = healthy/total, latency = min over
  healthy, block = max over healthy (no endpoint-count EWMA bias).

endpoint.Enabled state machine:
- Endpoint.RecordProbeVerdict(healthy, K) is the probe's proactive re-enable —
  acts ONLY on disabled endpoints (never resets an enabled endpoint's
  relay-driven refusal climb) and re-enables after K consecutive healthy cycles
  (default probing.DefaultProbeReEnableHysteresis = 3). Relay path stays the
  fast disabler (MarkUnhealthy at 50). Probe-driven disable deferred.

Tests cover every acceptance item: 4:1 relay:probe weighting, provider
aggregation, the Enabled state machine (incl. the anti-coupling guard), the
sync seam + fallback, and fraction-healthy decay.

Verification: gofmt clean; go build ./... green; go vet + go test across
chainstate/lavasession/probing/provideroptimizer/rpcsmartrouter/utils-score
pass; go test -race on lavasession/provideroptimizer pass.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(probing): Topic D — proactive health prober (MAG-2161)

Add the per-chain proactive health prober: on its own cadence
(defaultProbeCadence = 5s) it reads stored per-endpoint telemetry
(Topic A) plus the consensus baseline (Topic C) — making ZERO upstream
calls — renders a health verdict for EVERY direct-RPC endpoint (regular
AND backup), proactively re-enables recovered endpoints (the O1 win:
recovery in seconds, not at the 15-min epoch), and feeds ONE aggregated
QoS sample per provider (Topic E rule E2). Always-on; the legacy
synthetic direct-RPC probe feed is gated off for static providers.

Prober (rpcsmartrouter):
- runProbeLoop / runProbeCycle / runProbeCycleCore — the loop, the
  live-deps adapter, and a pure testable core.
- Baseline sourced from chainState.GetConsensusBaseline() (strict-
  majority), NOT the observed max-across-providers tip — closes the
  door on Finding 2 reincarnating in the probe path.
- Compile-time guard: var _ probeQoSAppender =
  (*provideroptimizer.ProviderOptimizer)(nil) so the optional-capability
  type assertion can't silently nil on signature drift.

Verdict engine (probing):
- verdict.go: RenderEndpointVerdict (alive = fresh & non-zero tip;
  keeping-up = within LagToleranceBlocks of baseline) + DefaultVerdictConfig.
- contract.go: per-dimension has-flags (HasLatency/HasBlock) so a
  healthy-but-unpolled endpoint never feeds a fake 0 latency/block;
  AggregateProviderSample = fraction-healthy availability, min latency,
  max block over healthy endpoints.

QoS feed (provideroptimizer):
- AppendProbeData gains per-dimension hasLatency/hasSync flags;
  availability always fed (probe weight 0.25 → 4:1 relay:probe).

Tests: probe_loop_test.go (re-enable, stale-stays-disabled, one-sample-
per-provider, backups-covered, no-observation-unhealthy), verdict_test.go,
contract_test.go. Full suite green; build + gofmt clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(rpcsmartrouter): resolve MAG-2160/2161 review findings + harden concurrency

Post-merge review of the ChainTracker/Probing redesign (Topics C/D/E) plus the
data-race hardening surfaced while satisfying the review's go test -race gate.

Review findings (F1-F7):
- F1: the probe re-enables a disabled endpoint only from DISTINCT successful
  polls produced strictly after its disable instant (edge-triggered disabledAt,
  stamped at BOTH disable write-sites). A pre-disable / relay-fed observation
  can never re-enable.
- F2: a genuine re-enable also restores the provider to routing
  (csm.RestoreRecoveredProvider) — regular -> validAddresses, backup -> out of
  blockedBackupProviders; overlap-safe and idempotent.
- F3: endpoint.Enabled reads go through the synchronized Endpoint.IsEnabled();
  the synthetic direct-RPC probe is removed for static providers.
- F4: the QoS sync reference is a per-sample, per-interface SyncReference
  resolved from THIS interface's ChainState — replacing the single mutable
  optimizer getter that the last interface to start would overwrite.
- F5: with no fresh consensus majority the sync update is OMITTED, never the
  legacy max-block-across-providers reference.
- F6: dedicated, validated --probe-loop-interval (ProbeLoopInterval) clock.
- F7: synthetic direct-RPC liveness feed dropped for static providers.

Concurrency hardening (go test -race green for lavasession + rpcsmartrouter):
- updateMetricsManager snapshots LatestBlock/RelayNum before its metrics
  goroutine, which outlived OnSessionDone's session Free (real production race).
- rpcclient handler.handleResponse sets op.err before op.resp<-msg for
  subscriptions, which was racing Subscribe's wait() (real production race).
- score_store getters take the type's existing RWMutex; rand.protocolRand ->
  atomic.Pointer; DebugProbes -> atomic global no longer bound at flag
  registration; chain_message drops its racy lazy request-hash cache (the relay
  processor runs concurrent attempts on one message); SetGlobalLoggingLevel test
  mock guarded by sync.Once; CSM unlocked log slice-reads and test-side
  shared-slice reads synchronized; mock WS server writes serialized.
- clock_injection decay test: deterministic ristretto build-up (was flaky under
  -race load because the async cache dropped accumulations) and corrected the
  warp_10h threshold to the true 0.122%.

Verified: go build, go vet, gofmt clean; non-race go test green; go test -race
green for lavasession, rpcsmartrouter, provideroptimizer, chainstate,
endpointstate, score, rand.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(chainlib): resolve pre-existing data races + failures in chainproxy/listeners/tracing

Pre-existing failures surfaced by the full go test -race / make test sweep, in
packages unrelated to the redesign. Two were real production bugs newly fatal
under stricter dependency versions.

- dyncodec/remote_relayer + chainproxy connector test: gRPC method names were
  missing the leading slash ("/pkg.Service/Method"). Newer gRPC rejects the
  slash-less form ("Unimplemented: malformed method name") and returns an empty
  response. Real bug in the relayer reflection path.
- chainproxy connector: addClientsAsynchronously read len(freeClients) without
  the lock; the tests read freeClients/usedClients directly instead of via the
  locked numberOf* accessors.
- WS chain listeners (jsonRPC / tendermint / rest / grpc): listeningAddress was
  a plain string written by a goroutine and read by GetListeningAddress ->
  atomic.Pointer[string].
- consumer_websocket_manager graceful shutdown: the writer goroutine called
  Close() on the gofiber conn, which unblocks the handler and lets gofiber
  recycle the conn (fasthttp pooling) while the writer is still inside Close()
  (use-after-recycle). Now SetReadDeadline unblocks the read loop so the conn's
  owner performs the single close.
- tracing: corrected a stale test expectation — the OTel SDK no longer collapses
  a 1.0 sampling ratio to AlwaysOnSampler; TraceIDRatioBased{1} is spec-compliant
  and samples 100%.

Verified: go test -race green for chainlib, chainproxy, rpcclient, tracing; the
full non-race make test now passes with zero failures.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(chainlib): route WS idle-close frame through the single writer goroutine

The idle-timeout path in the rate-limit goroutine called websocketConn.WriteMessage
directly, bypassing the websocketConnWriteChan single-writer goroutine — a second
concurrent writer, which gofiber/gorilla forbids (data race) and which contradicts
the single-writer discipline the surrounding code documents. Route it through
sendWS like every other write; the client closing on the frame unblocks the read
loop as before. (Found in review of PR #143; the GracefulShutdown tests don't
exercise the idle path so -race hadn't flagged it.)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(chainproxy): read connector.usedClients under the lock, not via atomic

usedClients (int64) was written/read as a plain field under connector.lock
everywhere (GetRpc/ReturnRpc/Close) EXCEPT numberOfUsedClients(), which used
atomic.LoadInt64 without the lock. A mutex-protected write and a lone atomic read
are different happens-before domains, so the async addClientsAsynchronouslyGrpc
goroutine's read raced GetRpc's `usedClients++` (a rare ~1/21 -race flake in the
chainlib suite, captured via a sustained hunt). Read under the RLock to match
numberOfFreeClients(); connector.closed keeps its (consistent) atomic access.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(chaintracker): five review findings — tip tags, baseline age, sync gen, SyncBlock floor

P1-#1 dyncodec: keep reflection callback/parser paths slash-less; the leading
slash is added only at the grpc.ClientConn.Invoke boundary. Regression test
asserts both the slash-less callback and the slashed invoke.

P1-#2 tip eligibility: harvest a relay's block as the current tip only when the
method's spec function tag is GET_BLOCKNUM, or GET_BLOCK_BY_NUM requesting LATEST.
RequestedBlock==LATEST and GetUsedDefaultValue (non-deterministic for
eth_blockNumber) cannot discriminate receipt/by-hash historical responses. All
downstream tip writes stay gated behind the single harvest chokepoint.

P2-#3 chainstate baseline: split the overloaded baselineAt into baselineAt (TTL
freshness, refreshed every confirming Recompute) and baselineSince (establishment
time, preserved while the block is unchanged, reset on forward/downward/regained
change). GetConsensusBaselineWithTime returns baselineSince for sync-lag so a
stuck baseline accrues real age while a reconfirmed one never expires.

P2-#4 tracker generation: register the tracker and allocate its observation
generation synchronously before the relay dispatch captures it. GetOrCreateTracker
does no network I/O (the poll loop stays async), so an early relay no longer
captures generation 0 and silently drops its harvested tip.

P2-#5 optimizer SyncBlock: serialize each provider's QoS read-modify-write with a
striped lock and hold the authoritative monotonic SyncBlock floor under it. The
ristretto cache is async/eventually-consistent and cannot enforce monotonicity, so
a stale read (incl. the legacy availability-only writer) could regress the block;
the floor guarantees SyncBlock never decreases. ResetState clears the floors.

Verification: gofmt, go build ./..., go vet ./..., make test (all green), and
targeted -race for dyncodec, chainstate, endpointstate, provideroptimizer,
rpcsmartrouter.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(endpointstate): tolerate an empty GET_BLOCKNUM template for gRPC

The per-endpoint poller hard-failed when a GET_BLOCKNUM parse directive had an empty
function_template, returning "GET_BLOCKNUM missing function template apiInterface=grpc"
and retrying ChainTracker creation forever. But for gRPC the spec's api_name carries the
method and the function_template is only the request PAYLOAD — which is legitimately empty
for a no-argument call such as cosmos Service/GetLatestBlock. So an empty gRPC template
means "empty body", not "undefined method"; the relay path already serves these calls with
an empty body, only the poll path rejected them.

Extract blockNumRequestBody(apiInterface, template): gRPC resolves an empty template to "{}"
(the canonical empty JSON message the codec marshals to an empty proto); REST (needs a path)
and Tendermint (needs a method) still hard-error on an empty template, since polling those
without one would send garbage. Unit-tested across all four interfaces.

Surfaced by the chaintracker-redesign: the previous single global ChainTracker bound to one
provider, so a non-gRPC primary left this gRPC gap latent; per-endpoint pollers now exercise
every interface and hit it.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(endpointstate): complete gRPC per-endpoint poll support (method header + response descriptor)

Two layers kept the per-endpoint gRPC ChainTracker poll from completing — both because the
poll path was built for text protocols (REST/JSON-RPC/Tendermint) and gRPC differs
structurally. Following the empty-template fix:

1. Method header: gRPC carries the method PATH in an x-grpc-method header, not the URL or
   body. EndpointPoller.sendRawRequest dropped the apiName, lumping gRPC into the generic
   JSON-POST branch, so every poll failed "gRPC method path not provided in headers". Set
   lavasession.GRPCMethodHeader for the gRPC interface (the relay path already does).

2. Response descriptor: a gRPC response is binary protobuf; decoding it needs the protobuf
   method descriptor. CraftChainMessage can't resolve one (reflection needs a live
   connection), so the crafted message had none and parsing failed "does not have a
   methodDescriptor set in grpcMessage". The DirectRPCConnection resolves+caches the
   descriptor during the send and exposes it via GetCachedMethodDescriptor; new
   chainlib.HydrateGrpcResponseParsing wires it (and a grpcurl FormatJSON formatter — the
   SAME one the relay path uses, so camelCase field names match spec result_parsing
   parser_args) onto the crafted message before FormatResponseForParsing.

Applied to BOTH spec-driven poll paths via a shared hydrateGrpcChainMessage helper:
FetchLatestBlockNum (GET_BLOCKNUM) and fetchSingleBlockHash (GET_BLOCK_BY_NUM, the
fork-detection hash fetch that runs when blocksToSave > 1).

Verified live: the gRPC ChainTracker now polls and advances block-to-block, tracking the
same tip as the REST/tendermintrpc interfaces. Surfaced by the chaintracker-redesign:
per-endpoint pollers now exercise every interface, where the single global tracker bound to
one provider did not.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* refactor(endpointtip): consolidate per-endpoint tip into one shared store

The per-endpoint observed block tip was written to two independent places:
Endpoint.LatestBlock (an atomic on the lavasession struct) and the
EndpointObservation block triple (in endpointstate's observation map). Worse,
BOTH writes to Endpoint.LatestBlock were ungated — the poll-path write in
NewLatestCallback and the relay-path write in harvestAndUpdateTipFromRelay
bypassed the generation + time-monotonic guards that the observation triple
enforces — so a stale or replaced-tracker observation could move the atomic
while being correctly rejected by the gated store, leaving the two copies
diverged.

Introduce protocol/endpointtip: a stdlib-only leaf package holding the single
source of truth for the tip triple {Block, ObservedAt, Source}, keyed by
chain|apiInterface|url. Being a leaf, it is importable by BOTH the high layer
(endpointstate observers) and the low layer (lavasession QoS reader) without the
import cycle that previously forced the value to be pushed down onto Endpoint.

- endpointstate/observation.go: recordPollObservation and RecordRelayObservation
  keep the generation gate and write the triple to the store (the time-monotonic
  guard moves into Store.Set, semantics preserved exactly — time-, not
  block-monotonic). The observation map now stores only poll-health;
  GetObservation/SnapshotObservations compose the triple back from the store at
  read time (delegation, not a second copy). Lock order is obsMu → store lock
  everywhere; onTipObservation still fires after obsMu is released.
- Delete Endpoint.LatestBlock + LastBlockUpdate (the latter had no readers) and
  the two ungated writes; the three readers (consumer_session_manager syncBlock,
  rpcsmartrouter syncGap + consistency fallback) read the store via a nil-safe
  helper.
- Monitor RemoveTracker/Stop evict the chain's keys so the global map stays
  bounded as endpoints churn.

Tests migrated to seed/read the store; build, vet, and -race all green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(chainstate): derive outlier threshold from chain block time

The anti-lie / downward-realignment outlier threshold was a fixed 100-block
constant. The threshold is denominated in blocks, but the quantity it bounds —
how far the optimistic tip may legitimately lead consensus before a write is
treated as a lie — is denominated in time. So a fixed block count meant ~240x
different poison tolerance across chains: ~100min on a 60s chain, ~20min on
Ethereum, ~40s on Solana. On slow chains the guard was uselessly loose.

Derive it in DefaultConfig from the chain's average block time (already passed
in for the staleness window, so no new plumbing):

    OutlierThreshold = clamp(round(OutlierTimeBudget / avgBlockTime), floor, ceil)
    OutlierTimeBudget = 1200s   floor = 32   ceil = 512

- 1200s keeps a 12s chain (Ethereum) at the historical 100 → backward compatible;
  the only chain whose behavior shifts is one that wasn't getting the right
  semantics before.
- floor 32 keeps slow chains above the ~10-20-block legitimate lead (staleness
  window in blocks + propagation) so the guard never false-rejects an honest
  advance.
- ceil 512 is the primary fast-chain safety knob: without it, anchoring ETH to
  100 derives ~3000 Solana slots, a guard so loose a liar could hold the tip
  3000 slots ahead.
- avgBlockTime <= 0 (spec without average_block_time) falls back to the fixed
  DefaultOutlierThreshold (100), never 0.

Both use-sites (SetLatestBlock write guard, Recompute downward realign) stay on
the one cfg.OutlierThreshold. Adds a table test across 60s/12s/6s/0.4s/0.25s
chains; corrects the one existing assertion that baked in the old fixed value.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(smart-router/debug): add endpoint-state, chain-state, provider-routing read endpoints (MAG-2202)

Three read-only GET debug endpoints so the MAG-2201 automation suite can read
ChainTracker/Probing-redesign state from the wire instead of scraping logs.
Registered only under --debug-address; GET-only (any other method -> 405);
nil-router safe (returns []).

Each endpoint returns a flat array of self-describing records (ChainID +
ApiInterface, plus NetworkAddress for endpoint-state) — flat keys, no object
nesting, deterministically sorted — honoring the ticket's response-shape rules:

- /debug/endpoint-state: per-endpoint poll-health + enable/recovery, joining
  EndpointMonitor.SnapshotObservations with a new Endpoint.HealthSnapshot
  (Enabled/DisabledAt/ConsecutiveHealthyProbes/LastSuccessfulPoll/LatestBlock/
  ObservedAt/Source) by NetworkAddress.
- /debug/chain-state: raw, non-TTL-gated ChainState.DebugSnapshot
  (ObservedTip/ConsensusBaseline/HasBaseline/Initialized + LastObservedAt/
  BaselineSince) so tests can assert TTL expiry / realignment / baseline clearing.
- /debug/provider-routing: ConsumerSessionManager.ProviderRoutingSnapshot
  (ValidAddresses/CurrentlyBlockedProviderAddresses/BlockedBackupProviders).

Keys mirror the Go identifiers; timestamps RFC3339 (empty string for zero);
address slices non-nil so empties encode as [] not null. ChainID/ApiInterface
are sourced from listenEndpoint / csm.RPCEndpoint(), never by splitting the key.
Backing state already exists on chaintracker-redesign, so this only adds read
accessors + handlers.

Stacked on chaintracker-redesign (PR #143); merges after it. Endpoint 4
(/debug/probe-loop) follows in a separate PR — it needs counter instrumentation
on the existing runProbeLoop, not new read plumbing.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(smart-router/debug): add /debug/probe-loop prober-cycle telemetry (MAG-2202 endpoint 4)

The fourth read-only debug endpoint, instrumenting the existing Topic-D prober
(runProbeLoop) with per-cycle counters — no behavior change, just observation.

- runProbeCycleCore now returns (scored, reEnabled, syncOmitted): endpoints
  scored, endpoints re-enabled this cycle (F1), and providers whose QoS sample
  fed no sync evidence (F5: no fresh consensus baseline). Existing callers ignore
  the returns; behavior is unchanged.
- runProbeCycle times each cycle and records the snapshot onto a per-server
  probeLoopStats (mutex-guarded, off the data plane); runProbeLoop publishes the
  configured --probe-loop-interval cadence before the first tick.
- GET /debug/probe-loop returns a flat array of per-chain records (ChainID +
  ApiInterface): CycleIntervalMs, CyclesCompleted, LastCycleStartedAt (RFC3339),
  LastCycleDurationMs, EndpointsScored, ReEnabledCount, SyncOmittedCount.
  Reuses writeDebugRows / debugTimeRFC3339 from the endpoints-1-3 PR. GET-only
  (405 otherwise), nil-router safe.

Lets the suite verify the prober runs on its own cadence (F6), per-cycle
re-enable counting (F1), and the sync-omitted count (F5).

Stacked on #155 (endpoints 1-3); retarget to chaintracker-redesign after it merges.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(rpcsmartrouter): gate tip-state writes on relay-observation acceptance

Addresses PR #143 review (P1 correctness): a stale relay still mutated
tip state after the generation gate rejected it.

recordRelayBlockObservation was generation-gated, but the tip-state writes
that followed in harvestAndUpdateTipFromRelay (updateLatestBlockHeight's
bootstrap atomic + router metric, and the per-endpoint latest-block metric)
ran unconditionally. After a same-URL tracker replacement bumped the
generation, a stale relay's observation was correctly dropped from the store
but still poisoned every other tip-state field.

RecordRelayObservation / recordRelayBlockObservation now return whether the
write was accepted (passed the generation + monotonic guards), and
harvestAndUpdateTipFromRelay early-returns when it was not, gating the
remaining writes.

Extends the generation-rejection harvest test to assert the bootstrap atomic
is not moved by a stale-generation relay (fails without the fix).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(chainstate): break majority-baseline window ties toward the highest

Addresses PR #143 review (P2 correctness): computeMajorityBaseline kept only
the first max-size window, understating the baseline on ties.

The clustering loop used a strict `count > bestCount`, so a later equal-size
window never replaced the incumbent. With observations [1000,1002,1004] and
BucketWidth 2 there are two overlapping 2-of-3 majority windows ([1000,1002]
and [1002,1004], sharing 1002), and the loop returned 1002 — understating the
consensus baseline that feeds sync scoring and outlier/realignment decisions.
The "strict majority cluster is unique" doc claim only rules out DISJOINT
rivals; overlapping equal-size majorities genuinely coexist.

Break window-size ties toward the most-advanced window
(count == bestCount && blocks[j] > bestMax), so the baseline is never
understated, and correct the doc comment accordingly.

Adds TestComputeMajorityBaseline_OverlappingTiePrefersHighest, which asserts
the staircase resolves to 1004 (fails returning 1002 without the fix).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(rpcsmartrouter): don't charge in-cluster endpoints a sync-gap

Addresses PR #143 review (correctness, low severity): because the consensus
baseline is the winning cluster's MOST-ADVANCED block, a majority sitting one
cluster-step below it was charged a sync-gap against a tip only the fastest
cluster member reported (e.g. [1000,1000,1002]/BucketWidth=2 → baseline 1002,
the two 1000-nodes charged syncGap 2).

Rather than change the baseline element (which would reverse the just-shipped
P2 highest-window fix and alter consensus semantics), fix it at the symptom:
an endpoint within BucketWidth of the baseline is, by definition, inside the
agreeing cluster, so Site C zeroes its sync-gap. The new clamp subsumes the
old `syncGap < 0` (endpoint-ahead) case.

Exposes ChainState.ConsensusBucketWidth() for the clamp and adds
TestSiteC_InClusterEndpointChargedNoSyncGap pinning the arithmetic.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* perf(chaintracker): skip block-gap machinery for flat per-endpoint trackers

Addresses PR #143 review (efficiency): flat per-endpoint trackers ran the
adaptive block-gap machinery whose result nothing consumes.

For a flat tracker (flatPollInterval > 0) computePollInterval returns a fixed
cadence and never consults the block-gap sweep, yet start() still fired
blockGapTicker (a Percentile+Stability sweep over blockEventsGap) and
AddBlockGap still appended per block — pure wasted CPU/allocation multiplied
across every direct-RPC endpoint.

The reviewer's premise that the result is "discarded" is incomplete: the sweep
also feeds updateAverageBlockTimeForRegistrations -> UpdateBlockTime. But
RegisterForBlockTimeUpdates has no production call sites (only a test, which
uses a NON-flat tracker), so for flat trackers that consumer is empty too and
the conclusion holds. So finding-4's "still runs for block-time consumers"
rationale does not apply to per-endpoint trackers.

Start blockGapTicker only for the global adaptive tracker (nil channel parks
the select case for flat trackers) and gate the per-block AddBlockGap append on
flatPollInterval == 0. setLatestChangeTime stays unconditional (it still drives
the not-updated/emergency path). The global adaptive tracker is unchanged.

Adds TestFlatTracker_SkipsBlockGapMachinery (blockEventsGap stays empty for a
flat tracker; fails without the gate). The existing adaptive block-time-update
test still passes, proving the global path is untouched.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* refactor(probing): reuse chainstate staleness multiplier as one source of truth

Addresses PR #143 review (cleanup): probing.defaultStalenessMultiplier (10) and
chainstate.DefaultStalenessMultiplier (10) independently encoded the same
"fresh/alive horizon = N x blockTime" concept, so tuning one would silently
diverge the per-endpoint liveness horizon from the per-chain consensus window.

chainstate is a stdlib-only leaf package (no import cycle), and the probing
comment already noted it "mirrors the chainstate consensus window", so make
chainstate.DefaultStalenessMultiplier the single named source: drop the local
duplicate and reference the exported constant from probing.DefaultVerdictConfig.
The floors stay package-specific (minProbeStaleness 5s vs minStalenessWindow 2s)
— only the shared multiplier is unified. Documented the dependency on both ends.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* refactor(lavasession): centralize F1 recovery-field resets behind helpers

Addresses PR #143 review (cleanup): the recovery-field reset block was
copy-pasted across six disable/enable sites, so "every Enabled transition
resets the recovery fields" was enforced only by remembering to copy it — a new
path that missed one would silently break F1 hysteresis with no compile error.

Extract two composed, lock-held helpers on Endpoint:
- clearRecoveryStreakLocked()    — zeroes the streak (consecutiveHealthyProbes +
  lastRecoveryPoll), leaving disabledAt; used by both disable sites (after
  stamping disabledAt) and the enabled-domain reset in RecordProbeVerdict.
- clearRecoveryTrackingLocked()  — streak + disabledAt; the full clean slate for
  every Enabled->true transition (RecordProbeVerdict re-enable, ResetHealth,
  the session reconnect path). This is the named helper the reviewer requested.

Deliberately NOT folded in: the invalid-evidence reset in RecordProbeVerdict
(the reviewer's listed "319"). It zeroes consecutiveHealthyProbes but PRESERVES
lastRecoveryPoll on purpose, so a probe cadence faster than the poll cadence
cannot re-count an already-counted poll across a streak break. Routing it
through a clear helper would silently break that F1 guard — exactly the hazard
the comment warns about — so it stays inline (documented on the helper).

Behavior-preserving: same field writes at every site; lock context verified
held at both session sites. The lavasession F1/hysteresis, probe-reenable, and
recovery suites all still pass.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* refactor(endpointstate): delete dead ValidateEndpointSync / GetSyncGap

Addresses PR #143 review (cleanup): both EndpointMonitor.ValidateEndpointSync
and GetSyncGap are callerless in production — a repo-wide search finds them only
in their own tests. Sync-gap is now computed inline against
chainState.GetConsensusBaseline (rpcsmartrouter_server.go:~1520), so these
methods are dead. Per CLAUDE.md's "minimise exported surface area", remove them
and their tests.

GetLatestBlockNum (which they delegated to) stays — it is still called in
production (rpcsmartrouter_server.go:1649, 1692).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* refactor(chainstate): drop test-only HasConsensusBaseline; use DebugSnapshot

Addresses PR #143 review (cleanup): HasConsensusBaseline was exported but had
only test callers, and it bypassed the TTL freshness gate — inviting a future
caller to read a stale baseline, the exact bug the design removed.

It is fully redundant with DebugSnapshot(), which already exposes the same raw,
non-TTL (ConsensusBaseline, HasBaseline) pair under the same read lock and IS
the telemetry path. Drop the method (minimise exported surface area) and migrate
the test callers across chainstate/rpcsmartrouter/endpointstate to DebugSnapshot.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(smart-router): address PR #143 review findings (#1-3, #5, #7-8)

Fixes from the PR #143 review (docs/PR-143-REVIEW.md). The top findings share
one root: getLatestBlock() now intentionally returns 0 on a TTL-stale tip, and
several consumers mishandle the 0.

- #1 archive routing: add getLatestBlockBestEffort() (strict tip, else last-known
  atomic, 0 only at cold-start) for the three getExtensionsFromDirectiveHeaders
  returns, so a stale window no longer forces every concrete-block request to the
  archive pool. Cache finalization stays on the strict getLatestBlock().
- #2 jsonRPC eth_call: guard the archive-depth subtraction (parsedBlock>=0 &&
  LatestBlock>ethCallArchiveBlockDepth) so uint64(0)-126 can't underflow and force
  archive for every eth_call (also covers chains younger than the depth).
- #3 averageBlockTime: floor the spec block time once (DefaultAverageBlockTime) and
  feed the same value to both ChainState and the EndpointMonitor, so the consensus
  window can't end up shorter than the poll cadence on no-average_block_time specs.
- #5 endpoint flap: dampen the poll-healthy/relay-failing flap with a capped,
  decaying re-enable hysteresis (reEnableAfterK << reenableProbeFlaps, cap 2; a
  successful relay decays it). Race-safe under e.mu; bounded well below the epoch.
- #7 hot path: cache the two tip-defining ApiNames once (sync.Once) so
  tipBlockFromRelay no longer takes the parser RWLock twice per relay.
- #8 cleanup: make endpointstate.ObservationSource a type alias of
  endpointtip.Source (drop the duplicate enum/String/converter).

#4 (cache finalization) and #6 (sub-majority sync) resolved as by-design with
hardened comments/rationale; two separate latent items recorded as follow-ups.

go build/vet clean; lavasession, endpointstate, chainstate, rpcsmartrouter,
chainlib suites green (incl. -race where applicable).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* removed doc

* build: bump Go toolchain to 1.26.5 to clear govulncheck GO-2026-5856

The govulncheck CI gate flagged GO-2026-5856 (Encrypted Client Hello
privacy leak in crypto/tls), reachable from our TLS-using code paths.
It is a Go standard-library advisory fixed in go1.26.5 — no application
code change applies. CI pins the scan's Go via go-version-file: go.mod,
so bumping the directive from 1.26.4 to 1.26.5 makes setup-go install the
patched stdlib and the scan goes clean.

Verified locally: govulncheck v1.5.0 built under go1.26.5 reports
"No vulnerabilities found" (exit 0); go build ./... passes under 1.26.5.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Took 28 minutes

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Anna Rayer <anna@magmadevs.com>
avitenzer pushed a commit that referenced this pull request Jul 21, 2026
…lock timestamps (MAG-2307 review)

Review follow-up to PR #222 (MAG-2307, merged as c011a80). Addresses two required
changes from VicSheCodes and all three Copilot comments.

Decoupling (VicSheCodes #1): /debug/time-warp is the optimizer/QoS clock, and the
black-box suite runs warp-then-reset-to-0 routinely to reset QoS state -- coupling
ChainState to it risked disturbing ChainState on a QoS reset. ChainState's TTL/staleness
clock now has its own endpoint, POST /debug/chain-state-time-warp (offset_seconds relative
to real time, 0 = real; same finite/>=0/<=24h validation), fully independent. Neither
/debug/time-warp nor /debug/reset-all touches ChainState any more.

Timestamp writes (VicSheCodes #2 = Copilot chain_state.go:258 and :377): SetLatestBlock
and Recompute stored lastObservedAt/baselineAt/baselineSince with the WARPED clock. A write
while a warp was active stamped a future time, so clearing the warp left real - future =
negative age, reading as fresh forever. Now freshness is EVALUATED against the warped clock
but stored timestamps always use the real base clock (cs.now()).

Doc fix (Copilot chain_state.go:408): GetConsensusBaselineWithTime returns baselineSince
(establishment time), not baselineAt (the internal TTL-freshness stamp, never exposed).

Tests: dedicated-endpoint HTTP e2e; /debug/time-warp independence (does not touch ChainState);
write-under-warp regressions pinning the real-clock timestamp and no negative-age freshness.
Full chainstate (-race) and rpcsmartrouter suites pass; go build ./... and gofmt clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
nimrod-teich pushed a commit that referenced this pull request Aug 16, 2026
…2724)

A provider's configured name was its routing identity, and the maps keyed by
that identity collapsed two entries sharing a name into one. One upstream
served every request while the other sat idle, and because the retry skip-list
used the same key, setting that name aside after a failure removed both and
left nothing to retry against — the customer got an error while a healthy
provider stood by.

Commit dffc0f2 established that duplicate names must be tracked separately, but
only for the startup failed-provider set. Everything downstream was still keyed
by name.

Two layers:

Fail fast. ValidateUniqueProviderNames rejects two providers sharing a name on
one chain+api-interface, before any listener binds. That scope is deliberate —
a session manager is built per chain+interface and owns its own pairing map, so
it is exactly where names collapse; reusing an operator label across chains
stays legal. Run per list at parse time and across static+backup together,
since the two are looked up against each other by address. Every collision is
reported at once and sorted, so one fix pass suffices and the message does not
vary between runs.

Name + index. ResolveDuplicateAddresses gives colliding providers distinct
identities "<name>#0", "<name>#1", indexed by config order so they are stable
across restarts. An unambiguous name keeps the bare configured name, leaving
pin headers, debug output and metric labels untouched for every provider not in
a collision. Unreachable behind the validation above; it exists so a list that
reaches the session manager by another path degrades to "both routable" rather
than "one silently dropped".

Address() replaces Name at all four identity sites: session registration,
mergeAbsentProviders, the restored-set match, and the metrics endpoint_id. That
last one matters — health metrics key off PublicLavaAddress, so leaving
RegisterEndpoint on Name would split a collided provider across two endpoint_id
values. The three restore-path sites compared config Name against session
PublicLavaAddress and agreed only because the two were identical; changing the
derivation without them would report every provider absent and merge duplicate
sessions fleet-wide.

Tests drive UpdateAllProviders and getValidProviderAddresses rather than
restating their logic, which was the flaw in the existing duplicate-name test.
Confirmed non-vacuous: without resolution the same path yields one pairing
entry, validAddresses [SimTwin SimTwin], and an empty selection.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
avitenzer added a commit that referenced this pull request Aug 17, 2026
…2724)

A provider's configured name was its routing identity, and the maps keyed by
that identity collapsed two entries sharing a name into one. One upstream
served every request while the other sat idle, and because the retry skip-list
used the same key, setting that name aside after a failure removed both and
left nothing to retry against — the customer got an error while a healthy
provider stood by.

Commit dffc0f2 established that duplicate names must be tracked separately, but
only for the startup failed-provider set. Everything downstream was still keyed
by name.

Two layers:

Fail fast. ValidateUniqueProviderNames rejects two providers sharing a name on
one chain+api-interface, before any listener binds. That scope is deliberate —
a session manager is built per chain+interface and owns its own pairing map, so
it is exactly where names collapse; reusing an operator label across chains
stays legal. Run per list at parse time and across static+backup together,
since the two are looked up against each other by address. Every collision is
reported at once and sorted, so one fix pass suffices and the message does not
vary between runs.

Name + index. ResolveDuplicateAddresses gives colliding providers distinct
identities "<name>#0", "<name>#1", indexed by config order so they are stable
across restarts. An unambiguous name keeps the bare configured name, leaving
pin headers, debug output and metric labels untouched for every provider not in
a collision. Unreachable behind the validation above; it exists so a list that
reaches the session manager by another path degrades to "both routable" rather
than "one silently dropped".

Address() replaces Name at all four identity sites: session registration,
mergeAbsentProviders, the restored-set match, and the metrics endpoint_id. That
last one matters — health metrics key off PublicLavaAddress, so leaving
RegisterEndpoint on Name would split a collided provider across two endpoint_id
values. The three restore-path sites compared config Name against session
PublicLavaAddress and agreed only because the two were identical; changing the
derivation without them would report every provider absent and merge duplicate
sessions fleet-wide.

Tests drive UpdateAllProviders and getValidProviderAddresses rather than
restating their logic, which was the flaw in the existing duplicate-name test.
Confirmed non-vacuous: without resolution the same path yields one pairing
entry, validAddresses [SimTwin SimTwin], and an empty selection.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants