Repository navigation
feat: initial smart router implementation - #1
Merged
Merged
Conversation
Multi-protocol RPC gateway that routes JSON-RPC, REST, gRPC, and Tendermint RPC requests to statically configured upstream endpoints with QoS-based provider selection, caching, and automatic failover. Components: - cmd/smartrouter, cmd/lavap: Cobra entrypoints for rpcsmartrouter and cache subcommands (lavap kept as a compat alias). - protocol/chainlib: per-interface parsers and proxies (JSON-RPC, REST, Tendermint RPC, gRPC) with chainproxy clients and rpcInterfaceMessages encoders. - protocol/rpcsmartrouter: router server, direct-RPC relay path, retry state machine, endpoint chain fetcher/tracker, WS/gRPC subscription managers. - protocol/relaycore, provideroptimizer, lavasession, chaintracker, qos: relay orchestration, QoS scoring and weighted provider selection, session/connection pooling, per-chain latest-block tracking. - protocol/metrics: Prometheus metrics and optional Kafka analytics. - ecosystem/cache: standalone cache server process the router connects to via --cache-be. - specs/: offline JSON specs for ethereum, lava, cosmossdk, tendermint, ibc, grpctest — loaded via --use-static-spec. - utils/, types/: shared logger (lavalog), scoring primitives, spec and relay types, spec fetcher. - scripts/pre_setups/: end-to-end bootstrap scripts for LAVA and ETH1. Provenance: hard fork of github.com/lavanet/lava/v5, rebranded to github.com/magma-Devs/smart-router. Scope is trimmed to the router and its transitive dependencies — no chain/node/keeper code — and no upstream merges are expected. "lavanet/lava" strings remaining in specs/lava.json and protocol/chainlib/rest_test.go are Cosmos REST paths served by the Lava node, not GitHub refs, and must not be rewritten. Took 2 minutes
Contributor
Test Results0 tests 0 ✅ 0s ⏱️ Results for commit ecbe90a. ♻️ This comment has been updated with latest results. |
The initial import carried over CI workflows verbatim from lavanet/lava/v5. Several of them reference directories and scripts that were stripped out of this fork (x/, testutil/e2e/, testutil/common/, scripts/protocgen.sh, cmd/lavad, cmd/lavavisor), causing every PR to fail 6+ checks. Removed (upstream-only, non-operational in this fork): - cli_tests.yml — references /home/runner/work/lava/lava paths; the only real test step was commented out. - proto.yml — runs scripts/protocgen.sh, which does not exist. - network-upgrades-tag-populate.yml — writes to the external lavanet/devops repo on every version tag. - lava_helm_charts_tags_populate.yml — targets magma-Devs/smart-router-helm-charts, flagged in-file as unverified. Rewrote lava.yml (now "Build and Test"): - Single go test ./... run instead of consensus/protocol/e2e splits that referenced non-existent packages. - Build matrix covers smartrouter + lavap across linux/darwin and amd64/arm64; removed lavad and lavavisor targets. - Docker build uses the top-level Dockerfile (which actually exists) instead of cmd/*/Dockerfile (which don't), and doesn't push on PRs. - Keeps junit reporting, Codecov upload, and EnricoMi's publish action for test result comments. Follow-up: release.yml still references cmd/lavad/Dockerfile and a cosmovisor image; it won't fail this PR (tag-triggered only) but should be addressed before cutting the first release. Took 35 minutes
- Rename .github/workflows/lava.yml -> smartrouter.yml so the filename reflects the fork's actual product. - Drop lavap from the build matrix. CI now produces only smartrouter binaries across linux/darwin x amd64/arm64 (4 artifacts instead of 8). Workflow display name remains "Build and Test". Took 6 minutes
Replaces remaining "Lava" branding that referred to the upstream
codebase's identity with "Smart Router" — this fork is the centralised
routing solution, distinct from the Lava decentralised network which
remains a legitimate routing target.
Kept intentionally (Lava-as-upstream-chain):
- specs/lava.json and Lava REST paths in tests/specs
- config/smartrouter_examples/smartrouter_lava.yml (target-chain name)
- "Lava blockchain via PublicNode" wording in docs
- cmd/lavap/ alias binary for legacy ops tooling
- LICENSE.md "Copyright 2022 Lava Network" (required by Apache 2.0)
Changes:
- .goreleaser.yaml: project_name lava -> smart-router; drop lavavisor
and lavad builds/archives; add smartrouter build and archive; keep
lavap build for legacy distribution.
- .github/workflows/release.yml: retitle to "Publish Smart Router
Release"; replace lavap/lavad docker matrix with a single
smart-router image built from the root Dockerfile.
- .github/pr_labeler.yml: drop dead path labels (cmd/lavad,
cmd/lavavisor, x/**, cookbook/**, proto/**, docs/**, Team:Consensus);
add labels for cmd/smartrouter, ecosystem, utils, types, scripts.
- .github/ISSUE_TEMPLATE/bug_report.yml: rename "Lava Version" field to
"Smart Router Version"; update placeholder.
- README.md: retitle to "Smart Router"; reframe intro as a centralised
gateway; update renamed script paths.
- protocol/rpcsmartrouter/README.md: "Lava's RPC gateway" ->
"centralised RPC gateway".
- CONTRIBUTING.md, CODING_GUIDELINES.md, CODE_OF_CONDUCT.md: retitle
and reword product identity references.
- scripts/pre_setups/init_lava_smartrouter_{eth,lava}.sh renamed to
init_smartrouter_{eth,lava}.sh; "killall lavap lavad" replaced with
"killall smartrouter lavap"; self-referencing comment updated.
Out of scope (Phase 2, separate PR):
- Go package renames (lavasession, lavaprotocol, lavalog, lavaslices).
Three collide with Go stdlib if renamed to the obvious targets; needs
a design decision.
- scripts/useful_commands.sh lavad helper functions (unused by current
scripts; would only matter if a developer calls them directly).
- /lava/health HTTP endpoint path (requires code changes).
Took 13 minutes
Docker build was failing because Dockerfile pinned Go 1.23 while go.mod declares 1.26.2; the builder image could not compile Go 1.26 syntax. Bumped ARG GO_VERSION to 1.26. Lint was failing because golangci-lint v1.62.0 (built with Go 1.23) refuses to lint a newer toolchain than itself. Migrated to: - golangci-lint-action v6 with version: latest (picks up v2.x). - .golangci.yml v2 schema: version: "2" header, linters moved under linters.default: none + enable list, settings nested in linters.settings, exclusions under linters.exclusions, and formatters (gofmt/gofumpt/goimports) hoisted into a dedicated formatters block per the v2 convention. Lint is temporarily advisory (continue-on-error on every retry) while the v2 migration settles. Once a pinned v2.x version is known to produce a clean run against this codebase, drop the continue-on-error flags to re-enforce the gate. Took 4 minutes
golang:1.26-alpine3.21 does not exist on Docker Hub — Alpine 3.21 only has golang:1.23 images. Go 1.26 was only published against alpine3.22 and alpine3.23. Confirmed via docker manifest inspect: - golang:1.26-alpine3.21 -> no such manifest - golang:1.26-alpine3.22 -> exists Took 12 minutes
Took 12 minutes
nimrod-teich
approved these changes
Apr 23, 2026
avitenzer
added a commit
that referenced
this pull request
Apr 30, 2026
…Sprint 3) Routes runtime config through the SmartRouterConfig dispatcher introduced in Sprint 2 — community now refuses non-EVM specs / non-jsonrpc interfaces / WS and gRPC transports at startup with the pinned error messages, and enterprise unlocks them after a valid embedded license is verified. Engine gates (§3.3.6 audit rows in protocol/rpcsmartrouter/rpcsmartrouter.go): - Rows #1, #2 — ValidateAPIInterface + ValidateSpec before NewChainParser - Row #3 — ValidateTransport on WS endpoints; HasPrefix stays as the discriminator, ValidateTransport is the gate - Row #3a — bare-host gRPC detection in convertProvidersToSessions; closure signature widened to return error so the gate can propagate via the existing errCh path - Rows #4, #7 — NewDirect*SubscriptionManager → ActiveConfig().Create*(); Start(ctx) lifecycle stays outside the factory via an inline interface assert (only Direct impls need starting; noops don't) - Row #5 — unchanged (existing NewNoOpWSSubscriptionManager fallback matches what the community factory would return) - Row #6 — defensive ValidateAPIInterface(grpc) precondition; documented as defense-in-depth against future row-#1 refactors Startup (cmd/smartrouter/): - main.go calls installLicenseCheck(rootCmd) — license validation fires only on the router command's PreRunE, not on version / cache / test, matching CLAUDE.md's "cache is community-available" rule. - startup_community.go (//go:build !enterprise) — INFO log line only. - startup_enterprise.go (//go:build enterprise) — validateAndActivateLicense reads the embedded license, dispatches on LicenseStatus, calls ActivateConfig, and starts expiryWatcher(ctx, license). Watcher cadence: hourly during grace, hourly in last 7 days, daily in last 30 days, daily poll outside the warning window so long-running processes detect entry. - validateAndActivateLicense takes the license key as a parameter so tests can mint per-state licenses with ephemeral keys (license_test.go pattern). CI guard: - scripts/check_gated_symbols.sh + make check-gates fail if any new caller reaches the gated constructors (NewDirect*SubscriptionManager, lavasession.NewDirectRPCConnection, chainlib.NewChainParser, RegisterEnterpriseConfig, the ws://-prefix discriminator) outside an explicit per-symbol allowlist. Test files are exempt. Verified: go build ./... and go build -tags enterprise ./... both clean; go vet both modes clean; full test suites pass both modes (rpcsmartrouter, licensing, cmd/smartrouter); make check-gates reports 0 violations across 6 symbols; go tool nm confirms community binary contains 0 enterpriseConfig / EmbeddedLicense / expiryWatcher symbols (enterprise binary has all 9 enterpriseConfig methods plus the license-flow functions). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Took 1 hour 1 minute
2 of 3 tasks
nimrod-teich
pushed a commit
that referenced
this pull request
Jun 18, 2026
…Sprint 3) Routes runtime config through the SmartRouterConfig dispatcher introduced in Sprint 2 — community now refuses non-EVM specs / non-jsonrpc interfaces / WS and gRPC transports at startup with the pinned error messages, and enterprise unlocks them after a valid embedded license is verified. Engine gates (§3.3.6 audit rows in protocol/rpcsmartrouter/rpcsmartrouter.go): - Rows #1, #2 — ValidateAPIInterface + ValidateSpec before NewChainParser - Row #3 — ValidateTransport on WS endpoints; HasPrefix stays as the discriminator, ValidateTransport is the gate - Row #3a — bare-host gRPC detection in convertProvidersToSessions; closure signature widened to return error so the gate can propagate via the existing errCh path - Rows #4, #7 — NewDirect*SubscriptionManager → ActiveConfig().Create*(); Start(ctx) lifecycle stays outside the factory via an inline interface assert (only Direct impls need starting; noops don't) - Row #5 — unchanged (existing NewNoOpWSSubscriptionManager fallback matches what the community factory would return) - Row #6 — defensive ValidateAPIInterface(grpc) precondition; documented as defense-in-depth against future row-#1 refactors Startup (cmd/smartrouter/): - main.go calls installLicenseCheck(rootCmd) — license validation fires only on the router command's PreRunE, not on version / cache / test, matching CLAUDE.md's "cache is community-available" rule. - startup_community.go (//go:build !enterprise) — INFO log line only. - startup_enterprise.go (//go:build enterprise) — validateAndActivateLicense reads the embedded license, dispatches on LicenseStatus, calls ActivateConfig, and starts expiryWatcher(ctx, license). Watcher cadence: hourly during grace, hourly in last 7 days, daily in last 30 days, daily poll outside the warning window so long-running processes detect entry. - validateAndActivateLicense takes the license key as a parameter so tests can mint per-state licenses with ephemeral keys (license_test.go pattern). CI guard: - scripts/check_gated_symbols.sh + make check-gates fail if any new caller reaches the gated constructors (NewDirect*SubscriptionManager, lavasession.NewDirectRPCConnection, chainlib.NewChainParser, RegisterEnterpriseConfig, the ws://-prefix discriminator) outside an explicit per-symbol allowlist. Test files are exempt. Verified: go build ./... and go build -tags enterprise ./... both clean; go vet both modes clean; full test suites pass both modes (rpcsmartrouter, licensing, cmd/smartrouter); make check-gates reports 0 violations across 6 symbols; go tool nm confirms community binary contains 0 enterpriseConfig / EmbeddedLicense / expiryWatcher symbols (enterprise binary has all 9 enterpriseConfig methods plus the license-flow functions). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Took 1 hour 1 minute
avitenzer
added a commit
that referenced
this pull request
Jun 23, 2026
…c gen, SyncBlock floor P1-#1 dyncodec: keep reflection callback/parser paths slash-less; the leading slash is added only at the grpc.ClientConn.Invoke boundary. Regression test asserts both the slash-less callback and the slashed invoke. P1-#2 tip eligibility: harvest a relay's block as the current tip only when the method's spec function tag is GET_BLOCKNUM, or GET_BLOCK_BY_NUM requesting LATEST. RequestedBlock==LATEST and GetUsedDefaultValue (non-deterministic for eth_blockNumber) cannot discriminate receipt/by-hash historical responses. All downstream tip writes stay gated behind the single harvest chokepoint. P2-#3 chainstate baseline: split the overloaded baselineAt into baselineAt (TTL freshness, refreshed every confirming Recompute) and baselineSince (establishment time, preserved while the block is unchanged, reset on forward/downward/regained change). GetConsensusBaselineWithTime returns baselineSince for sync-lag so a stuck baseline accrues real age while a reconfirmed one never expires. P2-#4 tracker generation: register the tracker and allocate its observation generation synchronously before the relay dispatch captures it. GetOrCreateTracker does no network I/O (the poll loop stays async), so an early relay no longer captures generation 0 and silently drops its harvested tip. P2-#5 optimizer SyncBlock: serialize each provider's QoS read-modify-write with a striped lock and hold the authoritative monotonic SyncBlock floor under it. The ristretto cache is async/eventually-consistent and cannot enforce monotonicity, so a stale read (incl. the legacy availability-only writer) could regress the block; the floor guarantees SyncBlock never decreases. ResetState clears the floors. Verification: gofmt, go build ./..., go vet ./..., make test (all green), and targeted -race for dyncodec, chainstate, endpointstate, provideroptimizer, rpcsmartrouter. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
avitenzer
added a commit
that referenced
this pull request
Jun 30, 2026
Fixes from the PR #143 review (docs/PR-143-REVIEW.md). The top findings share one root: getLatestBlock() now intentionally returns 0 on a TTL-stale tip, and several consumers mishandle the 0. - #1 archive routing: add getLatestBlockBestEffort() (strict tip, else last-known atomic, 0 only at cold-start) for the three getExtensionsFromDirectiveHeaders returns, so a stale window no longer forces every concrete-block request to the archive pool. Cache finalization stays on the strict getLatestBlock(). - #2 jsonRPC eth_call: guard the archive-depth subtraction (parsedBlock>=0 && LatestBlock>ethCallArchiveBlockDepth) so uint64(0)-126 can't underflow and force archive for every eth_call (also covers chains younger than the depth). - #3 averageBlockTime: floor the spec block time once (DefaultAverageBlockTime) and feed the same value to both ChainState and the EndpointMonitor, so the consensus window can't end up shorter than the poll cadence on no-average_block_time specs. - #5 endpoint flap: dampen the poll-healthy/relay-failing flap with a capped, decaying re-enable hysteresis (reEnableAfterK << reenableProbeFlaps, cap 2; a successful relay decays it). Race-safe under e.mu; bounded well below the epoch. - #7 hot path: cache the two tip-defining ApiNames once (sync.Once) so tipBlockFromRelay no longer takes the parser RWLock twice per relay. - #8 cleanup: make endpointstate.ObservationSource a type alias of endpointtip.Source (drop the duplicate enum/String/converter). #4 (cache finalization) and #6 (sub-majority sync) resolved as by-design with hardened comments/rationale; two separate latent items recorded as follow-ups. go build/vet clean; lavasession, endpointstate, chainstate, rpcsmartrouter, chainlib suites green (incl. -race where applicable). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
avitenzer
added a commit
that referenced
this pull request
Jul 8, 2026
…ations → ChainState consensus → proactive prober (#143) * refactor(endpointstate): extract per-endpoint ChainTracker into its own package Prep for the ChainTracker & Probing Redesign (MAG-2171). Splits the per-endpoint block-telemetry "data plane" out of rpcsmartrouter into a dedicated package so the upcoming probing "decision plane" has a clean, one-way boundary to build against (probing may read endpointstate; never the reverse). - Move endpoint_chain_tracker_manager.go -> endpointstate/endpoint_monitor.go (EndpointChainTrackerManager -> EndpointMonitor) - Move endpoint_chain_fetcher.go -> endpointstate/endpoint_poller.go (EndpointChainFetcher -> EndpointPoller) - Keep EndpointChainTrackerConfig / EndpointChainTrackerState names: they describe the underlying chaintracker concept, which is still accurate. - Add protocol/probing/doc.go: empty decision-plane shell staking out the two-plane boundary for a later extraction ticket. - Promote the in-package joinURLPath helper (surfaced as a cross-package dependency by the move) to common.JoinURLPath; relocate its test. - Relocate the cancelFuncs lifecycle + ResetAllLatestBlocks count-correctness tests into endpointstate (they poke unexported maps, now in-package). - Rewrite the rpcsmartrouter debug router-walk test to exported behavior only (real empty monitors + nil-safety); per-tracker count correctness is covered in endpointstate. Verified: go build ./..., go vet ./..., endpointstate (+ -race), full rpcsmartrouter suite, and common JoinURLPath all pass. One-way import edge confirmed (rpcsmartrouter -> endpointstate); no cycle. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(endpointstate): add per-endpoint observation contract (MAG-2158, Topic A) Topic A of the ChainTracker & Probing redesign: make the per-endpoint monitor the reliable, side-effect-free observation source that Topics B/C/D read. - EndpointObservation (observation.go): the locked contract — LatestBlock, ObservedAt, Source (Poll|Relay), LastPollAttempt, LastSuccessfulPoll, LastPollLatency, LastPollError, ConsecutivePollFailures. The block triple is written by whichever source produced the most recent block (ObservedAt monotonic); poll-health fields are written only by the poll path. - EndpointMonitor: RecordPollObservation / RecordRelayObservation / GetObservation (consistent-snapshot read). Backed by a dedicated obsMu so the hot observation path doesn't contend with tracker-lifecycle (mu); documented lock ordering. Records dropped on RemoveTracker/Stop to stay bounded. - EndpointPoller: an onPollObservation callback fires on every FetchLatestBlockNum round-trip (success or failure), with transport-only latency measured around sendRawRequest. Wired by the manager in GetOrCreateTracker. Additive and behavior-neutral: recording never touches QoS or endpoint.Enabled; Endpoint.LatestBlock/LastBlockUpdate are unchanged. RecordRelayObservation's call site is wired by Topic B (MAG-2159). Verified: go build ./..., go vet ./..., endpointstate (+ -race, incl. a concurrent poll/relay/read snapshot test), and the full rpcsmartrouter suite all pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(endpointstate): record poll observations on the SVM path; harden lifecycle The per-endpoint observation contract (MAG-2158) attached its poll instrumentation only to EndpointPoller.FetchLatestBlockNum, which Solana-family ChainTrackers never call -- SVMChainTracker fetches the latest slot via CustomMessage. So SOLANA/SOLANAT/KOII endpoints recorded zero poll observations (the main fast-chain/quota case). The direct Record* unit tests passed because nothing drove a real poll. - Add chaintracker.PollObserver (ObserveLatestBlockPoll). EndpointPoller implements it; SVMChainTracker.FetchLatestBlockNum calls it exactly once per poll with transport-only latency measured around CustomMessage. The default path keeps recording via FetchLatestBlockNum's defer; the two are mutually exclusive by chain family, so no double-recording. - RecordPollObservation -> internal recordPollObservation(url, gen, ...). A per-tracker generation gate rejects late callbacks from removed or replaced trackers (no recreate-after-remove, no overwrite of a new same-URL tracker); a sticky stopped flag refuses writes after Stop. - Monotonic poll-health: a stale poll (older than the last attempt) is dropped wholesale; equal timestamps apply (last-writer-wins). - ObservationSourceUnknown = iota, so a zero/failure-only record reports Unknown instead of masquerading as poll-sourced. RecordRelayObservation stays exported; its production call site remains deferred to MAG-2159. Tests: real-poll e2e through GetOrCreateTracker for non-SVM and SVM (success + failure); chaintracker-package tests asserting the SVM wrapper records exactly one observation per poll on every path; lifecycle/ generation/monotonic/source unit tests. Verified: gofmt; go test -race ./protocol/endpointstate; go test ./protocol/chaintracker ./protocol/lavasession ./protocol/rpcsmartrouter; go vet ./protocol/...; go build ./... Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(rpcsmartrouter, chaintracker): add relay-block harvest logic and floored poll cadence (MAG-2159) Implements relay-block harvesting into per-endpoint observation stores, keyed on NetworkAddress with Source=Relay, and ensures no impact on poll-health fields. Introduces a flat, floored poll cadence (avgBlockTime/2) for per-endpoint trackers, replacing adaptive tiers while keeping the global tracker unchanged. Adds comprehensive tests to validate behavior under both floored and legacy cadences. * fix(mag-2159): harden relay tip-harvest — eligibility, Solana slot, flat cadence, lifecycle Post-review hardening of the per-endpoint chain-tracker harvest path (MAG-2159 review findings 1-6): 1. Tip eligibility: harvest a relay block only when it is the current tip — EVM/generic requires RequestedBlock()==LATEST_BLOCK and reply.LatestBlock>0, so historical getBlockByNumber(N)/getBlockByHash/receipt/logs no longer poison the tip. New tipBlockFromRelay() gate. 2. Solana context.slot: chain-gated extractSolanaContextSlot() harvests result.context.slot from successful, non-batch JSON-RPC replies; rejects error/null/batch/coincidental-slot/non-positive. No global slot interpretation. 3. Startup scheduling: flat trackers arm the periodic timer only after init completes (first poll >= FlatPollInterval after the final attempt); init retries are ctx-cancellable spaced so failures cannot burst quota. 4. Flat cadence: FlatPollInterval (renamed from PollIntervalFloor) is the mode switch — >0 selects a fixed flat scheduler that ignores adaptive tiers and block-gap recalibration; 0 keeps legacy adaptive (global tracker only). 5. Lifecycle safety: RecordRelayObservation takes an endpoint generation and records only when the live generation exists and matches, under obsMu — safe across RemoveTracker, same-URL replacement, and Stop. 6. Operator config: --chain-tracker-polling-multiplier help notes it no longer affects per-endpoint trackers, with a runtime warning when set. Tests: real-parser EVM eligibility (eth_blockNumber -> LATEST_BLOCK harvested, concrete block rejected), Solana slot table, generation-safety suite, and real-StartAndServe startup-cadence timing tests. Note: full `go test -race ./protocol/chaintracker` still fails on pre-existing legacy block-gap races (smallestBlockGap/updatePollingTimeBasedOnBlockGap vs AddBlockGap); the flat path does not touch those and does not widen the race. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(mag-2159): traffic-gate the per-endpoint poll on fresh relay observations (Pass 2 B-D2) A per-endpoint dedicated poll now borrows a fresh relay-harvested tip instead of spending an upstream call — the core quota win Topic B exists to deliver. - EndpointPoller gains a nil-safe relayGate closure. FetchLatestBlockNum consults it BEFORE any upstream work: on a hit it returns the borrowed tip with no upstream round-trip and no poll observation (no poll happened, so poll-health stays owned by the poll path). It still returns the block so the ChainTracker advances its atomic latest — which GetSyncGap / consistency read (borrow-and-return, not skip-entirely; those readers are fed only by the poll loop, so skipping would stale the sync gap). - EndpointMonitor.freshRelayTip is the predicate: borrowable only when the latest observation is Source == Relay (anti-circularity — a poll-sourced obs must never suppress the next poll, or one poll throttles all polls) AND younger than relayGateFreshness (the gate freshness threshold, default averageBlockTime ~ one block stale, 2x the flat poll interval). Wired per endpoint in GetOrCreateTracker. Honest quota claim: eliminates the per-tick FetchLatestBlockNum; on a genuine tip advance the borrowed block still trips one FetchBlockHashByNum upstream (fork tracking, chain_tracker.go:435) — correct, not waste. Deferred (documented in freshRelayTip): suppression is unbounded, safe today because no decision-plane consumer reads poll-health (the gate is GetObservation's first production reader). A busy-endpoint poll ceiling (forced real poll every N intervals to catch a cached/lying upstream) lands with the Topic-D probe. Tests (protocol/endpointstate/traffic_gate_test.go): anti-circularity (relay suppresses, poll does not), freshness boundary, borrow-without-upstream-call via a counting connection (+ tracker atomic advances, no observation recorded), and nil-gate-always-polls. Full endpointstate suite green under -race. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(mag-2159): consolidate chain-tracker polling knobs into one floored control (Pass 2 B4) Collapse the multiplier soup into a single relief control and retire the dead knobs the per-endpoint flat cadence + Pass-1 hardening left behind. Removed: - config.PollingTimeMultiplier (per-tracker override) — set by NO caller in prod or test, so it was dead config. Its branch in newCustomChainTracker is gone; the multiplier now resolves solely via EffectivePollingMultiplier (flag override, else the built-in 16). - clampPollingMultiplier (the in-tracker divide-by-zero clamp) — provably redundant once the dead field is gone: the only surviving feeders of the adaptive tiers (base/(multiplier/4)) are the default 16 and the flag, which is range-validated to [4,16] before assignment, so every value is >= 4 by construction. Its direct unit test goes with it. - endpointstate.MinPollingInterval — defined, never read. Kept (re-documented): MinPollingTimeMultiplier survives as the single named [4,16] lower bound. The flag validation in rpcsmartrouter is now the SOLE divide-by-zero guard (commented as such) rather than a redundant first layer ahead of an in-tracker clamp. Scope: zero runtime cadence change. The global tracker still uses the multiplier-adaptive path; flattening it to a fixed interval is deferred to Topic C (its readers — fallback latest-block, syncGap, consistency — are not harvest-fed yet, so changing its cadence now would risk starving them). Runtime log + flag help refreshed to "affects only the global tracker" (the interim "until knob consolidation" wording is now stale). Verify: gofmt; go vet; go build ./...; go test ./protocol/chaintracker ./protocol/endpointstate ./protocol/rpcsmartrouter — all green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(mag-2159): relocate traffic gate to ChainTracker level — gate Solana + the fork-hash fetch (review F1/F2) The Pass-2 gate sat on EndpointPoller.FetchLatestBlockNum, which two things bypass: SVMChainTracker (it polls via CustomMessage, never through EndpointPoller — so Solana, the primary quota-sensitive chain, was never gated), and forkChanged's per-tick FetchBlockHashByNum (which runs even when the block is unchanged, so the old gate still paid one hash fetch per tick). Move the gate up to ChainTracker.fetchAllPreviousBlocksIfNecessary, ABOVE the generic/SVM iChainFetcherWrapper split, so it suppresses both families and the whole cycle: - ChainTrackerConfig.RelayTipFresh (set by EndpointMonitor from freshRelayTip) reports whether a fresh Relay-sourced observation covers the endpoint. - When it does, the cycle is SKIPPED entirely — no FetchLatestBlockNum and no fork-check hash fetch: genuinely zero upstream (F2). Skip-only, not borrow-and-return: a skip touches nothing, so SVM seenBlock/slot/hash caches stay correct when a relay supplies a slot but no hash (F1). - Bounded by MaxRelaySkipsBeforePoll (default 4): after N consecutive skips one real poll is forced for independent fork/liveness verification (the busy-endpoint ceiling). Tracker-atomic staleness is capped at ~2 blocks, well inside the 10-block EndpointLagThreshold consistency pre-validation uses. - The gate fires on relay-observation freshness alone (Source==Relay anti-circularity); the global tracker leaves RelayTipFresh nil and never skips. The old EndpointPoller.relayGate field + its borrow-and-return logic are removed. Tests: white-box full-cycle upstream-call counts for fresh-unchanged / fresh-advanced / stale / no-relay across BOTH the EVM and SVM wrappers (chain_tracker_gate_test.go, internal package, deterministic — no timer); an end-to-end SOLANA monitor test counting real getLatestBlockhash upstream calls; bounded-verification and nil-gate (global tracker) guards. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(mag-2159): gate endpoint tip-state on eligibility + capture generation before dispatch (review F3/F4) F4 — historical responses poisoned the endpoint tip. The relay success path unconditionally wrote every positive Reply.LatestBlock to targetEndpoint.LatestBlock, LastBlockUpdate, the global estimator, and the latest-block metric — including historical eth_getBlockByNumber(N) responses, whose block is NOT the current tip. Extract harvestAndUpdateTipFromRelay, which gates ALL of those tip-state writes on tipBlockFromRelay eligibility (the same rule the observation store uses). The block this response serviced (latestServicedBlock, for OnSessionDone) is kept separate and ungated. A historical response now moves none of the tip state; the relay path carries the tip for busy endpoints (tip-eligible only), backing the skip-only gate. F3 — generation captured after the relay. endpointObservationGeneration was read at harvest time, so an old in-flight relay could complete after a same-URL tracker replacement, read the replacement's live generation, and be accepted as the new tracker's. Ensure the tracker and capture the generation BEFORE relayInnerDirect, then thread the captured value to the harvest — so a relay that began against one incarnation carries that incarnation's generation and is rejected after replacement. Tests: real-parser integration through harvestAndUpdateTipFromRelay — a historical eth_getBlockByNumber(N) moves no tip state / no Relay observation, while eth_blockNumber moves all of it; a real-incarnation remove/recreate proves the captured-generation harvest is rejected after same-URL replacement and the live generation is accepted. (The harvest/eligibility tests switch to the lightweight real parser to avoid the CreateChainLibMocks SetGlobalLoggingLevel data race under -race.) Note: no full live-goroutine concurrency test for F3 — there is no successful-relay harness for sendRelayToDirectEndpoints (only a failure-path driver); the capture-before-dispatch ordering is a straight-line property verified by reading the call site plus the real-incarnation gate test. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(mag-2160): add per-chain ChainState tip with internal consensus (Topic C, phase 1) First phase of Topic C: the per-chain "current chain head" that will replace the three unreconciled latest-block sources (global ChainTracker, relay estimator, monotonic atomic). This commit adds the type + consensus in isolation — no wiring, no read-site migration, no deletions yet (those are the high-blast-radius later phases). protocol/chainstate.ChainState (built in-tree per T1, referencing lava's protocol/chainstate, not importing it): - SetLatestBlock: cheap monotonic + outlier-guarded write the relay-harvest and poll paths will call. Raises the tip monotonically; when a consensus baseline exists, rejects a block > baseline+OutlierThreshold as an anti-lie outlier. The tip only goes DOWN via Recompute, never a write. - Recompute: pulls a snapshot of per-endpoint BlockObservations and computes the strict-majority consensus baseline ITSELF (T2 inverted — data plane owns consensus, no external SetMajorityBaseline), then realigns the tip down when a fresh majority sits > OutlierThreshold below it. Consensus rules (locked): >50%, min-2 agreeing, dedup by URL, fresh-only, bucketed by BucketWidth. A strict majority is unique, so map-iteration order is deterministic. - GetLatestBlock / GetLatestBlockWithTime: TTL-fresh reads (T4); a stale tip reports not-found rather than freezing. - Decoupled via the BlockObservation DTO (no endpointstate import → no cycle); consensus is a pure function; clock injectable for deterministic TTL tests. - Proposed-default constants (tunable): BucketWidth=2, OutlierThreshold=100, StalenessWindow/TTL = DefaultStalenessMultiplier×avgBlockTime (floored), via DefaultConfig. Tests: monotonic + outlier guard (only with a baseline), TTL expiry + paired read, consensus cardinality (1/2/3-endpoint), freshness + dedup-by-URL, downward realignment + recovery, single-endpoint guard-off. Green under -race. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(mag-2160): wire observations into ChainState tip + consensus tick (Topic C, phase 2) Additive wiring — the legacy global tracker / estimator / atomic still stand and still serve all reads; this phase only feeds the new ChainState so phase 3 can migrate the reads onto it. - EndpointMonitor gains an OnTipObservation(block) hook, fired for every ACCEPTED positive block from the poll path (recordPollObservation) and the relay-harvest path (RecordRelayObservation). It is fired AFTER obsMu is released (registered before the unlock defer, so LIFO runs it last) so the ChainState lock never nests inside the observation lock. Rejected writes — failed poll, stale generation, non-positive or stale block — do not fire it. - EndpointMonitor.SnapshotObservations() returns a copy of all records under one read lock, for the consensus pull. - rpcsmartrouter constructs a per-chain chainstate.ChainState (DefaultConfig from the spec avgBlockTime), wires OnTipObservation -> ChainState.SetLatestBlock (the cheap monotonic tip), and runs runChainStateConsensusLoop: a ticker (avgBlockTime, floored at 1s) that pulls SnapshotObservations and calls ChainState.Recompute. The snapshot is taken under the monitor lock and released before Recompute takes the ChainState lock — the two locks never nest, no deadlock. Tests: OnTipObservation fires only for accepted positive poll/relay blocks (not failed/stale/rejected); SnapshotObservations returns an independent copy; and an end-to-end integration (real monitor + real ChainState) proving agreeing relay observations feed the monotonic tip, a recompute establishes the strict-majority baseline, and an anti-lie outlier is rejected. Green under -race. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(mag-2160): migrate the 3 latest-block read sites onto ChainState (Topic C, phase 3) Point every "what block is the chain at" read at the new per-chain ChainState tip instead of the global tracker. The global tracker is now constructed but UNREAD — phase 4 deletes it. - Site A getLatestBlock(): the router-wide tip (archive routing, cache finalization) is ChainState.GetLatestBlock() (TTL-fresh, consensus-guarded), replacing the global-tracker → estimator → atomic ladder. The monotonic latestBlockHeight atomic stays as the cold-start fallback so getLatestBlock never returns 0 in the bootstrap window before the first observation. - Site B (Reply.LatestBlock fallback): when the response carried no block, stamp THIS endpoint's own observed latest (endpointChainTrackerManager.GetLatestBlockNum for the endpoint URL) instead of the global tracker's provider[0] value — fixes the cross-attribution (S6) where one node's tip was stamped onto another endpoint's QoS/seenBlock. - Site C (syncGap): the chain tip is ChainState.GetLatestBlockWithTime(), so every endpoint is judged against the real consensus chain head rather than a single node (fixes the syncGap bias S5). The this-endpoint half stays the endpoint's own latest. The estimator now has no readers (Site A was its only one); it and the global tracker are removed in phase 4. Full rpcsmartrouter suite green (happy-path parity). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(mag-2160): delete the global ChainTracker + retire the estimator (Topic C, phase 4) With all three reads migrated to ChainState (phase 3), the global tracker and the estimator have no readers — remove them. Global ChainTracker (S1/S2/S5/S6/S10 — single point of trust bound to healthyStaticProviders[0], no failover/recovery/anti-lie): - Delete the construction + fire-and-forget StartAndServe goroutine in rpcsmartrouter.go (CreateSmartRouterEndpoint). - Drop the chainTracker field on RPCSmartRouterServer, the ServeRPCRequests ctor param, and the field assignment; remove the now-unused chaintracker import. - One fewer poller per pod. Estimator (relaycore/latest_block_estimator.go) — its only reader was Site A, now ChainState: - Delete the file + its test, the latestBlockEstimator field, and the Record call in updateLatestBlockHeight. The monotonic latestBlockHeight atomic is KEPT, but only as the cold-start fallback for getLatestBlock (so it never returns 0 in the bootstrap window before the first observation feeds ChainState). The chaintracker PACKAGE stays — the per-endpoint trackers in endpointstate use it as the generic poll engine. go build ./... green; relaycore + rpcsmartrouter suites green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(mag-2160): delete the dead gRPC IChainTracker service (Topic C, phase 5) The chaintracker gRPC service (chain_tracker_grpc.go + chain_tracker_service.go) exposed IChainTracker over gRPC. Nothing in-tree consumed ChainTrackerService / RegisterChainTrackerServiceServer / NewChainTrackerService outside the package, and the only thing that registered it was ChainTracker.serve() — which only ran when a tracker set a server address. No tracker ever did (the global tracker, now removed in phase 4, didn't set one either), so the whole path was dead. - Delete both gRPC files. - Remove ChainTracker.serve() entirely (it existed solely to host that service) and simplify StartAndServe to just start the poll loop. Drop the now-dead serverAddress field, the ServerAddress config knob, and the 7 imports that only serve() used (net/http, os, os/signal, grpcweb, http2, h2c, grpc). Zero behavior change — serve() was always a no-op (server address always empty). The chaintracker PACKAGE stays as the generic per-endpoint poll engine. go build ./... green; chaintracker suite green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(chainstate): resolve MAG-2160 review findings (tip/consensus/glue) Address the 9-finding review of Topic C (per-chain ChainState tip + global-tracker removal). The algorithm was sound; the bugs were in the rpcsmartrouter glue wiring old read sites onto the new APIs. ChainState internals (F2/F3/F4/F8): - F2: split GetLatestBlock (optimistic observed tip) from GetConsensusBaseline (strict-majority, TTL-gated via baselineAt) so sync scoring no longer measures endpoints against a lone optimistic reporter's tip. - F3: replace fixed block/BucketWidth buckets with distance-aware sliding-window clustering so two endpoints within BucketWidth always agree (1001/1002 no longer split across a boundary). - F4: track lastObservedAt and refresh it on equal-block confirmations so a live-but-stable tip does not expire by TTL while endpoints keep reporting it. - F8: floor a zero StalenessWindow at minStalenessWindow in New/NewWithClock so expiry can never be disabled. - Add sticky Initialized() and a NewWithClock clock seam. rpcsmartrouter glue (F1/F5/F6 + Site C): - F1: getLatestBlock falls back to the latestBlockHeight atomic only on genuine cold start (!Initialized); a TTL-expired tip returns unknown rather than reviving a frozen value. Remove the non-tip-eligible cache-hit atomic writer. - F5: recomputeChainStateConsensus no longer early-returns on an empty snapshot, so Recompute(empty) clears a stale baseline. - F6: Site B reads the freshest per-endpoint observation (GetObservation) instead of the MAG-2159-gated tracker atomic (GetLatestBlockNum). - Site C: syncGap scores against GetConsensusBaseline and stays 0 when no fresh majority exists. Dead polling-multiplier flag (F7): - Remove --chain-tracker-polling-multiplier, PollingTimeMultiplierOverride, EffectivePollingMultiplier, MinPollingTimeMultiplier, and startup logging. The adaptive engine branch stays, using the fixed built-in MostFrequentPollingMultiplier. Tests: focused integration tests for every corrected rpcss seam (chainstate_glue_test.go) plus ChainState unit tests for clustering boundary, equal-block freshness, empty-snapshot clear, observed-vs-consensus, and config defaults. Clean up stale "global estimator" comments. Verification: gofmt clean; go build ./... green; go test + go vet across chainstate/endpointstate/chaintracker/rpcsmartrouter pass; go test -race on chainstate/endpointstate pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(qos): Topic E — QoS + endpoint.Enabled contract (precedes MAG-2161/D) Build the contract both the prober (Topic D) and the relay path honor, ahead of D so the prober targets a settled API. Unit-testable without a running prober. QoS sync reference → consensus baseline (MAG-2160 Finding 2): - ProviderOptimizer gains a syncReference seam (SetSyncReferenceGetter): BOTH the relay and probe sync dimensions measure lag against ChainState's consensus baseline instead of max-block-across-providers (which one fast/lying reporter could inflate, dinging the whole pod). Falls back to the legacy reference when there is no fresh majority (cold start / single-endpoint). - chainstate.GetConsensusBaselineWithTime exposes the baseline + its timestamp for the lag calc. Wired live per chain interface in ServeRPCRequests. Probe QoS feed + fraction-healthy availability: - New ProviderOptimizer.AppendProbeData(provider, availability, latency, syncBlock, healthy) feeds all three dimensions at ProbeUpdateWeight. - AvailabilityScoreStore relaxed from strict {0,1} to [0,1] so a provider's fraction-of-endpoints-healthy sample decays the score (partial degradation). Provider aggregation (rule E2): - probing.AggregateProviderSample collapses per-endpoint verdicts into ONE sample/provider/cycle: availability = healthy/total, latency = min over healthy, block = max over healthy (no endpoint-count EWMA bias). endpoint.Enabled state machine: - Endpoint.RecordProbeVerdict(healthy, K) is the probe's proactive re-enable — acts ONLY on disabled endpoints (never resets an enabled endpoint's relay-driven refusal climb) and re-enables after K consecutive healthy cycles (default probing.DefaultProbeReEnableHysteresis = 3). Relay path stays the fast disabler (MarkUnhealthy at 50). Probe-driven disable deferred. Tests cover every acceptance item: 4:1 relay:probe weighting, provider aggregation, the Enabled state machine (incl. the anti-coupling guard), the sync seam + fallback, and fraction-healthy decay. Verification: gofmt clean; go build ./... green; go vet + go test across chainstate/lavasession/probing/provideroptimizer/rpcsmartrouter/utils-score pass; go test -race on lavasession/provideroptimizer pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(probing): Topic D — proactive health prober (MAG-2161) Add the per-chain proactive health prober: on its own cadence (defaultProbeCadence = 5s) it reads stored per-endpoint telemetry (Topic A) plus the consensus baseline (Topic C) — making ZERO upstream calls — renders a health verdict for EVERY direct-RPC endpoint (regular AND backup), proactively re-enables recovered endpoints (the O1 win: recovery in seconds, not at the 15-min epoch), and feeds ONE aggregated QoS sample per provider (Topic E rule E2). Always-on; the legacy synthetic direct-RPC probe feed is gated off for static providers. Prober (rpcsmartrouter): - runProbeLoop / runProbeCycle / runProbeCycleCore — the loop, the live-deps adapter, and a pure testable core. - Baseline sourced from chainState.GetConsensusBaseline() (strict- majority), NOT the observed max-across-providers tip — closes the door on Finding 2 reincarnating in the probe path. - Compile-time guard: var _ probeQoSAppender = (*provideroptimizer.ProviderOptimizer)(nil) so the optional-capability type assertion can't silently nil on signature drift. Verdict engine (probing): - verdict.go: RenderEndpointVerdict (alive = fresh & non-zero tip; keeping-up = within LagToleranceBlocks of baseline) + DefaultVerdictConfig. - contract.go: per-dimension has-flags (HasLatency/HasBlock) so a healthy-but-unpolled endpoint never feeds a fake 0 latency/block; AggregateProviderSample = fraction-healthy availability, min latency, max block over healthy endpoints. QoS feed (provideroptimizer): - AppendProbeData gains per-dimension hasLatency/hasSync flags; availability always fed (probe weight 0.25 → 4:1 relay:probe). Tests: probe_loop_test.go (re-enable, stale-stays-disabled, one-sample- per-provider, backups-covered, no-observation-unhealthy), verdict_test.go, contract_test.go. Full suite green; build + gofmt clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(rpcsmartrouter): resolve MAG-2160/2161 review findings + harden concurrency Post-merge review of the ChainTracker/Probing redesign (Topics C/D/E) plus the data-race hardening surfaced while satisfying the review's go test -race gate. Review findings (F1-F7): - F1: the probe re-enables a disabled endpoint only from DISTINCT successful polls produced strictly after its disable instant (edge-triggered disabledAt, stamped at BOTH disable write-sites). A pre-disable / relay-fed observation can never re-enable. - F2: a genuine re-enable also restores the provider to routing (csm.RestoreRecoveredProvider) — regular -> validAddresses, backup -> out of blockedBackupProviders; overlap-safe and idempotent. - F3: endpoint.Enabled reads go through the synchronized Endpoint.IsEnabled(); the synthetic direct-RPC probe is removed for static providers. - F4: the QoS sync reference is a per-sample, per-interface SyncReference resolved from THIS interface's ChainState — replacing the single mutable optimizer getter that the last interface to start would overwrite. - F5: with no fresh consensus majority the sync update is OMITTED, never the legacy max-block-across-providers reference. - F6: dedicated, validated --probe-loop-interval (ProbeLoopInterval) clock. - F7: synthetic direct-RPC liveness feed dropped for static providers. Concurrency hardening (go test -race green for lavasession + rpcsmartrouter): - updateMetricsManager snapshots LatestBlock/RelayNum before its metrics goroutine, which outlived OnSessionDone's session Free (real production race). - rpcclient handler.handleResponse sets op.err before op.resp<-msg for subscriptions, which was racing Subscribe's wait() (real production race). - score_store getters take the type's existing RWMutex; rand.protocolRand -> atomic.Pointer; DebugProbes -> atomic global no longer bound at flag registration; chain_message drops its racy lazy request-hash cache (the relay processor runs concurrent attempts on one message); SetGlobalLoggingLevel test mock guarded by sync.Once; CSM unlocked log slice-reads and test-side shared-slice reads synchronized; mock WS server writes serialized. - clock_injection decay test: deterministic ristretto build-up (was flaky under -race load because the async cache dropped accumulations) and corrected the warp_10h threshold to the true 0.122%. Verified: go build, go vet, gofmt clean; non-race go test green; go test -race green for lavasession, rpcsmartrouter, provideroptimizer, chainstate, endpointstate, score, rand. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(chainlib): resolve pre-existing data races + failures in chainproxy/listeners/tracing Pre-existing failures surfaced by the full go test -race / make test sweep, in packages unrelated to the redesign. Two were real production bugs newly fatal under stricter dependency versions. - dyncodec/remote_relayer + chainproxy connector test: gRPC method names were missing the leading slash ("/pkg.Service/Method"). Newer gRPC rejects the slash-less form ("Unimplemented: malformed method name") and returns an empty response. Real bug in the relayer reflection path. - chainproxy connector: addClientsAsynchronously read len(freeClients) without the lock; the tests read freeClients/usedClients directly instead of via the locked numberOf* accessors. - WS chain listeners (jsonRPC / tendermint / rest / grpc): listeningAddress was a plain string written by a goroutine and read by GetListeningAddress -> atomic.Pointer[string]. - consumer_websocket_manager graceful shutdown: the writer goroutine called Close() on the gofiber conn, which unblocks the handler and lets gofiber recycle the conn (fasthttp pooling) while the writer is still inside Close() (use-after-recycle). Now SetReadDeadline unblocks the read loop so the conn's owner performs the single close. - tracing: corrected a stale test expectation — the OTel SDK no longer collapses a 1.0 sampling ratio to AlwaysOnSampler; TraceIDRatioBased{1} is spec-compliant and samples 100%. Verified: go test -race green for chainlib, chainproxy, rpcclient, tracing; the full non-race make test now passes with zero failures. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(chainlib): route WS idle-close frame through the single writer goroutine The idle-timeout path in the rate-limit goroutine called websocketConn.WriteMessage directly, bypassing the websocketConnWriteChan single-writer goroutine — a second concurrent writer, which gofiber/gorilla forbids (data race) and which contradicts the single-writer discipline the surrounding code documents. Route it through sendWS like every other write; the client closing on the frame unblocks the read loop as before. (Found in review of PR #143; the GracefulShutdown tests don't exercise the idle path so -race hadn't flagged it.) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(chainproxy): read connector.usedClients under the lock, not via atomic usedClients (int64) was written/read as a plain field under connector.lock everywhere (GetRpc/ReturnRpc/Close) EXCEPT numberOfUsedClients(), which used atomic.LoadInt64 without the lock. A mutex-protected write and a lone atomic read are different happens-before domains, so the async addClientsAsynchronouslyGrpc goroutine's read raced GetRpc's `usedClients++` (a rare ~1/21 -race flake in the chainlib suite, captured via a sustained hunt). Read under the RLock to match numberOfFreeClients(); connector.closed keeps its (consistent) atomic access. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(chaintracker): five review findings — tip tags, baseline age, sync gen, SyncBlock floor P1-#1 dyncodec: keep reflection callback/parser paths slash-less; the leading slash is added only at the grpc.ClientConn.Invoke boundary. Regression test asserts both the slash-less callback and the slashed invoke. P1-#2 tip eligibility: harvest a relay's block as the current tip only when the method's spec function tag is GET_BLOCKNUM, or GET_BLOCK_BY_NUM requesting LATEST. RequestedBlock==LATEST and GetUsedDefaultValue (non-deterministic for eth_blockNumber) cannot discriminate receipt/by-hash historical responses. All downstream tip writes stay gated behind the single harvest chokepoint. P2-#3 chainstate baseline: split the overloaded baselineAt into baselineAt (TTL freshness, refreshed every confirming Recompute) and baselineSince (establishment time, preserved while the block is unchanged, reset on forward/downward/regained change). GetConsensusBaselineWithTime returns baselineSince for sync-lag so a stuck baseline accrues real age while a reconfirmed one never expires. P2-#4 tracker generation: register the tracker and allocate its observation generation synchronously before the relay dispatch captures it. GetOrCreateTracker does no network I/O (the poll loop stays async), so an early relay no longer captures generation 0 and silently drops its harvested tip. P2-#5 optimizer SyncBlock: serialize each provider's QoS read-modify-write with a striped lock and hold the authoritative monotonic SyncBlock floor under it. The ristretto cache is async/eventually-consistent and cannot enforce monotonicity, so a stale read (incl. the legacy availability-only writer) could regress the block; the floor guarantees SyncBlock never decreases. ResetState clears the floors. Verification: gofmt, go build ./..., go vet ./..., make test (all green), and targeted -race for dyncodec, chainstate, endpointstate, provideroptimizer, rpcsmartrouter. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(endpointstate): tolerate an empty GET_BLOCKNUM template for gRPC The per-endpoint poller hard-failed when a GET_BLOCKNUM parse directive had an empty function_template, returning "GET_BLOCKNUM missing function template apiInterface=grpc" and retrying ChainTracker creation forever. But for gRPC the spec's api_name carries the method and the function_template is only the request PAYLOAD — which is legitimately empty for a no-argument call such as cosmos Service/GetLatestBlock. So an empty gRPC template means "empty body", not "undefined method"; the relay path already serves these calls with an empty body, only the poll path rejected them. Extract blockNumRequestBody(apiInterface, template): gRPC resolves an empty template to "{}" (the canonical empty JSON message the codec marshals to an empty proto); REST (needs a path) and Tendermint (needs a method) still hard-error on an empty template, since polling those without one would send garbage. Unit-tested across all four interfaces. Surfaced by the chaintracker-redesign: the previous single global ChainTracker bound to one provider, so a non-gRPC primary left this gRPC gap latent; per-endpoint pollers now exercise every interface and hit it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(endpointstate): complete gRPC per-endpoint poll support (method header + response descriptor) Two layers kept the per-endpoint gRPC ChainTracker poll from completing — both because the poll path was built for text protocols (REST/JSON-RPC/Tendermint) and gRPC differs structurally. Following the empty-template fix: 1. Method header: gRPC carries the method PATH in an x-grpc-method header, not the URL or body. EndpointPoller.sendRawRequest dropped the apiName, lumping gRPC into the generic JSON-POST branch, so every poll failed "gRPC method path not provided in headers". Set lavasession.GRPCMethodHeader for the gRPC interface (the relay path already does). 2. Response descriptor: a gRPC response is binary protobuf; decoding it needs the protobuf method descriptor. CraftChainMessage can't resolve one (reflection needs a live connection), so the crafted message had none and parsing failed "does not have a methodDescriptor set in grpcMessage". The DirectRPCConnection resolves+caches the descriptor during the send and exposes it via GetCachedMethodDescriptor; new chainlib.HydrateGrpcResponseParsing wires it (and a grpcurl FormatJSON formatter — the SAME one the relay path uses, so camelCase field names match spec result_parsing parser_args) onto the crafted message before FormatResponseForParsing. Applied to BOTH spec-driven poll paths via a shared hydrateGrpcChainMessage helper: FetchLatestBlockNum (GET_BLOCKNUM) and fetchSingleBlockHash (GET_BLOCK_BY_NUM, the fork-detection hash fetch that runs when blocksToSave > 1). Verified live: the gRPC ChainTracker now polls and advances block-to-block, tracking the same tip as the REST/tendermintrpc interfaces. Surfaced by the chaintracker-redesign: per-endpoint pollers now exercise every interface, where the single global tracker bound to one provider did not. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(endpointtip): consolidate per-endpoint tip into one shared store The per-endpoint observed block tip was written to two independent places: Endpoint.LatestBlock (an atomic on the lavasession struct) and the EndpointObservation block triple (in endpointstate's observation map). Worse, BOTH writes to Endpoint.LatestBlock were ungated — the poll-path write in NewLatestCallback and the relay-path write in harvestAndUpdateTipFromRelay bypassed the generation + time-monotonic guards that the observation triple enforces — so a stale or replaced-tracker observation could move the atomic while being correctly rejected by the gated store, leaving the two copies diverged. Introduce protocol/endpointtip: a stdlib-only leaf package holding the single source of truth for the tip triple {Block, ObservedAt, Source}, keyed by chain|apiInterface|url. Being a leaf, it is importable by BOTH the high layer (endpointstate observers) and the low layer (lavasession QoS reader) without the import cycle that previously forced the value to be pushed down onto Endpoint. - endpointstate/observation.go: recordPollObservation and RecordRelayObservation keep the generation gate and write the triple to the store (the time-monotonic guard moves into Store.Set, semantics preserved exactly — time-, not block-monotonic). The observation map now stores only poll-health; GetObservation/SnapshotObservations compose the triple back from the store at read time (delegation, not a second copy). Lock order is obsMu → store lock everywhere; onTipObservation still fires after obsMu is released. - Delete Endpoint.LatestBlock + LastBlockUpdate (the latter had no readers) and the two ungated writes; the three readers (consumer_session_manager syncBlock, rpcsmartrouter syncGap + consistency fallback) read the store via a nil-safe helper. - Monitor RemoveTracker/Stop evict the chain's keys so the global map stays bounded as endpoints churn. Tests migrated to seed/read the store; build, vet, and -race all green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(chainstate): derive outlier threshold from chain block time The anti-lie / downward-realignment outlier threshold was a fixed 100-block constant. The threshold is denominated in blocks, but the quantity it bounds — how far the optimistic tip may legitimately lead consensus before a write is treated as a lie — is denominated in time. So a fixed block count meant ~240x different poison tolerance across chains: ~100min on a 60s chain, ~20min on Ethereum, ~40s on Solana. On slow chains the guard was uselessly loose. Derive it in DefaultConfig from the chain's average block time (already passed in for the staleness window, so no new plumbing): OutlierThreshold = clamp(round(OutlierTimeBudget / avgBlockTime), floor, ceil) OutlierTimeBudget = 1200s floor = 32 ceil = 512 - 1200s keeps a 12s chain (Ethereum) at the historical 100 → backward compatible; the only chain whose behavior shifts is one that wasn't getting the right semantics before. - floor 32 keeps slow chains above the ~10-20-block legitimate lead (staleness window in blocks + propagation) so the guard never false-rejects an honest advance. - ceil 512 is the primary fast-chain safety knob: without it, anchoring ETH to 100 derives ~3000 Solana slots, a guard so loose a liar could hold the tip 3000 slots ahead. - avgBlockTime <= 0 (spec without average_block_time) falls back to the fixed DefaultOutlierThreshold (100), never 0. Both use-sites (SetLatestBlock write guard, Recompute downward realign) stay on the one cfg.OutlierThreshold. Adds a table test across 60s/12s/6s/0.4s/0.25s chains; corrects the one existing assertion that baked in the old fixed value. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(smart-router/debug): add endpoint-state, chain-state, provider-routing read endpoints (MAG-2202) Three read-only GET debug endpoints so the MAG-2201 automation suite can read ChainTracker/Probing-redesign state from the wire instead of scraping logs. Registered only under --debug-address; GET-only (any other method -> 405); nil-router safe (returns []). Each endpoint returns a flat array of self-describing records (ChainID + ApiInterface, plus NetworkAddress for endpoint-state) — flat keys, no object nesting, deterministically sorted — honoring the ticket's response-shape rules: - /debug/endpoint-state: per-endpoint poll-health + enable/recovery, joining EndpointMonitor.SnapshotObservations with a new Endpoint.HealthSnapshot (Enabled/DisabledAt/ConsecutiveHealthyProbes/LastSuccessfulPoll/LatestBlock/ ObservedAt/Source) by NetworkAddress. - /debug/chain-state: raw, non-TTL-gated ChainState.DebugSnapshot (ObservedTip/ConsensusBaseline/HasBaseline/Initialized + LastObservedAt/ BaselineSince) so tests can assert TTL expiry / realignment / baseline clearing. - /debug/provider-routing: ConsumerSessionManager.ProviderRoutingSnapshot (ValidAddresses/CurrentlyBlockedProviderAddresses/BlockedBackupProviders). Keys mirror the Go identifiers; timestamps RFC3339 (empty string for zero); address slices non-nil so empties encode as [] not null. ChainID/ApiInterface are sourced from listenEndpoint / csm.RPCEndpoint(), never by splitting the key. Backing state already exists on chaintracker-redesign, so this only adds read accessors + handlers. Stacked on chaintracker-redesign (PR #143); merges after it. Endpoint 4 (/debug/probe-loop) follows in a separate PR — it needs counter instrumentation on the existing runProbeLoop, not new read plumbing. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(smart-router/debug): add /debug/probe-loop prober-cycle telemetry (MAG-2202 endpoint 4) The fourth read-only debug endpoint, instrumenting the existing Topic-D prober (runProbeLoop) with per-cycle counters — no behavior change, just observation. - runProbeCycleCore now returns (scored, reEnabled, syncOmitted): endpoints scored, endpoints re-enabled this cycle (F1), and providers whose QoS sample fed no sync evidence (F5: no fresh consensus baseline). Existing callers ignore the returns; behavior is unchanged. - runProbeCycle times each cycle and records the snapshot onto a per-server probeLoopStats (mutex-guarded, off the data plane); runProbeLoop publishes the configured --probe-loop-interval cadence before the first tick. - GET /debug/probe-loop returns a flat array of per-chain records (ChainID + ApiInterface): CycleIntervalMs, CyclesCompleted, LastCycleStartedAt (RFC3339), LastCycleDurationMs, EndpointsScored, ReEnabledCount, SyncOmittedCount. Reuses writeDebugRows / debugTimeRFC3339 from the endpoints-1-3 PR. GET-only (405 otherwise), nil-router safe. Lets the suite verify the prober runs on its own cadence (F6), per-cycle re-enable counting (F1), and the sync-omitted count (F5). Stacked on #155 (endpoints 1-3); retarget to chaintracker-redesign after it merges. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(rpcsmartrouter): gate tip-state writes on relay-observation acceptance Addresses PR #143 review (P1 correctness): a stale relay still mutated tip state after the generation gate rejected it. recordRelayBlockObservation was generation-gated, but the tip-state writes that followed in harvestAndUpdateTipFromRelay (updateLatestBlockHeight's bootstrap atomic + router metric, and the per-endpoint latest-block metric) ran unconditionally. After a same-URL tracker replacement bumped the generation, a stale relay's observation was correctly dropped from the store but still poisoned every other tip-state field. RecordRelayObservation / recordRelayBlockObservation now return whether the write was accepted (passed the generation + monotonic guards), and harvestAndUpdateTipFromRelay early-returns when it was not, gating the remaining writes. Extends the generation-rejection harvest test to assert the bootstrap atomic is not moved by a stale-generation relay (fails without the fix). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(chainstate): break majority-baseline window ties toward the highest Addresses PR #143 review (P2 correctness): computeMajorityBaseline kept only the first max-size window, understating the baseline on ties. The clustering loop used a strict `count > bestCount`, so a later equal-size window never replaced the incumbent. With observations [1000,1002,1004] and BucketWidth 2 there are two overlapping 2-of-3 majority windows ([1000,1002] and [1002,1004], sharing 1002), and the loop returned 1002 — understating the consensus baseline that feeds sync scoring and outlier/realignment decisions. The "strict majority cluster is unique" doc claim only rules out DISJOINT rivals; overlapping equal-size majorities genuinely coexist. Break window-size ties toward the most-advanced window (count == bestCount && blocks[j] > bestMax), so the baseline is never understated, and correct the doc comment accordingly. Adds TestComputeMajorityBaseline_OverlappingTiePrefersHighest, which asserts the staircase resolves to 1004 (fails returning 1002 without the fix). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(rpcsmartrouter): don't charge in-cluster endpoints a sync-gap Addresses PR #143 review (correctness, low severity): because the consensus baseline is the winning cluster's MOST-ADVANCED block, a majority sitting one cluster-step below it was charged a sync-gap against a tip only the fastest cluster member reported (e.g. [1000,1000,1002]/BucketWidth=2 → baseline 1002, the two 1000-nodes charged syncGap 2). Rather than change the baseline element (which would reverse the just-shipped P2 highest-window fix and alter consensus semantics), fix it at the symptom: an endpoint within BucketWidth of the baseline is, by definition, inside the agreeing cluster, so Site C zeroes its sync-gap. The new clamp subsumes the old `syncGap < 0` (endpoint-ahead) case. Exposes ChainState.ConsensusBucketWidth() for the clamp and adds TestSiteC_InClusterEndpointChargedNoSyncGap pinning the arithmetic. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * perf(chaintracker): skip block-gap machinery for flat per-endpoint trackers Addresses PR #143 review (efficiency): flat per-endpoint trackers ran the adaptive block-gap machinery whose result nothing consumes. For a flat tracker (flatPollInterval > 0) computePollInterval returns a fixed cadence and never consults the block-gap sweep, yet start() still fired blockGapTicker (a Percentile+Stability sweep over blockEventsGap) and AddBlockGap still appended per block — pure wasted CPU/allocation multiplied across every direct-RPC endpoint. The reviewer's premise that the result is "discarded" is incomplete: the sweep also feeds updateAverageBlockTimeForRegistrations -> UpdateBlockTime. But RegisterForBlockTimeUpdates has no production call sites (only a test, which uses a NON-flat tracker), so for flat trackers that consumer is empty too and the conclusion holds. So finding-4's "still runs for block-time consumers" rationale does not apply to per-endpoint trackers. Start blockGapTicker only for the global adaptive tracker (nil channel parks the select case for flat trackers) and gate the per-block AddBlockGap append on flatPollInterval == 0. setLatestChangeTime stays unconditional (it still drives the not-updated/emergency path). The global adaptive tracker is unchanged. Adds TestFlatTracker_SkipsBlockGapMachinery (blockEventsGap stays empty for a flat tracker; fails without the gate). The existing adaptive block-time-update test still passes, proving the global path is untouched. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(probing): reuse chainstate staleness multiplier as one source of truth Addresses PR #143 review (cleanup): probing.defaultStalenessMultiplier (10) and chainstate.DefaultStalenessMultiplier (10) independently encoded the same "fresh/alive horizon = N x blockTime" concept, so tuning one would silently diverge the per-endpoint liveness horizon from the per-chain consensus window. chainstate is a stdlib-only leaf package (no import cycle), and the probing comment already noted it "mirrors the chainstate consensus window", so make chainstate.DefaultStalenessMultiplier the single named source: drop the local duplicate and reference the exported constant from probing.DefaultVerdictConfig. The floors stay package-specific (minProbeStaleness 5s vs minStalenessWindow 2s) — only the shared multiplier is unified. Documented the dependency on both ends. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(lavasession): centralize F1 recovery-field resets behind helpers Addresses PR #143 review (cleanup): the recovery-field reset block was copy-pasted across six disable/enable sites, so "every Enabled transition resets the recovery fields" was enforced only by remembering to copy it — a new path that missed one would silently break F1 hysteresis with no compile error. Extract two composed, lock-held helpers on Endpoint: - clearRecoveryStreakLocked() — zeroes the streak (consecutiveHealthyProbes + lastRecoveryPoll), leaving disabledAt; used by both disable sites (after stamping disabledAt) and the enabled-domain reset in RecordProbeVerdict. - clearRecoveryTrackingLocked() — streak + disabledAt; the full clean slate for every Enabled->true transition (RecordProbeVerdict re-enable, ResetHealth, the session reconnect path). This is the named helper the reviewer requested. Deliberately NOT folded in: the invalid-evidence reset in RecordProbeVerdict (the reviewer's listed "319"). It zeroes consecutiveHealthyProbes but PRESERVES lastRecoveryPoll on purpose, so a probe cadence faster than the poll cadence cannot re-count an already-counted poll across a streak break. Routing it through a clear helper would silently break that F1 guard — exactly the hazard the comment warns about — so it stays inline (documented on the helper). Behavior-preserving: same field writes at every site; lock context verified held at both session sites. The lavasession F1/hysteresis, probe-reenable, and recovery suites all still pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(endpointstate): delete dead ValidateEndpointSync / GetSyncGap Addresses PR #143 review (cleanup): both EndpointMonitor.ValidateEndpointSync and GetSyncGap are callerless in production — a repo-wide search finds them only in their own tests. Sync-gap is now computed inline against chainState.GetConsensusBaseline (rpcsmartrouter_server.go:~1520), so these methods are dead. Per CLAUDE.md's "minimise exported surface area", remove them and their tests. GetLatestBlockNum (which they delegated to) stays — it is still called in production (rpcsmartrouter_server.go:1649, 1692). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(chainstate): drop test-only HasConsensusBaseline; use DebugSnapshot Addresses PR #143 review (cleanup): HasConsensusBaseline was exported but had only test callers, and it bypassed the TTL freshness gate — inviting a future caller to read a stale baseline, the exact bug the design removed. It is fully redundant with DebugSnapshot(), which already exposes the same raw, non-TTL (ConsensusBaseline, HasBaseline) pair under the same read lock and IS the telemetry path. Drop the method (minimise exported surface area) and migrate the test callers across chainstate/rpcsmartrouter/endpointstate to DebugSnapshot. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(smart-router): address PR #143 review findings (#1-3, #5, #7-8) Fixes from the PR #143 review (docs/PR-143-REVIEW.md). The top findings share one root: getLatestBlock() now intentionally returns 0 on a TTL-stale tip, and several consumers mishandle the 0. - #1 archive routing: add getLatestBlockBestEffort() (strict tip, else last-known atomic, 0 only at cold-start) for the three getExtensionsFromDirectiveHeaders returns, so a stale window no longer forces every concrete-block request to the archive pool. Cache finalization stays on the strict getLatestBlock(). - #2 jsonRPC eth_call: guard the archive-depth subtraction (parsedBlock>=0 && LatestBlock>ethCallArchiveBlockDepth) so uint64(0)-126 can't underflow and force archive for every eth_call (also covers chains younger than the depth). - #3 averageBlockTime: floor the spec block time once (DefaultAverageBlockTime) and feed the same value to both ChainState and the EndpointMonitor, so the consensus window can't end up shorter than the poll cadence on no-average_block_time specs. - #5 endpoint flap: dampen the poll-healthy/relay-failing flap with a capped, decaying re-enable hysteresis (reEnableAfterK << reenableProbeFlaps, cap 2; a successful relay decays it). Race-safe under e.mu; bounded well below the epoch. - #7 hot path: cache the two tip-defining ApiNames once (sync.Once) so tipBlockFromRelay no longer takes the parser RWLock twice per relay. - #8 cleanup: make endpointstate.ObservationSource a type alias of endpointtip.Source (drop the duplicate enum/String/converter). #4 (cache finalization) and #6 (sub-majority sync) resolved as by-design with hardened comments/rationale; two separate latent items recorded as follow-ups. go build/vet clean; lavasession, endpointstate, chainstate, rpcsmartrouter, chainlib suites green (incl. -race where applicable). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * removed doc * build: bump Go toolchain to 1.26.5 to clear govulncheck GO-2026-5856 The govulncheck CI gate flagged GO-2026-5856 (Encrypted Client Hello privacy leak in crypto/tls), reachable from our TLS-using code paths. It is a Go standard-library advisory fixed in go1.26.5 — no application code change applies. CI pins the scan's Go via go-version-file: go.mod, so bumping the directive from 1.26.4 to 1.26.5 makes setup-go install the patched stdlib and the scan goes clean. Verified locally: govulncheck v1.5.0 built under go1.26.5 reports "No vulnerabilities found" (exit 0); go build ./... passes under 1.26.5. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Took 28 minutes --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Anna Rayer <anna@magmadevs.com>
avitenzer
pushed a commit
that referenced
this pull request
Jul 21, 2026
…lock timestamps (MAG-2307 review) Review follow-up to PR #222 (MAG-2307, merged as c011a80). Addresses two required changes from VicSheCodes and all three Copilot comments. Decoupling (VicSheCodes #1): /debug/time-warp is the optimizer/QoS clock, and the black-box suite runs warp-then-reset-to-0 routinely to reset QoS state -- coupling ChainState to it risked disturbing ChainState on a QoS reset. ChainState's TTL/staleness clock now has its own endpoint, POST /debug/chain-state-time-warp (offset_seconds relative to real time, 0 = real; same finite/>=0/<=24h validation), fully independent. Neither /debug/time-warp nor /debug/reset-all touches ChainState any more. Timestamp writes (VicSheCodes #2 = Copilot chain_state.go:258 and :377): SetLatestBlock and Recompute stored lastObservedAt/baselineAt/baselineSince with the WARPED clock. A write while a warp was active stamped a future time, so clearing the warp left real - future = negative age, reading as fresh forever. Now freshness is EVALUATED against the warped clock but stored timestamps always use the real base clock (cs.now()). Doc fix (Copilot chain_state.go:408): GetConsensusBaselineWithTime returns baselineSince (establishment time), not baselineAt (the internal TTL-freshness stamp, never exposed). Tests: dedicated-endpoint HTTP e2e; /debug/time-warp independence (does not touch ChainState); write-under-warp regressions pinning the real-clock timestamp and no negative-age freshness. Full chainstate (-race) and rpcsmartrouter suites pass; go build ./... and gofmt clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
7 of 9 tasks
nimrod-teich
pushed a commit
that referenced
this pull request
Aug 16, 2026
…2724) A provider's configured name was its routing identity, and the maps keyed by that identity collapsed two entries sharing a name into one. One upstream served every request while the other sat idle, and because the retry skip-list used the same key, setting that name aside after a failure removed both and left nothing to retry against — the customer got an error while a healthy provider stood by. Commit dffc0f2 established that duplicate names must be tracked separately, but only for the startup failed-provider set. Everything downstream was still keyed by name. Two layers: Fail fast. ValidateUniqueProviderNames rejects two providers sharing a name on one chain+api-interface, before any listener binds. That scope is deliberate — a session manager is built per chain+interface and owns its own pairing map, so it is exactly where names collapse; reusing an operator label across chains stays legal. Run per list at parse time and across static+backup together, since the two are looked up against each other by address. Every collision is reported at once and sorted, so one fix pass suffices and the message does not vary between runs. Name + index. ResolveDuplicateAddresses gives colliding providers distinct identities "<name>#0", "<name>#1", indexed by config order so they are stable across restarts. An unambiguous name keeps the bare configured name, leaving pin headers, debug output and metric labels untouched for every provider not in a collision. Unreachable behind the validation above; it exists so a list that reaches the session manager by another path degrades to "both routable" rather than "one silently dropped". Address() replaces Name at all four identity sites: session registration, mergeAbsentProviders, the restored-set match, and the metrics endpoint_id. That last one matters — health metrics key off PublicLavaAddress, so leaving RegisterEndpoint on Name would split a collided provider across two endpoint_id values. The three restore-path sites compared config Name against session PublicLavaAddress and agreed only because the two were identical; changing the derivation without them would report every provider absent and merge duplicate sessions fleet-wide. Tests drive UpdateAllProviders and getValidProviderAddresses rather than restating their logic, which was the flaw in the existing duplicate-name test. Confirmed non-vacuous: without resolution the same path yields one pairing entry, validAddresses [SimTwin SimTwin], and an empty selection. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
8 of 12 tasks
avitenzer
added a commit
that referenced
this pull request
Aug 17, 2026
…2724) A provider's configured name was its routing identity, and the maps keyed by that identity collapsed two entries sharing a name into one. One upstream served every request while the other sat idle, and because the retry skip-list used the same key, setting that name aside after a failure removed both and left nothing to retry against — the customer got an error while a healthy provider stood by. Commit dffc0f2 established that duplicate names must be tracked separately, but only for the startup failed-provider set. Everything downstream was still keyed by name. Two layers: Fail fast. ValidateUniqueProviderNames rejects two providers sharing a name on one chain+api-interface, before any listener binds. That scope is deliberate — a session manager is built per chain+interface and owns its own pairing map, so it is exactly where names collapse; reusing an operator label across chains stays legal. Run per list at parse time and across static+backup together, since the two are looked up against each other by address. Every collision is reported at once and sorted, so one fix pass suffices and the message does not vary between runs. Name + index. ResolveDuplicateAddresses gives colliding providers distinct identities "<name>#0", "<name>#1", indexed by config order so they are stable across restarts. An unambiguous name keeps the bare configured name, leaving pin headers, debug output and metric labels untouched for every provider not in a collision. Unreachable behind the validation above; it exists so a list that reaches the session manager by another path degrades to "both routable" rather than "one silently dropped". Address() replaces Name at all four identity sites: session registration, mergeAbsentProviders, the restored-set match, and the metrics endpoint_id. That last one matters — health metrics key off PublicLavaAddress, so leaving RegisterEndpoint on Name would split a collided provider across two endpoint_id values. The three restore-path sites compared config Name against session PublicLavaAddress and agreed only because the two were identical; changing the derivation without them would report every provider absent and merge duplicate sessions fleet-wide. Tests drive UpdateAllProviders and getValidProviderAddresses rather than restating their logic, which was the flaw in the existing duplicate-name test. Confirmed non-vacuous: without resolution the same path yields one pairing entry, validAddresses [SimTwin SimTwin], and an empty selection. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
7 of 9 tasks
This was referenced Aug 19, 2026
8 of 12 tasks
8 of 12 tasks
10 of 13 tasks
This was referenced Oct 4, 2026
Merged
8 of 9 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Multi-protocol RPC gateway that routes JSON-RPC, REST, gRPC, and Tendermint RPC requests to statically configured upstream endpoints with QoS-based provider selection, caching, and automatic failover.
Components:
Provenance: hard fork of github.com/lavanet/lava/v5, rebranded to github.com/magma-Devs/smart-router. Scope is trimmed to the router and its transitive dependencies — no chain/node/keeper code — and no upstream merges are expected. "lavanet/lava" strings remaining in specs/lava.json and protocol/chainlib/rest_test.go are Cosmos REST paths served by the Lava node, not GitHub refs, and must not be rewritten.
Took 2 minutes