Skip to content

Releases

v1.6.8

Choose a tag to compare

@github-actions github-actions released this 10 Nov 13:54
b9b4af3

v1.6.8 — 2026-11-09

Highlights

Integrators upgrading to version 1.6.8 must account for several breaking changes across connection limits, caching, GraphQL handling, and health reporting. First, operators routing traffic through external proxies like CDNs must configure the new --websocket-trusted-proxies flag with their proxy IP ranges, because --limit-parallel-websocket-connections-per-ip now counts strictly by client address rather than a combination of address, X-Forwarded-For, and User-Agent. Second, requests containing lava-select-provider, lava-stickiness, or lava-providers-block headers now bypass all cache tiers to guarantee node execution, eliminating the need to append lava-force-cache-refresh. Third, the router enforces stricter GraphQL validation, returning HTTP 400 with a GraphQL error document instead of HTTP 500 for requests exceeding depth or field limits, and refusing payloads with duplicate fields, mismatched casing, or mixed mutation and query operations. Fourth, provider verification backoffs now retry nodes offline for over 16 minutes every 15 minutes instead of 3—reducing the frequency of retry: provider verification still failing log lines—while smartrouter health now reports PartialFail for failing add-on checks, removes URLs on explicit Fail states, and exposes new scope="none" and scope="service" metric series. Beyond these breaking changes, this release introduces GET /debug/verification to expose state transitions alongside GET /debug/relay-errors for tracking node and protocol error counts, and refines upstream error handling by correctly parsing Solana JSON-RPC errors during chain tracking, classifying XRP "not found" errors for proper retries, and respecting rate-limit wait times requested via gRPC headers or trailers.

Changes

⚠ Breaking changes

  • fix(chainlib)!: answer a GraphQL request the router refuses by its own limit with a client error and a GraphQL error document (MAG-4577) (#598) [2d99d4b]
    • a GraphQL request refused by the router's own limits (more than 256 levels of depth, more than 300 root fields, or more than 32 nested fragments) now gets HTTP 400 with a GraphQL error document instead of HTTP 500 with the error inside a JSON string.
  • feat!: say what a failed verification costs (Fail, PartialFail, Warning), and publish every failing check (MAG-3950) (#580) [67eeeba]
    • smartrouter health reports PartialFail for add-on and archive checks that read Fail; a url naming an undefined add-on is admitted with the name in refusedServices instead of refused; an explicit "Fail" on an add-on check now takes the url out; both verification metrics gain scope="none" series and scope="service" series keyed by undefined add-on names.
  • feat!: retry each failing verification on its own backoff, log its transitions, serve GET /debug/verification (MAG-3950) (#582) [2ab221f]
    • a provider out for ~16 min or more is retried every 15 min (+jitter) instead of every 3 min; the url grace's second strike comes ~30s after the first instead of at the next epoch, so a blip that outlasts the retry delay demotes the provider; Loki rules counting retry: provider verification still failing lines see about one line per 15 min.
  • fix(chainlib)!: count the websocket connection limit per client address, read through trusted proxies (MAG-3083) (#665) 104aead
    • --limit-parallel-websocket-connections-per-ip counts connections per client address instead of per address, X-Forwarded-For and User-Agent. Behind a proxy whose address is not loopback or private, such as a CDN, add its ranges to the new --websocket-trusted-proxies flag, or every client behind it shares one allowance.
  • fix(chainlib)!: refuse a GraphQL request whose wrapper the router and the node would read differently (MAG-4908) (#655) [46f4f4b]
    • a GraphQL request body that repeats query, operationName, variables or extensions, or spells one of them in a different letter case, is refused by the router with HTTP 400 instead of being relayed to a node. A GraphQL document that holds a mutation beside a query or a subscription is refused with HTTP 400 instead of running the operation operationName names.
  • fix(cache)!: never answer a pinned, sticky or node-excluding request from the cache (MAG-3520, MAG-3864) (#663) [b9b4af3]
    • a request carrying lava-select-provider, lava-stickiness or lava-providers-block is no longer answered from the cache, by either tier; it always reaches a node, and the node's answer is still cached for other requests. Sending lava-force-cache-refresh next to a pin is no longer needed for this.

Dependency updates

  • fix(deps): bump github.com/andybalholm/brotli from 1.1.0 to 1.2.8 (GO-2026-6888) (#657) 1e25c1d

New Features

  • feat!: say what a failed verification costs (Fail, PartialFail, Warning), and publish every failing check (MAG-3950) (#580) [67eeeba]
  • feat!: retry each failing verification on its own backoff, log its transitions, serve GET /debug/verification (MAG-3950) (#582) [2ab221f]
  • feat(debug): GET /debug/relay-errors reports node-error and protocol-error counts (MAG-4718) (#626) [e709626]

Bug fixes

  • fix(chainlib)!: answer a GraphQL request the router refuses by its own limit with a client error and a GraphQL error document (MAG-4577) (#598) [2d99d4b]
  • fix(chaintracker): count a Solana latest-block poll answered with a JSON-RPC error as a failed poll (MAG-4613) (#579) [c19b414]
  • fix(provideroptimizer): log one debug record per pick, whatever the number of candidates (MAG-3152) (#674) [7c4f843]
  • fix(chainlib): keep the catch-all node url when two urls share a method list (MAG-3081) (#673) [8116cca]
  • fix(rpcsmartrouter): answer a refused JSON-RPC 429 without lava-identified-node-error (MAG-3030) (#679) [52a648e]
  • fix(rpcsmartrouter): never cache node-local Bitcoin-family or walletd reads (MAG-4207, MAG-4793) (#672) [57dd388]
  • fix(chainlib,common): an XRP error inside result is a node error on every method (MAG-4817) (#649) [d1ad560]
  • fix(common): retry every XRP "not found" and classify the rest of rippled's errors (MAG-4817) (#649) [3603080]
  • fix(common): keep a node that does not serve pending state switched on, and ask a peer (MAG-5021) (#667) [22e12a1]
  • fix(rpcsmartrouter): send the lava-* headers when a request's budget runs out unanswered (MAG-3980) (#671) [6afab23]
  • fix(chainlib): send the request headers a spec declares on JSON-RPC and Tendermint, as on REST and gRPC (MAG-3079) (#662) [c793ca7]
  • fix(chaintracker): resume polling when a hold-off clears and don't probe-score a held-off endpoint (MAG-4387) (#537) [628e1e5]
  • fix(grpc): a DEADLINE_EXCEEDED the node sends at the router's deadline is the router's timeout, not the node's answer (MAG-4925) (#656) [efc90a5]
  • fix(rpcsmartrouter): say why a write's outcome is unclear (MAG-4380) (#546) [e90f7e3]
  • fix(grpc): hold a rate-limited gRPC endpoint off for the wait it asked for, header or trailer (MAG-4914, MAG-4915) (#678) 0e8b9bc
  • fix(chainlib)!: count the websocket connection limit per client address, read through trusted proxies (MAG-3083) (#665) 104aead
  • fix(cache): never cache a null for a block at or below the router's tip (MAG-4124) (#668) [5b9f81e]
  • fix(lavasession): release only a pinned request's own provider, and only a live one (MAG-4731) (#670) [4b1cc3c]
  • fix(chainlib)!: refuse a GraphQL request whose wrapper the router and the node would read differently (MAG-4908) (#655) [46f4f4b]
  • fix(chainstate): let a single-endpoint tip follow its endpoint through a block storm (MAG-4964) (#669) [9ec894f]
  • fix(rpcsmartrouter): tell the caller when an archive request degrades to a non-archive node (MAG-4729) (#675) [cc118a2]
  • fix(version): report the toolchain's VCS revision when -ldflags left no commit (MAG-3298) (#658) [f6f3214]
  • fix(cache)!: never answer a pinned, sticky or node-excluding request from the cache (MAG-3520, MAG-3864) (#663) [b9b4af3]
Read more

v1.6.7

v1.6.7 Pre-release
Pre-release

Choose a tag to compare

@github-actions github-actions released this 08 Nov 08:35
04c91a4

v1.6.7 — 2026-11-08

Highlights

Integrators must update client error handling and routing logic to accommodate three breaking changes in version 1.6.7. First, requests using the lava-stickiness header or setting lava-select-provider-strict: true no longer failover if the pinned provider fails, lags the chain tip, or is busy; clients now receive the direct node error or consistency rejection. Second, the router now returns upstream error responses byte-for-byte on all HTTP interfaces without a router envelope, moving the request GUID exclusively to the Lava-Guid HTTP header and shifting router-generated status codes to 502 for unreachable upstreams, 504 for timeouts, and 429 for rate limits. Third, operators must stop clients from calling RPC methods marked enabled: false in the lava-specs catalog (such as eth_newFilter on Etherlink), as the router now explicitly rejects these instead of relaying them. Beyond these breaking API changes, this release adds support for a RESP (Redis/Valkey) backend as a secondary cache tier, managed by a dedicated read breaker tied to the --cache-timeout flag. Finally, the WebSocket and gRPC managers now share a unified connection pool to prevent orphaned subscriptions, the router correctly parses EIP-1898 block objects for accurate QoS sync scoring, and configured upstream secrets are strictly redacted from all client replies and logs.

Changes

⚠ Breaking changes

  • fix(protocol)!: a sticky request is answered by its node or fails; lava-select-provider keeps failover unless lava-select-provider-strict (MAG-2280, MAG-4123, MAG-4826) (#578) [760b442]
    • a request with lava-stickiness, or lava-select-provider with lava-select-provider-strict: true, is no longer served by another node when its node fails, is slow, or lags the chain tip. It gets that node's error with Lava-Retries 0, or "pinned provider is too far behind the chain tip" from the consistency gate. A pod-local sticky session whose node is busy now fails instead of moving to another node. A plain lava-select-provider is unchanged.
  • fix(protocol)!: return the node's answer as is on every interface; 502/504 only when no node answered (MAG-4481) (#587) [e44a7d2]
    • a failed relay that a node answered now returns that node's status and body byte for byte on every HTTP interface, with no router envelope, no added jsonrpc field and no GUID in the body (it stays in the Lava-Guid header). All attempts rate limited is the node's 429 (was 503). No node answered is 502 "no answer from upstream: ", or 504 for a timeout or a hung upstream (was 500). A write that every node answered gets the node's answer like a read: all primaries down is the node's 503 (was 500), all rate limited is 429 (was 503).
  • fix(chainlib)!: refuse an api the spec disables instead of relaying it (#490) [8d5ec1b]
    • a method the spec marks enabled: false is refused with "api is disabled" instead of being relayed as a Default- api. Clients that got a node's answer for such a method before now get an error. The lava-specs catalog disables 412 apis, among them eth_newFilter, eth_getFilterChanges, eth_getProof and eth_syncing on Etherlink and Monad.

New Features

  • feat(rpcsmartrouter): an internal-probe request does not keep a chain awake (MAG-4797) ([#633]) [d0866a6]
  • feat(cache): let the secondary cache tier be a RESP (Redis/Valkey) backend (MAG-4778) ([#630]) [a1b89e2]

Bug fixes

  • fix(utils): make the log level and the stderr logger safe to change while others log (MAG-4740) ([#617]) [aca006b]
  • fix(chainlib,tracing): keep gRPC headers sendable and span strings valid UTF-8 (#555) [f585df3]
  • fix(chainlib): cut request bodies on every listener's log lines, and name the GUID on the error line (MAG-4454) (#559) [0b8a55e]
  • fix(metrics): keep the method label valid UTF-8 (MAG-4430) (#554) [7c1102e]
  • refactor(specfetcher): run the GitHub and GitLab contents listings through one flow (MAG-4244) ([#606]) [fe4ea6a]
  • fix(lavasession): a wss url no longer keeps an https + wss provider out of the last chance (MAG-4285) ([#620]) [010667c]
  • fix(lavasession): lend the last chance to every concurrent request when all URLs are off (MAG-4829) ([#637]) [acc4455]
  • fix(cache): warn at startup when a RESP primary is paired with a secondary cache (MAG-4777) ([#629]) [d765031]
  • fix(chainlib): a batch mixing a latest read with block 0 asks about latest, not block 0 (MAG-4346) (#593) [b875f86]
  • fix(endpointstate): report every answered poll on the url's series, not only one the tip store kept (MAG-4347) (#592) [1e9c85a]
  • fix(parser): read the height out of an EIP-1898 {"blockNumber": …} block object (MAG-4758) ([#619]) [d5b80c4]
  • refactor(provideroptimizer): score latency and sync lag through one normalizer (MAG-4244) ([#607]) [2b9df42]
  • refactor(lavasession): send both direct HTTP requests through one sender (MAG-4279) ([#608]) [924600c]
  • fix(rpcsmartrouter): a client that joins during the last client's unsubscribe gets its own subscription (MAG-4345) (#589) [322a967]
  • fix(rpcsmartrouter): tell a non-EVM node the unsubscribe its spec declares, and bound a subscription's restores (MAG-4278) (#590) [de08be4]
  • fix(rpcsmartrouter): tell the node to drop the subscription a restore replaced, and keep a redialed connection (MAG-4396) (#591) [04811cf]
  • refactor(rpcsmartrouter): one subscription core for the WebSocket and gRPC managers (MAG-4279) ([#612]) [9a865bf]
  • fix(rpcsmartrouter): one connection pool behind the WebSocket and gRPC subscriptions (MAG-4279) ([#614]) [ddf96f8]
  • refactor: mechanical de-duplication from the 2026-10-13 review, and a dupl lint guard (MAG-4280) ([#604]) [ad8f204]
  • refactor(rpcsmartrouter): one helper per primary/backup tier step instead of a pasted copy per tier (MAG-4279) ([#610]) [f837232]
  • refactor(chainlib): one handler for every HTTP listener route and one HTTP send for the proxies (MAG-4279) ([#611]) [278a4a0]
  • fix(common): retry Alchemy's -32600 "Unsupported method" on another provider ([#632]) [2953bc9]
  • fix(protocol)!: a sticky request is answered by its node or fails; lava-select-provider keeps failover unless lava-select-provider-strict (MAG-2280, MAG-4123, MAG-4826) (#578) [760b442]
  • fix(protocol)!: return the node's answer as is on every interface; 502/504 only when no node answered (MAG-4481) (#587) [e44a7d2]
  • fix(performance): close the RESP secondary's breaker within secondary-cache-timeout (MAG-4778) ([#630]) [7756ba0]
  • fix(cache): name secondary-resp-cache in its startup errors, and refuse its unused keys first (MAG-4778) ([#630]) [3caaab1]
  • fix(performance): close the RESP cache's read breaker within --cache-timeout (MAG-4872) ([#645]) [b7a3bd2]
  • fix(protocol): one error-classification path, and gRPC rate limits held off rather than scored (MAG-4279) ([#613]) [b791f5b]
  • fix(security): withhold configured upstream secrets from client replies and logs (MAG-4863) ([#648]) [3a3f24f]
  • fix(chainlib)!: refuse an api the spec disables instead of relaying it (#490) [8d5ec1b]
  • fix(rpcsmartrouter): keep the gRPC pool's reconnect on its last rung past the ladder (MAG-4873) ([#646]) [2d35f3d]
  • fix(rpcsmartrouter): log a consistency rejection as a warning that names the node (#459) [0cd9f50]
  • fix(chainlib): admit a provider pinned to a path that serves the root, whatever the collection order (MAG-4131) (#483) [6235ce9]
  • fix(chainlib): admit the pinned url in the re-verifier's and health's copy of the guard too (MAG-4131) (#483) [83bc1d0]
  • fix(rpcsmartrouter): a retry after a cached node error asks an upstream (MAG-4416) (#552) [0dc300e]
  • fix(rpcsmartrouter): give a cached node error the retry verdict a live one gets (MAG-4416) (#552) [9f2b91b]
  • fix(rpcsmartrouter): the router's own health check never switches a lent url back on (MAG-4790) ([#631]) [04c91a4]

Documentation updates

  • docs: say Smart Router runs on Kubernetes and whom to contact (MAG-4723) ([#616]) [869ad4b]
  • docs(metrics): say that outcome="error" also counts a refused cache entry (MAG-3827) ([#623]) [0bd92d6]
  • docs(relaypolicy): point the by-hash archive note at the MAG-4142 decision ([#642]) [b85bf5c]
  • docs(secondary-cache): name the ACL a read-only RESP secondary needs (MAG-4778) ([#630]) [3df63ba]

Build process updates

  • ci: run every unit test on each pull request and on main (MAG-4847) ([#635]) [6c6c704]
  • ci: run the unit tests under the race detector (MAG-4849) ([#640]) [feb6ea3]
  • ci: run protocol/rpcsmartrouter's tests as four concurrent shards (MAG-4856) ([#643]) [fed7c22]
  • ci: retire the dev-sim-prtests PR gate (MAG-4870) ([#644]) [5e049c5]

Other work

  • style: fix the three golangci-lint findings on main (MAG-4855) ([#641]) [e8bea92]
  • perf(grpc): give one descriptor resolver to a whole endpoint, not to each channel (MAG-4397) ([#601]) [157e1ee]
  • perf(metrics): update the rpc_endpoint_* series positionally, without a label map per call (MAG-4244) ([#609]) [2d2709e]
Read more

v1.6.6

v1.6.6 Pre-release
Pre-release

Choose a tag to compare

@github-actions github-actions released this 02 Nov 10:39
bb19283

v1.6.6 — 2026-11-02

Highlights

Integrators must update incoming requests to pass the customer-id HTTP header, as the router no longer reads the project-id header for customer identification. Failing to migrate to the new header name will break customer attribution for all routed RPC traffic. Beyond this breaking API change, version 1.6.6 corrects provider blocking telemetry for edge cases during upstream failover. The smartrouter_csm_provider_blocked metric now correctly increments for backup providers and providers that have no usable URL. This ensures operators alerting on this metric will catch routing dead-ends even when a primary upstream fails and the configured fallbacks are unreachable.

Changes

⚠ Breaking changes

  • feat(chainlib)!: read the customer id from a customer-id header, renamed from project-id (MAG-4727) (#621) 1fb9fc4

New Features

  • feat(chainlib)!: read the customer id from a customer-id header, renamed from project-id (MAG-4727) (#621) 1fb9fc4

Bug fixes

  • fix(lavasession): smartrouter_csm_provider_blocked covers backups and providers with no usable url (MAG-4735) (#624) bb19283

Documentation updates

  • docs(changelog): v1.6.5 readiness follows reachability; add MAG-4641 (MAG-4720) (#615) 0a97191

v1.6.5

v1.6.5 Pre-release
Pre-release

Choose a tag to compare

@github-actions github-actions released this 29 Oct 10:35
bd9f314

v1.6.5 — 2026-10-29

Highlights

Readiness follows reachability. A router's /readyz and smartrouter_overall_health now say whether it reaches its upstreams. Each chain+interface's health check ends in one of three verdicts:

  • No upstream answers: every try timed out, failed to connect, or had no upstream to go to. The chain turns unhealthy. A client relay that no upstream answered brings the check forward to within --relays-health-unhealthy-interval (default 15s). A chain+interface that turns unhealthy brings forward the checks of the router's other interfaces too. A router that loses its egress no longer waits out the 5-minute healthy cadence, and an idle interface no longer holds it Ready.
  • An upstream answers with an error, an error status or a rate limit: nothing changes. Every replica shares its upstreams and gets the same answer, so it is no reason to take one out of rotation. The answer does not prove the chain serves either, so a chain that is already unhealthy stays unhealthy.
  • An upstream answers with a result: the chain is healthy. Earlier releases passed the check on any reply, including a gRPC call that hit the router's own deadline, which kept a router without egress Ready indefinitely.

Who notices: operators. An upstream outage that every replica shares no longer empties a router's Service, and smartrouter_overall_health stays 1 through it, so alert on the error share for that, not on readiness. The Health Check Interval Check log line gains verdict (healthy, inconclusive or unreachable), and its health result is now the chain's health after the check. docs/METRICS.md ("When they change") states the rule. This is MAG-4617, MAG-4639 and MAG-4641. MAG-4641 landed inside #584's commit, so its line below was added by hand.

Release v1.6.5 also refines upstream health verification and router readiness by ensuring each node URL is verified over its own isolated connection and each health-check attempt uses a dedicated relay processor. Operators can also track exactly which configured providers failed verification—and which specific check triggered the failure—through newly published metrics. Finally, for gRPC traffic, this release corrects how connection failures and timeouts are classified during upstream routing. A call that hits the router's internal deadline is now properly recorded as a timeout, and a refused connection is marked as unreachable, preventing the router from misreporting either event as a valid answer from the upstream node.

Changes

New Features

  • feat(metrics): publish which configured providers failed verification, and which check (MAG-4594) (#572) 316fad1
  • feat(readiness): readiness follows reachability, three-valued probe (MAG-4641) (#586) bb43ae9
    • added by hand: #586 merged into #584's branch and reached main inside #584's squash commit, so the generator listed no line for it. This is the readiness rule in the Highlights.

Bug fixes

  • fix: verify every node url over its own connection (MAG-4231) (#463) dc512c7
  • fix(readiness): turn NotReady when a router's relays all fail (MAG-4617) (#581) b6d951c
    • its rule, that any reply but a result fails the check, was replaced before this release by MAG-4641: see the Highlights.
  • fix(grpc): a call that hits the router's own deadline is a timeout, not the node's answer (MAG-4625) (#583) ec999d5
  • fix(grpc): a refused connection is unreachable, not the node's answer (MAG-4355) (#585) f4dd073
  • fix(readiness): give each health-check try its own relay processor (MAG-4639) (#584) bb43ae9
    • this commit also carries MAG-4641 (#586).

Documentation updates

  • docs(changelog): say v1.6.4 drops the two distrusted roots, and correct v1.6.2 (MAG-4596) (#575) ec7a319

v1.6.4

v1.6.4 Pre-release
Pre-release

Choose a tag to compare

@github-actions github-actions released this 23 Oct 11:19

v1.6.4 — 2026-10-23

Highlights

Security: the image no longer trusts the two roots removed over the eMudhra compromise. v1.6.4 builds on gcr.io/distroless/static-debian13:debug, whose trust store (ca-certificates 20250419+deb13u1) no longer contains emSign ECC Root CA - G3 or AAA Certificate Services, the two roots Debian and Mozilla removed over the eMudhra intermediate compromise. The previous base, static-debian12, is no longer updated by distroless, so v1.6.3 and every earlier image still trust both. Correction to the v1.6.2 notes: they say the fix is a rebuild "not a change to the image's CA bundle". The root removals had already been published, and this release carries them. The Go 1.26.10 check from v1.6.2 still rejects any chain through the eight compromised intermediates; the trust store now refuses chains through those two roots as well, including chains through eMudhra CAs outside Go's list. A host whose certificate chain ends only at one of the two roots now fails TLS verification, as it already does in Mozilla, Chrome and Debian. The same fix for the v1.5 line is v1.5.10.

The release also introduces a breaking change to error handling and event logging that requires operators to update their telemetry pipelines to relay_end schema 2. The status field for a failed relay no longer writes 0; it now records the upstream's actual HTTP status code, or 502 when all transport attempts fail. Additionally, the router no longer intercepts errors to return a generic HTTP 500 text response, meaning client applications will now receive the upstream node's exact status code and response body. Integrators must consult docs/RELAY-END-EVENT.md to verify the schema 2 contract and update clients to parse upstream-native error payloads. Beyond these protocol corrections, this release adjusts the failover routing logic to include a lastChanceBorrow condition, which permits the router to lend a head-stalled upstream URL when no other healthy nodes are available to serve the request.

Changes

⚠ Breaking changes

  • fix(protocol)!: return the upstream's own error to the client instead of the router's text (MAG-4218) (#551) 8645963
    • relay_end is schema 2. status on a failed relay is the status the client gets: the upstream's own status when an upstream answered, 502 when every attempt failed in transport, where schema 1 wrote 0. docs/RELAY-END-EVENT.md is the contract for schema 2 and lists the versions. The reply the client reads on this path changes too: the node's status and body instead of HTTP 500 with the router's text.

Bug fixes

  • fix(protocol)!: return the upstream's own error to the client instead of the router's text (MAG-4218) (#551) 8645963
  • fix(lavasession): the last chance lends a head-stalled URL when nothing else can serve (MAG-4589) (#570) 9eda1ae
  • refactor(lavasession): name the last-chance condition lastChanceBorrow (MAG-4589) (#570) 926b0c1
  • fix(docker): move the runtime base to distroless static-debian13 (MAG-4596) (#573) 1a74447
    • this is the security fix above: the image's trust store no longer contains emSign ECC Root CA - G3 or AAA Certificate Services.

v1.5.10

Choose a tag to compare

@github-actions github-actions released this 23 Oct 11:11

v1.5.10 — 2026-10-23

Highlights

Smart Router v1.5.10 is v1.5.9 on a current runtime base; the router's own code is unchanged. The image now builds on gcr.io/distroless/static-debian13:debug, whose trust store (ca-certificates 20250419+deb13u1) no longer contains emSign ECC Root CA - G3 or AAA Certificate Services, the two roots Debian and Mozilla removed over the eMudhra intermediate compromise. The previous base, static-debian12, is no longer updated by distroless, so v1.5.9 and every earlier image still trust both.

Correction to the v1.5.9 notes: they describe the fix as a rebuild "rather than a change to the image's CA bundle". The root removals had already been published, and this release carries them. The Go 1.26.10 check from v1.5.9 still rejects any chain through the eight compromised intermediates; the trust store now refuses chains through those two roots as well, as a second and independent layer. A host whose certificate chain ends only at one of the two roots now fails TLS verification, as it already does in Mozilla, Chrome and Debian.

Changes

Bug fixes

  • fix(docker): move the runtime base to distroless static-debian13 (MAG-4596), the same change as #573 on main

v1.6.3

v1.6.3 Pre-release
Pre-release

Choose a tag to compare

@github-actions github-actions released this 22 Oct 09:28

v1.6.3 — 2026-10-22

Highlights

Smart Router v1.6.3 expands the relay-end event payload with deeper transaction and caller telemetry for production observability. Integrators can now track submitted transactions directly via the new tx_hash, tx_status, and tx_error fields, while identity fields like customer, request_id, task_id, and tx_id map upstream requests back to the originating client. The event payload also introduces request and reply sizes, block heights, and the upstream node_code, complemented by two new narrative log lines written per relay when the router operates at the Debug log level. Finally, this release resolves REST routing bugs by ensuring REST health checks correctly send their GET template as the URL without a body, and by applying a unified body-method rule for REST relays that preserves the original API name when given an empty template.

Changes

New Features

  • feat(relay-end): name the transaction a submit sends (MAG-4455) (#556) 07bf136
  • feat(relay-end): name the customer and the caller's ids (MAG-4456) (#556) 2d42393
  • feat(relay-end): add request and reply sizes, blocks and node codes (MAG-4457) (#556) 10f1ae0
  • feat(logs): write two per-relay narrative lines at Debug (MAG-4054) (#569) 5144af3

Bug fixes

  • fix(rpcsmartrouter): a REST health check sends its GET template as the URL, with no body (MAG-4516) (#561) d44e0f6
  • fix(rpcsmartrouter): one body-method rule for REST relays, and an empty template keeps the api name (MAG-4516) (#561) 7373514

Documentation updates

  • docs(changelog): say v1.6.2 distrusts the compromised eMudhra intermediates (MAG-4522) (#564) c3a120f
  • docs(relay-end): document tx_hash, tx_status and tx_error by chain (MAG-4455) (#556) 8095142
  • docs(relay-end): document customer, request_id, task_id and tx_id (MAG-4456) (#556) f8b606f
  • docs(relay-end): document sizes, blocks and node_code (MAG-4457) (#556) 4d7d9bf

v1.6.2

v1.6.2 Pre-release
Pre-release

Choose a tag to compare

@github-actions github-actions released this 21 Oct 17:07

v1.6.2 — 2026-10-21

Highlights

Security: v1.6.2 distrusts the compromised eMudhra intermediate CAs. It is built on Go 1.26.10, which makes crypto/x509 reject any certificate chain that contains one of eight eMudhra intermediate CA certificates whose private keys were compromised (GO-2026-6809, CVE-2026-111943). With one of those keys, anyone who can intercept the router's TLS connection to an upstream node or a TLS cache can present a certificate for that host that every earlier release accepts, then read or alter the relayed traffic, including upstream API keys carried in URLs or headers. v1.6.2 refuses such a certificate whichever trust store it verifies against. The check is compiled into the binary, so the fix is this rebuild, not a change to the image's CA bundle. Connections pinned to a private CA (an endpoint's cacert-pem, or the RESP cache's tls.ca-file) were never exposed. Every earlier release is affected: move to v1.6.2, or to v1.5.9 on the 1.5 line.

The release also introduces three breaking changes affecting GraphQL request validation and write broadcasting during rate limits. First, the router now outright refuses GraphQL requests that expand to more than 300 root fields or nest fragments deeper than 32 levels at the root, rather than relaying them to upstream nodes. Second, when a GraphQL request combines a root under an enabled cross-validation policy with one under a forbid-caller-cv policy, the router now enforces cross-validation under the enabled policy without the caller's headers, rather than routing the request to a single provider. Third, write broadcasting behavior during rate-limit hold-offs has changed: writes are now broadcast to all enabled providers regardless of their hold-off status, meaning lava-fast-tx-participants will list every provider, recovered providers may serve the winning reply indicated by the Lava-Provider-Address header, and still-limited vendors will receive the write and return a 429. Beyond these breaking changes, this release fixes an issue where a single block-less element would prevent the router from caching batch or multi-root GraphQL requests.

Changes

⚠ Breaking changes

  • fix(chainlib)!: bound the root fields a GraphQL request expands to (MAG-4360) (#549) d3b38e6
    • a GraphQL request that expands to more than 300 root fields, or nests fragments more than 32 deep at its root, is refused by the router instead of being relayed to a node.
  • fix(rpcsmartrouter)!: an enabled cross-validation policy applies beside a forbidden GraphQL root (MAG-4361) (#550) 0d38f68
    • a GraphQL request that selects a root under an enabled cross-validation policy together with a root under a forbid-caller-cv policy is cross-validated under the enabled policy, without the caller's headers, instead of being served by a single provider.
  • fix(lavasession)!: broadcast a write to every provider during a rate-limit hold-off (MAG-4388) (#538) 52c429c
    • during a rate-limit hold-off a write is broadcast to every enabled provider, held off or not, instead of only to the providers that are not held off (or, with every provider held off, to the soonest to expire). lava-fast-tx-participants lists every provider again, a held-off provider that has recovered can be the one whose reply is returned (Lava-Provider-Address), and a vendor that is still rate-limiting receives the write and answers it with a 429.

Bug fixes

  • fix(chainlib)!: bound the root fields a GraphQL request expands to (MAG-4360) (#549) d3b38e6
  • fix(rpcsmartrouter)!: an enabled cross-validation policy applies beside a forbidden GraphQL root (MAG-4361) (#550) 0d38f68
  • fix(chainlib): a block-less element no longer makes a batch or multi-root GraphQL request uncacheable (MAG-4381) (#547) 871d770
  • fix(lavasession)!: broadcast a write to every provider during a rate-limit hold-off (MAG-4388) (#538) 52c429c

Build process updates

  • build(deps): move to Go 1.26.10 so govulncheck passes (#560) 2639009
    • this is the security fix above: Go 1.26.10 rejects any certificate chain through the eight compromised eMudhra intermediate CAs (GO-2026-6809).

v1.6.1

v1.6.1 Pre-release
Pre-release

Choose a tag to compare

@github-actions github-actions released this 20 Oct 09:08
150d616

v1.6.1 — 2026-10-20

Highlights

Smart Router v1.6.1 introduces four breaking changes to gRPC TLS validation and GraphQL error handling that operators must address before upgrading. First, gRPC TLS settings are now strictly enforced across all dialers: a cacert-pem that is unreadable or fails to verify the dialed host will now refuse the connection instead of falling back to system roots, requiring operators to provide valid certificates or explicitly set the allow-insecure flag for self-signed nodes. Second, GraphQL routing changes three behaviors: queries nested deeper than 256 levels are outright refused, 2xx responses lacking both data and errors objects are scored as node failures rather than valid answers, and 4xx responses containing JSON-RPC error envelopes trigger a retry on a new provider while filing as NODE_METHOD_NOT_ALLOWED in smartrouter_errors_total instead of returning immediately to the client. Beyond these breaking changes, this release consolidates request telemetry into a single relay_end log event that records the chain name, skipped upstreams, stale fallbacks, and a detailed history of every retry attempt with its send time and retryable verdict. Finally, the router optimizes upstream connection management by honoring hold-offs from JSON-RPC 429 rate limit bodies, sharing a single gRPC stream per subscription, and actively dropping dead WebSocket connections and slow subscription clients.

Changes

⚠ Breaking changes

  • fix(common)!: one TLS config and dial builder for every gRPC dialer, one reply-header filter for every tier (MAG-4279) (#524) 16a42b9
    • the gRPC TLS settings (cacert-pem, cert-pem/key-pem, allow-insecure) now mean the same thing on every dialer, and cacert-pem is enforced. The unary connector verifies the server against cacert-pem: it used to skip verification once a CA was loaded, so a cacert-pem that did not sign the server, or whose certificate does not name the dialled host or IP, now refuses the dial (set allow-insecure for a self-signed development node). Subscription streams now verify against cacert-pem and present cert-pem/key-pem; they used to read allow-insecure alone. cert-pem/key-pem is presented even when cacert-pem is set. A cacert-pem that cannot be loaded (unreadable, or holding no PEM certificate) now fails every dial instead of falling back to the system roots.
  • fix(chainlib)!: a JSON-RPC error envelope under a caller's status on a GraphQL endpoint is the gateway refusing the route (MAG-4379) (#545) 5eff4a0
    • on the graphql interface, a 4xx reply carrying a JSON-RPC error envelope under a status the registry files as the caller's (400, 406, 411, 412, 414-417, 422, 428, 431) is no longer returned to the client as the answer. It is retried on another provider, so the body, Lava-Retries and Lava-Provider-Address the client gets change, and it is filed as NODE_METHOD_NOT_ALLOWED instead of USER_INVALID_REQUEST in smartrouter_errors_total.
  • fix(chainlib)!: a GraphQL 2xx with neither data nor errors is a node error, not the answer (MAG-4369) (#534) [f14fdac]
    • a GraphQL reply under a 2xx that carries neither a data object nor errors is no longer the answer. A read is retried on another provider, so the body, Lava-Retries and Lava-Provider-Address the client gets change; a write waits for another provider's answer instead of returning the gateway's body at once; and the endpoint that sent it is scored as failing instead of having its failure count reset.
  • fix(chainlib)!: bound how deeply a GraphQL document may nest (MAG-4359) (#548) 150d616
    • a GraphQL request nested more than 256 levels deep is refused by the router as a request it cannot parse, instead of being relayed to a node.

New Features

  • feat(utils): a GUID carries the second it was minted (MAG-4399) (#539) [f017c87]
  • feat(utils): counter, not random bits, below the GUID second (MAG-4399) (#539) 16a167f
  • feat(logs): one relay_end event per relay, at every log level (MAG-4400) (#543) [9fc7505]
  • feat(logs): name the chain on relay_end (MAG-4400) (#543) [9e23602]
  • feat(logs): list relay_end attempts whenever a relay took more than one (MAG-4400) (#543) 4a7f9ae
  • feat(logs): give each relay_end attempt its batch and send time (MAG-4400) (#543) 4d9a48c
  • feat(logs): write each failed attempt's retryable verdict on relay_end (MAG-4400) (#543) [f5a7f0a]
  • feat(logs): give each WebSocket frame its own GUID (MAG-4400) (#543) 552fd47
  • feat(logs): name skipped upstreams and stale fallbacks on relay_end (MAG-4400) (#543) [ec71d3a]
  • feat(logs): read a Cosmos transaction's code into relay_end (MAG-4400) (#543) [af0da11]
  • feat(logs): write relay_end for subscription requests (MAG-4400) (#543) 7058ea7

Bug fixes

  • fix: JSON-RPC 429 bodies are rate limits; tracker honours hold-offs (MAG-4165) (#488) 8ad0ff6
  • fix(rpcsmartrouter): retire dead WebSocket connections, drop slow subscription clients, and share one gRPC stream per subscribe (MAG-4242) (#519) [9d1c7fc]
  • fix(lavasession): leave the endpoint alone on a plain session acquisition (MAG-4242) (#520) 027cc1c
  • fix: eleven behaviour drifts between duplicated code paths (MAG-4278) (#523) [f4ebf95]
  • fix(common)!: one TLS config and dial builder for every gRPC dialer, one reply-header filter for every tier (MAG-4279) (#524) 16a42b9
  • fix(lavasession): one descriptor resolver for the unary connection and the subscription stream pool (MAG-4279) (#525) 409e5cd
  • fix(rpcsmartrouter): read no tip out of a GraphQL user relay's reply (MAG-4373) (#535) [e7c6e4c]
  • fix(holdoff): count a 429 burst as one strike (MAG-4386) (#536) [d3413bb]
  • fix(chainlib): type a GraphQL RESOURCE_EXHAUSTED head-poll answer as a rate limit (MAG-4378) (#544) [fc5d729]
  • fix(chainlib)!: a JSON-RPC error envelope under a caller's status on a GraphQL endpoint is the gateway refusing the route (MAG-4379) (#545) 5eff4a0
  • fix(metrics): keep the customer label valid UTF-8, and count an empty project-id as no customer (#553) 64cb8f1
  • refactor(logs): name the event writer utils.LogEvent; lint fixes (MAG-4400) (#543) [dbb6ff1]
  • fix(logs): write relay_end stateful as a JSON boolean (MAG-4400) (#543) 9d03a90
  • fix(logs): withhold an unlisted method only when the URL named it (MAG-4400) (#543) [dfdc63f]
  • fix(logs): write a lone slash for relay_end's root-path method (MAG-4400) (#543) [e17bde7]
  • fix(logs): write no failed-processing line beside relay_end (MAG-4400) (#543) 8a7b3a9
  • fix(relaycore): classify a GraphQL node fault by its error code, not by the 200 it arrived in (MAG-4375) (#541) 2ecb74f
  • fix(chainlib)!: a GraphQL 2xx with neither data nor errors is a node error, not the answer (MAG-4369) (#534) [f14fdac]
  • fix(chainlib)!: bound how deeply a GraphQL document may nest (MAG-4359) (#548) 150d616

Documentation updates

  • docs(changelog): rewrite the v1.6.0 highlights and breaking changes (#527) [a70ce1c]
  • docs(logs): relay_end error is prose; match on outcome and error_code (MAG-4400) (#543) 154be71
  • docs(logs): select relay_end writes without the json label clash (MAG-4400) (#543) [ddb149f]
Read more

v1.6.0

v1.6.0 Pre-release
Pre-release

Choose a tag to compare

@github-actions github-actions released this 14 Oct 13:28
149d2f5

v1.6.0 — 2026-10-14

Highlights

v1.6.0 changes seven behaviours a client or an operator may depend on. They are listed under ⚠ Breaking changes below, each with the surface it touches and what to do: four change what a client sees on a reply (upstream response headers, lava-relay-timeout, retries after a node refusal, hedge accounting), one can stop the router at boot (cross-validation policy validation), one makes a transaction submission ignore the caller's cross-validation headers, and one changes which requests reach an archive node.

Archive routing is decided once, before the first attempt

The router decides whether a request needs an archive node from the request itself, before it picks an endpoint. A request is historical, and goes only to the endpoints that declare the archive extension, when any of these holds:

  • it names a block deeper than the chain's archive depth, the rule.block of the archive extension in the chain's spec (127 blocks behind the head on Ethereum);
  • it names the earliest block;
  • it carries lava-extension: archive.

Endpoints without the extension are skipped whichever tier they are in: on a fleet where only a backup declares archive, a historical request goes straight to that backup, and a request for a recent block stays on the primaries. An archive-routed request counts CU at the extension's cu_multiplier. A retry re-sends the same request to another endpoint of the same pool; it no longer adds archive on the second attempt ([#389]), which used to pull a plain latest read into the backup tier, at archive cost, after one failure. The router also reads more of the request than before ([#514]): a JSON-RPC batch is judged on its deepest element, and a range method such as trace_filter on its fromBlock.

Requests whose block the router cannot read are not routed to archive on their own: methods that name a block or transaction by hash (eth_getTransactionReceipt, eth_getBlockByHash, debug_traceTransaction, trace_transaction, Tendermint block_by_hash, and the like) and EIP-1898 block objects ({"blockHash": …} or {"blockNumber": …}) on eth_call, eth_getBalance, eth_getStorageAt, eth_getTransactionCount, eth_getCode and eth_getProof. They stay on the primaries unless the caller sends lava-extension: archive. When a caller asks for an extension no node on the router offers, the request is still served and the reply says so in the new Lava-Extension-Unavailable header, counted in smartrouter_extension_unavailable_total ([#450]).

Also new since v1.4.0, the last graduated release

v1.5.0 through v1.5.8 were pre-releases, so an upgrade from v1.4.0 brings everything in their sections below as well. Three of those changes are large enough to repeat here.

  • REST errors are classified the way JSON-RPC errors are. Until v1.5.5 a REST reply counted as the node's answer unless its status was 5xx or 429, so a Horizon transaction_failed, an Aptos resource_not_found or a Cosmos tx-not-found was filed as a success, never classified, and on a write the first of them ended the broadcast while a sibling node was still executing the transaction. Every status outside 2xx is now a node error with its own retry and fault verdict ([#448], v1.5.8): a 404 is data the node does not hold, retried on another node and never scored; 401, 402, 403, 407, 426 and 451 are the endpoint refusing the router, retried elsewhere without blaming the node; 409 is a transaction already known, so in a broadcast the accepting node's reply wins over the duplicates; a 400 "block beyond head" is a missing block; malformed-request statuses (400, 406, 411, 412, 414 to 417, 422, 428, 431) are the caller's error. A 5xx that carries the node's own JSON body reaches the caller as-is instead of a router-made 500, an empty 404 or 405 from a gateway in front of the node is a route refusal rather than the node's answer ([#435], v1.5.5), and a node answer the endpoint is not at fault for logs at DEBUG rather than ERROR. The rule was probed live against twelve REST API families; the full status map is section 9 of docs/ERROR-REGISTRY-DESIGN.md. v1.6.0 continues it: five NODE_* refusal codes are retried on another provider (breaking change 2), a "data not held" answer scores on sync only ([#460]), and an upstream EOF, DNS failure or http-as-https counts against the endpoint ([#518]).
  • Solana replies are read by path, not decoded. Every served reply on a Solana router used to be decoded whole to find result.context.slot; a 5.9 MB getBlock reply cost a full copy and 2.8 ms for a member that getBlock never carries. Since v1.5.4 ([#428]) the slot is read by path, getBlock replies are skipped outright (8 ns, no allocation), a reply whose result is a list is answered from its first byte, and a truncated reply yields no slot instead of a wrong one. The cache write snapshots the reply instead of deep-copying it through JSON, which took 33 ms and 15 to 20 MB per block and now takes about 100 ns ([#431]), and block-height extraction no longer decodes a whole block to run a block-hash rule that could only return 0 ([#430]). In v1.6.0 the frozen-head detector counts a url's highest slot, so a Solana node whose context.slot and finalized slot disagree while both stand still is still taken out of rotation ([#494]).
  • Heavy responses cost less memory and CPU. A node url can ask its upstream for gzip with accept-encoding: gzip ([#432], v1.5.4); the reply is inflated while it is read, with pooled readers, and a block shrinks four to six times on the wire. An upstream body whose length is announced is read into one buffer of that size instead of a chain of growing buffers ([#429]; for 5.9 MB, 5.6 MiB and 3 allocations instead of 13.9 MiB and 31). JSON-RPC block-height extraction stops at 1 MiB, since a larger JSON-RPC reply is a log filter, a full block or a trace, never the tip ([#430]). Since v1.5.6, --cache-max-entry-bytes (default 1 MiB, 0 for no cap) serves a reply above the cap without writing it to the cache, which protects the hit rate of ordinary entries and the cache server's CPU; smartrouter_cache_write_skipped_total{reason="size"} and smartrouter_cache_entry_bytes show what it skips ([#434]). v1.6.0 adds the audit's efficiency fixes ([#511]): regexps compiled once instead of per error, metric increments counted inline instead of on a goroutine each, and the node-errors header built in one buffer.

What's new

  • Chain tracker idle mode ([#505]), off by default. --chain-tracker-idle-after <duration> pauses a chain's dedicated upstream polls once it has served no client relay for that long; --chain-tracker-idle-poll-interval keeps a slow heartbeat instead (0 polls nothing while idle). The next client relay wakes the chain with one poll of every endpoint before it is routed. smartrouter_chain_idle reports the state; the operator guide is docs/CHAIN-TRACKER-IDLE-MODE.md.
  • A url stuck on a frozen head leaves rotation ([#494]). A node that keeps answering with the same latest block while another url of the chain has moved past it is taken out of rotation after 10 × --chain-tracker-poll-divisor tracker cycles (at least 3), and returns once it answers above the block it was stuck on. A provider with every url out is blocked, so traffic moves to the next primary or to the backup tier.
  • One latest-block series per url ([#501]). rpc_endpoint_url_latest_block{spec, apiInterface, endpoint_id, url, internal_path} carries each url's own head (the url label is scheme and host plus an 8-hex fingerprint, never the api key), so an alert can tell a frozen url from one that trails. rpc_endpoint_latest_block now reports a provider's lowest url head. A removed url's series is deleted, not left standing.
  • Per-customer metrics from the project-id header ([#513]). smartrouter_customer_requests_total and smartrouter_customer_requests_failed_total (spec, apiInterface, customer, method), smartrouter_customer_end_to_end_latency_milliseconds (spec, apiInterface, customer) and smartrouter_customer_overflow_total (spec). A request without the header counts as DefaultDappID; each spec admits 512 distinct customers and later values collapse to other. The header never reaches an upstream or the cache key.
  • Native GraphQL API interface ([#480]), first used for Sui. Each root field is its own API, so cross-validation policies, the stateful-write guard, compute units and tip parsing apply per operation instead of collapsing onto one /graphql path.
  • Cross-validation names the plurality on failure ([#491]). lava-cross-validation-plurality-size and lava-cross-validation-plurality-providers carry the largest set of providers that returned the same response when the quorum is not reached; the providers list is empty on a tie.
  • Chain tip regressions are counted ([#506]). smartrouter_latest_block_regressions_total{spec, apiInterface, reason} (reason is consensus or stale_tip) with an ERROR log, and smartrouter_latest_block follows the tip down.
  • GET /debug/sticky-claims ([#442]) reports the sticky-session claim outcomes per endpoint, cumulative since the pod started.
  • Endpoint observations travel on the RESP cache backend ([#390]), so the fleet tracker gate works there too and replicas that share a cache stop each polling every upstream.

Notable fixes

  • Failover and sessions. A blocked provider's disabled endpoint is lent once per request as a real last chance instead of "No pairings available" ([#504]). An upstream connection that answers only 5xx is retired after two in a row, at most once per 5 s, so a load balancer pinned to a dead backend no longer holds the router on 503s until a restart ([#507]). A relay is never handed a WebSocket endpoint ([#487]); a direct RPC session is tied to the endpoint selection chose ([#493]); every provider in `lava...
Read more