Repository navigation
feat(rpcsmartrouter): report the sticky-session claim outcomes on GET /debug/sticky-claims (MAG-3860) - #442
Conversation
… /debug/sticky-claims (MAG-3860) A test cannot tell a cross-pod sticky session from a pod-local one by the reply: Lava-Provider-Address reads the same whether a pod used its own claim or a peer's. The signal is smartrouter_csm_sticky_claims_total at outcome="adopted", and the metrics port needs a port-forward a person opens by hand, so no scheduled run can reach it. The session manager now keeps the seven outcome counts in process beside the counter it already feeds, and GET /debug/sticky-claims reports them per endpoint: ChainID, ApiInterface, SharedSticky (whether the fleet-wide claim registry is wired) and Outcomes, with every outcome present. Without the registry every count stays zero, so SharedSticky is what tells "off" from "never fired". The counts are cumulative since the pod started, like the counter; a test reads the change around the requests it sends. The route is read-only and answers on a router without a cache or --shared-state, the precedent being /debug/provider-scores (MAG-2707). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…and say what adopted does not prove (MAG-3860) Review findings on the route: - Rows now carry PodID (endpointstate.LocalPodID: the pod name and a per-process suffix). The chart's debug route and the k3d NodePort both reach a Service that balances across the router's pods, and the claim registry is only wired where there is more than one pod, so any reading that can be non-zero may come from any of them. PodID lets a test pair its before and after readings, and see a restart between them. - adopted is not proof on its own that a session crossed pods. A pod that drops its local pin (invalidated) keeps the fleet claim by design, so its next request reads its own claim back and counts adopted. The route comment, docs/METRICS.md, the counter's help, the sticky-test README and the tests said otherwise; a test now pins the single-pod path. - The outcomes are a typed enum. Declaring one gives it its count slot, so the route cannot miss an outcome the counter counts, and the slot test ranges over every declared outcome. - The route test drives a peer pod's claim, so its row shows adopted, and sends the pod-local endpoint real sticky traffic, so its zeros mean something. The method and nil-router checks join stateEndpointPaths. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Review: manual pass (high effort) on
|
Correction to my review above: finding 1 is wrongI flagged that the That test already exists in this PR. for outcome := range numStickyOutcomes {
require.NotEmpty(t, stickyOutcomes[outcome], "outcome %d has no label", outcome)It ranges over the declared outcomes, so a new constant without a label fails there, and the final It is also the half of this PR whose tests can actually run: So: no findings from me on this PR. Two things still worth a reply, neither a defect in the code:
|
Third pass on
|
…, and why SharedSticky is false (MAG-3860) Review of #442 (third pass, finding 1): the route's comment and docs/METRICS.md said a pod reads its own claim back as `adopted` after /debug/reset-all. It cannot. The reset clears the pod's local pins (ResetTransientFailureState) and flushes the cache that holds the claims (deps.cache is the backend NewCacheStickyStore wraps; redisstore.Purge unlinks prefix:* which covers the SetStickyIfAbsent keys, and ristrettoStore.Purge clears stickyPins), so the next request re-claims: `claimed`, not `adopted`. Both places now say so, and name the confound that is real: the reset runs only in the pod that received it, so its peers keep their confirmed local pins and keep serving the old upstream from `local_hit` until those pins age out. The pins go through Clear(), so `invalidated` does not move on a reset either; its only increment site is invalidateStickyPin. Finding 3: `SharedSticky: false` conflated "--shared-state not set" with "set, but the cache backend cannot hold claims" (NewCacheStickyStore returned nil). The session manager now records that a registry was requested, and each row carries `SharedStickyReason` with one of three values. The docs say the counts are per process and that a load-balanced read hits one pod, so a reader pins a pod or compares same-PodID readings, and that `adopted` cannot be split into a peer's claim and the pod's own at the registry interface. Tests: TestStickyClaimCounts_AResetReclaimsRatherThanAdopts (session manager: reset plus purge leaves claimed=2, adopted=0, invalidated=0, and the peer's pin survives as local_hit); TestDebugStickyClaims_ResetAllReclaimsRatherThanAdopts (the real POST /debug/reset-all against a registry that is also the flushed cache: the counts survive the reset, the next request counts claimed); TestStickyClaimCounts_SharedStickyRequestedTellsRefusedFromNeverAsked; and the route test now asserts PodID is the host name plus an 8-hex per-process suffix, identical on every row, and reads all three SharedStickyReason values. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…s (MAG-3860) Review of #442 (third pass, finding 2): nothing passed --debug-address, so the route this PR documents was served by no deployed or harness router (the Helm chart 6.8.0 only opens the port; this compose file did not pass the flag). The three harness replicas now start the debug server on :9999, published as :9801-:9803 beside the metrics ports. The README describes the route next to the metrics scrape, including that the counts are per replica and that `SharedStickyReason` reads `--shared-state not set` under the negative control, and run.sh dumps the route per replica on a failed run. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
Third-pass finding 1 is right, and I checked the chain you gave: Finding 3. Finding 2. Confirmed: nothing passed Full |
Re-review on
|
| # | Finding | Status |
|---|---|---|
| Pass 3 #1 | Blocking — the documented /debug/reset-all case cannot happen on any shipped backend |
fixed |
| Pass 3 #2 | relayed, not verified — is anything serving the route? | fixed |
| Pass 3 #3a | the PodID assertion is near-tautological |
fixed |
| Pass 3 #3b | the counts are per-process and the read is load-balanced | fixed |
| Pass 3 #3c | adopted cannot be split at the interface |
disclosed, better than I suggested |
| Pass 3 #3d | SharedSticky: false conflates two states |
fixed |
#1 is fixed with the mechanism spelled out, and the real confound named. METRICS.md now says the reset clears the receiving pod's local pins and flushes the shared claims with them (one cache backs both), so that pod's next request reads claimed. It then names the thing that actually confuses a reader — the reset runs only in the pod that received it, so its peers keep their confirmed local pins and keep serving the old upstream from local_hit until those pins age out, and the fleet is split until then. It even carries the one exception where a claim does survive to be read back: a skipped flush against a cache pod predating the FlushCache RPC, which the reset's own response reveals by omitting cache-be from its cleared list. That is more precise than the finding asked for, and TestDebugStickyClaims_ResetAllReclaimsRatherThanAdopts pins it.
#2 is fixed where it had to be — in a thing that runs. The three harness replicas now start the debug server on :9999, published as :9801-:9803 beside the metrics ports; the README documents the route; and run.sh dumps it per replica on a failed run, which is when anyone will want it. METRICS.md is also honest about the deployment gap: "--debug-address; nothing passes that flag by default, so the deployment or the harness has to."
#3a is fixed properly rather than deleted. The assertion is now structural — HasPrefix(row.PodID, host+"/"), ^[0-9a-f]{8}$ on the suffix, and "one process, one PodID on every row" — so an empty or constant PodID fails, which was the whole point. Cross-process distinctness still is not asserted (it cannot be, in one process), but the shape assertion is what makes the "compare readings only when PodID matches" protocol trustworthy.
#3b is fixed in both places a reader will look, with the operational instruction rather than just the caveat: "The counts are per process and cumulative for its lifetime: the debug Service balances across a router's pods, so pin one pod (a port-forward) or compare two readings only when their PodID matches." The harness README says the same for the three ports.
#3c is answered better than my suggestion. I proposed an adopted_peer / adopted_self split or an owner field. The PR instead states the interface limit plainly — "the registry does not say whose claim it holds, so the route cannot split the two further" — and hands QA a usable substitute: "A rise in adopted on a pod whose invalidated did not move is a peer's claim." That is a deterministic-enough signal from the data that exists, without changing SharedStickyStore.Fetch. Good trade.
I cannot verify from here whether MAG-3860 was updated to record that adopted is not by itself the signal that sharing crossed pods — that was part of the finding and it is a Jira edit. Worth confirming, since the ticket's premise rests on it.
#3d is fixed — SharedStickyReason now distinguishes "--shared-state never requested" from "requested and refused by the cache backend", documented in METRICS.md and surfaced on the route, with the harness README noting it reads --shared-state not set under the negative control.
One stale paragraph, shared across this stack
The MAG-3859 / "does not compile on main" caveat is no longer true — origin/main renamed the duplicate helper apart, and the runs above are on this head with nothing uncommitted.
Still true from the earlier passes
My own retraction stands: first-pass finding 1 (the label table going incomplete without a compile error) was wrong, and I withdrew it. Nothing in this delta changes that.
🤖 Generated with Claude Code
AnnaR-prog
left a comment
There was a problem hiding this comment.
Approving 1c0415b: every finding is fixed. Outside the PR: MAG-3860's description still calls adopted the only cross-pod signal, so the ticket is worth updating.
🤖 Generated with Claude Code
…, and why SharedSticky is false (MAG-3860) Review of #442 (third pass, finding 1): the route's comment and docs/METRICS.md said a pod reads its own claim back as `adopted` after /debug/reset-all. It cannot. The reset clears the pod's local pins (ResetTransientFailureState) and flushes the cache that holds the claims (deps.cache is the backend NewCacheStickyStore wraps; redisstore.Purge unlinks prefix:* which covers the SetStickyIfAbsent keys, and ristrettoStore.Purge clears stickyPins), so the next request re-claims: `claimed`, not `adopted`. Both places now say so, and name the confound that is real: the reset runs only in the pod that received it, so its peers keep their confirmed local pins and keep serving the old upstream from `local_hit` until those pins age out. The pins go through Clear(), so `invalidated` does not move on a reset either; its only increment site is invalidateStickyPin. Finding 3: `SharedSticky: false` conflated "--shared-state not set" with "set, but the cache backend cannot hold claims" (NewCacheStickyStore returned nil). The session manager now records that a registry was requested, and each row carries `SharedStickyReason` with one of three values. The docs say the counts are per process and that a load-balanced read hits one pod, so a reader pins a pod or compares same-PodID readings, and that `adopted` cannot be split into a peer's claim and the pod's own at the registry interface. Tests: TestStickyClaimCounts_AResetReclaimsRatherThanAdopts (session manager: reset plus purge leaves claimed=2, adopted=0, invalidated=0, and the peer's pin survives as local_hit); TestDebugStickyClaims_ResetAllReclaimsRatherThanAdopts (the real POST /debug/reset-all against a registry that is also the flushed cache: the counts survive the reset, the next request counts claimed); TestStickyClaimCounts_SharedStickyRequestedTellsRefusedFromNeverAsked; and the route test now asserts PodID is the host name plus an 8-hex per-process suffix, identical on every row, and reads all three SharedStickyReason values. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…counter (MAG-3860) The metrics reference listed the sticky-sessions gauge but not smartrouter_csm_sticky_claims_total, the counter that says whether a lava-stickiness id crossed replicas. It now does, with what each outcome means, and names GET /debug/sticky-claims (Magma-Devs/smart-router#442): the same counts per endpoint on the debug server, readable without the metrics port, with SharedSticky telling "off" from "never fired". The --debug-address row in the CLI reference links to it. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Jira ticket: MAG-3860
Implements MAG-3860 (Medium) for the sticky-session QA coverage in MAG-3550.
Why
A test cannot tell a cross-pod sticky session from a pod-local one by the reply:
Lava-Provider-Addressreads the same whether a pod used its own claim or a peer's. The signal issmartrouter_csm_sticky_claims_total{outcome="adopted"}, on the metrics port, which needs a port-forward a person opens by hand, so no scheduled run can read it.The route
GET /debug/sticky-claimson the debug server (--debug-address). It returns a flat array of self-describing rows like the other/debug/*routes, one per endpoint:[{"ChainID":"ETH1","ApiInterface":"jsonrpc","PodID":"eth-router-7d9f8b6c5-x2kqp/3fa9c1d2","SharedSticky":true, "Outcomes":{"adopted":1,"claimed":0,"error":0,"invalidated":0,"local_hit":0,"lost_race":0,"no_candidate":0}}]Outcomesholds the seven counts the counter carries, keyed by itsoutcomelabel values, every one present, zero included. They are cumulative since the process started, like the counter, so a test reads the change around the requests it sends.SharedStickysays whether the fleet-wide claim registry is wired (--shared-statewith a cache backend). Without it every count stays0, so this field is what tells "off" from "never fired" (acceptance criterion 2).PodIDnames the process that answered. It isendpointstate.LocalPodID(): the pod name, then a suffix that changes when the process restarts.PodIDmatches. Acceptance criterion 1 asks for the counts "for that router pod".The counts are kept in process by the session manager, beside the counter it already feeds (
recordStickyOutcome), so the route does not depend on the metrics manager or its port. The outcomes are a typed enum. Declaring an outcome is what gives it a count slot, so the route cannot miss one the counter counts. The precedent is/debug/provider-scores(MAG-2707).What
adoptedshows, and what it does notadoptedcounts a session routed by a live claim read from the registry rather than one held in memory. That is how a peer's claim shows up. It is not proof on its own:invalidated) keeps the fleet claim by design, so its next request reads its own claim back and countsadoptedtoo./debug/reset-all, if the claim outlived the reset.adoptedon a pod whoseinvalidateddid not move, with no reset in between, is a peer's claim.The counter's help text,
docs/METRICS.mdanddocker/sticky-test/README.mdsaidadoptedproved a crossed session. This PR corrects all three.Evidence
Live, using the binary built from a89aeb0 against provider_simulator:
--resp-cache-addressesand--shared-state.lava-stickiness: user-42request went twice to pod A and once to pod B, withlava-force-cache-refreshso no reply came from the response cache./debug/sticky-claimsLava-Provider-Address: eth-sim-1, both timesclaimed:1, local_hit:1, SharedSticky:trueLava-Provider-Address: eth-sim-1adopted:1, SharedSticky:true--shared-state)0,SharedSticky:falseThe header is identical on both pods, and only the route shows that B's session crossed pods. Each pod's
PodIDcarried its own per-process suffix, a POST answered 405, and each pod's counts matched itssmartrouter_csm_sticky_claims_total.Tests.
TestStickyClaimCounts_FollowRealResolutionsresolves two pods through one fake registry and checksclaimed,local_hit,adoptedanderror.TestStickyClaimCounts_APodReadingBackItsOwnClaimCountsAdoptedcovers the single-podadoptedabove: claim, drop the local pin, resolve again.TestStickyClaimCounts_WithoutARegistryTheFeatureReadsOffchecks that a router without the registry reads as off, with zero counts.TestStickyClaimCounts_EveryOutcomeHasASlotrecords every declared outcome and reads each back under its own label. An outcome added without a label, or sharing one, fails it.TestDebugStickyClaims_ReportsEachEndpointsOutcomesdrives real resolutions throughGetSessionson direct-RPC session managers. A peer pod claims a session and the router's pod adopts it, while a pod-local endpoint serves sticky traffic that resolves no claim. The test checks the rows,PodID, the flag and all seven counts, and that a second read changes nothing.stateEndpointPaths, the shared GET-only and nil-router checks of the other state routes.PodIDwrong;adoptednot counted, caught in both the unit and the route tests;SharedStickyalways true.Local runs from a clean worktree of a89aeb0:
go build ./...andgo vet ./...pass.protocol/rpcsmartrouter,protocol/lavasessionandprotocol/metricssuites pass.-race.golangci-lint --new-from-rev origin/mainreports 0 issues, test files included.CI runs no
go test.Updated by @AnnaR-prog on 2026-10-05, from review: this paragraph used to say
main'sprotocol/rpcsmartroutertest package does not compile until #433 lands, so the runs above used #433's copy of the file. That is no longer true: #433 (f404669) renamed the duplicatesolanaGetBlockReplyhelper apart, this branch contains it, andgo vet ./protocol/rpcsmartrouter/is clean on1c0415bwith nothing uncommitted, so the package's tests compile as committed, with no overlay.Docs
docs/METRICS.mdnames the route beside the counter and correctsadopted. The product docs change is a companion PR, Magma-Devs/docs#50. It adds the counter todocs/reference/metrics.md, which did not list it, and the route, withPodIDand theadoptedcaveat.docs/reference/cli.mdpoints--debug-addressat it.Delivery
Merging deploys nothing. The route exists only on a router started with
--debug-address, which the k3d nightly sets. It reaches the nightly with the next image built from main.Review round 3 (a89aeb0 → 1c0415b)
/debug/reset-allis not anadoptedcase. The reset clears the receiving pod's local pins and flushes the shared claims with them (one cache backs both), so that pod's next request re-claims (claimed), while its peers keep their confirmed local pins and serve the old upstream fromlocal_hituntil those pins age out;invalidateddoes not move on a reset because it drops the pins throughClear(). The route comment,docs/METRICS.mdand the disambiguation rule now say so (4b2a5ad);TestDebugStickyClaims_ResetAllReclaimsRatherThanAdoptsdrives the realPOST /debug/reset-alland pins it.SharedStickyReasonon each row says whether a registry was never requested (--shared-state not set), requested and wired (registry wired), or requested and refused by the cache backend.PodIDis the host name plus an 8-hex per-process suffix, identical on every row; a reader pins one pod (port-forward) or compares same-PodIDreadings. The peer/self split ofadoptedis a limit of the registry interface (Fetchreturns no claimant); anadopted_peer/adopted_selfsplit is a follow-up.--debug-address, the compose harness included.docker/sticky-testnow starts the debug server on the three replicas (:9801-:9803) and its README documents the route (1c0415b). The Helm chart and the k3d nightly still do not pass the flag; that needsadditionalFlagsor a values change outside this repo.GET /debug/sticky-claims, fieldsPodID,SharedSticky,SharedStickyReason,Outcomes) is handed to Victoria for the MAG-3550 assertions once this merges; the nightly serves it only after--debug-addressis passed.🤖 Generated with Claude Code