Skip to content

feat(rpcsmartrouter): report the sticky-session claim outcomes on GET /debug/sticky-claims (MAG-3860) - #442

Merged
avitenzer merged 6 commits into
mainfrom
feat/MAG-3860-debug-sticky-claims
Oct 5, 2026
Merged

avitenzer merged 6 commits into
mainfrom
feat/MAG-3860-debug-sticky-claims

Conversation

@avitenzer

@avitenzer avitenzer commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

Jira ticket: MAG-3860

Implements MAG-3860 (Medium) for the sticky-session QA coverage in MAG-3550.

Why

A test cannot tell a cross-pod sticky session from a pod-local one by the reply: Lava-Provider-Address reads the same whether a pod used its own claim or a peer's. The signal is smartrouter_csm_sticky_claims_total{outcome="adopted"}, on the metrics port, which needs a port-forward a person opens by hand, so no scheduled run can read it.

The route

GET /debug/sticky-claims on the debug server (--debug-address). It returns a flat array of self-describing rows like the other /debug/* routes, one per endpoint:

[{"ChainID":"ETH1","ApiInterface":"jsonrpc","PodID":"eth-router-7d9f8b6c5-x2kqp/3fa9c1d2","SharedSticky":true,
  "Outcomes":{"adopted":1,"claimed":0,"error":0,"invalidated":0,"local_hit":0,"lost_race":0,"no_candidate":0}}]
  • Outcomes holds the seven counts the counter carries, keyed by its outcome label values, every one present, zero included. They are cumulative since the process started, like the counter, so a test reads the change around the requests it sends.
  • SharedSticky says whether the fleet-wide claim registry is wired (--shared-state with a cache backend). Without it every count stays 0, so this field is what tells "off" from "never fired" (acceptance criterion 2).
  • PodID names the process that answered. It is endpointstate.LocalPodID(): the pod name, then a suffix that changes when the process restarts.
    • The chart reaches the debug port through a Service that balances across the router's pods, and it wires the claim registry only when the router can have more than one pod. So consecutive reads can come from different pods.
    • A test compares two readings only when their PodID matches. Acceptance criterion 1 asks for the counts "for that router pod".
  • The route is GET-only and changes nothing (criterion 3). It is nil-router safe, like its neighbours.

The counts are kept in process by the session manager, beside the counter it already feeds (recordStickyOutcome), so the route does not depend on the metrics manager or its port. The outcomes are a typed enum. Declaring an outcome is what gives it a count slot, so the route cannot miss one the counter counts. The precedent is /debug/provider-scores (MAG-2707).

What adopted shows, and what it does not

adopted counts a session routed by a live claim read from the registry rather than one held in memory. That is how a peer's claim shows up. It is not proof on its own:

  • A pod that drops its local pin (invalidated) keeps the fleet claim by design, so its next request reads its own claim back and counts adopted too.
  • So does a pod after /debug/reset-all, if the claim outlived the reset.
  • A rise in adopted on a pod whose invalidated did not move, with no reset in between, is a peer's claim.

The counter's help text, docs/METRICS.md and docker/sticky-test/README.md said adopted proved a crossed session. This PR corrects all three.

Evidence

Live, using the binary built from a89aeb0 against provider_simulator:

  • Pods A and B shared one throwaway Redis through --resp-cache-addresses and --shared-state.
  • Pod C ran without either.
  • The same lava-stickiness: user-42 request went twice to pod A and once to pod B, with lava-force-cache-refresh so no reply came from the response cache.
pod reply header /debug/sticky-claims
A (shared) Lava-Provider-Address: eth-sim-1, both times claimed:1, local_hit:1, SharedSticky:true
B (shared) Lava-Provider-Address: eth-sim-1 adopted:1, SharedSticky:true
C (no cache, no --shared-state) — all seven 0, SharedSticky:false

The header is identical on both pods, and only the route shows that B's session crossed pods. Each pod's PodID carried its own per-process suffix, a POST answered 405, and each pod's counts matched its smartrouter_csm_sticky_claims_total.

Tests.

  • TestStickyClaimCounts_FollowRealResolutions resolves two pods through one fake registry and checks claimed, local_hit, adopted and error.
  • TestStickyClaimCounts_APodReadingBackItsOwnClaimCountsAdopted covers the single-pod adopted above: claim, drop the local pin, resolve again.
  • TestStickyClaimCounts_WithoutARegistryTheFeatureReadsOff checks that a router without the registry reads as off, with zero counts.
  • TestStickyClaimCounts_EveryOutcomeHasASlot records every declared outcome and reads each back under its own label. An outcome added without a label, or sharing one, fails it.
  • TestDebugStickyClaims_ReportsEachEndpointsOutcomes drives real resolutions through GetSessions on direct-RPC session managers. A peer pod claims a session and the router's pod adopts it, while a pod-local endpoint serves sticky traffic that resolves no claim. The test checks the rows, PodID, the flag and all seven counts, and that a second read changes nothing.
  • The route joins stateEndpointPaths, the shared GET-only and nil-router checks of the other state routes.
  • Each was seen to fail on a deliberate break:
    • PodID wrong;
    • adopted not counted, caught in both the unit and the route tests;
    • an outcome declared without a label;
    • the GET-only check removed;
    • a dropped pin kept;
    • SharedSticky always true.

Local runs from a clean worktree of a89aeb0:

  • go build ./... and go vet ./... pass.
  • The full protocol/rpcsmartrouter, protocol/lavasession and protocol/metrics suites pass.
  • The new tests pass under -race.
  • golangci-lint --new-from-rev origin/main reports 0 issues, test files included.

CI runs no go test.

Updated by @AnnaR-prog on 2026-10-05, from review: this paragraph used to say main's protocol/rpcsmartrouter test package does not compile until #433 lands, so the runs above used #433's copy of the file. That is no longer true: #433 (f404669) renamed the duplicate solanaGetBlockReply helper apart, this branch contains it, and go vet ./protocol/rpcsmartrouter/ is clean on 1c0415b with nothing uncommitted, so the package's tests compile as committed, with no overlay.

Docs

docs/METRICS.md names the route beside the counter and corrects adopted. The product docs change is a companion PR, Magma-Devs/docs#50. It adds the counter to docs/reference/metrics.md, which did not list it, and the route, with PodID and the adopted caveat. docs/reference/cli.md points --debug-address at it.

Delivery

Merging deploys nothing. The route exists only on a router started with --debug-address, which the k3d nightly sets. It reaches the nightly with the next image built from main.

Review round 3 (a89aeb0 → 1c0415b)

  • /debug/reset-all is not an adopted case. The reset clears the receiving pod's local pins and flushes the shared claims with them (one cache backs both), so that pod's next request re-claims (claimed), while its peers keep their confirmed local pins and serve the old upstream from local_hit until those pins age out; invalidated does not move on a reset because it drops the pins through Clear(). The route comment, docs/METRICS.md and the disambiguation rule now say so (4b2a5ad); TestDebugStickyClaims_ResetAllReclaimsRatherThanAdopts drives the real POST /debug/reset-all and pins it.
  • SharedStickyReason on each row says whether a registry was never requested (--shared-state not set), requested and wired (registry wired), or requested and refused by the cache backend.
  • Reads are per process. PodID is the host name plus an 8-hex per-process suffix, identical on every row; a reader pins one pod (port-forward) or compares same-PodID readings. The peer/self split of adopted is a limit of the registry interface (Fetch returns no claimant); an adopted_peer/adopted_self split is a follow-up.
  • Serving the route. Nothing passed --debug-address, the compose harness included. docker/sticky-test now starts the debug server on the three replicas (:9801-:9803) and its README documents the route (1c0415b). The Helm chart and the k3d nightly still do not pass the flag; that needs additionalFlags or a values change outside this repo.
  • Definition-of-done item 3: the route name (GET /debug/sticky-claims, fields PodID, SharedSticky, SharedStickyReason, Outcomes) is handed to Victoria for the MAG-3550 assertions once this merges; the nightly serves it only after --debug-address is passed.
  • Companion docs PR docs(reference): name GET /debug/sticky-claims and the sticky-claims counter (MAG-3860) docs#50 carries the reset-all wording and needs the same corrections.

🤖 Generated with Claude Code

… /debug/sticky-claims (MAG-3860)

A test cannot tell a cross-pod sticky session from a pod-local one by the
reply: Lava-Provider-Address reads the same whether a pod used its own
claim or a peer's. The signal is smartrouter_csm_sticky_claims_total at
outcome="adopted", and the metrics port needs a port-forward a person
opens by hand, so no scheduled run can reach it.

The session manager now keeps the seven outcome counts in process beside
the counter it already feeds, and GET /debug/sticky-claims reports them
per endpoint: ChainID, ApiInterface, SharedSticky (whether the fleet-wide
claim registry is wired) and Outcomes, with every outcome present. Without
the registry every count stays zero, so SharedSticky is what tells "off"
from "never fired". The counts are cumulative since the pod started, like
the counter; a test reads the change around the requests it sends. The
route is read-only and answers on a router without a cache or
--shared-state, the precedent being /debug/provider-scores (MAG-2707).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…and say what adopted does not prove (MAG-3860)

Review findings on the route:

- Rows now carry PodID (endpointstate.LocalPodID: the pod name and a
  per-process suffix). The chart's debug route and the k3d NodePort both
  reach a Service that balances across the router's pods, and the claim
  registry is only wired where there is more than one pod, so any reading
  that can be non-zero may come from any of them. PodID lets a test pair
  its before and after readings, and see a restart between them.
- adopted is not proof on its own that a session crossed pods. A pod that
  drops its local pin (invalidated) keeps the fleet claim by design, so
  its next request reads its own claim back and counts adopted. The route
  comment, docs/METRICS.md, the counter's help, the sticky-test README and
  the tests said otherwise; a test now pins the single-pod path.
- The outcomes are a typed enum. Declaring one gives it its count slot, so
  the route cannot miss an outcome the counter counts, and the slot test
  ranges over every declared outcome.
- The route test drives a peer pod's claim, so its row shows adopted, and
  sends the pod-local endpoint real sticky traffic, so its zeros mean
  something. The method and nil-router checks join stateEndpointPaths.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@AnnaR-prog

Copy link
Copy Markdown
Contributor

Review: manual pass (high effort) on a89aeb0

Reviewed directly against the PR base 887e2bc (no /code-review fan-out). No correctness defects found. One low finding on the new label table, plus two invariants I checked because the docs lean on them.

1. Low: the label table can go incomplete without a compile error

The comment on stickyOutcome claims the structure makes omissions impossible:

Declaring an outcome here is what gives it a count slot, so the route cannot miss one the metric counts.

True for the count slot — [numStickyOutcomes]atomic.Uint64 grows with the iota block. Not true for the label:

var stickyOutcomes = [numStickyOutcomes]string{
	stickyOutcomeLocalHit: "local_hit",
	...
}

A new constant added before numStickyOutcomes without a matching entry compiles cleanly and yields the zero value. recordStickyOutcome then calls RecordStickyClaim(chain, api, ""), creating a permanent outcome="" series on smartrouter_csm_sticky_claims_total, and StickyClaimCounts() returns a map with an empty-string key that /debug/sticky-claims serves as "": N. The closed-set property the comment relies on — and that the doc's "every outcome is present, zero included" promises — is the half that is not enforced.

Cheapest structural fix is an init() or a TestStickyOutcomeLabelsAreComplete that asserts every index is non-empty and the labels are distinct. One loop, and it makes the comment's claim true.

Verified and not flagged

  • "Without the registry every count stays zero" holds. I checked this because both METRICS.md and the handler comment make SharedSticky the discriminator between "off" and "never fired", and local_hit looked like it could fire on a pod-local sticky hit. It cannot: every recordStickyOutcome call site — the six in resolveStickyPin and the one in invalidateStickyPin — sits inside if stickiness != "" && selectedProvider == "" && csm.sharedSticky != nil at consumer_session_manager.go:1643. invalidateStickyPin is reached only from that block's defer.
  • The adopted semantic change is documentation only. No stickyOutcomeAdopted call site moved; the diff corrects three descriptions of existing behaviour (a pod reading back its own claim after dropping the local copy). Worth saying explicitly because the METRICS.md and Prometheus Help text both read like a behaviour change.
  • One row per endpoint, no duplicates. rpsr.sessionManagers is map[string]*ConsumerSessionManager keyed by rpcEndpoint.Key() (chainID-apiInterface), so ChainID + ApiInterface uniquely identifies a row and two readings from the same PodID are comparable field-for-field.
  • Holding deps.router.mu in the handler is the established pattern — every other /debug/* handler in buildDebugMux does the same, and the work under the lock here is a map walk plus seven atomic loads per endpoint.
  • StickyClaimCounts() reads csm.sharedSticky without csm.lock, which matches the existing read at :1643 and the lock-free write in SetSharedStickyStore (construction-time, once). Consistent with the file, so not flagged — but if that field ever becomes settable after boot, this handler is a new reader to remember.
  • Counters are cumulative and survive /debug/reset-all, which is what the doc's "compare two readings" advice needs.
  • PodID caveat is the right one to document. The debug Service load-balances across pods, so the doc telling readers to compare only same-PodID readings is the difference between this endpoint being usable and misleading.

Merge order

All nine of #436–#444 branch from 887e2bc and none contains another, so each "Files changed" is its own. git merge-tree finds no conflict with any sibling. Shared files, all clean: protocol/lavasession/consumer_session_manager.go with #439 and #443, protocol/metrics/smartrouter_metrics_manager.go with #443, protocol/rpcsmartrouter/rpcsmartrouter.go with #439 and #441, docs/METRICS.md with #436, #440 and #443 (different table sections). No ordering constraint.

Base

go test ./protocol/rpcsmartrouter/ does not build at 887e2bc: solanaGetBlockReply is declared twice (direct_rpc_integration_test.go:872 and solana_context_slot_bench_test.go:75). Main fixed it in f404669, whose message notes CI never compiles the test binary. So debug_sticky_claims_test.go and the debug_server_test.go additions have not run anywhere — not on CI, not locally at this head. Rebase onto current main. I confirmed the test binary compiles cleanly once f404669 is applied (go vet ./protocol/rpcsmartrouter/ passes on this head with it cherry-picked), so a rebase is the whole fix. (protocol/lavasession, where sticky_claim_counts_test.go lives, is green — I ran it.)

@AnnaR-prog

Copy link
Copy Markdown
Contributor

Correction to my review above: finding 1 is wrong

I flagged that the stickyOutcomes label table could go incomplete without a compile error, and suggested "an init() or a TestStickyOutcomeLabelsAreComplete that asserts every index is non-empty and the labels are distinct."

That test already exists in this PR. protocol/lavasession/sticky_claim_counts_test.go, TestStickyClaimCounts_EveryOutcomeHasASlot:

for outcome := range numStickyOutcomes {
    require.NotEmpty(t, stickyOutcomes[outcome], "outcome %d has no label", outcome)

It ranges over the declared outcomes, so a new constant without a label fails there, and the final require.Equal on the whole map catches two outcomes sharing a label. That is exactly the invariant I said was unenforced. I had only read this PR's test names, not the bodies — my mistake, and the finding does not stand.

It is also the half of this PR whose tests can actually run: protocol/lavasession compiles at this base, and go test ./protocol/lavasession/ -run Sticky is green at a89aeb0 (I ran it). The protocol/rpcsmartrouter half — debug_sticky_claims_test.go and the debug_server_test.go addition — still cannot run until this rebases onto current main, for the solanaGetBlockReply reason in my Base note.

So: no findings from me on this PR. Two things still worth a reply, neither a defect in the code:

  • MAG-3860's definition of done item 3 — "The route name is passed to Victoria, so the sticky-session tests under MAG-3550 can be written against it" — is not mentioned in the PR body. Item 2 (product docs under docs/reference/) is covered, disclosed as companion PR docs(reference): name GET /debug/sticky-claims and the sticky-claims counter (MAG-3860) docs#50. Item 3 has no equivalent, so merging this and closing the ticket would drop it silently. One line in the PR body, or a comment on MAG-3860, closes it.
  • /debug/sticky-claims joins stateEndpointPaths, so TestDebugStateEndpoints_MethodNotAllowed covers acceptance criterion 3 (read-only, non-GET returns 405) without a separate test. Worth knowing that the criterion is covered there rather than in the new file.

@AnnaR-prog

Copy link
Copy Markdown
Contributor

Third pass on a89aeb0: retracting "no findings", one blocking

My correction above ended with "no findings from me on this PR." That is now wrong. An adversarial pass found a false statement in the operator docs this PR ships, and I verified it against both cache backends myself.

1. Blocking — the documented /debug/reset-all case cannot happen on any shipped backend

Two places state it without hedging:

  • protocol/rpcsmartrouter/rpcsmartrouter.go:1735 — "A pod that dropped its local copy of its own claim (invalidated, or /debug/reset-all) reads it back as adopted too."
  • docs/METRICS.md:424 — "adopted (... or this pod's own read back after it dropped its local copy — see invalidated — or after /debug/reset-all)"

/debug/reset-all flushes the store that holds the claims, so there is nothing to read back. The chain:

  1. rpcsmartrouter.go:1443-1445 calls deps.cache.Flush(flushCtx).
  2. deps.cache is options.cache — the same value passed to performance.NewCacheStickyStore(options.cache) at :2646. One backend, both roles.
  3. Both implementations destroy the pins:
    • RESP: redisstore.Store.Purge matches s.prefix + ":*", and SetStickyIfAbsent writes at s.key(key) = prefix + ":" + key. Inside the purge.
    • cache-be (ristretto): ristrettoStore.Purge ends with r.cs.stickyPins.clear() outright.

So after a reset-all the fleet claim is gone and the next sticky request re-claims — claimed, not adopted. The documented case exists only in the codes.Unimplemented rolling-upgrade branch at :1450, where the cache pod predates the FlushCache RPC. Nothing tests it: sticky_claim_counts_test.go:49-59 covers only the invalidateStickyPin variant, where invalidated really does move.

Why this blocks rather than being a wording nit: it is load-bearing. It is one of the two reasons the PR gives for adopted being ambiguous, and it is the reason the disambiguation rule carries the escape clause "with no reset in between." A QA engineer writing the MAG-3550 assertions will design around a confound that is not there, and the same wording is going into Magma-Devs/docs#50, so it reaches customers. Deleting the reset-all clause from both places fixes it.

While that line is open, the confound that is real is a different one, and it is not documented: /debug/reset-all runs only in the process that received the request. That pod clears its local pins and flushes the shared claims, while its peers keep their confirmed local pins and keep serving the old upstream from local_hit — and the receiving pod re-claims, possibly a different upstream, because its optimizer was reset too. The fleet is split until those peer pins age out. Also worth knowing: ResetTransientFailureState clears the sticky table via Clear(), which bypasses invalidateStickyPin, so invalidated does not move on that path — the only increment site is consumer_session_manager.go:4110.

2. Relayed, not verified by me — is there anything serving the route?

I could not check this within my constraints, so treat it as a question rather than a finding. The PR body says "the route exists only on a router started with --debug-address, which the k3d nightly sets." The adversarial pass reports that (a) git grep -i k3d in this repo matches only bytes inside a PNG, (b) the Helm chart's router deployment never passes --debug-address — it opens containerPort: 9999 named debug and a Service targeting it, but the router only starts the debug server when DebugAddress != "", so with debug.enabled=true the port is open with nothing listening unless an operator injects the flag through additionalFlags, and (c) docker/sticky-test/docker-compose.sticky.yml does not pass it either, while this PR edits that harness's README and leaves README.md:52-58 still telling the reader to scrape :7801/metrics.

If that holds, MAG-3860's definition-of-done item 3 lands on a route no deployed router serves until the chart or the nightly's values change — worth either a chart/harness change or a line in the body naming who does it.

3. Worth knowing before QA builds on it

  • The PodID assertion is near-tautological. debug_sticky_claims_test.go:128 compares row.PodID against endpointstate.LocalPodID() — the same memoised function that produced it. It passes for any value, "" included, and nothing checks that two processes get different IDs. Cross-process distinctness is the only property the "compare readings only when PodID matches" protocol rests on.
  • The counts are per-process and the read is load-balanced. With replicas, a single read hits a random pod, and the route offers no enumeration, aggregation or way to pin a read — so a MAG-3550 assertion built on it is probabilistic. The PR discloses the symptom ("compare two readings only when their PodID matches") but not this consequence.
  • adopted still cannot be split at the interface. SharedStickyStore.Fetch returns (provider, epoch, found, err) with no claimant identity, so peer-vs-self is unknowable where the count is taken. This PR correctly retracts the old "took a claim a peer had made" wording in three places, which disproves the ticket's premise that adopted is the signal that sharing crossed pods — and MAG-3860 is not updated to say so. An adopted_peer / adopted_self split, or an owner field, is what would make the deterministic assertion possible.
  • SharedSticky: false conflates two states: --shared-state never asked for, and asked for but the backend cannot hold claims (NewCacheStickyStore returns nil, with only a one-shot warning log to tell them apart).

Still standing from my earlier comments

The enum conversion is fully contained — every user of the changed names is inside the diff (checked with a positive control). TestStickyClaimCounts_EveryOutcomeHasASlot does enforce label completeness, and protocol/lavasession is green at this head (go test ./protocol/lavasession/ -run Sticky), so that half is proven. /debug/sticky-claims joining stateEndpointPaths covers acceptance criterion 3. The rpcsmartrouter half still cannot run until this rebases onto current main.

avitenzer and others added 3 commits September 30, 2026 19:08
…, and why SharedSticky is false (MAG-3860)

Review of #442 (third pass, finding 1): the route's comment and docs/METRICS.md
said a pod reads its own claim back as `adopted` after /debug/reset-all. It
cannot. The reset clears the pod's local pins (ResetTransientFailureState) and
flushes the cache that holds the claims (deps.cache is the backend
NewCacheStickyStore wraps; redisstore.Purge unlinks prefix:* which covers the
SetStickyIfAbsent keys, and ristrettoStore.Purge clears stickyPins), so the
next request re-claims: `claimed`, not `adopted`. Both places now say so, and
name the confound that is real: the reset runs only in the pod that received
it, so its peers keep their confirmed local pins and keep serving the old
upstream from `local_hit` until those pins age out. The pins go through
Clear(), so `invalidated` does not move on a reset either; its only increment
site is invalidateStickyPin.

Finding 3: `SharedSticky: false` conflated "--shared-state not set" with
"set, but the cache backend cannot hold claims" (NewCacheStickyStore returned
nil). The session manager now records that a registry was requested, and each
row carries `SharedStickyReason` with one of three values. The docs say the
counts are per process and that a load-balanced read hits one pod, so a reader
pins a pod or compares same-PodID readings, and that `adopted` cannot be split
into a peer's claim and the pod's own at the registry interface.

Tests: TestStickyClaimCounts_AResetReclaimsRatherThanAdopts (session manager:
reset plus purge leaves claimed=2, adopted=0, invalidated=0, and the peer's
pin survives as local_hit); TestDebugStickyClaims_ResetAllReclaimsRatherThanAdopts
(the real POST /debug/reset-all against a registry that is also the flushed
cache: the counts survive the reset, the next request counts claimed);
TestStickyClaimCounts_SharedStickyRequestedTellsRefusedFromNeverAsked; and the
route test now asserts PodID is the host name plus an 8-hex per-process suffix,
identical on every row, and reads all three SharedStickyReason values.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…s (MAG-3860)

Review of #442 (third pass, finding 2): nothing passed --debug-address, so the
route this PR documents was served by no deployed or harness router (the Helm
chart 6.8.0 only opens the port; this compose file did not pass the flag). The
three harness replicas now start the debug server on :9999, published as
:9801-:9803 beside the metrics ports. The README describes the route next to
the metrics scrape, including that the counts are per replica and that
`SharedStickyReason` reads `--shared-state not set` under the negative control,
and run.sh dumps the route per replica on a failed run.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@avitenzer

Copy link
Copy Markdown
Contributor Author

Third-pass finding 1 is right, and I checked the chain you gave: deps.cache is the same options.cache that NewCacheStickyStore wraps, both purges drop the pins, and Clear() never reaches invalidateStickyPin. Fixed in 4b2a5ad: the reset-all clause is gone from the route comment and METRICS.md, the disambiguation rule lost its "no reset in between" escape, and both now describe the confound that is real. The reset runs in one pod, which re-claims (claimed), while its peers keep their confirmed pins and serve the old upstream from local_hit until they age out, with invalidated not moving. TestDebugStickyClaims_ResetAllReclaimsRatherThanAdopts drives the real POST /debug/reset-all with a registry that is also the flushed cache, and TestStickyClaimCounts_AResetReclaimsRatherThanAdopts pins it at the session manager; both fail if the flush stops emptying the registry.

Finding 3. PodID is now asserted as host name plus an 8-hex per-process suffix, identical on every row; the docs say the counts are per process and a load-balanced read hits one pod. SharedSticky: false no longer conflates the two states: each row carries SharedStickyReason (registry wired / --shared-state not set / --shared-state set, but the cache backend cannot hold claims), from a flag the session manager records when a registry is requested. The peer/self split of adopted is documented as a limit of the registry interface; an adopted_peer/adopted_self split needs the store to return the claimant, which is a follow-up.

Finding 2. Confirmed: nothing passed --debug-address, the compose harness included. 1c0415b starts the debug server on the three harness replicas (:9801-:9803) and the README documents the route beside the metrics scrape. The chart and the nightly still need the flag; that is outside this repo. The definition-of-done item 3 sentence is in the PR body now, and Magma-Devs/docs#50 needs the same reset-all corrections.

Full rpcsmartrouter and lavasession suites pass on the branch with current main merged in (b68b1c2, so the reviewed commits keep their SHAs); the changed tests pass under -race; golangci-lint reports 0 new issues.

@AnnaR-prog

Copy link
Copy Markdown
Contributor

Re-review on 1c0415b — everything fixed, including the four "worth knowing" items

Re-reviewed the two author commits since my 2026-09-27 pass (4b2a5ad8, 1c0415b9) plus the origin/main merge. Both sticky-claims tests pass, and there is a new one named for the finding: TestDebugStickyClaims_ResetAllReclaimsRatherThanAdopts.

Prior findings

# Finding Status
Pass 3 #1 Blocking — the documented /debug/reset-all case cannot happen on any shipped backend fixed
Pass 3 #2 relayed, not verified — is anything serving the route? fixed
Pass 3 #3a the PodID assertion is near-tautological fixed
Pass 3 #3b the counts are per-process and the read is load-balanced fixed
Pass 3 #3c adopted cannot be split at the interface disclosed, better than I suggested
Pass 3 #3d SharedSticky: false conflates two states fixed

#1 is fixed with the mechanism spelled out, and the real confound named. METRICS.md now says the reset clears the receiving pod's local pins and flushes the shared claims with them (one cache backs both), so that pod's next request reads claimed. It then names the thing that actually confuses a reader — the reset runs only in the pod that received it, so its peers keep their confirmed local pins and keep serving the old upstream from local_hit until those pins age out, and the fleet is split until then. It even carries the one exception where a claim does survive to be read back: a skipped flush against a cache pod predating the FlushCache RPC, which the reset's own response reveals by omitting cache-be from its cleared list. That is more precise than the finding asked for, and TestDebugStickyClaims_ResetAllReclaimsRatherThanAdopts pins it.

#2 is fixed where it had to be — in a thing that runs. The three harness replicas now start the debug server on :9999, published as :9801-:9803 beside the metrics ports; the README documents the route; and run.sh dumps it per replica on a failed run, which is when anyone will want it. METRICS.md is also honest about the deployment gap: "--debug-address; nothing passes that flag by default, so the deployment or the harness has to."

#3a is fixed properly rather than deleted. The assertion is now structural — HasPrefix(row.PodID, host+"/"), ^[0-9a-f]{8}$ on the suffix, and "one process, one PodID on every row" — so an empty or constant PodID fails, which was the whole point. Cross-process distinctness still is not asserted (it cannot be, in one process), but the shape assertion is what makes the "compare readings only when PodID matches" protocol trustworthy.

#3b is fixed in both places a reader will look, with the operational instruction rather than just the caveat: "The counts are per process and cumulative for its lifetime: the debug Service balances across a router's pods, so pin one pod (a port-forward) or compare two readings only when their PodID matches." The harness README says the same for the three ports.

#3c is answered better than my suggestion. I proposed an adopted_peer / adopted_self split or an owner field. The PR instead states the interface limit plainly — "the registry does not say whose claim it holds, so the route cannot split the two further" — and hands QA a usable substitute: "A rise in adopted on a pod whose invalidated did not move is a peer's claim." That is a deterministic-enough signal from the data that exists, without changing SharedStickyStore.Fetch. Good trade.

I cannot verify from here whether MAG-3860 was updated to record that adopted is not by itself the signal that sharing crossed pods — that was part of the finding and it is a Jira edit. Worth confirming, since the ticket's premise rests on it.

#3d is fixed — SharedStickyReason now distinguishes "--shared-state never requested" from "requested and refused by the cache backend", documented in METRICS.md and surfaced on the route, with the harness README noting it reads --shared-state not set under the negative control.

One stale paragraph, shared across this stack

The MAG-3859 / "does not compile on main" caveat is no longer true — origin/main renamed the duplicate helper apart, and the runs above are on this head with nothing uncommitted.

Still true from the earlier passes

My own retraction stands: first-pass finding 1 (the label table going incomplete without a compile error) was wrong, and I withdrew it. Nothing in this delta changes that.

🤖 Generated with Claude Code

https://claude.ai/code/session_0154pmYvf8T1kbvB9pYjihvq

@AnnaR-prog AnnaR-prog left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving 1c0415b: every finding is fixed. Outside the PR: MAG-3860's description still calls adopted the only cross-pod signal, so the ticket is worth updating.

🤖 Generated with Claude Code

@avitenzer
avitenzer merged commit 764559f into main Oct 5, 2026
avitenzer added a commit that referenced this pull request Oct 5, 2026
…, and why SharedSticky is false (MAG-3860)

Review of #442 (third pass, finding 1): the route's comment and docs/METRICS.md
said a pod reads its own claim back as `adopted` after /debug/reset-all. It
cannot. The reset clears the pod's local pins (ResetTransientFailureState) and
flushes the cache that holds the claims (deps.cache is the backend
NewCacheStickyStore wraps; redisstore.Purge unlinks prefix:* which covers the
SetStickyIfAbsent keys, and ristrettoStore.Purge clears stickyPins), so the
next request re-claims: `claimed`, not `adopted`. Both places now say so, and
name the confound that is real: the reset runs only in the pod that received
it, so its peers keep their confirmed local pins and keep serving the old
upstream from `local_hit` until those pins age out. The pins go through
Clear(), so `invalidated` does not move on a reset either; its only increment
site is invalidateStickyPin.

Finding 3: `SharedSticky: false` conflated "--shared-state not set" with
"set, but the cache backend cannot hold claims" (NewCacheStickyStore returned
nil). The session manager now records that a registry was requested, and each
row carries `SharedStickyReason` with one of three values. The docs say the
counts are per process and that a load-balanced read hits one pod, so a reader
pins a pod or compares same-PodID readings, and that `adopted` cannot be split
into a peer's claim and the pod's own at the registry interface.

Tests: TestStickyClaimCounts_AResetReclaimsRatherThanAdopts (session manager:
reset plus purge leaves claimed=2, adopted=0, invalidated=0, and the peer's
pin survives as local_hit); TestDebugStickyClaims_ResetAllReclaimsRatherThanAdopts
(the real POST /debug/reset-all against a registry that is also the flushed
cache: the counts survive the reset, the next request counts claimed);
TestStickyClaimCounts_SharedStickyRequestedTellsRefusedFromNeverAsked; and the
route test now asserts PodID is the host name plus an 8-hex per-process suffix,
identical on every row, and reads all three SharedStickyReason values.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@avitenzer
avitenzer deleted the feat/MAG-3860-debug-sticky-claims branch October 5, 2026 12:41
AnnaR-prog pushed a commit to Magma-Devs/docs that referenced this pull request Oct 7, 2026
…counter (MAG-3860)

The metrics reference listed the sticky-sessions gauge but not
smartrouter_csm_sticky_claims_total, the counter that says whether a
lava-stickiness id crossed replicas. It now does, with what each outcome
means, and names GET /debug/sticky-claims (Magma-Devs/smart-router#442):
the same counts per endpoint on the debug server, readable without the
metrics port, with SharedSticky telling "off" from "never fired". The
--debug-address row in the CLI reference links to it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants