Skip to content

Report a lagging name registration as pending, not Completed - #59

Open
requilence wants to merge 1 commit into
mainfrom
fix/ns-operation-completed-without-registry
Open

requilence wants to merge 1 commit into
mainfrom
fix/ns-operation-completed-without-registry

Conversation

@requilence

Copy link
Copy Markdown
Contributor

Fixes GO-7482.

Summary

any-ns-node could report a name registration as Completed to the payment node while the name was never written to the name-service cache. Because IsNameAvailable is cache-only, the name then read as available to every client forever, and because Completed is terminal on the payment-node side, nothing ever retried. Users were charged and the name was permanently unusable.

Two bugs combined:

  • cache.UpdateInCache returned nil (success) when the ENS registry answered with a zero owner address, so the caller believed the cache had been updated.
  • anynsaarpc.GetOperation published status.OperationState before that check, so Completed was reported even when the cache update did nothing.

The trigger is a cross-provider race. The ERC-4337 receipt is polled through the Alchemy bundler, while the registry read goes to a different provider (ConnectToRegistryContract() → ethclient.Dial(config.GethUrl)) at latest block. In the observed incident on nsnode1 the two reads were 99 ms apart and the registry provider had not caught up:

03:27:04.747  any-ns.aa-rpc  operation completed, updating cache  {"FullName": "privanode.any"}
03:27:04.846  any-ns.cache   received owner address               {"Owner addr": "0x0000...0000"}
03:27:04.846  any-ns.cache   name is not registered yet...

Replaying the identical operation 48 days later returned the real owner and wrote the cache row with name_expires exactly 12 months after the original attempt, proving the name had been on chain the entire time. A sub-second provider lag became a permanent failure purely because the failure branch returned nil.

Changes:

  • cache — add an exported ErrNameNotRegistered sentinel and return it whenever the registry has no owner for the name: both on the zero-address read and on the "not found" error path, which previously returned a raw error. Both spellings now mean one thing to callers.
  • anynsaarpc — when the cache update fails with that sentinel, report nsp.OperationState_PendingOrNotFound instead of Completed, so the payment node keeps polling. Every other error keeps the existing failed to update in cache behaviour.
  • anynsaarpc — retry the cache update 3 times, 1 s apart, honouring ctx cancellation. This is a latency optimization only. The correctness mechanism is the payment node's re-poll, which handles a lag of any length; the retry just avoids waiting a whole poll interval for the common sub-second case.
  • anynsrpc — a name missing from the registry is still available. Without this, IsNameAvailable on the readFromCache: false path would turn every free name into an RPC error.
  • contracts — GetOwnerForNamehash built bind.CallOpts{} with a nil Context, so go-ethereum fell back to context.Background() and the caller's ctx was silently discarded. Pre-existing, but the retry is what makes it matter: without it a hung provider is unbounded and cancellation is honoured only between attempts.

The retry deliberately lives in anynsaarpc.GetOperation and not in UpdateInCache. UpdateInCache's other caller is anynsrpc.IsNameAvailable, where on the readFromCache: false path every free name — the common answer during signup — would otherwise cost 3 registry RPCs and 2 s of in-request sleep. BatchIsNameAvailable loops sequentially, so a 10-name batch would go from sub-second to ~20 s, and there is no server-side DRPC handler timeout or client deadline, so it would hang rather than error. Production is readFromCache: true today, but that flag is a candidate for another fix, so this must not be left as a landmine.

The retry knobs are constants, not config. A missing YAML key would leave the count at 0, the loop would never run, addr would stay zero and every registration would report "not registered" — a total cache-population outage from an absent key. The delay is a var rather than a const only so tests can shrink it.

Why PendingOrNotFound and not an error

Both options avoid the Completed latch, but they are not equivalent. Verified against any-pp-node at 77a4f78 rather than reasoning from the enum alone:

  • periodic/periodic_common.go:225-241 processNSOperationPending persists resp.OperationState only in the Error branch. Completed goes terminal, Error is stored as Error, and anything else — PendingOrNotFound included — falls through, returns nil, and the stored op state stays Pending. The item stays in StatusPendingNSOperation and ProcessNSOperation re-handles Pending on every tick. So PendingOrNotFound is never stored and cannot latch.
  • Returning an error instead would go processItems → increaseRetryCountOnError (periodic/periodic.go:76-107, and the v2 twin at periodic/periodic_v2.go:116-152), bumping OpsRetryCount. At maxRetries = 5 the subscription/user is forced to db.StatusError with a Sentry capture and a Slack alert — and once out of StatusPendingNSOperation it is no longer picked up by searchPendingNS. That is a different terminal latch needing manual recovery, reached after only five polls of a condition that is expected and self-healing.

So the error path would trade a silent permanent failure for a noisy one. PendingOrNotFound gives unbounded, cost-free polling and keeps the operation id in the response. Genuine infrastructure failures (Mongo write errors, GetAdditionalNameInfo failures) still return an error, which is exactly where the retry-count and alert escalation belongs.

Known remaining gap

GetOperation still reports Completed with no cache write when the operation has no row in Mongo (anynsaarpc.go:124,143). db.GetOperation returns mongo.ErrNoDocuments verbatim, so on a restored backup or a wiped collection the whole cache block is skipped — byte-for-byte the GO-7482 symptom.

This is deliberately not forced to a non-Completed state: some operations legitimately have no Mongo row (AdminFundUserAccount), and with the row absent there is no FullName to tell a registration from a funding operation. It now emits a log.Warn so the case is at least visible.

Test plan

  • go build ./... — clean
  • go test -count=1 -p 1 ./... — all packages pass
  • cache: zero-address owner returns ErrNameNotRegistered and writes nothing to Mongo
  • cache: the registry "not found" path returns the same sentinel
  • cache: happy path still writes the cache row (existing tests)
  • anynsaarpc: registry never catches up → PendingOrNotFound, no error, UpdateInCache called exactly updateCacheRetryCount times
  • anynsaarpc: registry catches up on the second try → Completed
  • anynsaarpc: closed context stops the retry after one attempt
  • anynsaarpc: a non-sentinel cache failure is still an error
  • anynsrpc: a name missing from the registry reports Available: true with no error
  • Each fix mutation-tested: reverting any one of the five production changes individually makes at least one test fail

Note: plain go test ./... (packages in parallel) is flaky on this repo and on a clean main — every package fixture connects to mongodb://localhost:27017 and drops the same any-ns-test database, so parallel packages wipe each other's data. Which test loses the race varies per run. -p 1 is green on both. Unrelated to this change; listed as a follow-up.

Follow-ups (not in this PR)

  • any-pp-node should handle PendingOrNotFound explicitly. This fix's safety rests on an untested property of the consumer: processNSOperationPending happens not to persist the state, and ProcessNSOperation's switch has no case for it — it would fall to default: "NS operation state not supported". A future "record the state we got" cleanup there would silently convert this fix into a mass StatusError incident. Recommend an explicit case OperationState_PendingOrNotFound: return nil plus a test on the pp-node side.
  • Escalation for a permanent zero owner (GO-7483). A misconfigured AddrRegistry, or GethUrl pointed at the wrong chain, now yields unbounded silent polling: no Sentry, no Slack, no age check, and the user sits in StatusPendingNSOperation — a paid subscription that never activates, with no alarm. Pre-fix that produced "wrongly Completed"; post-fix it is "nothing works, quietly". Stating the tradeoff plainly: this PR removes a silent data-corruption bug and replaces its worst case with a silent stall. The escalation belongs on the pp-node side.
  • Make IsNameAvailable fall back to the contract on a cache miss — wider blast radius, needs its own design decision.
  • Per-package Mongo test databases, to de-flake parallel go test ./....
  • anynsaarpc.go:405 CreateUserOperation builds out and then return nil, nil, discarding it — callers get a nil response with no error.
  • No unique index on ns.cache.name (cache.go:49 // TODO: index it) allows a concurrent double-insert. Pre-existing and not widened by this change.

any-ns-node could tell the payment node that a name registration was Completed
while the name was not written to the name-service cache at all:

- cache.UpdateInCache returned nil when the ENS registry answered with a zero
  owner address, so the caller believed the cache had been updated
- anynsaarpc.GetOperation published status.OperationState before that check, so
  Completed was reported even when the cache update did nothing

The registry is read through a different RPC provider than the one that gives us
the user operation receipt, so it can lag behind for a fraction of a second right
after the registration is mined. That short lag turned into a permanent failure:
IsNameAvailable is cache-only, so the name kept reading as available forever, and
Completed is terminal for the payment node, so nothing ever retried.

- cache: add ErrNameNotRegistered and return it whenever the registry has no
  owner for the name, instead of nil (zero address) or a raw error ("not found")
- anynsaarpc: report PendingOrNotFound instead of Completed when the cache was
  not updated, so the payment node keeps polling
- anynsaarpc: retry the cache update 3 times, 1s apart, honouring ctx
  cancellation. this only shortens the wait - correctness comes from the re-poll
- anynsrpc: a name missing from the registry is still "available", do not turn
  it into an RPC error on the direct-read path
- contracts: pass ctx to the registry CallOpts, it was silently dropped

Operations with no row in Mongo are still reported as Completed without a cache
write. That path has no FullName, so a registration cannot be told apart from a
funding operation there; it now logs a warning.
@github-actions

github-actions Bot commented Sep 1, 2026

Copy link
Copy Markdown

New Coverage 39.4% of statements
Patch Coverage 88.2% of changed statements (30/34)

Coverage provided by https://github.com/seriousben/go-patch-cover-action

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant