WALM-295: Owner-scoped memory read API (namespaces/memories/agents)

[harrymove-ctrl]

Summary

Adds the owner-scoped, cursor-paginated read API Console syncs from (WALM-295):

• GET /v1/owners/{owner}/namespaces — rollup (memory count, storage used) per namespace, paginated.
• GET /v1/owners/{owner}/memories — keyset pagination on (owner, updated_at, id), includes agent_id/package_id/status per memory.
• GET /v1/owners/{owner}/agents — live on-chain read of the account's delegate keys, short-TTL cached.

All three reuse the existing signature-verified auth (/api/restore's pattern) — the {owner} path segment must match the authenticated identity or the request gets 403.

Also plumbs agent_id/package_id into all 5 real write paths of insert_vector (remember, remember-manual/analyze, restore recovery) so the new memories response has real provenance data, not just nulls.

Migration

New migration set 010–013 on vector_entries (adds updated_at, agent_id, package_id, and a (owner, updated_at, id) index). Split across 4 separate transactions rather than one, so the backfill UPDATE only needs a ROW EXCLUSIVE lock (doesn't block reads) instead of holding ACCESS EXCLUSIVE across the whole backfill.

Design doc / plan

• docs/superpowers/specs/2026-08-04-memory-read-api-design.md
• docs/superpowers/plans/2026-08-04-walm-295-memory-read-api.md
• docs/api/memory-read-api.md — Console-facing contract (response shapes + auth headers)

Test plan

• cargo test --bins — 429 passed, 0 failed, 30 ignored
• cargo build --bins / cargo fmt --check — clean
• Verified against local Postgres+Redis (docker compose -f services/server/docker-compose.yml) throughout — migrations apply cleanly, keyset pagination confirmed gap/duplicate-free including a forced updated_at tie, planner confirmed index usage (no full-table scan)

Follow-ups (non-blocking, tracked here for visibility)

• AppState.delegate_keys_cache (the new in-process cache backing /agents) has no eviction/cap — grows unbounded, keyed by every distinct owner that calls /agents over the server's lifetime. Slow memory growth only, not a correctness issue. Needs a periodic sweep or bounded/LRU cache.
• docs/api/memory-read-api.md's 429 example uses illustrative layer/retry_after_seconds values that don't match the real constants in rate_limit.rs (real layers are snake_case like account_burst, fixed at 60s/300s). Doc-only, no functional impact.

Scope note

WALM-296 (per-memory expiry: end_epoch/expires_at) extends the same memories response this PR creates and is tracked as a separate follow-on PR — not included here.

[ducnmm]

Found five issues that should be addressed before merge:

1. High — /agents fails on testnet. list_owner_agents always calls the JSON-RPC-only list_delegate_keys_cached, even though testnet requires SUI_GRPC_URL and JSON-RPC is explicitly unsupported. Auth can succeed through gRPC, then every cold/expired /agents lookup returns 500. Please implement delegate-key listing through state.sui_grpc_client, with JSON-RPC only as a supported fallback.

2. High — /namespaces does not support incremental sync. Its updated_after cursor is only the last namespace name (HAVING namespace > $2). If Console has synced alpha and a later write changes alpha's count/storage, polling with the prior cursor never returns that changed rollup. This conflicts with WALM-295's incremental-listing scope. The cursor needs change/high-watermark semantics, or the API contract and ticket scope need to be revised explicitly.

3. High — /memories does not return a terminal watermark. next_cursor is generated only when has_more. An owner whose result fits in one page receives no checkpoint at all, and after the final page of a multi-page traversal the latest row is likewise not checkpointed. Clients therefore cannot poll incrementally without rereading data. Return the last emitted cursor independently of has_more, or add a separate continuation watermark.

4. High — migrations 010–012 have a rolling-deploy race. Migration 010 adds nullable updated_at without a default, 011 backfills and commits, and only 012 installs the default plus NOT NULL. An old replica can insert a new NULL row between 011 and 012, causing SET NOT NULL to fail and new replicas to crash-loop. Install DEFAULT NOW() when the column is introduced, before the backfill, then enforce non-nullability.

5. Medium — incremental sync cannot communicate deletion. Forget and expired-blob cleanup hard-delete vector_entries; later updated_after reads only scan rows that still exist. Console can retain a deleted memory indefinitely. Add tombstones/change events, or explicitly require and support periodic full reconciliation.

CI is green, but current tests do not cover testnet gRPC behavior, rolling deployment, or incremental lifecycle behavior after update/delete.

[ducnmm]

Found two blocking issues and two pagination-contract issues on the current head (c09bdd05):

1. High — rolling deployment can fail the updated_at NOT NULL migration. services/server/migrations/010_memory_read_api_columns.sql:24 adds the column nullable without a default. Existing replicas can insert NULL rows while 011 backfills; 012 then fails at services/server/migrations/012_memory_read_api_updated_at_not_null.sql:13. Set the default before the backfill so concurrent writers cannot create new NULLs.

2. High — /agents bypasses the configured gRPC client. services/server/src/routes/memory_read.rs:357-363 passes only the HTTP client/JSON-RPC URL, and services/server/src/storage/sui.rs:221-227 always uses JSON-RPC. Testnet requires gRPC, so cold-cache /agents requests can return 500 despite auth succeeding through the gRPC client. Route this through the configured Sui client path.

3. Medium — final memory page drops the sync watermark. memory_read.rs:249-253 returns a cursor only when another row already exists. One-page or completed traversals end with next_cursor: null, so clients have no (updated_at, id) watermark for future incremental polling.

4. Medium — namespace continuation cannot surface updates to earlier-sorting namespaces. memory_read.rs:93-100 uses HAVING namespace > cursor and does not filter/order by MAX(updated_at). Once a namespace sorts before the cursor, later changes to it are invisible to that incremental traversal.

Verification: complete diff and surrounding auth/storage/migrations reviewed; GitHub checks and diff check pass; focused cursor and delegate-key tests pass. DB-backed migration and handler tests could not be run without DATABASE_URL.

[ducnmm]

Additional review findings on current HEAD (c09bdd05):

6. High — list_owner_agents signature mismatch with gRPC path
list_delegate_keys_cached takes (cache, http_client, rpc_url, account_id, package_id) but does not accept the grpc_client that verify_delegate_key_onchain already uses for auth. Fixing issue #1 (testnet gRPC support) requires threading grpc_client into this function signature as well.

7. Medium — SNAPSHOT_VERSION = 1 may need a bump
The 010–013 migrations add agent_id/package_id as NULL for all pre-existing rows. Console receiving agent_id: null for old memories may misinterpret this as "no agent" rather than "unknown/pre-migration". If this constitutes a semantic change requiring Console to re-reconcile, SNAPSHOT_VERSION should be bumped.

8. Medium — Namespace response missing updated_at
The SQL computes MAX(updated_at) but the response struct NamespaceSummary does not include it. Console has no way to know when a namespace's rollup last changed, making it harder to build incremental sync on top of the namespace cursor.

9. Low — Memory items don't expose updated_at in the response
The query fetches updated_at for cursor encoding, but MemoryItem excludes it from the serialized response. If Console needs "last updated" timestamps or snapshot-diffing, the data is discarded unnecessarily.

---

Regarding existing findings #1–#5 from your earlier reviews: I verified them against the current HEAD and they all still apply. #1 (testnet gRPC) and #4 (migration rolling-deploy race) remain the highest priority blockers before merge.

[harrymove-ctrl]

Addressed the contract-clarification blockers in 19ea3490 (just pushed to feat/memory-read-api). Mapping each requirement to what changed:

1-3. Durable checkpoint / one-page / final-page / empty-sync behavior — next_cursor is now always returned for any non-empty page (final page or single-page result included), and a separate has_more: bool is the explicit end-of-data signal instead of overloading next_cursor: null for two different meanings. next_cursor is null only for a genuinely empty page, in which case the client keeps its previous cursor. Documented in the new "Cursor semantics" section of docs/api/memory-read-api.md, and covered by query_owner_namespaces_exactly_fitting_page_still_returns_watermark / query_owner_memories_exactly_fitting_page_still_returns_watermark.

4. Same-updated_at rows not skipped — unchanged from the original implementation: (updated_at, id) keyset tie-break for /memories, tested via query_owner_memories_handles_updated_at_tie_via_id_tiebreak (forced collision).

5. Previously-seen namespace resurfacing on later change — fixed. /namespaces' cursor is now a (MAX(updated_at), namespace) watermark instead of a bare name, so a namespace that sorts before the cursor still comes back if it was touched since. Test: query_owner_namespaces_cursor_resurfaces_earlier_name_updated_after_watermark. Note this is a breaking change from the old alphabetical-by-name ordering — flagged explicitly in the doc.

6. snapshot_version reset/reconciliation — partially done. SNAPSHOT_VERSION is bumped 1 -> 2 for this exact contract change (cursor wire format + new updated_at/has_more fields), with a code comment explaining why. Still missing: an explicit doc section telling clients what to do when snapshot_version changes on a poll (i.e. "discard your cursor, do a full resync") and a test asserting that behavior. Will add before this merges — flagging so it isn't lost.

Deletion propagation — confirmed not implemented, and no longer overclaimed in the docs. forget() (POST /api/forget → delete_by_namespace, storage/db.rs:647) hard-deletes vector_entries rows; the namespace rollup's updated_at watermark doesn't advance on a delete, so a namespace whose only change since a client's last sync was a deletion won't resurface via updated_after. docs/api/memory-read-api.md now says this explicitly and recommends a periodic full (cursor-less) resync as the workaround until it's addressed. Not spinning this into a separate ticket — tracking it here as an open item on this same contract, since fixing it (soft-delete column + status: "deleted" + tombstone cleanup) is itself a contract change that should go through the same Console sign-off you're asking for below.

Agreed this is a contract change (breaking: namespace ordering + cursor wire format) and should get Console-team sign-off before we consider this mergeable — not just a code review pass. Requested acceptance tests: one-page-then-insert, update-after-checkpoint, same-updated_at tie, namespace-changed-after-sync, and empty-incremental-sync are covered by the tests named above plus query_owner_namespaces_empty_for_unknown_owner; will add an explicit "poll with valid cursor, zero new rows, cursor preserved" case alongside the snapshot_version test since that's the one scenario not yet directly covered.

[jessiemongeon1]

Style Guide Audit

Audited 1 file(s) against the Sui Documentation Style Guide.

46 violation(s) found. All must be fixed before merge.

docs/api/memory-read-api.md (46 violation(s))

46 violation(s) (46 regex, 0 claude)

• Line 1 — H1 only in frontmatter
  • Current: # Memory Read API (WALM-295)
  • Fix: Use ## or lower for section headings
• Line 14 — No em dashes in prose
  • Current: —
  • Fix: Rewrite with comma, parentheses, or split sentence
• Line 28 — Capitalize testnet in prose
  • Current: testnet
  • Fix: Testnet
• Line 28 — Use present tense, not future
  • Current: will
  • Fix: Use present tense verb
• Line 39 — No em dashes in prose
  • Current: —
  • Fix: Rewrite with comma, parentheses, or split sentence
• Line 40 — No Latin abbreviations
  • Current: e.g.
  • Fix: for example
• Line 40 — No em dashes in prose
  • Current: —
  • Fix: Rewrite with comma, parentheses, or split sentence
• Line 41 — No em dashes in prose
  • Current: —
  • Fix: Rewrite with comma, parentheses, or split sentence
• Line 42 — No Latin abbreviations
  • Current: e.g.
  • Fix: for example
• Line 44 — No em dashes in prose
  • Current: —
  • Fix: Rewrite with comma, parentheses, or split sentence
• Line 46 — No em dashes in prose
  • Current: —
  • Fix: Rewrite with comma, parentheses, or split sentence
• Line 47 — No em dashes in prose
  • Current: —
  • Fix: Rewrite with comma, parentheses, or split sentence
• Line 48 — Use present tense, not future
  • Current: will
  • Fix: Use present tense verb
• Line 53 — "on-chain" is one word → "onchain"
  • Current: on-chain
  • Fix: onchain
• Line 55 — "on-chain" is one word → "onchain"
  • Current: on-chain
  • Fix: onchain
• Line 68 — No em dashes in prose
  • Current: —
  • Fix: Rewrite with comma, parentheses, or split sentence
• Line 70 — "on-chain" is one word → "onchain"
  • Current: on-chain
  • Fix: onchain
• Line 74 — No em dashes in prose
  • Current: —
  • Fix: Rewrite with comma, parentheses, or split sentence
• Line 91 — "on-chain" is one word → "onchain"
  • Current: on-chain
  • Fix: onchain
• Line 112 — No em dashes in prose
  • Current: —
  • Fix: Rewrite with comma, parentheses, or split sentence
• Line 118 — No em dashes in prose
  • Current: —
  • Fix: Rewrite with comma, parentheses, or split sentence
• Line 131 — Use "through" not "via"
  • Current: via
  • Fix: through
• Line 133 — No em dashes in prose
  • Current: —
  • Fix: Rewrite with comma, parentheses, or split sentence
• Line 136 — No em dashes in prose
  • Current: —
  • Fix: Rewrite with comma, parentheses, or split sentence
• Line 140 — No em dashes in prose
  • Current: —
  • Fix: Rewrite with comma, parentheses, or split sentence
• Line 141 — Use "basic" not "simple"
  • Current: simply
  • Fix: (remove)
• Line 161 — No em dashes in prose
  • Current: —
  • Fix: Rewrite with comma, parentheses, or split sentence
• Line 191 — No em dashes in prose
  • Current: —
  • Fix: Rewrite with comma, parentheses, or split sentence
• Line 195 — No em dashes in prose
  • Current: —
  • Fix: Rewrite with comma, parentheses, or split sentence
• Line 198 — No em dashes in prose
  • Current: —
  • Fix: Rewrite with comma, parentheses, or split sentence
• Line 203 — No em dashes in prose
  • Current: —
  • Fix: Rewrite with comma, parentheses, or split sentence
• Line 210 — No em dashes in prose
  • Current: —
  • Fix: Rewrite with comma, parentheses, or split sentence
• Line 216 — No em dashes in prose
  • Current: —
  • Fix: Rewrite with comma, parentheses, or split sentence
• Line 225 — No em dashes in prose
  • Current: —
  • Fix: Rewrite with comma, parentheses, or split sentence
• Line 227 — No Latin abbreviations
  • Current: e.g.
  • Fix: for example
• Line 227 — Use second person (you), not first person
  • Current: only when we ship a breaking contract change (e.g.
  • Fix: Rewrite using 'you'
• Line 234 — No em dashes in prose
  • Current: —
  • Fix: Rewrite with comma, parentheses, or split sentence
• Line 236 — No em dashes in prose
  • Current: —
  • Fix: Rewrite with comma, parentheses, or split sentence
• Line 241 — No Latin abbreviations
  • Current: e.g.
  • Fix: for example
• Line 242 — No em dashes in prose
  • Current: —
  • Fix: Rewrite with comma, parentheses, or split sentence
• Line 248 — No em dashes in prose
  • Current: —
  • Fix: Rewrite with comma, parentheses, or split sentence
• Line 252 — Use "because" not causal "since"
  • Current: since
  • Fix: because
• Line 253 — No Latin abbreviations
  • Current: e.g.
  • Fix: for example
• Line 259 — "on-chain" is one word → "onchain"
  • Current: on-chain
  • Fix: onchain
• Line 276 — No em dashes in prose
  • Current: —
  • Fix: Rewrite with comma, parentheses, or split sentence
• Line 286 — "on-chain" is one word → "onchain"
  • Current: on-chain
  • Fix: onchain

---

Automated audit using the Sui Documentation Style Guide.

[harrymove-ctrl]

Re-review — WALM-295

Traced all 5 acceptance criteria plus the migration lock-splitting and provenance-plumbing claims against the diff (not the description). Approve — this is genuinely correct, well-reasoned work, but there are two things to close before merge:

Verified:

• 3 endpoints share the existing signature-verified auth middleware (main.rs protected_routes group) and correctly 403 on owner mismatch — not a separate/weaker auth path.
• Real keyset pagination: (updated_at, id) > (cursor) with a peek-one-extra-row has_more, cursor built from the last emitted row (not the peeked one) — no off-by-one gap. /namespaces' HAVING (MAX(updated_at), namespace) > (...) is a correct, non-obvious choice given it must apply post-aggregation.
• No plaintext/ciphertext/key material in any response DTO.
• Migrations 010–013 genuinely execute as separate transactions (storage/db.rs:592-618, one raw_sql().execute() per file) — the lock-avoidance claim is real, and the SET DEFAULT-as-own-statement trick in 010 (avoiding Postgres's fast-default attmissingval defeating 011's backfill) is correct and non-obvious.
• agent_id/package_id genuinely flow through all real write paths via the shared insert_vector_and_mark_remember_done helper (3 call sites) plus walrus_seal.rs/admin.rs — "5 write paths" in the PR body is shorthand for call-site count, not 5 literal insert_vector calls, worth a 1-line doc nit but not a blocker.

To close before merge:

1. PR WALM-295/296/297: owner-scoped memory read API + bearer token auth for Console #554 duplication. WALM-295/296/297: owner-scoped memory read API + bearer token auth for Console #554 (feat/console-integration-test → dev) is a self-declared superset — its own PR body says it "already contains the full history of WALM-295: Owner-scoped memory read API (namespaces/memories/agents) #537, Add per-memory storage expiry (end_epoch/expires_at) #543, and WALM-297: Owner-scoped bearer token auth for the read API (Phase 1) #546... recommend closing those in favor of this one once reviewed," and its first 5 commits are verbatim copies of this branch's commits. Merging WALM-295: Owner-scoped memory read API (namespaces/memories/agents) #537 now is fine (clean subset), but whoever owns WALM-295/296/297: owner-scoped memory read API + bearer token auth for Console #554 needs to rebase it past this merge or close it in favor of landing Add per-memory storage expiry (end_epoch/expires_at) #543/WALM-297: Owner-scoped bearer token auth for the read API (Phase 1) #546 separately — otherwise it'll either no-op-conflict or silently re-diverge the agent_id/package_id plumbing on a manual rebase. Can you confirm who's driving WALM-295/296/297: owner-scoped memory read API + bearer token auth for Console #554 so this doesn't land twice?
2. Style-guide-audit bot flagged 46 violations in docs/api/memory-read-api.md on 2026-08-10 ("all must be fixed before merge") — no commits since then addressing it. Please run the fixes (mostly em-dash/e.g./on-chain→onchain mechanical rewrites) before merge; happy to take a pass if useful.

Depth: targeted (data-layer + auth-reuse changes; migrations warranted the closer read given they touch a live table).