Add per-memory storage expiry (end_epoch/expires_at) - #543
Add per-memory storage expiry (end_epoch/expires_at)#543harrymove-ctrl wants to merge 18 commits into
Conversation
…d regression tests
…ix synced_at index
…g JSON-RPC-sunset migration
/api/remember returns {"job_id": ..., "status": ...}. The 64KB and
large-size test functions asserted "id" in result, so they always
reported failure even when the underlying job completed successfully.
|
Architecture question after confirming the response contract with the Console team:
Since the existing V1 rows have already been backfilled and V1 leases cannot be renewed, can we simplify WALM-296 to a DB-only indexed flow?
This keeps reads cheap and avoids repeatedly scanning owner collections or checking each blob only to determine whether it has expired. It also removes the sweep retry/throughput/multi-replica concerns. The main implementation requirement becomes ensuring that all write and recovery paths preserve expiry metadata; nullable expiry remains available for genuinely unresolved/error cases. If continuous reconciliation of external deletions is a Phase 1 requirement, could we document that requirement and its freshness/SLA explicitly? Otherwise I would prefer treating it as a later reconciliation feature rather than coupling it to this read-API change. |
Style Guide AuditAudited 2 file(s) against the Sui Documentation Style Guide. 1 violation(s) found. All must be fixed before merge.
|
Style Guide AuditAudited 2 file(s) against the Sui Documentation Style Guide. 26 violation(s) found. All must be fixed before merge.
|
Status check — WALM-296Following the Linear thread: the
Once rebased and CI is clean I'll do the full read-through (Nikola's item #2, the sweep's 24h-predicate/backlog concern, is filed as WALM-323 already so not blocking here). Also flagging: this PR isn't linked in the Notion "GitHub PR" field for WALM-296 — worth adding so it's discoverable from the board. |
4 participants
Summary
WALM-296: adds
end_epoch/expires_atto every item inGET /v1/owners/{owner}/memories(built by WALM-295, PR #537, not yet merged — this PR targetsfeat/memory-read-apidirectly rather thandev, so the diff here is scoped to WALM-296 only).vector_entriescolumns:end_epoch,expires_at,expiry_synced_at(migration 014), plus an index onexpiry_synced_at(migration 015).end_epochis populated at write time on the 2 write paths that have it for free (primary upload + seal-store); the other 3 (metadata-transfer recovery, finalize-recovery, restore) startNULL.main.rs, mirrors the existingevict_expired_delegate_keys/fail_stale_remember_jobspattern) queries the Walrus chain for storage-lease end epochs and converts them toexpires_attimestamps, batched by owner, never touching the hot listing path.expires_at_from_epoch) is anchored on the current epoch, not Walrus genesis —expires_at = now + (end_epoch - current_epoch) * epoch_duration. This self-corrects every sweep cycle regardless of whetherepoch_durationever changes, and avoids extrapolating from a fixed point over potentially hundreds of epochs.statusonMemoryItemchanges from a hardcoded"active"to a real derivation:"expired"ifexpires_atis in the past,"active"otherwise (including when not yet synced — an intentional optimistic default).Migration notes
ADD COLUMNs, no rewrite).CREATE INDEX CONCURRENTLY, its own file/transaction (required — cannot run inside a transaction).Test plan
cargo test --bins, against real local Postgres).NULLS LASTwhen the query needsNULLS FIRST— verified viaEXPLAIN, cost dropped from 163 to 5.96), and an arithmetic overflow panic in the epoch-delta computation (function runs in an unsupervised background task with no restart wrapper).Follow-ups (non-blocking, deferred)
nonexistentBlobIdsverification (for blobs no longer on-chain) is fetched and parsed but not yet consumed — those rows stayNULL/"active"indefinitely rather than reflecting a more accurate state.end_epoch IS NULLfirst) once the table grows past that.SuiClient/gRPC client instance is now constructed (separate quota fromsecurity_delete_sui) — small, accepted duplication, not unified.docs/api/memory-read-api.mddocuments the final shape.Scope note
This targets
feat/memory-read-api(PR #537), notdev— WALM-296 fundamentally extends WALM-295'sMemoryItem/query_owner_memories, so it's stacked on top rather than independent.