Update / indexed db account ops integration

[gergana95]

Issue is https://github.com/AmbireTech/ambire-app/issues/7264
Related ambire-app PR https://github.com/AmbireTech/ambire-app/pull/7697

Move transaction history to IndexedDB

Makes IndexedDB the source of truth for accountsOps (transaction history), with a
one-time migration from the legacy chrome.storage.local blob.

Why

accountsOps was stored as a single key-value blob, so every new transaction rewrote the
entire history. That cost grows without bound — a heavy account is tens of MB re-serialised
on every write, and the whole blob is read on every service-worker wake-up.

IndexedDB gives row-level writes and a bounded startup read: pending ops in full, plus the
20 most recent finalized ops per (account, chain). Older history is lazy-loaded on demand.

Mobile is unaffected — it has no IndexedDB and keeps using the key-value backend.

---

What reviewers should focus on

In rough order of "if this is wrong, it matters most":

1. ActivityController.#load() — migration must complete before the first read, and
   #load() must never reject. It is launched from the constructor and assigned to
   #initialLoadPromise, which every public method awaits; a rejection there breaks the
   controller for the whole session and raises an unhandled rejection.
2. #mergeOpsIntoCache — the lazy-load merges by id and keeps the existing object
   on a collision. Replacing the array would drop ops that are only in memory and detach
   objects that updateAccountsOpsStatuses mutates in place across long provider awaits.
3. Ordering in addAccountOp — persistence comes last, after emitUpdate(). On the
   key-value backend putSingleOp rewrites the whole blob, so awaiting it earlier blocks
   the UI on a full serialisation. There is a comment saying not to "fix" this; please leave
   it alone.
4. #fallBackToKeyValueForThisSession() — if the migration fails, the session must read
   and write the legacy blob. Reading the blob while still writing to IndexedDB puts one
   row in the empty store, which makes the isEmpty() guard skip the retry forever.
5. Migration handlers are synchronous (idbDatabase.ts). To transform rows you must
   chain off the read, not await it — awaiting a non-IDB promise lets the versionchange
   transaction commit and the writes vanish silently. Verified in Chrome and Firefox
   against 14,000 rows. There is a comment warning against tidying it into an await.
6. The transaction total is cached, and IDB-only. getTotalOpsCountForAccount() exists
   because BannerController evaluates minTxnsTotal/maxTxnsTotal in a synchronous
   callback that cannot query IndexedDB, and the in-memory lengths are only the startup
   window. On the key-value backend the count is summed live instead — caching there could
   only ever be staler, since that backend's in-memory blob is the whole history. The
   cache is refreshed by recount, not by +1 on write, because putSingleOp can evict a
   row on its own and make the net change zero.

Design decisions worth knowing

• Schema is declarative. AMBIRE_IDB_SCHEMA is the single source of truth;
  reconcileSchema() creates any missing store or index on every upgrade, idempotently. So
  a purely additive change needs only a manifest entry plus a dbVersion bump. Migration
  handlers exist for transforming existing rows, not for creating structure.
• The legacy key is retained, not deleted. It is a recovery floor if something goes
  wrong. It is frozen at migration time, so it is never a source of truth. Open question
  for this PR: when do we delete it? Keeping it forever means every migrated user stores
  the history twice.
• activityIdbMigrated records that history lives in IndexedDB. Nothing reads it
  today — it is written (#recordHistoryLivesInIdb) because it can only be recorded while
  IndexedDB is working, and a session that cannot open IndexedDB can no longer distinguish
  "this user never had transactions" from "this user's history is in IndexedDB and
  unreachable". Keeping the writer means the marker is in place for whatever consumes it
  later. Deliberately accessed via narrow casts rather than added to StorageProps, because
  it is provisional. Reviewers: flag this if you would rather not carry a write-only key.
• A dbVersion bump is effectively unrollbackable. A build pinned to the old version
  cannot open an upgraded database. dbVersion is 1 here — the phishing store was
  prepared and then intentionally left out of the manifest, so this PR ships a single
  version. Add it in the same change that wires PhishingController.

---

Manual QA

The whole checklist below has been run and passes — on Chrome, and B1/B2 on Firefox too.
The checkboxes are left unticked on purpose so it stays usable as a fresh checklist for
anyone re-verifying. The table here is the record of what was actually covered.

Status — all green

✅	B1 — a broadcast transaction is no longer dropped from memory
✅	B2 — pending ops no longer block the lazy-load; all 35 reachable
✅	B3a / B3b / B3c — including the IndexedDB-unavailable fallback: the wallet stays usable and quiet, reading the retained legacy blob
✅	B4 — account removal clears the expansion markers, a re-add pages fully again
✅	B5 — no regressions: ordering, chain filtering, both address-poisoning cases, repeat reloads, clean console
✅	B6 — B1 and B2 repeated on Firefox
✅	Schema-upgrade machinery and a data-transforming migration handler rewriting 14,000 rows — Chrome and Firefox, using throwaway version bumps since reverted
✅	Connection recovery: blocking() closes, terminated() drops the cache, #openTx reopens a dead handle

PART A — Setup (required)

• A1. yarn setup from the repo root.
  • ⚠️ Do not run npm install inside src/ambire-common before building. The Jest suite
    needs that nested node_modules, but the LavaMoat build fails on it
    (unknown package directory for @babel/runtime) — which is exactly why setup.sh
    deletes it. Tests and builds want opposite states; run yarn setup again after testing.
• A2. yarn build:web:webkit
  • Pass: build/webkit-prod/manifest.json exists.
• A3. Delete the local database on every browser you tested with.
  • It is at v3 from the liveness probe; the code is back to v1, so a v1 build
    cannot open a v3 database (VersionError) and would silently fall back to
    key-value storage.

  indexedDB.deleteDatabase('ambire')

• A4. Reload the extension, confirm a clean start in the SW console:

  [AmbireIdb] Upgrading "ambire" v0 → v1
  [AmbireIdb] created index "by-account-chain-timestamp" on "accountsOps"
  [AmbireIdb] created index "by-account-chain-status" on "accountsOps"
  [AmbireIdb] v1: initial schema applied
  

  • Pass: v0 → v1 (not v3), no VersionError.
• A5. Use a throwaway profile with a disposable account for anything below
  that sends a transaction.

---

PART B — Fixes you can verify in a browser

B1 — A broadcast transaction is no longer dropped from memory ⭐ most important

This is the bug that could stop a transaction being polled to confirmation. It needed
an active chain-filtered session and a group inside the 20-op startup window.

• B1.1 Seed a small group (under the window) so the lazy-load path is live:

  (async () => {
    const CHAIN = '8453' // Base
    const accounts = JSON.parse((await chrome.storage.local.get('accounts')).accounts)
    const ACC = accounts[0].addr
    const big = (n) => ({ $bigint: String(n) })
    const ops = Array.from({ length: 5 }, (_, i) => ({
      id: `seed-${i}`, accountAddr: ACC, chainId: big(CHAIN), signingKeyAddr: ACC,
      signingKeyType: 'internal', nonce: big(i),
      calls: [{ to: '0xF0cD725D2195b1D3f4BD038c3786005B793237DB', value: big(0), data: '0x' }],
      gasLimit: null, gasFeePayment: null, accountOpToExecuteBefore: null,
      txnId: `0xseed${i}`, identifiedBy: { type: 'Transaction', identifier: `0xseed${i}` },
      status: 'success', timestamp: Date.now() - i * 3600000,
      isSingletonDeploy: false, flags: {}
    }))
    await chrome.storage.local.set({ accountsOps: JSON.stringify({ [ACC]: { [CHAIN]: ops } }) })
    await chrome.storage.local.remove('activityIdbMigrated')
    await new Promise((r) => { const q = indexedDB.deleteDatabase('ambire'); q.onsuccess = q.onerror = q.onblocked = () => r() })
    console.log(`seeded 5 ops on chain ${CHAIN} for ${ACC} — reload the extension`)
  })()

• B1.2 Reload the extension.
• B1.3 Open Activity / history and filter to Base — leave that view open.
  This is what creates the filtered session that used to destroy the new op.
• B1.4 With that view still open, send a small transaction on the same chain.
• B1.5 Pass: the new transaction appears in the list immediately, shows as
  pending, and progresses to confirmed on its own.
  Fail: it appears briefly then vanishes, or stays pending forever — that is the
  old bug (in-memory op discarded → never polled).
• B1.6 Confirm it also reached disk:

  (async () => {
    const db = await new Promise((r) => { const q = indexedDB.open('ambire'); q.onsuccess = () => r(q.result) })
    const rows = await new Promise((r) => { const q = db.transaction('accountsOps').objectStore('accountsOps').getAll(); q.onsuccess = () => r(q.result) })
    db.close()
    console.log('newest:', rows.sort((a, b) => b.timestamp - a.timestamp)[0]?.id)
  })()

B2 — Pending ops no longer block the lazy-load

The old gate was inMemoryCount > 20. Pending ops are exempt from the 20-op cap, so a
group could arrive at 25 and be mistaken for "already fully expanded", freezing
pagination at 25 of 35.

• B2.1 Seed 5 pending + 30 finalized on one chain:

  (async () => {
    const CHAIN = '8453' // Base
    const accounts = JSON.parse((await chrome.storage.local.get('accounts')).accounts)
    const ACC = accounts[0].addr
    const big = (n) => ({ $bigint: String(n) })
    const mk = (id, ts, status) => ({
      id, accountAddr: ACC, chainId: big(CHAIN), signingKeyAddr: ACC, signingKeyType: 'internal',
      nonce: big(0), calls: [{ to: '0xF0cD725D2195b1D3f4BD038c3786005B793237DB', value: big(0), data: '0x' }],
      gasLimit: null, gasFeePayment: null, accountOpToExecuteBefore: null,
      txnId: `0x${id}`, identifiedBy: { type: 'Transaction', identifier: `0x${id}` },
      status, timestamp: ts, isSingletonDeploy: false, flags: {}
    })
    const pending = Array.from({ length: 5 }, (_, i) => mk(`pend-${i}`, Date.now() - i * 1000, 'broadcasted-but-not-confirmed'))
    const finalized = Array.from({ length: 30 }, (_, i) => mk(`fin-${i}`, Date.now() - (i + 100) * 3600000, 'success'))
    await chrome.storage.local.set({ accountsOps: JSON.stringify({ [ACC]: { [CHAIN]: [...pending, ...finalized] } }) })
    await chrome.storage.local.remove('activityIdbMigrated')
    await new Promise((r) => { const q = indexedDB.deleteDatabase('ambire'); q.onsuccess = q.onerror = q.onblocked = () => r() })
    console.log(`seeded 5 pending + 30 finalized on chain ${CHAIN} — reload the extension`)
  })()

• B2.2 Reload the extension, open Activity, filter to Base.
• B2.3 Pass: all 35 are reachable — page through to the oldest (fin-29)
  and check the total/page count reflects 35.
  Fail: only ~25 visible with no further pages — the old heuristic.
• B2.4 All 5 pending ops must be visible regardless of paging (pending ops are
  always loaded in full).

B3 — The migration-completion flag is recorded for users with no legacy blob

Previously the flag was only set after migrating a legacy blob, so it was never recorded for
anyone who installed after IDB became the default — the one moment it can be written is
while IDB still works, and those users skipped it entirely.

No transaction needed. Ops are written straight into IDB below, which is exactly the
state a post-IDB user reaches naturally. Note that clearing accountsOps does not
remove your account — only the transaction history.

B3a — A genuinely empty wallet must NOT arm the flag

• B3a.1 Clear history, the flag, and the database:

  (async () => {
    await chrome.storage.local.remove(['accountsOps', 'activityIdbMigrated'])
    await new Promise((r) => { const q = indexedDB.deleteDatabase('ambire'); q.onsuccess = q.onerror = q.onblocked = () => r() })
    console.log('clean slate — reload the extension')
  })()

• B3a.2 Reload, then check:

  chrome.storage.local.get('activityIdbMigrated').then((r) => console.log('flag:', r.activityIdbMigrated ?? '(unset)'))

  • Pass: (unset). With no history there is nothing to warn about, so a brand-new
    wallet must not be told it lost something.

B3b — Ops in IDB (no legacy blob) DO arm the flag

• B3b.1 Write rows directly into IDB, bypassing any migration:

  (async () => {
    const CHAIN = '8453' // Base
    const accounts = JSON.parse((await chrome.storage.local.get('accounts')).accounts)
    const ACC = accounts[0].addr
    // No legacy blob and no flag — the state a post-IDB install is in
    await chrome.storage.local.remove(['accountsOps', 'activityIdbMigrated'])
  
    const db = await new Promise((res, rej) => {
      const req = indexedDB.open('ambire', 1)
      req.onupgradeneeded = () => {
        const s = req.result.createObjectStore('accountsOps', { keyPath: ['accountAddr', 'chainId', 'id'] })
        s.createIndex('by-account-chain-timestamp', ['accountAddr', 'chainId', 'timestamp'])
        s.createIndex('by-account-chain-status', ['accountAddr', 'chainId', 'status'])
      }
      req.onsuccess = () => res(req.result)
      req.onerror = () => rej(req.error)
    })
    await new Promise((res, rej) => {
      const tx = db.transaction('accountsOps', 'readwrite')
      // chainId is a STRING on the row; op.chainId stays a real BigInt in the payload
      tx.objectStore('accountsOps').put({
        accountAddr: ACC, chainId: CHAIN, id: 'idb-native-1', timestamp: Date.now(), status: 'success',
        op: { id: 'idb-native-1', accountAddr: ACC, chainId: BigInt(CHAIN), calls: [], gasFeePayment: null,
              status: 'success', timestamp: Date.now(), identifiedBy: { type: 'Transaction', identifier: '0xnative1' } }
      })
      tx.oncomplete = res
      tx.onerror = () => rej(tx.error)
    })
    db.close()
    console.log('wrote 1 row straight into IDB, no legacy blob — reload the extension')
  })()

• B3b.2 Reload, then check the flag again with the B3a.2 snippet.
  • Pass: true. History now lives in IDB, so the net is armed.
  • Fail: still (unset) — #recordHistoryLivesInIdb is not running.

B3c — IDB unavailable degrades to the legacy blob without breaking

There is deliberately no banner and no error toast for this state. The condition is
detected inside #load(), i.e. on service-worker startup, where errors only reach the UI
through a live port — no window is open, so a toast is emitted into nothing. A banner was
implemented and then removed: the user cannot act on it, and the retained legacy blob means
history still renders. What must hold is that the fallback is quiet and functional.

To make IDB genuinely unavailable, put the database at a version the build cannot open —
deleting it is not enough, that leaves IDB usable and merely empty.

• B3c.1 With the flag set from B3b, force a higher version:

  (async () => {
    const flag = (await chrome.storage.local.get('activityIdbMigrated')).activityIdbMigrated
    console.log('flag before:', flag, '(must be "true" — run B3b first if not)')
    // The build opens at dbVersion 1, so a v99 database makes openAmbireIdb() throw
    // VersionError, which is what "IDB unavailable" means in production.
    await new Promise((res) => {
      const req = indexedDB.open('ambire', 99)
      req.onsuccess = () => { req.result.close(); res() }
      req.onerror = () => res()
      req.onblocked = () => res()
    })
    console.log('database forced to v99 — reload the extension')
  })()

• B3c.2 Reload the extension. In the SW console expect the fallback to engage:

  [background] Failed to open IDB, activity will fall back to storage  (VersionError)
  

• B3c.3 Open the extension.
  • Pass: the wallet is fully usable. History renders from the retained legacy blob,
    no error banner, no error toast, no [Emitted error…] line in the console.
  • Fail: an unhandled rejection, a hung Activity screen, or any public method on
    ActivityController never resolving — that would mean #load() rejected.
• B3c.4 Send a transaction while still on the fallback.
  • Pass: it appears and persists across a reload (the key-value backend is handling
    both reads and writes, not a mixed state).
• B3c.5 ⚠️ Clean up — required. The database is at v99 and the build cannot
  open it, so the extension stays on the key-value fallback until you remove it:

  indexedDB.deleteDatabase('ambire')

  Then reload and confirm a normal [AmbireIdb] Upgrading "ambire" v0 → v1.

B4 — Removing an account clears its expansion markers

A stale marker would make a re-added account look already-expanded and permanently
show only the startup window.

• B4.1 With more than 20 ops on a chain (reuse B2's seed), open Activity,
  filter to that chain, and page to the end so the group is fully expanded.
• B4.2 Remove the account from the wallet.
• B4.3 Re-import the same account.
• B4.4 Open Activity, filter to the same chain.
• B4.5 Pass: history loads and pages fully again.
  Fail: only ~20 visible with no further pages — a stale marker.

B5 — Nothing regressed

• B5.1 History renders newest-first, no duplicates.
• B5.2 Chain filter switches without losing rows.
• B5.3 Send screen: a known recipient is not flagged "first time sending".
• B5.4 Send screen: a lookalike of a known recipient does raise the
  poisoning warning.
• B5.5 Reload twice — row count stable, no duplicate migration in the log.
• B5.6 No unexpected InvalidStateError or VersionError in the SW console.

B6 — Firefox

• B6.1 yarn build:web:gecko, load via about:debugging.
• B6.2 Repeat B1 and B2 — those are the two with UI-visible behaviour.

---

PART C — Fixes that are NOT browser-reachable

Each needs a specific call to fail on demand, which cannot be induced from the console
without patching the build. Listed so the gap is explicit.

Fix	Why not reachable	Covered by
Key-value fallback when the migration fails	Needs migrateFromStorage to throw	a failed migration keeps writes out of IDB…
#load never rejects (post-load guard)	Needs storage.get('activityIdbMigrated') to throw	init survives the post-load history checks throwing
Failed startup read must not mark history expanded	Needs loadStartupOps to throw	a failed startup read does not permanently mark…
updateOps aborts on a malformed op	Needs an op with no timestamp reaching updateOps	a malformed op aborts the batch…
Merge keeps in-memory-only ops	Needs putSingleOp to fail while the op stays in memory	an op that failed to persist still survives…

All five were verified by removing the fix and confirming the test fails, so they are
not resting on a green-suite assumption.

---

Automated tests

193 tests, 7 suites — 167 of them new in this PR.

Suite	Tests	New?	Scope
activityIdb.test.ts	67	new	IndexedDB backend: CRUD, startup window, 1000-row cap, bigint round-trip, batch atomicity, malformed-row filtering, connection reopen
idbIntegration.test.ts	33	new	reconcileSchema, applyMigrations, manifest↔handler consistency guards, handler row-transformation
activityIdbMigration.test.ts	33	new	#load() wiring: migration ordering, legacy retention, the flag, fallback, expansion, method interactions
phishingIdb.test.ts	26	new	Reference implementation (store not in the manifest yet)
idbDatabase.test.ts	8	new	Singleton, store/index creation, blocking()
activity.test.ts	20	unchanged	Exercises the key-value/mobile path, so it doubles as the mobile regression guard
externalAccountOps.test.ts	6	unchanged	Unrelated

activity.test.ts is worth calling out: it constructs the controller without an idb
argument, so every one of its 20 tests runs the key-value backend. It passing unchanged is
the main evidence that mobile behaviour is untouched.

Every new test was sabotage-verified

Each fix was removed individually to confirm the intended test fails. This caught four
tests that passed with the fix deleted — each because the test setup happened to satisfy
the precondition the test depended on, so the scenario never reached the code under test. One
example: a test meant to prove the startup window is respected used a load helper that
expanded the full history as a side effect.

A green suite gives no signal about that, so please apply the same check to anything added on
top of this.

One test is knowingly not sabotage-isolable and says so in a comment: "the mobile count
reflects a newly added op". It passes with its guard removed, because a second gate upstream
makes the guarded branch unreachable on that backend. It is kept as a behavioural pin rather
than deleted or dressed up as verified.

---

Known limitations, deliberately not fixed

None of these are known-broken behaviour — they are accepted trade-offs, each documented at
the code that implements it.

1. isEmpty() migration guard. It cannot distinguish "never migrated" from "migrated,
   wiped, then partially repopulated". If IndexedDB is wiped while running and anything is
   written before the next restart, that row makes the guard skip and the legacy copy is
   never read again. Wipe → restart with no write in between does recover, which is the
   realistic path — the extension has unlimitedStorage, so routine eviction is not a
   factor. Documented at activityIdb.ts ensureMigrated() and in controllers/AGENTS.md.
2. activityIdbMigrated is write-only. Nothing reads it. It is written because it can
   only be recorded while IndexedDB works — a session that cannot open IndexedDB can no
   longer tell "never had transactions" from "history is in IndexedDB and unreachable", and
   by then it is too late to write the flag that would have distinguished them. Keeping the
   writer means the marker is in place for whatever consumes it later.
   Reviewers: flag this if you would rather not carry a write-only key.
3. hasAccountOpsSentTo still expands full history. It answers "have I sent here?" and
   the address-poisoning lookalike check, both properties of the whole history, so on a miss
   it loads everything into memory — for every account when accountId is empty. The fast
   path (sentToHistory.recipients) is only populated by addAccountOp, and there is no
   backfill from existing history, so for users with pre-existing history the expensive
   path runs on recipients they have used many times before. This is the largest remaining
   optimization; see services/storage/README.md for the full write-up and the fix.
4. The cached transaction total lags by one update. #refreshTotalOpsCount runs after
   emitUpdate() in addAccountOp (persistence must come last), so a banner gated on
   minTxnsTotal sees the new total on the next update. Refreshing earlier would block the
   UI on an IndexedDB read; emitting a second update would re-serialize the whole controller
   state per broadcast. The thresholds are coarse buckets.
5. No migration telemetry. A failed migration falls back silently, so there is no
   aggregate view of how many users land on the fallback. The right place is where
   openAmbireIdb() fails in background.ts, which currently only logs to console.
6. MAX_IDB_GROUP_SIZE cannot converge downward — putSingleOp adds one row and
   deletes at most one, so an oversized group holds steady rather than shrinking.
   Unreachable today because the in-memory trim() keeps groups at ≤1000.

Suggested follow-ups

• Decide legacy-key retention (bounded retention is my suggestion: delete once the
  migration has proven itself over a release).
• Backfill sentToHistory.recipients from full history at migration time, then drop
  #ensureFullHistoryLoaded() from the address-poisoning path — this removes a mechanism
  rather than adding one. Deliberately not in this PR: it changes security-relevant
  behaviour and deserves its own review.
• Add the telemetry from (5) before a wide rollout.
• Wire PhishingController to PhishingIdbStorage and add the store to the manifest (in
  one change — the dbVersion bump is unrollbackable).

Rollout note

There is no feature flag: the read-path flip lands for every existing user at once. The
retained legacy blob makes it recoverable, but recovery means shipping again. Worth deciding
whether to gate it or stage the rollout.

[socket-security]

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	idb@​8.0.3
	fake-indexeddb@​6.2.5

View full report

[PetromirDev]

I love the idea of using indexeddb but don't think that investing much time into a POC is that viable. IMO you cannot compare inserting a single account op without serializing it to serializing up to 1000 ops per network for each account and then persisting that with chrome.storage.local. Let's have the three of us discuss this (@gergana95, @superKalo and me) and ship this. I have some considerations with the current state of the POC, mainly about migrations and not having an abstraction over storage. I won't share them yet because I know that this is a POC but want to discuss them and proceed with this one quicker as I think that it will be a great improvement.

[gergana95]

/review

[github-actions]

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 4 🔵🔵🔵🔵⚪

🔒 No security concerns identified

✅ No TODO sections

⚡ Recommended focus areas for review Silent error swallowing persistAccountsOps catches persistence failures with console.error instead of this.emitError(...), violating the invariant that errors in controller methods must not be swallowed silently. Every other persistence error handler in this file (putSingleOp, deleteAccount, #refreshTotalOpsCount, etc.) correctly uses this.emitError({ level, message, error }). Since persistAccountsOps is on the call path of public methods (backfillAccountOpBalanceChangesAndPersist, #updateAccountsOpsStatuses), a failed updateOps call — e.g. an IDB write failure during a status or balance-change update — would not reach Sentry. The in-memory state would diverge from persisted state, and the team would have no visibility into the failure, making stale-on-restart issues impossible to diagnose. private async persistAccountsOps(changedOps: SubmittedAccountOp[]) { try { await this.#persistence.updateOps(changedOps) } catch (error) { console.error('ActivityController: Failed to persist updated ops', error) }

[PetromirDev]

Also, please update the description of the PR

[PetromirDev]

"Adding it would require a
// dbVersion bump, and a shipped bump cannot be rolled back" - it is not shipped yet so I see no reason to not add it there