✨ Cache the extracted core.sqlite_zip database files

[giovannipizzi]

By default, the SQLite database contained in a core.sqlite_zip archive is extracted (and for remote archives, downloaded) to a temporary file again for every Python session. This commit adds an optional local cache for the extracted databases, stored in the cache/sqlite_zip subdirectory of the AiiDA configuration folder.

The cache is content-addressed: entries are named after the CRC-32 checksum and size of the uncompressed database, as recorded in the central directory of the zip archive. Validating a cache entry therefore only requires reading the central directory (for remote archives, a small HTTP range request), never the database itself, and identical archives at different locations share a single entry. No metadata files are needed and concurrent processes are safe: entries are written to a temporary file and moved in place atomically. Since entries are shared between profiles and processes, cached databases are always opened read-only (sqlite URI options mode=ro&immutable=1), so no consumer can corrupt an entry for the others.

A valid cache entry is always used automatically. The cache is only written when explicitly requested:

• verdi --use-cache -p <profile-or-archive-location> <command> caches the database for a single invocation. The flag is recorded in the storage configuration of a detached copy of the profile, so it is never persisted to the configuration file.
  The configuration version is not bumped, since the content of the storage.config is not validated, the keys have safe defaults if not defined, and if AiiDA is downgraded, those new keys are simply ignored.
• verdi profile setup core.sqlite_zip --use-cache caches the database at setup and records it under the cached_database key of the storage configuration in the configuration file, together with the archive schema version under cached_database_version.

For profiles recording a cached_database, a deleted cache entry is transparently recreated on the next load. If instead the archive content no longer matches, loading fails with an error pointing to the new verdi profile cache-refresh command, which re-caches the database, updates the recorded keys, and deletes the previously recorded cache entry if no other profile still records it. The new verdi profile cache-clear command deletes all cached files; entries are recreated on the next load, although users relying on --force-cache with an unreachable archive should be aware that clearing the cache deletes their only usable copy of the data.

Finally, the top-level verdi --force-cache option uses the recorded cached database directly, without accessing the archive at all. This allows for example working offline with a remote archive, at the cost of the data not being validated (and repository files, which are always read from the archive itself, potentially being unavailable). The recorded schema version is checked even in this mode, so a cached database that predates an upgrade of the archive format fails with a clear error pointing to verdi profile cache-refresh, instead of obscure ORM errors.

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 745dd5a6-4419-4b41-8edf-1b64f58ae0c3

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[giovannipizzi]

Note 1: developed with support of Claude Fable.

Note 2: to be merged only after #7432

[giovannipizzi]

Potential discussion points:

1. Cache key = CRC-32 + size from the zip central directory. Pros: validation needs only the
   central directory (tiny range request), dedup across locations, no metadata files. Cons:
   32-bit checksum (accidental collision astronomically unlikely with size included, but not
   impossible); the value is declared by the archive/server, not computed — a corrupt or
   malicious archive can alias an existing entry and the user silently gets the wrong database
   (also relevant with remote URLs, where the server is a third party). Alternatives: hash the
   actual bytes on write and re-verify periodically; include the archive schema version in the
   key; per-source namespacing (loses dedup). Is CRC+size an acceptable trust model for the team?

2. Opt-in writes vs. always-on cache. Current design: reads always hit the cache, writes only
   with --use-cache/profile setup flag. Simpler alternative: always cache (as pip/uv do) with a
   size cap + LRU eviction, removing the flags entirely. The opt-in design avoids surprising disk
   usage but adds two CLI flags, a config key and a "pointer" concept — is the added surface worth
   it?

3. No eviction or size limit. verdi profile cache-clear is all-or-nothing and manual.
   For large archives the cache can silently grow to many GBs. Do we want a
   configurable max size, LRU eviction, per-entry/per-profile clearing, or at least showing the
   cache size somewhere (verdi status, verdi profile list)?

4. Naming of the flags. In AiiDA, "caching" strongly connotes calculation caching
   (verdi config set caching.default_enabled); a top-level verdi --use-cache may mislead.
   Also --force-cache sounds like "force (re)writing the cache" while it means "use the cache
   without validating, possibly offline". Candidates: --cache-db/--db-cache and --offline.
   Renaming after release will require deprecation, so best settled now.

5. Two overlapping mechanisms in the storage config: use_cache vs cached_database.
   A profile set up with --use-cache persists both use_cache: true and the cached_database
   pointer, and either alone already triggers cache repopulation. Could the persisted model be
   reduced to the pointer only (with use_cache strictly a CLI-transient flag)? Fewer states to
   reason about, and it makes the "flags are never persisted" invariant structural.

6. force_cache as a config key. It is meant to be CLI-only, but it lives in
   storage_config. Now, accidental persistence is prevented (cache-refresh persists
   the pristine profile, and Config.update_profile always strips the key before storing), and a
   stale-schema cached database now fails clearly even in this mode thanks to the recorded
   cached_database_version. Still possible: hand-editing force_cache: true into
   config.json, permanently disabling archive validation for a profile. Remaining decision:
   make the invariant structural by keeping the flag out of storage_config entirely (e.g. a
   Profile attribute), or accept and document the hand-edit escape hatch.

7. Config schema version not bumped despite new recognized keys (use_cache,
   cached_database, and after review also cached_database_version). Rationale:
   storage config content is not validated, keys have safe defaults, downgrades ignore
   them. That is true today; agree as a team that this is the general policy for additive
   storage-config keys (and note it somewhere), since a future stricter validation of
   storage_config would turn these silently-ignored keys into errors on downgrade.

8. Python API parity. Caching is currently driven via CLI flags and raw storage_config
   keys; from Python one has to mutate profile.storage_config dicts by hand (as the tests do).
   If notebook users are a target audience (likely, for archive profiles), consider a supported
   API, e.g. SqliteZipBackend.create_profile(..., use_cache=True) or arguments on
   load_profile.

9. Concurrent-use guarantees. Writes are atomic (temp file + rename), but there is no
   locking between a reader holding an open session on a cache entry and cache-clear unlinking
   it (fine on POSIX as inode stays alive, but fails/behaves differently on Windows) and no
   protection against two processes downloading the same large remote database simultaneously
   (wasted bandwidth, last-write-wins). Acceptable, or worth a lock file, or reusing somehow the
   already existing methods for profile locking in AiiDA? (Note: probably the latter is not possible as
   they are per-profile)

[Bud-Macaulay]

Seems somewhat to address #7405 - which will help me on MaterialsCloud.

[Bud-Macaulay]

Hi, am currently testing this.

I haven't looked into it but a quick optimization would be to check the profile name isn't already taken before downloading anything since at the moment

verdi profile setup core.sqlite_zip -n --profile-name temp --filepath ... churns for quite a while if temp profile exists.

Additionally - just a thought, perhaps the same sort of feature could be ported to verdi archive import ...,

---

NOTE: After testing - I can comment this "just works" for my current usecase HERE (downloading and querying db),

I haven't used the file-access aspect of the PR, nor have i looked at the code at all, however.

[Bud-Macaulay]

Quick thought is to check how this behaves if a user swaps the file at the remote destination with a different AiiDA profile or even a malformed/faulty file...

If we understand this behaviour i will know whether i will be able to hotswap .aiida profiles with updated records without causing issues

It could also be a potential security risk - someone points users to his web .aiida file and then after connection changes the file to something malicious (a zip bomb or something) ...