fix(storage): cache open HDF5 read handles for multi-bag bag files

agents/DEVELOPING.md

@@ -95,7 +95,7 @@ just want to *use* the library (decorator, config, backends), see
 **Storage layout (`storage/`):**
 - `base.py` — `OperationContext(ABC)` (the new root after PR #604) exposes the `_operation_context(key, *, intent=Intent.WRITE)` context-manager hook — chain via `super()._operation_context(key, intent=intent)`; the `intent: Intent` kw-only param (PR #601) is a `StrEnum` with `WRITE` (the default — takes the exclusive lock) and `READ` (a **no-op** today: the lock mixins short-circuit it and acquire nothing; reserved for a future reader-writer/shared lock, so it must never guard a read-modify-write). `CacheStack._operation_context` enters its non-`stack[0]` members with `READ`, so a transfer into a stack only really locks `stack[0]` where the write lands. `KeyManagement(OperationContext)` adds the abstract `list`/`_evict`/`_contains` plus concrete `evict`/`contains`/`expand`/`shrink`/`_normalize_key` (`expand` raises `KeyError` for prefixes shorter than 4 chars). `StorageBackend(KeyManagement)` adds `put`/`get`. Domain ABCs `ValueStorage`, `CallStorage` (the latter also exposes `transform(func=None)` — re-saves every entry through *func*; entries whose `to_lookup_key()` changes are saved under the new key and the old one is evicted, unchanged ones are re-saved idempotently. `func=None` applies the identity, which is the form to use after a hash-function change. The analogous `Cache.redigest` does its own loop because it must update `values` too). **Bridge mixins** `ValueMixin`/`CallMixin` implement `save`/`load`[/`query`] on top of `put`/`get`. Also: `SaveError`, `AmbiguousDigestError`, `_resolve_prefix` (used by both base and `Sql`). `OperationContext`, `KeyManagement`, `Intent`, `StorageBackend`, `ValueStorage`/`ValueMixin`, `CallStorage`/`CallMixin` are all re-exported from `fleche.storage`.
 - `file.py` — `FileStorage` base for disk-backed backends (also re-exported from `fleche.storage` for subclassing). Locking is via the `filelock` package: writes use `filelock.FileLock(lock_path, timeout=lock_timeout)` directly; reads go through `_file_read_lock_with_fallback`, which on lock timeout logs a warning and proceeds **without** the lock (a missing/torn file then surfaces as `KeyError`). `root` is resolved (`expanduser`+`absolute`+`resolve`) in `__post_init__`. Subclasses implement `_to_file`/`_from_file`; compression and signing live in `pickle_file.py`, not here.
-- `memory.py`, `void.py`, `pickle_file.py` (+`PickleFileBackend.with_pickle`/`with_cloudpickle`/`with_dill`; `compress_all`/`decompress_all` migration helpers; gzip auto-detected by `\x1f\x8b` magic on read), `bagofholding_file.py` (+`rebag(version_validator="none")` re-saves every bag through a chosen validator, useful when older bags would fail strict version checks; `_from_file` opens the bag with `H5Bag(path, _skip_load=True).load(...)` so the file is opened exactly once per read instead of twice — PR #616; optional `prefix_length: int | None = 2` on `BagOfHoldingH5FileBackend` multiplexes keys sharing the first `prefix_length` chars into one `root/{prefix}.h5` file as sibling groups named by the full key, via bagofholding 0.1.12's `file.h5/group` path addressing — single-level, fixed-length split only; the default `2` multi-bags into up to 256 files, `prefix_length=0` restores the one-file-per-key layout, and `prefix_length=None` infers the length from the files already in `root` (empty root → the default `2`) — PR #746 closes #556. The PR #748 add-ons: `refix(n)` copies every entry into a new `prefix_length` layout and returns the re-fixed storage, leaving the receiver invariant (target must be an explicit int, `0` = per-key; fails fast on unreadable entries; resumable — entries already in the target layout are skipped), `__post_init__` raises `ValueError` when the configured `prefix_length` doesn't match the layout of files already in `root` (skippable via the init-only `check_consistency=False`, which requires an explicit `prefix_length` and makes the instance blind to all other layouts' files), and the `consolidate(root, n)` classmethod repairs a mixed-layout root by refixing every other prefix length it finds to `n`), `sql.py` — concrete backends (each exposes `Value*` and/or `Call*` classes; `sql.py` only has `Sql` for calls). `MemoryBackend.put`/`get` deep-copy values, so mutating a stored object after retrieval does not affect later reads. `Sql.query` always pushes name/module/version/code_digest/result and argument filters down to SQL (arguments are compared as `digest(value)` strings via `JOIN`s on the `arguments` table). Metadata filters are pushed down only when *every* filter value is a simple scalar (`str`/`bool`/`int`/`float`) via JSON-extract `as_*` casts; any `None` or complex (e.g. list) value disables metadata pushdown only — name/argument filters still apply at SQL level — and the post-load `meta_matches` check runs on every yielded result regardless. `Sql` does not inherit from `CallMixin` — it implements `CallStorage.save` directly, folding the existing-row check into a private `_persist_call` helper that runs in one transaction (no separate `contains`+`evict` round-trip), with `PerKeyLockMixin` keeping concurrent saves serialised (see `regression/test_sql_concurrent_save.py`). The companion private helper `_fetch_call` materialises a `DigestedCall` from a row plus its argument/metadata child rows; `Sql` does not expose any `put`/`get` methods (it isn't a `StorageBackend` subclass). `_coerce_sqlite_url` accepts a bare path (treated as sqlite, parent dir auto-created), a `sqlite:` URL, or any other SQLAlchemy URL (e.g. `postgresql://`, `mysql+pymysql://`) verbatim. `:memory:` is special-cased. SQLite-only PRAGMAs are enabled via a `connect` event listener (gated on `dialect.name == "sqlite"`): `foreign_keys=ON` always, plus `journal_mode=WAL` for file-backed DBs (skipped for `:memory:`). The schema is three tables: `calls`, `arguments` (one row per arg, `UNIQUE(call_key, name)`, ordered by `position`), `metadata` (one JSON blob per metadata namespace, `UNIQUE(call_key, name)`). `arguments.call_key`/`metadata.call_key` are FKs with `ondelete="CASCADE"`, so `_evict` is a single `session.execute(delete(CallModel).where(...))` (SQLAlchemy DSL, no raw SQL or ORM materialisation) and lets the DB clean up child rows (SQLite needs the `foreign_keys=ON` PRAGMA the dialect listener installs; Postgres/MySQL enforce FK cascades natively). MySQL/MariaDB needs explicit `VARCHAR(255)` for indexable name columns; other dialects get unbounded `TEXT` via `String().with_variant(...)`. `Sql` also accepts `check_same_thread=False` only for sqlite drivers (skipped for Postgres/MySQL where the flag would raise at connect).
+- `memory.py`, `void.py`, `pickle_file.py` (+`PickleFileBackend.with_pickle`/`with_cloudpickle`/`with_dill`; `compress_all`/`decompress_all` migration helpers; gzip auto-detected by `\x1f\x8b` magic on read), `bagofholding_file.py` (+`rebag(version_validator="none")` re-saves every bag through a chosen validator, useful when older bags would fail strict version checks; `_from_file` opens the bag with `H5Bag(path, _skip_load=True).load(...)` so the file is opened exactly once per read instead of twice — PR #616; optional `prefix_length: int | None = 2` on `BagOfHoldingH5FileBackend` multiplexes keys sharing the first `prefix_length` chars into one `root/{prefix}.h5` file as sibling groups named by the full key, via bagofholding 0.1.12's `file.h5/group` path addressing — single-level, fixed-length split only; the default `2` multi-bags into up to 256 files, `prefix_length=0` restores the one-file-per-key layout, and `prefix_length=None` infers the length from the files already in `root` (empty root → the default `2`) — PR #746 closes #556. The PR #748 add-ons: `refix(n)` copies every entry into a new `prefix_length` layout and returns the re-fixed storage, leaving the receiver invariant (target must be an explicit int, `0` = per-key; fails fast on unreadable entries; resumable — entries already in the target layout are skipped), `__post_init__` raises `ValueError` when the configured `prefix_length` doesn't match the layout of files already in `root` (skippable via the init-only `check_consistency=False`, which requires an explicit `prefix_length` and makes the instance blind to all other layouts' files), and the `consolidate(root, n)` classmethod repairs a mixed-layout root by refixing every other prefix length it finds to `n`. The #625 multi-bag perf fix adds a **process-wide read-handle cache** (`_BagHandleCache` / module-level `_bag_handles`): `contains`/`list` reuse open read-only `h5py.File` handles instead of paying a full HDF5 open per call, indexed in a `WeakValueDictionary` by absolute path with a strong-ref MRU of `_MAX_OPEN_BAGS` keeping them alive between operations, validated against an `(inode, mtime_ns, size)` stat signature on every acquisition. Handles are opened with `locking=False` so they don't block other processes' writes; the flip side is that HDF5 rejects *any* same-process open whose mode or locking flags differ from a live handle's, so every path that lets H5Bag open the file itself (`_to_file`, `_from_file`, `rebag`) or writes via `h5py.File(..., "a")` (`_evict`) must run inside `_bag_writer(key)`, which closes the cached handle and holds the per-bag in-process lock across the operation; readers hold the same lock for the duration of their handle use (`_bag_reader`). The cache is keyed by path process-wide — not per-instance — precisely because a handle cached by one storage instance must be closable by any other instance addressing the same file. `evict` still pays the write open/close (structural: a durable per-key delete in a shared HDF5 file)), `sql.py` — concrete backends (each exposes `Value*` and/or `Call*` classes; `sql.py` only has `Sql` for calls). `MemoryBackend.put`/`get` deep-copy values, so mutating a stored object after retrieval does not affect later reads. `Sql.query` always pushes name/module/version/code_digest/result and argument filters down to SQL (arguments are compared as `digest(value)` strings via `JOIN`s on the `arguments` table). Metadata filters are pushed down only when *every* filter value is a simple scalar (`str`/`bool`/`int`/`float`) via JSON-extract `as_*` casts; any `None` or complex (e.g. list) value disables metadata pushdown only — name/argument filters still apply at SQL level — and the post-load `meta_matches` check runs on every yielded result regardless. `Sql` does not inherit from `CallMixin` — it implements `CallStorage.save` directly, folding the existing-row check into a private `_persist_call` helper that runs in one transaction (no separate `contains`+`evict` round-trip), with `PerKeyLockMixin` keeping concurrent saves serialised (see `regression/test_sql_concurrent_save.py`). The companion private helper `_fetch_call` materialises a `DigestedCall` from a row plus its argument/metadata child rows; `Sql` does not expose any `put`/`get` methods (it isn't a `StorageBackend` subclass). `_coerce_sqlite_url` accepts a bare path (treated as sqlite, parent dir auto-created), a `sqlite:` URL, or any other SQLAlchemy URL (e.g. `postgresql://`, `mysql+pymysql://`) verbatim. `:memory:` is special-cased. SQLite-only PRAGMAs are enabled via a `connect` event listener (gated on `dialect.name == "sqlite"`): `foreign_keys=ON` always, plus `journal_mode=WAL` for file-backed DBs (skipped for `:memory:`). The schema is three tables: `calls`, `arguments` (one row per arg, `UNIQUE(call_key, name)`, ordered by `position`), `metadata` (one JSON blob per metadata namespace, `UNIQUE(call_key, name)`). `arguments.call_key`/`metadata.call_key` are FKs with `ondelete="CASCADE"`, so `_evict` is a single `session.execute(delete(CallModel).where(...))` (SQLAlchemy DSL, no raw SQL or ORM materialisation) and lets the DB clean up child rows (SQLite needs the `foreign_keys=ON` PRAGMA the dialect listener installs; Postgres/MySQL enforce FK cascades natively). MySQL/MariaDB needs explicit `VARCHAR(255)` for indexable name columns; other dialects get unbounded `TEXT` via `String().with_variant(...)`. `Sql` also accepts `check_same_thread=False` only for sqlite drivers (skipped for Postgres/MySQL where the flag would raise at connect).
 - `destructuring.py` — `DestructuringMixin` for recursive value splitting + `Digested` ABC with markers `DigestedIterable` (lists/tuples), `DigestedDict`, and `DigestedFields` (shared base reconstructing instances via `object.__new__` + `__setattr__`, bypassing `__init__`/`__post_init__`, so `init=False`/`InitVar`/frozen/slots fields round-trip) with two concrete subclasses `DigestedDataclass` (stdlib dataclasses) and `DigestedAttrs` (`attrs`-decorated classes). All preserve digest equivalence via `__digest__`. `DestructuringMixin` is a `ValueStorage` subclass — operates at the `save`/`load` layer, not `put`/`get`; compose **above** `ValueMixin` in the MRO. `remaining_depth` (default `1`) controls how deep structures are split across keys. `child_digests(key)→set[Digest]` returns the immediate digest references of a stored entry (raw, pre-`mend`); `count_reuses()` tallies how often each key is referenced as a sub-component (useful for GC-style audits). The `HasChildDigests` `runtime_checkable` Protocol declares the `child_digests` shape so `Cache.gc()` can opt-in to transitive walks via plain `isinstance` — any future value storage that exposes `child_digests` satisfies it without explicit registration. NamedTuples are deliberately **not** destructured (`_is_trojan_tuple` guard). `register_destructurer(pred, fn)` (re-exported from `fleche.storage`) appends a new entry to the module-level `_DESTRUCTURERS` list — first match wins, so registering a handler for a brand-new container type is safe; ordering matters if you want to override list/dict/dataclass/attrs.
 - `thread_safe.py` — `SerializingMixin(OperationContext)` (single `_PicklableRLock`), `PerKeyLockMixin(OperationContext)` (striped locks via a module-level `_per_instance_locks: WeakKeyDictionary`; per-instance `WeakValueDictionary[key, RLock]` — so the storage instance must be **hashable**, which all the frozen-dataclass concrete classes are). Both mixins override `_operation_context(key, *, intent=Intent.WRITE)`; chain via `super()._operation_context(key, intent=intent)`. Both short-circuit `intent=Intent.READ` to a plain `yield` (no lock acquired) — the no-op shared-lock placeholder. Inheriting from `OperationContext` directly (rather than `KeyManagement`) is what lets them attach at *either* the storage layer or the cache layer — `BaseCache(OperationContext)` since PR #622 closed #569. `_PicklableLock`/`_PicklableRLock` survive pickle round-trip with state not preserved — in-process only, NOT inter-process synchronisation.