Skip to content

feat(storage): cache filesystem paths by content - #797

Open
pmrv wants to merge 48 commits into
mainfrom
temppath
Open

feat(storage): cache filesystem paths by content#797
pmrv wants to merge 48 commits into
mainfrom
temppath

Conversation

@pmrv

@pmrv pmrv commented Jul 27, 2026

Copy link
Copy Markdown
Owner

Stacked on #793 (prepared-call) — merge that first; this PR shows only the path-caching commits on top of it.

What

First-class support for Path values in cached functions (#516, #33, #517): files and directories are stored by content and rematerialized under temporary paths on cache hits.

  • Identity: a file is (basename, content); a directory is its tree alone (child names in, root name out); plain bytes is the content-only escape hatch. Nonexistent paths warn and run uncached (What's the digest of a non existing file? #517).
  • Storage: PathValueMixin converts paths to FileBlob/DirectoryBlob records referencing content-addressed bytes blobs (identical bodies deduplicate across names, directories, and bytes values), wired into every default value storage. Digest salts mirror storage exactly, preserving digest(path) == save(path).
  • Materialization: hits return TempPaths — Path subclasses whose backing temp tree lives while any derived path is referenced (3.12+ via with_segments; 3.11 via a weak registry of live roots).
  • Destructuring: paths are found inside dicts / lists / tuples / dataclasses / attrs, as values or keys, at any depth. DigestedDict is generalized to a type-preserving DigestedMapping; the destructurer predicate is an exact-type allowlist (dict, OrderedDict, list, tuple) since subclasses may repurpose the type(value)(pairs) reconstruction contract (defaultdict, Counter).

Docs

  • docs/usage/file_semantics.rst — the full user-facing contract (identity, materialization, lifetime, nesting, opaque containers, argument mutation, fidelity limits). Black-box validated: agents restricted to reading only this page wrote assertion scripts against it over three rounds; every failed or ambiguous expectation was folded back into the text.
  • Recipes page, dev-internals page, and a runnable notebooks/PathsInContainers.ipynb exploring the container-nesting edge cases.

Interplay with #793

Path arguments make argument mutation easy (writing into a received directory), so this branch is where the two-phase save protocol pays off visibly: tests/integration/test_path_mutation.py pins that mutating consumers hit on honest repeats, the mutated tree is a distinct call, and a mutated argument passed back out is captured in its final state.

Full suite green on Python 3.13 and 3.11.

🤖 Generated with Claude Code

pmrv and others added 7 commits July 26, 2026 12:43
The wrapper digested arguments twice from live state: once before the
function body (the lookup key) and once after (when Cache.save stashed
the call record).  A function that mutated an argument — e.g. appended
to a received list, or wrote into a received directory — was therefore
recorded under the post-mutation content: honest repeat calls could
never hit it, while a call passing the mutated state would false-hit a
result computed from different input.

Fleche commits pure functions: the recorded identity of a call is now
sealed from the arguments as passed, via a two-phase save protocol:

- cache.prepare(call) -> PreparedCall stores argument values and seals
  the lookup key before the body runs; PreparedCall.commit(result,
  metadata) stores the result as returned (a mutated argument passed
  back out is captured in its final state) and files the record;
  PreparedCall.abandon() releases without recording (body raised or
  result uncacheable).  PreparedCall is also a context manager that
  abandons when the block exits uncommitted.
- prepare is implemented once on BaseCache over a new save_value
  primitive (the write-side counterpart of load_value), which wrappers
  forward, CacheStack routes to stack[0], and SshCache sends over the
  wire; read-only caches admit digest-only (nothing written, and the
  commit after the body is rejected — matching the previous post-body
  rejection behavior).
- Cache.save takes the fully digested record; a live Call is still
  accepted as the degenerate one-shot form (values static, nothing to
  drift), so post-hoc saving — transfers, redigest, external callers —
  keeps working unchanged.

BREAKING CHANGE: calls of argument-mutating functions are now recorded
under the pre-call argument state; caches populated by such functions
will re-execute once and re-record under the correct key.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Rename DigestedDict to DigestedMapping and make it generic over any
Mapping rather than only dict, preserving the concrete mapping type
(e.g. OrderedDict) and subclass identity through a save/load round-trip.

A backward-compatible DigestedDict alias is kept (re-exported from
fleche.caches and fleche.storage.destructuring) so existing imports keep
working.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Store and retrieve pathlib.Path values by their content rather than as
opaque path strings, so caching a function that takes or returns a Path is
portable and reproducible across machines. Following git's split — content
is content-addressed, names live in trees:

- a file is keyed on (name, content): it keeps its name and extension on a
  cache hit, and its content (plain bytes) deduplicates — shared across
  names and with bytes values. To key on content alone, return bytes.
- a directory is keyed on its tree alone; its incidental root name is
  dropped (a reloaded directory is named by its digest, its children by
  their real names).

PathValueMixin owns the traversal and is wired into the default value
storages (memory, pickle, H5) between DestructuringMixin and ValueMixin: a
file becomes a FileBlob(name, content) record, a directory a DirectoryBlob
tree, and content lives in plain bytes blobs. The digest Path arm mirrors
storage so digest(path) == values.save(path) — the invariant cached lookups
of path arguments and results depend on.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…otebook

Add a light recipes page and a path-storage dev-guide deep dive — content
addressing, the digest(path) == values.save(path) invariant, deduplication,
and the name-aware-files / content-only-directories model — wire both into
the toctree, and add a runnable Files.ipynb walkthrough registered in the
notebook integration test.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Companion to Files.ipynb: shows the intended UX of Path values nested in
dicts/lists/dataclasses, then the edge cases — hits changing location and
type, aliasing loss, Path dict-keys mending into new keys, per-hit
materialization and temp-file lifetime, opaque containers storing paths by
location, and the exact-type destructuring allowlist.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
New docs/usage/file_semantics.rst spells out the full user-facing contract
for Path values: call identity ((basename, content) for files, tree-only
for directories, content-keyed bytes), materialization on hits (fresh
temp copies, faithful basenames, deterministic digest root names, order
preservation), lifetime of materialized paths, nesting in destructured
containers (dict keys, aliasing, exact-type allowlist), opaque-container
location semantics, the do-not-mutate-arguments rule, and fidelity limits
(permissions, symlinks, metadata).

The page was validated black-box: agents restricted to reading only this
page wrote assertion scripts against it in three rounds; every failed or
ambiguous expectation was folded back into the text.  The mutation rule
and fidelity limits were discovered this way.

Also cross-links the page from the TL;DR, the files recipe, and the path
storage internals page.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Integration coverage for the two-phase save protocol meeting path
storage: a function writing into a received directory hits on honest
repeats, the mutated tree is a distinct call, and a mutated argument
passed back out is captured in its final state.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@claude claude Bot added the benchmark Run benchmark action label Jul 27, 2026
@claude

claude Bot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

🤖 CI failure summary for 6ed3580 on temppath

Summary
Type checking failed with three type errors in src/fleche/storage/. The code is calling object.__init__() and type() constructors with arguments where the type signature expects none.

Root cause
Two instances of calling type constructors with an iterable argument, and one unsupported class base:

error[too-many-positional-arguments]: Too many positional arguments to `object.__init__`: expected 1, got 2
   --> src/fleche/storage/destructuring.py:133:13 & 146:28

error[unsupported-base]: Unsupported class base
  --> src/fleche/storage/paths.py:12:16
  class TempPath(type(Path())):

Suggested fix

  1. In destructuring.py:133 and 146: replace direct type construction with a dict constructor or explicit object initialization
  2. In paths.py:12: change the base class from type(Path()) to pathlib.Path directly, or use a compatible mixin approach
  3. Run ty check src/ locally to verify the fixes

Generated by Claude Haiku 4.5.

@github-actions

Copy link
Copy Markdown
Contributor

Benchmark Results

Baseline: efeeebc (recomputed) · HEAD~: 2da4b10

Delta

Significant changes (|Δ| > 10%): 7 — 233 other rows hidden

Integration

configuration workload function time @ efeeebc time @ head Δ vs base Δ vs 2da4b10
SizeLimitedCache(Memory,max=10) data_heavy hit 190 µs 190 µs +0.0% 🟢 -64.8%
H5+Sql lightweight miss 7.3 ms 6.1 ms 🟢 -16.4% +1.7%
H5+Sql data_heavy miss 7.8 ms 6.8 ms 🟢 -12.8% +0.0%
H5+Sql compute_heavy miss 7.9 ms 6.9 ms 🟢 -12.7% +0.0%
SizeLimitedCache(Memory,max=10) compute_heavy hit 170 µs 190 µs 🔴 +11.8% +5.6%

Value Storage

configuration workload function time @ efeeebc time @ head Δ vs base Δ vs 2da4b10
CloudpickleFile_Signed nested_structures save 360 µs 390 µs +8.3% 🔴 +11.4%
DillFile numpy_arrays save 620 µs 660 µs +6.5% 🔴 +10.0%

Full results

Call Storage

calls

configuration contains_hit contains_miss evict load save
🔴 SqlFile 🟩 270 µs 🟩 250 µs 🟩 480 µs 🟩 750 µs 🟥 2.2 ms
🔴 SqlMemory 🟩 260 µs 🟩 250 µs 🟩 290 µs 🟩 730 µs 🟧 2 ms
Digest
workload digest
Dict (small) 🟩 1.5 ms
Float 🟩 270 µs
Integer 🟩 140 µs
List (integers, len<100) 🟩 970 µs
List (integers, len>100) 🟥 28 ms
Nested (Random Hypothesis) 🟩 3.7 ms
None 🟩 120 µs
Numpy (integers, len<100) 🟩 940 µs
Numpy (integers, len>100) 🟩 2.7 ms
String (len<100) 🟩 140 µs
String (len>100) 🟩 230 µs
Integration

compute_heavy

configuration contains_hit contains_miss hit miss
🟠 H5+Sql 🟩 390 µs 🟩 390 µs 🟨 3 ms 🟥 6.9 ms
🟢 Pickle+Sql 🟩 390 µs 🟩 380 µs 🟩 1.4 ms 🟨 4 ms
🟣 Memory 🟩 81 µs 🟩 82 µs 🟩 160 µs 🟩 1.2 ms
🟣 Memory(Raw) 🟩 76 µs 🟩 76 µs 🟩 150 µs 🟩 1.1 ms
🟣 SizeLimitedCache(Memory,max=10) 🟩 84 µs 🟩 85 µs 🟩 190 µs 🟩 1.2 ms
🟣 SizeLimitedCache(Memory,max=100) 🟩 84 µs 🟩 84 µs 🟩 160 µs 🟩 1.2 ms
🟤 Memory+Sqlite(:memory:) 🟩 380 µs 🟩 370 µs 🟩 980 µs 🟨 2.7 ms

data_heavy

configuration contains_hit contains_miss hit miss
🟠 H5+Sql 🟩 400 µs 🟩 380 µs 🟨 3.1 ms 🟥 6.8 ms
🟢 Pickle+Sql 🟩 390 µs 🟩 380 µs 🟩 1.4 ms 🟨 3.5 ms
🟣 Memory 🟩 82 µs 🟩 82 µs 🟩 170 µs 🟩 490 µs
🟣 Memory(Raw) 🟩 75 µs 🟩 76 µs 🟩 160 µs 🟩 450 µs
🟣 SizeLimitedCache(Memory,max=10) 🟩 84 µs 🟩 84 µs 🟩 190 µs 🟩 500 µs
🟣 SizeLimitedCache(Memory,max=100) 🟩 84 µs 🟩 84 µs 🟩 180 µs 🟩 510 µs
🟤 Memory+Sqlite(:memory:) 🟩 370 µs 🟩 370 µs 🟩 1 ms 🟩 2 ms

lightweight

configuration contains_hit contains_miss hit miss
🟠 H5+Sql 🟩 390 µs 🟩 370 µs 🟨 2.9 ms 🟥 6.1 ms
🟢 Pickle+Sql 🟩 390 µs 🟩 380 µs 🟩 1.4 ms 🟨 3.3 ms
🟣 Memory 🟩 83 µs 🟩 83 µs 🟩 160 µs 🟩 350 µs
🟣 Memory(Raw) 🟩 74 µs 🟩 75 µs 🟩 150 µs 🟩 310 µs
🟣 SizeLimitedCache(Memory,max=10) 🟩 83 µs 🟩 84 µs 🟩 180 µs 🟩 380 µs
🟣 SizeLimitedCache(Memory,max=100) 🟩 84 µs 🟩 85 µs 🟩 170 µs 🟩 360 µs
🟤 Memory+Sqlite(:memory:) 🟩 380 µs 🟩 370 µs 🟩 1 ms 🟩 1.8 ms
Value Storage

nested_structures

configuration contains_hit contains_miss evict load save
🔵 CloudpickleFile 🟩 33 µs 🟩 33 µs 🟩 57 µs 🟩 290 µs 🟩 380 µs
🔵 CloudpickleFile_Signed 🟩 33 µs 🟩 33 µs 🟩 57 µs 🟩 300 µs 🟩 390 µs
🟠 BagOfHoldingH5File 🟩 270 µs 🟩 33 µs 🟩 260 µs 🟧 1.1 ms 🟥 1.5 ms
🟡 DillFile 🟩 33 µs 🟩 33 µs 🟩 59 µs 🟩 290 µs 🟩 400 µs
🟡 DillFile_Signed 🟩 33 µs 🟩 34 µs 🟩 57 µs 🟩 300 µs 🟩 400 µs
🟢 PickleFile 🟩 33 µs 🟩 33 µs 🟩 57 µs 🟩 280 µs 🟩 350 µs
🟢 PickleFile_Signed 🟩 33 µs 🟩 33 µs 🟩 55 µs 🟩 290 µs 🟩 380 µs
🟣 Memory 🟩 7 µs 🟩 6.9 µs 🟩 7.3 µs 🟩 8.8 µs 🟩 9.8 µs
🟣 Memory(Raw) 🟩 1.9 µs 🟩 1.8 µs 🟩 2.1 µs 🟩 3.3 µs 🟩 3.7 µs

numpy_arrays

configuration contains_hit contains_miss evict load save
🔵 CloudpickleFile 🟩 33 µs 🟩 34 µs 🟩 120 µs 🟩 310 µs 🟩 470 µs
🔵 CloudpickleFile_Signed 🟩 33 µs 🟩 34 µs 🟩 110 µs 🟩 380 µs 🟩 540 µs
🟠 BagOfHoldingH5File 🟩 270 µs 🟩 34 µs 🟩 690 µs 🟧 1.8 ms 🟥 2.3 ms
🟡 DillFile 🟩 33 µs 🟩 34 µs 🟩 120 µs 🟩 330 µs 🟩 660 µs
🟡 DillFile_Signed 🟩 33 µs 🟩 33 µs 🟩 120 µs 🟩 400 µs 🟩 720 µs
🟢 PickleFile 🟩 34 µs 🟩 34 µs 🟩 120 µs 🟩 320 µs 🟩 440 µs
🟢 PickleFile_Signed 🟩 33 µs 🟩 33 µs 🟩 120 µs 🟩 380 µs 🟩 510 µs
🟣 Memory 🟩 6.9 µs 🟩 6.9 µs 🟩 7.5 µs 🟩 14 µs 🟩 15 µs
🟣 Memory(Raw) 🟩 1.9 µs 🟩 1.8 µs 🟩 2.1 µs 🟩 7.8 µs 🟩 8.3 µs

small_strings

configuration contains_hit contains_miss evict load save
🔵 CloudpickleFile 🟩 33 µs 🟩 32 µs 🟩 100 µs 🟩 280 µs 🟩 350 µs
🔵 CloudpickleFile_Signed 🟩 32 µs 🟩 33 µs 🟩 110 µs 🟩 290 µs 🟩 360 µs
🟠 BagOfHoldingH5File 🟩 280 µs 🟩 33 µs 🟨 600 µs 🟥 1.7 ms 🟥 1.7 ms
🟡 DillFile 🟩 33 µs 🟩 33 µs 🟩 110 µs 🟩 290 µs 🟩 370 µs
🟡 DillFile_Signed 🟩 32 µs 🟩 33 µs 🟩 110 µs 🟩 300 µs 🟩 380 µs
🟢 PickleFile 🟩 33 µs 🟩 33 µs 🟩 110 µs 🟩 280 µs 🟩 350 µs
🟢 PickleFile_Signed 🟩 33 µs 🟩 33 µs 🟩 110 µs 🟩 290 µs 🟩 350 µs
🟣 Memory 🟩 7 µs 🟩 6.9 µs 🟩 7 µs 🟩 8.7 µs 🟩 9.2 µs
🟣 Memory(Raw) 🟩 1.9 µs 🟩 1.8 µs 🟩 1.9 µs 🟩 3.2 µs 🟩 3.6 µs

Address review on #793:

- `BaseCache.save_value` is gone (and with it the new abstract method on
  `BaseCache`, which broke the API for existing subclasses).  `Cache.prepare`
  now hands its whole value storage to `Call.stash`, which already leaves an
  unknown result as `None`.
- `PreparedCall.commit` no longer stores the result itself: it attaches the
  live value to the record and lets `Cache.save` store it at the end, next to
  the one-shot `Call` path.
- `PreparedCall` drops the redundant `call` and `key` fields; callers that
  need the key have the `Call` or the `DigestedCall` at hand.
- `BaseCache.prepare` keeps a concrete digest-only default (key sealed,
  nothing written), so third-party caches keep working and `ReadOnlyMixin`
  only restates it to win the MRO over `CacheWrapper`.  Wrappers and stacks
  delegate storage inward and rebind the cache so the commit runs through
  their own `save` policy.
- Over SSH, `save_value` (one round trip per value) is replaced by a single
  `prepare` RPC that stashes the arguments remotely and returns the sealed
  record; read-only remotes short-circuit to the local digest-only admission.
- `CacheStack.save` now returns the record key, which `commit` hands back.

Co-authored-by: Marvin Poul <2719909+pmrv@users.noreply.github.com>

@pmrv pmrv left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

revise

Comment thread docs/usage/file_semantics.rst Outdated
fleche caches *pure* functions. What a function does to its arguments
without passing it back out is invisible to the cache — treat received paths
as read-only and write outputs to a fresh directory (``tempfile.mkdtemp``).
A mutated argument that *is* returned is captured faithfully in its final,

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

feels almost more like it should go into a more generic section rather than file specific.

Comment thread docs/usage/file_semantics.rst Outdated
Paths nested inside containers
------------------------------

Paths are found and content-stored inside the containers fleche takes apart:

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cross ref destructuring here, since that's the key

Comment thread notebooks/Files.ipynb

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

save executed version

Comment thread notebooks/Files.ipynb

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

move the completedprocess digester into fleche itself

@pmrv

pmrv commented Aug 5, 2026

Copy link
Copy Markdown
Owner Author

Interactions with ssh cache unclear

claude added 3 commits August 6, 2026 16:13
…he record

commit now builds the filed record with dataclasses.replace, so the live
result is never parked on the prepared record after save, and raises
RuntimeError on a second commit or a commit after abandon (abandon stays
idempotent).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QT4DxU9dVALrMvff7setnk
Mirror the existing Rejected handling on save: a cache that refuses
admission logs a warning and the call executes uncached instead of
failing before the body runs.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QT4DxU9dVALrMvff7setnk
- SshCache.prepare: live wire round trip (arguments stashed server-side
  before the body, mutation-coherent commit) and the read-only local
  digest-only admission with zero RPCs.
- PreparedCall: double commit and commit-after-abandon raise; commit
  leaves the prepared record unmutated.
- The end-to-end tests now run on a private Cache instance: populating
  the shared named-memory singleton (load_cache_config caches it in
  _live_caches) leaked into any later test that round-trips that cache's
  config, e.g. test_run_server_serves_active_cache_over_stdio_until_eof.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QT4DxU9dVALrMvff7setnk
claude added 3 commits August 6, 2026 16:38
… stores it

Combines the #827 sketch with the replace-based commit: DigestedCall goes
back to holding digests only, PreparedCall carries the live result and
metadata privately, and Cache.save stores the result when handed the
PreparedCall itself.  PreparedCall pickles without its process-local
cache binding so a commit against SshCache ships only the sealed record
and the pending result.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QT4DxU9dVALrMvff7setnk
Policy rejections keep their warning; actual errors (storage faults,
lost connections) are logged with traceback and the body still runs.
Also drop the dead call.metadata assignment left over from the commit
protocol switch.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QT4DxU9dVALrMvff7setnk
SshCache.save now recreates Cache.save's ending for a PreparedCall: a
write-gated save_value RPC stores the result value remotely, then the
plain digested record is filed — only Call and DigestedCall ever cross
the wire, so PreparedCall.__getstate__ goes away.  The resolve step
shared by both cache types moves onto PreparedCall.  Server-side,
save_value routes through prepare with a synthetic never-committed call,
so wrappers and stacks direct the value to the same storage their saves
use without any new public cache method.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QT4DxU9dVALrMvff7setnk
pmrv pushed a commit that referenced this pull request Aug 6, 2026
…edProcess

Addresses the four review threads on #797.

- The "argument mutation" rules were stated as a file-specific caveat, but
  they are general: a mutated list argument behaves identically (verified —
  cold call mutates, hit leaves the argument untouched, and a mutated
  argument that *is* returned is stored post-mutation). Move the contract to
  a new `usage/purity` page covering keyed-as-passed, mutation not replayed,
  and side effects being cold-call-only; `file_semantics` keeps the path
  instance and links out.

- Cross-reference destructuring from "Paths nested inside containers", since
  that is the mechanism the section is really about: only destructured
  children reach the path machinery, so `_DESTRUCTURERS` *is* the list of
  places a nested path gets content treatment. Explains the container list,
  the exact-types caveat, and the opaque-container section in one stroke.

- Move the `CompletedProcess` digester out of `notebooks/Files.ipynb` and
  into `digest.py` as a match arm (args + returncode + stdout + stderr).
  `run()`'s result is neither a dataclass nor iterable, so it was
  `Indigestible` and every shell-wrapping function needed the same hand-rolled
  hook. Purely additive — those values raised before, so no stored digest
  changes and no `hash_version` bump.

- Save the executed `Files.ipynb` (the notebook now demonstrates the built-in
  digest, no `add_hook` cell). Same for `PathsInContainers.ipynb`, which was
  added without a `docs/notebooks/` symlink, a toctree entry, or a slot in
  `test_notebooks.py` — all three now wired up.

Also records issue #829 (making paths genuinely work over SSH by doing the
blob reduction client-side) in `remote.py`, `dev/path_storage`, and the
DEVELOPING design-theme cheat sheet, and fixes a short title underline in
`usage/helpers`.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RascUYS7JgCMHHpXsYbPTf
…tocol

Implements the changes approved in the #793 review threads, revised per
the #825 review ("combine" with the #827 sketch; broadened prepare
guard; two-trip remote commit). Stacked onto `prepared-call` for merging
into that branch.

## Changes

- **Pending result lives on `PreparedCall`, never on `DigestedCall`**
(combining #827 with the replace-based commit): `PreparedCall` carries
`_result`/`_metadata` as private fields, `commit` hands the whole
`PreparedCall` to `cache.save`, and the cache stores the result and
files a record built via `PreparedCall.resolve` — `DigestedCall.result`
is back to a checkable `Digest | None`, and `commit` never mutates
`digested`. A bare `DigestedCall` still files as-is.
- **`PreparedCall` never goes over the wire**: `SshCache.save` recreates
`Cache.save`'s ending in two trips — a write-gated `save_value` RPC
stores the result value remotely, then the plain `DigestedCall` is
filed. Only `Call` and `DigestedCall` cross the wire (asserted by a spy
in the round-trip test). Server-side, `save_value` routes through
`prepare` with a synthetic never-committed call, so wrappers and stacks
direct the value to the same storage their saves use without adding a
public cache method; a failure between the trips leaves a
content-addressed orphan for gc, like any abandoned call.
- **`commit` is single-shot**: a second `commit` or a `commit` after
`abandon` raises `RuntimeError`; `abandon` stays idempotent so cleanup
paths can't trip it.
- **Wrapper degrades gracefully on any `prepare` failure**: `Rejected`
logs a policy warning; any other `Exception` (storage fault, lost
connection) logs with traceback — in both cases the body runs and
returns uncached instead of failing before it ever ran. Also drops the
dead `call.metadata` assignment.
- **Remote `prepare` tests**: live wire round trip (arguments stashed
server-side before the body runs, sealed key survives argument mutation,
two-trip commit) and the read-only branch (local digest-only admission,
zero RPCs, commit rejected locally before the first trip).
- **Test hygiene fix (drive-by)**: the e2e tests in
`test_prepared_call.py` activated the shared named-`"memory"` cache
stickily; that singleton lives in `config._live_caches`, so populating
it leaked into any later test that round-trips the named cache's config
(`test_run_server_serves_active_cache_over_stdio_until_eof[memory]`
crashes in `dataclasses.asdict` when the files run back to back — masked
in the full suite by `tests/unit/config`'s `_live_caches.clear()`
fixture). The tests now scope a private `Cache` instance.

Parked separately: the gc/in-flight race is #826.

## Testing

Full suite: 1659 passed, 11 skipped. `ty check src/` clean. The affected
file pair also passes in the previously-failing collection order.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01QT4DxU9dVALrMvff7setnk
pmrv and others added 3 commits August 6, 2026 16:30
Applies the remaining comment trims flagged in the #793 review (the rest
already landed with #825). Narration and before/after history go;
invariants stay:

- `BaseCache.prepare` docstring: 20 → 15 lines; keeps the override
contract, the digest-only base semantics, and the `digest(x) ==
values.save(x)` key-equality invariant.
- `Cache.prepare`: drops the result-lifecycle narration (documented on
`PreparedCall`/`save`); keeps the pre-body stash rationale and the
no-cache-level-lock explanation.
- `ReadOnlyMixin.prepare`: drops "the behavior save() rejection produced
before the two-phase protocol, minus the wasted writes"; keeps the MRO
reason and the seal-without-writing behavior.
- `SshCache.prepare`: drops the ", as before" tail; keeps the
Path-over-SSH caveat.

Comment-only diff. Affected test dirs pass (191 tests), `ty check src/`
clean.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01QT4DxU9dVALrMvff7setnk

---
_Generated by [Claude
Code](https://claude.ai/code/session_01QT4DxU9dVALrMvff7setnk)_

Co-authored-by: Claude <noreply@anthropic.com>
…edProcess (#831)

Answers the four inline review threads on #797.

- Move the general "fleche caches pure functions" contract — arguments keyed
  as passed, mutation and other side effects not replayed — out of
  file_semantics into a new usage/purity page. Verified it is not
  path-specific: a mutated list argument behaves identically.

- Reframe "Paths nested inside containers" around destructuring, since only
  destructured children reach the path machinery, so _DESTRUCTURERS is the
  list of places a nested path gets content treatment. Also document why
  "any depth" holds at any remaining_depth: a Path matches no destructurer,
  so it is always written out as its own entry rather than inlined.

- Move the CompletedProcess digester out of notebooks/Files.ipynb into
  digest.py as a match arm (args + returncode + stdout + stderr). Purely
  additive: those values raised Indigestible before, so no stored digest
  changes and no hash_version bump.

- Commit Files.ipynb and PathsInContainers.ipynb executed, and wire
  PathsInContainers into docs/notebooks/, the toctree, and test_notebooks.py.
  Replace the generic query cell in Files.ipynb with the value store's own
  view of a stored path. Reframe the PathsInContainers "Edge N" sections as
  Beware (consequences of caching by value) vs Caveat (real limits of the
  mending machinery).
`ty check src/` passes on `main` but reports three diagnostics here, so the
ty workflow fails for every PR into this branch (and now for the branch
itself).  Both are false positives on code that is correct at runtime:

- `DigestedMapping.mend` / `_rebuild_plain` call `type(value)(...)`, which ty
  narrows to `type[Mapping]`.  `Mapping` is an ABC with no `__init__`, so the
  call resolves to `object.__init__` and the argument reads as one positional
  too many.  The runtime type is always a concrete `dict`/`OrderedDict`.

- `TempPath(type(Path()))` uses a dynamic base, through which ty cannot
  resolve an MRO.  The indirection is required while Python 3.11 is
  supported — bare `Path` only became subclassable in 3.12.

Suppressed rather than reworked: a `cast` on the two constructors would
trade a checker complaint for an indirection, and the `TempPath` base cannot
change until the 3.11 floor moves.  Comments are anchored on the exact lines
ty reports, which is where it binds them.

`ty check src/` (pinned 0.0.62, as CI runs it): All checks passed!

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RascUYS7JgCMHHpXsYbPTf

pmrv commented Aug 6, 2026

Copy link
Copy Markdown
Owner Author

Pushed 4e2083e here as you asked (from the #831 thread) — ty check src/ is green on main but was reporting three diagnostics on this branch, so the ty workflow was failing for every PR into it, and after #831 landed, for the branch itself.

Three suppression comments, no behaviour change:

-            (self.get(storage, k), self.get(storage, v))
+            (self.get(storage, k), self.get(storage, v))  # ty: ignore[too-many-positional-arguments]

-        return type(value)(zip(children[:n], children[n:]))
+        return type(value)(zip(children[:n], children[n:]))  # ty: ignore[too-many-positional-arguments]

-class TempPath(type(Path())):
+class TempPath(type(Path())):  # ty: ignore[unsupported-base]

Both are false positives on code that's correct at runtime. In DigestedMapping, ty narrows type(value) to type[Mapping] — an ABC with no __init__ — so the call resolves to object.__init__ and the argument reads as one positional too many; the runtime type is always a concrete dict/OrderedDict. For TempPath, ty can't resolve an MRO through a dynamic base, and that indirection has to stay while 3.11 is supported, since bare Path only became subclassable in 3.12.

I suppressed rather than reworked: a cast on the two constructors trades a checker complaint for an indirection that earns nothing, and the TempPath base can't change until the 3.11 floor moves. Say the word if you'd rather have the cast version in destructuring.py. Placement matters — ty binds a suppression to the line it reports, which for the first one is the generator expression, not the return type(self.items)( above it.

Verified with the pinned ty == 0.0.62 CI uses: All checks passed! Suite unchanged at 1763 passed / 11 skipped.

#828 is rebased on top of this and green locally (1788 passed / 11 skipped).


Generated by Claude Code

pmrv and others added 2 commits August 6, 2026 16:59
Answers your comment on #797, ["Interactions with ssh cache
unclear"](#797 (comment)).
Targets `temppath`.

The four **inline** review threads on #797 moved to #831, which has
since **merged** — `temppath` was merged back in here (`f938975`, no
conflicts), so this PR is the `SshCache` × `Path` work only.

## What the interaction actually was

Probed with a real `python -m fleche remote --serve` subprocess, client
and server in different working directories so the same relative name
denotes different files on each side. Three behaviours, all silent:

| | before |
|---|---|
| server *can* see something at that name | stores **its** bytes,
returns **its** digest — `digest(x) == save_value(x)` broken, so the
record is filed under a key no client recomputes, and a load returns the
wrong content |
| server can see nothing there | `Indigestible` surfaces from inside an
RPC |
| loading a path stored remotely | server materializes into a temp dir
on **its** disk and sends back only the name — dangling here, and
unlinked there as soon as its own reference dies |

The last one is not fixable by holding a reference:
`PurePath.__reduce__` rebuilds from `parts` alone, so `TempPath`'s
`_temp_root` is dropped in transit and `_live_roots` is per-process. The
`save_value` comment already said "paths over SSH remain unsupported" —
nothing enforced it.

## What this does

Refuses, in both directions, via a new `RemotePathUnsupported`. It
subclasses `SaveError`, so the two-phase-save degradations already in
the codebase carry it and no caller special-cases anything:

- path **argument** → digest-only reference, **digest computed
locally**. The seal stays intact, lookups hit and miss correctly; only
the bytes aren't retrievable from the remote record.
- path **result** → `Rejected`. The call runs, returns the file, is
logged as not cached.
- **load** → raises, lazily. A record whose result is a path still loads
and queries; only touching the path value raises.

Detection is `storage.paths.find_path`, walking a value through a new
`storage.destructuring.child_slots` — the read-only half of
`_intern_rec`'s dispatch — so nesting is covered exactly as far as a
destructuring save reaches, and no further (a path inside an opaque
object is still stored by location, same as locally).

Escape hatches, both documented: return `bytes`, or put a local layer in
front of the remote one so saves never reach the wire.

## Follow-up (second commit)

Per your review, the problem itself is tracked as **#829** rather than
left as a comment. The shape is already implied by `PathValueMixin`: it
reduces a path to `bytes` plus a `FileBlob`/`DirectoryBlob`, all of
which ship fine, and those blobs' `__digest__` is *defined* to match the
`Path` digest arm — so running the reduction **client-side** keeps the
seal intact by construction, and needs no server change for saves (a
`FileBlob` isn't a `Path`, so it falls straight through the server's
path layer). The one genuinely new verb is an unmended `load_value` that
returns the blob instead of materializing it on the server. Recorded in
`RemotePathUnsupported`, `dev/path_storage`, and the DEVELOPING
design-theme cheat sheet.

## Docs

- `usage/file_semantics.rst`: new "Paths stop at a remote (SSH) cache"
contract section + a quick-reference row.
- `dev/path_storage.rst`: the mechanism, and the #829 sketch.
- `recipes/files_and_paths.rst`: a recipe. `SshCache`'s own docstring
too.

## Tests

`tests/unit/test_remote.py` — bare and nested
(list/tuple/nested-dict/dataclass) rejection with nothing reaching the
server; the `bytes` hatch still working; `prepare()` keeping
`prepared.key == call.to_lookup_key()` for a path argument (the
regression the guard buys); live-`Call` rejection; load-side rejection
incl. nested; a `LazyCall` whose arguments still read fine and only
`.result` raises.

`tests/unit/storage/test_paths.py` — `find_path` descent, opaque-leaf
boundary, cycle termination.

`tests/integration/test_remote.py` — the divergent-cwd reproduction
above, and a path-returning `@fleche` function running uncached against
a remote instead of breaking.

Re-verified after merging `temppath`: **1788 passed / 11 skipped**, docs
build succeeds. `ty check src/` reports **3** diagnostics — all already
on `temppath`, none from this PR; see [the #831
thread](#831 (comment))
for the diagnosis and a verified fix, still pending your call on where
it should land.

---------

Co-authored-by: Claude <noreply@anthropic.com>
Conflict in `SshCache`, in `remote.py` — and not only textual.  Both sides
changed `save`, and `prepared-call` adds `prepare`, an RPC that carries
argument *values* rather than digests.  That is a fourth way a `Path` can
reach the server, which the guard added in #828 did not cover: shipping the
live call hands the far side a path *string* to resolve against its own
filesystem, breaking `digest(x) == save_value(x)` exactly as before.

Resolved by keeping both sides and closing that route:

- `save` handles `PreparedCall` (ship result, then file the record) as well
  as the live-`Call` and `DigestedCall` forms, routing the result through
  `save_value` rather than `_rpc` so the path guard applies to it.
- `prepare` falls back to the local `BaseCache` two-phase prepare when any
  argument carries a path.  Each argument then goes through `save_value`,
  where a path degrades to a digest-only reference computed *locally* — the
  seal stays intact and lookups still hit, which is the same degradation
  #828 established for the other routes.

Also widens `find_path` itself, which was too narrow to be safe.  It mirrored
*destructuring*, but the invariant at stake is `digest(x) == save_value(x)`,
and `digest` recurses further than destructuring does: into namedtuples,
sets, and frozensets, reading the file in each.  A path hidden in one of
those therefore still decides the key while passing the guard, so
`Bundle(path, 0.5)` crossed the wire and the server digested that name
against its own disk.  The walk now follows `digest` instead, stopping where
`digest` stops (an arbitrary object is `Indigestible`, so no path inside it
can decide a key) and declining to consume one-shot iterables.

1807 passed, 11 skipped; `ty check src/` clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RascUYS7JgCMHHpXsYbPTf

pmrv commented Aug 6, 2026

Copy link
Copy Markdown
Owner Author

Conflict resolved in fa5d2d2 — but it wasn't only textual, and your instinct about the path/destructuring interaction was right, just in a different place than depth. Two real bugs came out of it.

1. prepare is a fourth values-over-the-wire route

prepared-call adds SshCache.prepare, which ships the whole live call so the remote stashes argument values before the body runs. #828's guard covered save_value, save, and load_value — not this. So a path argument would have gone over as a bare string for the server to resolve against its own filesystem: the same silent digest(x) == save_value(x) break, via a new door. (The old comment there even said "paths over SSH are unsupported" — again, nothing enforced it.)

Resolved by keeping both sides and closing the route:

  • save now handles PreparedCall as well as the live-Call and DigestedCall forms, and routes the result through save_value rather than _rpc so the guard applies to it.
  • prepare falls back to the local BaseCache two-phase prepare when any argument carries a path. Each argument then goes through save_value, where a path degrades to a digest-only reference computed locally — seal intact, lookups still hit. Same degradation fix(remote): refuse Path values at the SshCache boundary #828 established elsewhere, no new concept. The fallback is narrow: ordinary calls keep the one-round-trip path, pinned by a test.

2. The guard was mirroring the wrong walk

Chasing your concern I checked remaining_depth first, and it's clean — 36/36 across remaining_depth ∈ {0,1,2,3,5,10} × nesting depth ∈ {0,1,2,3,5,8}, with the original file deleted before load, and with the setting verified to actually change storage layout (17 → 10 → 3 → 1 entries). A Path matches no destructurer, so its depth stays inf, it's never inlined, and PathValueMixin always gets it.

But the check turned up something worse. find_path mirrored destructuring — and the invariant at stake is digest(x) == save_value(x), while digest recurses further than destructuring does:

value destructuring digest reads the file guard saw it (before)
[path] descends yes yes
Bundle(path, 0.5) (namedtuple) opaque yes no
{path} (set) opaque yes no
frozenset({path}) opaque yes no

A path hidden in any of those still decides the key, so it crossed the wire and the server digested that name against its own disk. Locally this is harmless — same process, same filesystem — which is exactly why it wasn't visible before.

find_path now follows digest, stopping where digest stops (an arbitrary object is Indigestible, so no path inside it can decide a key) and declining to consume one-shot iterables rather than exhausting them as a side effect.

Verification

1807 passed / 11 skipped, ty check src/ clean. New tests pin: prepare not shipping a path-carrying call while still using the remote for ordinary ones, commit rejecting a path result, and namedtuple/set paths being refused at the boundary.

Two API changes on your branch that my #828 tests had to follow: PreparedCall.keyto_lookup_key(), and Cache.save_value is gone in favour of the two-phase protocol.


Generated by Claude Code

GitHub flagged this PR un-mergeable, so this merge is to clear that rather
than to stay current: main's #856 rewrote the `agents/DEVELOPING.md` notebook
enumeration this branch also edits.  Resolved to the union — twelve notebooks
and twelve symlinks, main's `FiveMinuteTour` alongside this branch's `Files`
and `PathsInContainers`, and this branch's `purity`, `file_semantics`,
`path_storage` and `recipes/files_and_paths` pages.  `docs/index.rst`
auto-merged; verified its entries survived.

Two things come in with it that are main's, not this branch's, and I have not
silently fixed either:

**CI will be red on `FiveMinuteTour.ipynb`.**  It imports `ase`, which is a
dependency nowhere in `pyproject.toml` and is not installed by `tests.yml`.
`test_notebooks.py` globs every notebook with no skip, the `changes` gate
skips the suite for notebook-only PRs, and pushes to `main` don't run tests at
all — so it has never executed in CI, and the first PR touching `src/` after
it landed is the one that goes red.  Held this merge back for a day hoping for
a fix; taking the red now because a conflicted PR is worse than a red one.
Everything else passes: 1909 passed, 11 skipped, that one deselected.

**Sphinx now emits one warning**: `duplicate object description of fleche.D,
other instance in digests/digests_as_args`.  Checked rather than assumed —
`origin/main` builds clean in a worktree, this tree warns reproducibly across
two builds, and the file itself is byte-identical to main's, so it is the
larger page set here shifting autoapi's read order relative to the page that
carries `:no-index:`.  The build still succeeds; left for pmrv since the fix
belongs in the autoapi config, not in path work.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RascUYS7JgCMHHpXsYbPTf

pmrv commented Aug 11, 2026

Copy link
Copy Markdown
Owner Author

Merged main in ad7b883, which means CI here will go red on FiveMinuteTour.ipynb — the ase problem from my last comment, still unfixed on main.

I held the merge back for a day to keep this PR green. What forced it: #856 rewrote the same agents/DEVELOPING.md notebook enumeration this branch edits, so GitHub flipped #797 to dirty. A conflicted PR can't be merged at all, a red one can be read — so I took the red rather than leave the conflict sitting.

Everything except that one notebook passes: 1909 passed, 11 skipped, FiveMinuteTour deselected. The three options are unchanged (add ase to the tests extra / skip notebooks whose imports are unavailable / drop the ASE cell) and I still haven't picked one — say the word and I'll do it here or in a separate PR against main.

The conflict itself resolved to the union: twelve notebooks, twelve symlinks, main's FiveMinuteTour beside this branch's Files and PathsInContainers, and this branch's purity / file_semantics / path_storage / recipes/files_and_paths pages. docs/index.rst auto-merged and I verified its entries survived.

One new thing worth your eyes, unrelated to the above. Sphinx now emits a warning that it didn't before:

docs/autoapi/fleche/index.rst:256: WARNING: duplicate object description of
fleche.D, other instance in digests/digests_as_args, use :no-index: for one of them

I checked this rather than guessing at it:

  • origin/main builds clean — I built it in a separate worktree to be sure.
  • This tree warns reproducibly, twice in a row, fresh output dir each time.
  • docs/digests/digests_as_args.rst is byte-identical to main's and already carries :no-index:.

So it isn't a change to that page — it's that the extra pages here shift the order in which Sphinx reads documents relative to the autoapi-generated one, and the generated instance is the one without :no-index:. The build still succeeds (exit 0, one warning), so nothing is broken, but it would presumably show up on the next docs build on main once this lands. The fix belongs in the autoapi config rather than in path work, so I've left it alone — happy to take it if you'd like it fixed here.

Findings 1–5 and 7 remain done; still awaiting you on finding 6, the object-dtype ndarray digest issue, dropping child_slots, and the store-before-register gc window.


Generated by Claude Code

@claude

claude Bot commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

🤖 CI failure summary for ad7b883 on temppath

Summary
Notebook test FiveMinuteTour.ipynb fails on both Python 3.13 and 3.14 due to a missing dependency. The notebook execution fails when trying to import the ase module.

Root cause
The ase package is not included in the test environment dependencies:

ModuleNotFoundError: No module named 'ase'

This occurs when executing the FiveMinuteTour.ipynb notebook, suggesting it was recently added or its imports changed without updating the test environment setup.

Suggested fix

  1. Identify where ase is imported in the notebook and verify it's necessary
  2. Add ase to the project's test dependencies (likely setup.py, requirements.txt, or pyproject.toml)
  3. Re-run the tests to confirm the fix

Generated by Claude Haiku 4.5.

Main's #859 refreshes the `agents/DEVELOPING.md` line counts and records
#854/#855 and PR #858.  Auto-merged — it edited the decisions log while this
branch's changes are in the docs/notebooks inventory, so the two did not
overlap for once.

Verified rather than trusting the clean exit: this branch's `purity`,
`file_semantics`, `path_storage` and `recipes/files_and_paths` entries survive,
as do the twelve-notebook counts.  Only `agents/DEVELOPING.md` is in the merge,
so the suite and the Sphinx build are untouched; duplicate-definition scan
clean.

CI stays red on `FiveMinuteTour.ipynb` (missing `ase`) — unchanged, main's
defect, reported on #797 and awaiting a decision there.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RascUYS7JgCMHHpXsYbPTf
@claude

claude Bot commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

🤖 CI failure summary for 4468aaf on temppath

Summary

The test notebook FiveMinuteTour.ipynb fails during execution because the ase (Atomic Simulation Environment) module is not installed. This affects both Python 3.11 and 3.12 test runs (1 failure out of 1920+ tests per run).

Root cause

ModuleNotFoundError: No module named 'ase'

The notebook attempts to import from ase.build and ase.calculators.emt, but the package is missing from the test environment.

Suggested fix

Add ase to the project's test dependencies in setup.py, pyproject.toml, or requirements-dev.txt. This dependency is needed for the FiveMinuteTour tutorial notebook to execute during CI validation.


Generated by Claude Haiku 4.5.

Main's #857 guards the `FiveMinuteTour` ASE imports with try/except and skips
that section when `ase` is missing — the fix for the failure this branch has
been carrying.

It does not clear CI here, though: the notebook still fails, now one cell
later.  Ran it after merging rather than assuming the fix was complete:

    ModuleNotFoundError: No module named 'executorlib'

Cell 30 does a bare `from executorlib import SingleNodeExecutor`, and
`executorlib` is its own optional extra rather than part of `tests`, which is
all `tests.yml` installs.  Same shape as the `ase` problem, same remedy — the
guard #857 added just needs to cover this import too.  Left for pmrv, who is
evidently mid-fix, rather than racing them with a duplicate patch in a path PR.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RascUYS7JgCMHHpXsYbPTf

pmrv commented Aug 12, 2026

Copy link
Copy Markdown
Owner Author

Thanks for #857 — merged it here in f2ec078. It doesn't clear CI yet, though: the same notebook still fails, one cell later.

I ran it after merging rather than assuming the fix was complete:

ModuleNotFoundError: No module named 'executorlib'

Cell 30:

from executorlib import SingleNodeExecutor as Executor

Bare import, no guard — and executorlib is its own optional extra in pyproject.toml, not part of tests, which is all tests.yml installs. So it's the same shape as the ase case: the notebook reaches for an optional dep that CI never has.

The try/except ImportError you added for the ASE section applied to this import would do it:

try:
    from executorlib import SingleNodeExecutor as Executor
except ImportError:
    print("executorlib not installed, skipping this section")
else:
    ...

I haven't pushed that — it's your notebook and you're evidently mid-fix, so a duplicate patch from a path PR would just collide with whatever you land next. Say the word if you'd rather I did it here.

Worth noting the two filters that let this class of thing through, since it'll keep happening: the changes gate skips the suite for notebook-only PRs, and tests.yml doesn't run on pushes to main — so a notebook that reaches for an optional dep goes unnoticed until some unrelated PR touching src/ picks up the red. That's how #841 landed, and why #857's remaining half is only visible from here.

Everything else on this branch is unchanged and green: 1909 passed, 11 skipped with that notebook deselected.

Still open for you, unchanged: the Sphinx fleche.D duplicate-object warning (main builds clean, this tree warns reproducibly — fix belongs in the autoapi config), finding 6, the object-dtype ndarray digest issue, dropping the now-unused child_slots, and whether to close the store-before-register gc window with a readers-writer lock.


Generated by Claude Code

@claude

claude Bot commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

🤖 CI failure summary for f2ec078 on temppath

Summary

Notebook integration tests are failing because executorlib is not installed. The FiveMinuteTour.ipynb notebook attempts to import SingleNodeExecutor from executorlib but the dependency is missing from the test environment.

Root cause

ModuleNotFoundError: No module named 'executorlib'
  File FiveMinuteTour.ipynb, Cell In[16], line 1
    from executorlib import SingleNodeExecutor as Executor

Both Python 3.12 and 3.14 environments are affected.

Suggested fix

Add executorlib to the project's test dependencies (requirements.txt, setup.py, or pyproject.toml). If this is a new dependency on the temppath branch, ensure it's properly declared before merging.


Generated by Claude Haiku 4.5.

Main's #862 guards the `FiveMinuteTour` executorlib import, the half of #857
that was still failing here, and re-executes ten notebooks.

That clears the last red: this branch is green again for the first time since
#841 landed.  Verified after merging rather than assuming — the notebook that
had been failing now passes on its own, and the full suite is **1910 passed,
11 skipped, zero failures**.  `ty` 0.0.65 clean.

Sphinx now builds with **no warnings at all**: the `duplicate object
description of fleche.D` warning reported earlier is gone, without anything on
this branch changing.  It was an ordering effect between the autoapi-generated
page and `digests_as_args`, and main's docs churn has evidently shifted the
read order back.  Nothing here to fix, and nothing left to raise with pmrv on
that point.

This branch's content is intact across the merge: `docs/index.rst` keeps
`dev/path_storage`, `notebooks/Files` and `notebooks/PathsInContainers`, and
`agents/DEVELOPING.md` keeps the twelve-notebook counts.  Duplicate-definition
scan clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RascUYS7JgCMHHpXsYbPTf

pmrv commented Aug 12, 2026

Copy link
Copy Markdown
Owner Author

#862 does it — merged in d032f71, and this branch is green again, for the first time since #841 landed.

Verified after merging rather than assuming: the notebook that had been failing now passes on its own, and the full suite is 1910 passed, 11 skipped, zero failures. ty 0.0.65 clean.

The Sphinx fleche.D warning is also gone — and nothing on this branch changed to fix it. As I said when I reported it, it was an ordering effect between the autoapi-generated page and digests_as_args rather than a content problem; your docs churn since has evidently shifted the read order back. The build is now warning-free here. So that one is closed, no action needed from you.

Where #797 stands:

Finding Status
1 — gc() destroys stored Path values fixed, 28df4cd
2 — gc() evicts in-flight arguments fixed, a9e1a13 + ff168a0 + 35c3aae
3 — SSH path fallback degrades all arguments fixed, 4c10b92
4 — find_path misses re-iterable containers fixed, 4d6ded1
5 — unreadable paths raise fixed, 59f17e6
7 — test leaks the sticky cache fixed, 58a0cfe
6 — three round trips per remote miss your call, untouched

Still open for you, all small and all previously described:

  1. Finding 6 — collapse the commit's two trips into one commit RPC. You called it take-or-leave; it also closes the partial-failure window your own comment on save flags, and it interacts with the finding-3 fix (a path-carrying prepare now costs a trip per argument), so both places want batching together if you want it at all.
  2. child_slots — I added it in fix(remote): refuse Path values at the SshCache boundary #828 to back the old destructuring-shaped find_path; nothing calls or tests it now, but it is exported in fleche.storage.__all__. Drop it or keep it as a general helper?
  3. Object-dtype ndarray digests are address-baseddigest(np.array([mk()], dtype=object)) differs between two equal arrays, so such arguments never hit across processes. Pre-existing, unrelated to this PR, semver-breaking to fix. Want an issue?
  4. The gc store-before-register window — the one race the ordering fix does not close, pinned by test_gc_may_evict_a_value_stored_but_not_yet_registered. Closing it needs a readers-writer lock shared by writers and the sweep; it is the same window one-shot save has always had, so it is arguably a separate change.

Nothing here blocks a merge — 1, 3 and 4 are questions, 2 is optional cleanup. Happy to take any of them on your word.


Generated by Claude Code

claude added 3 commits August 13, 2026 19:45
Six commits: the storage-hierarchy devnote promoted to docs (#849), Runtime
cputime/systime metadata (#851), a `_lazy_default` state refactor (#858),
coverage and entry-point-log fixes (#860, #864), and a ty bump to 0.0.69
(#863).

Conflict in `agents/DEVELOPING.md` again — main rewrote the `docs/` inventory
around the two new `dev/` pages and the new figure set, while this branch's
side still listed the `devnotes/storage-hierarchy.*` files main has now
deleted.  Rebuilt on main's newer text rather than reconciling line by line,
folding this branch's four additions back in: `usage/purity`,
`usage/file_semantics`, `dev/path_storage`, `recipes/files_and_paths`, and the
twelve-notebook symlink count (main's text said nine plus `Destructuring`,
which predates `Files`/`PathsInContainers`).

`docs/index.rst`, `src/fleche/digest.py` and `tests/unit/test_remote.py`
auto-merged; verified this branch's three toctree entries survived.

Checked at the new pin rather than the old one: `ty` 0.0.69 clean, full suite
1916 passed / 11 skipped, Sphinx builds with no warnings, duplicate-definition
scan clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RascUYS7JgCMHHpXsYbPTf
Main's #865 records the #806/#807 landings, marks #808 obsolete, and notes the
#861 notebook fix.  Touches only `agents/DEVELOPING.md`, and it auto-merged —
its edits are in the decisions log rather than the docs inventory this branch
keeps colliding with.

Verified rather than trusting the clean exit: this branch's `usage/purity`,
`usage/file_semantics`, `dev/path_storage` and `recipes/files_and_paths`
entries survive, as do the twelve-notebook counts.  No source, docs, tests or
notebooks in the merge and the `ty` pin is unchanged at 0.0.69, so the suite,
the Sphinx build and the type check are untouched; duplicate-definition scan
clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RascUYS7JgCMHHpXsYbPTf
Picks up #871 (records the 2026-08-14 refactor proposals #866-#869 in
agents/DEVELOPING.md). Auto-merged cleanly; the only touched file is
agents/DEVELOPING.md, where main's new proposal entries sit alongside
this branch's docs-layout and notebook-count text.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RascUYS7JgCMHHpXsYbPTf

pmrv commented Aug 15, 2026

Copy link
Copy Markdown
Owner Author

test-minimum-deps went red on 30a9ccc — pre-existing digest collision on main, not this branch

30a9ccc is a merge of main that touches only agents/DEVELOPING.md, so I treated any red as new and mine. It isn't — but what it found is a real correctness bug in main, so I'd rather flag it than re-run past it silently.

What failed

Hypothesis found a falsifying example for tests/unit/storage/test_destructuring_storage.py::test_roundtrip:

value  = {None: [-512], '': -2.220446049250313e-16}
loaded = {None: [-2.220446049250313e-16], '': -2.220446049250313e-16}

The -512 came back as -2.220446049250313e-16 because the two share a digest:

>>> digest(-512) == digest(-2.220446049250313e-16)
True
>>> -512 == -2.220446049250313e-16
False

Why

The Number() arm of _digest_bytes ends with value = hash(value); return _digest_bytes(value) — a float's digest is the digest of the int hash(value). hash(-2**-52) == -512, so the float and the int -512 are indistinguishable to the cache.

That is the documented mechanism (docs/digests/digest_equivalence.rst: the type-name salt is deliberately not carried into number digests, which is what makes digest(1) == digest(1.0) == digest(1+0j)). What the docs assume, though, is that hash is injective. It isn't — it's a 61-bit fold — so the intended equal-values-collide becomes unequal-values-collide. Every non-integral float aliases some int:

value hash(value) equal? same digest?
0.5 1152921504606846976 no yes
0.1 230584300921369408 no yes
1e308 156575653125701 no yes
Fraction(1,3) 1537228672809129301 no yes
Decimal("0.5") 1152921504606846976 no yes

It reaches cache keys, not just digest():

>>> f.digest(0.5) == f.digest(1152921504606846976)
True

— i.e. f(0.5) can return the cached result of f(1152921504606846976).

Not from this branch

  • The collision reproduces against origin/main's src/fleche/digest.py in a clean worktree (COLLIDES ON MAIN: True).
  • This branch does not touch the Number arm; its digest.py changes are the Path / CompletedProcess arms and OPAQUE_ITERABLES.
  • tests/strategies.py — which generates the example — is unchanged here, so main's own CI can draw this same example on an unlucky seed.

Seed-dependent: 3000 derandomized examples pass on both main and this branch, which is why it has stayed hidden.

What I'm not doing

Not fixing it here. Salting number digests by type (or by exact value) changes the digest of every float, complex, Fraction and Decimal, invalidating every persisted call that has one as an argument or result — a migration decision for a persistent cache, and squarely yours, not something to smuggle into a path-caching PR. Happy to take it in a follow-up if you want it, in whichever direction you prefer (full type+value salt, or keep cross-type equivalence but digest the exact value rather than hash).

I've re-run the failed job so the PR's CI state isn't left red on a seed accident. Flagging this alongside the four questions in this comment — still all awaiting your call, none started.


Generated by Claude Code

claude added 2 commits August 16, 2026 04:45
Picks up #872 (records the PR #870 MPI collective() close-out as
out-of-scope in agents/DEVELOPING.md). Auto-merged cleanly; the only
touched file is agents/DEVELOPING.md, where main's new close-out note
sits alongside this branch's docs-layout and notebook-count text.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RascUYS7JgCMHHpXsYbPTf
pmrv added a commit that referenced this pull request Aug 21, 2026
…625 refresh (#882)

Scheduled audit of `agents/DEVELOPING.md` against the tracker state as
of 2026-08-21. `AGENTS.md` and `agents/USAGE.md` checked, no changes
needed (no source changes on `main` since the last pass).

- Perf-audit pointer: #625 was refreshed 2026-08-20 (doc said
2026-08-13).
- #861 (FiveMinuteTour `executorlib` guard) closed 2026-08-21 — removed
from the open-bugs list; the decisions bullet notes the closure date.
- #709 now has PR #881 in flight (serializer registry replacing the
three `with_*` classmethods) — recorded on the 2026-07-03 proposal
bullet.
- The Path/file-handling theme now names PR #797 (the `temppath` branch,
open since 2026-07-27) as the in-flight implementation, with a one-line
sketch of its content-addressed blob storage and `TempPath`
rematerialization; the #829 bullet cross-references it.

Docs-audit PRs #873#878 are all still open — no state change to record
there.

---
_Generated by [Claude
Code](https://claude.ai/code/session_017eFADsswQu1uvyacei8Cj8)_

Co-authored-by: Claude <noreply@anthropic.com>
pmrv added a commit that referenced this pull request Aug 22, 2026
…for #840 (#888)

Scheduled audit of `AGENTS.md` and its linked guides against the tracker
state as of 2026-08-22. No code landed on `main` since the previous
audit (#882), so the Quick Reference and Architecture claims are
unchanged; the delta is issue-tracker state only.

- Add the 2026-08-21 refactor cohort under open proposals: #883
(`_raw_sub_digests` hardcodes built-in `Digested` subtypes, breaking
`gc()`/`count_reuses()` for `register_destructurer` extensions — the one
correctness gap of the batch), #884 (sqlite URL handling hand-rolled
twice and drifted on `~`-expansion), #885 (`XDG_CACHE_HOME` companion to
#868), #886 (`benchmark_storage.py` five-op timing harness duplicated).
- Record PR #887 in flight against bug #840 (`BoundWrapper.__reduce__`
pre-serialises `func`: `pickle` by reference, `cloudpickle` by value).
- Split the single Bugs/Feature-request paragraph into one-item-per-line
sub-bullets so independent updates stop colliding in merges.

Verified still open and accurately described: PRs #797, #837, #873-#878,
#879, #881.

---
_Generated by [Claude
Code](https://claude.ai/code/session_01J6LqSMaWKWzDfF7hF51Syq)_

Co-authored-by: Claude <noreply@anthropic.com>
pmrv added a commit that referenced this pull request Aug 23, 2026
…theme (#889)

Scheduled AGENTS.md audit, 2026-08-23.

No code has landed on `main` since the 2026-08-21 audit (63a0584), and
the issue/PR state recorded in `agents/DEVELOPING.md` is still current:
in-flight PRs #873#878, #881, #887, and #797 all remain open; no new
issues since #883#886. Spot-checks of the Quick Reference and
Architecture claims against `src/` all pass (public `__all__`,
`_lazy_default`/`_sticky_set`/`_hard_set`, `PreparedCall`/two-phase
save, `register_storage`, `_CACHE_TEMPLATES`,
`Runtime.cputime`/`systime`, module line counts quoted in #789/#832).
`AGENTS.md` and `agents/USAGE.md` need no changes.

One gap found: the performance theme names "pooled file handles in
`bagofholding_file.py`" as a fix candidate without noting that draft PR
#786 (open since 2026-07-23) already implements it, and PR #804 (open
since 2026-07-31) — the benchmark-harness fix that removes the
always-evicting `SizeLimitedCache(max_size=10)` config — was recorded
nowhere. Both are now listed under the perf theme as
check-before-duplicating entries, added as separate paragraphs to keep
future edits conflict-free.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01H9uSJpX8SLJrtYbc2ZaSv5

---
_Generated by [Claude
Code](https://claude.ai/code/session_01H9uSJpX8SLJrtYbc2ZaSv5)_

Co-authored-by: claude[bot] <claude[bot]@users.noreply.github.com>
@pmrv

pmrv commented Aug 25, 2026

Copy link
Copy Markdown
Owner Author

Believe it or not the solution to the gc() vs Path problem is more abstract indirection! Behind both path and destructuring mixin we add another interface that recurses into selected items and route all item access through that. path and destructuring then implement that for their specific types.

pmrv added a commit that referenced this pull request Aug 28, 2026
…qlFile fsync cheap fix (#898)

Weekly AGENTS.md/DEVELOPING.md audit. Only one change since the
2026-08-27 pass (#897): issue #625's perf audit was refreshed the same
morning.

- Update the perf-audit pointer in `agents/DEVELOPING.md` from
"refreshed 2026-08-20" to 2026-08-27 and record the run's verdict (no
new source-caused regressions; flagged rows are the chronic
`BagOfHoldingH5File` per-op open cost or noise).
- Record the still-open SQL-side hot spot the refreshed audit re-flags:
`Sql` fsyncs once per key on `save`/`evict` because
`_configure_sqlite_pragmas` (`src/fleche/storage/sql.py:190-227`) sets
`journal_mode=WAL` but no `synchronous` pragma; the cheap fix (`PRAGMA
synchronous=NORMAL`) has been flagged in every audit since 2026-05-07.
Verified against the source — the function sets only `foreign_keys` and
`journal_mode`.

Everything else checked and current: no merges to `main` since #897;
open PRs (#873, #874, #887, #892, #894, #896, #797, #786, #804) and
issues (#893, #895, #625) are all already recorded.
`eisenforschung/landau` was audited in the same pass and needs no update
(nothing landed since its 2026-08-25 pass; in-flight PRs
#391/#394/#395/#414/#422 unchanged; spot-checked claims hold).

---
_Generated by [Claude
Code](https://claude.ai/code/session_012SCLy9y7aUR7F44Q7UviGo)_

Co-authored-by: Claude <noreply@anthropic.com>
pmrv added a commit that referenced this pull request Aug 29, 2026
…racker items (#904)

Scheduled AGENTS.md audit. Changes since the 2026-08-27 pass:

- Release-state claims updated: PR #793 (two-phase save) and PR #843
(lock-free pickle-family backends) shipped in 0.22.0 on 2026-08-28
(release PR #837); the three "unreleased as of 2026-08-27" qualifiers
are gone. Noted that `fix(query)` #894 is an ancestor of the 0.22.0 tag
(verified via `git tag --contains c82c691`) even though the generated
changelog entry omits it.
- PR #894 (`latest()`/`oldest()` raise `ValueError` when no matching
call carries `Runtime` metadata, `IndexError` on empty), PR #892 (gc
mid-sweep guards + 441-case digest product collapse, suite 1738 → 1299),
and docs-audit PRs #873/#874 moved from the in-flight section to
decisions landed; the in-flight section is now a one-liner pointing at
the five PRs still open (#887, #804, #797, #786, #523), each covered
under its theme.
- New tracker items recorded: the 2026-08-28 refactor cohort #899#902,
bug #903 (`put()` `filelock.Timeout` fatality on ≤0.21.2,
dedup-concentrated lock contention; resolved for pickle-family by
0.22.0's atomic rename, bagofholding multi-bag still exposed per #893),
and the #895 resolution (documented limitation per the 2026-08-28
maintainer comment). #900 cross-referenced at the `pypi-publish.yml`
line in the workflows list.
- The #893 bullet's `lock_timeout` mitigation sentence rephrased to the
current state: the `FutureWarning` drop covers only pickle-family
configs, a bagofholding `values.lock_timeout` still works on 0.22.0
(checked against `config.py:466`).
- USAGE.md: query section states the `latest()`/`oldest()`
`ValueError`/`IndexError` contract (verified against
`query.py:149-188`).

landau's AGENTS.md was audited in the same pass and left unchanged — no
code or tracker movement since its 2026-08-24 update (only docs commits
on main; issues #423#425 and PRs #422/#414/#395/#394/#391 unchanged).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_014myhh6xKWC2VhQeB92mGSn

---
_Generated by [Claude
Code](https://claude.ai/code/session_014myhh6xKWC2VhQeB92mGSn)_

Co-authored-by: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants