Conversation
The wrapper digested arguments twice from live state: once before the function body (the lookup key) and once after (when Cache.save stashed the call record). A function that mutated an argument — e.g. appended to a received list, or wrote into a received directory — was therefore recorded under the post-mutation content: honest repeat calls could never hit it, while a call passing the mutated state would false-hit a result computed from different input. Fleche commits pure functions: the recorded identity of a call is now sealed from the arguments as passed, via a two-phase save protocol: - cache.prepare(call) -> PreparedCall stores argument values and seals the lookup key before the body runs; PreparedCall.commit(result, metadata) stores the result as returned (a mutated argument passed back out is captured in its final state) and files the record; PreparedCall.abandon() releases without recording (body raised or result uncacheable). PreparedCall is also a context manager that abandons when the block exits uncommitted. - prepare is implemented once on BaseCache over a new save_value primitive (the write-side counterpart of load_value), which wrappers forward, CacheStack routes to stack[0], and SshCache sends over the wire; read-only caches admit digest-only (nothing written, and the commit after the body is rejected — matching the previous post-body rejection behavior). - Cache.save takes the fully digested record; a live Call is still accepted as the degenerate one-shot form (values static, nothing to drift), so post-hoc saving — transfers, redigest, external callers — keeps working unchanged. BREAKING CHANGE: calls of argument-mutating functions are now recorded under the pre-call argument state; caches populated by such functions will re-execute once and re-record under the correct key. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Rename DigestedDict to DigestedMapping and make it generic over any Mapping rather than only dict, preserving the concrete mapping type (e.g. OrderedDict) and subclass identity through a save/load round-trip. A backward-compatible DigestedDict alias is kept (re-exported from fleche.caches and fleche.storage.destructuring) so existing imports keep working. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Store and retrieve pathlib.Path values by their content rather than as opaque path strings, so caching a function that takes or returns a Path is portable and reproducible across machines. Following git's split — content is content-addressed, names live in trees: - a file is keyed on (name, content): it keeps its name and extension on a cache hit, and its content (plain bytes) deduplicates — shared across names and with bytes values. To key on content alone, return bytes. - a directory is keyed on its tree alone; its incidental root name is dropped (a reloaded directory is named by its digest, its children by their real names). PathValueMixin owns the traversal and is wired into the default value storages (memory, pickle, H5) between DestructuringMixin and ValueMixin: a file becomes a FileBlob(name, content) record, a directory a DirectoryBlob tree, and content lives in plain bytes blobs. The digest Path arm mirrors storage so digest(path) == values.save(path) — the invariant cached lookups of path arguments and results depend on. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…otebook Add a light recipes page and a path-storage dev-guide deep dive — content addressing, the digest(path) == values.save(path) invariant, deduplication, and the name-aware-files / content-only-directories model — wire both into the toctree, and add a runnable Files.ipynb walkthrough registered in the notebook integration test. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Companion to Files.ipynb: shows the intended UX of Path values nested in dicts/lists/dataclasses, then the edge cases — hits changing location and type, aliasing loss, Path dict-keys mending into new keys, per-hit materialization and temp-file lifetime, opaque containers storing paths by location, and the exact-type destructuring allowlist. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
New docs/usage/file_semantics.rst spells out the full user-facing contract for Path values: call identity ((basename, content) for files, tree-only for directories, content-keyed bytes), materialization on hits (fresh temp copies, faithful basenames, deterministic digest root names, order preservation), lifetime of materialized paths, nesting in destructured containers (dict keys, aliasing, exact-type allowlist), opaque-container location semantics, the do-not-mutate-arguments rule, and fidelity limits (permissions, symlinks, metadata). The page was validated black-box: agents restricted to reading only this page wrote assertion scripts against it in three rounds; every failed or ambiguous expectation was folded back into the text. The mutation rule and fidelity limits were discovered this way. Also cross-links the page from the TL;DR, the files recipe, and the path storage internals page. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Integration coverage for the two-phase save protocol meeting path storage: a function writing into a received directory hits on honest repeats, the mutated tree is a distinct call, and a mutated argument passed back out is captured in its final state. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
🤖 CI failure summary for Summary Root cause Suggested fix
Generated by Claude Haiku 4.5. |
Benchmark ResultsBaseline: efeeebc (recomputed) · HEAD~: 2da4b10 DeltaSignificant changes (|Δ| > 10%): 7 — 233 other rows hiddenIntegration
Value Storage
Full resultsCall Storagecalls
Digest
Integrationcompute_heavy
data_heavy
lightweight
Value Storagenested_structures
numpy_arrays
small_strings
|
Address review on #793: - `BaseCache.save_value` is gone (and with it the new abstract method on `BaseCache`, which broke the API for existing subclasses). `Cache.prepare` now hands its whole value storage to `Call.stash`, which already leaves an unknown result as `None`. - `PreparedCall.commit` no longer stores the result itself: it attaches the live value to the record and lets `Cache.save` store it at the end, next to the one-shot `Call` path. - `PreparedCall` drops the redundant `call` and `key` fields; callers that need the key have the `Call` or the `DigestedCall` at hand. - `BaseCache.prepare` keeps a concrete digest-only default (key sealed, nothing written), so third-party caches keep working and `ReadOnlyMixin` only restates it to win the MRO over `CacheWrapper`. Wrappers and stacks delegate storage inward and rebind the cache so the commit runs through their own `save` policy. - Over SSH, `save_value` (one round trip per value) is replaced by a single `prepare` RPC that stashes the arguments remotely and returns the sealed record; read-only remotes short-circuit to the local digest-only admission. - `CacheStack.save` now returns the record key, which `commit` hands back. Co-authored-by: Marvin Poul <2719909+pmrv@users.noreply.github.com>
| fleche caches *pure* functions. What a function does to its arguments | ||
| without passing it back out is invisible to the cache — treat received paths | ||
| as read-only and write outputs to a fresh directory (``tempfile.mkdtemp``). | ||
| A mutated argument that *is* returned is captured faithfully in its final, |
There was a problem hiding this comment.
feels almost more like it should go into a more generic section rather than file specific.
| Paths nested inside containers | ||
| ------------------------------ | ||
|
|
||
| Paths are found and content-stored inside the containers fleche takes apart: |
There was a problem hiding this comment.
cross ref destructuring here, since that's the key
There was a problem hiding this comment.
move the completedprocess digester into fleche itself
|
Interactions with ssh cache unclear |
…he record commit now builds the filed record with dataclasses.replace, so the live result is never parked on the prepared record after save, and raises RuntimeError on a second commit or a commit after abandon (abandon stays idempotent). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QT4DxU9dVALrMvff7setnk
Mirror the existing Rejected handling on save: a cache that refuses admission logs a warning and the call executes uncached instead of failing before the body runs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QT4DxU9dVALrMvff7setnk
- SshCache.prepare: live wire round trip (arguments stashed server-side before the body, mutation-coherent commit) and the read-only local digest-only admission with zero RPCs. - PreparedCall: double commit and commit-after-abandon raise; commit leaves the prepared record unmutated. - The end-to-end tests now run on a private Cache instance: populating the shared named-memory singleton (load_cache_config caches it in _live_caches) leaked into any later test that round-trips that cache's config, e.g. test_run_server_serves_active_cache_over_stdio_until_eof. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QT4DxU9dVALrMvff7setnk
… stores it Combines the #827 sketch with the replace-based commit: DigestedCall goes back to holding digests only, PreparedCall carries the live result and metadata privately, and Cache.save stores the result when handed the PreparedCall itself. PreparedCall pickles without its process-local cache binding so a commit against SshCache ships only the sealed record and the pending result. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QT4DxU9dVALrMvff7setnk
Policy rejections keep their warning; actual errors (storage faults, lost connections) are logged with traceback and the body still runs. Also drop the dead call.metadata assignment left over from the commit protocol switch. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QT4DxU9dVALrMvff7setnk
SshCache.save now recreates Cache.save's ending for a PreparedCall: a write-gated save_value RPC stores the result value remotely, then the plain digested record is filed — only Call and DigestedCall ever cross the wire, so PreparedCall.__getstate__ goes away. The resolve step shared by both cache types moves onto PreparedCall. Server-side, save_value routes through prepare with a synthetic never-committed call, so wrappers and stacks direct the value to the same storage their saves use without any new public cache method. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QT4DxU9dVALrMvff7setnk
…edProcess Addresses the four review threads on #797. - The "argument mutation" rules were stated as a file-specific caveat, but they are general: a mutated list argument behaves identically (verified — cold call mutates, hit leaves the argument untouched, and a mutated argument that *is* returned is stored post-mutation). Move the contract to a new `usage/purity` page covering keyed-as-passed, mutation not replayed, and side effects being cold-call-only; `file_semantics` keeps the path instance and links out. - Cross-reference destructuring from "Paths nested inside containers", since that is the mechanism the section is really about: only destructured children reach the path machinery, so `_DESTRUCTURERS` *is* the list of places a nested path gets content treatment. Explains the container list, the exact-types caveat, and the opaque-container section in one stroke. - Move the `CompletedProcess` digester out of `notebooks/Files.ipynb` and into `digest.py` as a match arm (args + returncode + stdout + stderr). `run()`'s result is neither a dataclass nor iterable, so it was `Indigestible` and every shell-wrapping function needed the same hand-rolled hook. Purely additive — those values raised before, so no stored digest changes and no `hash_version` bump. - Save the executed `Files.ipynb` (the notebook now demonstrates the built-in digest, no `add_hook` cell). Same for `PathsInContainers.ipynb`, which was added without a `docs/notebooks/` symlink, a toctree entry, or a slot in `test_notebooks.py` — all three now wired up. Also records issue #829 (making paths genuinely work over SSH by doing the blob reduction client-side) in `remote.py`, `dev/path_storage`, and the DEVELOPING design-theme cheat sheet, and fixes a short title underline in `usage/helpers`. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RascUYS7JgCMHHpXsYbPTf
…tocol Implements the changes approved in the #793 review threads, revised per the #825 review ("combine" with the #827 sketch; broadened prepare guard; two-trip remote commit). Stacked onto `prepared-call` for merging into that branch. ## Changes - **Pending result lives on `PreparedCall`, never on `DigestedCall`** (combining #827 with the replace-based commit): `PreparedCall` carries `_result`/`_metadata` as private fields, `commit` hands the whole `PreparedCall` to `cache.save`, and the cache stores the result and files a record built via `PreparedCall.resolve` — `DigestedCall.result` is back to a checkable `Digest | None`, and `commit` never mutates `digested`. A bare `DigestedCall` still files as-is. - **`PreparedCall` never goes over the wire**: `SshCache.save` recreates `Cache.save`'s ending in two trips — a write-gated `save_value` RPC stores the result value remotely, then the plain `DigestedCall` is filed. Only `Call` and `DigestedCall` cross the wire (asserted by a spy in the round-trip test). Server-side, `save_value` routes through `prepare` with a synthetic never-committed call, so wrappers and stacks direct the value to the same storage their saves use without adding a public cache method; a failure between the trips leaves a content-addressed orphan for gc, like any abandoned call. - **`commit` is single-shot**: a second `commit` or a `commit` after `abandon` raises `RuntimeError`; `abandon` stays idempotent so cleanup paths can't trip it. - **Wrapper degrades gracefully on any `prepare` failure**: `Rejected` logs a policy warning; any other `Exception` (storage fault, lost connection) logs with traceback — in both cases the body runs and returns uncached instead of failing before it ever ran. Also drops the dead `call.metadata` assignment. - **Remote `prepare` tests**: live wire round trip (arguments stashed server-side before the body runs, sealed key survives argument mutation, two-trip commit) and the read-only branch (local digest-only admission, zero RPCs, commit rejected locally before the first trip). - **Test hygiene fix (drive-by)**: the e2e tests in `test_prepared_call.py` activated the shared named-`"memory"` cache stickily; that singleton lives in `config._live_caches`, so populating it leaked into any later test that round-trips the named cache's config (`test_run_server_serves_active_cache_over_stdio_until_eof[memory]` crashes in `dataclasses.asdict` when the files run back to back — masked in the full suite by `tests/unit/config`'s `_live_caches.clear()` fixture). The tests now scope a private `Cache` instance. Parked separately: the gc/in-flight race is #826. ## Testing Full suite: 1659 passed, 11 skipped. `ty check src/` clean. The affected file pair also passes in the previously-failing collection order. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01QT4DxU9dVALrMvff7setnk
Applies the remaining comment trims flagged in the #793 review (the rest already landed with #825). Narration and before/after history go; invariants stay: - `BaseCache.prepare` docstring: 20 → 15 lines; keeps the override contract, the digest-only base semantics, and the `digest(x) == values.save(x)` key-equality invariant. - `Cache.prepare`: drops the result-lifecycle narration (documented on `PreparedCall`/`save`); keeps the pre-body stash rationale and the no-cache-level-lock explanation. - `ReadOnlyMixin.prepare`: drops "the behavior save() rejection produced before the two-phase protocol, minus the wasted writes"; keeps the MRO reason and the seal-without-writing behavior. - `SshCache.prepare`: drops the ", as before" tail; keeps the Path-over-SSH caveat. Comment-only diff. Affected test dirs pass (191 tests), `ty check src/` clean. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01QT4DxU9dVALrMvff7setnk --- _Generated by [Claude Code](https://claude.ai/code/session_01QT4DxU9dVALrMvff7setnk)_ Co-authored-by: Claude <noreply@anthropic.com>
…edProcess (#831) Answers the four inline review threads on #797. - Move the general "fleche caches pure functions" contract — arguments keyed as passed, mutation and other side effects not replayed — out of file_semantics into a new usage/purity page. Verified it is not path-specific: a mutated list argument behaves identically. - Reframe "Paths nested inside containers" around destructuring, since only destructured children reach the path machinery, so _DESTRUCTURERS is the list of places a nested path gets content treatment. Also document why "any depth" holds at any remaining_depth: a Path matches no destructurer, so it is always written out as its own entry rather than inlined. - Move the CompletedProcess digester out of notebooks/Files.ipynb into digest.py as a match arm (args + returncode + stdout + stderr). Purely additive: those values raised Indigestible before, so no stored digest changes and no hash_version bump. - Commit Files.ipynb and PathsInContainers.ipynb executed, and wire PathsInContainers into docs/notebooks/, the toctree, and test_notebooks.py. Replace the generic query cell in Files.ipynb with the value store's own view of a stored path. Reframe the PathsInContainers "Edge N" sections as Beware (consequences of caching by value) vs Caveat (real limits of the mending machinery).
`ty check src/` passes on `main` but reports three diagnostics here, so the ty workflow fails for every PR into this branch (and now for the branch itself). Both are false positives on code that is correct at runtime: - `DigestedMapping.mend` / `_rebuild_plain` call `type(value)(...)`, which ty narrows to `type[Mapping]`. `Mapping` is an ABC with no `__init__`, so the call resolves to `object.__init__` and the argument reads as one positional too many. The runtime type is always a concrete `dict`/`OrderedDict`. - `TempPath(type(Path()))` uses a dynamic base, through which ty cannot resolve an MRO. The indirection is required while Python 3.11 is supported — bare `Path` only became subclassable in 3.12. Suppressed rather than reworked: a `cast` on the two constructors would trade a checker complaint for an indirection, and the `TempPath` base cannot change until the 3.11 floor moves. Comments are anchored on the exact lines ty reports, which is where it binds them. `ty check src/` (pinned 0.0.62, as CI runs it): All checks passed! Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RascUYS7JgCMHHpXsYbPTf
|
Pushed Three suppression comments, no behaviour change: - (self.get(storage, k), self.get(storage, v))
+ (self.get(storage, k), self.get(storage, v)) # ty: ignore[too-many-positional-arguments]
- return type(value)(zip(children[:n], children[n:]))
+ return type(value)(zip(children[:n], children[n:])) # ty: ignore[too-many-positional-arguments]
-class TempPath(type(Path())):
+class TempPath(type(Path())): # ty: ignore[unsupported-base]Both are false positives on code that's correct at runtime. In I suppressed rather than reworked: a Verified with the pinned #828 is rebased on top of this and green locally (1788 passed / 11 skipped). Generated by Claude Code |
Answers your comment on #797, ["Interactions with ssh cache unclear"](#797 (comment)). Targets `temppath`. The four **inline** review threads on #797 moved to #831, which has since **merged** — `temppath` was merged back in here (`f938975`, no conflicts), so this PR is the `SshCache` × `Path` work only. ## What the interaction actually was Probed with a real `python -m fleche remote --serve` subprocess, client and server in different working directories so the same relative name denotes different files on each side. Three behaviours, all silent: | | before | |---|---| | server *can* see something at that name | stores **its** bytes, returns **its** digest — `digest(x) == save_value(x)` broken, so the record is filed under a key no client recomputes, and a load returns the wrong content | | server can see nothing there | `Indigestible` surfaces from inside an RPC | | loading a path stored remotely | server materializes into a temp dir on **its** disk and sends back only the name — dangling here, and unlinked there as soon as its own reference dies | The last one is not fixable by holding a reference: `PurePath.__reduce__` rebuilds from `parts` alone, so `TempPath`'s `_temp_root` is dropped in transit and `_live_roots` is per-process. The `save_value` comment already said "paths over SSH remain unsupported" — nothing enforced it. ## What this does Refuses, in both directions, via a new `RemotePathUnsupported`. It subclasses `SaveError`, so the two-phase-save degradations already in the codebase carry it and no caller special-cases anything: - path **argument** → digest-only reference, **digest computed locally**. The seal stays intact, lookups hit and miss correctly; only the bytes aren't retrievable from the remote record. - path **result** → `Rejected`. The call runs, returns the file, is logged as not cached. - **load** → raises, lazily. A record whose result is a path still loads and queries; only touching the path value raises. Detection is `storage.paths.find_path`, walking a value through a new `storage.destructuring.child_slots` — the read-only half of `_intern_rec`'s dispatch — so nesting is covered exactly as far as a destructuring save reaches, and no further (a path inside an opaque object is still stored by location, same as locally). Escape hatches, both documented: return `bytes`, or put a local layer in front of the remote one so saves never reach the wire. ## Follow-up (second commit) Per your review, the problem itself is tracked as **#829** rather than left as a comment. The shape is already implied by `PathValueMixin`: it reduces a path to `bytes` plus a `FileBlob`/`DirectoryBlob`, all of which ship fine, and those blobs' `__digest__` is *defined* to match the `Path` digest arm — so running the reduction **client-side** keeps the seal intact by construction, and needs no server change for saves (a `FileBlob` isn't a `Path`, so it falls straight through the server's path layer). The one genuinely new verb is an unmended `load_value` that returns the blob instead of materializing it on the server. Recorded in `RemotePathUnsupported`, `dev/path_storage`, and the DEVELOPING design-theme cheat sheet. ## Docs - `usage/file_semantics.rst`: new "Paths stop at a remote (SSH) cache" contract section + a quick-reference row. - `dev/path_storage.rst`: the mechanism, and the #829 sketch. - `recipes/files_and_paths.rst`: a recipe. `SshCache`'s own docstring too. ## Tests `tests/unit/test_remote.py` — bare and nested (list/tuple/nested-dict/dataclass) rejection with nothing reaching the server; the `bytes` hatch still working; `prepare()` keeping `prepared.key == call.to_lookup_key()` for a path argument (the regression the guard buys); live-`Call` rejection; load-side rejection incl. nested; a `LazyCall` whose arguments still read fine and only `.result` raises. `tests/unit/storage/test_paths.py` — `find_path` descent, opaque-leaf boundary, cycle termination. `tests/integration/test_remote.py` — the divergent-cwd reproduction above, and a path-returning `@fleche` function running uncached against a remote instead of breaking. Re-verified after merging `temppath`: **1788 passed / 11 skipped**, docs build succeeds. `ty check src/` reports **3** diagnostics — all already on `temppath`, none from this PR; see [the #831 thread](#831 (comment)) for the diagnosis and a verified fix, still pending your call on where it should land. --------- Co-authored-by: Claude <noreply@anthropic.com>
Conflict in `SshCache`, in `remote.py` — and not only textual. Both sides changed `save`, and `prepared-call` adds `prepare`, an RPC that carries argument *values* rather than digests. That is a fourth way a `Path` can reach the server, which the guard added in #828 did not cover: shipping the live call hands the far side a path *string* to resolve against its own filesystem, breaking `digest(x) == save_value(x)` exactly as before. Resolved by keeping both sides and closing that route: - `save` handles `PreparedCall` (ship result, then file the record) as well as the live-`Call` and `DigestedCall` forms, routing the result through `save_value` rather than `_rpc` so the path guard applies to it. - `prepare` falls back to the local `BaseCache` two-phase prepare when any argument carries a path. Each argument then goes through `save_value`, where a path degrades to a digest-only reference computed *locally* — the seal stays intact and lookups still hit, which is the same degradation #828 established for the other routes. Also widens `find_path` itself, which was too narrow to be safe. It mirrored *destructuring*, but the invariant at stake is `digest(x) == save_value(x)`, and `digest` recurses further than destructuring does: into namedtuples, sets, and frozensets, reading the file in each. A path hidden in one of those therefore still decides the key while passing the guard, so `Bundle(path, 0.5)` crossed the wire and the server digested that name against its own disk. The walk now follows `digest` instead, stopping where `digest` stops (an arbitrary object is `Indigestible`, so no path inside it can decide a key) and declining to consume one-shot iterables. 1807 passed, 11 skipped; `ty check src/` clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RascUYS7JgCMHHpXsYbPTf
|
Conflict resolved in 1.
|
| value | destructuring | digest reads the file |
guard saw it (before) |
|---|---|---|---|
[path] |
descends | yes | yes |
Bundle(path, 0.5) (namedtuple) |
opaque | yes | no |
{path} (set) |
opaque | yes | no |
frozenset({path}) |
opaque | yes | no |
A path hidden in any of those still decides the key, so it crossed the wire and the server digested that name against its own disk. Locally this is harmless — same process, same filesystem — which is exactly why it wasn't visible before.
find_path now follows digest, stopping where digest stops (an arbitrary object is Indigestible, so no path inside it can decide a key) and declining to consume one-shot iterables rather than exhausting them as a side effect.
Verification
1807 passed / 11 skipped, ty check src/ clean. New tests pin: prepare not shipping a path-carrying call while still using the remote for ordinary ones, commit rejecting a path result, and namedtuple/set paths being refused at the boundary.
Two API changes on your branch that my #828 tests had to follow: PreparedCall.key → to_lookup_key(), and Cache.save_value is gone in favour of the two-phase protocol.
Generated by Claude Code
GitHub flagged this PR un-mergeable, so this merge is to clear that rather than to stay current: main's #856 rewrote the `agents/DEVELOPING.md` notebook enumeration this branch also edits. Resolved to the union — twelve notebooks and twelve symlinks, main's `FiveMinuteTour` alongside this branch's `Files` and `PathsInContainers`, and this branch's `purity`, `file_semantics`, `path_storage` and `recipes/files_and_paths` pages. `docs/index.rst` auto-merged; verified its entries survived. Two things come in with it that are main's, not this branch's, and I have not silently fixed either: **CI will be red on `FiveMinuteTour.ipynb`.** It imports `ase`, which is a dependency nowhere in `pyproject.toml` and is not installed by `tests.yml`. `test_notebooks.py` globs every notebook with no skip, the `changes` gate skips the suite for notebook-only PRs, and pushes to `main` don't run tests at all — so it has never executed in CI, and the first PR touching `src/` after it landed is the one that goes red. Held this merge back for a day hoping for a fix; taking the red now because a conflicted PR is worse than a red one. Everything else passes: 1909 passed, 11 skipped, that one deselected. **Sphinx now emits one warning**: `duplicate object description of fleche.D, other instance in digests/digests_as_args`. Checked rather than assumed — `origin/main` builds clean in a worktree, this tree warns reproducibly across two builds, and the file itself is byte-identical to main's, so it is the larger page set here shifting autoapi's read order relative to the page that carries `:no-index:`. The build still succeeds; left for pmrv since the fix belongs in the autoapi config, not in path work. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RascUYS7JgCMHHpXsYbPTf
|
Merged I held the merge back for a day to keep this PR green. What forced it: #856 rewrote the same Everything except that one notebook passes: 1909 passed, 11 skipped, The conflict itself resolved to the union: twelve notebooks, twelve symlinks, main's One new thing worth your eyes, unrelated to the above. Sphinx now emits a warning that it didn't before: I checked this rather than guessing at it:
So it isn't a change to that page — it's that the extra pages here shift the order in which Sphinx reads documents relative to the autoapi-generated one, and the generated instance is the one without Findings 1–5 and 7 remain done; still awaiting you on finding 6, the object-dtype ndarray digest issue, dropping Generated by Claude Code |
|
🤖 CI failure summary for Summary Root cause This occurs when executing the Suggested fix
Generated by Claude Haiku 4.5. |
Main's #859 refreshes the `agents/DEVELOPING.md` line counts and records #854/#855 and PR #858. Auto-merged — it edited the decisions log while this branch's changes are in the docs/notebooks inventory, so the two did not overlap for once. Verified rather than trusting the clean exit: this branch's `purity`, `file_semantics`, `path_storage` and `recipes/files_and_paths` entries survive, as do the twelve-notebook counts. Only `agents/DEVELOPING.md` is in the merge, so the suite and the Sphinx build are untouched; duplicate-definition scan clean. CI stays red on `FiveMinuteTour.ipynb` (missing `ase`) — unchanged, main's defect, reported on #797 and awaiting a decision there. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RascUYS7JgCMHHpXsYbPTf
|
🤖 CI failure summary for SummaryThe test notebook Root causeThe notebook attempts to import from Suggested fixAdd Generated by Claude Haiku 4.5. |
Main's #857 guards the `FiveMinuteTour` ASE imports with try/except and skips that section when `ase` is missing — the fix for the failure this branch has been carrying. It does not clear CI here, though: the notebook still fails, now one cell later. Ran it after merging rather than assuming the fix was complete: ModuleNotFoundError: No module named 'executorlib' Cell 30 does a bare `from executorlib import SingleNodeExecutor`, and `executorlib` is its own optional extra rather than part of `tests`, which is all `tests.yml` installs. Same shape as the `ase` problem, same remedy — the guard #857 added just needs to cover this import too. Left for pmrv, who is evidently mid-fix, rather than racing them with a duplicate patch in a path PR. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RascUYS7JgCMHHpXsYbPTf
|
Thanks for #857 — merged it here in I ran it after merging rather than assuming the fix was complete: Cell 30: from executorlib import SingleNodeExecutor as ExecutorBare import, no guard — and The try:
from executorlib import SingleNodeExecutor as Executor
except ImportError:
print("executorlib not installed, skipping this section")
else:
...I haven't pushed that — it's your notebook and you're evidently mid-fix, so a duplicate patch from a path PR would just collide with whatever you land next. Say the word if you'd rather I did it here. Worth noting the two filters that let this class of thing through, since it'll keep happening: the Everything else on this branch is unchanged and green: 1909 passed, 11 skipped with that notebook deselected. Still open for you, unchanged: the Sphinx Generated by Claude Code |
|
🤖 CI failure summary for SummaryNotebook integration tests are failing because Root causeBoth Python 3.12 and 3.14 environments are affected. Suggested fixAdd Generated by Claude Haiku 4.5. |
Main's #862 guards the `FiveMinuteTour` executorlib import, the half of #857 that was still failing here, and re-executes ten notebooks. That clears the last red: this branch is green again for the first time since #841 landed. Verified after merging rather than assuming — the notebook that had been failing now passes on its own, and the full suite is **1910 passed, 11 skipped, zero failures**. `ty` 0.0.65 clean. Sphinx now builds with **no warnings at all**: the `duplicate object description of fleche.D` warning reported earlier is gone, without anything on this branch changing. It was an ordering effect between the autoapi-generated page and `digests_as_args`, and main's docs churn has evidently shifted the read order back. Nothing here to fix, and nothing left to raise with pmrv on that point. This branch's content is intact across the merge: `docs/index.rst` keeps `dev/path_storage`, `notebooks/Files` and `notebooks/PathsInContainers`, and `agents/DEVELOPING.md` keeps the twelve-notebook counts. Duplicate-definition scan clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RascUYS7JgCMHHpXsYbPTf
|
#862 does it — merged in Verified after merging rather than assuming: the notebook that had been failing now passes on its own, and the full suite is 1910 passed, 11 skipped, zero failures. The Sphinx Where #797 stands:
Still open for you, all small and all previously described:
Nothing here blocks a merge — 1, 3 and 4 are questions, 2 is optional cleanup. Happy to take any of them on your word. Generated by Claude Code |
Six commits: the storage-hierarchy devnote promoted to docs (#849), Runtime cputime/systime metadata (#851), a `_lazy_default` state refactor (#858), coverage and entry-point-log fixes (#860, #864), and a ty bump to 0.0.69 (#863). Conflict in `agents/DEVELOPING.md` again — main rewrote the `docs/` inventory around the two new `dev/` pages and the new figure set, while this branch's side still listed the `devnotes/storage-hierarchy.*` files main has now deleted. Rebuilt on main's newer text rather than reconciling line by line, folding this branch's four additions back in: `usage/purity`, `usage/file_semantics`, `dev/path_storage`, `recipes/files_and_paths`, and the twelve-notebook symlink count (main's text said nine plus `Destructuring`, which predates `Files`/`PathsInContainers`). `docs/index.rst`, `src/fleche/digest.py` and `tests/unit/test_remote.py` auto-merged; verified this branch's three toctree entries survived. Checked at the new pin rather than the old one: `ty` 0.0.69 clean, full suite 1916 passed / 11 skipped, Sphinx builds with no warnings, duplicate-definition scan clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RascUYS7JgCMHHpXsYbPTf
Main's #865 records the #806/#807 landings, marks #808 obsolete, and notes the #861 notebook fix. Touches only `agents/DEVELOPING.md`, and it auto-merged — its edits are in the decisions log rather than the docs inventory this branch keeps colliding with. Verified rather than trusting the clean exit: this branch's `usage/purity`, `usage/file_semantics`, `dev/path_storage` and `recipes/files_and_paths` entries survive, as do the twelve-notebook counts. No source, docs, tests or notebooks in the merge and the `ty` pin is unchanged at 0.0.69, so the suite, the Sphinx build and the type check are untouched; duplicate-definition scan clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RascUYS7JgCMHHpXsYbPTf
Picks up #871 (records the 2026-08-14 refactor proposals #866-#869 in agents/DEVELOPING.md). Auto-merged cleanly; the only touched file is agents/DEVELOPING.md, where main's new proposal entries sit alongside this branch's docs-layout and notebook-count text. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RascUYS7JgCMHHpXsYbPTf
|
| value | hash(value) |
equal? | same digest? |
|---|---|---|---|
0.5 |
1152921504606846976 |
no | yes |
0.1 |
230584300921369408 |
no | yes |
1e308 |
156575653125701 |
no | yes |
Fraction(1,3) |
1537228672809129301 |
no | yes |
Decimal("0.5") |
1152921504606846976 |
no | yes |
It reaches cache keys, not just digest():
>>> f.digest(0.5) == f.digest(1152921504606846976)
True— i.e. f(0.5) can return the cached result of f(1152921504606846976).
Not from this branch
- The collision reproduces against
origin/main'ssrc/fleche/digest.pyin a clean worktree (COLLIDES ON MAIN: True). - This branch does not touch the
Numberarm; itsdigest.pychanges are thePath/CompletedProcessarms andOPAQUE_ITERABLES. tests/strategies.py— which generates the example — is unchanged here, somain's own CI can draw this same example on an unlucky seed.
Seed-dependent: 3000 derandomized examples pass on both main and this branch, which is why it has stayed hidden.
What I'm not doing
Not fixing it here. Salting number digests by type (or by exact value) changes the digest of every float, complex, Fraction and Decimal, invalidating every persisted call that has one as an argument or result — a migration decision for a persistent cache, and squarely yours, not something to smuggle into a path-caching PR. Happy to take it in a follow-up if you want it, in whichever direction you prefer (full type+value salt, or keep cross-type equivalence but digest the exact value rather than hash).
I've re-run the failed job so the PR's CI state isn't left red on a seed accident. Flagging this alongside the four questions in this comment — still all awaiting your call, none started.
Generated by Claude Code
Picks up #872 (records the PR #870 MPI collective() close-out as out-of-scope in agents/DEVELOPING.md). Auto-merged cleanly; the only touched file is agents/DEVELOPING.md, where main's new close-out note sits alongside this branch's docs-layout and notebook-count text. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RascUYS7JgCMHHpXsYbPTf
…625 refresh (#882) Scheduled audit of `agents/DEVELOPING.md` against the tracker state as of 2026-08-21. `AGENTS.md` and `agents/USAGE.md` checked, no changes needed (no source changes on `main` since the last pass). - Perf-audit pointer: #625 was refreshed 2026-08-20 (doc said 2026-08-13). - #861 (FiveMinuteTour `executorlib` guard) closed 2026-08-21 — removed from the open-bugs list; the decisions bullet notes the closure date. - #709 now has PR #881 in flight (serializer registry replacing the three `with_*` classmethods) — recorded on the 2026-07-03 proposal bullet. - The Path/file-handling theme now names PR #797 (the `temppath` branch, open since 2026-07-27) as the in-flight implementation, with a one-line sketch of its content-addressed blob storage and `TempPath` rematerialization; the #829 bullet cross-references it. Docs-audit PRs #873–#878 are all still open — no state change to record there. --- _Generated by [Claude Code](https://claude.ai/code/session_017eFADsswQu1uvyacei8Cj8)_ Co-authored-by: Claude <noreply@anthropic.com>
…for #840 (#888) Scheduled audit of `AGENTS.md` and its linked guides against the tracker state as of 2026-08-22. No code landed on `main` since the previous audit (#882), so the Quick Reference and Architecture claims are unchanged; the delta is issue-tracker state only. - Add the 2026-08-21 refactor cohort under open proposals: #883 (`_raw_sub_digests` hardcodes built-in `Digested` subtypes, breaking `gc()`/`count_reuses()` for `register_destructurer` extensions — the one correctness gap of the batch), #884 (sqlite URL handling hand-rolled twice and drifted on `~`-expansion), #885 (`XDG_CACHE_HOME` companion to #868), #886 (`benchmark_storage.py` five-op timing harness duplicated). - Record PR #887 in flight against bug #840 (`BoundWrapper.__reduce__` pre-serialises `func`: `pickle` by reference, `cloudpickle` by value). - Split the single Bugs/Feature-request paragraph into one-item-per-line sub-bullets so independent updates stop colliding in merges. Verified still open and accurately described: PRs #797, #837, #873-#878, #879, #881. --- _Generated by [Claude Code](https://claude.ai/code/session_01J6LqSMaWKWzDfF7hF51Syq)_ Co-authored-by: Claude <noreply@anthropic.com>
…theme (#889) Scheduled AGENTS.md audit, 2026-08-23. No code has landed on `main` since the 2026-08-21 audit (63a0584), and the issue/PR state recorded in `agents/DEVELOPING.md` is still current: in-flight PRs #873–#878, #881, #887, and #797 all remain open; no new issues since #883–#886. Spot-checks of the Quick Reference and Architecture claims against `src/` all pass (public `__all__`, `_lazy_default`/`_sticky_set`/`_hard_set`, `PreparedCall`/two-phase save, `register_storage`, `_CACHE_TEMPLATES`, `Runtime.cputime`/`systime`, module line counts quoted in #789/#832). `AGENTS.md` and `agents/USAGE.md` need no changes. One gap found: the performance theme names "pooled file handles in `bagofholding_file.py`" as a fix candidate without noting that draft PR #786 (open since 2026-07-23) already implements it, and PR #804 (open since 2026-07-31) — the benchmark-harness fix that removes the always-evicting `SizeLimitedCache(max_size=10)` config — was recorded nowhere. Both are now listed under the perf theme as check-before-duplicating entries, added as separate paragraphs to keep future edits conflict-free. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01H9uSJpX8SLJrtYbc2ZaSv5 --- _Generated by [Claude Code](https://claude.ai/code/session_01H9uSJpX8SLJrtYbc2ZaSv5)_ Co-authored-by: claude[bot] <claude[bot]@users.noreply.github.com>
# Conflicts: # agents/DEVELOPING.md
|
Believe it or not the solution to the gc() vs Path problem is more abstract indirection! Behind both path and destructuring mixin we add another interface that recurses into selected items and route all item access through that. path and destructuring then implement that for their specific types. |
…qlFile fsync cheap fix (#898) Weekly AGENTS.md/DEVELOPING.md audit. Only one change since the 2026-08-27 pass (#897): issue #625's perf audit was refreshed the same morning. - Update the perf-audit pointer in `agents/DEVELOPING.md` from "refreshed 2026-08-20" to 2026-08-27 and record the run's verdict (no new source-caused regressions; flagged rows are the chronic `BagOfHoldingH5File` per-op open cost or noise). - Record the still-open SQL-side hot spot the refreshed audit re-flags: `Sql` fsyncs once per key on `save`/`evict` because `_configure_sqlite_pragmas` (`src/fleche/storage/sql.py:190-227`) sets `journal_mode=WAL` but no `synchronous` pragma; the cheap fix (`PRAGMA synchronous=NORMAL`) has been flagged in every audit since 2026-05-07. Verified against the source — the function sets only `foreign_keys` and `journal_mode`. Everything else checked and current: no merges to `main` since #897; open PRs (#873, #874, #887, #892, #894, #896, #797, #786, #804) and issues (#893, #895, #625) are all already recorded. `eisenforschung/landau` was audited in the same pass and needs no update (nothing landed since its 2026-08-25 pass; in-flight PRs #391/#394/#395/#414/#422 unchanged; spot-checked claims hold). --- _Generated by [Claude Code](https://claude.ai/code/session_012SCLy9y7aUR7F44Q7UviGo)_ Co-authored-by: Claude <noreply@anthropic.com>
…racker items (#904) Scheduled AGENTS.md audit. Changes since the 2026-08-27 pass: - Release-state claims updated: PR #793 (two-phase save) and PR #843 (lock-free pickle-family backends) shipped in 0.22.0 on 2026-08-28 (release PR #837); the three "unreleased as of 2026-08-27" qualifiers are gone. Noted that `fix(query)` #894 is an ancestor of the 0.22.0 tag (verified via `git tag --contains c82c691`) even though the generated changelog entry omits it. - PR #894 (`latest()`/`oldest()` raise `ValueError` when no matching call carries `Runtime` metadata, `IndexError` on empty), PR #892 (gc mid-sweep guards + 441-case digest product collapse, suite 1738 → 1299), and docs-audit PRs #873/#874 moved from the in-flight section to decisions landed; the in-flight section is now a one-liner pointing at the five PRs still open (#887, #804, #797, #786, #523), each covered under its theme. - New tracker items recorded: the 2026-08-28 refactor cohort #899–#902, bug #903 (`put()` `filelock.Timeout` fatality on ≤0.21.2, dedup-concentrated lock contention; resolved for pickle-family by 0.22.0's atomic rename, bagofholding multi-bag still exposed per #893), and the #895 resolution (documented limitation per the 2026-08-28 maintainer comment). #900 cross-referenced at the `pypi-publish.yml` line in the workflows list. - The #893 bullet's `lock_timeout` mitigation sentence rephrased to the current state: the `FutureWarning` drop covers only pickle-family configs, a bagofholding `values.lock_timeout` still works on 0.22.0 (checked against `config.py:466`). - USAGE.md: query section states the `latest()`/`oldest()` `ValueError`/`IndexError` contract (verified against `query.py:149-188`). landau's AGENTS.md was audited in the same pass and left unchanged — no code or tracker movement since its 2026-08-24 update (only docs commits on main; issues #423–#425 and PRs #422/#414/#395/#394/#391 unchanged). 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_014myhh6xKWC2VhQeB92mGSn --- _Generated by [Claude Code](https://claude.ai/code/session_014myhh6xKWC2VhQeB92mGSn)_ Co-authored-by: Claude <noreply@anthropic.com>
Stacked on #793 (
prepared-call) — merge that first; this PR shows only the path-caching commits on top of it.What
First-class support for
Pathvalues in cached functions (#516, #33, #517): files and directories are stored by content and rematerialized under temporary paths on cache hits.(basename, content); a directory is its tree alone (child names in, root name out); plainbytesis the content-only escape hatch. Nonexistent paths warn and run uncached (What's the digest of a non existing file? #517).PathValueMixinconverts paths toFileBlob/DirectoryBlobrecords referencing content-addressedbytesblobs (identical bodies deduplicate across names, directories, and bytes values), wired into every default value storage. Digest salts mirror storage exactly, preservingdigest(path) == save(path).TempPaths —Pathsubclasses whose backing temp tree lives while any derived path is referenced (3.12+ viawith_segments; 3.11 via a weak registry of live roots).DigestedDictis generalized to a type-preservingDigestedMapping; the destructurer predicate is an exact-type allowlist (dict,OrderedDict,list,tuple) since subclasses may repurpose thetype(value)(pairs)reconstruction contract (defaultdict,Counter).Docs
docs/usage/file_semantics.rst— the full user-facing contract (identity, materialization, lifetime, nesting, opaque containers, argument mutation, fidelity limits). Black-box validated: agents restricted to reading only this page wrote assertion scripts against it over three rounds; every failed or ambiguous expectation was folded back into the text.notebooks/PathsInContainers.ipynbexploring the container-nesting edge cases.Interplay with #793
Path arguments make argument mutation easy (writing into a received directory), so this branch is where the two-phase save protocol pays off visibly:
tests/integration/test_path_mutation.pypins that mutating consumers hit on honest repeats, the mutated tree is a distinct call, and a mutated argument passed back out is captured in its final state.Full suite green on Python 3.13 and 3.11.
🤖 Generated with Claude Code