diff --git a/agents/DEVELOPING.md b/agents/DEVELOPING.md index 3b25ae54..98a77385 100644 --- a/agents/DEVELOPING.md +++ b/agents/DEVELOPING.md @@ -174,8 +174,8 @@ Shared fixtures (in `fixtures.py`): - `benchmarks/` — `benchmark_{digest,integration,storage}.py`, `run_benchmarks.py`, `compare_results.py` (diff two `results.csv` runs), `utils.py`, `profile_digest_types.py` (per-type cProfile harness), `results.csv`. - `devnotes/storage-hierarchy.{dot,md,svg}` — rendered inheritance diagram for the storage classes. -- `docs/` — Sphinx sources, grouped by topic: root holds `index`, `installation`, `parallel_execution`; `usage/` holds `tldr`, `helpers`, `lazy_call`, `query` (`tldr` first, PR #745); `digests/` holds `digests_as_args`, `digest_equivalence`, `entry_points` (added by PR #752 as the dedicated fleche-ase / third-party plug-in page); `storage/` holds `configuration`, `cache_stack`, `security`; `dev/` holds `custom_digests`, `developer`, `ssh_cache`. `docs/notebooks/` is **symlinks** into `../../notebooks/` (six entries — `Caches` and `TransferWorkflow` are not exposed in docs); the `rendernb.yml` workflow re-executes `notebooks/*.ipynb` in place when a PR carries the `rendernb` label. -- `notebooks/` — usage examples (`GettingStarted`, `Caches`, `CacheStack`, `StorageBackends`, `SecureStorage`, `ConcurrentExecution`, `ExtraMethods`, `TransferWorkflow`); five of these (all except `Caches`, `ConcurrentExecution`, `TransferWorkflow`) are executed by `tests/integration/test_notebooks.py`. +- `docs/` — Sphinx sources, grouped by topic: root holds `index`, `installation`, `parallel_execution`; `usage/` holds `tldr`, `purity`, `helpers`, `file_semantics`, `lazy_call`, `query` (`tldr` first, PR #745; `purity` carries the general "fleche caches pure functions" contract — arguments keyed as passed, mutation and other side effects not replayed — that `file_semantics` states for paths); `digests/` holds `digests_as_args`, `digest_equivalence`, `entry_points` (added by PR #752 as the dedicated fleche-ase / third-party plug-in page); `storage/` holds `configuration`, `cache_stack`, `security`; `dev/` holds `custom_digests`, `developer`, `path_storage`, `ssh_cache`; `recipes/` holds `files_and_paths`. `docs/notebooks/` is **symlinks** into `../../notebooks/` (eight entries — `Caches` and `TransferWorkflow` are not exposed in docs); the `rendernb.yml` workflow re-executes `notebooks/*.ipynb` in place when a PR carries the `rendernb` label. +- `notebooks/` — usage examples (`GettingStarted`, `Caches`, `CacheStack`, `StorageBackends`, `SecureStorage`, `ConcurrentExecution`, `ExtraMethods`, `Files`, `PathsInContainers`, `TransferWorkflow`); seven of these (all except `Caches`, `ConcurrentExecution`, `TransferWorkflow`) are executed by `tests/integration/test_notebooks.py`. Committed **with outputs** — `rendernb.yml` re-executes them in place on a PR labelled `rendernb`, so an edited notebook should be re-run before it lands. - `.github/workflows/` — CI: `tests.yml` (PR sweep across 3.11–3.14 + `sql-backends` job that boots Postgres 16 + MariaDB 11 service containers and sets `FLECHE_TEST_{POSTGRES,MYSQL}_URL`), `ty.yml`, `test-minimum-deps.yml` (PR #645, installs every direct dep at its declared floor via `uv pip install --resolution lowest-direct -e ".[tests]"` and runs the suite — `uv pip install` rather than `uv sync` because the universal lock floats deps up through cross-extra constraints and would mask the real minimums), `benchmarks.yml`/`benchmarks-main.yml`/`updatebenchmarks.yml`, `perf-triage.yml` (PR #628 — Haiku reads the PR diff/description and adds the `benchmark` label when the change touches a hot path — that label is the existing `benchmarks.yml` trigger), `rendernb.yml` (re-executes `notebooks/*.ipynb` on PRs labelled `rendernb`), `release-please.yml`, `pypi-publish.yml` (trusted-publisher upload triggered by `release: published`), `claude.yaml` + `claude_ci_details.yaml` (the latter exposes CI status as a tool the in-PR Claude can call). Releases use **release-please** (`release-please-config.json`, `.release-please-manifest.json`) — release PRs are opened automatically from conventional-commit history on `main`. ## Design themes / open scope (issue tracker) @@ -217,6 +217,7 @@ Cheat sheet of what's been considered. Issue numbers are the entry points — fe - `CachePool(ReadOnlyMixin, _MultiCache)` — a read-only unordered collection of caches (PR #689). Extracted the `CacheStack` read fan-out into a shared `_MultiCache(BaseCache)` base (`_first_hit`/`_collect`/`_foreach` + `contains`/`load_value`/`expand`/`_shrink`/`_query`) so `CacheStack` and `CachePool` share one implementation. `save`/`evict` raise `Rejected`; config dispatches `{"pool": [...]}` alongside the existing list→`CacheStack` and `type="ssh"`→`SshCache` paths. Pinned in `tests/unit/caches/test_cache_pool.py`. - Cache-level race fixes on top of #569: #217 → PR #629, #451 → PR #631, #452 → PR #630, #485 → PR #627. Regression-pinned in `tests/regression/test_issue_{217,451,452,485}.py`; concurrency stress tests share `run_workers()` from `tests/fixtures.py` (PR #633). - Digest dispatch grew a `builtins`-type arm and a `not isinstance(value, type)` guard on the dataclass/attrs arms so `digest(int)` / `digest(SomeDataclass)` (the class object) stop raising — PR #651, closes #469. +- `digest()` grew a `subprocess.CompletedProcess` arm (args + returncode + stdout + stderr) so functions wrapping command-line tools hash without a user hook — `run()`'s result is neither a dataclass nor iterable, so it was `Indigestible` before. Purely additive (those values raised previously), so no `hash_version` bump. Pinned in `tests/unit/digest/test_digest.py::test_completedprocess_*`; `notebooks/Files.ipynb` used to register this by hand via `add_hook` and no longer does. - `digest()` grew explicit `pd.DataFrame` / `pd.Series` / `pd.Index` arms so pandas inputs hash by content (columns/name + dtype + index + `hash_pandas_object`), not by the column names yielded by `iter()` — PR #675. Silent change with no `hash_version` bump (previous pandas cache entries are now unreachable). Pinned in `tests/unit/digest/test_digest.py::test_pandas_{dataframe,series,index}_hashes_by_content`. - `SizeLimitedMixin` swapped its raw `threading.RLock` for `_PicklableRLock` so `SizeLimitedCache` survives `pickle.dumps` (PR #664, closes #663). - `MetaData.keys` is uniformly a `@property` across every built-in (`Runtime`/`Environment`/`Git`/`Tags`) instead of the prior mix of class-level dict attributes for the zero-arg three plus a property for `Tags` (PR #690, closes #637). The abstract-property-overridden-by-class-attribute smell on the ABC is gone. PR #763 (closes #738) then single-sourced the static-schema built-ins by moving the shared schema to a `_keys: ClassVar[dict[str, type]]` class attribute on the ABC (default `{}`) with `MetaData.keys` a concrete property returning `self._keys`; `Runtime`/`Environment`/`Git` now declare `_keys` once (dropping their per-class `keys` override), while `Tags` keeps overriding `keys` directly since its schema is per-instance. Drift guard: `tests/unit/metadata/test_metadata.py::test_builtin_metadata_pre_post_keys_match_schema` asserts `set(pre_output) | set(post_output) == set(cls._keys)` for each of the three static built-ins. `name` is still set by `@configurable` (`Runtime`/`Environment`/`Git`) or as a manual dataclass attribute (`Tags`) — unchanged. diff --git a/docs/index.rst b/docs/index.rst index 394d79a2..41b9581e 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -31,6 +31,7 @@ Welcome to the **Fleche** library documentation. :caption: Using Fleche usage/tldr + usage/purity usage/helpers usage/file_semantics usage/lazy_call @@ -84,6 +85,7 @@ Welcome to the **Fleche** library documentation. notebooks/CacheStack notebooks/ConcurrentExecution notebooks/Files + notebooks/PathsInContainers .. toctree:: :maxdepth: 2 diff --git a/docs/notebooks/PathsInContainers.ipynb b/docs/notebooks/PathsInContainers.ipynb new file mode 120000 index 00000000..6382d357 --- /dev/null +++ b/docs/notebooks/PathsInContainers.ipynb @@ -0,0 +1 @@ +../../notebooks/PathsInContainers.ipynb \ No newline at end of file diff --git a/docs/usage/file_semantics.rst b/docs/usage/file_semantics.rst index 25e297ee..d8d5d50a 100644 --- a/docs/usage/file_semantics.rst +++ b/docs/usage/file_semantics.rst @@ -44,19 +44,20 @@ functions returning identical files share one stored body (see Argument mutation ~~~~~~~~~~~~~~~~~ -A call is keyed on its arguments **as passed**: argument content is captured -*before* the function body runs. A function that mutates its own argument — -most commonly, writing an output file *into* a directory it received — is -still recorded under the pre-call content, so honest repeat calls hit. The -mutation itself, however, is neither recorded nor replayed: a cache hit -leaves the argument untouched, so a side effect on the input happens on cold -calls only. - -fleche caches *pure* functions. What a function does to its arguments -without passing it back out is invisible to the cache — treat received paths -as read-only and write outputs to a fresh directory (``tempfile.mkdtemp``). -A mutated argument that *is* returned is captured faithfully in its final, -post-mutation state: if the mutation is the point, return it. +Arguments are keyed **as passed** — captured before the body runs — and a +mutation the body performs on one is neither recorded nor replayed. That is +a general rule (:ref:`argument-mutation`); its most common instance here is a +function that writes an output file *into* a directory it received. The call +is recorded under the directory's pre-call tree, so honest repeat calls hit, +but the written file does not reappear on a hit — it exists only on cold +calls. + +Returning what you wrote to is the normal shape, not a workaround for this: a +function that writes into a directory and hands that directory back is recorded +with it in its final, post-mutation state, and that is the intended way to +produce files. What is not replayed is a write to something you never return — +so a path that is pure *input* should be treated as read-only, with new files +written somewhere you do return (``tempfile.mkdtemp``). Only *content* changes count as mutation: permissions are not part of identity (see :ref:`fidelity-limits`), so a ``chmod`` on a received path is @@ -146,13 +147,30 @@ the file. If you need the file at a stable location, copy it out: Paths nested inside containers ------------------------------ -Paths are found and content-stored inside the containers fleche takes apart: -``dict``, ``OrderedDict``, ``list``, ``tuple`` (**exact types** — see below), -``dataclasses`` and ``attrs`` classes — nested to any depth, as values *or* as -dict keys. Everything above about identity, materialization, and lifetime -applies to each nested path individually. Container structure is otherwise -faithful on a hit: lists and tuples keep their element order, and a mended -dict keeps the insertion order the stored value had. +Whether a nested path is stored by content comes down to one thing: +**destructuring** — whether storage takes the surrounding container apart into +independently-stored children, or pickles it whole as one opaque value. Only +children reach the path machinery, so the list of containers fleche +destructures *is* the list of places a nested path gets content treatment. +That list is +:data:`~fleche.storage.destructuring._DESTRUCTURERS`: ``dict``, +``OrderedDict``, ``list``, ``tuple`` (**exact types** — see below), +``dataclasses`` and ``attrs`` classes. See :ref:`extending-destructurer` for +the mechanism and how to add your own container to it. + +Within those, paths are found nested to any depth, as values *or* as dict +keys, and everything above about identity, materialization, and lifetime +applies to each one individually. Container structure is otherwise faithful +on a hit: lists and tuples keep their element order, and a mended dict keeps +the insertion order the stored value had. + +"Any depth" is not a figure of speech, and no storage setting narrows it. A +``Path`` matches no destructurer, so it is always written out as its own stored +entry rather than inlined into the container above it — and being written out is +exactly what hands it to the content machinery. The +``remaining_depth`` knob only decides how eagerly *destructurable* nodes are +split into separate entries, so it cannot put a path out of reach however it is +set. Three caveats specific to nesting: @@ -175,7 +193,8 @@ Opaque containers store paths by *location* ------------------------------------------- A path inside an opaque value (a ``namedtuple``, a ``set``, an arbitrary -object) never reaches the content machinery. The call is still *keyed* +object) is never destructured out of it, and so never reaches the content +machinery. The call is still *keyed* correctly — the digest layer does look inside — but what is stored is the path object itself, pointing at wherever the file was when it was saved. A hit returns that original *location* (whether as the same object or an equal copy @@ -183,8 +202,9 @@ is backend-dependent — rely on neither): if the file has since been deleted or edited, the hit hands you a dangling or stale path, **without any warning**. Rule of thumb: return paths in plain dicts / lists / tuples / dataclasses. If -you need a custom container to participate, see -:func:`~fleche.storage.destructuring.register_destructurer`. +you need a custom container to participate, register a destructurer for it +with :func:`~fleche.storage.destructuring.register_destructurer` — see +:ref:`extending-destructurer`. .. _file-dedup: diff --git a/docs/usage/helpers.rst b/docs/usage/helpers.rst index 001da91e..4df64fb1 100644 --- a/docs/usage/helpers.rst +++ b/docs/usage/helpers.rst @@ -38,7 +38,7 @@ Attempts to load the result of a specific call from the cache. If the result is Returns ``True`` if the result for the given call is already present in the cache, ``False`` otherwise. ``.query(*args, metadata={}, **kwargs)`` -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Returns matching cached calls from the active cache. Any argument passed as ``None`` acts as a wildcard, matching any stored value for that parameter. The ``metadata`` keyword argument accepts a dictionary of metadata tags to further filter results (e.g., ``metadata={"tags": {"project": "alpha"}}``). @@ -102,7 +102,9 @@ Functions Returning ``None`` Functions that return ``None`` are **never cached**. When a decorated function returns ``None``, ``fleche`` logs a ``WARNING`` and skips the save step entirely. Subsequent calls will execute the function again rather than -returning a cached value. +returning a cached value — including any side effects they have, which for a +cached function would otherwise happen on the cold call only (see +:doc:`purity`). This applies to all code paths, with one difference for ``.rerun()``: diff --git a/docs/usage/purity.rst b/docs/usage/purity.rst new file mode 100644 index 00000000..e70dfe0e --- /dev/null +++ b/docs/usage/purity.rst @@ -0,0 +1,101 @@ +.. _purity: + +Purity and Side Effects +======================= + +What fleche assumes about the functions you decorate, and what therefore does +*not* survive a cache hit. The rules here apply to every argument and result +type; :doc:`file_semantics` is the same contract seen through ``Path`` values. + +The assumption +-------------- + +fleche caches **pure** functions: the result is determined by the arguments, +and anything else the body does is incidental. A cache hit replays the +*result* and nothing else — so every effect a function has besides returning +a value happens on cold calls only. + +That is not a restriction fleche can check, and it does not raise if you break +it. It shows up as a function that behaves differently the second time you +call it. + +.. _argument-mutation: + +Arguments are keyed as passed +----------------------------- + +Argument content is captured **before** the body runs, so a function that +mutates its own argument is still recorded under the pre-call content and +honest repeat calls hit. The mutation itself is neither recorded nor +replayed: + +.. code-block:: python + + @fleche + def append_and_report(xs, n): + xs.append(n) + return len(xs) + + a = [1, 2] + append_and_report(a, 9) # 3 — body runs, and a is now [1, 2, 9] + + b = [1, 2] + append_and_report(b, 9) # 3 — cache hit; b is still [1, 2] + +Both calls return ``3``, because that is the recorded result. Only the first +one changed its argument. Nothing about this is specific to lists: a +``dict``, a numpy array, and a directory a function writes into all behave the +same way. + +Return what you mutate +~~~~~~~~~~~~~~~~~~~~~~ + +An argument that is mutated **and returned** is captured faithfully in its +final, post-mutation state — the result is stored after the body runs: + +.. code-block:: python + + @fleche + def append_and_return(xs, n): + xs.append(n) + return xs + + append_and_return([1, 2], 9) # [1, 2, 9] — body runs + append_and_return([1, 2], 9) # [1, 2, 9] — cache hit, same value + +If the mutation is the point of the function, return it. Treat arguments you +receive as read-only otherwise, and build results fresh. + +Other side effects +------------------ + +Everything else a body does — printing, logging, writing files outside the +returned value, sending a request, inserting a row — happens on the cold call +and never again: + +.. code-block:: python + + @fleche + def record(x): + db.insert(x) # runs once, ever + return x * 2 + +If an effect must happen on every call, it belongs outside the cached +function; keep the cached part to the computation whose result you want +stored. + +Two related cases +----------------- + +* A function returning ``None`` is never cached, so it re-executes every time + — including its side effects. See :ref:`none-not-cached`. +* Paths follow all of the above, with the mutation case made concrete: + a directory a function receives and writes into is recorded under its + pre-call tree, and the written file does not reappear on a hit. See + :doc:`file_semantics`. + +See also +-------- + +* :doc:`file_semantics` — the same contract for files and directories. +* :doc:`helpers` — ``.rerun()`` for forcing a cold call deliberately. diff --git a/docs/usage/tldr.rst b/docs/usage/tldr.rst index e3332e9c..3aa938cf 100644 --- a/docs/usage/tldr.rst +++ b/docs/usage/tldr.rst @@ -60,8 +60,9 @@ directories are cached by **content** (not by path string), and a cache hit returns a freshly materialized copy under a temporary path. See :doc:`file_semantics` for the exact contract. -That is all you need for everyday use. The rest of this section covers the -helper methods in depth (:doc:`helpers`), the file/path contract +That is all you need for everyday use. The rest of this section covers what +fleche assumes about the functions you decorate (:doc:`purity`), the helper +methods in depth (:doc:`helpers`), the file/path contract (:doc:`file_semantics`), lazy loading of large cached objects (:doc:`lazy_call`), and querying stored calls (:doc:`query`); storage backends and configuration details live under :doc:`/storage/configuration`. diff --git a/notebooks/Files.ipynb b/notebooks/Files.ipynb index e3e72b12..f76494f3 100644 --- a/notebooks/Files.ipynb +++ b/notebooks/Files.ipynb @@ -8,8 +8,8 @@ "# Caching functions that read and write files\n", "\n", "`fleche` is a content-addressed cache. When a cached function takes or returns a\n", - "`pathlib.Path`, fleche stores the file's (or directory tree's) **contents** \u2014 not\n", - "just the path string \u2014 so results are portable and reproducible across machines.\n", + "`pathlib.Path`, fleche stores the file's (or directory tree's) **contents** — not\n", + "just the path string — so results are portable and reproducible across machines.\n", "\n", "A path is keyed by its content, so:\n", "\n", @@ -18,19 +18,37 @@ " file lives at a different location.\n", "\n", "The in-memory cache (`cache(\"memory\")`) and every default value storage now carry\n", - "this behaviour out of the box \u2014 no custom storage composition required." + "this behaviour out of the box — no custom storage composition required." ] }, { "cell_type": "code", + "execution_count": 1, "id": "cell-01", - "metadata": {}, - "execution_count": null, - "outputs": [], + "metadata": { + "execution": { + "iopub.execute_input": "2026-08-06T19:58:48.374404Z", + "iopub.status.busy": "2026-08-06T19:58:48.374124Z", + "iopub.status.idle": "2026-08-06T19:58:49.119391Z", + "shell.execute_reply": "2026-08-06T19:58:49.117923Z" + } + }, + "outputs": [ + { + "data": { + "text/plain": [ + "['PerKeyLockMixin', 'DestructuringMixin', 'PathValueMixin', 'ValueMixin']" + ] + }, + "execution_count": 1, + "metadata": {}, + "output_type": "execute_result" + } + ], "source": [ "import tempfile\n", "from pathlib import Path\n", - "from subprocess import run, CompletedProcess\n", + "from subprocess import run\n", "\n", "import fleche as fl\n", "from fleche import fleche\n", @@ -44,10 +62,28 @@ }, { "cell_type": "code", + "execution_count": 2, "id": "cell-02", - "metadata": {}, - "execution_count": null, - "outputs": [], + "metadata": { + "execution": { + "iopub.execute_input": "2026-08-06T19:58:49.122636Z", + "iopub.status.busy": "2026-08-06T19:58:49.122082Z", + "iopub.status.idle": "2026-08-06T19:58:49.128658Z", + "shell.execute_reply": "2026-08-06T19:58:49.126851Z" + } + }, + "outputs": [ + { + "data": { + "text/plain": [ + "PosixPath('/tmp/tmpyxng_i3t-fleche-files')" + ] + }, + "execution_count": 2, + "metadata": {}, + "output_type": "execute_result" + } + ], "source": [ "# A scratch directory for the files our functions produce.\n", "WORK = Path(tempfile.mkdtemp(suffix=\"-fleche-files\"))\n", @@ -68,10 +104,26 @@ }, { "cell_type": "code", + "execution_count": 3, "id": "cell-04", - "metadata": {}, - "execution_count": null, - "outputs": [], + "metadata": { + "execution": { + "iopub.execute_input": "2026-08-06T19:58:49.132107Z", + "iopub.status.busy": "2026-08-06T19:58:49.131691Z", + "iopub.status.idle": "2026-08-06T19:58:49.141581Z", + "shell.execute_reply": "2026-08-06T19:58:49.140088Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + " [write] running: 'hello'\n", + "returned: PosixPath -> hellohello\n" + ] + } + ], "source": [ "@fleche\n", "def write(text, repeat=1, name=\"out.txt\"):\n", @@ -86,10 +138,25 @@ }, { "cell_type": "code", + "execution_count": 4, "id": "cell-05", - "metadata": {}, - "execution_count": null, - "outputs": [], + "metadata": { + "execution": { + "iopub.execute_input": "2026-08-06T19:58:49.144539Z", + "iopub.status.busy": "2026-08-06T19:58:49.144174Z", + "iopub.status.idle": "2026-08-06T19:58:49.151092Z", + "shell.execute_reply": "2026-08-06T19:58:49.149640Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "from cache: hellohello\n" + ] + } + ], "source": [ "# Same arguments -> cache hit -> the body does NOT run (no \"[write] running\").\n", "again = write(\"hello\", 2)\n", @@ -109,10 +176,28 @@ }, { "cell_type": "code", + "execution_count": 5, "id": "cell-07", - "metadata": {}, - "execution_count": null, - "outputs": [], + "metadata": { + "execution": { + "iopub.execute_input": "2026-08-06T19:58:49.153583Z", + "iopub.status.busy": "2026-08-06T19:58:49.153313Z", + "iopub.status.idle": "2026-08-06T19:58:49.166569Z", + "shell.execute_reply": "2026-08-06T19:58:49.165422Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + " [write] running: 'a quick brown fox'\n", + " [wordcount] running: sentence.txt\n", + "count: 4\n", + "count again: 4\n" + ] + } + ], "source": [ "@fleche\n", "def wordcount(path: Path):\n", @@ -137,10 +222,35 @@ }, { "cell_type": "code", + "execution_count": 6, "id": "cell-09", - "metadata": {}, - "execution_count": null, - "outputs": [], + "metadata": { + "execution": { + "iopub.execute_input": "2026-08-06T19:58:49.169824Z", + "iopub.status.busy": "2026-08-06T19:58:49.169546Z", + "iopub.status.idle": "2026-08-06T19:58:49.183369Z", + "shell.execute_reply": "2026-08-06T19:58:49.181356Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + " [make_tree] running: alpha\n" + ] + }, + { + "data": { + "text/plain": [ + "['sub', 'sub/leaf.bin', 'top.txt']" + ] + }, + "execution_count": 6, + "metadata": {}, + "output_type": "execute_result" + } + ], "source": [ "@fleche\n", "def make_tree(seed):\n", @@ -163,10 +273,35 @@ }, { "cell_type": "code", + "execution_count": 7, "id": "cell-10", - "metadata": {}, - "execution_count": null, - "outputs": [], + "metadata": { + "execution": { + "iopub.execute_input": "2026-08-06T19:58:49.186305Z", + "iopub.status.busy": "2026-08-06T19:58:49.186041Z", + "iopub.status.idle": "2026-08-06T19:58:49.212754Z", + "shell.execute_reply": "2026-08-06T19:58:49.211003Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + " [total_bytes] running: ff88b2cec3e828c231a8a9df977be4f4cfa2e938cc0cdc41aac1e03350f5af8b\n" + ] + }, + { + "data": { + "text/plain": [ + "20" + ] + }, + "execution_count": 7, + "metadata": {}, + "output_type": "execute_result" + } + ], "source": [ "# End-to-end cache hit: make_tree(\"alpha\") and total_bytes both come from cache.\n", "total_bytes(make_tree(\"alpha\"))" @@ -185,10 +320,27 @@ }, { "cell_type": "code", + "execution_count": 8, "id": "cell-12", - "metadata": {}, - "execution_count": null, - "outputs": [], + "metadata": { + "execution": { + "iopub.execute_input": "2026-08-06T19:58:49.215720Z", + "iopub.status.busy": "2026-08-06T19:58:49.215476Z", + "iopub.status.idle": "2026-08-06T19:58:49.223612Z", + "shell.execute_reply": "2026-08-06T19:58:49.222050Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + " [write] running: 'shared body'\n", + " [write] running: 'shared body'\n", + "shared body stored once: True\n" + ] + } + ], "source": [ "# Two calls, different names, identical body -> the content is stored once.\n", "write(\"shared body\", 1, name=\"left.txt\")\n", @@ -200,13 +352,37 @@ }, { "cell_type": "code", + "execution_count": 9, "id": "cell-13", - "metadata": {}, - "execution_count": null, - "outputs": [], + "metadata": { + "execution": { + "iopub.execute_input": "2026-08-06T19:58:49.227426Z", + "iopub.status.busy": "2026-08-06T19:58:49.227174Z", + "iopub.status.idle": "2026-08-06T19:58:49.232930Z", + "shell.execute_reply": "2026-08-06T19:58:49.231276Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "FileBlob('out.txt', '785d68f8426805e292630852bdedb46dd56ac44dcb7047740d30704f3d84d4fa')\n", + "FileBlob('sentence.txt', '0f61b76af53fa2dc41528c3866206d22ceb3b7f560e7f42aacc006fc5ace228c')\n", + "DirectoryBlob({'leaf.bin': 'c70f6db1a5371bc6046fb5a040fd13bd5220c78908eac8126c1361daad854904'})\n", + "DirectoryBlob({'sub': '7aa02a10cd58fe7b0f16ef0e06b255d7842ce362687e489e941281d27761d95a', 'top.txt': '8eb42147b1727df4b082ebc0bdfc5fbaea064308411a4d801cae67c908ce4287'})\n", + "FileBlob('left.txt', '32cbd77d1dbff488cd42dc84ea72ebd47358fbf321412b35a5c1084e36f5b775')\n", + "FileBlob('right.txt', '32cbd77d1dbff488cd42dc84ea72ebd47358fbf321412b35a5c1084e36f5b775')\n" + ] + } + ], "source": [ - "# Introspect the recorded calls of any cached function.\n", - "write.query().table()" + "# What a `Path` actually becomes in storage: its content as plain `bytes` under\n", + "# its own digest, plus a small record pairing that content with a name. Note\n", + "# that left.txt and right.txt reference the *same* content digest.\n", + "for blob in c.values.storage.values():\n", + " if type(blob).__name__ in (\"FileBlob\", \"DirectoryBlob\"):\n", + " print(blob)\n" ] }, { @@ -220,22 +396,26 @@ "step runs in a working directory, produces files, and the whole pipeline is\n", "cached by content.\n", "\n", - "A `subprocess.CompletedProcess` isn't digestible out of the box, so we register a\n", - "digest hook describing how to fingerprint one." + "Each step returns the `subprocess.CompletedProcess` that `run()` produced. fleche\n", + "digests those directly — by `args`, `returncode`, `stdout`, and `stderr` — so the\n", + "captured output participates in the key with no extra setup. For a type fleche\n", + "does not know, `fl.digest.add_hook((TheType, fn))` is the extension point." ] }, { "cell_type": "code", + "execution_count": 10, "id": "cell-15", - "metadata": {}, - "execution_count": null, + "metadata": { + "execution": { + "iopub.execute_input": "2026-08-06T19:58:49.235969Z", + "iopub.status.busy": "2026-08-06T19:58:49.235619Z", + "iopub.status.idle": "2026-08-06T19:58:49.245255Z", + "shell.execute_reply": "2026-08-06T19:58:49.243354Z" + } + }, "outputs": [], "source": [ - "def digest_completedprocess(cp):\n", - " return fl.digest.digest((type(cp).__name__, cp.args, cp.returncode, cp.stdout, cp.stderr))\n", - "\n", - "fl.digest.add_hook((CompletedProcess, digest_completedprocess))\n", - "\n", "@fleche\n", "def shell(cwd, prog, args=(), stdin=b\"\"):\n", " print(\" [shell] running:\", prog, *args)\n", @@ -255,10 +435,30 @@ }, { "cell_type": "code", + "execution_count": 11, "id": "cell-16", - "metadata": {}, - "execution_count": null, - "outputs": [], + "metadata": { + "execution": { + "iopub.execute_input": "2026-08-06T19:58:49.248014Z", + "iopub.status.busy": "2026-08-06T19:58:49.247696Z", + "iopub.status.idle": "2026-08-06T19:58:49.269994Z", + "shell.execute_reply": "2026-08-06T19:58:49.268579Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "--- first run ---\n", + " [pipeline] running: b'hello world'\n", + " [shell] running: cp input.txt copy.txt\n", + " [shell] running: tr a-z A-Z\n", + "produced: ['copy.txt', 'input.txt', 'shout.txt']\n", + "shout.txt: HELLO WORLD\n" + ] + } + ], "source": [ "print(\"--- first run ---\")\n", "out = pipeline(b\"hello world\")\n", @@ -268,10 +468,26 @@ }, { "cell_type": "code", + "execution_count": 12, "id": "cell-17", - "metadata": {}, - "execution_count": null, - "outputs": [], + "metadata": { + "execution": { + "iopub.execute_input": "2026-08-06T19:58:49.272480Z", + "iopub.status.busy": "2026-08-06T19:58:49.272198Z", + "iopub.status.idle": "2026-08-06T19:58:49.280138Z", + "shell.execute_reply": "2026-08-06T19:58:49.277972Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "--- second run: fully cached (no body / shell prints) ---\n", + "same files: ['copy.txt', 'input.txt', 'shout.txt']\n" + ] + } + ], "source": [ "print(\"--- second run: fully cached (no body / shell prints) ---\")\n", "out2 = pipeline(b\"hello world\")\n", @@ -280,10 +496,81 @@ }, { "cell_type": "code", + "execution_count": 13, "id": "cell-18", - "metadata": {}, - "execution_count": null, - "outputs": [], + "metadata": { + "execution": { + "iopub.execute_input": "2026-08-06T19:58:49.283069Z", + "iopub.status.busy": "2026-08-06T19:58:49.282802Z", + "iopub.status.idle": "2026-08-06T19:58:49.301525Z", + "shell.execute_reply": "2026-08-06T19:58:49.300406Z" + } + }, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
namemoduletimestarttimestopwalltime
6f89shell__main__2026-08-06 19:58:49.251591921+00:002026-08-06 19:58:49.259691+00:000.008099
a668shell__main__2026-08-06 19:58:49.261044025+00:002026-08-06 19:58:49.264387608+00:000.003344
\n", + "
" + ], + "text/plain": [ + " name module timestart \\\n", + "6f89 shell __main__ 2026-08-06 19:58:49.251591921+00:00 \n", + "a668 shell __main__ 2026-08-06 19:58:49.261044025+00:00 \n", + "\n", + " timestop walltime \n", + "6f89 2026-08-06 19:58:49.259691+00:00 0.008099 \n", + "a668 2026-08-06 19:58:49.264387608+00:00 0.003344 " + ] + }, + "execution_count": 13, + "metadata": {}, + "output_type": "execute_result" + } + ], "source": [ "# Every shell invocation fleche recorded:\n", "shell.query().table()" @@ -297,7 +584,16 @@ "name": "python3" }, "language_info": { - "name": "python" + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.11.15" } }, "nbformat": 4, diff --git a/notebooks/PathsInContainers.ipynb b/notebooks/PathsInContainers.ipynb index 65ed0006..98c8849e 100644 --- a/notebooks/PathsInContainers.ipynb +++ b/notebooks/PathsInContainers.ipynb @@ -5,7 +5,7 @@ "id": "9050c7cf", "metadata": {}, "source": [ - "# Paths nested inside containers: the usual UX, and the edges\n", + "# Paths nested inside containers: the usual UX, and what to beware of\n", "\n", "[Files.ipynb](Files.ipynb) shows the sunny-day behaviour of `Path` arguments and\n", "return values: files and directories are stored by **content** and rematerialized\n", @@ -13,10 +13,19 @@ "\n", "This notebook explores what happens when paths are *nested inside other values* —\n", "dicts, lists, dataclasses — which fleche's `DestructuringMixin` takes apart and\n", - "reassembles (\"mends\") around the path machinery. The first half is the intended\n", - "UX; the second half collects the edge cases where a cache hit is **not** a faithful\n", - "replay of the original call: location changes, aliasing loss, path dict-keys, and\n", - "container types whose mending is incomplete or outright broken." + "reassembles (\"mends\") around the path machinery.\n", + "\n", + "The first half is the intended UX. The second half is what to watch out for, in\n", + "two distinct flavours:\n", + "\n", + "- **Beware** — consequences of what caching a pure function by content *means*.\n", + " A cached call is replayed by its **value**, so a file comes back as a copy: its\n", + " location, its identity, and whatever sat next to it on disk were never part of\n", + " that value. Code that leaned on them was relying on something the cache never\n", + " promised. These are not defects and they are not going to change.\n", + "- **Caveat** — real limits of the mending machinery, where a hit hands back\n", + " something less faithful than it could: paths used as dict keys, and paths\n", + " hidden inside containers fleche does not destructure.\n" ] }, { @@ -25,10 +34,10 @@ "id": "1ba85e32", "metadata": { "execution": { - "iopub.execute_input": "2026-07-25T19:23:11.833017Z", - "iopub.status.busy": "2026-07-25T19:23:11.832839Z", - "iopub.status.idle": "2026-07-25T19:23:12.221170Z", - "shell.execute_reply": "2026-07-25T19:23:12.220562Z" + "iopub.execute_input": "2026-08-06T19:58:50.542910Z", + "iopub.status.busy": "2026-08-06T19:58:50.542683Z", + "iopub.status.idle": "2026-08-06T19:58:51.227501Z", + "shell.execute_reply": "2026-08-06T19:58:51.225855Z" } }, "outputs": [ @@ -67,10 +76,10 @@ "id": "6825eb50", "metadata": { "execution": { - "iopub.execute_input": "2026-07-25T19:23:12.222514Z", - "iopub.status.busy": "2026-07-25T19:23:12.222262Z", - "iopub.status.idle": "2026-07-25T19:23:12.224735Z", - "shell.execute_reply": "2026-07-25T19:23:12.224328Z" + "iopub.execute_input": "2026-08-06T19:58:51.230438Z", + "iopub.status.busy": "2026-08-06T19:58:51.229894Z", + "iopub.status.idle": "2026-08-06T19:58:51.235385Z", + "shell.execute_reply": "2026-08-06T19:58:51.233846Z" } }, "outputs": [], @@ -102,10 +111,10 @@ "id": "96e644a1", "metadata": { "execution": { - "iopub.execute_input": "2026-07-25T19:23:12.226046Z", - "iopub.status.busy": "2026-07-25T19:23:12.225909Z", - "iopub.status.idle": "2026-07-25T19:23:12.231118Z", - "shell.execute_reply": "2026-07-25T19:23:12.230608Z" + "iopub.execute_input": "2026-08-06T19:58:51.238073Z", + "iopub.status.busy": "2026-08-06T19:58:51.237818Z", + "iopub.status.idle": "2026-08-06T19:58:51.248360Z", + "shell.execute_reply": "2026-08-06T19:58:51.247018Z" } }, "outputs": [ @@ -114,7 +123,7 @@ "output_type": "stream", "text": [ " [fit] running: alpha\n", - "cold: PosixPath -> /tmp/claude-1000/tmpddupkllp-fleche-nested/alpha-fit.txt\n" + "cold: PosixPath -> /tmp/tmp4rik9ggl-fleche-nested/alpha-fit.txt\n" ] } ], @@ -142,10 +151,10 @@ "id": "74dac313", "metadata": { "execution": { - "iopub.execute_input": "2026-07-25T19:23:12.232200Z", - "iopub.status.busy": "2026-07-25T19:23:12.232055Z", - "iopub.status.idle": "2026-07-25T19:23:12.234825Z", - "shell.execute_reply": "2026-07-25T19:23:12.234336Z" + "iopub.execute_input": "2026-08-06T19:58:51.250981Z", + "iopub.status.busy": "2026-08-06T19:58:51.250755Z", + "iopub.status.idle": "2026-08-06T19:58:51.257133Z", + "shell.execute_reply": "2026-08-06T19:58:51.255733Z" } }, "outputs": [ @@ -153,7 +162,7 @@ "name": "stdout", "output_type": "stream", "text": [ - "warm: TempPath -> /tmp/claude-1000/tmphvfdmrplfleche/alpha-fit.txt\n", + "warm: TempPath -> /tmp/tmpqu7_eme9fleche/alpha-fit.txt\n", "content: loss=5 | name kept: alpha-fit.txt\n" ] } @@ -180,10 +189,10 @@ "id": "1a38c092", "metadata": { "execution": { - "iopub.execute_input": "2026-07-25T19:23:12.236049Z", - "iopub.status.busy": "2026-07-25T19:23:12.235897Z", - "iopub.status.idle": "2026-07-25T19:23:12.239485Z", - "shell.execute_reply": "2026-07-25T19:23:12.239067Z" + "iopub.execute_input": "2026-08-06T19:58:51.259707Z", + "iopub.status.busy": "2026-08-06T19:58:51.259397Z", + "iopub.status.idle": "2026-08-06T19:58:51.266926Z", + "shell.execute_reply": "2026-08-06T19:58:51.265381Z" } }, "outputs": [ @@ -209,15 +218,18 @@ "id": "4db3a027", "metadata": {}, "source": [ - "## Edge 1: a hit changes *where* (and what type) the path is\n", + "## Beware: a hit gives you a *copy*, somewhere else\n", "\n", "The cold call returns whatever the function returned — the real location in\n", "`WORK`, as a plain `Path`. A warm hit returns a `TempPath` in a fresh temporary\n", "directory. Content is identical; **location is not**.\n", "\n", - "The classic footgun: code that resolves *siblings* of a returned path\n", - "(`p.parent / \"meta.json\"`) works on the first call and breaks on every hit,\n", - "because the materialized file sits alone in its temp directory." + "The function's value is the file it returned, and a location is not part of a\n", + "file's content — so this is content addressing working exactly as advertised, not\n", + "a fidelity gap. The practical consequence: code that resolves *siblings* of a\n", + "returned path (`p.parent / \"meta.json\"`) works on the first call and breaks on\n", + "every hit, because only what was returned got captured — the sibling was never\n", + "part of the value.\n" ] }, { @@ -226,10 +238,10 @@ "id": "7fe21179", "metadata": { "execution": { - "iopub.execute_input": "2026-07-25T19:23:12.240747Z", - "iopub.status.busy": "2026-07-25T19:23:12.240600Z", - "iopub.status.idle": "2026-07-25T19:23:12.245110Z", - "shell.execute_reply": "2026-07-25T19:23:12.244626Z" + "iopub.execute_input": "2026-08-06T19:58:51.269472Z", + "iopub.status.busy": "2026-08-06T19:58:51.269194Z", + "iopub.status.idle": "2026-08-06T19:58:51.278562Z", + "shell.execute_reply": "2026-08-06T19:58:51.277139Z" } }, "outputs": [ @@ -239,7 +251,7 @@ "text": [ " [produce] running: beta\n", "cold sibling exists: True\n", - "warm location: /tmp/claude-1000/tmptm88ttu4fleche/beta-data.csv\n", + "warm location: /tmp/tmp822dn7defleche/beta-data.csv\n", "warm sibling exists: False\n" ] } @@ -273,11 +285,15 @@ "id": "8fcb480d", "metadata": {}, "source": [ - "## Edge 2: aliasing is not preserved\n", + "## Beware: aliasing is not part of a value\n", "\n", "Return the *same* path twice and the cold result holds one object in two slots.\n", "The warm hit mends each slot independently: two separate materializations, in two\n", - "different temp directories. Equal content, unequal (and non-identical) paths." + "different temp directories. Equal content, unequal (and non-identical) paths.\n", + "\n", + "Object identity is a property of one process's memory, not of the value being\n", + "cached — nothing about \"these two files are the same object\" survives a round\n", + "trip through storage, and nothing could. Compare content, never `is`.\n" ] }, { @@ -286,10 +302,10 @@ "id": "67496526", "metadata": { "execution": { - "iopub.execute_input": "2026-07-25T19:23:12.246344Z", - "iopub.status.busy": "2026-07-25T19:23:12.246201Z", - "iopub.status.idle": "2026-07-25T19:23:12.251214Z", - "shell.execute_reply": "2026-07-25T19:23:12.250671Z" + "iopub.execute_input": "2026-08-06T19:58:51.281270Z", + "iopub.status.busy": "2026-08-06T19:58:51.281055Z", + "iopub.status.idle": "2026-08-06T19:58:51.291226Z", + "shell.execute_reply": "2026-08-06T19:58:51.289901Z" } }, "outputs": [ @@ -323,7 +339,7 @@ "id": "78097da1", "metadata": {}, "source": [ - "## Edge 3: paths as dict *keys* mend into different keys\n", + "## Caveat: paths as dict *keys* mend into different keys\n", "\n", "Dict keys are destructured like values. A `Path` key comes back as a `TempPath`\n", "at a new location — so the mended dict has a *different key* than the original,\n", @@ -337,10 +353,10 @@ "id": "77ba67df", "metadata": { "execution": { - "iopub.execute_input": "2026-07-25T19:23:12.252600Z", - "iopub.status.busy": "2026-07-25T19:23:12.252431Z", - "iopub.status.idle": "2026-07-25T19:23:12.258461Z", - "shell.execute_reply": "2026-07-25T19:23:12.257826Z" + "iopub.execute_input": "2026-08-06T19:58:51.293639Z", + "iopub.status.busy": "2026-08-06T19:58:51.293387Z", + "iopub.status.idle": "2026-08-06T19:58:51.302382Z", + "shell.execute_reply": "2026-08-06T19:58:51.300962Z" } }, "outputs": [ @@ -349,8 +365,8 @@ "output_type": "stream", "text": [ " [index] running: delta\n", - "cold key: /tmp/claude-1000/tmpddupkllp-fleche-nested/delta-k.txt\n", - "warm key: /tmp/claude-1000/tmpxk4r0zg6fleche/delta-k.txt\n", + "cold key: /tmp/tmp4rik9ggl-fleche-nested/delta-k.txt\n", + "warm key: /tmp/tmpbt0e_qsqfleche/delta-k.txt\n", "kw[orig_key] works: False\n" ] } @@ -376,12 +392,18 @@ "id": "f5ea33da", "metadata": {}, "source": [ - "## Edge 4: every hit materializes a fresh copy, with temp-file lifetime\n", + "## Beware: each hit is its own copy, with temp-file lifetime\n", "\n", "Each hit copies the stored bytes into a new temporary directory (large files: mind\n", "the churn). The temp tree lives exactly as long as some `TempPath` derived from\n", "it is referenced — keep only a `str` of the location and the file is gone once the\n", - "path object is collected." + "path object is collected.\n", + "\n", + "Again this follows from the model rather than working against it: the cache owns\n", + "the content, and hands you a copy to use. It cannot know when you are finished\n", + "with that copy except by watching the reference you were given, so hold the\n", + "`Path` object for as long as you need the file, and `shutil.copy` it out if you\n", + "need it at a location of your own.\n" ] }, { @@ -390,10 +412,10 @@ "id": "88ba415b", "metadata": { "execution": { - "iopub.execute_input": "2026-07-25T19:23:12.259737Z", - "iopub.status.busy": "2026-07-25T19:23:12.259570Z", - "iopub.status.idle": "2026-07-25T19:23:12.314448Z", - "shell.execute_reply": "2026-07-25T19:23:12.313281Z" + "iopub.execute_input": "2026-08-06T19:58:51.304989Z", + "iopub.status.busy": "2026-08-06T19:58:51.304725Z", + "iopub.status.idle": "2026-08-06T19:58:51.389293Z", + "shell.execute_reply": "2026-08-06T19:58:51.388102Z" } }, "outputs": [ @@ -431,7 +453,7 @@ "id": "d8593d07", "metadata": {}, "source": [ - "## Edge 5: paths hidden in *opaque* containers are stored by location, not content\n", + "## Caveat: paths hidden in *opaque* containers are stored by location, not content\n", "\n", "Destructuring only recurses into what it knows: lists, tuples, dicts, dataclasses,\n", "attrs classes. Everything else — namedtuples (deliberately treated as opaque),\n", @@ -451,10 +473,10 @@ "id": "6e5db0b0", "metadata": { "execution": { - "iopub.execute_input": "2026-07-25T19:23:12.316163Z", - "iopub.status.busy": "2026-07-25T19:23:12.316065Z", - "iopub.status.idle": "2026-07-25T19:23:12.319779Z", - "shell.execute_reply": "2026-07-25T19:23:12.319316Z" + "iopub.execute_input": "2026-08-06T19:58:51.391523Z", + "iopub.status.busy": "2026-08-06T19:58:51.391305Z", + "iopub.status.idle": "2026-08-06T19:58:51.399526Z", + "shell.execute_reply": "2026-08-06T19:58:51.398213Z" } }, "outputs": [ @@ -463,7 +485,7 @@ "output_type": "stream", "text": [ " [bundle] running: zeta\n", - "warm type: PosixPath -> /tmp/claude-1000/tmpddupkllp-fleche-nested/zeta-nt.txt\n", + "warm type: PosixPath -> /tmp/tmp4rik9ggl-fleche-nested/zeta-nt.txt\n", "points at the ORIGINAL location: True\n" ] } @@ -488,10 +510,10 @@ "id": "5c4884f9", "metadata": { "execution": { - "iopub.execute_input": "2026-07-25T19:23:12.323854Z", - "iopub.status.busy": "2026-07-25T19:23:12.323678Z", - "iopub.status.idle": "2026-07-25T19:23:12.326503Z", - "shell.execute_reply": "2026-07-25T19:23:12.326114Z" + "iopub.execute_input": "2026-08-06T19:58:51.401931Z", + "iopub.status.busy": "2026-08-06T19:58:51.401678Z", + "iopub.status.idle": "2026-08-06T19:58:51.406926Z", + "shell.execute_reply": "2026-08-06T19:58:51.405586Z" } }, "outputs": [ @@ -499,7 +521,7 @@ "name": "stdout", "output_type": "stream", "text": [ - "hit returns: /tmp/claude-1000/tmpddupkllp-fleche-nested/zeta-nt.txt\n", + "hit returns: /tmp/tmp4rik9ggl-fleche-nested/zeta-nt.txt\n", "exists: False <- dangling, no warning\n" ] } @@ -531,7 +553,7 @@ "id": "ab85f64e", "metadata": {}, "source": [ - "## Edge 6: container subclasses are opaque — deliberately\n", + "## Caveat: container subclasses are opaque — deliberately\n", "\n", "Mending rebuilds containers via `type(value)()`, a contract subclasses\n", "may repurpose: `defaultdict`'s first argument is a factory (would crash),\n", @@ -550,10 +572,10 @@ "id": "d19c337b", "metadata": { "execution": { - "iopub.execute_input": "2026-07-25T19:23:12.328117Z", - "iopub.status.busy": "2026-07-25T19:23:12.328034Z", - "iopub.status.idle": "2026-07-25T19:23:12.330973Z", - "shell.execute_reply": "2026-07-25T19:23:12.330638Z" + "iopub.execute_input": "2026-08-06T19:58:51.409279Z", + "iopub.status.busy": "2026-08-06T19:58:51.409056Z", + "iopub.status.idle": "2026-08-06T19:58:51.417030Z", + "shell.execute_reply": "2026-08-06T19:58:51.415673Z" } }, "outputs": [ @@ -562,7 +584,7 @@ "output_type": "stream", "text": [ " [by_kind] running: eta\n", - "warm: defaultdict {'files': [PosixPath('/tmp/claude-1000/tmpddupkllp-fleche-nested/eta-dd.txt')]}\n", + "warm: defaultdict {'files': [PosixPath('/tmp/tmp4rik9ggl-fleche-nested/eta-dd.txt')]}\n", "but the nested path is the ORIGINAL location: True\n" ] } @@ -587,10 +609,10 @@ "id": "ef2d0d13", "metadata": { "execution": { - "iopub.execute_input": "2026-07-25T19:23:12.332260Z", - "iopub.status.busy": "2026-07-25T19:23:12.332187Z", - "iopub.status.idle": "2026-07-25T19:23:12.335016Z", - "shell.execute_reply": "2026-07-25T19:23:12.334601Z" + "iopub.execute_input": "2026-08-06T19:58:51.419207Z", + "iopub.status.busy": "2026-08-06T19:58:51.419003Z", + "iopub.status.idle": "2026-08-06T19:58:51.425975Z", + "shell.execute_reply": "2026-08-06T19:58:51.424622Z" } }, "outputs": [ @@ -605,7 +627,7 @@ "\n", "register_destructurer(pred: Callable[[Any], bool], fn: Callable) -> None\n", " Register a custom container destructurer.\n", - "\n", + " \n", " *pred(value)* should return ``True`` for values this destructurer handles.\n", " *fn* must accept ``(intern, value)`` where *intern* is\n", " :meth:`DestructuringMixin._intern_rec`. Entries are appended after the\n", @@ -643,17 +665,18 @@ " values, or nested anywhere inside dicts / lists / tuples / dataclasses / attrs\n", " classes — arbitrarily deep. Content-addressed, dedup'd, survives deletion of\n", " the originals.\n", - "- **A hit is a copy, not a replay:** returned paths live in fresh temp\n", - " directories. Don't resolve siblings, don't compare locations, don't expect\n", - " aliasing, and keep a reference to the `Path` object for as long as you need the\n", - " file.\n", - "- **Don't key dicts by `Path`** if you'll look them up afterwards — keys mend\n", - " into new locations. Use `str(path)` or a stable ID.\n", - "- **Don't hide paths in opaque containers** (namedtuples, sets, plain classes,\n", - " and any container *subclass* — only exact `dict` / `OrderedDict` / `list` /\n", - " `tuple` are destructured): they are stored by location and come back stale or\n", - " dangling after the original moves on. `register_destructurer` is the opt-in\n", - " door for well-behaved custom containers." + "- **Beware — a hit is a copy, because a value is all that is cached:** returned\n", + " paths live in fresh temp directories. Don't resolve siblings, don't compare\n", + " locations, don't expect aliasing, and keep a reference to the `Path` object for\n", + " as long as you need the file. None of this is pending a fix; it is what\n", + " caching by content means.\n", + "- **Caveat — don't key dicts by `Path`** if you'll look them up afterwards — keys\n", + " mend into new locations. Use `str(path)` or a stable ID.\n", + "- **Caveat — don't hide paths in opaque containers** (namedtuples, sets, plain\n", + " classes, and any container *subclass* — only exact `dict` / `OrderedDict` /\n", + " `list` / `tuple` are destructured): they are stored by location and come back\n", + " stale or dangling after the original moves on. `register_destructurer` is the\n", + " opt-in door for well-behaved custom containers.\n" ] } ], @@ -673,7 +696,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.13.11" + "version": "3.11.15" } }, "nbformat": 4, diff --git a/src/fleche/digest.py b/src/fleche/digest.py index f4a36537..fc7dcf6e 100644 --- a/src/fleche/digest.py +++ b/src/fleche/digest.py @@ -6,6 +6,7 @@ import numbers from numbers import Number import struct +import subprocess from pathlib import Path import types import importlib.metadata @@ -345,6 +346,12 @@ def _digest_bytes(value: Any) -> bytes: m.update(_digest_bytes(value.__func__)) case property(): m.update(_digest_bytes((value.fget, value.fset, value.fdel))) + case subprocess.CompletedProcess(): + m.update( + _digest_bytes( + (value.args, value.returncode, value.stdout, value.stderr) + ) + ) case _ if isinstance(value, type) and value.__module__ == 'builtins': # Digest a built-in type (int, str, list, …) by its qualified name. # Restricted to the builtins module; user-defined types remain Indigestible. diff --git a/tests/integration/test_notebooks.py b/tests/integration/test_notebooks.py index e08dd491..6a58daeb 100644 --- a/tests/integration/test_notebooks.py +++ b/tests/integration/test_notebooks.py @@ -10,6 +10,7 @@ "SecureStorage.ipynb", "CacheStack.ipynb", "Files.ipynb", + "PathsInContainers.ipynb", ] diff --git a/tests/unit/digest/test_digest.py b/tests/unit/digest/test_digest.py index 4de054d2..2b80fc76 100644 --- a/tests/unit/digest/test_digest.py +++ b/tests/unit/digest/test_digest.py @@ -1,6 +1,8 @@ import cmath import datetime import struct +import subprocess +import sys import collections import collections.abc import types as types_module @@ -945,3 +947,65 @@ def test_non_builtin_type_raises_indigestible(t): """Non-builtin type objects (user-defined classes, dataclass classes) raise Indigestible.""" with pytest.raises(Indigestible): digest(t) + + +# --------------------------------------------------------------------------- +# subprocess.CompletedProcess +# --------------------------------------------------------------------------- + + +def _cp(args=("echo", "hi"), returncode=0, stdout=b"hi\n", stderr=b""): + return subprocess.CompletedProcess( + args=list(args), returncode=returncode, stdout=stdout, stderr=stderr + ) + + +def test_completedprocess_hashes_by_its_fields(): + """Wrapping shell tools is a first-class use case; `run()`'s result must hash.""" + assert digest(_cp()) == digest(_cp()) + + +@pytest.mark.parametrize( + "changed", + [ + pytest.param({"args": ("echo", "bye")}, id="args"), + pytest.param({"returncode": 1}, id="returncode"), + pytest.param({"stdout": b"other\n"}, id="stdout"), + pytest.param({"stderr": b"boom\n"}, id="stderr"), + ], +) +def test_completedprocess_distinguishes_each_field(changed): + assert digest(_cp()) != digest(_cp(**changed)) + + +def test_completedprocess_is_not_digest_equal_to_its_field_tuple(): + """The type name salts the hash, as for every other arm.""" + cp = _cp() + assert digest(cp) != digest((cp.args, cp.returncode, cp.stdout, cp.stderr)) + + +@pytest.mark.parametrize( + "kwargs", + [ + pytest.param({"stdout": "hi\n", "stderr": ""}, id="text-mode-str"), + pytest.param({"stdout": None, "stderr": None}, id="streams-not-captured"), + ], +) +def test_completedprocess_handles_uncaptured_and_text_streams(kwargs): + """`capture_output=False` leaves None; `text=True` leaves str.""" + assert digest(_cp(**kwargs)) == digest(_cp(**kwargs)) + + +def test_completedprocess_from_a_real_run(): + a = subprocess.run([sys.executable, "-c", "print('x')"], capture_output=True) + b = subprocess.run([sys.executable, "-c", "print('x')"], capture_output=True) + c = subprocess.run([sys.executable, "-c", "print('y')"], capture_output=True) + assert digest(a) == digest(b) + assert digest(a) != digest(c) + + +def test_completedprocess_nested_in_a_result(): + """The notebook's shape: a function returning (path-ish, CompletedProcess).""" + cp = _cp() + assert digest(("workdir", cp)) == digest(("workdir", _cp())) + assert digest({"ret": cp}) != digest({"ret": _cp(returncode=2)})