docs: split the purity contract out of file semantics; digest CompletedProcess

agents/DEVELOPING.md

@@ -174,8 +174,8 @@ Shared fixtures (in `fixtures.py`):
 
 - `benchmarks/` — `benchmark_{digest,integration,storage}.py`, `run_benchmarks.py`, `compare_results.py` (diff two `results.csv` runs), `utils.py`, `profile_digest_types.py` (per-type cProfile harness), `results.csv`.
 - `devnotes/storage-hierarchy.{dot,md,svg}` — rendered inheritance diagram for the storage classes.
-- `docs/` — Sphinx sources, grouped by topic: root holds `index`, `installation`, `parallel_execution`; `usage/` holds `tldr`, `helpers`, `lazy_call`, `query` (`tldr` first, PR #745); `digests/` holds `digests_as_args`, `digest_equivalence`, `entry_points` (added by PR #752 as the dedicated fleche-ase / third-party plug-in page); `storage/` holds `configuration`, `cache_stack`, `security`; `dev/` holds `custom_digests`, `developer`, `ssh_cache`. `docs/notebooks/` is **symlinks** into `../../notebooks/` (six entries — `Caches` and `TransferWorkflow` are not exposed in docs); the `rendernb.yml` workflow re-executes `notebooks/*.ipynb` in place when a PR carries the `rendernb` label.
-- `notebooks/` — usage examples (`GettingStarted`, `Caches`, `CacheStack`, `StorageBackends`, `SecureStorage`, `ConcurrentExecution`, `ExtraMethods`, `TransferWorkflow`); five of these (all except `Caches`, `ConcurrentExecution`, `TransferWorkflow`) are executed by `tests/integration/test_notebooks.py`.
+- `docs/` — Sphinx sources, grouped by topic: root holds `index`, `installation`, `parallel_execution`; `usage/` holds `tldr`, `purity`, `helpers`, `file_semantics`, `lazy_call`, `query` (`tldr` first, PR #745; `purity` carries the general "fleche caches pure functions" contract — arguments keyed as passed, mutation and other side effects not replayed — that `file_semantics` states for paths); `digests/` holds `digests_as_args`, `digest_equivalence`, `entry_points` (added by PR #752 as the dedicated fleche-ase / third-party plug-in page); `storage/` holds `configuration`, `cache_stack`, `security`; `dev/` holds `custom_digests`, `developer`, `path_storage`, `ssh_cache`; `recipes/` holds `files_and_paths`. `docs/notebooks/` is **symlinks** into `../../notebooks/` (eight entries — `Caches` and `TransferWorkflow` are not exposed in docs); the `rendernb.yml` workflow re-executes `notebooks/*.ipynb` in place when a PR carries the `rendernb` label.
+- `notebooks/` — usage examples (`GettingStarted`, `Caches`, `CacheStack`, `StorageBackends`, `SecureStorage`, `ConcurrentExecution`, `ExtraMethods`, `Files`, `PathsInContainers`, `TransferWorkflow`); seven of these (all except `Caches`, `ConcurrentExecution`, `TransferWorkflow`) are executed by `tests/integration/test_notebooks.py`. Committed **with outputs** — `rendernb.yml` re-executes them in place on a PR labelled `rendernb`, so an edited notebook should be re-run before it lands.
 - `.github/workflows/` — CI: `tests.yml` (PR sweep across 3.11–3.14 + `sql-backends` job that boots Postgres 16 + MariaDB 11 service containers and sets `FLECHE_TEST_{POSTGRES,MYSQL}_URL`), `ty.yml`, `test-minimum-deps.yml` (PR #645, installs every direct dep at its declared floor via `uv pip install --resolution lowest-direct -e ".[tests]"` and runs the suite — `uv pip install` rather than `uv sync` because the universal lock floats deps up through cross-extra constraints and would mask the real minimums), `benchmarks.yml`/`benchmarks-main.yml`/`updatebenchmarks.yml`, `perf-triage.yml` (PR #628 — Haiku reads the PR diff/description and adds the `benchmark` label when the change touches a hot path — that label is the existing `benchmarks.yml` trigger), `rendernb.yml` (re-executes `notebooks/*.ipynb` on PRs labelled `rendernb`), `release-please.yml`, `pypi-publish.yml` (trusted-publisher upload triggered by `release: published`), `claude.yaml` + `claude_ci_details.yaml` (the latter exposes CI status as a tool the in-PR Claude can call). Releases use **release-please** (`release-please-config.json`, `.release-please-manifest.json`) — release PRs are opened automatically from conventional-commit history on `main`.
 
 ## Design themes / open scope (issue tracker)
@@ -217,6 +217,7 @@ Cheat sheet of what's been considered. Issue numbers are the entry points — fe
 - `CachePool(ReadOnlyMixin, _MultiCache)` — a read-only unordered collection of caches (PR #689). Extracted the `CacheStack` read fan-out into a shared `_MultiCache(BaseCache)` base (`_first_hit`/`_collect`/`_foreach` + `contains`/`load_value`/`expand`/`_shrink`/`_query`) so `CacheStack` and `CachePool` share one implementation. `save`/`evict` raise `Rejected`; config dispatches `{"pool": [...]}` alongside the existing list→`CacheStack` and `type="ssh"`→`SshCache` paths. Pinned in `tests/unit/caches/test_cache_pool.py`.
 - Cache-level race fixes on top of #569: #217 → PR #629, #451 → PR #631, #452 → PR #630, #485 → PR #627. Regression-pinned in `tests/regression/test_issue_{217,451,452,485}.py`; concurrency stress tests share `run_workers()` from `tests/fixtures.py` (PR #633).
 - Digest dispatch grew a `builtins`-type arm and a `not isinstance(value, type)` guard on the dataclass/attrs arms so `digest(int)` / `digest(SomeDataclass)` (the class object) stop raising — PR #651, closes #469.
+- `digest()` grew a `subprocess.CompletedProcess` arm (args + returncode + stdout + stderr) so functions wrapping command-line tools hash without a user hook — `run()`'s result is neither a dataclass nor iterable, so it was `Indigestible` before. Purely additive (those values raised previously), so no `hash_version` bump. Pinned in `tests/unit/digest/test_digest.py::test_completedprocess_*`; `notebooks/Files.ipynb` used to register this by hand via `add_hook` and no longer does.
 - `digest()` grew explicit `pd.DataFrame` / `pd.Series` / `pd.Index` arms so pandas inputs hash by content (columns/name + dtype + index + `hash_pandas_object`), not by the column names yielded by `iter()` — PR #675. Silent change with no `hash_version` bump (previous pandas cache entries are now unreachable). Pinned in `tests/unit/digest/test_digest.py::test_pandas_{dataframe,series,index}_hashes_by_content`.
 - `SizeLimitedMixin` swapped its raw `threading.RLock` for `_PicklableRLock` so `SizeLimitedCache` survives `pickle.dumps` (PR #664, closes #663).
 - `MetaData.keys` is uniformly a `@property` across every built-in (`Runtime`/`Environment`/`Git`/`Tags`) instead of the prior mix of class-level dict attributes for the zero-arg three plus a property for `Tags` (PR #690, closes #637). The abstract-property-overridden-by-class-attribute smell on the ABC is gone. PR #763 (closes #738) then single-sourced the static-schema built-ins by moving the shared schema to a `_keys: ClassVar[dict[str, type]]` class attribute on the ABC (default `{}`) with `MetaData.keys` a concrete property returning `self._keys`; `Runtime`/`Environment`/`Git` now declare `_keys` once (dropping their per-class `keys` override), while `Tags` keeps overriding `keys` directly since its schema is per-instance. Drift guard: `tests/unit/metadata/test_metadata.py::test_builtin_metadata_pre_post_keys_match_schema` asserts `set(pre_output) | set(post_output) == set(cls._keys)` for each of the three static built-ins. `name` is still set by `@configurable` (`Runtime`/`Environment`/`Git`) or as a manual dataclass attribute (`Tags`) — unchanged.

docs/index.rst

@@ -31,6 +31,7 @@ Welcome to the **Fleche** library documentation.
    :caption: Using Fleche
 
    usage/tldr
+   usage/purity
    usage/helpers
    usage/file_semantics
    usage/lazy_call
@@ -84,6 +85,7 @@ Welcome to the **Fleche** library documentation.
    notebooks/CacheStack
    notebooks/ConcurrentExecution
    notebooks/Files
+   notebooks/PathsInContainers
 
 .. toctree::
    :maxdepth: 2

docs/notebooks/PathsInContainers.ipynb

@@ -0,0 +1 @@
+../../notebooks/PathsInContainers.ipynb

docs/usage/file_semantics.rst

@@ -44,19 +44,20 @@ functions returning identical files share one stored body (see
 Argument mutation
 ~~~~~~~~~~~~~~~~~
 
-A call is keyed on its arguments **as passed**: argument content is captured
-*before* the function body runs.  A function that mutates its own argument —
-most commonly, writing an output file *into* a directory it received — is
-still recorded under the pre-call content, so honest repeat calls hit.  The
-mutation itself, however, is neither recorded nor replayed: a cache hit
-leaves the argument untouched, so a side effect on the input happens on cold
-calls only.
-
-fleche caches *pure* functions.  What a function does to its arguments
-without passing it back out is invisible to the cache — treat received paths
-as read-only and write outputs to a fresh directory (``tempfile.mkdtemp``).
-A mutated argument that *is* returned is captured faithfully in its final,
-post-mutation state: if the mutation is the point, return it.
+Arguments are keyed **as passed** — captured before the body runs — and a
+mutation the body performs on one is neither recorded nor replayed.  That is
+a general rule (:ref:`argument-mutation`); its most common instance here is a
+function that writes an output file *into* a directory it received.  The call
+is recorded under the directory's pre-call tree, so honest repeat calls hit,
+but the written file does not reappear on a hit — it exists only on cold
+calls.
+
+Returning what you wrote to is the normal shape, not a workaround for this: a
+function that writes into a directory and hands that directory back is recorded
+with it in its final, post-mutation state, and that is the intended way to
+produce files.  What is not replayed is a write to something you never return —
+so a path that is pure *input* should be treated as read-only, with new files
+written somewhere you do return (``tempfile.mkdtemp``).
 
 Only *content* changes count as mutation: permissions are not part of
 identity (see :ref:`fidelity-limits`), so a ``chmod`` on a received path is
@@ -146,13 +147,30 @@ the file.  If you need the file at a stable location, copy it out:
 Paths nested inside containers
 ------------------------------
 
-Paths are found and content-stored inside the containers fleche takes apart:
-``dict``, ``OrderedDict``, ``list``, ``tuple`` (**exact types** — see below),
-``dataclasses`` and ``attrs`` classes — nested to any depth, as values *or* as
-dict keys.  Everything above about identity, materialization, and lifetime
-applies to each nested path individually.  Container structure is otherwise
-faithful on a hit: lists and tuples keep their element order, and a mended
-dict keeps the insertion order the stored value had.
+Whether a nested path is stored by content comes down to one thing:
+**destructuring** — whether storage takes the surrounding container apart into
+independently-stored children, or pickles it whole as one opaque value.  Only
+children reach the path machinery, so the list of containers fleche
+destructures *is* the list of places a nested path gets content treatment.
+That list is
+:data:`~fleche.storage.destructuring._DESTRUCTURERS`: ``dict``,
+``OrderedDict``, ``list``, ``tuple`` (**exact types** — see below),
+``dataclasses`` and ``attrs`` classes.  See :ref:`extending-destructurer` for
+the mechanism and how to add your own container to it.
+
+Within those, paths are found nested to any depth, as values *or* as dict
+keys, and everything above about identity, materialization, and lifetime
+applies to each one individually.  Container structure is otherwise faithful
+on a hit: lists and tuples keep their element order, and a mended dict keeps
+the insertion order the stored value had.
+
+"Any depth" is not a figure of speech, and no storage setting narrows it.  A
+``Path`` matches no destructurer, so it is always written out as its own stored
+entry rather than inlined into the container above it — and being written out is
+exactly what hands it to the content machinery.  The
+``remaining_depth`` knob only decides how eagerly *destructurable* nodes are
+split into separate entries, so it cannot put a path out of reach however it is
+set.
 
 Three caveats specific to nesting:
 
@@ -175,16 +193,18 @@ Opaque containers store paths by *location*
 -------------------------------------------
 
 A path inside an opaque value (a ``namedtuple``, a ``set``, an arbitrary
-object) never reaches the content machinery.  The call is still *keyed*
+object) is never destructured out of it, and so never reaches the content
+machinery.  The call is still *keyed*
 correctly — the digest layer does look inside — but what is stored is the path
 object itself, pointing at wherever the file was when it was saved.  A hit
 returns that original *location* (whether as the same object or an equal copy
 is backend-dependent — rely on neither): if the file has since been deleted or
 edited, the hit hands you a dangling or stale path, **without any warning**.
 
 Rule of thumb: return paths in plain dicts / lists / tuples / dataclasses.  If
-you need a custom container to participate, see
-:func:`~fleche.storage.destructuring.register_destructurer`.
+you need a custom container to participate, register a destructurer for it
+with :func:`~fleche.storage.destructuring.register_destructurer` — see
+:ref:`extending-destructurer`.
 
 .. _file-dedup:
 

docs/usage/helpers.rst

@@ -38,7 +38,7 @@ Attempts to load the result of a specific call from the cache. If the result is
 Returns ``True`` if the result for the given call is already present in the cache, ``False`` otherwise.
 
 ``.query(*args, metadata={}, **kwargs)``
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 Returns matching cached calls from the active cache. Any argument passed as ``None`` acts as a wildcard, matching any stored value for that parameter. The ``metadata`` keyword argument accepts a dictionary of metadata tags to further filter results (e.g., ``metadata={"tags": {"project": "alpha"}}``).
 
@@ -102,7 +102,9 @@ Functions Returning ``None``
 Functions that return ``None`` are **never cached**.  When a decorated function
 returns ``None``, ``fleche`` logs a ``WARNING`` and skips the save step
 entirely.  Subsequent calls will execute the function again rather than
-returning a cached value.
+returning a cached value — including any side effects they have, which for a
+cached function would otherwise happen on the cold call only (see
+:doc:`purity`).
 
 This applies to all code paths, with one difference for ``.rerun()``:
 

docs/usage/purity.rst

@@ -0,0 +1,101 @@
+.. _purity:
+
+Purity and Side Effects
+=======================
+
+What fleche assumes about the functions you decorate, and what therefore does
+*not* survive a cache hit.  The rules here apply to every argument and result
+type; :doc:`file_semantics` is the same contract seen through ``Path`` values.
+
+The assumption
+--------------
+
+fleche caches **pure** functions: the result is determined by the arguments,
+and anything else the body does is incidental.  A cache hit replays the
+*result* and nothing else — so every effect a function has besides returning
+a value happens on cold calls only.
+
+That is not a restriction fleche can check, and it does not raise if you break
+it.  It shows up as a function that behaves differently the second time you
+call it.
+
+.. _argument-mutation:
+
+Arguments are keyed as passed
+-----------------------------
+
+Argument content is captured **before** the body runs, so a function that
+mutates its own argument is still recorded under the pre-call content and
+honest repeat calls hit.  The mutation itself is neither recorded nor
+replayed:
+
+.. code-block:: python
+
+   @fleche
+   def append_and_report(xs, n):
+       xs.append(n)
+       return len(xs)
+
+   a = [1, 2]
+   append_and_report(a, 9)   # 3 — body runs, and a is now [1, 2, 9]
+
+   b = [1, 2]
+   append_and_report(b, 9)   # 3 — cache hit; b is still [1, 2]
+
+Both calls return ``3``, because that is the recorded result.  Only the first
+one changed its argument.  Nothing about this is specific to lists: a
+``dict``, a numpy array, and a directory a function writes into all behave the
+same way.
+
+Return what you mutate
+~~~~~~~~~~~~~~~~~~~~~~
+
+An argument that is mutated **and returned** is captured faithfully in its
+final, post-mutation state — the result is stored after the body runs:
+
+.. code-block:: python
+
+   @fleche
+   def append_and_return(xs, n):
+       xs.append(n)
+       return xs
+
+   append_and_return([1, 2], 9)   # [1, 2, 9] — body runs
+   append_and_return([1, 2], 9)   # [1, 2, 9] — cache hit, same value
+
+If the mutation is the point of the function, return it.  Treat arguments you
+receive as read-only otherwise, and build results fresh.
+
+Other side effects
+------------------
+
+Everything else a body does — printing, logging, writing files outside the
+returned value, sending a request, inserting a row — happens on the cold call
+and never again:
+
+.. code-block:: python
+
+   @fleche
+   def record(x):
+       db.insert(x)        # runs once, ever
+       return x * 2
+
+If an effect must happen on every call, it belongs outside the cached
+function; keep the cached part to the computation whose result you want
+stored.
+
+Two related cases
+-----------------
+
+* A function returning ``None`` is never cached, so it re-executes every time
+  — including its side effects.  See :ref:`none-not-cached`.
+* Paths follow all of the above, with the mutation case made concrete:
+  a directory a function receives and writes into is recorded under its
+  pre-call tree, and the written file does not reappear on a hit.  See
+  :doc:`file_semantics`.
+
+See also
+--------
+
+* :doc:`file_semantics` — the same contract for files and directories.
+* :doc:`helpers` — ``.rerun()`` for forcing a cold call deliberately.

docs/usage/tldr.rst

@@ -60,8 +60,9 @@ directories are cached by **content** (not by path string), and a cache hit
 returns a freshly materialized copy under a temporary path.  See
 :doc:`file_semantics` for the exact contract.
 
-That is all you need for everyday use.  The rest of this section covers the
-helper methods in depth (:doc:`helpers`), the file/path contract
+That is all you need for everyday use.  The rest of this section covers what
+fleche assumes about the functions you decorate (:doc:`purity`), the helper
+methods in depth (:doc:`helpers`), the file/path contract
 (:doc:`file_semantics`), lazy loading of large cached objects
 (:doc:`lazy_call`), and querying stored calls (:doc:`query`); storage backends
 and configuration details live under :doc:`/storage/configuration`.