Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 41 additions & 0 deletions docs/dev/path_storage.rst
Original file line number Diff line number Diff line change
Expand Up @@ -93,6 +93,47 @@ name is treated as incidental (typically a temp dir). If a root name carries
meaning, name a *child* meaningfully instead, or wrap the tree's identity in your
own value.

Why paths stop at the SSH boundary
----------------------------------

:class:`~fleche.remote.SshCache` is the one cache where "store this path"
cannot mean what it means everywhere else, because the two sides do not share
a filesystem. Values cross the wire by cloudpickle, and a pickled ``Path`` is
only its string — so a path handed to the remote is a *name it resolves
itself*. Three things follow, and each of them is silent:

* ``save_value`` on the remote runs :class:`PathValueMixin` against the
**server's** disk. If a file happens to sit at that name it is stored, under
the digest of *those* bytes; ``digest(path) == values.save(path)`` — the
invariant the seal in :meth:`~fleche.caches.BaseCache.prepare` rests on —
is broken, and the record lands under a key no client ever recomputes.
* if nothing sits there, the server raises ``Indigestible`` from the middle of
an RPC.
* ``load_value`` materializes into a temp directory on the **server** and can
only ship the name back. The :class:`TempPath` guard does not survive the
hop either — ``PurePath.__reduce__`` reconstructs from ``parts`` alone, so
the ``_temp_root`` attribute is dropped and the class-level ``_live_roots``
registry is per-process — so the server frees the tree as soon as its own
reference dies, and the client is left with a dangling name for a filesystem
it cannot see.

So :class:`~fleche.remote.SshCache` refuses instead, via
:func:`~fleche.storage.paths.find_path` (which walks a value exactly the way a
destructuring save does, using
:func:`~fleche.storage.destructuring.child_slots`) and
:class:`~fleche.remote.RemotePathUnsupported`. That exception subclasses
:class:`~fleche.storage.SaveError` so the existing two-phase-save degradations
carry it: a path argument becomes a digest-only reference **whose digest was
computed locally**, which keeps the seal intact and lookups correct, and a path
result becomes :class:`~fleche.caches.Rejected`, so the call runs uncached
rather than wrongly cached. See :ref:`file-remote-caches` for the user-facing
version.

Making paths genuinely work over SSH is a separate feature: it needs the
*content* on the wire (client-side blob conversion before the call, plus a way
to fetch a stored record without the server materializing it first), which is a
change to the RPC surface rather than to this module.

See also
--------

Expand Down
18 changes: 18 additions & 0 deletions docs/recipes/files_and_paths.rst
Original file line number Diff line number Diff line change
Expand Up @@ -78,3 +78,21 @@ their names.
directory's own ``.name`` is a hash (its children's names are faithful).
Don't rely on the top-level directory name surviving a cache hit; do rely on
everything inside it.

Sharing files with a remote (SSH) cache
---------------------------------------

Return ``bytes``. A :class:`~pathlib.Path` cannot cross a
:class:`~fleche.remote.SshCache` — only the path string would travel, and the
remote would resolve it against its own filesystem — so fleche refuses it and
runs the call uncached. Content ships and deduplicates normally:

.. code-block:: python

@fleche
def render(text) -> bytes: # not `-> Path`
return _render_to_pdf(text)

If you want path semantics *and* a remote cache, put a local layer in front of
it: saves land in the local layer, so paths never reach the wire. See
:ref:`file-remote-caches` for the full contract.
36 changes: 36 additions & 0 deletions docs/usage/file_semantics.rst
Original file line number Diff line number Diff line change
Expand Up @@ -197,6 +197,41 @@ and re-caching adds a tiny name record, never a second copy of the bytes. If
you want *keys* (not just storage) to ignore the name too, return ``bytes``
instead of a ``Path``; that is the content-only escape hatch.

.. _file-remote-caches:

Paths stop at a remote (SSH) cache
----------------------------------

Everything above assumes the cache and your process see the **same
filesystem**. A :class:`~fleche.remote.SshCache` does not: it forwards each
operation to a fleche running on another machine, and values travel by
cloudpickle — which reduces a ``Path`` to its *string*. Only the name would
arrive, and the remote would resolve it against its own filesystem, storing
whatever happens to sit there (or nothing) under a digest that no longer
matches yours.

Rather than store something else under your key, fleche refuses:
:class:`~fleche.remote.RemotePathUnsupported` is raised when a path — bare or
nested in a container — would cross the wire. Because it is a
:class:`~fleche.storage.SaveError`, the usual degradations apply and your code
does not have to catch anything:

* a path **argument** falls back to a digest-only reference. The call is still
keyed correctly (the digest is computed *here*, from your file), so lookups
hit and miss exactly as they should; only the file's bytes are not retrievable
from the remote record.
* a path **result** is rejected: the call runs, returns your file, and is
logged as not cached.
* **loading** a path stored on the remote raises too. The remote materializes
into a temp directory on *its* disk and can only send back the name, which
means nothing here. This is lazy — a record whose result is a path can still
be loaded and queried; only touching the path value raises.

To share file content across machines, return the file's ``bytes``: they
travel and deduplicate normally. To keep full path semantics, put a local
cache layer in front of the remote one (see :doc:`/notebooks/CacheStack`) — the
local layer is where saves land, so paths never reach the wire.

Quick reference
---------------

Expand All @@ -212,6 +247,7 @@ Return same path twice Two independent copies
Return path inside namedtuple/set/custom class The *original* location (may be stale)
Pass a nonexistent ``Path`` argument No caching: warns, always executes
Write into a directory you received Hit (keyed as passed); write not replayed
Return a ``Path`` through an ``SshCache`` Refused: runs, warns, not cached
=============================================== ======================================

See also
Expand Down
77 changes: 73 additions & 4 deletions src/fleche/remote.py
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,8 @@
from .caches import BaseCache, Rejected
from .call import DigestedCall, LazyCall, QueryCall
from .digest import Digest
from .storage.base import SaveError
from .storage.paths import find_path

logger = logging.getLogger("fleche.remote")

Expand Down Expand Up @@ -346,6 +348,31 @@ class RemoteConnectionError(RuntimeError):
"""Raised when the SSH subprocess cannot be reached or has died."""


class RemotePathUnsupported(SaveError):
"""Raised when a :class:`~pathlib.Path` would cross the SSH boundary.

Path values are stored **by content** (see
:class:`~fleche.storage.paths.PathValueMixin`), but the RPC ships values
by cloudpickle, and a pickled ``Path`` is just its *string*. Letting one
through would hand the server a name to resolve against its **own**
filesystem: it stores whatever happens to sit at that location — a
different file, or nothing — under a digest that no longer matches the
client's ``digest(path)``. The mismatch silently breaks the
``digest(x) == save_value(x)`` seal every lookup depends on, so the
record is filed under a key no client can ever reproduce.

Subclasses :class:`~fleche.storage.base.SaveError` so the standard
save-side degradations apply and nothing has to special-case it: a path
*argument* falls back to a digest-only reference (correct local digest,
lookups still hit, only the bytes are unavailable remotely), and a path
*result* turns into :class:`~fleche.caches.Rejected` — the call runs and
returns normally, it just is not cached. On the **load** side it
propagates to the caller instead: a path the server materialized into
its own temp directory is meaningless here, and the guard's whole point
is to say so rather than hand back a dangling name.
"""


def _warn_on_version_skew(info: dict[str, Any]) -> None:
"""Log a warning when the remote's fleche or cloudpickle version differs.

Expand Down Expand Up @@ -707,6 +734,13 @@ class SshCache(BaseCache):
ControlPersist in ``~/.ssh/config`` or via *ssh_options* to share the
underlying connection across multiple fleche runs.

**Filesystem paths do not cross this cache.** Values travel by
cloudpickle, so a :class:`~pathlib.Path` would arrive as a bare string for
the remote to resolve against its own disk; both directions raise
:class:`RemotePathUnsupported` instead. Ship the file's ``bytes``, or put
a local layer in front of the remote one — see that exception and
:ref:`file-remote-caches`.

Args:
host: SSH target, e.g. ``"user@host"`` or any alias from
``~/.ssh/config``.
Expand Down Expand Up @@ -785,20 +819,55 @@ def _rpc(self, name: str, *args: Any) -> Any:
raise Rejected(self, *args)
return spec.unwrap(self, self._conn.call(name, *args))

@staticmethod
def _reject_path(value: Any, what: str) -> None:
"""Raise :class:`RemotePathUnsupported` if *value* carries a ``Path``.

Scans the same object graph a destructuring save would walk, so a
path nested in a list, dict, dataclass, or ``attrs`` instance is
caught too — those reach the server's value storage exactly like a
bare one and fail the same way.
"""
path = find_path(value)
if path is not None:
raise RemotePathUnsupported(
f"{what} carries the filesystem path {str(path)!r}, which cannot "
"cross an SshCache: values travel by cloudpickle, so only the "
"path string would arrive and the remote would resolve it "
"against its own filesystem. Return the file's `bytes` instead "
"to store its content, or keep path-valued calls in a local "
"cache layer."
)

def save(self, call: DigestedCall | _call.Call) -> str:
# A DigestedCall carries only digests; the degenerate live-`Call`
# form still carries values, and the server would stash them itself.
if isinstance(call, _call.Call):
try:
self._reject_path(dict(call.arguments), "the call's arguments")
self._reject_path(call.result, "the call's result")
except RemotePathUnsupported as e:
raise Rejected(e) from None
return self._rpc("save", call)

def load(self, key: str) -> LazyCall:
return self._rpc("load", key)

def load_value(self, key: str) -> Any:
return self._rpc("load_value", key)
# The server materializes a stored path into a temp directory on
# *its* filesystem and can only ship us the name — which points
# nowhere here, and which the server unlinks as soon as its own
# reference dies. Refuse it rather than return a dangling path.
value = self._rpc("load_value", key)
self._reject_path(value, f"the value at {key}")
return value

def save_value(self, value: Any) -> Digest:
# One round trip per value (no batching yet). Values travel by
# cloudpickle, so a Path value ships its path *string*, not its
# content — the same limitation the previous ship-the-whole-Call
# protocol had; paths over SSH remain unsupported.
# cloudpickle, so a Path would ship its path *string*, not its
# content — see `RemotePathUnsupported` for why that is refused
# rather than silently stored against the server's filesystem.
self._reject_path(value, "the value")
return self._rpc("save_value", value)

def evict(self, key: str | Digest) -> None:
Expand Down
6 changes: 4 additions & 2 deletions src/fleche/storage/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,8 +16,8 @@
CallStorage,
CallMixin,
)
from .destructuring import DestructuringMixin, register_destructurer
from .paths import PathValueMixin, TempPath, FileBlob, DirectoryBlob
from .destructuring import DestructuringMixin, child_slots, register_destructurer
from .paths import PathValueMixin, TempPath, FileBlob, DirectoryBlob, find_path
from .memory import ValueMemory, CallMemory
from .void import ValueVoid, CallVoid
from .file import FileStorage
Expand All @@ -38,11 +38,13 @@
"CallStorage",
"CallMixin",
"DestructuringMixin",
"child_slots",
"register_destructurer",
"PathValueMixin",
"TempPath",
"FileBlob",
"DirectoryBlob",
"find_path",
"ValueMemory",
"CallMemory",
"ValueVoid",
Expand Down
30 changes: 30 additions & 0 deletions src/fleche/storage/destructuring.py
Original file line number Diff line number Diff line change
Expand Up @@ -243,6 +243,36 @@ def _field_items(value: Any) -> list[tuple[str, Any]]:
]


def child_slots(value: Any) -> list[tuple[Any, Any]] | None:
"""Enumerate the ``(label, child)`` pairs a save would recurse into.

The read-only half of :meth:`DestructuringMixin._intern_rec`'s dispatch:
same leaf rules (numbers / strings / bytes / digests / namedtuples are
opaque), same ``_DESTRUCTURERS`` lookup, but it only *reports* the
children instead of interning them. ``None`` means "opaque leaf" — the
value has no children a destructuring save would look inside.

Labels follow the underlying :meth:`Digested._slots`: field names for
dataclass / ``attrs`` instances, ``None`` for the positional slots of
lists, tuples, and mappings.

A destructurer registered through :func:`register_destructurer` is only
visible here if it is a :class:`Digested` classmethod (i.e. exposes
``_slots``); anything else is reported as an opaque leaf, since there is
no way to enumerate its children without also storing them.
"""
if isinstance(value, (digest.Digest, Number, str, bytes)):
return None
if isinstance(value, tuple) and DestructuringMixin._is_trojan_tuple(value):
return None
for pred, sunder_fn in _DESTRUCTURERS:
if pred(value):
owner = getattr(sunder_fn, "__self__", None)
slots = getattr(owner, "_slots", None)
return None if slots is None else slots(value)
return None


def register_destructurer(pred: Callable[[Any], bool], fn: Callable) -> None:
"""Register a custom container destructurer.

Expand Down
33 changes: 33 additions & 0 deletions src/fleche/storage/paths.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,42 @@
import weakref

from . import base
from . import destructuring
from .. import digest


def find_path(value: Any) -> Path | None:
"""Return a :class:`~pathlib.Path` a save of *value* would reach, if any.

Walks *value* exactly the way a destructuring save does — via
:func:`~fleche.storage.destructuring.child_slots`, so lists, tuples,
dicts, dataclasses, and ``attrs`` instances are descended into and
everything else is a leaf — and stops at the first ``Path`` it reaches,
returning ``None`` if the value carries no path at all. Which path comes
back when there are several is unspecified: this answers "is there one",
and the path itself is there to name in the resulting error.

This is a *predicate helper*, not part of the storage protocol: it exists
so a caller that cannot honour path semantics (notably
:class:`fleche.remote.SshCache`, where a path's meaning does not survive
the hop to another filesystem) can detect the situation up front instead
of silently storing something else. Shared cycles are visited once.
"""
seen: set[int] = set()
stack: list[Any] = [value]
while stack:
item = stack.pop()
if isinstance(item, Path):
return item
if id(item) in seen:
continue
seen.add(id(item))
slots = destructuring.child_slots(item)
if slots:
stack.extend(child for _, child in slots)
return None


class TempPath(type(Path())):
"""
A Path that deletes its backing temp tree when no references remain.
Expand Down
Loading
Loading