Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
48 commits
Select commit Hold shift + click to select a range
2da4b10
feat(caches)!: seal call identity before the function body runs
pmrv Jul 26, 2026
a37ef21
feat(storage): generalize DigestedDict to DigestedMapping
pmrv Jun 21, 2026
d917df3
feat(storage): cache filesystem paths by content
pmrv Jun 21, 2026
4f8198e
docs(paths): document path caching with recipes, a dev guide, and a n…
pmrv Jun 21, 2026
2cdb745
docs(storage): add notebook on paths nested inside containers
pmrv Jul 25, 2026
2fed836
docs(paths): add black-box-tested file semantics contract page
pmrv Jul 25, 2026
6ed3580
test(paths): pin pre-call keying for mutated Path arguments
pmrv Jul 26, 2026
b8b3865
refactor(caches): drop save_value, seal via values storage in prepare
github-actions[bot] Aug 5, 2026
4025229
fix(call): enforce single-shot PreparedCall.commit without mutating t…
claude Aug 6, 2026
ab77c07
fix(wrapper): run uncached when the cache rejects prepare
claude Aug 6, 2026
404bff7
test: cover remote prepare, commit guards, and hermetic e2e caches
claude Aug 6, 2026
a611788
refactor(caches): carry the pending result on PreparedCall until save…
claude Aug 6, 2026
5d5a67a
fix(wrapper): degrade to uncached on any prepare failure
claude Aug 6, 2026
b23ada9
refactor(remote): ship the committed result and record in two trips
claude Aug 6, 2026
fefdbfe
Merge pull request #825: review follow-ups for the two-phase save pro…
pmrv Aug 6, 2026
ae092dd
docs(caches): trim protocol comments to their invariants (#830)
pmrv Aug 6, 2026
e2fe718
docs: split the purity contract out of file semantics; digest Complet…
pmrv Aug 6, 2026
4e2083e
fix(ci): silence the three ty diagnostics this branch introduces
claude Aug 6, 2026
0ddf18f
fix(remote): refuse Path values at the SshCache boundary (#828)
pmrv Aug 6, 2026
fa5d2d2
Merge branch 'prepared-call' into temppath
claude Aug 6, 2026
8fe8227
docs(paths): explain why remaining_depth cannot reach a nested path
claude Aug 6, 2026
add1dea
Merge branch 'main' into temppath
claude Aug 7, 2026
4336827
Merge branch 'main' into temppath
claude Aug 7, 2026
28df4cd
fix(storage): keep path content reachable from the gc walk
claude Aug 7, 2026
59f17e6
fix(digest): degrade unreadable paths to Indigestible
claude Aug 7, 2026
a9e1a13
fix(caches): keep in-flight call arguments reachable from gc
claude Aug 7, 2026
ff168a0
fix(caches): order gc's reads so a mid-sweep commit cannot slip through
claude Aug 7, 2026
4c10b92
fix(remote): degrade only the path arguments, not the whole call
claude Aug 7, 2026
4d6ded1
fix(storage): walk any re-iterable container when looking for a path
claude Aug 7, 2026
58a0cfe
test(paths): stop leaking the sticky cache out of the mutation tests
claude Aug 8, 2026
3c33028
Merge branch 'main' into temppath
claude Aug 8, 2026
c8a5861
Merge branch 'main' into temppath
claude Aug 8, 2026
35c3aae
test(gc): scope the concurrent-sweep test to what the sweep guarantees
claude Aug 8, 2026
aac0e06
Merge branch 'main' into temppath
claude Aug 8, 2026
fd73b8a
Merge branch 'main' into temppath
claude Aug 9, 2026
c108ec2
Merge branch 'main' into temppath
claude Aug 9, 2026
ad7b883
Merge branch 'main' into temppath
claude Aug 11, 2026
4468aaf
Merge branch 'main' into temppath
claude Aug 12, 2026
f2ec078
Merge branch 'main' into temppath
claude Aug 12, 2026
d032f71
Merge branch 'main' into temppath
claude Aug 12, 2026
4cb00fa
Merge branch 'main' into temppath
claude Aug 13, 2026
0c2b8d4
Merge branch 'main' into temppath
claude Aug 14, 2026
30a9ccc
Merge branch 'main' into temppath
claude Aug 15, 2026
a4c6c82
Merge branch 'main' into temppath
claude Aug 16, 2026
0f79627
Merge remote-tracking branch 'origin/main' into temppath
claude Aug 20, 2026
6c45c90
Merge remote-tracking branch 'origin/main' into temppath
claude Aug 21, 2026
f1be424
Merge remote-tracking branch 'origin/main' into temppath
claude Aug 22, 2026
cfe0ed9
Merge remote-tracking branch 'origin/main' into temppath
claude Aug 23, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 5 additions & 3 deletions agents/DEVELOPING.md
Original file line number Diff line number Diff line change
Expand Up @@ -174,9 +174,10 @@ Shared fixtures (in `fixtures.py`):
## Other directories

- `benchmarks/` — `benchmark_{digest,integration,storage}.py`, `run_benchmarks.py`, `compare_results.py` (diff two `results.csv` runs), `utils.py`, `profile_digest_types.py` (per-type cProfile harness), `results.csv`.
- `docs/` — Sphinx sources, grouped by topic: root holds `index`, `installation`, `parallel_execution`; `usage/` holds `tldr`, `helpers`, `lazy_call`, `query` (`tldr` first, PR #745); `digests/` holds `digests_as_args`, `digest_equivalence`, `entry_points` (added by PR #752 as the dedicated fleche-ase / third-party plug-in page); `storage/` holds `configuration`, `destructuring` (figures + prose on destructuring/dedup/`remaining_depth`), `cache_stack`, `security`; `dev/` holds `call_lifecycle` (the two-phase `prepare`/`commit`/`abandon` save protocol and the miss/hit sequences behind a decorated call), `custom_digests`, `extending_destructurer`, `function_profile`, `ssh_cache`, `sql_test_backends`, `storage_hierarchy` (the former grab-bag `dev/developer.rst` was flattened into those standalone pages, keeping its `.. _…:` anchor labels). `docs/figures/` holds `gen_diagrams.py` plus the three destructuring SVGs it generates — the script builds the depicted layouts in real `ValueMemory` stores so the digest labels in the figures match what `notebooks/Destructuring.ipynb` computes — and the two hand-written Graphviz sources `storage_hierarchy.dot` / `storage_mro.dot` with their rendered SVGs (`dot -Tsvg <src> -o <out>`; these replaced the old `devnotes/storage-hierarchy.{dot,md,svg}`, which was stale from PR #245 — it predated `OperationContext`, the thread-safety mixins, and destructuring becoming default on every value storage), plus `gen_sequence.py`, which emits `storage_sequence.svg` (a destructured save/load through the MRO, showing the per-entry lock scopes) and `cache_sequence.svg` (cache miss/hit around the two-phase save) — its panels are transcribed from instrumented traces, not from reading the code. Every figure SVG is checked in rather than rendered at build time because `.readthedocs.yaml` declares no `apt_packages`, so `dot` cannot be assumed on the builder. The HTML theme is `shibuya`, skinned solarized-light/dark with the green accent via CSS custom properties in `docs/_static/custom.css` (which also keeps the "on this page" rail in-flow down to 720px-wide viewports and sets an 18px base font); the figure palette in `gen_diagrams.py` is the matching solarized-light set, so retheming the docs means updating both and rerunning the generator. `docs/notebooks/` is **symlinks** into `../../notebooks/` (all nine non-`Destructuring` entries — `Caches`/`TransferWorkflow` added by PR #839, `FiveMinuteTour` by PR #841 — plus `Destructuring` since the destructuring-docs work); the `rendernb.yml` workflow re-executes `notebooks/*.ipynb` in place when a PR carries the `rendernb` label.
- `docs/` — Sphinx sources, grouped by topic: root holds `index`, `installation`, `parallel_execution`; `usage/` holds `tldr`, `purity`, `helpers`, `file_semantics`, `lazy_call`, `query` (`tldr` first, PR #745; `purity` carries the general "fleche caches pure functions" contract — arguments keyed as passed, mutation and other side effects not replayed — that `file_semantics` states for paths); `digests/` holds `digests_as_args`, `digest_equivalence`, `entry_points` (added by PR #752 as the dedicated fleche-ase / third-party plug-in page); `storage/` holds `configuration`, `destructuring` (figures + prose on destructuring/dedup/`remaining_depth`), `cache_stack`, `security`; `dev/` holds `call_lifecycle` (the two-phase `prepare`/`commit`/`abandon` save protocol and the miss/hit sequences behind a decorated call), `custom_digests`, `extending_destructurer`, `function_profile`, `path_storage`, `ssh_cache`, `sql_test_backends`, `storage_hierarchy` (the former grab-bag `dev/developer.rst` was flattened into those standalone pages, keeping its `.. _…:` anchor labels); `recipes/` holds `files_and_paths`. `docs/figures/` holds `gen_diagrams.py` plus the three destructuring SVGs it generates — the script builds the depicted layouts in real `ValueMemory` stores so the digest labels in the figures match what `notebooks/Destructuring.ipynb` computes — and the two hand-written Graphviz sources `storage_hierarchy.dot` / `storage_mro.dot` with their rendered SVGs (`dot -Tsvg <src> -o <out>`; these replaced the old `devnotes/storage-hierarchy.{dot,md,svg}`, which was stale from PR #245 — it predated `OperationContext`, the thread-safety mixins, and destructuring becoming default on every value storage), plus `gen_sequence.py`, which emits `storage_sequence.svg` (a destructured save/load through the MRO, showing the per-entry lock scopes) and `cache_sequence.svg` (cache miss/hit around the two-phase save) — its panels are transcribed from instrumented traces, not from reading the code. Every figure SVG is checked in rather than rendered at build time because `.readthedocs.yaml` declares no `apt_packages`, so `dot` cannot be assumed on the builder. The HTML theme is `shibuya`, skinned solarized-light/dark with the green accent via CSS custom properties in `docs/_static/custom.css` (which also keeps the "on this page" rail in-flow down to 720px-wide viewports and sets an 18px base font); the figure palette in `gen_diagrams.py` is the matching solarized-light set, so retheming the docs means updating both and rerunning the generator. `docs/notebooks/` is **symlinks** into `../../notebooks/` — all twelve entries, and all twelve are in the toctree (`Caches`/`TransferWorkflow` added by PR #839, `FiveMinuteTour` by PR #841, `Destructuring` by PR #844, `Files`/`PathsInContainers` by PR #797); the `rendernb.yml` workflow re-executes `notebooks/*.ipynb` in place when a PR carries the `rendernb` label.

- `notebooks/` — usage examples (`FiveMinuteTour`, `GettingStarted`, `Caches`, `CacheStack`, `Destructuring`, `StorageBackends`, `SecureStorage`, `ConcurrentExecution`, `ExtraMethods`, `TransferWorkflow`); all are executed by `tests/integration/test_notebooks.py` since PR #839, which flipped the parametrisation from a hand-maintained list to `sorted(notebooks/*.ipynb)` so a notebook added later cannot silently rot out of coverage. `Destructuring.ipynb` embeds the `docs/figures/` figures as **2x PNG** markdown-cell attachments (no display code; PNG because JupyterLab refuses to render `image/svg+xml` attachments) and rebuilds every depicted storage layout in `ValueMemory` so readers can play; its digest outputs match the figure labels by content-addressing. Running `python docs/figures/gen_diagrams.py` regenerates the SVGs **and** refreshes the notebook's embedded PNGs (`refresh_notebook_attachments`, rasterizing via `rsvg-convert` or `cairosvg`); each SVG carries a generated-by comment pointing back at the script.

- `notebooks/` — usage examples (`FiveMinuteTour`, `GettingStarted`, `Caches`, `CacheStack`, `Destructuring`, `StorageBackends`, `SecureStorage`, `ConcurrentExecution`, `ExtraMethods`, `Files`, `PathsInContainers`, `TransferWorkflow`); all twelve are executed by `tests/integration/test_notebooks.py`, which since PR #839 parametrises over `sorted(notebooks/*.ipynb)` rather than a hand-maintained list, so a notebook added later cannot silently rot out of coverage. Committed **with outputs** — `rendernb.yml` re-executes them in place on a PR labelled `rendernb`, so an edited notebook should be re-run before it lands. `Destructuring.ipynb` embeds the `docs/figures/` figures as **2x PNG** markdown-cell attachments (no display code; PNG because JupyterLab refuses to render `image/svg+xml` attachments) and rebuilds every depicted storage layout in `ValueMemory` so readers can play; its digest outputs match the figure labels by content-addressing. Running `python docs/figures/gen_diagrams.py` regenerates the SVGs **and** refreshes the notebook's embedded PNGs (`refresh_notebook_attachments`, rasterizing via `rsvg-convert` or `cairosvg`); each SVG carries a generated-by comment pointing back at the script.
- `.github/workflows/` — CI: `tests.yml` (PR sweep across 3.11–3.14 + `sql-backends` job that boots Postgres 16 + MariaDB 11 service containers and sets `FLECHE_TEST_{POSTGRES,MYSQL}_URL`), `ty.yml`, `test-minimum-deps.yml` (PR #645, installs every direct dep at its declared floor via `uv pip install --resolution lowest-direct -e ".[tests]"` and runs the suite — `uv pip install` rather than `uv sync` because the universal lock floats deps up through cross-extra constraints and would mask the real minimums), `benchmarks.yml`/`benchmarks-main.yml`/`updatebenchmarks.yml`, `perf-triage.yml` (PR #628 — Haiku reads the PR diff/description and adds the `benchmark` label when the change touches a hot path — that label is the existing `benchmarks.yml` trigger), `rendernb.yml` (re-executes `notebooks/*.ipynb` on PRs labelled `rendernb`), `release-please.yml`, `pypi-publish.yml` (trusted-publisher upload triggered by `release: published`), `claude.yaml` + `claude_ci_details.yaml` (the latter exposes CI status as a tool the in-PR Claude can call). Releases use **release-please** (`release-please-config.json`, `.release-please-manifest.json`) — release PRs are opened automatically from conventional-commit history on `main`.

## Design themes / open scope (issue tracker)
Expand All @@ -193,7 +194,7 @@ Cheat sheet of what's been considered. Issue numbers are the entry points — fe

**PR #804** (opened 2026-07-31) drops the `SizeLimitedCache(max_size=10)` config from `benchmarks/benchmark_integration.py`: uniform-random eviction under the 60 call records that config accumulates across the three benchmarked functions meant most "hit"-phase samples were silently re-measuring misses — hit-phase numbers for that config in older `results.csv` runs are unreliable; the surviving `max_size=100` config never evicts.

- **Distributed / remote caching** (#552; #551 **landed**). Still open: #552 — a `TieredValues` + `GlobusValues` cold tier for HPC value blobs that `Cache.query()` never touches, only `Cache.load_value()` on a hot miss.
- **Distributed / remote caching** (#552; #551 **landed**). `SshCache` refuses `Path` values outright (`RemotePathUnsupported`, a `SaveError` subclass so a path argument degrades to a locally-computed digest-only reference and a path result becomes `Rejected`) — a pickled `Path` is only its string, so the server was resolving it against its own filesystem and filing records under keys no client recomputes. **#829** tracks making them actually work: run the path→blob reduction client-side (the blobs' `__digest__` already matches the `Path` digest arm) plus one new unmended `load_value` verb. Still open: #552 — a `TieredValues` + `GlobusValues` cold tier for HPC value blobs that `Cache.query()` never touches, only `Cache.load_value()` on a hot miss.

- **Config redesign** (#568). Next-generation TOML/YAML schema with top-level `value`/`call`/`stash`/`metadata` namespaces and cross-file named references. Supersedes the just-landed merged-discovery model (PR #553) once the schema firms up; expect a `hash_version`-style migration story.

Expand Down Expand Up @@ -304,6 +305,7 @@ Cheat sheet of what's been considered. Issue numbers are the entry points — fe
- Cache-level race fixes on top of #569: #217 → PR #629, #451 → PR #631, #452 → PR #630, #485 → PR #627. Regression-pinned in `tests/regression/test_issue_{217,451,452,485}.py`; concurrency stress tests share `run_workers()` from `tests/fixtures.py` (PR #633).

- Digest dispatch grew a `builtins`-type arm and a `not isinstance(value, type)` guard on the dataclass/attrs arms so `digest(int)` / `digest(SomeDataclass)` (the class object) stop raising — PR #651, closes #469.
- `digest()` grew a `subprocess.CompletedProcess` arm (args + returncode + stdout + stderr) so functions wrapping command-line tools hash without a user hook — `run()`'s result is neither a dataclass nor iterable, so it was `Indigestible` before. Purely additive (those values raised previously), so no `hash_version` bump. Pinned in `tests/unit/digest/test_digest.py::test_completedprocess_*`; `notebooks/Files.ipynb` used to register this by hand via `add_hook` and no longer does.

- `digest()` grew explicit `pd.DataFrame` / `pd.Series` / `pd.Index` arms so pandas inputs hash by content (columns/name + dtype + index + `hash_pandas_object`), not by the column names yielded by `iter()` — PR #675. Silent change with no `hash_version` bump (previous pandas cache entries are now unreachable). Pinned in `tests/unit/digest/test_digest.py::test_pandas_{dataframe,series,index}_hashes_by_content`.

Expand Down
Loading
Loading