remote caching for downloaded artifacts

docs/docs/using-pants/remote-caching-and-execution/remote-caching.mdx

@@ -94,6 +94,20 @@ remote_cache_read = true
 remote_cache_write = true
 ```
 
+## Caching of downloads
+
+In addition to process results, Pants caches URL downloads (`http_source` sources and the external tools Pants itself fetches, such as interpreters, linters and formatters) in the remote cache. The downloaded bytes are stored in the remote store, along with an entry recording that the URL was observed to serve exactly those bytes. A machine with a cold local cache — such as an ephemeral CI runner — is then served a previously-verified download entirely from the remote cache, without contacting the origin server (e.g. GitHub) at all, protecting builds from origin outages.
+
+This behavior is on by default whenever remote cache reads or writes are enabled, and can be disabled with `[GLOBAL].remote_cache_downloads` (for example, during a cache-corruption investigation, or if your organization audits origin fetches at an egress proxy). A download is only ever served from the cache when some machine previously fetched the same URL and verified the same digest declared in the build; content is always re-verified against that digest as it is fetched. Note that rotating `[GLOBAL].process_execution_cache_namespace` does not affect cached downloads (their content is configuration-independent): use the option or server-side deletion instead. Also note that disabling the option (or deleting entries server-side) stops new remote-cache serving, but does not revoke the local record on a machine that was already served a download: such a machine keeps serving that (URL, digest) pair from its local caches until its local store evicts the file. A strict origin-only posture therefore also requires clearing local caches on warm machines.
+
+Operational notes:
+
+- Cache uploads happen in the background at the end of the run. A very short-lived run can exit before an upload completes, in which case the next machine re-downloads from the origin and retries the upload; this self-heals, and is bounded by the number of distinct tools.
+- Cached downloads stay warm best on servers which validate that an action result's referenced blobs still exist (e.g. bazel-remote's default completeness checking): there, every cache read refreshes both the entry and the bytes. On fleets where machines with `remote_cache_write` are long-lived (and so rarely re-download), it also helps to run at least one write-enabled builder that cycles its local cache, so evicted entries get re-minted.
+- Downloads of `file:` URLs and of URLs containing userinfo (`user:password@`) never participate. Presigned URLs which differ on every run work correctly but create one dead cache entry per distinct URL.
+- Downloads using `auth_headers` do participate: if the origin is private, anyone with read access to the remote cache can read the downloaded content. Disable `[GLOBAL].remote_cache_downloads` or `[GLOBAL].remote_cache_write` if this does not suit your cache's trust domain.
+- Cache entry lookups use the `[GLOBAL].remote_cache_rpc_timeout_millis` and `[GLOBAL].remote_cache_rpc_concurrency` options, with the retry count coming from `[GLOBAL].remote_store_rpc_retries`; byte transfers use the `[GLOBAL].remote_store_*` options. If the cache is unreachable, each download waits up to (`remote_store_rpc_retries` + 1) × `remote_cache_rpc_timeout_millis` (about 90 seconds with default settings) before falling back to the origin, with a warning throttled by `[GLOBAL].remote_cache_warnings`. If a degraded cache is adding too much latency, disable `[GLOBAL].remote_cache_downloads` (or remote caching) until it recovers.
+
 ## Reference
 
 Run `pants help-advanced global` or refer to [Global options](../../../reference/global-options.mdx). Most remote execution and caching options begin with the prefix `--remote`.

docs/notes/2.34.x.md

@@ -26,9 +26,12 @@ Published Pants binaries are now compiled with a new `dist` Cargo profile that e
 
 Pants option config files are now parsed as TOML 1.1 rather than TOML 1.0. This covers `pants.toml` and any other file named by `[GLOBAL].pants_config_files`, the rcfiles named by `[GLOBAL].pantsrc_files` (`/etc/pantsrc`, `~/.pants.rc` and `.pants.rc` by default), and `.toml` files referenced by `@fromfile` option values. Inline tables may now span multiple lines and end with a trailing comma, strings may use the `\e` and `\xHH` escapes, and times may omit their seconds. TOML 1.1 only adds syntax to TOML 1.0, so existing files continue to parse unchanged. TOML files read by backends, such as `pyproject.toml`, are unaffected.
 
-The `remote_cache_rpc_timeout_millis` & `remote_cache_rpc_concurrency` options are now correctly used again after being silently ignored since 2.19.  The documented default for `remote_cache_rpc_timeout_millis` is now `30000` to align with the actual behavior since 2.19.  If you used the defaults then nothing changes; if you explicitly set them they now take effect.
+The `remote_cache_rpc_timeout_millis` & `remote_cache_rpc_concurrency` options are now correctly used again after being silently ignored since 2.19.  The documented default for `remote_cache_rpc_timeout_millis` is now `30000` to align with the actual behavior since 2.19.  If you used the defaults then nothing changes; if you explicitly set them they now take effect. Conversely, since 2.19 `remote_store_rpc_timeout_millis` and `remote_store_rpc_concurrency` accidentally governed cache (ActionCache) RPCs as well: if you set the store options to tune cache lookups, set the corresponding `remote_cache_rpc_*` options now, as cache RPCs otherwise revert to the cache options' defaults.
+
+Fixed a bug where `remote_store_rpc_retries` was silently ignored by the REAPI store and cache providers, which always used 2 retries (since 2.2). If you used the default nothing changes; if you explicitly set it, it now takes effect for both store and cache RPCs — for example, a value of 8 now makes up to 9 attempts (previously 3), and a value of 0 now means a single attempt. Also fixed a related bug where the experimental OpenDAL provider used one more than `remote_store_rpc_retries` as the number of retries.
+
+Downloads (e.g. `http_source` sources and the external tools Pants itself fetches, such as interpreters and linters) now participate in remote caching. When remote caching is configured, the bytes of a downloaded file are stored in the remote store along with a record that the URL was observed to serve exactly those bytes, so machines with cold local caches (e.g. ephemeral CI runners) are served previously-verified downloads entirely from the remote cache instead of re-fetching from the origin (and failing when, say, GitHub has an outage). Content is always re-verified against the digest declared in the build as it is fetched. This is enabled by default whenever remote cache reads or writes are enabled, and can be disabled with the new `[GLOBAL].remote_cache_downloads` option.
 
-Fixed a bug where `remote_store_rpc_retries` was silently ignored by the REAPI cache provider, which always used 2 retries (since 2.2). If you used the default nothing changes; if you explicitly set it, it now takes effect. Also fixed a related bug where the experimental OpenDAL provider used one more than `remote_store_rpc_retries` as the number of retries.
 
 ### Goals
 

src/python/pants/engine/internals/buildbarn_integration_tests/BUILD

@@ -6,6 +6,7 @@ python_sources(dependencies=[":config"])
 python_tests(
     name="tests",
     overrides={
+        "buildbarn_download_cache_integration_test.py": {"timeout": 300},
         "buildbarn_remote_cache_integration_test.py": {"timeout": 180},
         "buildbarn_remote_execution_integration_test.py": {"timeout": 300},
     },