remote caching for downloaded artifacts

[cburroughs]

So, a common frustration I run into looks something like this:

• In CI a build fails with native_engine.IntrinsicError: Server error (504) downloading file. This is most often an external tool from GitHub.
• Someone links https://mrshu.github.io/github-statuses/ and points out the "one 9" of GitHub reliability. :-/
• Someone else asks something along the lines of "we have so many caches that it is hard to keep track of them all, why aren't they helping"?

The short answer to the last question is that we have long cached downloads locally, but never remotely. And the naive "just use the digest" strategy would suffer from the bug described in issue #13255: stale-but-present digests would silently serve cached bytes and thus dead or wrong URLs could persist in configuration indefinitely. In other words you would get the right bytes, but with no guarantee that the bytes came from the URL you expected.

(I've continued to try cranking up file_downloads_max_attempts, but GitHub reliability really hasn't been great. A secondary motivation is that on AWS, managed NAT gateway bandwidth is at a surprisingly high cost premium and unless you are all on ipv6 it is easy to rack up surprisingly high bills.)

The approach here (inspired by my understanding of bazel/bb-remote-asset) is to add a "synthetic" (in the sense that it isn't a "real" command) entry that represents the URL-->Digest mapping. So you can only get a cache hit with the expected URL, the digest alone isn't sufficient.

I spent a while trying to understand the Remote Asset API and how it fits in. I came away with the conclusion that server support is still uneven and is potentially complicated to operate. And notably the way Buildbarn implements the Remote Asset API is on top of the Remote Execution API with synthetic actions
https://www.youtube.com/watch?v=Fn035DBkVXo so I think it is a reasonable pattern with prior art. (Also we already construct the local cache keys since #13266.)

remote_cache_downloads (default: on) enables this behavior (I knob seemed like the safe thing to provide in case of bugs, but absent bugs I'm not sure why one would want to disable this.)

Notice: This code was generated Elephant/Goldfish style https://drensin.medium.com/elephants-goldfish-and-the-new-golden-age-of-software-engineering-c33641a48874 with Fable. The design document is at
https://gist.github.com/cburroughs/f7c522be75ad68de973d406f7ad683df and it is an actual design document that I have edited, read, reviewed etc, not just a LLM dumping scratchpad.

[cburroughs]

(this one really is still in draft; working with @JettJones)

[cburroughs]

I tried doing something like this with a large CI system:

• Run through CI with the latest release, everything green.
• Blackhole github.com api.github.com codeload.github.com objects.githubusercontent.com raw.githubusercontent.com release-assets.githubusercontent.com with /etc/hosts. --> So much red.

And then:

• Run through CI with this branch, passes normally.
• While making sure the named_caches are not saved, blackhole github.com api.github.com codeload.github.com objects.githubusercontent.com raw.githubusercontent.com release-assets.githubusercontent.com with /etc/hosts.
• Stuff mostly, passes. No no issues turned up.

The "mostly" is that many tools (such as Helm) do their own downloads, and so REAPI can't save them. I still think this is a net improvement, but there is still more work to do if someone has the goal of "immune to GitHub outages".

[JettJones]

I tried doing something like this with a large CI system:

* Run through CI with the latest release, everything green.

* Blackhole  `github.com api.github.com codeload.github.com objects.githubusercontent.com raw.githubusercontent.com release-assets.githubusercontent.com` with `/etc/hosts`. --> So much red.

And then:

* Run through CI with this branch, passes normally.

Nice - that's all I was thinking of when testing this, so sounds reasonable to me 👍