oci: Retry transient registry failures when pulling

[cgwalters-bot]

Pulls through skopeo have no retries at all, so a single 503 or dropped connection from e.g. quay.io fails the whole pull. That's a regular source of CI flakes for bootc (bootc-dev/bootc#2177). The proxy doesn't retry for us: in podman the retry loop lives in the caller (c/common pkg/retry), so this adds one here.

The proxy decides which errors are worth retrying. Since skopeo 1.19 it tags each failed reply with c/common's IsErrorRetryable() verdict, the same check podman's retry loop uses, and cgwalters-forge/containers-image-proxy-rs#2 exposes that as Error::is_retryable(). composefs-rs does no error-text matching of its own. With an older skopeo nothing gets retried, same as today. Only proxy errors count, so a local failure to import verified data is never retried.

Retries happen per proxy operation: opening the image, fetching the manifest, the config, and each layer or delta blob. That way a failed layer doesn't refetch the others. The schedule is podman's default, 3 retries after 1s, 2s and 4s, and a fixed delay can be set like --retry-delay. Only docker:// pulls retry. PullOptions gains a public retry: RetryPolicy field to tune or disable it. Since PullOptions isn't #[non_exhaustive], code that builds it with a struct literal and no ..Default::default() needs updating. cfsctl oci pull --retry N exposes the count (--retry 0 fails fast); the varlink API keeps the default. Note that pull_image() now retries by default too. Each attempt restarts the blob transfer, so a layer's Started progress event is sent again per attempt, and cfsctl replaces that layer's bar.

The first commit makes cfsctl print progress messages when stderr isn't a terminal. indicatif drops them otherwise, so retry warnings would never show up in CI logs. It could land on its own.

Before merging: the second commit points containers-image-proxy at the forge branch through [patch.crates-io]. The plan is to land cgwalters-forge/containers-image-proxy-rs#2 and release it as 0.11.1 first, then replace the [patch] with a version bump. The retry loop stays in composefs-rs for now; bootc already has its own, and switching bootc's classifier to is_retryable() is a separate follow-up (bootc-dev/bootc#2466).

The new integration test serves an OCI layout from a small in-process registry. The registry answers the first manifest request with a 503 and hangs up on the first request for each blob. The test checks that cfsctl oci pull docker://... recovers and yields the pinned image ID, and that --retry 0 fails. It goes through the real skopeo, so it also checks that skopeo classifies both failures as retryable. That classification needs skopeo 1.19, so the test parses skopeo --version and skips with older versions (upstream's ubuntu-24.04 runners have 1.13.3). The skopeo check is now one shared helper in the integration tests' main.rs; before, old_format.rs had its own copy.

Testing ran on a 16-core devspace at head 39b7496, with proxy-rs pulled from the forge branch. Two containers were used:

• quay.io/fedora/fedora:latest (rustc 1.98, skopeo 1.22.3): just fmt-check, just clippy and just check-feature-combos were clean, and cargo test -p composefs-oci -p composefs-ctl passed (174 composefs-oci tests, including the retry and fetch_layer unit tests). just test-integration passed 112, including test_pull_retries_transient_registry_errors and the new test_parse_skopeo_version. It failed test_ostree_pull_local_all_modes and test_varlink_open_repository_invalid_spec, and main fails the same two in that container.
• ubuntu:24.04 (skopeo 1.13.3, rust stable), as in upstream's smoke job: the retry test reports skopeo (1, 13, 3) does not classify retryable errors (needs (1, 19, 0)), skipping registry retry test and passes, and test_parse_skopeo_version passes. I stopped the full unprivileged suite there after it sat for 30 minutes with idle cfsctl processes. I didn't dig into that, and main wasn't run there for comparison.

The fs-verity-dependent unit tests, just check-fuzz and the privileged VM tests weren't run for this revision. The earlier revision passed check-fuzz.

Closes composefs#348

Generated-by: https://github.com/cgwalters/#llms

---

Review draft in cgwalters-forge, not upstream yet. This section is removed when the PR is opened upstream.

• Upstream: composefs/composefs-rs, base main
• Board item: PVTI_lAHOAQ_SPs4Bj2Gizg8PhYk
• Fork CI: off; the devspace testing described above is this PR's CI, and upstream CI runs once it is opened there

To review:

• Approve, or comment /promote on a line of its own, to open it upstream, ready for review. Either covers only the commits pushed so far.
• If upstream requires DCO, approving also signs off: promote adds Signed-off-by: Colin Walters <walters@verbum.org> to the commits lacking it (the bot's and yours; anyone else's only if you ask), with you as committer.
• Add a /draft line (in the same comment or before) to open it upstream as a draft (/ready undoes that).
• Close to drop it.
• Edit the title and description freely: they become the upstream PR's. Review comments are squashed into the commits they concern, with a reply here.

[cgwalters]

This is too ugly, we need to reuse the logic from the proxy, is that not possible?

[cgwalters-bot]

Yes. Since 1.19, skopeo tags every failed reply with c/common's IsErrorRetryable() verdict (error_code: "retryable"), but containers-image-proxy-rs was dropping it. cgwalters-forge/containers-image-proxy-rs#2 exposes it as Error::is_retryable(), and retry.rs now relies on that alone, so the error-string patterns, the jitter and the delay cap are all gone. That leaves podman's 1s/2s/4s loop, which is short, and neither proxy-rs nor composefs-rs had one to reuse. This needs a proxy-rs release. Until then, the new second commit pulls proxy-rs in from git; the rebased and squashed branch is at 9dbdcf0.

Generated-by: https://github.com/cgwalters/#llms

[cgwalters-bot]

Review guide for head 9dbdcf02cd: where to look closely and what is safe to skim. It is advice from the bot's reviewer and doesn't replace reading the diff; the review app walks it.

Retries transient registry failures during docker:// pulls. Retries are per proxy operation (OpenImage, manifest, config, each layer and delta blob) on podman's default schedule: 3 retries after 1s, 2s and 4s. The only classifier is containers-image-proxy's new Error::is_retryable() from forge proxy-rs#2, so no error-text matching is left. With skopeo older than 1.19, nothing is retried. The [patch.crates-io] commit is temporary and must become a 0.11.x version bump once proxy-rs is released. The main risk is the new integration test. It needs skopeo 1.19 or later but only checks that skopeo exists. The upstream Integration job runs the unprivileged tests on the ubuntu-24.04 host, whose skopeo (from plucky) is 1.13, so the test will fail there. Also look at the layer import refactor. The proxy driver is now always awaited, and nothing is registered until the proxy has verified the blob. Digest mismatches are no longer retried, which matches podman.

Hotspots

1. look-closely · api — Cargo.toml:60-63 (a85d634): Temporary git patch pointing at the forge fork. Before merge, drop it for a containers-image-proxy 0.11.x version bump in both the composefs-oci and integration-tests Cargo.toml. crates.io 0.11.0 has no is_retryable().
2. risky · test-gap — crates/composefs-integration-tests/src/tests/registry_retry.rs:275-279 (9dbdcf0): This only checks that skopeo exists, but the test needs skopeo 1.19 or later. The upstream Integration job runs unprivileged tests on the ubuntu-24.04 host with skopeo 1.13 from plucky, so the first case will fail. Skip when skopeo --version is below 1.19.
3. look-closely · logic — crates/composefs-oci/src/retry.rs:122-157 (8565d72): The retry loop itself: the attempt count, the backoff and the give-up context. max_retries=3 means 4 attempts.
4. look-closely · error-handling — crates/composefs-oci/src/retry.rs:82-111 (8565d72): ProxyAndImportError, plus its special case in is_transient, is the only machinery beyond the loop. Wrapping the proxy error with the import error as anyhow context would keep the proxy error in the chain and remove the need for this type.
5. note · logic — crates/composefs-oci/src/skopeo.rs:102-111 (8565d72): Only Transport::Registry retries; local transports get RetryPolicy::none().
6. look-closely · logic — crates/composefs-oci/src/skopeo.rs:531-568 (8565d72): Per-layer retry. Started is sent again for each attempt, and fetch() must start a fresh GetBlob every time. The semaphore permit stays held during backoff sleeps.
7. look-closely · logic — crates/composefs-oci/src/skopeo.rs:626-676 (8565d72): Behavior change from main: the driver is now awaited even when the import failed. The reader is dropped first, so skopeo gets EPIPE and FinishPipe returns instead of hanging. The layer is only registered after the driver reports success.
8. note · api — crates/composefs-oci/src/lib.rs:349-353 (8565d72): New pub field on PullOptions, which is not non_exhaustive, so struct literals without ..Default::default() break. pull_image() also retries by default now.
9. note · logic — crates/composefs-ctl/src/lib.rs:169-177 (c2afac4): When stderr is not a tty, every progress message now goes to it. This is intended, so that retry warnings reach CI logs.

Safe to skim

• crates/composefs-oci/src/retry.rs:159-374: Table-driven unit tests
• crates/composefs-oci/src/skopeo.rs:798-1094: Unit tests with a scripted fake fetcher
• crates/composefs-integration-tests/src/tests/registry_retry.rs:1-270: Minimal flaky HTTP registry fixture
• crates/composefs-integration-tests/src/tests/zstd_chunked.rs: have_skopeo() moved to main.rs
• crates/composefs-oci/Cargo.toml: Adds the tokio time feature

[cgwalters]

/draft

[cgwalters-bot]

Signed off 4 commit(s) with Signed-off-by: Colin Walters <walters@verbum.org> for composefs/composefs-rs's DCO check, on cgwalters's approval #1 (review) of 39b749612a61. Same trees; the head is now a594bfe1f9c2.

[cgwalters-bot]

Opened upstream as composefs#407 (draft). Closing this review draft.