You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Integration CI: Karpenter chart pull from public.ecr.aws is rate-limited (unauthenticated) #8894
Both specs in the karpenter integration suite failed because the Karpenter Helm chart could not be
fetched from public.ecr.aws:
Error: failed to install Karpenter: failed to install Karpenter chart: failed to locate chart:
failed to perform "FetchReference" on source:
GET "https://public.ecr.aws/v2/karpenter/karpenter/manifests/1.6.3"
The failure landed ~19 minutes into the spec, after the cluster and nodegroup were already built, so
each occurrence costs a full cluster create/delete cycle.
Why it matters
public.ecr.aws rate-limits unauthenticated pulls. CI pulls the chart anonymously
(helmChartName = "oci://public.ecr.aws/karpenter/karpenter", pkg/karpenter/karpenter.go), so this
will keep recurring, and it fails the suite for a reason that has nothing to do with eksctl's
behaviour.
This was one of two independent causes of a red integration run; the other (EKS ResourceInUseException
409 races in three other suites) is addressed in #8893.
Suggested fix
Preference order, most durable first:
Authenticate the ECR pull in CI. An authenticated public.ecr.aws pull gets a substantially
higher rate limit. This fixes the problem where it actually is — in the CI environment — and adds
nothing for end users to carry. aws ecr-public get-login-password in the integration-test
workflow before the suite runs.
Pre-pull / cache the chart in the workflow so the suite doesn't hit the registry at all.
Retry the chart fetch in eksctl. Possible — pkg/utils/retry already provides the idiomatic
backoff used elsewhere in the repo — but it needs care: InstallChart is not idempotent in
general, so a retry has to be scoped to chart-location failures and not to failures that occur
after anything has been applied to the cluster. That distinction currently requires matching on
error text, which is fragile. Worth doing only if 1 and 2 are rejected.
I'd lean towards 1: it's the smallest change, it has no user-visible surface, and an anonymous pull
in CI is the actual defect.
Versions
Reproduced against main (0.231.0-dev), integration suite `karpenter`, us-west-2.
Chart: oci://public.ecr.aws/karpenter/karpenter 1.6.3
Activity