Skip to content

Integration CI: Karpenter chart pull from public.ecr.aws is rate-limited (unauthenticated) #8894

Open

Description

@gustavodiaz7722

What happened?

Both specs in the karpenter integration suite failed because the Karpenter Helm chart could not be
fetched from public.ecr.aws:

Error: failed to install Karpenter: failed to install Karpenter chart: failed to locate chart:
failed to perform "FetchReference" on source:
GET "https://public.ecr.aws/v2/karpenter/karpenter/manifests/1.6.3"

The failure landed ~19 minutes into the spec, after the cluster and nodegroup were already built, so
each occurrence costs a full cluster create/delete cycle.

Why it matters

public.ecr.aws rate-limits unauthenticated pulls. CI pulls the chart anonymously
(helmChartName = "oci://public.ecr.aws/karpenter/karpenter", pkg/karpenter/karpenter.go), so this
will keep recurring, and it fails the suite for a reason that has nothing to do with eksctl's
behaviour.

This was one of two independent causes of a red integration run; the other (EKS ResourceInUseException
409 races in three other suites) is addressed in #8893.

Suggested fix

Preference order, most durable first:

  1. Authenticate the ECR pull in CI. An authenticated public.ecr.aws pull gets a substantially
    higher rate limit. This fixes the problem where it actually is — in the CI environment — and adds
    nothing for end users to carry. aws ecr-public get-login-password in the integration-test
    workflow before the suite runs.
  2. Pre-pull / cache the chart in the workflow so the suite doesn't hit the registry at all.
  3. Retry the chart fetch in eksctl. Possible — pkg/utils/retry already provides the idiomatic
    backoff used elsewhere in the repo — but it needs care: InstallChart is not idempotent in
    general, so a retry has to be scoped to chart-location failures and not to failures that occur
    after anything has been applied to the cluster. That distinction currently requires matching on
    error text, which is fragile. Worth doing only if 1 and 2 are rejected.

I'd lean towards 1: it's the smallest change, it has no user-visible surface, and an anonymous pull
in CI is the actual defect.

Versions

Reproduced against main (0.231.0-dev), integration suite `karpenter`, us-west-2.
Chart: oci://public.ecr.aws/karpenter/karpenter 1.6.3
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/tech-debtLeftover improvements in code, testing and buildingkind/bug

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions