[kube-prometheus-stack] Alertmanager PVC stuck Pending on GKE since v82+ — CSI rejects operator-injected empty selector

[bmailhe]

Describe the bug a clear and concise description of what the bug is.

Describe the bug

After upgrading kube-prometheus-stack from 81.5.0 to 83.4.2, any Alertmanager with alertmanager.alertmanagerSpec.storage.volumeClaimTemplate configured fails to provision on GKE clusters. The PVC is rejected by the GCE PD CSI driver with:

ProvisioningFailed  pd.csi.storage.gke.io  claim Selector is not supported

The resulting Alertmanager pod is stuck Pending with FailedScheduling: running PreBind plugin "VolumeBinding": binding volumes: context deadline exceeded.

Downgrading back to 81.5.0 fixes it. prometheus volumeClaimTemplate appears unaffected in our setup — only alertmanager hits this.

Workaround

Pin chart to 81.5.0.

What's your helm version?

version.BuildInfo{Version:"v3.20.0", GitCommit:"b2e4314fa0f229a1de7b4c981273f61d69ee5a59", GitTreeState:"clean", GoVersion:"go1.25.6"}

What's your kubectl version?

Client Version: v1.32.3 Kustomize Version: v5.5.0 Server Version: v1.34.4-gke.1047000

Which chart?

kube-prometheus-stack

What's the chart version?

83.4.2

What happened?

The PersistentVolumeClaim materialized from the Alertmanager STS volumeClaimTemplates has spec.selector: {} — an empty but non-nil selector. GKE's PD CSI external-provisioner rejects any PVC with a selector, including empty.

Evidence on a running cluster (chart 83.4.2, operator v0.90.1):

• Helm-rendered Alertmanager CR has no selector field
• kubectl get --raw on the STS spec.volumeClaimTemplates[0].spec has no selector field
• kubectl get --raw on the resulting PVC has "selector": {}
• managedFields show kube-controller-manager owns f:selector: {} (the STS controller materializes it during PVC creation from the template)

What you expected to happen?

What you expected to happen?

The PVC should be created without a selector field (matching the Alertmanager CR and STS template), so the CSI provisioner accepts it.

How to reproduce it?

How to reproduce it?

1. Deploy kube-prometheus-stack 83.4.2 on a GKE cluster (any PD-backed storage class: standard-rwo, premium-rwo, etc. — all funnel through pd.csi.storage.gke.io via CSI Migration).
2. Configure alertmanager with persistence:

   alertmanager:
     alertmanagerSpec:
       storage:
         volumeClaimTemplate:
           spec:
             storageClassName: standard-rwo
             accessModes: ["ReadWriteOnce"]
             resources:
               requests:
                 storage: 5Gi

3. PVC stays Pending with claim Selector is not supported; pod stays Pending.

Enter the changed values of values.yaml?

  alertmanager:
    alertmanagerSpec:
      storage:
        volumeClaimTemplate:
          spec:
            storageClassName: standard-rwo
            accessModes: ["ReadWriteOnce"]
            resources:
              requests:
                storage: 5Gi

Enter the command that you execute and failing/misfunctioning.

helm secrets upgrade -i prometheus-operator . -n monitoring --create-namespace -f myvalues.yaml

Anything else we need to know?

No response

[DrFaust92]

I can't reproduce the empty selector from the chart side.

Clean kind cluster (k8s 1.37), chart 83.4.2 with its own CRDs and operator v0.90.1 — your exact combination — and your values verbatim:

$ kubectl get alertmanager -o json | jq '.items[0].spec.storage.volumeClaimTemplate.spec | keys'
["accessModes","resources","storageClassName"]
$ kubectl get sts -o json | jq '.items[0].spec.volumeClaimTemplates[0].spec | has("selector")'
false
$ kubectl get pvc -o json | jq '.items[0].spec | has("selector")'
false

The PVC binds, and there is no selector key anywhere in the chain. Same result on 91.2.1 with operator v0.94.0. That fits the chart code — alertmanager.yaml passes alertmanagerSpec.storage through with toYaml and adds nothing, which is also why your rendered CR has no selector.

So something in your cluster is adding it between the StatefulSet template and the PVC, which matches your own finding that kube-controller-manager owns f:selector on the PVC but the STS template has no such field. Two things worth checking: whether a mutating webhook is in play (kubectl get mutatingwebhookconfigurations — Policy Controller, Kyverno, Config Sync and friends all like to normalise PVCs), and whether that PVC predates the upgrade. StatefulSet PVCs are created once and never reconciled, so a claim created under an older release keeps whatever fields it had, and the chart version you are on at the time makes no difference to it.

If you still have the cluster, kubectl get pvc <name> -o json | jq '{created: .metadata.creationTimestamp, managedFields: .metadata.managedFields}' would settle which manager wrote the field and when.