[kube-prometheus-stack] KubeHpaMaxedOut fires false positives when HPA is at maxReplicas but not ScalingLimited

[julienbaladier]

Describe the bug a clear and concise description of what the bug is.

The KubeHpaMaxedOut alert fires false positives when an HPA is running at maxReplicas but is not actually constrained: ScalingLimited=False with reason DesiredWithinRange. This means the HPA computed exactly this replica count and is satisfied; it is not trying to scale further and being blocked.

This is particularly visible with KEDA-managed HPAs (cron or fixed-scale triggers) but can occur with any HPA where the desired replica count happens to equal maxReplicas without the HPA being resource-starved.

What's your helm version?

version.BuildInfo{Version:"v3.20.0", GitCommit:"b2e4314fa0f229a1de7b4c981273f61d69ee5a59", GitTreeState:"clean", GoVersion:"go1.25.6"}

What's your kubectl version?

Client Version: v1.34.2 Kustomize Version: v5.7.1 Server Version: v1.34.9-eks-bca9cf6

Which chart?

kube-prometheus-stack

What's the chart version?

88.6.5

What happened?

Multiple HPAs continuously fire KubeHpaMaxedOut while ScalingLimited=False, indicating no actual scaling constraint. The alert intent is to detect HPAs that cannot scale enough to meet demand but it fires even when demand is fully met at maxReplicas.

What you expected to happen?

The alert should only fire when the HPA is genuinely blocked from scaling further, i.e. when ScalingLimited=True.

How to reproduce it?

Without KEDA:

1. Create a Deployment with an HPA: minReplicas: 1, maxReplicas: 2, scaling on any metric
2. Ensure the desired replica count stabilizes at 2 (e.g. metric consistently above threshold for 1 replica)
3. Verify kube_horizontalpodautoscaler_status_condition{condition="ScalingLimited",status="true"} == 0
4. After 15 minutes, KubeHpaMaxedOut fires despite the HPA being satisfied

With KEDA:

1. Create a ScaledObject with minReplicaCount: 0, maxReplicaCount: 2 using a cron trigger
2. During the active cron window, the deployment runs at 2 replicas: exactly what the scaler wants
3. After 15 minutes, KubeHpaMaxedOut fires

Enter the changed values of values.yaml?

No custom values, reproduces with default chart configuration.

Enter the command that you execute and failing/misfunctioning.

# Check ScalingLimited condition on HPAs that are at maxReplicas
kubectl get hpa -A
kubectl get hpa <name> -n <namespace> -o jsonpath='{.status.conditions}'

Anything else we need to know?

The kube_horizontalpodautoscaler_status_condition metric exposed by kube-state-metrics is the correct signal. Adding a ScalingLimited=True check eliminates the false positive without changing behavior for genuine alerts:

(
  kube_horizontalpodautoscaler_status_current_replicas{job="kube-state-metrics", namespace=~".*"}
    ==
  kube_horizontalpodautoscaler_spec_max_replicas{job="kube-state-metrics", namespace=~".*"}
)
and on (namespace, horizontalpodautoscaler, cluster) (
  kube_horizontalpodautoscaler_spec_max_replicas{job="kube-state-metrics", namespace=~".*"}
    !=
  kube_horizontalpodautoscaler_spec_min_replicas{job="kube-state-metrics", namespace=~".*"}
)
and on (namespace, horizontalpodautoscaler, cluster) (
  kube_horizontalpodautoscaler_status_condition{condition="ScalingLimited", status="true"} == 1
)