KubeHpaMaxedOut should consider ScalingLimited

[thewisenerd]

What is missing?
KubeHpaMaxedOut does not consider ScalingLimited when on max replicas

Why do we need it?
we have these weird HPAs (max=2, min=1) and others which keep triggering KubeHpaMaxedOut
but HPA itself isn't looking to scale up beyond.

why we probably should not do this:

• HPA is usually? a gradual ramp
• logically if a HPA reaches max replicas it probably gonna want to scale up further
• we may reduce the time-to-alert-signal (or whatever's the technical term here) for a valid use-case

Environment

• kube-prometheus version:

  ce1ae0fdefb7545269831e5435c9bcd91d4acd7d

kube-prometheus/manifests/kubernetesControlPlane-prometheusRule.yaml

Line 254 in ce1ae0f

- alert: KubeHpaMaxedOut

Anything else we need to know?:

• on our prod cluster, we had 7 noisy alerts because of this (not all cases max=2,min=1)
• we're not really on the latest and greatest (kube-prometheus-stack 63.1.0), but the following patch's what we've used

expr: |-
  (
    kube_horizontalpodautoscaler_status_current_replicas{job="kube-state-metrics"}
      ==
    kube_horizontalpodautoscaler_spec_max_replicas{job="kube-state-metrics"}
  )
  # upstream chart 82.10.2 introduced this guard in 1c0c69b0a.
  and on (namespace, horizontalpodautoscaler, cluster) (
    kube_horizontalpodautoscaler_spec_max_replicas{job="kube-state-metrics"}
      !=
    kube_horizontalpodautoscaler_spec_min_replicas{job="kube-state-metrics"}
  )
  # avoid false positives when max replicas is the healthy desired count.
  and on (namespace, horizontalpodautoscaler, cluster) (
    kube_horizontalpodautoscaler_status_condition{
      job="kube-state-metrics",
      condition="ScalingLimited",
      status="true"
    } == 1
  )