feat: add ScaleGate field to ScaleStrategy for fast fallback on quota exceeded - #5094
Open
gangli113 wants to merge 1 commit into
Open
feat: add ScaleGate field to ScaleStrategy for fast fallback on quota exceeded#5094gangli113 wants to merge 1 commit into
gangli113 wants to merge 1 commit into
Conversation
… exceeded Add a new optional ScaleGate []string field to the ScaleStrategy struct in the v1 RayCluster API. External controllers (e.g. Kueue) append a scaling gate such as "kueue.k8s.io/quota-exceeded" to a worker group's scaleGate when it cannot scale up due to insufficient quota. The Ray Autoscaler observes a non-empty scaleGate and falls back to a lower-priority worker group immediately, instead of waiting for the allocate status timeout. KubeRay defines the field but does not read or write it. The operator preserves the value across reconciles by muting it in GenerateHashWithoutReplicasAndWorkersToDelete, so an external write does not trigger a new RayCluster preparation. Regenerated CRDs, deepcopy, applyconfiguration, Helm chart CRDs, and API reference docs via make manifests generate + make helm. Signed-off-by: Gang Li <ganglica@google.com>
gangli113
requested review from
MortalHappiness,
andrewsykim,
kevin85421 and
rueian
as code owners
August 5, 2026 23:34
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why are these changes needed?
This change adds a new optional
ScaleGate []stringfield to theScaleStrategystruct in the v1RayClusterAPI.When KubeRay runs together with Kueue, quota limits are enforced through scheduling gates. If a worker group exceeds its quota, Kueue keeps the newly created Pods gated (
spec.schedulingGatescontainskueue.x-k8s.io/admission) and marks the WorkloadQuotaReserved=False. From the Ray Autoscaler's point of view, the gated Pod looks like it is simply launching slowly rather than failing, so the request transitions toALLOCATEDand the autoscaler waits forRAY_AUTOSCALER_RECONCILE_ALLOCATE_STATUS_TIMEOUT_S(default 60 minutes) before attempting fallback. This is an unacceptable delay for time-sensitive jobs.ScaleGateprovides a group-level signal carried in the RayCluster spec itself. An external controller (e.g. Kueue) appends a gate such askueue.k8s.io/quota-exceededto the affected worker group'sscaleStrategy.scaleGatewhen it cannot admit the group due to insufficient quota. The Ray Autoscaler observes a non-emptyscaleGateand immediately falls back to a lower-priority worker group (see Priority-Aware Worker Group Selection), bypassing the 60-minute allocate timeout. Once quota becomes available, the controller clears the gate.KubeRay defines the field but does not read or write it. To ensure an external write does not trigger a new RayCluster preparation, the operator preserves the value across reconciles by muting
scaleGateinGenerateHashWithoutReplicasAndWorkersToDelete(the same treatment asworkersToDelete, tolerations, and scheduling gates).This is the KubeRay half of the "Fast Fallback on Quota Exceeded" design; the corresponding Ray Autoscaler and Kueue changes consume/produce this field.
Included in this PR:
ScaleStrategy.ScaleGatefield inray-operator/apis/ray/v1/raycluster_types.go.scaleGatemuted inGenerateHashWithoutReplicasAndWorkersToDelete(ray-operator/controllers/ray/utils/util.go).zz_generated.deepcopy.go, applyconfiguration, CRDs (config/crd/bases/*), Helm chart CRDs, and the API reference docs viamake manifests generate+make helm.Related issue number
Labels
doc-updates-requiredlabel.breaking-changelabel.Checks