Skip to content

feat: add ScaleGate field to ScaleStrategy for fast fallback on quota exceeded - #5094

Open
gangli113 wants to merge 1 commit into
ray-project:masterfrom
gangli113:fast-fallback
Open

feat: add ScaleGate field to ScaleStrategy for fast fallback on quota exceeded#5094
gangli113 wants to merge 1 commit into
ray-project:masterfrom
gangli113:fast-fallback

Conversation

@gangli113

Copy link
Copy Markdown
Contributor

Why are these changes needed?

This change adds a new optional ScaleGate []string field to the ScaleStrategy struct in the v1 RayCluster API.

When KubeRay runs together with Kueue, quota limits are enforced through scheduling gates. If a worker group exceeds its quota, Kueue keeps the newly created Pods gated (spec.schedulingGates contains kueue.x-k8s.io/admission) and marks the Workload QuotaReserved=False. From the Ray Autoscaler's point of view, the gated Pod looks like it is simply launching slowly rather than failing, so the request transitions to ALLOCATED and the autoscaler waits for RAY_AUTOSCALER_RECONCILE_ALLOCATE_STATUS_TIMEOUT_S (default 60 minutes) before attempting fallback. This is an unacceptable delay for time-sensitive jobs.

ScaleGate provides a group-level signal carried in the RayCluster spec itself. An external controller (e.g. Kueue) appends a gate such as kueue.k8s.io/quota-exceeded to the affected worker group's scaleStrategy.scaleGate when it cannot admit the group due to insufficient quota. The Ray Autoscaler observes a non-empty scaleGate and immediately falls back to a lower-priority worker group (see Priority-Aware Worker Group Selection), bypassing the 60-minute allocate timeout. Once quota becomes available, the controller clears the gate.

KubeRay defines the field but does not read or write it. To ensure an external write does not trigger a new RayCluster preparation, the operator preserves the value across reconciles by muting scaleGate in GenerateHashWithoutReplicasAndWorkersToDelete (the same treatment as workersToDelete, tolerations, and scheduling gates).

This is the KubeRay half of the "Fast Fallback on Quota Exceeded" design; the corresponding Ray Autoscaler and Kueue changes consume/produce this field.

Included in this PR:

  • New ScaleStrategy.ScaleGate field in ray-operator/apis/ray/v1/raycluster_types.go.
  • scaleGate muted in GenerateHashWithoutReplicasAndWorkersToDelete (ray-operator/controllers/ray/utils/util.go).
  • Regenerated zz_generated.deepcopy.go, applyconfiguration, CRDs (config/crd/bases/*), Helm chart CRDs, and the API reference docs via make manifests generate + make helm.

Related issue number

Labels

  • If this PR has user-facing changes that require documentation updates at release time, I have added the doc-updates-required label.
  • If this PR contains breaking changes, I have added the breaking-change label.

Checks

  • I've made sure the tests are passing.
  • Testing Strategy
    • Unit tests
    • Manual tests
    • This PR is not tested :(

… exceeded

Add a new optional ScaleGate []string field to the ScaleStrategy struct in
the v1 RayCluster API. External controllers (e.g. Kueue) append a scaling
gate such as "kueue.k8s.io/quota-exceeded" to a worker group's scaleGate
when it cannot scale up due to insufficient quota. The Ray Autoscaler
observes a non-empty scaleGate and falls back to a lower-priority worker
group immediately, instead of waiting for the allocate status timeout.

KubeRay defines the field but does not read or write it. The operator
preserves the value across reconciles by muting it in
GenerateHashWithoutReplicasAndWorkersToDelete, so an external write does
not trigger a new RayCluster preparation.

Regenerated CRDs, deepcopy, applyconfiguration, Helm chart CRDs, and API
reference docs via make manifests generate + make helm.

Signed-off-by: Gang Li <ganglica@google.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant