[RayJob] Don't fail a running Ray job when its submitter dies

[1fanwang]

Why are these changes needed?

Stacked on #5099 — the first commit here is that PR. Review only [RayJob] Don't fail a running Ray job when its submitter dies; the diff collapses once #5099 lands.

A submitter killed by the infrastructure — OOMKill, eviction, preemption — exhausts its backoff and fails the submitter Job, and KubeRay reads that as the RayJob failing. The driver runs on the head and is unaffected: the job carries on, often succeeds, and the CR reports SubmissionFailed forever. That is the contradictory status pair in #2314, open since 2024.

#2317 asked for this; its repro script is "delete the submitter pod while a job is running". The submission-ID half was fixed by #2579, but that script still discards a healthy job on master.

Once the job has been observed RUNNING the submission plainly succeeded, so SubmissionFailed is wrong twice over: it discards a live job and blames a submission that worked. Those are left to #5099's submitter-finished timeout, which decides on live cluster state:

shouldUpdate, condition = checkK8sJobStatus(job)
shouldUpdate = shouldUpdate && !rayJobIsRunning(rayJob)

A failed submitter Job is terminal, so getJobFinishedCondition sets finishedAt and that path always has a reference point. No extra API calls here — it is a field read.

A submission that never produced a running job still fails immediately and keeps its SubmissionFailed or AppFailed reason. Both SidecarMode submitter-failure paths get the same treatment.

Related issue number

Closes #2314. Finishes the second half of #2317.

Labels

• doc-updates-required
• breaking-change

Checks

• I've made sure the tests are passing.
• Testing Strategy
  • Unit tests
  • Manual tests

Manual test instructions

kind, operator built from this branch, rayproject/ray:2.55.1, a 240s entrypoint ending in a marker. Run #2317's script: delete the submitter pod mid-job.

Before, on master — the submitter Job fails, and the CR contradicts GCS permanently:

===== GROUND TRUTH: Ray GCS =====
Job 'submitter-kill-repro-hkr46' succeeded

----- driver output -----
tick 46
tick 47
RAY_JOB_REALLY_SUCCEEDED

===== RayJob CR =====
  jobStatus            = RUNNING
  jobDeploymentStatus  = Failed
  reason               = SubmissionFailed
  message              = Job submission has failed. Reason: BackoffLimitExceeded. Message: Job has reached the specified backoff limit

After — same manifest, same kill, submitter Job still Failed=True:

[23:41:05Z] killed submitter pod
[23:41:32Z] js=RUNNING   dep=Running  | k8sJob: FailureTarget=True Failed=True
[23:43:12Z] js=RUNNING   dep=Running  | k8sJob: FailureTarget=True Failed=True
[23:44:28Z] js=RUNNING   dep=Running  | k8sJob: FailureTarget=True Failed=True
[23:44:53Z] js=SUCCEEDED dep=Complete | k8sJob: FailureTarget=True Failed=True

Realistic config, side by side. Head + 2 workers, a driver fanning real Ray tasks to both workers and checkpointing each step, shutdownAfterJobFinishes: true, ttlSecondsAfterFinished: 0, backoffLimit: 0. Same trigger: kill the head's dashboard JobHead module mid-job.

On master, the cluster is deleted eight seconds later, a third of the way in:

[16:08:58Z] killed JobHead        | progress: step 12/40
[16:08:58Z] dep=Running             clusters=1 runningPods=4
[16:09:06Z] dep=Failed/SubmissionFailed  clusters=0 runningPods=0

On this branch, same manifest, same trigger at step 11/40:

[16:11:14Z] js=RUNNING   dep=Running  | step 14/40
[16:12:40Z] js=RUNNING   dep=Running  | step 31/40
[16:13:29Z] js=RUNNING   dep=Running  | step 40/40
[16:13:41Z] js=SUCCEEDED dep=Complete | JOB_COMPLETED_ALL_STEPS, cluster then cleaned up

Triage note: with the default backoffLimit: 2 a non-zero exit is retried and the retry reattaches to the running job, which masked this twice. The damage needs an attempt with no retry left (#2314).

SidecarMode, same two runs. The guard applies there too:

master:  [18:04:18Z] killed JobHead | step 10/40 | dep=Running   clusters=1
         [18:04:28Z] dep=Failed/SubmissionFailed              clusters=0

branch:  [17:59:48Z] js=RUNNING   dep=Running | sidecar exit=1 | step 10/40
         [18:02:19Z] js=RUNNING   dep=Running | sidecar exit=1 | step 40/40
         [18:02:31Z] js=SUCCEEDED dep=Complete

A genuinely dead job is still failed. The guard keys on RUNNING, so the worry is a stranded job whose cluster is gone. Force-deleting the head mid-job, guard active and CR reading RUNNING, still goes terminal in seconds:

[06:38:46Z] force-deleted the head pod
[06:39:09Z] dep=Failed reason=AppFailed
            "Submitter completed but Ray job not found in RayCluster."

Killing the head's DashboardAgent is likewise still terminal: the node goes DEAD, and #5099's driver-liveness check lets the submitter-finished timeout fire as before.

Unit tests

=== RUN   TestCheckSubmitterAndUpdateStatusIfNeeded_SidecarMode
    --- PASS: .../job_running_is_left_to_the_submitter-finished_timeout
    --- PASS: .../job_never_observed_is_a_genuine_submission_failure
    --- PASS: .../job_already_failed_keeps_the_application_failure_reason
--- PASS: TestCheckSubmitterAndUpdateStatusIfNeeded_SidecarMode (0.04s)
=== RUN   TestCheckSubmitterAndUpdateStatusIfNeeded_K8sJobMode
    --- PASS: .../job_running_is_left_to_the_submitter-finished_timeout
    --- PASS: .../job_never_observed_is_a_genuine_submission_failure
    --- PASS: .../job_already_failed_keeps_the_application_failure_reason
--- PASS: TestCheckSubmitterAndUpdateStatusIfNeeded_K8sJobMode (0.04s)

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{Reviewed by Cursor Bugbot for commit 567335e. Configure here.}