[RayJob] Don't fail a running Ray job when the submitter exits

[1fanwang]

Why are these changes needed?

KubeRay treats the submitter finishing as evidence the Ray job is terminal. It isn't. ray job logs --follow, the submitter's last command in K8sJobMode and SidecarMode, returns 0 on any non-abnormal WebSocket close (sdk.py):

elif msg.type == aiohttp.WSMsgType.CLOSED:
    if ws.close_code == aiohttp.WSCloseCode.ABNORMAL_CLOSURE:
        raise RuntimeError(...)
    break                                  # normal close: no error
...
return _log_job_status(client, job_id)     # only logs; never exits non-zero

checkSubmitterFinishedTimeoutAndUpdateStatusIfNeeded then waits 30s, overwrites RUNNING with FAILED, and under shutdownAfterJobFinishes deletes the cluster from under the driver. It also produces the inconsistent status pair in #2314.

Ask the cluster instead of guessing

The timeout can't just be removed: #4091 added it for a status that never resolves after a head restart. Both cases arrive with JobStatus: RUNNING.

A job status is only current while the node running its driver is alive. Ray already reports driver_node_id on the job; this deserializes it and adds an IsNodeAlive check against the dashboard's node list:

if rayJobIsActive(rayJob) {
    alive, err := rayJobDriverIsAlive(ctx, dashboardClient, jobInfo)
    ...
    if alive { return false, nil }   // job is genuinely running; leave it
}

The check moves after the job-info poll, where jobInfo and the client are already available. It compares no timestamps and reads no Pod lifecycle, so it is unaffected by readiness probes flapping, by a sidecar submitter dying, and by whether the head restarted before or after the submitter exited, earlier revisions got each of those wrong, and Bugbot caught them.

Scoped to RUNNING on purpose: Ray assigns driver_node_id when it schedules the job's supervisor, so a PENDING job has none to check against. Those keep today's behavior rather than being guessed at. An unreachable dashboard returns an error and requeues instead of counting as a dead driver, and a jobInfo that has already gone terminal is left to the normal handling.

The e2e test RayJob has exceed SubmitterFinishedTimeout asserted the old behaviour; it is rewritten to assert the job survives, and renamed.

Related issue number

Addresses #4921. Related: #2314.

Labels

• doc-updates-required
• breaking-change

Checks

• I've made sure the tests are passing.
• Testing Strategy
  • Unit tests
  • Manual tests

Manual test instructions

kind, operator built from this branch, rayproject/ray:2.55.1.

The bug, on unpatched master. Stock submitter, nothing simulated, kill the head's DashboardAgent mid-job:

[02:22:08Z] killed DashboardAgent on the head
[02:22:24Z] submitter=Succeeded/0 | js=RUNNING dep=Running
[02:22:54Z] submitter=Succeeded/0 | js=FAILED  dep=Failed
            reason=JobDeploymentStatusTransitionGracePeriodExceeded

Its own last two lines before exiting 0:

INFO cli.py:91 -- Status for job '...': RUNNING
INFO cli.py:93 -- Status message: Job is currently running.

One run died 47s into a 240s job; kept alive for inspection, that job printed its completion marker and GCS reported success while the CR said Failed.

Healthy cluster, submitter exits 0, patched. The job outlives the grace period and finishes:

[07:09:51Z] js=RUNNING   dep=Running
...
[07:13:32Z] js=RUNNING   dep=Running
[07:13:52Z] js=SUCCEEDED dep=Complete

Driver node gone, patched, still fails. Killing the agent degrades the node, so it stops reporting ALIVE and the timeout fires as before:

[07:07:58Z] killed DashboardAgent
[07:08:29Z] submitter=Succeeded/0 | js=RUNNING dep=Running  headRestarts=0
[07:08:59Z] submitter=Succeeded/0 | js=FAILED  dep=Failed
            reason=JobDeploymentStatusTransitionGracePeriodExceeded

Same for a head restart, the driver's node is absent from the live set afterwards:

old driver node: 3bbbda71075d7929d056881d8921592b969874dd65ea3ee2d2067ca8
ALIVE nodes:    ['cb015fd19db8268abf47d287acf4c588ae1bba9f63bb971e1a06272d']
old driver node still ALIVE? False

Unit tests

=== RUN   TestCheckSubmitterFinishedTimeoutAndUpdateStatusIfNeeded
    --- PASS: .../submitter_still_running
    --- PASS: .../grace_period_not_yet_elapsed
    --- PASS: .../job_RUNNING_on_a_live_driver_node_is_left_alone
    --- PASS: .../job_PENDING_on_a_live_driver_node_is_left_alone
    --- PASS: .../job_RUNNING_but_the_driver_node_is_gone_times_out
    --- PASS: .../job_RUNNING_with_no_driver_node_reported_times_out
    --- PASS: .../job_never_observed_times_out
--- PASS: TestCheckSubmitterFinishedTimeoutAndUpdateStatusIfNeeded (0.00s)

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{Reviewed by Cursor Bugbot for commit f01abb6. Configure here.}