History Server: collect rotated Ray logs before overwrite

[AdibaAdi]

Why are these changes needed?

Ray rotates logs by default to limit local disk usage. The History Server collector previously relied primarily on collecting the remaining Ray log directory during shutdown or recovery from prev-logs.

For long-running workloads, this means an older rotated backup can be overwritten before the collector uploads it, leaving part of the log history unrecoverable even when object storage is available.

This PR adds live rotated-log collection for both head and worker collectors.

When a completed Ray rotation backup appears, the collector preserves its inode with a hard link before Ray can roll it out of the local backup ring. Each captured generation receives a unique .rotated.<capture-id> name, so successive rotations that reuse filenames such as .1 remain distinct.

Filesystem notifications provide prompt discovery, with periodic reconciliation as a backstop for missed or coalesced events. Captures are staged on the shared Ray filesystem, uploaded asynchronously, and reconstructed from staging after collector restart.

The upload lifecycle keeps pending and successfully uploaded captures distinct, retries transport failures, avoids duplicate uploads, and preserves the existing non-rotated log collection path.

Session transitions also preserve the session and Ray node identity associated with each capture. Known outgoing sessions are relocated individually before the existing broad leftover sweep, preventing unrelated sessions from being attributed to the same node ID.

To prevent rotated-log staging from growing indefinitely during an object-store outage, backpressure is based on bytes retained solely because the collector owns the final hard link to an inode. Captures that Ray still owns do not count toward this limit. New capture pauses at a 1 GiB high watermark and resumes below 512 MiB while uploads, retries, promotions, and releases continue.

Related issue number

Closes #4830

Labels

• If this PR has user-facing changes that require documentation updates at release time, I have added the doc-updates-required label.
• If this PR contains breaking changes, I have added the breaking-change label.

Checks

• I've made sure the tests are passing.
• Testing Strategy
  • Unit tests
  • Manual tests
  • This PR is not tested :(

Validation

go test -race -count=1 ./pkg/collector/logcollector/runtime/logcollector
go test -race -count=1 ./pkg/collector/...
go test -count=1 ./pkg/storage/...
go test -count=1 ./pkg/utils/...
go vet ./...
go build ./...
git diff --check upstream/master...HEAD

The rotated-log package was also exercised repeatedly under the race detector.

Backpressure regression coverage verifies that:

• captures still linked by Ray do not count toward retained-byte pressure
• captures Ray has rolled out of its backup ring do count
• sustained storage failure pauses new intake without evicting already captured logs
• upload and retry work continues while intake is paused
• successful recovery releases retained blocks and resumes intake
• retained-byte accounting is reconstructed from filesystem link counts after restart

Manual test instructions

I validated the collector and History Server flow locally using a native arm64 Kind cluster, the KubeRay operator, Ray 2.52.0, and MinIO.

For the test, Ray rotation was configured with:

RAY_ROTATION_MAX_BYTES=65536
RAY_ROTATION_BACKUP_COUNT=5

I ran one Ray task pinned to the head node and another pinned to the worker node. Each task emitted 6,400 uniquely marked log lines, producing substantially more rotations than Ray's five-file local backup ring could retain.

Observed results:

• 25 marked rotated objects were uploaded for the head node
• 25 marked rotated objects were uploaded for the worker node
• all 50 objects were stored under the expected session and Ray node IDs
• no head/worker cross-attribution was observed
• the objects were present in MinIO while both original Ray pods were still Running and Ready
• Ray and collector container restart counts remained zero
• MinIO preserved workload sequences 1-6399 for both nodes
• the final live Ray backup rings retained only approximately sequences 5120-6400

This confirms that rotations already overwritten from Ray's local backup ring remained recoverable from object storage.

After deleting the RayCluster, all 50 marked rotated objects remained in MinIO.

Using the archived History Server path, I then verified that:

• the archived session was discovered successfully
• all 25 marked rotated logs were listed for the head node
• all 25 marked rotated logs were listed for the worker node
• a marked head rotation could be downloaded with the expected head marker and no worker marker
• a marked worker rotation could be downloaded with the expected worker marker and no head marker
• an ordinary non-rotated raylet.out log could still be listed and downloaded successfully

The live History Server path continues to proxy Ray's dashboard and therefore exposes Ray's local .1 through .5 backup filenames. Once the cluster is archived, the object-storage path exposes the collector's .rotated.<capture-id> objects.

Scope of manual validation

These results are from local Kind + MinIO integration testing and are not production cloud-storage validation.

The existing History Server line-oriented download path omits a final trailing newline when serving a log file. Marker content was preserved; this behavior is pre-existing and unrelated to rotated-log capture.

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{Reviewed by Cursor Bugbot for commit 2d45c6f. Configure here.}