[DO NOT MERGE] repro TestRayServiceIncrementalUpgradeWithLocust RPS<=400 flakiness

[justinyeh1995]

Why are these changes needed?

Reproducing RPS <= 400 flakiness on buildkite with 25 distinct jobs on TestRayServiceIncrementalUpgradeWithLocust and TestRayServiceIncrementalUpgradeRollbackMatrixWithLocust to observe the fail-before cases.

some ci failing cases related to this issue.

https://buildkite.com/ray-project/ray-ecosystem-ci-kuberay-ci/builds/16269/list?sid=019fc713-155e-4397-965e-1f48c754c305&tab=output -> shows failures happened in TestRayServiceIncrementalUpgradeWithLocust:BlueGreen、AggressiveGradual、ConservativeGradual

https://buildkite.com/ray-project/ray-ecosystem-ci-kuberay-ci/builds/16394/list?jid=019fd75f-afcb-4932-981c-e6ea5d32bbe0&tab=output -> shoes failures happened in TestRayServiceIncrementalUpgradeRollbackMatrixWithLocust:ThirdSpec、EarlyRollback、FastRollback

Related issue number

related to #4782

Fail Before Repro

commit fc20caa and the two builds above showed this flakiness. so it is worth the fix.

notice that max_ongoing_request is ignored (misplaced, Serve default applied)

---

Checkpoint 1: no changes introduced. Ran it 25 times.

fc20caa 1/25 fails due to the RPS <= 400 issue

a4fd846 0/25 fails due to the RPS issue, but three jobs failed due to the clean up issue mention below. #5098 (comment)

---

Checkpoint 2:

Fixed the canonical form, max_ongoing_requests is not part of the autoscalingConfig.

• Tune target_on_going to 10 and max_ongoing_target to 20.

  • the rationale is we have a set 10 concurrent requests all the time, why don't we allow it on one replicas to reduce the latency and bump max_ongoing_target to way more than 10 so we reduce queue time.

• 376d838 6 out of 25 failed due to the RPS issue (not sure why but the observation is the truth)
  1 out of 25 failed due to clean up issue.

---

Checkpoint 3:

• Tune target_on_going to 2 (the original value) and max_ongoing_target to 20.

• 7507d84 1/25 is related to the RPS issue. another job failed due to the clean up issue. [DO NOT MERGE] repro TestRayServiceIncrementalUpgradeWithLocust RPS<=400 flakiness #5098 (comment)

• 565aad5 4/25 failed due to the RPS issue.

---

Checkpoint 4.

Keep the canonical format change, but revert the max_ongoing_requests back to the original value, 6.

f6a9042 1/25 failed with the RPS check.
b74f79a 0/25 failed due to the RPS check. 2/25 failed due to the flakiness due to the issue mention below.

Results: 1/50 failed to meet the RPS >= 400 requirement. (not a regression from the current set up but fix the canonical format)

---

Suggestion.

After conducting the experiment, I would suggest max_ongoing_requests stays at the intended 6 since there is no regression and it fits the orignal intent more. The fix will fix the canonical format issue, and use the value, 6.

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{Reviewed by Cursor Bugbot for commit 2802941. Configure here.}

[justinyeh1995]

there is a second, unrelated flaky failure in https://buildkite.com/ray-project/ray-ecosystem-ci-kuberay-ci/builds/16452/list?jid=019fdc92-39a1-4bbd-8b37-56c1182ddbd6&tab=output where Locust exits with code 2 during its SIGINT-triggered shutdown.

[2026-08-07T14:40:14Z]           File "/home/ray/anaconda3/lib/python3.10/site-packages/locust/stats.py", line 588, in get_response_time_percentile
--
[2026-08-07T14:40:14Z]             return calculate_response_time_percentile(self.response_times, self.num_requests, percent)
[2026-08-07T14:40:14Z]           File "/home/ray/anaconda3/lib/python3.10/site-packages/locust/stats.py", line 145, in calculate_response_time_percentile
[2026-08-07T14:40:14Z]             num_of_request = int(num_requests * percent)
[2026-08-07T14:40:14Z]         KeyboardInterrupt
...
[2026-08-07T14:41:19Z]     rayservice_incremental_upgrade_test.go:671: Locust load test failed: command terminated with exit code 

gevent is delivering deliberate SIGINT into Locust's background stats-CSV-writer greenlet mid-write. The test's cleanup

kuberay/ray-operator/test/e2eincrementalupgrade/rayservice_incremental_upgrade_test.go

Lines 661 to 673 in 7f67392

	defer func() {
	LogWithTimestamp(test.T(), "Stopping Locust load test")
	_, _, _ = ExecPodCmdWithError(
	test,
	locustHeadPod,
	common.RayHeadContainer,
	[]string{"pkill", "-SIGINT", "-f", "locust --headless"},
	)
	LogWithTimestamp(test.T(), "Waiting for Locust load test goroutine to finish")
	if err := eg.Wait(); err != nil && !test.T().Failed() {
	test.T().Errorf("Locust load test failed: %v", err)
	}
	}()

then treats that nonzero exit as a hard test failure even when every real assertion in the test had already passed by that point.

A likely fix is to drop that error check since warmupLocust's own timeout already catches the case where Locust genuinely never starts/ramps up. Not fixing this here since it's unrelated to serveConfigV2 tuning. Will test it in another follow-up. this still requires further investigations.