Skip to content

[LLM][3/N] Update Ray LLM to consume TPUAcceleratorConfig from Serve - #63216

Closed
ryanaoleary wants to merge 21 commits into
ray-project:masterfrom
ryanaoleary:e-llm-accelerator-config
Closed

[LLM][3/N] Update Ray LLM to consume TPUAcceleratorConfig from Serve#63216
ryanaoleary wants to merge 21 commits into
ray-project:masterfrom
ryanaoleary:e-llm-accelerator-config

Conversation

@ryanaoleary

@ryanaoleary ryanaoleary commented May 8, 2026

Copy link
Copy Markdown
Contributor

Description

This PR enables Ray LLM to use the new TPUAcceleratorConfig from Serve implemented in #63179, replacing the deferred PG logic for scheduling workers. This resolves a critical bug where deferred placement groups left LLMServer replicas scheduled without accelerator constraints.

Instead of deferring PG creation, we now pass the hardware topology directly to Serve. This ensures replicas are correctly constrained, allowing Serve to take ownership of the placement group lifecycle and successfully clean up temporary head placement groups upon readiness.

Key Changes:

  • Updated LLMServer.get_deployment_options to translate TPUConfig into Serve TPUAcceleratorConfig` and inject it into the Serve decorator options.
  • Refactored multi-host TPU tests to pass options via decorator, and added robust assertions verifying successful head-PG teardown.

This PR includes the changes from #63177 and #63179 so that the integration tests work. This PR only modifies the following files, the rest are from the PR this is rebased on:

  • python/ray/llm/_internal/serve/core/configs/accelerators.py
  • python/ray/llm/_internal/serve/core/server/llm_server.py
  • python/ray/llm/tests/serve/cpu/deployments/conftest.py
  • python/ray/llm/tests/serve/cpu/deployments/llm/test_llm_engine_tpu.py
  • python/ray/llm/tests/serve/cpu/deployments/llm/test_llm_server.py
  • python/ray/serve/tests/test_accelerator_config.py

Related issues

#57137

Additional information

Optional: Add implementation details, API changes, usage examples, screenshots, etc.

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

community-contribution Contributed by the community llm serve Ray Serve Related Issue stale The issue is stale. It will be closed within 7 days unless there are further conversation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants