[CI] main2mian v0.24.0 by zhangxinyuehfad · Pull Request #11227 · vllm-project/vllm-ascend

zhangxinyuehfad · 2026-07-01T01:32:51Z

What this PR does / why we need it?

This PR upgrades the verified vLLM main commit to ee0da84ab9e04ac7610e28580af62c365e898389 (v0.24.0 tag) and adapts vLLM Ascend to upstream API changes introduced after the previous verified commit.
Changes

Changes

`examples/offline_data_parallel.py`

Slice ASCEND_RT_VISIBLE_DEVICES according to the local DP rank before creating the LLM instance.
Fall back to torch.npu.device_count() when ASCEND_RT_VISIBLE_DEVICES is not set.
Keep application-level DP compatible with the upstream removal of automatic device isolation.
Upstream source: vllm#45026.

`tests/e2e/conftest.py`

Slice ASCEND_RT_VISIBLE_DEVICES for each DP rank before creating LLM() instances or worker processes.
Fall back to torch.npu.device_count() when the environment variable is unavailable.
Ensure each test process uses an independent NPU subset after upstream removed automatic device isolation.
Upstream source: vllm#45026.

`tests/ut/quantization/methods/test_w4a16_mxfp4.py`

Skip the test to match the upstream behavior.

`tests/ut/spec_decode/test_speculators_vwn_eagle3.py`

Mock vllm.v1.attention.selector._cached_get_attn_backend() for non-NPU unit tests.
Avoid attention backend initialization failures introduced by the updated v0.24.0 initialization path.
Keep spec decode unit tests runnable without physical NPU devices.

`vllm_ascend/_310p/worker_310p.py`

Use MemorySnapshot(device=device) for non-0.23.0 releases.
Align the 310P worker with the updated worker implementation.
Related upstream changes:
- vllm#30868

`vllm_ascend/patch/platform/patch_dp_device_ids.py`

Add a patch for get_physical_gpu_ids_for_local_dp_rank().
Support pre-sharded ASCEND_RT_VISIBLE_DEVICES by avoiding the upstream DP-rank offset when device isolation is already handled externally.
Prevent IndexError for application-level DP after the upstream device isolation changes.
Wire the patch into the platform initialization for non-0.23.0.
Upstream source: vllm#45026.

`vllm_ascend/patch/platform/patch_kv_cache_utils.py`

Patch _get_kv_cache_config_deepseek_v4() on v0.23.0.
Patch _get_kv_cache_config_packed() for non-0.23.0 by reusing the existing non-packed implementation.
Avoid excessive KV-cache allocation introduced by the packed KV-cache layout and prevent NPU OOM.
Upstream source: vllm#46205.

`vllm_ascend/patch/worker/init.py`

Monkey-patch vllm.v1.utils.CpuGpuBuffer.__init__() to remap torch.uint64 to torch.int64 on non-0.23.0.
Preserve the original behavior for all other data types.
Avoid the Ascend runtime failure caused by unsupported DT_UINT64 in aclnnInplaceZero.
Upstream source: vllm#44665.

`vllm_ascend/spec_decode/ngram_proposer.py`

Preserve the original initialization path on v0.23.0.
Delay the Ascend-specific propose() implementation during base-class initialization on newer releases.
Avoid eager GPU-only initialization introduced upstream while preserving the original behavior afterward.
Upstream source: vllm#29184.

`vllm_ascend/worker/utils.py`

Correct the type annotation of kernel_block_sizes to match its actual nested-list usage.
Keep the implementation consistent with the upstream interface.

`vllm_ascend/worker/worker.py`

Adjust the local device rank for single-node application-level DP on non-Ray backends.
Match the upstream device assignment behavior after automatic device isolation was removed.
Ensure each DP worker is mapped to the correct NPU subset.
Upstream source: vllm#45026.

`vllm_ascend/init.py`

Reorganize imports to eliminate circular dependencies during package initialization.

Does this PR introduce any user-facing change?

How was this patch tested?

vLLM version: v0.23.0
vLLM main: vllm-project/vllm@b9a7cd4

gemini-code-assist · 2026-07-01T01:33:19Z

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request updates the vllm_ascend repository to support vLLM version 0.24.0. It introduces robust conditional logic to handle breaking changes in upstream vLLM APIs, ensuring that the Ascend NPU platform remains compatible with both older (0.23.0) and newer versions. The changes span across tool call parsing, speculative decoding, model runner initialization, and graph management.

Highlights

Version Compatibility: Implemented version-specific logic using vllm_version_is('0.23.0') to maintain compatibility with upstream vLLM API changes.
API Adaptations: Updated tool parsers, speculative decoding logic, and graph management components to accommodate upstream API removals and signature changes.
Infrastructure and Testing: Updated CI configurations and added conditional test skipping to ensure stability across different vLLM versions.

New Features

🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Ignored Files

Ignored by pattern: .github/workflows/** (1)
- .github/workflows/pr_test.yaml

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

github-actions · 2026-07-01T01:34:08Z

👋 Hi! Thank you for contributing to the vLLM Ascend project. The following points will speed up your PR merge:‌‌

A PR should do only one thing, smaller PRs enable faster reviews.
Every PR should include unit tests and end-to-end tests ‌to ensure it works and is not broken by other future PRs.
Write the commit message by fulfilling the PR description to help reviewer and future developers understand.

If CI fails, you can run linting and testing checks locally according Contributing and Testing.

gemini-code-assist · 2026-07-01T01:49:04Z

Warning

Gemini encountered an error creating the review. You can try again by commenting /gemini review.

github-actions · 2026-07-01T10:38:41Z

This pull request has conflicts, please resolve those before we can evaluate the pull request.

github-actions · 2026-07-02T07:02:57Z

This pull request has conflicts, please resolve those before we can evaluate the pull request.

MengqingCao · 2026-07-04T12:34:24Z


-    # CUDA_VISIBLE_DEVICES for each DP rank is set automatically inside the
-    # engine processes.
+    # vLLM v0.24.0 (PR #45026) stopped isolating devices per worker process


I think we should add _build_device_ids in vllm/entrypoints/openai/dp_supervisor.py in the example.

…BLE_DEVICES unset vLLM v0.24.0 (PR #45026) removed per-process device isolation. For application-level DP (separate LLM() per rank), both ranks now see all NPU devices, causing HCCL MoE all-to-all communicators to fail with same physical device errors. Auto-discover device count via torch.npu.device_count() when the env var is absent, and assign each DP rank a distinct slice. Signed-off-by: hfadzxy <starmoon_zhang@163.com>

MengqingCao · 2026-07-04T12:50:52Z

+        self.assertEqual(layer.w13_weight.shape, (8, 128, 256))
        self.assertEqual(layer.w13_weight_scale.shape, (8, 4, 256))
-        self.assertEqual(layer.w2_weight.shape, (8, 256, 16))
+        self.assertEqual(layer.w2_weight.shape, (8, 256, 128))
        self.assertEqual(layer.w2_weight_scale.shape, (8, 8, 128))
+        self.assertEqual(layer.w13_weight.dtype, torch.float32)
+        self.assertEqual(layer.w2_weight.dtype, torch.float32)


why do we change the assertion check in the test case?

This pr skips this use case and notes subsequent fixes.No need to modify it now

MengqingCao · 2026-07-04T12:58:28Z

    import vllm_ascend.patch.worker.patch_fused_moe  # noqa

+# Ascend NPU does not support DT_UINT64 in aclnnInplaceZero.
+# CpuGpuBuffer with dtype=torch.uint64 (used by MambaCopyBuffers)


It seems more reasonable to patch MambaCopyBuffers instead of CpuGpuBuffer?

Sure, this is more elegant

MengqingCao · 2026-07-04T13:01:14Z

    # [num_speculative_steps]
    synthetic_conditional_rates: torch.Tensor | None = None,
    use_fp64: bool = False,
+    use_block_verification: bool = False,


plz add a todo here as this is a new feature for speculative decoding in vllm, but we already implement the block verify in vllm-ascend before, maybe we should refactor it later

MengqingCao · 2026-07-04T13:03:54Z

    register_model()
+
+
+import vllm_ascend.logger  # noqa: E402, F401


why moving this here?

To avoid circular imports when vLLM's plugin chain triggers during logger loading.

MengqingCao · 2026-07-04T13:11:41Z

+                    if dp_local_rank is None:
+                        dp_local_rank = parallel_config.data_parallel_index
+                    self.local_rank += dp_local_rank * tp_pp_world_size
+


I think you can adapt the init_device just as that of gpu_worker.py https://github.com/vllm-project/vllm/pull/45026/changes?diff=unified#diff-62b35645ec2e0a565f1367fda869f582ca181cdc04d7b8ca36db71218216d6d7

I will align it

Signed-off-by: hfadzxy <starmoon_zhang@163.com>

MengqingCao · 2026-07-05T06:41:43Z

+                # DP local_rank shift below would push local_rank past the
+                # length of the per-rank device list and trip the assert
+                # in this same method. Skip the shift in that case.
+                and parallel_config.assigned_physical_gpu_ids is None


when using ray as the distributed backend, device-ids doesn't have effect, thus let's remove this check.

MengqingCao · 2026-07-05T10:48:32Z

all ci passed in https://github.com/vllm-project/vllm-ascend/actions/runs/28728597438/job/85190525764

### What this PR does / why we need it? This PR upgrades the verified vLLM main commit to ee0da84ab9e04ac7610e28580af62c365e898389 (v0.24.0 tag) and adapts vLLM Ascend to upstream API changes introduced after the previous verified commit. Changes # Changes ### `examples/offline_data_parallel.py` - Slice `ASCEND_RT_VISIBLE_DEVICES` according to the local DP rank before creating the `LLM` instance. - Fall back to `torch.npu.device_count()` when `ASCEND_RT_VISIBLE_DEVICES` is not set. - Keep application-level DP compatible with the upstream removal of automatic device isolation. - Upstream source: [vllm#45026](vllm-project/vllm#45026). --- ### `tests/e2e/conftest.py` - Slice `ASCEND_RT_VISIBLE_DEVICES` for each DP rank before creating `LLM()` instances or worker processes. - Fall back to `torch.npu.device_count()` when the environment variable is unavailable. - Ensure each test process uses an independent NPU subset after upstream removed automatic device isolation. - Upstream source: [vllm#45026](vllm-project/vllm#45026). --- ### `tests/ut/quantization/methods/test_w4a16_mxfp4.py` - Skip the test to match the upstream behavior. --- ### `tests/ut/spec_decode/test_speculators_vwn_eagle3.py` - Mock `vllm.v1.attention.selector._cached_get_attn_backend()` for non-NPU unit tests. - Avoid attention backend initialization failures introduced by the updated v0.24.0 initialization path. - Keep spec decode unit tests runnable without physical NPU devices. --- ### `vllm_ascend/_310p/worker_310p.py` - Use `MemorySnapshot(device=device)` for non-0.23.0 releases. - Align the 310P worker with the updated worker implementation. - Related upstream changes: - [vllm#30868](vllm-project/vllm#30868) --- ### `vllm_ascend/patch/platform/patch_dp_device_ids.py` - Add a patch for `get_physical_gpu_ids_for_local_dp_rank()`. - Support pre-sharded `ASCEND_RT_VISIBLE_DEVICES` by avoiding the upstream DP-rank offset when device isolation is already handled externally. - Prevent `IndexError` for application-level DP after the upstream device isolation changes. - Wire the patch into the platform initialization for non-0.23.0. - Upstream source: [vllm#45026](vllm-project/vllm#45026). --- ### `vllm_ascend/patch/platform/patch_kv_cache_utils.py` - Patch `_get_kv_cache_config_deepseek_v4()` on v0.23.0. - Patch `_get_kv_cache_config_packed()` for non-0.23.0 by reusing the existing non-packed implementation. - Avoid excessive KV-cache allocation introduced by the packed KV-cache layout and prevent NPU OOM. - Upstream source: [vllm#46205](vllm-project/vllm#46205). --- ### `vllm_ascend/patch/worker/__init__.py` - Monkey-patch `vllm.v1.utils.CpuGpuBuffer.__init__()` to remap `torch.uint64` to `torch.int64` on non-0.23.0. - Preserve the original behavior for all other data types. - Avoid the Ascend runtime failure caused by unsupported `DT_UINT64` in `aclnnInplaceZero`. - Upstream source: [vllm#44665](vllm-project/vllm#44665). --- ### `vllm_ascend/spec_decode/ngram_proposer.py` - Preserve the original initialization path on v0.23.0. - Delay the Ascend-specific `propose()` implementation during base-class initialization on newer releases. - Avoid eager GPU-only initialization introduced upstream while preserving the original behavior afterward. - Upstream source: [vllm#29184](vllm-project/vllm#29184). --- ### `vllm_ascend/worker/utils.py` - Correct the type annotation of `kernel_block_sizes` to match its actual nested-list usage. - Keep the implementation consistent with the upstream interface. --- ### `vllm_ascend/worker/worker.py` - Adjust the local device rank for single-node application-level DP on non-Ray backends. - Match the upstream device assignment behavior after automatic device isolation was removed. - Ensure each DP worker is mapped to the correct NPU subset. - Upstream source: [vllm#45026](vllm-project/vllm#45026). --- ### `vllm_ascend/__init__.py` - Reorganize imports to eliminate circular dependencies during package initialization. - vLLM version: v0.23.0 - vLLM main: vllm-project/vllm@b9a7cd4 --------- Signed-off-by: zhangxinyuehfad <starmoon_zhang@163.com> Signed-off-by: hfadzxy <starmoon_zhang@163.com>

zhangxinyuehfad requested review from MengqingCao, Yikun, realliujiaxu, wangxiyuan, whx-sjtu and zzzzwwjj as code owners July 1, 2026 01:32

zhangxinyuehfad added the ready enable e2e test for PR label Jul 1, 2026

github-actions Bot added ci/build module:tests module:ops module:core labels Jul 1, 2026

github-actions Bot added the merge-conflicts label Jul 1, 2026

wangxiyuan force-pushed the zxy_main2main_v0.24.0 branch from 990d8ca to fd8387f Compare July 2, 2026 04:36

github-actions Bot removed the merge-conflicts label Jul 2, 2026

github-actions Bot added the merge-conflicts label Jul 2, 2026

zyz111222 mentioned this pull request Jul 2, 2026

[Hardware][Ascend][Model] Optimize Qwen3-TTS on 310P vllm-project/vllm-omni#4841

Merged

zhangxinyuehfad force-pushed the zxy_main2main_v0.24.0 branch from 2432904 to 14970a8 Compare July 2, 2026 13:17

github-actions Bot removed the merge-conflicts label Jul 2, 2026

zhangxinyuehfad force-pushed the zxy_main2main_v0.24.0 branch from 14970a8 to 3a79707 Compare July 2, 2026 14:06

zhangxinyuehfad removed the ready enable e2e test for PR label Jul 3, 2026

zhangxinyuehfad force-pushed the zxy_main2main_v0.24.0 branch from 3a79707 to 5ccbef3 Compare July 3, 2026 04:15

zhangxinyuehfad added the ready enable e2e test for PR label Jul 3, 2026

zhangxinyuehfad force-pushed the zxy_main2main_v0.24.0 branch 3 times, most recently from f295369 to a282451 Compare July 3, 2026 07:39

zhangxinyuehfad force-pushed the zxy_main2main_v0.24.0 branch 2 times, most recently from 6d6dcf1 to 09ea2dc Compare July 4, 2026 12:18

MengqingCao reviewed Jul 4, 2026

View reviewed changes

zhangxinyuehfad force-pushed the zxy_main2main_v0.24.0 branch from 09ea2dc to f13be37 Compare July 4, 2026 12:37

MengqingCao reviewed Jul 4, 2026

View reviewed changes

Comment thread tests/e2e/conftest.py

MengqingCao reviewed Jul 4, 2026

View reviewed changes

Comment thread tests/ut/core/test_profiling_chunk.py

MengqingCao reviewed Jul 4, 2026

View reviewed changes

zhangxinyuehfad added 3 commits July 4, 2026 21:45

use _build_device_ids from vllm no-v0.23.0

a0702cc

Signed-off-by: hfadzxy <starmoon_zhang@163.com>

revert test_process_weights_transposes_weights

cd7600a

Signed-off-by: hfadzxy <starmoon_zhang@163.com>

replace unit64 to int64 by MambaCopyBuffers

663eeed

Signed-off-by: hfadzxy <starmoon_zhang@163.com>

zhangxinyuehfad force-pushed the zxy_main2main_v0.24.0 branch from 01914d8 to 6bb24f7 Compare July 4, 2026 14:42

align _init_device with upstream gpu_worker.py init_device

ec74a26

Signed-off-by: hfadzxy <starmoon_zhang@163.com>

zhangxinyuehfad force-pushed the zxy_main2main_v0.24.0 branch from 6bb24f7 to ec74a26 Compare July 4, 2026 15:39

fix

954d773

Signed-off-by: hfadzxy <starmoon_zhang@163.com>

MengqingCao reviewed Jul 5, 2026

View reviewed changes

zhangxinyuehfad force-pushed the zxy_main2main_v0.24.0 branch 3 times, most recently from 31bf142 to 954d773 Compare July 5, 2026 10:37

MengqingCao approved these changes Jul 5, 2026

View reviewed changes

MengqingCao merged commit 05e095a into vllm-project:main Jul 5, 2026
54 of 65 checks passed

CubeLander mentioned this pull request Jul 31, 2026

[BugFix][Worker] Align MRV2 with verified vLLM APIs vLLM-HUST/vllm-ascend-hust#190

Closed

		register_model()


		import vllm_ascend.logger # noqa: E402, F401

Uh oh!

Conversation

zhangxinyuehfad commented Jul 1, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this PR does / why we need it?

Changes

examples/offline_data_parallel.py

tests/e2e/conftest.py

tests/ut/quantization/methods/test_w4a16_mxfp4.py

tests/ut/spec_decode/test_speculators_vwn_eagle3.py

vllm_ascend/_310p/worker_310p.py

vllm_ascend/patch/platform/patch_dp_device_ids.py

vllm_ascend/patch/platform/patch_kv_cache_utils.py

vllm_ascend/patch/worker/__init__.py

vllm_ascend/spec_decode/ngram_proposer.py

vllm_ascend/worker/utils.py

vllm_ascend/worker/worker.py

vllm_ascend/__init__.py

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

gemini-code-assist Bot commented Jul 1, 2026

Summary of Changes

Highlights

Footnotes

Uh oh!

github-actions Bot commented Jul 1, 2026

Uh oh!

gemini-code-assist Bot commented Jul 1, 2026

Uh oh!

github-actions Bot commented Jul 1, 2026

Uh oh!

github-actions Bot commented Jul 2, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MengqingCao Jul 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MengqingCao commented Jul 5, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

zhangxinyuehfad commented Jul 1, 2026 •

edited by github-actions Bot

Loading

`examples/offline_data_parallel.py`

`tests/e2e/conftest.py`

`tests/ut/quantization/methods/test_w4a16_mxfp4.py`

`tests/ut/spec_decode/test_speculators_vwn_eagle3.py`

`vllm_ascend/_310p/worker_310p.py`

`vllm_ascend/patch/platform/patch_dp_device_ids.py`

`vllm_ascend/patch/platform/patch_kv_cache_utils.py`

`vllm_ascend/patch/worker/init.py`

`vllm_ascend/spec_decode/ngram_proposer.py`

`vllm_ascend/worker/utils.py`

`vllm_ascend/worker/worker.py`

`vllm_ascend/init.py`

MengqingCao Jul 4, 2026 •

edited

Loading