[BugFix][MRV2] Fix DSpark spec decoding on model runner v2 - #14696
Conversation
Signed-off-by: root <root@vllm-ascend.local>
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request addresses critical bugs encountered during speculative decoding on Model Runner v2. It ensures proper memory allocation for speculative hidden states across various methods and corrects an attribute reference error in the metadata builder, improving stability during memory profiling and draft proposal phases. Highlights
New Features🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
|
👋 Hi! Thank you for contributing to the vLLM Ascend project. The following points will speed up your PR merge:
If CI fails, you can run linting and testing checks locally according Contributing and Testing. Tip 💡 Consider Linking a Related Issue or RFCYour PR title contains the [BugFix] tag, indicating a bug fix or new feature. Linking a related issue or RFC in the PR description is strongly encouraged — it gives reviewers helpful context and speeds up the review. You can use any of these keywords:
🙏 Thanks for helping us keep the project well-organized! |
There was a problem hiding this comment.
Code Review
Suggested PR Title:
[Attention][BugFix] Fix block size usage and MTP hidden buffer allocationSuggested PR Summary:
### What this PR does / why we need it?
This PR updates the attention metadata building to use `storage_block_size` instead of `block_size` in `dsa_v1.py`. It also refactors the allocation of `_mtp_hidden_buffer` in DeepSeek-V4 model to only allocate on the last pipeline parallel rank when MTP hidden states are needed (either via Eagle or a draft model).
However, there is an issue in the implementation: `uses_draft_model` is a property on `SpeculativeConfig`, not a method, so calling it as `spec_config.uses_draft_model()` will raise a `TypeError`. This needs to be corrected to `spec_config.uses_draft_model`.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
CI/CD tests.| needs_mtp_hidden_states = spec_config is not None and ( | ||
| spec_config.use_eagle() or spec_config.uses_draft_model() | ||
| ) |
There was a problem hiding this comment.
In vLLM, uses_draft_model is a property on SpeculativeConfig, not a method. Calling spec_config.uses_draft_model() will raise a TypeError: 'bool' object is not callable. Please access it as a property without parentheses: spec_config.uses_draft_model.
Additionally, verify if use_eagle is also a property or a method on your SpeculativeConfig class to avoid similar errors.
| needs_mtp_hidden_states = spec_config is not None and ( | |
| spec_config.use_eagle() or spec_config.uses_draft_model() | |
| ) | |
| needs_mtp_hidden_states = spec_config is not None and ( | |
| spec_config.use_eagle() or spec_config.uses_draft_model | |
| ) |
Temporarily apply the fix from vllm-project#14696 so the dedicated A3 CI run validates it together with the runner-routing changes. Signed-off-by: MrZ20 <2609716663@qq.com>
Temporarily apply the fix from vllm-project#14696 so the dedicated A3 CI run validates it together with the runner-routing changes. Signed-off-by: MrZ20 <2609716663@qq.com>
Temporarily apply the fix from vllm-project#14696 so the dedicated A3 CI run validates it together with the runner-routing changes. Signed-off-by: MrZ20 <2609716663@qq.com>
What this PR does / why we need it?
Allocate the MTP pre-hc_head residual buffer for every speculative method that consumes target hidden states (eagle/eagle3/mtp/dflash/dspark/draft_model) instead of only mtp, and only on the last PP rank.
Previously DSpark crashed during the memory-profiling dummy run with "TypeError: 'NoneType' object is not subscriptable".
Use storage_block_size instead of the non-existent block_size attribute
when building DSpark SWA indices in AscendDSAMetadataBuilder, fixing "AttributeError: 'AscendDSAMetadataBuilder' object has no attribute 'block_size'" in the draft propose path.
Does this PR introduce any user-facing change?
N/A
How was this patch tested?
CI passed with new added/existing test.