Speculative Decoding support for intel_xpu attention backend on XPU target - #30548
Speculative Decoding support for intel_xpu attention backend on XPU target#30548ANSHUMAN87 wants to merge 10 commits into
Conversation
|
Warning You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again! |
12e32ea to
38dbf85
Compare
|
@mingfeima @Valentine233 please help review. Thanks! |
There was a problem hiding this comment.
- Why not support topk>1 in this PR? This is essential to all tree-based methods.
- Could you provide the SD perf results with intel_xpu attention backend?
- Please also modify the hard-coded attention backend in
test/registered/spec/eagle/test_spec_eagle_parity.py.
|
5d07314 to
bdf55ad
Compare
|
Could you please provide the SD perf results with intel_xpu attention backend? |
Currently the speedup is only 1.3x . I am checking if it can be improved. |
What is the detailed config for benchmark? Could you compare the perf of these three: cuda/xpu with triton backend/xpu with intel_xpu backend? |
|
For topk>1, please validate with the fixing PR: sgl-project/sgl-kernel-xpu#330. |
| ), "XPUAttentionBackend XPU graph only supports decode / target-verify / draft-extend modes" | ||
| assert not ( | ||
| (is_verify or is_draft_decode or is_draft_extend) and self.topk > 1 | ||
| ), "XPUAttentionBackend XPU graph spec decoding supports topk <= 1 only" |
There was a problem hiding this comment.
so only greedy mode supported?
There was a problem hiding this comment.
When topk=1, draft tree becomes draft chain.
|
Generally LGTM. As |
|
@ANSHUMAN87 Please try to land this ASAP, thanks! |
| ) | ||
| from sglang.test.server_fixtures.spec_eagle_fixture import Eagle3Base, EagleLlama2Base | ||
|
|
||
| register_xpu_ci(est_time=480, suite="nightly-xpu-1-gpu", nightly=True) |
There was a problem hiding this comment.
why this is not in the pr test?
|
@ANSHUMAN87 is this one ready? |
@mingfeima yes this PR is ready for review. Except currently topk=1 is supported. topk>1 is facing some hang issue @Valentine233 is working on it. I will raise separate PR for topk>1 support once @Valentine233 is ready with sgl-kernel-xpu changes. |
4 participants
This PR enables Speculative Decoding to run with intel_xpu attention backend for XPU targets.
NOTE: Currently only topk=1 is supported.
JH...!
CI States
Latest PR Test (Base): ❌ Run #29554133821
Latest PR Test (Extra): ❌ Run #32336227677
Latest PR Test (AMD ROCm 7.2): ➖ No AMD PR run found for this commit.