Add DeepSeek V3 support - #1379
Open
SamJSui wants to merge 1 commit into
Open
Conversation
SamJSui
marked this pull request as ready for review
August 13, 2026 23:10
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds Liger Kernel support for Hugging Face Transformers
deepseek_v3/ DeepSeek V3. This is distinct from the existingdeepseek_v32support work.apply_liger_kernel_to_deepseek_v3into public exports and model-type autodispatch.logits_to_keep, and explicitshift_labels.Closes #623.
Details
DeepSeek V3 uses partial, interleaved RoPE that is incompatible with Liger's generic RoPE swap. Passing
rope=Truetherefore emits a warning and leaves both upstream DeepSeek V3 RoPE functions unchanged.The routed expert implementation also remains upstream. Liger patches dense MLPs and sparse-layer
shared_expertswith the existing Qwen3 MoE-compatible SwiGLU wrapper; a local Triton 3.2 bf16 fused-MoE backward probe failed becauseatomic_adddoes not support bf16.The FLCE forward follows the current Transformers API contract, including
@can_return_tuple,logits_to_keep, explicitshift_labels, and the shared PEFT/FSDP-safe LM-head loss helper.Testing Done
make testto ensure correctnessmake checkstyleto ensure code stylemake test-convergenceto ensure convergenceThe checked boxes mean the commands were run. The two broad GPU targets did not exit zero locally; every nonzero case reproduced unchanged on clean
origin/mainat780e76b, while all DeepSeek V3 checks passed.Clean validation environment: Python 3.10.12, PyTorch 2.13.0+cu130, Triton 3.7.1, Transformers 5.15.0.
Focused checks completed successfully: