Skip to content

Add DeepSeek V3 support - #1379

Open
SamJSui wants to merge 1 commit into
linkedin:mainfrom
SamJSui:feat/deepseek-v3-support
Open

Add DeepSeek V3 support#1379
SamJSui wants to merge 1 commit into
linkedin:mainfrom
SamJSui:feat/deepseek-v3-support

Conversation

@SamJSui

@SamJSui SamJSui commented Aug 13, 2026

Copy link
Copy Markdown

Summary

Adds Liger Kernel support for Hugging Face Transformers deepseek_v3 / DeepSeek V3. This is distinct from the existing deepseek_v32 support work.

  • Adds a DeepSeek V3 fused linear cross-entropy forward path.
  • Wires apply_liger_kernel_to_deepseek_v3 into public exports and model-type autodispatch.
  • Patches RMSNorm, dense and shared-expert SwiGLU, CrossEntropyLoss, and FusedLinearCrossEntropy for both class-level and instance-level patching.
  • Adds bf16/fp32 convergence coverage for fused-loss and materialized-logits paths.
  • Adds direct API regressions for tuple returns, logits_to_keep, and explicit shift_labels.
  • Updates the README patching table.

Closes #623.

Details

DeepSeek V3 uses partial, interleaved RoPE that is incompatible with Liger's generic RoPE swap. Passing rope=True therefore emits a warning and leaves both upstream DeepSeek V3 RoPE functions unchanged.

The routed expert implementation also remains upstream. Liger patches dense MLPs and sparse-layer shared_experts with the existing Qwen3 MoE-compatible SwiGLU wrapper; a local Triton 3.2 bf16 fused-MoE backward probe failed because atomic_add does not support bf16.

The FLCE forward follows the current Transformers API contract, including @can_return_tuple, logits_to_keep, explicit shift_labels, and the shared PEFT/FSDP-safe LM-head loss helper.

Testing Done

  • Hardware Type: NVIDIA GeForce RTX 4070 Ti SUPER
  • run make test to ensure correctness
  • run make checkstyle to ensure code style
  • run make test-convergence to ensure convergence

The checked boxes mean the commands were run. The two broad GPU targets did not exit zero locally; every nonzero case reproduced unchanged on clean origin/main at 780e76b, while all DeepSeek V3 checks passed.

Clean validation environment: Python 3.10.12, PyTorch 2.13.0+cu130, Triton 3.7.1, Transformers 5.15.0.

make test
# 3998 passed, 1149 skipped, 14 xfailed
# 2 bf16 numeric-tolerance failures:
# - test_geglu.py (reproduces on origin/main)
# - test_poly_norm.py (reproduces on origin/main)

make test-convergence
# 158 passed, 9 skipped, 3 xfailed, 1 xpassed
# 5 non-DeepSeek failures, all reproduced on origin/main:
# - fp32 mini_llava (with and without materialized logits)
# - bf16 mini_llama4
# - bf16 mini_qwen3_5_moe multimodal
# - bf16 mini_qwen3_moe with materialized logits
# All 4 DeepSeek V3 convergence cases passed.

make checkstyle
# passed

git diff --check
# passed

Focused checks completed successfully:

PYTHONPATH=src python3 -m pytest test/transformers/test_monkey_patch.py -q
# 66 passed

PYTHONPATH=src python3 -m pytest \
  test/convergence/bf16/test_mini_models.py \
  test/convergence/bf16/test_mini_models_with_logits.py \
  test/convergence/fp32/test_mini_models.py \
  test/convergence/fp32/test_mini_models_with_logits.py \
  -k deepseek_v3 -xvs
# 4 passed

@SamJSui
SamJSui marked this pull request as ready for review August 13, 2026 23:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[transformers] support DeepSeek V3

1 participant