Skip to content

feat(ds4): support qtype 105/106 across heterogeneous GPUs - #600

Merged
davide221 merged 11 commits into
mainfrom
codex/ds4-0731-mix-tp-clean
Aug 13, 2026
Merged

feat(ds4): support qtype 105/106 across heterogeneous GPUs#600
davide221 merged 11 commits into
mainfrom
codex/ds4-0731-mix-tp-clean

Conversation

@davide221

@davide221 davide221 commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

Summary

  • support embedded qtype 105/106 expert weights when DeepSeek 4 is split across two AMD GPUs
  • register each GPU-owned expert with its embedded decode table and unregister it before releasing owner buffers
  • enable registry-aware mixed-expert prefill on same-runtime GPU tensor parallelism
  • scale automatic route balancing with the model's expert top-k, preserving top-4 behavior and supporting the model-default top-6 path

This PR is stacked on #577. Its own diff contains the mixed-quantization support and top-k route scaling; the routing profiles and benchmark framework remain in the base stack.

Validation

Tested with the 0731 mixed-qtype model on an R9700 plus Radeon 8060S, using all six routed experts and the DSpark draft.

  • dual-architecture HIP Release build (gfx1151;gfx1201): passed
  • heterogeneous planner/storage tests: 10/10 passed
  • mixed registry teardown: passed independently on both GPUs
  • full DeepSeek 4 GPU unit suite: passed independently on both GPUs
  • two warmups plus three measured deterministic 128-token replies
  • client decode: 77.184-77.267 tok/s; median 77.221 tok/s
  • model-side decode: 68.1-68.2 tok/s
  • draft acceptance: 100%
  • exact response hash matched on every run

The automatic route split selected 5.0 of 6 expert slots for the faster owner at the qualified 4.4 device-rate ratio.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 49 files

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

Comment thread server/deps/llama.cpp/ggml/include/ggml-cuda.h
Comment thread server/scripts/qualify_ds4_q5_amd.sh
Comment thread server/src/common/moe_hybrid_ffn_eval.cpp
Comment thread server/deps/llama.cpp/ggml/src/ggml-cuda/fattn.cu
Comment thread server/src/deepseek4/deepseek4_fused_verify.inc
Comment thread server/deps/llama.cpp/ggml/src/ggml-cuda/moe-fused.cu
Comment thread server/src/common/moe_hybrid_ffn_eval.cpp Outdated
Comment thread server/docs/HETEROGENEOUS_STAGE_PLANNER.md Outdated
Comment thread server/docs/DS4_R9700_STRIX_PROFILE.md Outdated
Comment thread server/src/deepseek4/deepseek4_dspark_spec.cpp

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 23 files (changes from recent commits).

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

Comment thread server/src/deepseek4/deepseek4_dspark_spec.cpp
Comment thread server/src/common/heterogeneous_stage_planner.cpp Outdated
Comment thread server/tests/test_ds4_publication_tools.py Outdated
Comment thread server/scripts/qualify_ds4_q5_amd.sh Outdated
Comment thread server/src/deepseek4/deepseek4_backend.cpp Outdated
Comment thread server/src/common/moe_hybrid_ffn_eval.cpp Outdated

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 7 files (changes from recent commits).

Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic

Comment thread server/src/common/heterogeneous_stage_planner.cpp Outdated

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 13 files (changes from recent commits).

Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic

Comment thread server/scripts/qualify_ds4_q5_amd.sh Outdated
Comment thread server/deps/llama.cpp/ggml/src/ggml-cuda/top-k.cu Outdated
@davide221
davide221 force-pushed the codex/ds4-0731-mix-tp-clean branch from 0307661 to 40b983c Compare August 12, 2026 12:04
@davide221 davide221 changed the title feat(ds4): support 0731 mixed expert parallelism on R9700 + Strix feat(ds4): support qtype 105/106 across heterogeneous GPUs Aug 12, 2026
@davide221
davide221 force-pushed the codex/ds4-0731-mix-tp-clean branch 3 times, most recently from b03b8af to a601dc1 Compare August 12, 2026 15:01
@davide221
davide221 changed the base branch from main to codex/generic-heterogeneous-stage-planner August 12, 2026 15:01
@davide221
davide221 force-pushed the codex/generic-heterogeneous-stage-planner branch from 1e54c5d to c029f90 Compare August 12, 2026 18:20
@davide221
davide221 force-pushed the codex/ds4-0731-mix-tp-clean branch from a601dc1 to f56a8a7 Compare August 12, 2026 18:20
@davide221
davide221 force-pushed the codex/generic-heterogeneous-stage-planner branch from c029f90 to a209c34 Compare August 12, 2026 18:46
@davide221
davide221 force-pushed the codex/ds4-0731-mix-tp-clean branch 3 times, most recently from 9a4feb2 to 9d59e07 Compare August 12, 2026 19:18
@davide221
davide221 force-pushed the codex/generic-heterogeneous-stage-planner branch from a209c34 to 3ff79a6 Compare August 12, 2026 19:36
@davide221
davide221 force-pushed the codex/ds4-0731-mix-tp-clean branch from 9d59e07 to f169291 Compare August 12, 2026 19:37
@davide221
davide221 force-pushed the codex/generic-heterogeneous-stage-planner branch from 3ff79a6 to a46f877 Compare August 12, 2026 19:39
@davide221
davide221 force-pushed the codex/ds4-0731-mix-tp-clean branch from f169291 to d8d2394 Compare August 12, 2026 19:39
@davide221
davide221 force-pushed the codex/generic-heterogeneous-stage-planner branch from a46f877 to 5dd1cfc Compare August 12, 2026 20:11
@davide221
davide221 force-pushed the codex/ds4-0731-mix-tp-clean branch from d8d2394 to 2c7cdb3 Compare August 12, 2026 20:12
@davide221
davide221 force-pushed the codex/generic-heterogeneous-stage-planner branch from 5dd1cfc to b69f52e Compare August 13, 2026 13:02
@davide221
davide221 force-pushed the codex/ds4-0731-mix-tp-clean branch from 2c7cdb3 to f4c72d9 Compare August 13, 2026 13:02
@davide221

Copy link
Copy Markdown
Contributor Author

@cubic-dev-ai review the current head after the branch rebase. Please focus on correctness, maintainability, security, and heterogeneous qtype 105/106 lifecycle safety.

@cubic-dev-ai

cubic-dev-ai Bot commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

@cubic-dev-ai review the current head after the branch rebase. Please focus on correctness, maintainability, security, and heterogeneous qtype 105/106 lifecycle safety.

@davide221 I have started the AI code review. It will take a few minutes to complete.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 26 files

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

Comment thread server/deps/llama.cpp/ggml/src/ggml-cuda/rocmfp3_mix.cu
Comment thread server/src/deepseek4/deepseek4_backend.cpp
Comment thread harness/qualification/deepseek4/qualify_ds4_q5_amd.sh Outdated
Comment thread harness/tests/test_ds4_benchmark_tools.py Outdated
@davide221
davide221 force-pushed the codex/generic-heterogeneous-stage-planner branch from b69f52e to 0f83e60 Compare August 13, 2026 13:55
@davide221
davide221 force-pushed the codex/ds4-0731-mix-tp-clean branch from f4c72d9 to 1d25cb0 Compare August 13, 2026 13:55
@davide221

Copy link
Copy Markdown
Contributor Author

@cubic-dev-ai review the current rebased head. Please check correctness, maintainability, security, and the qtype 105/106 heterogeneous routing boundaries.

@cubic-dev-ai

cubic-dev-ai Bot commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

@cubic-dev-ai review the current rebased head. Please check correctness, maintainability, security, and the qtype 105/106 heterogeneous routing boundaries.

@davide221 I have started the AI code review. It will take a few minutes to complete.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 25 files

Requires human review: Auto-approval blocked because this review re-detected 1 unresolved issue already reported by Cubic.

Re-trigger cubic

Comment thread server/deps/llama.cpp/ggml/src/ggml.c Outdated
Comment thread server/src/common/moe_hybrid_ffn_eval.cpp
Comment thread server/src/common/moe_hybrid_ffn_eval.cpp
@davide221
davide221 force-pushed the codex/generic-heterogeneous-stage-planner branch from 0f83e60 to 2102a4c Compare August 13, 2026 14:18
@davide221
davide221 force-pushed the codex/ds4-0731-mix-tp-clean branch from 1d25cb0 to a92c514 Compare August 13, 2026 14:22
@davide221
davide221 changed the base branch from codex/generic-heterogeneous-stage-planner to main August 13, 2026 14:40
@davide221
davide221 merged commit 90f85fa into main Aug 13, 2026
3 of 6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant