Skip to content

Load Qwen3.8 GGUFs with Embedded NextN/MTP Blocks - #610

Merged
davide221 merged 1 commit into
Luce-Org:mainfrom
jkyamog:feat-qwen35-embedded-mtp-target
Aug 15, 2026
Merged

Load Qwen3.8 GGUFs with Embedded NextN/MTP Blocks#610
davide221 merged 1 commit into
Luce-Org:mainfrom
jkyamog:feat-qwen35-embedded-mtp-target

Conversation

@jkyamog

@jkyamog jkyamog commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

Qwen3.8-27B GGUFs can contain the normal target transformer layers followed by
embedded NextN/MTP predictor blocks. The current loader treated every GGUF
block as a target layer and rejected the model during startup when the total
block count was not divisible by Qwen's full-attention interval.

This change separates the raw GGUF block count from the target layer count:

block_count = 65
nextn_predict_layers = 1
target_layer_count = 64
full_attention_interval = 4

The target loader now validates the 64-layer trunk and ignores the trailing
embedded MTP block for ordinary target execution.

Reproduction

Model:

Qwen3.8-27B-Q6_K.gguf

Before this change, startup failed with:

target load: block_count=65 not divisible by full_attention_interval=4
[backend_factory] Qwen35Backend init failed
[server] backend creation failed

The GGUF metadata was verified directly:

general.architecture = qwen35
general.name = Qwen3.8-27B
qwen35.block_count = 65
qwen35.nextn_predict_layers = 1
qwen35.full_attention_interval = 4
blk.64.nextn.eh_proj.weight
blk.64.nextn.enorm.weight
blk.64.nextn.hnorm.weight
blk.64.nextn.shared_head_norm.weight

Implementation

  • Adds a shared helper to derive the effective target layer count.
  • Applies the embedded-block rule only to qwen35 and qwen35moe metadata.
  • Rejects invalid metadata where nextn_predict_layers >= block_count.
  • Verifies each ignored trailing block contains a blk.<index>.nextn. tensor.
  • Excludes embedded NextN tensors from the ordinary target-layer load.
  • Preserves raw block_count and nextn_predict_layers in model inspection.
  • Reports both raw and effective counts in loader errors.

Non-Qwen architectures are not reinterpreted, even if they contain similarly
named metadata.

Review in cubic

@jkyamog

jkyamog commented Aug 14, 2026

Copy link
Copy Markdown
Contributor Author

works with current 3.6 drafter, but its slow. I guess we need a 3.8 drafter.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 4 files

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

Comment thread server/test/test_server_unit.cpp
Comment thread server/src/common/gguf_inspect.h Outdated

@howard0su howard0su left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@jkyamog
jkyamog force-pushed the feat-qwen35-embedded-mtp-target branch from 2ca858b to 8d28de3 Compare August 15, 2026 01:21
@jkyamog

jkyamog commented Aug 15, 2026

Copy link
Copy Markdown
Contributor Author

updated to address issues reported by cubic-dev-ai

@davide221
davide221 merged commit 3e59047 into Luce-Org:main Aug 15, 2026
3 of 6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants