Cannot launch single-file MTP model using llama.cpp engine (Qwen3.6-MTP gguf support request)

[Side514]

System Info / 系統信息

• CUDA Version: 13.0
• xllamacpp: 2026.8.10229
• python: 3.12.13
• system: Ubuntu 24.04 LTS

Running Xinference with Docker? / 是否使用 Docker 运行 Xinfernece？

• docker / docker
• pip install / 通过 pip install 安装
• installation from source / 从源码安装

Version info / 版本信息

xprobe/xinference:v3.1.0 (ec41459d15cc)

The command used to start Xinference / 用以启动 xinference 的命令

using default inference/xinference/deploy/docker/docker-compose.yml

$ docker compose up -d

Reproduction / 复现过程

Xinference WebUI -> Launch Model -> qwen3.6 -> Launch

• Model Engine: llama.cpp
• Model Format: ggufv2
• Model Size: 35
• Quantization: UD-Q4_K_M (I replace the model file with APEX-MTP-I-Balanced because "Register Models" doesn't work for me)
• Multimodel Projector: mmproj-BF16.gguf
• ......
• Advanced Configuration
  • ......
  • Engine Parameters (Additional Parameters: llama.cpp)
    • n_ctx: 983040
    • n_parallel: 5
    • enable_mtp: True

here's the xinference log:

2026-08-16T07:17:19.252Z DEBUG xinference.core.supervisor pid:80 role:local address:0.0.0.0:16691 node:0b0dae70dc29 [request 845939a6-9942-11f1-a9c1-a2fa2837c84e] Enter launch_builtin_model, args: <xinference.core.supervisor.SupervisorActor object at 0x736b89eca7b0>, kwargs: model_uid=qwen3.6,model_name=qwen3.6,model_engine=llama.cpp,model_size_in_billions=35,model_format=ggufv2,quantization=UD-Q4_K_M,model_type=LLM,replica=1,n_gpu=auto,request_limits=None,wait_ready=True,peft_model_config=None,worker_ip=None,gpu_idx=None,download_hub=modelscope,model_path=None,enable_virtual_env=None,virtual_env_packages=None,envs=None,enable_thinking=False,multimodal_projector=mmproj-BF16.gguf,n_ctx=983040,n_parallel=5,enable_mtp=True
2026-08-16T07:17:19.253Z DEBUG xinference.core.supervisor pid:80 role:local address:0.0.0.0:16691 node:0b0dae70dc29 Enter launch_builtin_model, model_uid: qwen3.6, model_name: qwen3.6, model_size: 35, model_format: ggufv2, quantization: UD-Q4_K_M, replica: 1, enable_xavier: False, worker_ip: None, kwargs: {'enable_thinking': False, 'multimodal_projector': 'mmproj-BF16.gguf', 'n_ctx': 983040, 'n_parallel': 5, 'enable_mtp': True}
2026-08-16T07:17:19.253Z DEBUG xinference.core.worker pid:80 role:local address:0.0.0.0:16691 node:0b0dae70dc29 Enter get_model_count, args: <xinference.core.worker.WorkerActor object at 0x736b8929c770>, kwargs:
2026-08-16T07:17:19.254Z DEBUG xinference.core.worker pid:80 role:local address:0.0.0.0:16691 node:0b0dae70dc29 Leave get_model_count, elapsed time: 0 s
2026-08-16T07:17:19.254Z DEBUG xinference.core.worker pid:80 role:local address:0.0.0.0:16691 node:0b0dae70dc29 [request 84597f24-9942-11f1-a9c1-a2fa2837c84e] Enter get_gpu_allocation_status, args: <xinference.core.worker.WorkerActor object at 0x736b8929c770>, kwargs:
2026-08-16T07:17:19.254Z DEBUG xinference.core.worker pid:80 role:local address:0.0.0.0:16691 node:0b0dae70dc29 [request 84597f24-9942-11f1-a9c1-a2fa2837c84e] Leave get_gpu_allocation_status, elapsed time: 0 s
2026-08-16T07:17:19.254Z DEBUG xinference.core.supervisor pid:80 role:local address:0.0.0.0:16691 node:0b0dae70dc29 Worker candidates for qwen3.6: [{'addr': '0.0.0.0:16691', 'count': 0}]
2026-08-16T07:17:19.255Z DEBUG xinference.core.supervisor pid:80 role:local address:0.0.0.0:16691 node:0b0dae70dc29 GPU allocation snapshots: [{'addr': '0.0.0.0:16691', 'alloc': {'total': [0], 'models': {}, 'user_specified': {0: []}, 'allow_multi_replica_per_gpu': True}}]
2026-08-16T07:17:19.255Z DEBUG xinference.core.supervisor pid:80 role:local address:0.0.0.0:16691 node:0b0dae70dc29 Replica 0 assigned to 0.0.0.0:16691 (count: 1)
2026-08-16T07:17:19.255Z DEBUG xinference.core.supervisor pid:80 role:local address:0.0.0.0:16691 node:0b0dae70dc29 Launching 1 replicas in parallel for model qwen3.6
2026-08-16T07:17:19.256Z INFO xinference.core.worker pid:80 role:local address:0.0.0.0:16691 node:0b0dae70dc29 [request 8459b822-9942-11f1-a9c1-a2fa2837c84e] Enter launch_builtin_model, args: <xinference.core.worker.WorkerActor object at 0x736b8929c770>, kwargs: model_uid=qwen3.6-rep0,model_name=qwen3.6,model_size_in_billions=35,model_format=ggufv2,quantization=UD-Q4_K_M,model_engine=llama.cpp,model_type=LLM,n_gpu=auto,request_limits=None,peft_model_config=None,gpu_idx=[0],download_hub=modelscope,model_path=None,enable_virtual_env=None,virtual_env_packages=None,envs=None,xavier_config=None,enable_thinking=False,multimodal_projector=mmproj-BF16.gguf,n_ctx=983040,n_parallel=5,enable_mtp=True
2026-08-16T07:17:19.256Z INFO xinference.core.worker pid:80 role:local address:0.0.0.0:16691 node:0b0dae70dc29 You specify to launch the model: qwen3.6 on GPU index: [0] of the worker: 0.0.0.0:16691, xinference will automatically ignore the `n_gpu` option.
2026-08-16T07:17:19.256Z INFO xinference.core.worker pid:80 role:local address:0.0.0.0:16691 node:0b0dae70dc29 Launch started: model_name=qwen3.6, model_uid=qwen3.6-rep0 (active: 1/5, queued: 0)
2026-08-16T07:17:19.259Z DEBUG xinference.core.worker pid:80 role:local address:0.0.0.0:16691 node:0b0dae70dc29 Skipped .pth write (content unchanged): /root/.xinference/virtualenv/v4/qwen3.6/llama.cpp/3.12.13/local/lib/python3.12/dist-packages/_xinference_parent.pth
2026-08-16T07:17:19.260Z DEBUG xinference.core.progress_tracker pid:80 role:local address:0.0.0.0:16691 node:0b0dae70dc29 Setting progress, request id: launching-qwen3.6-rep0, progress: 0.0
2026-08-16T07:17:19.261Z WARNING xinference.model.llm.llm_family pid:80 role:local address:0.0.0.0:16691 node:0b0dae70dc29 Bypassing engine compatibility checks for llama.cpp due to virtualenv marker.
2026-08-16T07:17:19.262Z DEBUG xinference.model.llm.core pid:80 role:local address:0.0.0.0:16691 node:0b0dae70dc29 Launching qwen3.6-rep0 with XllamaCppModel
2026-08-16T07:17:19.263Z ERROR xinference.core.worker pid:80 role:local address:0.0.0.0:16691 node:0b0dae70dc29 Failed to load model qwen3.6-rep0
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/xinference/core/worker.py", line 3473, in launch_builtin_model
    model = await asyncio.to_thread(
            ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/asyncio/threads.py", line 25, in to_thread
    return await loop.run_in_executor(None, func_call)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/concurrent/futures/thread.py", line 59, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/xinference/model/core.py", line 48, in create_model_instance
    return create_llm_model_instance(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/xinference/model/llm/core.py", line 394, in create_llm_model_instance
    kwargs["draft_model_path"] = LLMCacheManager(
                                 ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/xinference/model/llm/cache_manager.py", line 67, in __init__
    raise ValueError(
ValueError: Model qwen3.6 (ggufv2, 35b, UD-Q4_K_M) does not declare a drafter model on hub modelscope. Please pass `draft_model_path` to use a local drafter.
2026-08-16T07:17:19.267Z DEBUG xinference.core.worker pid:80 role:local address:0.0.0.0:16691 node:0b0dae70dc29 [request 845b69c4-9942-11f1-a9c1-a2fa2837c84e] Enter _update_model_state, args: <xinference.core.worker.WorkerActor object at 0x736b8929c770>,qwen3.6-rep0,error, kwargs:
2026-08-16T07:17:19.267Z DEBUG xinference.core.progress_tracker pid:80 role:local address:0.0.0.0:16691 node:0b0dae70dc29 Setting progress, request id: launching-qwen3.6-rep0, progress: 0.8
2026-08-16T07:17:19.268Z DEBUG xinference.core.progress_tracker pid:80 role:local address:0.0.0.0:16691 node:0b0dae70dc29 Setting progress, request id: launching-qwen3.6-rep0, progress: 0.8
2026-08-16T07:17:19.268Z DEBUG xinference.core.worker pid:80 role:local address:0.0.0.0:16691 node:0b0dae70dc29 [request 845b69c4-9942-11f1-a9c1-a2fa2837c84e] Leave _update_model_state, elapsed time: 0 s
2026-08-16T07:17:19.268Z INFO xinference.core.worker pid:80 role:local address:0.0.0.0:16691 node:0b0dae70dc29 Launch finished: model_name=qwen3.6, model_uid=qwen3.6-rep0 (active: 0/5, queued: 0)
2026-08-16T07:17:19.268Z ERROR xinference.core.worker pid:80 role:local address:0.0.0.0:16691 node:0b0dae70dc29 [request 8459b822-9942-11f1-a9c1-a2fa2837c84e] Leave launch_builtin_model, error: Model qwen3.6 (ggufv2, 35b, UD-Q4_K_M) does not declare a drafter model on hub modelscope. Please pass `draft_model_path` to use a local drafter., elapsed time: 0 s
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/xinference/core/utils.py", line 98, in wrapped
    ret = await func(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/xinference/core/worker.py", line 3473, in launch_builtin_model
    model = await asyncio.to_thread(
            ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/asyncio/threads.py", line 25, in to_thread
    return await loop.run_in_executor(None, func_call)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/concurrent/futures/thread.py", line 59, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/xinference/model/core.py", line 48, in create_model_instance
    return create_llm_model_instance(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/xinference/model/llm/core.py", line 394, in create_llm_model_instance
    kwargs["draft_model_path"] = LLMCacheManager(
                                 ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/xinference/model/llm/cache_manager.py", line 67, in __init__
    raise ValueError(
ValueError: Model qwen3.6 (ggufv2, 35b, UD-Q4_K_M) does not declare a drafter model on hub modelscope. Please pass `draft_model_path` to use a local drafter.
2026-08-16T07:17:19.272Z ERROR xinference.core.supervisor pid:80 role:local address:0.0.0.0:16691 node:0b0dae70dc29 Failed to launch replica qwen3.6-rep0: [address=0.0.0.0:16691, pid=80] Model qwen3.6 (ggufv2, 35b, UD-Q4_K_M) does not declare a drafter model on hub modelscope. Please pass `draft_model_path` to use a local drafter.
2026-08-16T07:17:19.273Z DEBUG xinference.core.supervisor pid:80 role:local address:0.0.0.0:16691 node:0b0dae70dc29 [request 845c583e-9942-11f1-a9c1-a2fa2837c84e] Enter terminate_model, args: <xinference.core.supervisor.SupervisorActor object at 0x736b89eca7b0>,qwen3.6, kwargs: suppress_exception=True
2026-08-16T07:17:19.273Z DEBUG xinference.core.supervisor pid:80 role:local address:0.0.0.0:16691 node:0b0dae70dc29 [request 845c583e-9942-11f1-a9c1-a2fa2837c84e] Leave terminate_model, elapsed time: 0 s
2026-08-16T07:17:19.273Z ERROR xinference.core.supervisor pid:80 role:local address:0.0.0.0:16691 node:0b0dae70dc29 [request 845939a6-9942-11f1-a9c1-a2fa2837c84e] Leave launch_builtin_model, error: [address=0.0.0.0:16691, pid=80] Model qwen3.6 (ggufv2, 35b, UD-Q4_K_M) does not declare a drafter model on hub modelscope. Please pass `draft_model_path` to use a local drafter., elapsed time: 0 s
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/xinference/core/utils.py", line 98, in wrapped
    ret = await func(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/xinference/core/supervisor.py", line 2415, in launch_builtin_model
    await _launch_model()
  File "/usr/local/lib/python3.12/dist-packages/xinference/core/supervisor.py", line 2349, in _launch_model
    raise result
  File "/usr/local/lib/python3.12/dist-packages/xinference/core/supervisor.py", line 2146, in _launch_one_model
    subpool_address = await worker_ref.launch_builtin_model(
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/xoscar/backends/context.py", line 262, in send
    return self._process_result_message(result)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/xoscar/backends/context.py", line 111, in _process_result_message
    raise message.as_instanceof_cause()
  File "/usr/local/lib/python3.12/dist-packages/xoscar/backends/pool.py", line 689, in send
    result = await self._run_coro(message.message_id, coro)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/xoscar/backends/pool.py", line 389, in _run_coro
    return await coro
           ^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/xoscar/api.py", line 418, in __on_receive__
    return await super().__on_receive__(message)  # type: ignore
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "xoscar/core.pyx", line 564, in __on_receive__
  File "xoscar/core.pyx", line 526, in xoscar.core._BaseActor.__on_receive__
  File "xoscar/core.pyx", line 527, in xoscar.core._BaseActor.__on_receive__
  File "xoscar/core.pyx", line 532, in xoscar.core._BaseActor.__on_receive__
  File "/usr/local/lib/python3.12/dist-packages/xinference/core/utils.py", line 98, in wrapped
    ret = await func(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/xinference/core/worker.py", line 3473, in launch_builtin_model
    model = await asyncio.to_thread(
            ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/asyncio/threads.py", line 25, in to_thread
    return await loop.run_in_executor(None, func_call)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/concurrent/futures/thread.py", line 59, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/xinference/model/core.py", line 48, in create_model_instance
    return create_llm_model_instance(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/xinference/model/llm/core.py", line 394, in create_llm_model_instance
    kwargs["draft_model_path"] = LLMCacheManager(
                                 ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/xinference/model/llm/cache_manager.py", line 67, in __init__
    raise ValueError(
ValueError: [address=0.0.0.0:16691, pid=80] Model qwen3.6 (ggufv2, 35b, UD-Q4_K_M) does not declare a drafter model on hub modelscope. Please pass `draft_model_path` to use a local drafter.
2026-08-16T07:17:19.284Z ERROR xinference.api.restful_api pid:1 role:local address:0.0.0.0:9997 node:0b0dae70dc29 [address=0.0.0.0:16691, pid=80] Model qwen3.6 (ggufv2, 35b, UD-Q4_K_M) does not declare a drafter model on hub modelscope. Please pass `draft_model_path` to use a local drafter.
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/xinference/api/restful_api.py", line 900, in launch_model
    model_uid = await (await self._get_supervisor_ref()).launch_builtin_model(
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/xoscar/backends/context.py", line 262, in send
    return self._process_result_message(result)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/xoscar/backends/context.py", line 111, in _process_result_message
    raise message.as_instanceof_cause()
  File "/usr/local/lib/python3.12/dist-packages/xoscar/backends/pool.py", line 689, in send
    result = await self._run_coro(message.message_id, coro)
    ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/xoscar/backends/pool.py", line 389, in _run_coro
    return await coro
  File "/usr/local/lib/python3.12/dist-packages/xoscar/api.py", line 418, in __on_receive__
    return await super().__on_receive__(message)  # type: ignore
    ^^^^^^^^^^^^^^^^^
  File "xoscar/core.pyx", line 564, in __on_receive__
    raise ex
  File "xoscar/core.pyx", line 526, in xoscar.core._BaseActor.__on_receive__
    async with self._lock:
    ^^^^^^^^^^^^^^^^^
  File "xoscar/core.pyx", line 527, in xoscar.core._BaseActor.__on_receive__
    with debug_async_timeout('actor_lock_timeout',
    ^^^^^^^^^^^^^^^^^
  File "xoscar/core.pyx", line 532, in xoscar.core._BaseActor.__on_receive__
    result = await result
    ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/xinference/core/utils.py", line 98, in wrapped
    ret = await func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/xinference/core/supervisor.py", line 2415, in launch_builtin_model
    await _launch_model()
    ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/xinference/core/supervisor.py", line 2349, in _launch_model
    raise result
  File "/usr/local/lib/python3.12/dist-packages/xinference/core/supervisor.py", line 2146, in _launch_one_model
    subpool_address = await worker_ref.launch_builtin_model(
    ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/xoscar/backends/context.py", line 262, in send
    return self._process_result_message(result)
    ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/xoscar/backends/context.py", line 111, in _process_result_message
    raise message.as_instanceof_cause()
    ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/xoscar/backends/pool.py", line 689, in send
    result = await self._run_coro(message.message_id, coro)
    ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/xoscar/backends/pool.py", line 389, in _run_coro
    return await coro
  File "/usr/local/lib/python3.12/dist-packages/xoscar/api.py", line 418, in __on_receive__
    return await super().__on_receive__(message)  # type: ignore
    ^^^^^^^^^^^^^^^^^
  File "xoscar/core.pyx", line 564, in __on_receive__
    raise ex
  File "xoscar/core.pyx", line 526, in xoscar.core._BaseActor.__on_receive__
    async with self._lock:
    ^^^^^^^^^^^^^^^^^
  File "xoscar/core.pyx", line 527, in xoscar.core._BaseActor.__on_receive__
    with debug_async_timeout('actor_lock_timeout',
    ^^^^^^^^^^^^^^^^^
  File "xoscar/core.pyx", line 532, in xoscar.core._BaseActor.__on_receive__
    result = await result
    ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/xinference/core/utils.py", line 98, in wrapped
    ret = await func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/xinference/core/worker.py", line 3473, in launch_builtin_model
    model = await asyncio.to_thread(
    ^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/asyncio/threads.py", line 25, in to_thread
    return await loop.run_in_executor(None, func_call)
      ^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/concurrent/futures/thread.py", line 59, in run
    result = self.fn(*self.args, **self.kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/xinference/model/core.py", line 48, in create_model_instance
    return create_llm_model_instance(
    ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/xinference/model/llm/core.py", line 394, in create_llm_model_instance
    kwargs["draft_model_path"] = LLMCacheManager(
    ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/xinference/model/llm/cache_manager.py", line 67, in __init__
    raise ValueError(
ValueError: [address=0.0.0.0:16691, pid=80] Model qwen3.6 (ggufv2, 35b, UD-Q4_K_M) does not declare a drafter model on hub modelscope. Please pass `draft_model_path` to use a local drafter.
2026-08-16T07:17:19.291Z INFO uvicorn.access pid:1 role:local address:0.0.0.0:9997 node:0b0dae70dc29 10.121.69.50:10213 - "POST /v1/models HTTP/1.1" 400

I know that's because there's no draft_model_id for qwen3.6 in .xinference/model/v2/builtin/llm/llm_models.json.

but xllamacpp custom parameters specified in Xinference WebUI will also not work, because they always seem to be treated as str.

• ......
• Advanced Configuration
  • ......
  • Engine Parameters (Additional Parameters: llama.cpp)
    • n_ctx: 983040
    • n_parallel: 5
    • speculative.types: [3]

......
2026-08-16T07:35:01.162Z ERROR xinference.model.llm.llama_cpp.core pid:20582 role:local address:0.0.0.0:16691 node:0b0dae70dc29 Failed to set the param multimodal_projector = mmproj-BF16.gguf, error: 'xllamacpp.xllamacpp.CommonParams' object has no attribute 'multimodal_projector'
2026-08-16T07:35:01.162Z ERROR xinference.model.llm.llama_cpp.core pid:20582 role:local address:0.0.0.0:16691 node:0b0dae70dc29 Failed to set the param speculative.types = [3], error: Argument 'values' has incorrect type (expected list, got str)
2026-08-16T07:35:01.162Z ERROR xinference.model.llm.llama_cpp.core pid:20582 role:local address:0.0.0.0:16691 node:0b0dae70dc29 Failed to set the param use_mmap = False, error: 'xllamacpp.xllamacpp.CommonParams' object has no attribute 'use_mmap'
2026-08-16T07:35:01.162Z ERROR xinference.model.llm.llama_cpp.core pid:20582 role:local address:0.0.0.0:16691 node:0b0dae70dc29 Failed to set the param use_mlock = True, error: 'xllamacpp.xllamacpp.CommonParams' object has no attribute 'use_mlock'
......

Expected behavior / 期待表现

When --enable_mtp true is specified, a separate drafter is no longer strictly required. (Support Qwen3.6-MTP)

Or users can add custom parameters for xllamacpp.

---

llama.cpp supports single-file MTP models (llama.cpp docs).

xllamacpp supports them too.

import xllamacpp

params = xllamacpp.CommonParams()

# I replaced the builtin non-MTP model with a MTP version
params.model.path = "/root/.xinference/modelscope/models/unsloth--Qwen3.6-35B-A3B-GGUF/snapshots/master/Qwen3.6-35B-A3B-UD-Q4_K_M.gguf"
params.n_ctx = 1024

params.speculative.types = [
    xllamacpp.common_speculative_type.COMMON_SPECULATIVE_TYPE_DRAFT_MTP
]
draft = params.speculative.draft
draft.n_max = 2
draft.n_min = 1

server = xllamacpp.Server(params)

result = server.handle_completions({
    "max_tokens": 256,
    "prompt": "I believe the meaning of life is",
    "temperature": 0.1,
})

print(result["choices"][0]["text"])

the output:

0.00.000.766 I srv          init: build_info: b10229-c745be2a2
0.00.000.774 I cmn  common_param: common_params_print_info: verbosity = 3 (adjust with the `-lv N` CLI arg)
0.00.108.011 W srv          init: -----------------
0.00.108.016 W srv          init: CORS is set to allow all origins ('*') and no API key is set
0.00.108.016 W srv          init: this can be a security risk (cross-origin attacks)
0.00.108.016 W srv          init: more info: https://github.com/ggml-org/llama.cpp/pull/25655
0.00.108.017 W srv          init: -----------------
0.00.109.242 I srv          init: loading model
0.00.109.269 I srv    load_model: loading model '/root/.xinference/modelscope/models/unsloth--Qwen3.6-35B-A3B-GGUF/snapshots/master/Qwen3.6-35B-A3B-UD-Q4_K_M.gguf'
0.05.012.482 I common_speculative_init_result: creating MTP draft context against the target model '/root/.xinference/modelscope/models/unsloth--Qwen3.6-35B-A3B-GGUF/snapshots/master/Qwen3.6-35B-A3B-UD-Q4_K_M.gguf'
0.05.055.531 I srv    load_model: initializing, n_slots = 1, n_ctx_slot = 1024, kv_unified = 'false'
0.05.154.922 I srv          init: chat template supports preserving reasoning, consider enabling it via --reasoning-preserve
0.05.154.939 I srv          init: model loaded
0.05.154.940 I srv          init: listening on http://127.0.0.1:46085
1.39.322.828 I slot get_availabl: id  0 | task -1 | selected slot by LRU, t_last = -1
1.39.322.888 I slot launch_slot_: id  0 | task 0 | processing task, is_child = 0
1.40.772.635 I slot print_timing: id  0 | task 0 | prompt eval time =     157.93 ms /     7 tokens (   22.56 ms per token,    44.32 tokens per second)
1.40.772.642 I slot print_timing: id  0 | task 0 |        eval time =    1291.80 ms /   256 tokens (    5.05 ms per token,   198.17 tokens per second)
1.40.772.643 I slot print_timing: id  0 | task 0 |       total time =    1449.72 ms /   263 tokens
1.40.772.643 I slot print_timing: id  0 | task 0 |    graphs reused =        104
1.40.772.645 I slot print_timing: id  0 | task 0 | draft acceptance = 0.70952 (  149 accepted /   210 generated), mean len =  2.42
1.40.772.672 I slot      release: id  0 | task 0 | stop processing: n_tokens = 262, truncated = 0

Thanks for your works!

[github-actions]

This issue is stale because it has been open for 14 days with no activity.