System Info / 系統信息
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 610.57.04 KMD Version: 610.57.04 CUDA UMD Version: 13.3 |
+-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA A100-SXM4-80GB On | 00000000:50:00.0 Off | 0 |
| N/A 29C P0 46W / 400W | 72933MiB / 81920MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+------------------------+----------------------+
| 1 NVIDIA A100-SXM4-80GB On | 00000000:51:00.0 Off | 0 |
| N/A 31C P0 49W / 400W | 72933MiB / 81920MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+------------------------+----------------------+
Running Xinference with Docker? / 是否使用 Docker 运行 Xinfernece？
Version info / 版本信息
xinferencev3.1.0
The command used to start Xinference / 用以启动 xinference 的命令
docker run -d
--name xinference310
--gpus all
--restart always
--ipc=host
--cap-add=SYS_PTRACE
--security-opt seccomp=unconfined
-p 9997:9997
-v /data/xinference/models:/root/.xinference
-v /data/xinference/xinference_auth.json:/root/.xinference/xinference_auth.json:ro
-e VLLM_DISABLE_COMPILE=1
-e TORCH_COMPILE_DISABLE=1
-e XINFERENCE_MODEL_SRC=modelscope
-e XINFERENCE_LOG_CONSOLE=false
-e VLLM_LOGGING_LEVEL=INFO
-e NCCL_DEBUG=INFO
-e HF_HUB_OFFLINE=1
docker.1ms.run/xprobe/xinference:v3.1.0
xinference-local -H 0.0.0.0 --log-level info
Reproduction / 复现过程
安装启动docker后默认的推理引擎是vllm0.21.0，无法进行双卡并行推理，会错误退出，必须手动进入虚拟环境重装vllm和flashinfer-python到0.23.0和0.6.12
‘’‘
Using Python 3.12.13 environment at: /root/.xinference/virtualenv/v4/qwen3.5/vllm/3.12.13
Resolved 2 packages in 7.15s
Uninstalled 2 packages in 256ms
Installed 2 packages in 162ms
- flashinfer-cubin==0.6.11.post3
- flashinfer-cubin==0.6.8.post1
- flashinfer-python==0.6.11.post3
- flashinfer-python==0.6.8.post1
WARNING 08-14 01:40:36 [config.py:70] Support for Transformers v4 is deprecated. The Transformers v4 codepath will become unmaintained in vLLM
v0.22.0 and will be removed in vLLM v0.24.0. Please upgrade to Transformers v5: pip install --upgrade transformers
WARNING 08-14 01:40:43 [envs.py:1866] Unknown vLLM environment variable detected: VLLM_DISTRIBUTED_EXECUTOR_BACKEND
loading configuration file /root/.xinference/cache/v2/qwen3_5-pytorch-27b-none/config.json
loading configuration file /root/.xinference/cache/v2/qwen3_5-pytorch-27b-none/config.json
loading configuration file /root/.xinference/cache/v2/qwen3_5-pytorch-27b-none/config.json
Model config Qwen3_5Config {
"architectures": [
"Qwen3_5ForConditionalGeneration"
],
"image_token_id": 248056,
"model_type": "qwen3_5",
"text_config": {
"attention_bias": false,
"attention_dropout": 0.0,
"attn_output_gate": true,
"dtype": "bfloat16",
"eos_token_id": 248044,
"full_attention_interval": 4,
"head_dim": 256,
"hidden_act": "silu",
"hidden_size": 5120,
"initializer_range": 0.02,
"intermediate_size": 17408,
"layer_types": [
"linear_attention",
"linear_attention",
"linear_attention",
"full_attention",
"linear_attention",
"linear_attention",
"linear_attention",
"full_attention",
"linear_attention",
"linear_attention",
"linear_attention",
"full_attention",
"linear_attention",
"linear_attention",
"linear_attention",
"full_attention",
"linear_attention",
"linear_attention",
"linear_attention",
"full_attention",
"linear_attention",
"linear_attention",
"linear_attention",
"full_attention",
"linear_attention",
"linear_attention",
"linear_attention",
"full_attention",
"linear_attention",
"linear_attention",
"linear_attention",
"full_attention",
"linear_attention",
"linear_attention",
"linear_attention",
"full_attention",
"linear_attention",
"linear_attention",
"linear_attention",
"full_attention",
"linear_attention",
"linear_attention",
"linear_attention",
"full_attention",
"linear_attention",
"linear_attention",
"linear_attention",
"full_attention",
"linear_attention",
"linear_attention",
"linear_attention",
"full_attention",
"linear_attention",
"linear_attention",
"linear_attention",
"full_attention",
"linear_attention",
"linear_attention",
"linear_attention",
"full_attention",
"linear_attention",
"linear_attention",
"linear_attention",
"full_attention"
],
"linear_conv_kernel_dim": 4,
"linear_key_head_dim": 128,
"linear_num_key_heads": 16,
"linear_num_value_heads": 48,
"linear_value_head_dim": 128,
"mamba_ssm_dtype": "float32",
"max_position_embeddings": 262144,
"mlp_only_layers": [],
"model_type": "qwen3_5_text",
"mtp_num_hidden_layers": 1,
"mtp_use_dedicated_embeddings": false,
"num_attention_heads": 24,
"num_hidden_layers": 64,
"num_key_value_heads": 4,
"partial_rotary_factor": 0.25,
"rms_norm_eps": 1e-06,
"rope_parameters": {
"mrope_interleaved": true,
"mrope_section": [
11,
11,
10
],
"partial_rotary_factor": 0.25,
"rope_theta": 10000000,
"rope_type": "default"
},
"use_cache": true,
"vocab_size": 248320
},
"tie_word_embeddings": false,
"transformers_version": "4.57.6",
"video_token_id": 248057,
"vision_config": {
"deepstack_visual_indexes": [],
"depth": 27,
"hidden_act": "gelu_pytorch_tanh",
"hidden_size": 1152,
"in_channels": 3,
"initializer_range": 0.02,
"intermediate_size": 4304,
"model_type": "qwen3_5",
"num_heads": 16,
"num_position_embeddings": 2304,
"out_hidden_size": 5120,
"patch_size": 16,
"spatial_merge_size": 2,
"temporal_patch_size": 2
},
"vision_end_token_id": 248054,
"vision_start_token_id": 248053
}
Offline mode: forcing local_files_only=True
INFO 08-14 01:40:43 [model.py:568] Resolved architecture: Qwen3_5ForConditionalGeneration
INFO 08-14 01:40:43 [model.py:1697] Using max model len 200000
INFO 08-14 01:40:43 [scheduler.py:239] Chunked prefill is enabled with max_num_batched_tokens=2048.
INFO 08-14 01:40:43 [vllm.py:886] Asynchronous scheduling is enabled.
INFO 08-14 01:40:43 [kernel.py:212] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms
_norm=['native'])
loading configuration file /root/.xinference/cache/v2/qwen3_5-pytorch-27b-none/config.json
loading configuration file /root/.xinference/cache/v2/qwen3_5-pytorch-27b-none/config.json
Model config Qwen3_5Config {
"architectures": [
"Qwen3_5ForConditionalGeneration"
],
"image_token_id": 248056,
"model_type": "qwen3_5",
"text_config": {
"attention_bias": false,
"attention_dropout": 0.0,
"attn_output_gate": true,
"dtype": "bfloat16",
"eos_token_id": 248044,
"full_attention_interval": 4,
"head_dim": 256,
"hidden_act": "silu",
"hidden_size": 5120,
"initializer_range": 0.02,
"intermediate_size": 17408,
"layer_types": [
"linear_attention",
"linear_attention",
"linear_attention",
"full_attention",
"linear_attention",
"linear_attention",
"linear_attention",
"full_attention",
"linear_attention",
"linear_attention",
"linear_attention",
"full_attention",
"linear_attention",
"linear_attention",
"linear_attention",
"full_attention",
"linear_attention",
"linear_attention",
"linear_attention",
"full_attention",
"linear_attention",
"linear_attention",
"linear_attention",
"full_attention",
"linear_attention",
"linear_attention",
"linear_attention",
"full_attention",
"linear_attention",
"linear_attention",
"linear_attention",
"full_attention",
"linear_attention",
"linear_attention",
"linear_attention",
"full_attention",
"linear_attention",
"linear_attention",
"linear_attention",
"full_attention",
"linear_attention",
"linear_attention",
"linear_attention",
"full_attention",
"linear_attention",
"linear_attention",
"linear_attention",
"full_attention",
"linear_attention",
"linear_attention",
"linear_attention",
"full_attention",
"linear_attention",
"linear_attention",
"linear_attention",
"full_attention",
"linear_attention",
"linear_attention",
"linear_attention",
"full_attention",
"linear_attention",
"linear_attention",
"linear_attention",
"full_attention"
],
"linear_conv_kernel_dim": 4,
"linear_key_head_dim": 128,
"linear_num_key_heads": 16,
"linear_num_value_heads": 48,
"linear_value_head_dim": 128,
"mamba_ssm_dtype": "float32",
"max_position_embeddings": 262144,
"mlp_only_layers": [],
"model_type": "qwen3_5_text",
"mtp_num_hidden_layers": 1,
"mtp_use_dedicated_embeddings": false,
"num_attention_heads": 24,
"num_hidden_layers": 64,
"num_key_value_heads": 4,
"partial_rotary_factor": 0.25,
"rms_norm_eps": 1e-06,
"rope_parameters": {
"mrope_interleaved": true,
"mrope_section": [
11,
11,
10
],
"partial_rotary_factor": 0.25,
"rope_theta": 10000000,
"rope_type": "default"
},
"use_cache": true,
"vocab_size": 248320
},
"tie_word_embeddings": false,
"transformers_version": "4.57.6",
"video_token_id": 248057,
"vision_config": {
"deepstack_visual_indexes": [],
"depth": 27,
"hidden_act": "gelu_pytorch_tanh",
"hidden_size": 1152,
"in_channels": 3,
"initializer_range": 0.02,
"intermediate_size": 4304,
"model_type": "qwen3_5",
"num_heads": 16,
"num_position_embeddings": 2304,
"out_hidden_size": 5120,
"patch_size": 16,
"spatial_merge_size": 2,
"temporal_patch_size": 2
},
"vision_end_token_id": 248054,
"vision_start_token_id": 248053
}
Offline mode: forcing local_files_only=True
Offline mode: forcing local_files_only=True
loading file vocab.json
loading file merges.txt
loading file tokenizer.json
loading file added_tokens.json
loading file special_tokens_map.json
loading file tokenizer_config.json
loading file chat_template.jinja
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
loading configuration file /root/.xinference/cache/v2/qwen3_5-pytorch-27b-none/generation_config.json
Generate config GenerationConfig {
"bos_token_id": 248044,
"do_sample": true,
"eos_token_id": [
248046,
248044
],
"pad_token_id": 248044,
"temperature": 0.6,
"top_k": 20,
"top_p": 0.95
}
Offline mode: forcing local_files_only=True
loading configuration file /root/.xinference/cache/v2/qwen3_5-pytorch-27b-none/preprocessor_config.json
Image processor Qwen2VLImageProcessorFast {
"crop_size": null,
"data_format": "channels_first",
"default_to_square": true,
"device": null,
"disable_grouping": null,
"do_center_crop": null,
"do_convert_rgb": true,
"do_normalize": true,
"do_pad": null,
"do_rescale": true,
"do_resize": true,
"image_mean": [
0.5,
0.5,
0.5
],
"image_processor_type": "Qwen2VLImageProcessorFast",
"image_std": [
0.5,
0.5,
0.5
],
"input_data_format": null,
"max_pixels": null,
"merge_size": 2,
"min_pixels": null,
"pad_size": null,
"patch_size": 16,
"processor_class": "Qwen3VLProcessor",
"resample": 3,
"rescale_factor": 0.00392156862745098,
"return_tensors": null,
"size": {
"longest_edge": 16777216,
"shortest_edge": 65536
},
"temporal_patch_size": 2
}
Offline mode: forcing local_files_only=True
loading file vocab.json
loading file merges.txt
loading file tokenizer.json
loading file added_tokens.json
loading file special_tokens_map.json
loading file tokenizer_config.json
loading file chat_template.jinja
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Offline mode: forcing local_files_only=True
loading configuration file /root/.xinference/cache/v2/qwen3_5-pytorch-27b-none/video_preprocessor_config.json
Video processor Qwen3VLVideoProcessor {
"crop_size": null,
"data_format": "channels_first",
"default_to_square": true,
"device": null,
"do_center_crop": null,
"do_convert_rgb": true,
"do_normalize": true,
"do_rescale": true,
"do_resize": true,
"do_sample_frames": true,
"fps": 2,
"image_mean": [
0.5,
0.5,
0.5
],
"image_std": [
0.5,
0.5,
0.5
],
"input_data_format": null,
"max_frames": 768,
"merge_size": 2,
"min_frames": 4,
"num_frames": null,
"pad_size": null,
"patch_size": 16,
"processor_class": "Qwen3VLProcessor",
"resample": 3,
"rescale_factor": 0.00392156862745098,
"return_metadata": false,
"size": {
"longest_edge": 25165824,
"shortest_edge": 4096
},
"temporal_patch_size": 2,
"video_metadata": null,
"video_processor_type": "Qwen3VLVideoProcessor"
}
Offline mode: forcing local_files_only=True
loading configuration file None
Processor Qwen3VLProcessor:
-
image_processor: Qwen2VLImageProcessorFast {
"crop_size": null,
"data_format": "channels_first",
"default_to_square": true,
"device": null,
"disable_grouping": null,
"do_center_crop": null,
"do_convert_rgb": true,
"do_normalize": true,
"do_pad": null,
"do_rescale": true,
"do_resize": true,
"image_mean": [
0.5,
0.5,
0.5
],
"image_processor_type": "Qwen2VLImageProcessorFast",
"image_std": [
0.5,
0.5,
0.5
],
"input_data_format": null,
"max_pixels": null,
"merge_size": 2,
"min_pixels": null,
"pad_size": null,
"patch_size": 16,
"processor_class": "Qwen3VLProcessor",
"resample": 3,
"rescale_factor": 0.00392156862745098,
"return_tensors": null,
"size": {
"longest_edge": 16777216,
"shortest_edge": 65536
},
"temporal_patch_size": 2
}
-
tokenizer: Qwen2TokenizerFast(name_or_path='/root/.xinference/cache/v2/qwen3_5-pytorch-27b-none', vocab_size=248044, model_max_length=262144,
is_fast=True, padding_side='right', truncation_side='right', special_tokens={'eos_token': '<|im_end|>', 'pad_token': '<|endoftext|>', 'additio
nal_special_tokens': ['<|im_start|>', '<|im_end|>', '<|object_ref_start|>', '<|object_ref_end|>', '<|box_start|>', '<|box_end|>', '<|quad_start
|>', '<|quad_end|>', '<|vision_start|>', '<|vision_end|>', '<|vision_pad|>', '<|image_pad|>', '<|video_pad|>'], 'audio_bos_token': '<|audio_sta
rt|>', 'audio_eos_token': '<|audio_end|>', 'audio_token': '<|audio_pad|>', 'image_token': '<|image_pad|>', 'video_token': '<|video_pad|>', 'vis
ion_bos_token': '<|vision_start|>', 'vision_eos_token': '<|vision_end|>'}, clean_up_tokenization_spaces=False, added_tokens_decoder={
248044: AddedToken("<|endoftext|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
248045: AddedToken("<|im_start|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
248046: AddedToken("<|im_end|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
248047: AddedToken("<|object_ref_start|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
248048: AddedToken("<|object_ref_end|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
248049: AddedToken("<|box_start|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
248050: AddedToken("<|box_end|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
248051: AddedToken("<|quad_start|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
248052: AddedToken("<|quad_end|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
248053: AddedToken("<|vision_start|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
248054: AddedToken("<|vision_end|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
248055: AddedToken("<|vision_pad|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
248056: AddedToken("<|image_pad|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
248057: AddedToken("<|video_pad|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
248058: AddedToken("<tool_call>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
248059: AddedToken("</tool_call>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
248060: AddedToken("<|fim_prefix|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
248061: AddedToken("<|fim_middle|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
248062: AddedToken("<|fim_suffix|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
248063: AddedToken("<|fim_pad|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
248064: AddedToken("<|repo_name|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
248065: AddedToken("<|file_sep|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
248066: AddedToken("<tool_response>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
248067: AddedToken("</tool_response>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
248068: AddedToken("", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
248069: AddedToken("", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
248070: AddedToken("<|audio_start|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
248071: AddedToken("<|audio_end|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
248072: AddedToken("<tts_pad>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
248073: AddedToken("<tts_text_bos>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
248074: AddedToken("<tts_text_eod>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
248075: AddedToken("<tts_text_bos_single>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
248076: AddedToken("<|audio_pad|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
}
)
-
video_processor: Qwen3VLVideoProcessor {
"crop_size": null,
"data_format": "channels_first",
"default_to_square": true,
"device": null,
"do_center_crop": null,
"do_convert_rgb": true,
"do_normalize": true,
"do_rescale": true,
"do_resize": true,
"do_sample_frames": true,
"fps": 2,
"image_mean": [
0.5,
0.5,
0.5
],
"image_std": [
0.5,
0.5,
0.5
],
"input_data_format": null,
"max_frames": 768,
"merge_size": 2,
"min_frames": 4,
"num_frames": null,
"pad_size": null,
"patch_size": 16,
"processor_class": "Qwen3VLProcessor",
"resample": 3,
"rescale_factor": 0.00392156862745098,
"return_metadata": false,
"size": {
"longest_edge": 25165824,
"shortest_edge": 4096
},
"temporal_patch_size": 2,
"video_metadata": null,
"video_processor_type": "Qwen3VLVideoProcessor"
}
{
"processor_class": "Qwen3VLProcessor"
}
Offline mode: forcing local_files_only=True
loading configuration file /root/.xinference/cache/v2/qwen3_5-pytorch-27b-none/preprocessor_config.json
Image processor Qwen2VLImageProcessorFast {
"crop_size": null,
"data_format": "channels_first",
"default_to_square": true,
"device": null,
"disable_grouping": null,
"do_center_crop": null,
"do_convert_rgb": true,
"do_normalize": true,
"do_pad": null,
"do_rescale": true,
"do_resize": true,
"image_mean": [
0.5,
0.5,
0.5
],
"image_processor_type": "Qwen2VLImageProcessorFast",
"image_std": [
0.5,
0.5,
0.5
],
"input_data_format": null,
"max_pixels": null,
"merge_size": 2,
"min_pixels": null,
"pad_size": null,
"patch_size": 16,
"processor_class": "Qwen3VLProcessor",
"resample": 3,
"rescale_factor": 0.00392156862745098,
"return_tensors": null,
"size": {
"longest_edge": 16777216,
"shortest_edge": 65536
},
"temporal_patch_size": 2
}
Offline mode: forcing local_files_only=True
loading file vocab.json
loading file merges.txt
loading file tokenizer.json
loading file added_tokens.json
loading file special_tokens_map.json
loading file tokenizer_config.json
loading file chat_template.jinja
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Offline mode: forcing local_files_only=True
loading configuration file /root/.xinference/cache/v2/qwen3_5-pytorch-27b-none/video_preprocessor_config.json
Video processor Qwen3VLVideoProcessor {
"crop_size": null,
"data_format": "channels_first",
"default_to_square": true,
"device": null,
"do_center_crop": null,
"do_convert_rgb": true,
"do_normalize": true,
"do_rescale": true,
"do_resize": true,
"do_sample_frames": true,
"fps": 2,
"image_mean": [
0.5,
0.5,
0.5
],
"image_std": [
0.5,
0.5,
0.5
],
"input_data_format": null,
"max_frames": 768,
"merge_size": 2,
"min_frames": 4,
"num_frames": null,
"pad_size": null,
"patch_size": 16,
"processor_class": "Qwen3VLProcessor",
"resample": 3,
"rescale_factor": 0.00392156862745098,
"return_metadata": false,
"size": {
"longest_edge": 25165824,
"shortest_edge": 4096
},
"temporal_patch_size": 2,
"video_metadata": null,
"video_processor_type": "Qwen3VLVideoProcessor"
}
Offline mode: forcing local_files_only=True
loading configuration file None
Processor Qwen3VLProcessor:
-
image_processor: Qwen2VLImageProcessorFast {
"crop_size": null,
"data_format": "channels_first",
"default_to_square": true,
"device": null,
"disable_grouping": null,
"do_center_crop": null,
"do_convert_rgb": true,
"do_normalize": true,
"do_pad": null,
"do_rescale": true,
"do_resize": true,
"image_mean": [
0.5,
0.5,
0.5
],
"image_processor_type": "Qwen2VLImageProcessorFast",
"image_std": [
0.5,
0.5,
0.5
],
"input_data_format": null,
"max_pixels": null,
"merge_size": 2,
"min_pixels": null,
"pad_size": null,
"patch_size": 16,
"processor_class": "Qwen3VLProcessor",
"resample": 3,
"rescale_factor": 0.00392156862745098,
"return_tensors": null,
"size": {
"longest_edge": 16777216,
"shortest_edge": 65536
},
"temporal_patch_size": 2
}
-
tokenizer: CachedQwen2TokenizerFast(name_or_path='/root/.xinference/cache/v2/qwen3_5-pytorch-27b-none', vocab_size=248044, model_max_length=2
62144, is_fast=True, padding_side='right', truncation_side='left', special_tokens={'eos_token': '<|im_end|>', 'pad_token': '<|endoftext|>', 'ad
ditional_special_tokens': ['<|im_start|>', '<|im_end|>', '<|object_ref_start|>', '<|object_ref_end|>', '<|box_start|>', '<|box_end|>', '<|quad_
start|>', '<|quad_end|>', '<|vision_start|>', '<|vision_end|>', '<|vision_pad|>', '<|image_pad|>', '<|video_pad|>'], 'audio_bos_token': '<|audi
o_start|>', 'audio_eos_token': '<|audio_end|>', 'audio_token': '<|audio_pad|>', 'image_token': '<|image_pad|>', 'video_token': '<|video_pad|>',
'vision_bos_token': '<|vision_start|>', 'vision_eos_token': '<|vision_end|>'}, clean_up_tokenization_spaces=False, added_tokens_decoder={
248044: AddedToken("<|endoftext|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
248045: AddedToken("<|im_start|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
248046: AddedToken("<|im_end|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
248047: AddedToken("<|object_ref_start|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
248048: AddedToken("<|object_ref_end|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
248049: AddedToken("<|box_start|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
248050: AddedToken("<|box_end|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
248051: AddedToken("<|quad_start|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
248052: AddedToken("<|quad_end|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
248053: AddedToken("<|vision_start|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
248054: AddedToken("<|vision_end|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
248055: AddedToken("<|vision_pad|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
248056: AddedToken("<|image_pad|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
248057: AddedToken("<|video_pad|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
248058: AddedToken("<tool_call>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
248059: AddedToken("</tool_call>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
248060: AddedToken("<|fim_prefix|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
248061: AddedToken("<|fim_middle|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
248062: AddedToken("<|fim_suffix|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
248063: AddedToken("<|fim_pad|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
248064: AddedToken("<|repo_name|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
248065: AddedToken("<|file_sep|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
248066: AddedToken("<tool_response>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
248067: AddedToken("</tool_response>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
248068: AddedToken("", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
248069: AddedToken("", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
248070: AddedToken("<|audio_start|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
248071: AddedToken("<|audio_end|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
248072: AddedToken("<tts_pad>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
248073: AddedToken("<tts_text_bos>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
248074: AddedToken("<tts_text_eod>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
248075: AddedToken("<tts_text_bos_single>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
248076: AddedToken("<|audio_pad|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
}
)
-
video_processor: Qwen3VLVideoProcessor {
"crop_size": null,
"data_format": "channels_first",
"default_to_square": true,
"device": null,
"do_center_crop": null,
"do_convert_rgb": true,
"do_normalize": true,
"do_rescale": true,
"do_resize": true,
"do_sample_frames": true,
"fps": 2,
"image_mean": [
0.5,
0.5,
0.5
],
"image_std": [
0.5,
0.5,
0.5
],
"input_data_format": null,
"max_frames": 768,
"merge_size": 2,
"min_frames": 4,
"num_frames": null,
"pad_size": null,
"patch_size": 16,
"processor_class": "Qwen3VLProcessor",
"resample": 3,
"rescale_factor": 0.00392156862745098,
"return_metadata": false,
"size": {
"longest_edge": 25165824,
"shortest_edge": 4096
},
"temporal_patch_size": 2,
"video_metadata": null,
"video_processor_type": "Qwen3VLVideoProcessor"
}
{
"processor_class": "Qwen3VLProcessor"
}
WARNING 08-14 01:40:50 [system_utils.py:157] We must use the spawn multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to '
spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: CUDA is initialized
/usr/local/lib/python3.12/dist-packages/torch/cuda/init.py:61: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py
instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
import pynvml # type: ignore[import]
WARNING 08-14 01:40:55 [config.py:70] Support for Transformers v4 is deprecated. The Transformers v4 codepath will become unmaintained in vLLM
v0.22.0 and will be removed in vLLM v0.24.0. Please upgrade to Transformers v5: pip install --upgrade transformers
(EngineCore pid=923) INFO 08-14 01:41:00 [core.py:109] Initializing a V1 LLM engine (v0.21.0) with config: model='/root/.xinference/cache/v2/qw
en3_5-pytorch-27b-none', speculative_config=None, tokenizer='/root/.xinference/cache/v2/qwen3_5-pytorch-27b-none', skip_tokenizer_init=False, t
okenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=200000, download_dir=Non
e, load_format=auto, tensor_parallel_size=2, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=a
g_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=False, enable_return_routed_experts=False, kv
cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable
additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityCon
fig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sam
ple=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_loggi
ng_iteration_details=False), seed=0, served_model_name=/root/.xinference/cache/v2/qwen3_5-pytorch-27b-none, enable_prefix_caching=False, enable
chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir':
'', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vll
m::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'v
llm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_for
ward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::unifie
d_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token
budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_
ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': Fals
e, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FUL
L_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64], 'cudagraph_copy_i
nputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_qu
ant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, '
max_cudagraph_capture_size': 64, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_b
it_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_pri
ority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=False, moe_backend='auto')
WARNING 08-14 01:41:04 [config.py:70] Support for Transformers v4 is deprecated. The Transformers v4 codepath will become unmaintained in vLLM
v0.22.0 and will be removed in vLLM v0.24.0. Please upgrade to Transformers v5: pip install --upgrade transformers
WARNING 08-14 01:41:05 [config.py:70] Support for Transformers v4 is deprecated. The Transformers v4 codepath will become unmaintained in vLLM
v0.22.0 and will be removed in vLLM v0.24.0. Please upgrade to Transformers v5: pip install --upgrade transformers
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] EngineCore failed to start.
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] Traceback (most recent call last):
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] File "/root/.xinference/virtualenv/v4/qwen3.5/vllm/3.12.13/lib/python3.12/site-packa
ges/vllm/v1/engine/core.py", line 1114, in run_engine_core
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] File "/root/.xinference/virtualenv/v4/qwen3.5/vllm/3.12.13/lib/python3.12/site-packa
ges/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] return func(*args, **kwargs)
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] File "/root/.xinference/virtualenv/v4/qwen3.5/vllm/3.12.13/lib/python3.12/site-packa
ges/vllm/v1/engine/core.py", line 880, in init
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] super().init(
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] File "/root/.xinference/virtualenv/v4/qwen3.5/vllm/3.12.13/lib/python3.12/site-packa
ges/vllm/v1/engine/core.py", line 118, in init
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] self.model_executor = executor_class(vllm_config)
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] File "/opt/inference/xinference/model/llm/vllm/distributed_executor_v1.py", line 154
, in init
(EngineCore pid=923) Process EngineCore:
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] Executor.init(self, vllm_config, *args, **kwargs)
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] File "/root/.xinference/virtualenv/v4/qwen3.5/vllm/3.12.13/lib/python3.12/site-packa
ges/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] return func(*args, **kwargs)
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] File "/root/.xinference/virtualenv/v4/qwen3.5/vllm/3.12.13/lib/python3.12/site-packa
ges/vllm/v1/executor/abstract.py", line 109, in init
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] self._init_executor()
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] File "/opt/inference/xinference/model/llm/vllm/distributed_executor_v1.py", line 175
, in _init_executor
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] refs: List[xo.ActorRefType[WorkerActor]] = [fut.result() for fut in futures]
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] ^^^^^^^^^^^^
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] File "/usr/lib/python3.12/concurrent/futures/_base.py", line 456, in result
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] return self.__get_result()
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] ^^^^^^^^^^^^^^^^^^^
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] File "/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] raise self._exception
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] File "/usr/local/lib/python3.12/dist-packages/xoscar/api.py", line 79, in create_act
or
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] return await ctx.create_actor(actor_cls, *args, uid=uid, address=address, **kwargs
)
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
^
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] File "/usr/local/lib/python3.12/dist-packages/xoscar/backends/context.py", line 152,
in create_actor
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] return self._process_result_message(result)
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] File "/usr/local/lib/python3.12/dist-packages/xoscar/backends/context.py", line 111,
in _process_result_message
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] raise message.as_instanceof_cause()
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] File "/usr/local/lib/python3.12/dist-packages/xoscar/backends/pool.py", line 441, in
process_message
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] processor.result = await self._run_coro(
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] ^^^^^^^^^^^^^^^^^
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] File "/usr/local/lib/python3.12/dist-packages/xoscar/backends/pool.py", line 389, in
_run_coro
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] return await coro
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] File "/usr/local/lib/python3.12/dist-packages/xoscar/backends/pool.py", line 923, in
create_actor
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] await self.notify_main_pool_to_create(message)
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] ^^^^^^^^^^^^^^^^^
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] File "/usr/local/lib/python3.12/dist-packages/xoscar/backends/pool.py", line 917, in
notify_main_pool_to_create
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] await self.call(self._main_address, reg_message)
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] ^^^^^^^^^^^^^^^^^
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] File "/usr/local/lib/python3.12/dist-packages/xoscar/backends/pool.py", line 448, in
call
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] return await self._caller.call(self._router, dest_address, message) # type: ignor
e
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] ^^^^^^^^^^^^^^^^^
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] File "/usr/local/lib/python3.12/dist-packages/xoscar/backends/core.py", line 222, in
call
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] return await self.call_with_client(client, message, wait)
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] ^^^^^^^^^^^^^^^^^
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] File "/usr/local/lib/python3.12/dist-packages/xoscar/backends/core.py", line 167, in
call_with_client
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] r = await wait_response
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] ^^^^^^^^^^^^^^^^^
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] File "/usr/local/lib/python3.12/dist-packages/xoscar/backends/core.py", line 104, in
_listen
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] raise ServerClosed(
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] ^^^^^^^^^^^^^^^^^
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] xoscar.errors.ServerClosed: [address=0.0.0.0:44277, pid=772] Remote server unixsocket:
///8060928 closed: 0 bytes read on a total of 11 expected bytes
(EngineCore pid=923) Traceback (most recent call last):
(EngineCore pid=923) File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
(EngineCore pid=923) self.run()
(EngineCore pid=923) File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
(EngineCore pid=923) self._target(*self._args, **self._kwargs)
(EngineCore pid=923) File "/root/.xinference/virtualenv/v4/qwen3.5/vllm/3.12.13/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 11
44, in run_engine_core
(EngineCore pid=923) raise e
(EngineCore pid=923) File "/root/.xinference/virtualenv/v4/qwen3.5/vllm/3.12.13/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 11
14, in run_engine_core
(EngineCore pid=923) engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(EngineCore pid=923) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=923) File "/root/.xinference/virtualenv/v4/qwen3.5/vllm/3.12.13/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178,
in sync_wrapper
(EngineCore pid=923) return func(*args, **kwargs)
(EngineCore pid=923) ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=923) File "/root/.xinference/virtualenv/v4/qwen3.5/vllm/3.12.13/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 88
0, in init
(EngineCore pid=923) super().init(
(EngineCore pid=923) File "/root/.xinference/virtualenv/v4/qwen3.5/vllm/3.12.13/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 11
8, in init
(EngineCore pid=923) self.model_executor = executor_class(vllm_config)
(EngineCore pid=923) ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=923) File "/opt/inference/xinference/model/llm/vllm/distributed_executor_v1.py", line 154, in init
(EngineCore pid=923) Executor.init(self, vllm_config, *args, **kwargs)
(EngineCore pid=923) File "/root/.xinference/virtualenv/v4/qwen3.5/vllm/3.12.13/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178,
in sync_wrapper
(EngineCore pid=923) return func(*args, **kwargs)
(EngineCore pid=923) ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=923) File "/root/.xinference/virtualenv/v4/qwen3.5/vllm/3.12.13/lib/python3.12/site-packages/vllm/v1/executor/abstract.py", l
ine 109, in init
(EngineCore pid=923) self._init_executor()
(EngineCore pid=923) File "/opt/inference/xinference/model/llm/vllm/distributed_executor_v1.py", line 175, in _init_executor
(EngineCore pid=923) refs: List[xo.ActorRefType[WorkerActor]] = [fut.result() for fut in futures]
(EngineCore pid=923) ^^^^^^^^^^^^
(EngineCore pid=923) File "/usr/lib/python3.12/concurrent/futures/_base.py", line 456, in result
(EngineCore pid=923) return self.__get_result()
(EngineCore pid=923) ^^^^^^^^^^^^^^^^^^^
(EngineCore pid=923) File "/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
(EngineCore pid=923) raise self._exception
(EngineCore pid=923) File "/usr/local/lib/python3.12/dist-packages/xoscar/api.py", line 79, in create_actor
(EngineCore pid=923) return await ctx.create_actor(actor_cls, *args, uid=uid, address=address, **kwargs)
(EngineCore pid=923) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=923) File "/usr/local/lib/python3.12/dist-packages/xoscar/backends/context.py", line 152, in create_actor
(EngineCore pid=923) return self._process_result_message(result)
(EngineCore pid=923) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=923) File "/usr/local/lib/python3.12/dist-packages/xoscar/backends/context.py", line 111, in _process_result_message
(EngineCore pid=923) raise message.as_instanceof_cause()
(EngineCore pid=923) File "/usr/local/lib/python3.12/dist-packages/xoscar/backends/pool.py", line 441, in process_message
(EngineCore pid=923) processor.result = await self._run_coro(
(EngineCore pid=923) ^^^^^^^^^^^^^^^^^
(EngineCore pid=923) File "/usr/local/lib/python3.12/dist-packages/xoscar/backends/pool.py", line 389, in _run_coro
(EngineCore pid=923) return await coro
(EngineCore pid=923) File "/usr/local/lib/python3.12/dist-packages/xoscar/backends/pool.py", line 923, in create_actor
(EngineCore pid=923) await self.notify_main_pool_to_create(message)
(EngineCore pid=923) ^^^^^^^^^^^^^^^^^
(EngineCore pid=923) File "/usr/local/lib/python3.12/dist-packages/xoscar/backends/pool.py", line 917, in notify_main_pool_to_create
(EngineCore pid=923) await self.call(self._main_address, reg_message)
(EngineCore pid=923) ^^^^^^^^^^^^^^^^^
(EngineCore pid=923) File "/usr/local/lib/python3.12/dist-packages/xoscar/backends/pool.py", line 448, in call
(EngineCore pid=923) return await self._caller.call(self._router, dest_address, message) # type: ignore
(EngineCore pid=923) ^^^^^^^^^^^^^^^^^
(EngineCore pid=923) File "/usr/local/lib/python3.12/dist-packages/xoscar/backends/core.py", line 222, in call
(EngineCore pid=923) return await self.call_with_client(client, message, wait)
(EngineCore pid=923) ^^^^^^^^^^^^^^^^^
(EngineCore pid=923) File "/usr/local/lib/python3.12/dist-packages/xoscar/backends/core.py", line 167, in call_with_client
(EngineCore pid=923) r = await wait_response
(EngineCore pid=923) ^^^^^^^^^^^^^^^^^
(EngineCore pid=923) File "/usr/local/lib/python3.12/dist-packages/xoscar/backends/core.py", line 104, in _listen
(EngineCore pid=923) raise ServerClosed(
(EngineCore pid=923) ^^^^^^^^^^^^^^^^^
(EngineCore pid=923) xoscar.errors.ServerClosed: [address=0.0.0.0:44277, pid=772] Remote server unixsocket:///8060928 closed: 0 bytes read on a
total of 11 expected bytes
’‘’
Expected behavior / 期待表现
能不能不要明确支持的模型的默认虚拟环境都是存在问题的，如果不能确保支持，至少在说明描述中提及明确支持的运行环境要求以及未测试的边缘场景可能性以方便排查错误
此外，现在如果不设置XINFERENCE_LOG_CONSOLE=false，dockerlogs信息中会在启动模型时有大量来自web查询进度的get请求刷屏，完全屏蔽其他信息，关闭后又会有其他信息不全，这些明显无价值的信息最好能默认屏蔽
System Info / 系統信息
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 610.57.04 KMD Version: 610.57.04 CUDA UMD Version: 13.3 |
+-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA A100-SXM4-80GB On | 00000000:50:00.0 Off | 0 |
| N/A 29C P0 46W / 400W | 72933MiB / 81920MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+------------------------+----------------------+
| 1 NVIDIA A100-SXM4-80GB On | 00000000:51:00.0 Off | 0 |
| N/A 31C P0 49W / 400W | 72933MiB / 81920MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+------------------------+----------------------+
Running Xinference with Docker? / 是否使用 Docker 运行 Xinfernece？
Version info / 版本信息
xinferencev3.1.0
The command used to start Xinference / 用以启动 xinference 的命令
docker run -d
--name xinference310
--gpus all
--restart always
--ipc=host
--cap-add=SYS_PTRACE
--security-opt seccomp=unconfined
-p 9997:9997
-v /data/xinference/models:/root/.xinference
-v /data/xinference/xinference_auth.json:/root/.xinference/xinference_auth.json:ro
-e VLLM_DISABLE_COMPILE=1
-e TORCH_COMPILE_DISABLE=1
-e XINFERENCE_MODEL_SRC=modelscope
-e XINFERENCE_LOG_CONSOLE=false
-e VLLM_LOGGING_LEVEL=INFO
-e NCCL_DEBUG=INFO
-e HF_HUB_OFFLINE=1
docker.1ms.run/xprobe/xinference:v3.1.0
xinference-local -H 0.0.0.0 --log-level info
Reproduction / 复现过程
安装启动docker后默认的推理引擎是vllm0.21.0，无法进行双卡并行推理，会错误退出，必须手动进入虚拟环境重装vllm和flashinfer-python到0.23.0和0.6.12
‘’‘
Using Python 3.12.13 environment at: /root/.xinference/virtualenv/v4/qwen3.5/vllm/3.12.13
Resolved 2 packages in 7.15s
Uninstalled 2 packages in 256ms
Installed 2 packages in 162ms
WARNING 08-14 01:40:36 [config.py:70] Support for Transformers v4 is deprecated. The Transformers v4 codepath will become unmaintained in vLLM
v0.22.0 and will be removed in vLLM v0.24.0. Please upgrade to Transformers v5: pip install --upgrade transformers
WARNING 08-14 01:40:43 [envs.py:1866] Unknown vLLM environment variable detected: VLLM_DISTRIBUTED_EXECUTOR_BACKEND
loading configuration file /root/.xinference/cache/v2/qwen3_5-pytorch-27b-none/config.json
loading configuration file /root/.xinference/cache/v2/qwen3_5-pytorch-27b-none/config.json
loading configuration file /root/.xinference/cache/v2/qwen3_5-pytorch-27b-none/config.json
Model config Qwen3_5Config {
"architectures": [
"Qwen3_5ForConditionalGeneration"
],
"image_token_id": 248056,
"model_type": "qwen3_5",
"text_config": {
"attention_bias": false,
"attention_dropout": 0.0,
"attn_output_gate": true,
"dtype": "bfloat16",
"eos_token_id": 248044,
"full_attention_interval": 4,
"head_dim": 256,
"hidden_act": "silu",
"hidden_size": 5120,
"initializer_range": 0.02,
"intermediate_size": 17408,
"layer_types": [
"linear_attention",
"linear_attention",
"linear_attention",
"full_attention",
"linear_attention",
"linear_attention",
"linear_attention",
"full_attention",
"linear_attention",
"linear_attention",
"linear_attention",
"full_attention",
"linear_attention",
"linear_attention",
"linear_attention",
"full_attention",
"linear_attention",
"linear_attention",
"linear_attention",
"full_attention",
"linear_attention",
"linear_attention",
"linear_attention",
"full_attention",
"linear_attention",
"linear_attention",
"linear_attention",
"full_attention",
"linear_attention",
"linear_attention",
"linear_attention",
"full_attention",
"linear_attention",
"linear_attention",
"linear_attention",
"full_attention",
"linear_attention",
"linear_attention",
"linear_attention",
"full_attention",
"linear_attention",
"linear_attention",
"linear_attention",
"full_attention",
"linear_attention",
"linear_attention",
"linear_attention",
"full_attention",
"linear_attention",
"linear_attention",
"linear_attention",
"full_attention",
"linear_attention",
"linear_attention",
"linear_attention",
"full_attention",
"linear_attention",
"linear_attention",
"linear_attention",
"full_attention",
"linear_attention",
"linear_attention",
"linear_attention",
"full_attention"
],
"linear_conv_kernel_dim": 4,
"linear_key_head_dim": 128,
"linear_num_key_heads": 16,
"linear_num_value_heads": 48,
"linear_value_head_dim": 128,
"mamba_ssm_dtype": "float32",
"max_position_embeddings": 262144,
"mlp_only_layers": [],
"model_type": "qwen3_5_text",
"mtp_num_hidden_layers": 1,
"mtp_use_dedicated_embeddings": false,
"num_attention_heads": 24,
"num_hidden_layers": 64,
"num_key_value_heads": 4,
"partial_rotary_factor": 0.25,
"rms_norm_eps": 1e-06,
"rope_parameters": {
"mrope_interleaved": true,
"mrope_section": [
11,
11,
10
],
"partial_rotary_factor": 0.25,
"rope_theta": 10000000,
"rope_type": "default"
},
"use_cache": true,
"vocab_size": 248320
},
"tie_word_embeddings": false,
"transformers_version": "4.57.6",
"video_token_id": 248057,
"vision_config": {
"deepstack_visual_indexes": [],
"depth": 27,
"hidden_act": "gelu_pytorch_tanh",
"hidden_size": 1152,
"in_channels": 3,
"initializer_range": 0.02,
"intermediate_size": 4304,
"model_type": "qwen3_5",
"num_heads": 16,
"num_position_embeddings": 2304,
"out_hidden_size": 5120,
"patch_size": 16,
"spatial_merge_size": 2,
"temporal_patch_size": 2
},
"vision_end_token_id": 248054,
"vision_start_token_id": 248053
}
Offline mode: forcing local_files_only=True
INFO 08-14 01:40:43 [model.py:568] Resolved architecture: Qwen3_5ForConditionalGeneration
INFO 08-14 01:40:43 [model.py:1697] Using max model len 200000
INFO 08-14 01:40:43 [scheduler.py:239] Chunked prefill is enabled with max_num_batched_tokens=2048.
INFO 08-14 01:40:43 [vllm.py:886] Asynchronous scheduling is enabled.
INFO 08-14 01:40:43 [kernel.py:212] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms
_norm=['native'])
loading configuration file /root/.xinference/cache/v2/qwen3_5-pytorch-27b-none/config.json
loading configuration file /root/.xinference/cache/v2/qwen3_5-pytorch-27b-none/config.json
Model config Qwen3_5Config {
"architectures": [
"Qwen3_5ForConditionalGeneration"
],
"image_token_id": 248056,
"model_type": "qwen3_5",
"text_config": {
"attention_bias": false,
"attention_dropout": 0.0,
"attn_output_gate": true,
"dtype": "bfloat16",
"eos_token_id": 248044,
"full_attention_interval": 4,
"head_dim": 256,
"hidden_act": "silu",
"hidden_size": 5120,
"initializer_range": 0.02,
"intermediate_size": 17408,
"layer_types": [
"linear_attention",
"linear_attention",
"linear_attention",
"full_attention",
"linear_attention",
"linear_attention",
"linear_attention",
"full_attention",
"linear_attention",
"linear_attention",
"linear_attention",
"full_attention",
"linear_attention",
"linear_attention",
"linear_attention",
"full_attention",
"linear_attention",
"linear_attention",
"linear_attention",
"full_attention",
"linear_attention",
"linear_attention",
"linear_attention",
"full_attention",
"linear_attention",
"linear_attention",
"linear_attention",
"full_attention",
"linear_attention",
"linear_attention",
"linear_attention",
"full_attention",
"linear_attention",
"linear_attention",
"linear_attention",
"full_attention",
"linear_attention",
"linear_attention",
"linear_attention",
"full_attention",
"linear_attention",
"linear_attention",
"linear_attention",
"full_attention",
"linear_attention",
"linear_attention",
"linear_attention",
"full_attention",
"linear_attention",
"linear_attention",
"linear_attention",
"full_attention",
"linear_attention",
"linear_attention",
"linear_attention",
"full_attention",
"linear_attention",
"linear_attention",
"linear_attention",
"full_attention",
"linear_attention",
"linear_attention",
"linear_attention",
"full_attention"
],
"linear_conv_kernel_dim": 4,
"linear_key_head_dim": 128,
"linear_num_key_heads": 16,
"linear_num_value_heads": 48,
"linear_value_head_dim": 128,
"mamba_ssm_dtype": "float32",
"max_position_embeddings": 262144,
"mlp_only_layers": [],
"model_type": "qwen3_5_text",
"mtp_num_hidden_layers": 1,
"mtp_use_dedicated_embeddings": false,
"num_attention_heads": 24,
"num_hidden_layers": 64,
"num_key_value_heads": 4,
"partial_rotary_factor": 0.25,
"rms_norm_eps": 1e-06,
"rope_parameters": {
"mrope_interleaved": true,
"mrope_section": [
11,
11,
10
],
"partial_rotary_factor": 0.25,
"rope_theta": 10000000,
"rope_type": "default"
},
"use_cache": true,
"vocab_size": 248320
},
"tie_word_embeddings": false,
"transformers_version": "4.57.6",
"video_token_id": 248057,
"vision_config": {
"deepstack_visual_indexes": [],
"depth": 27,
"hidden_act": "gelu_pytorch_tanh",
"hidden_size": 1152,
"in_channels": 3,
"initializer_range": 0.02,
"intermediate_size": 4304,
"model_type": "qwen3_5",
"num_heads": 16,
"num_position_embeddings": 2304,
"out_hidden_size": 5120,
"patch_size": 16,
"spatial_merge_size": 2,
"temporal_patch_size": 2
},
"vision_end_token_id": 248054,
"vision_start_token_id": 248053
}
Offline mode: forcing local_files_only=True
Offline mode: forcing local_files_only=True
loading file vocab.json
loading file merges.txt
loading file tokenizer.json
loading file added_tokens.json
loading file special_tokens_map.json
loading file tokenizer_config.json
loading file chat_template.jinja
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
loading configuration file /root/.xinference/cache/v2/qwen3_5-pytorch-27b-none/generation_config.json
Generate config GenerationConfig {
"bos_token_id": 248044,
"do_sample": true,
"eos_token_id": [
248046,
248044
],
"pad_token_id": 248044,
"temperature": 0.6,
"top_k": 20,
"top_p": 0.95
}
Offline mode: forcing local_files_only=True
loading configuration file /root/.xinference/cache/v2/qwen3_5-pytorch-27b-none/preprocessor_config.json
Image processor Qwen2VLImageProcessorFast {
"crop_size": null,
"data_format": "channels_first",
"default_to_square": true,
"device": null,
"disable_grouping": null,
"do_center_crop": null,
"do_convert_rgb": true,
"do_normalize": true,
"do_pad": null,
"do_rescale": true,
"do_resize": true,
"image_mean": [
0.5,
0.5,
0.5
],
"image_processor_type": "Qwen2VLImageProcessorFast",
"image_std": [
0.5,
0.5,
0.5
],
"input_data_format": null,
"max_pixels": null,
"merge_size": 2,
"min_pixels": null,
"pad_size": null,
"patch_size": 16,
"processor_class": "Qwen3VLProcessor",
"resample": 3,
"rescale_factor": 0.00392156862745098,
"return_tensors": null,
"size": {
"longest_edge": 16777216,
"shortest_edge": 65536
},
"temporal_patch_size": 2
}
Offline mode: forcing local_files_only=True
loading file vocab.json
loading file merges.txt
loading file tokenizer.json
loading file added_tokens.json
loading file special_tokens_map.json
loading file tokenizer_config.json
loading file chat_template.jinja
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Offline mode: forcing local_files_only=True
loading configuration file /root/.xinference/cache/v2/qwen3_5-pytorch-27b-none/video_preprocessor_config.json
Video processor Qwen3VLVideoProcessor {
"crop_size": null,
"data_format": "channels_first",
"default_to_square": true,
"device": null,
"do_center_crop": null,
"do_convert_rgb": true,
"do_normalize": true,
"do_rescale": true,
"do_resize": true,
"do_sample_frames": true,
"fps": 2,
"image_mean": [
0.5,
0.5,
0.5
],
"image_std": [
0.5,
0.5,
0.5
],
"input_data_format": null,
"max_frames": 768,
"merge_size": 2,
"min_frames": 4,
"num_frames": null,
"pad_size": null,
"patch_size": 16,
"processor_class": "Qwen3VLProcessor",
"resample": 3,
"rescale_factor": 0.00392156862745098,
"return_metadata": false,
"size": {
"longest_edge": 25165824,
"shortest_edge": 4096
},
"temporal_patch_size": 2,
"video_metadata": null,
"video_processor_type": "Qwen3VLVideoProcessor"
}
Offline mode: forcing local_files_only=True
loading configuration file None
Processor Qwen3VLProcessor:
image_processor: Qwen2VLImageProcessorFast {
"crop_size": null,
"data_format": "channels_first",
"default_to_square": true,
"device": null,
"disable_grouping": null,
"do_center_crop": null,
"do_convert_rgb": true,
"do_normalize": true,
"do_pad": null,
"do_rescale": true,
"do_resize": true,
"image_mean": [
0.5,
0.5,
0.5
],
"image_processor_type": "Qwen2VLImageProcessorFast",
"image_std": [
0.5,
0.5,
0.5
],
"input_data_format": null,
"max_pixels": null,
"merge_size": 2,
"min_pixels": null,
"pad_size": null,
"patch_size": 16,
"processor_class": "Qwen3VLProcessor",
"resample": 3,
"rescale_factor": 0.00392156862745098,
"return_tensors": null,
"size": {
"longest_edge": 16777216,
"shortest_edge": 65536
},
"temporal_patch_size": 2
}
tokenizer: Qwen2TokenizerFast(name_or_path='/root/.xinference/cache/v2/qwen3_5-pytorch-27b-none', vocab_size=248044, model_max_length=262144,
is_fast=True, padding_side='right', truncation_side='right', special_tokens={'eos_token': '<|im_end|>', 'pad_token': '<|endoftext|>', 'additio
nal_special_tokens': ['<|im_start|>', '<|im_end|>', '<|object_ref_start|>', '<|object_ref_end|>', '<|box_start|>', '<|box_end|>', '<|quad_start
|>', '<|quad_end|>', '<|vision_start|>', '<|vision_end|>', '<|vision_pad|>', '<|image_pad|>', '<|video_pad|>'], 'audio_bos_token': '<|audio_sta
rt|>', 'audio_eos_token': '<|audio_end|>', 'audio_token': '<|audio_pad|>', 'image_token': '<|image_pad|>', 'video_token': '<|video_pad|>', 'vis
ion_bos_token': '<|vision_start|>', 'vision_eos_token': '<|vision_end|>'}, clean_up_tokenization_spaces=False, added_tokens_decoder={
248044: AddedToken("<|endoftext|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
248045: AddedToken("<|im_start|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
248046: AddedToken("<|im_end|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
248047: AddedToken("<|object_ref_start|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
248048: AddedToken("<|object_ref_end|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
248049: AddedToken("<|box_start|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
248050: AddedToken("<|box_end|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
248051: AddedToken("<|quad_start|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
248052: AddedToken("<|quad_end|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
248053: AddedToken("<|vision_start|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
248054: AddedToken("<|vision_end|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
248055: AddedToken("<|vision_pad|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
248056: AddedToken("<|image_pad|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
248057: AddedToken("<|video_pad|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
248058: AddedToken("<tool_call>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
248059: AddedToken("</tool_call>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
248060: AddedToken("<|fim_prefix|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
248061: AddedToken("<|fim_middle|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
248062: AddedToken("<|fim_suffix|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
248063: AddedToken("<|fim_pad|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
248064: AddedToken("<|repo_name|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
248065: AddedToken("<|file_sep|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
248066: AddedToken("<tool_response>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
248067: AddedToken("</tool_response>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
248068: AddedToken("", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
248069: AddedToken("", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
248070: AddedToken("<|audio_start|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
248071: AddedToken("<|audio_end|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
248072: AddedToken("<tts_pad>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
248073: AddedToken("<tts_text_bos>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
248074: AddedToken("<tts_text_eod>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
248075: AddedToken("<tts_text_bos_single>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
248076: AddedToken("<|audio_pad|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
}
)
video_processor: Qwen3VLVideoProcessor {
"crop_size": null,
"data_format": "channels_first",
"default_to_square": true,
"device": null,
"do_center_crop": null,
"do_convert_rgb": true,
"do_normalize": true,
"do_rescale": true,
"do_resize": true,
"do_sample_frames": true,
"fps": 2,
"image_mean": [
0.5,
0.5,
0.5
],
"image_std": [
0.5,
0.5,
0.5
],
"input_data_format": null,
"max_frames": 768,
"merge_size": 2,
"min_frames": 4,
"num_frames": null,
"pad_size": null,
"patch_size": 16,
"processor_class": "Qwen3VLProcessor",
"resample": 3,
"rescale_factor": 0.00392156862745098,
"return_metadata": false,
"size": {
"longest_edge": 25165824,
"shortest_edge": 4096
},
"temporal_patch_size": 2,
"video_metadata": null,
"video_processor_type": "Qwen3VLVideoProcessor"
}
{
"processor_class": "Qwen3VLProcessor"
}
Offline mode: forcing local_files_only=True
loading configuration file /root/.xinference/cache/v2/qwen3_5-pytorch-27b-none/preprocessor_config.json
Image processor Qwen2VLImageProcessorFast {
"crop_size": null,
"data_format": "channels_first",
"default_to_square": true,
"device": null,
"disable_grouping": null,
"do_center_crop": null,
"do_convert_rgb": true,
"do_normalize": true,
"do_pad": null,
"do_rescale": true,
"do_resize": true,
"image_mean": [
0.5,
0.5,
0.5
],
"image_processor_type": "Qwen2VLImageProcessorFast",
"image_std": [
0.5,
0.5,
0.5
],
"input_data_format": null,
"max_pixels": null,
"merge_size": 2,
"min_pixels": null,
"pad_size": null,
"patch_size": 16,
"processor_class": "Qwen3VLProcessor",
"resample": 3,
"rescale_factor": 0.00392156862745098,
"return_tensors": null,
"size": {
"longest_edge": 16777216,
"shortest_edge": 65536
},
"temporal_patch_size": 2
}
Offline mode: forcing local_files_only=True
loading file vocab.json
loading file merges.txt
loading file tokenizer.json
loading file added_tokens.json
loading file special_tokens_map.json
loading file tokenizer_config.json
loading file chat_template.jinja
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Offline mode: forcing local_files_only=True
loading configuration file /root/.xinference/cache/v2/qwen3_5-pytorch-27b-none/video_preprocessor_config.json
Video processor Qwen3VLVideoProcessor {
"crop_size": null,
"data_format": "channels_first",
"default_to_square": true,
"device": null,
"do_center_crop": null,
"do_convert_rgb": true,
"do_normalize": true,
"do_rescale": true,
"do_resize": true,
"do_sample_frames": true,
"fps": 2,
"image_mean": [
0.5,
0.5,
0.5
],
"image_std": [
0.5,
0.5,
0.5
],
"input_data_format": null,
"max_frames": 768,
"merge_size": 2,
"min_frames": 4,
"num_frames": null,
"pad_size": null,
"patch_size": 16,
"processor_class": "Qwen3VLProcessor",
"resample": 3,
"rescale_factor": 0.00392156862745098,
"return_metadata": false,
"size": {
"longest_edge": 25165824,
"shortest_edge": 4096
},
"temporal_patch_size": 2,
"video_metadata": null,
"video_processor_type": "Qwen3VLVideoProcessor"
}
Offline mode: forcing local_files_only=True
loading configuration file None
Processor Qwen3VLProcessor:
image_processor: Qwen2VLImageProcessorFast {
"crop_size": null,
"data_format": "channels_first",
"default_to_square": true,
"device": null,
"disable_grouping": null,
"do_center_crop": null,
"do_convert_rgb": true,
"do_normalize": true,
"do_pad": null,
"do_rescale": true,
"do_resize": true,
"image_mean": [
0.5,
0.5,
0.5
],
"image_processor_type": "Qwen2VLImageProcessorFast",
"image_std": [
0.5,
0.5,
0.5
],
"input_data_format": null,
"max_pixels": null,
"merge_size": 2,
"min_pixels": null,
"pad_size": null,
"patch_size": 16,
"processor_class": "Qwen3VLProcessor",
"resample": 3,
"rescale_factor": 0.00392156862745098,
"return_tensors": null,
"size": {
"longest_edge": 16777216,
"shortest_edge": 65536
},
"temporal_patch_size": 2
}
tokenizer: CachedQwen2TokenizerFast(name_or_path='/root/.xinference/cache/v2/qwen3_5-pytorch-27b-none', vocab_size=248044, model_max_length=2
62144, is_fast=True, padding_side='right', truncation_side='left', special_tokens={'eos_token': '<|im_end|>', 'pad_token': '<|endoftext|>', 'ad
ditional_special_tokens': ['<|im_start|>', '<|im_end|>', '<|object_ref_start|>', '<|object_ref_end|>', '<|box_start|>', '<|box_end|>', '<|quad_
start|>', '<|quad_end|>', '<|vision_start|>', '<|vision_end|>', '<|vision_pad|>', '<|image_pad|>', '<|video_pad|>'], 'audio_bos_token': '<|audi
o_start|>', 'audio_eos_token': '<|audio_end|>', 'audio_token': '<|audio_pad|>', 'image_token': '<|image_pad|>', 'video_token': '<|video_pad|>',
'vision_bos_token': '<|vision_start|>', 'vision_eos_token': '<|vision_end|>'}, clean_up_tokenization_spaces=False, added_tokens_decoder={
248044: AddedToken("<|endoftext|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
248045: AddedToken("<|im_start|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
248046: AddedToken("<|im_end|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
248047: AddedToken("<|object_ref_start|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
248048: AddedToken("<|object_ref_end|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
248049: AddedToken("<|box_start|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
248050: AddedToken("<|box_end|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
248051: AddedToken("<|quad_start|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
248052: AddedToken("<|quad_end|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
248053: AddedToken("<|vision_start|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
248054: AddedToken("<|vision_end|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
248055: AddedToken("<|vision_pad|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
248056: AddedToken("<|image_pad|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
248057: AddedToken("<|video_pad|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
248058: AddedToken("<tool_call>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
248059: AddedToken("</tool_call>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
248060: AddedToken("<|fim_prefix|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
248061: AddedToken("<|fim_middle|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
248062: AddedToken("<|fim_suffix|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
248063: AddedToken("<|fim_pad|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
248064: AddedToken("<|repo_name|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
248065: AddedToken("<|file_sep|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
248066: AddedToken("<tool_response>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
248067: AddedToken("</tool_response>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
248068: AddedToken("", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
248069: AddedToken("", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
248070: AddedToken("<|audio_start|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
248071: AddedToken("<|audio_end|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
248072: AddedToken("<tts_pad>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
248073: AddedToken("<tts_text_bos>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
248074: AddedToken("<tts_text_eod>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
248075: AddedToken("<tts_text_bos_single>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
248076: AddedToken("<|audio_pad|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
}
)
video_processor: Qwen3VLVideoProcessor {
"crop_size": null,
"data_format": "channels_first",
"default_to_square": true,
"device": null,
"do_center_crop": null,
"do_convert_rgb": true,
"do_normalize": true,
"do_rescale": true,
"do_resize": true,
"do_sample_frames": true,
"fps": 2,
"image_mean": [
0.5,
0.5,
0.5
],
"image_std": [
0.5,
0.5,
0.5
],
"input_data_format": null,
"max_frames": 768,
"merge_size": 2,
"min_frames": 4,
"num_frames": null,
"pad_size": null,
"patch_size": 16,
"processor_class": "Qwen3VLProcessor",
"resample": 3,
"rescale_factor": 0.00392156862745098,
"return_metadata": false,
"size": {
"longest_edge": 25165824,
"shortest_edge": 4096
},
"temporal_patch_size": 2,
"video_metadata": null,
"video_processor_type": "Qwen3VLVideoProcessor"
}
{
"processor_class": "Qwen3VLProcessor"
}
WARNING 08-14 01:40:50 [system_utils.py:157] We must use the
spawnmultiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: CUDA is initialized
/usr/local/lib/python3.12/dist-packages/torch/cuda/init.py:61: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py
instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
import pynvml # type: ignore[import]
WARNING 08-14 01:40:55 [config.py:70] Support for Transformers v4 is deprecated. The Transformers v4 codepath will become unmaintained in vLLM
v0.22.0 and will be removed in vLLM v0.24.0. Please upgrade to Transformers v5: pip install --upgrade transformers
(EngineCore pid=923) INFO 08-14 01:41:00 [core.py:109] Initializing a V1 LLM engine (v0.21.0) with config: model='/root/.xinference/cache/v2/qw
en3_5-pytorch-27b-none', speculative_config=None, tokenizer='/root/.xinference/cache/v2/qwen3_5-pytorch-27b-none', skip_tokenizer_init=False, t
okenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=200000, download_dir=Non
e, load_format=auto, tensor_parallel_size=2, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=a
g_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=False, enable_return_routed_experts=False, kv
cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable
additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityCon
fig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sam
ple=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_loggi
ng_iteration_details=False), seed=0, served_model_name=/root/.xinference/cache/v2/qwen3_5-pytorch-27b-none, enable_prefix_caching=False, enable
chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir':
'', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vll
m::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'v
llm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_for
ward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::unifie
d_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token
budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_
ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': Fals
e, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FUL
L_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64], 'cudagraph_copy_i
nputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_qu
ant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, '
max_cudagraph_capture_size': 64, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_b
it_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_pri
ority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=False, moe_backend='auto')
WARNING 08-14 01:41:04 [config.py:70] Support for Transformers v4 is deprecated. The Transformers v4 codepath will become unmaintained in vLLM
v0.22.0 and will be removed in vLLM v0.24.0. Please upgrade to Transformers v5: pip install --upgrade transformers
WARNING 08-14 01:41:05 [config.py:70] Support for Transformers v4 is deprecated. The Transformers v4 codepath will become unmaintained in vLLM
v0.22.0 and will be removed in vLLM v0.24.0. Please upgrade to Transformers v5: pip install --upgrade transformers
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] EngineCore failed to start.
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] Traceback (most recent call last):
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] File "/root/.xinference/virtualenv/v4/qwen3.5/vllm/3.12.13/lib/python3.12/site-packa
ges/vllm/v1/engine/core.py", line 1114, in run_engine_core
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] File "/root/.xinference/virtualenv/v4/qwen3.5/vllm/3.12.13/lib/python3.12/site-packa
ges/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] return func(*args, **kwargs)
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] File "/root/.xinference/virtualenv/v4/qwen3.5/vllm/3.12.13/lib/python3.12/site-packa
ges/vllm/v1/engine/core.py", line 880, in init
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] super().init(
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] File "/root/.xinference/virtualenv/v4/qwen3.5/vllm/3.12.13/lib/python3.12/site-packa
ges/vllm/v1/engine/core.py", line 118, in init
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] self.model_executor = executor_class(vllm_config)
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] File "/opt/inference/xinference/model/llm/vllm/distributed_executor_v1.py", line 154
, in init
(EngineCore pid=923) Process EngineCore:
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] Executor.init(self, vllm_config, *args, **kwargs)
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] File "/root/.xinference/virtualenv/v4/qwen3.5/vllm/3.12.13/lib/python3.12/site-packa
ges/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] return func(*args, **kwargs)
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] File "/root/.xinference/virtualenv/v4/qwen3.5/vllm/3.12.13/lib/python3.12/site-packa
ges/vllm/v1/executor/abstract.py", line 109, in init
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] self._init_executor()
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] File "/opt/inference/xinference/model/llm/vllm/distributed_executor_v1.py", line 175
, in _init_executor
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] refs: List[xo.ActorRefType[WorkerActor]] = [fut.result() for fut in futures]
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] ^^^^^^^^^^^^
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] File "/usr/lib/python3.12/concurrent/futures/_base.py", line 456, in result
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] return self.__get_result()
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] ^^^^^^^^^^^^^^^^^^^
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] File "/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] raise self._exception
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] File "/usr/local/lib/python3.12/dist-packages/xoscar/api.py", line 79, in create_act
or
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] return await ctx.create_actor(actor_cls, *args, uid=uid, address=address, **kwargs
)
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
^
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] File "/usr/local/lib/python3.12/dist-packages/xoscar/backends/context.py", line 152,
in create_actor
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] return self._process_result_message(result)
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] File "/usr/local/lib/python3.12/dist-packages/xoscar/backends/context.py", line 111,
in _process_result_message
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] raise message.as_instanceof_cause()
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] File "/usr/local/lib/python3.12/dist-packages/xoscar/backends/pool.py", line 441, in
process_message
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] processor.result = await self._run_coro(
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] ^^^^^^^^^^^^^^^^^
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] File "/usr/local/lib/python3.12/dist-packages/xoscar/backends/pool.py", line 389, in
_run_coro
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] return await coro
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] File "/usr/local/lib/python3.12/dist-packages/xoscar/backends/pool.py", line 923, in
create_actor
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] await self.notify_main_pool_to_create(message)
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] ^^^^^^^^^^^^^^^^^
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] File "/usr/local/lib/python3.12/dist-packages/xoscar/backends/pool.py", line 917, in
notify_main_pool_to_create
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] await self.call(self._main_address, reg_message)
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] ^^^^^^^^^^^^^^^^^
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] File "/usr/local/lib/python3.12/dist-packages/xoscar/backends/pool.py", line 448, in
call
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] return await self._caller.call(self._router, dest_address, message) # type: ignor
e
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] ^^^^^^^^^^^^^^^^^
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] File "/usr/local/lib/python3.12/dist-packages/xoscar/backends/core.py", line 222, in
call
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] return await self.call_with_client(client, message, wait)
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] ^^^^^^^^^^^^^^^^^
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] File "/usr/local/lib/python3.12/dist-packages/xoscar/backends/core.py", line 167, in
call_with_client
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] r = await wait_response
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] ^^^^^^^^^^^^^^^^^
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] File "/usr/local/lib/python3.12/dist-packages/xoscar/backends/core.py", line 104, in
_listen
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] raise ServerClosed(
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] ^^^^^^^^^^^^^^^^^
(EngineCore pid=923) ERROR 08-14 01:41:09 [core.py:1140] xoscar.errors.ServerClosed: [address=0.0.0.0:44277, pid=772] Remote server unixsocket:
///8060928 closed: 0 bytes read on a total of 11 expected bytes
(EngineCore pid=923) Traceback (most recent call last):
(EngineCore pid=923) File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
(EngineCore pid=923) self.run()
(EngineCore pid=923) File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
(EngineCore pid=923) self._target(*self._args, **self._kwargs)
(EngineCore pid=923) File "/root/.xinference/virtualenv/v4/qwen3.5/vllm/3.12.13/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 11
44, in run_engine_core
(EngineCore pid=923) raise e
(EngineCore pid=923) File "/root/.xinference/virtualenv/v4/qwen3.5/vllm/3.12.13/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 11
14, in run_engine_core
(EngineCore pid=923) engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(EngineCore pid=923) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=923) File "/root/.xinference/virtualenv/v4/qwen3.5/vllm/3.12.13/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178,
in sync_wrapper
(EngineCore pid=923) return func(*args, **kwargs)
(EngineCore pid=923) ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=923) File "/root/.xinference/virtualenv/v4/qwen3.5/vllm/3.12.13/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 88
0, in init
(EngineCore pid=923) super().init(
(EngineCore pid=923) File "/root/.xinference/virtualenv/v4/qwen3.5/vllm/3.12.13/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 11
8, in init
(EngineCore pid=923) self.model_executor = executor_class(vllm_config)
(EngineCore pid=923) ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=923) File "/opt/inference/xinference/model/llm/vllm/distributed_executor_v1.py", line 154, in init
(EngineCore pid=923) Executor.init(self, vllm_config, *args, **kwargs)
(EngineCore pid=923) File "/root/.xinference/virtualenv/v4/qwen3.5/vllm/3.12.13/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178,
in sync_wrapper
(EngineCore pid=923) return func(*args, **kwargs)
(EngineCore pid=923) ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=923) File "/root/.xinference/virtualenv/v4/qwen3.5/vllm/3.12.13/lib/python3.12/site-packages/vllm/v1/executor/abstract.py", l
ine 109, in init
(EngineCore pid=923) self._init_executor()
(EngineCore pid=923) File "/opt/inference/xinference/model/llm/vllm/distributed_executor_v1.py", line 175, in _init_executor
(EngineCore pid=923) refs: List[xo.ActorRefType[WorkerActor]] = [fut.result() for fut in futures]
(EngineCore pid=923) ^^^^^^^^^^^^
(EngineCore pid=923) File "/usr/lib/python3.12/concurrent/futures/_base.py", line 456, in result
(EngineCore pid=923) return self.__get_result()
(EngineCore pid=923) ^^^^^^^^^^^^^^^^^^^
(EngineCore pid=923) File "/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
(EngineCore pid=923) raise self._exception
(EngineCore pid=923) File "/usr/local/lib/python3.12/dist-packages/xoscar/api.py", line 79, in create_actor
(EngineCore pid=923) return await ctx.create_actor(actor_cls, *args, uid=uid, address=address, **kwargs)
(EngineCore pid=923) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=923) File "/usr/local/lib/python3.12/dist-packages/xoscar/backends/context.py", line 152, in create_actor
(EngineCore pid=923) return self._process_result_message(result)
(EngineCore pid=923) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=923) File "/usr/local/lib/python3.12/dist-packages/xoscar/backends/context.py", line 111, in _process_result_message
(EngineCore pid=923) raise message.as_instanceof_cause()
(EngineCore pid=923) File "/usr/local/lib/python3.12/dist-packages/xoscar/backends/pool.py", line 441, in process_message
(EngineCore pid=923) processor.result = await self._run_coro(
(EngineCore pid=923) ^^^^^^^^^^^^^^^^^
(EngineCore pid=923) File "/usr/local/lib/python3.12/dist-packages/xoscar/backends/pool.py", line 389, in _run_coro
(EngineCore pid=923) return await coro
(EngineCore pid=923) File "/usr/local/lib/python3.12/dist-packages/xoscar/backends/pool.py", line 923, in create_actor
(EngineCore pid=923) await self.notify_main_pool_to_create(message)
(EngineCore pid=923) ^^^^^^^^^^^^^^^^^
(EngineCore pid=923) File "/usr/local/lib/python3.12/dist-packages/xoscar/backends/pool.py", line 917, in notify_main_pool_to_create
(EngineCore pid=923) await self.call(self._main_address, reg_message)
(EngineCore pid=923) ^^^^^^^^^^^^^^^^^
(EngineCore pid=923) File "/usr/local/lib/python3.12/dist-packages/xoscar/backends/pool.py", line 448, in call
(EngineCore pid=923) return await self._caller.call(self._router, dest_address, message) # type: ignore
(EngineCore pid=923) ^^^^^^^^^^^^^^^^^
(EngineCore pid=923) File "/usr/local/lib/python3.12/dist-packages/xoscar/backends/core.py", line 222, in call
(EngineCore pid=923) return await self.call_with_client(client, message, wait)
(EngineCore pid=923) ^^^^^^^^^^^^^^^^^
(EngineCore pid=923) File "/usr/local/lib/python3.12/dist-packages/xoscar/backends/core.py", line 167, in call_with_client
(EngineCore pid=923) r = await wait_response
(EngineCore pid=923) ^^^^^^^^^^^^^^^^^
(EngineCore pid=923) File "/usr/local/lib/python3.12/dist-packages/xoscar/backends/core.py", line 104, in _listen
(EngineCore pid=923) raise ServerClosed(
(EngineCore pid=923) ^^^^^^^^^^^^^^^^^
(EngineCore pid=923) xoscar.errors.ServerClosed: [address=0.0.0.0:44277, pid=772] Remote server unixsocket:///8060928 closed: 0 bytes read on a
total of 11 expected bytes
’‘’
Expected behavior / 期待表现
能不能不要明确支持的模型的默认虚拟环境都是存在问题的，如果不能确保支持，至少在说明描述中提及明确支持的运行环境要求以及未测试的边缘场景可能性以方便排查错误
此外，现在如果不设置XINFERENCE_LOG_CONSOLE=false，dockerlogs信息中会在启动模型时有大量来自web查询进度的get请求刷屏，完全屏蔽其他信息，关闭后又会有其他信息不全，这些明显无价值的信息最好能默认屏蔽