VideoChat3

A Fully Open Video MLLM for Efficient, Generalist Video Understanding

Paper     Homepage     Models & Data

One model for fine-grained motion, long-video reasoning, temporal grounding, and online proactive response.

✨ Overview

VideoChat3 is a 4B generalist video MLLM built to understand video across time—from subtle motion and hour-long stories to precise temporal evidence and live streams.

It combines I3D-ViT for 16× spatiotemporal compression with Adaptive Frame Resolution for evidence-aware streaming, trained on Academic2M, LV116K, and OL617K.

🔥 Updates

• 2026/07/27: 🔥🔥🔥Release the full-stage training data and training checkpoints for each stage of VideoChat3.
• 2026/07/27: 🔥🔥🔥Release the training code for VideoChat3.
• 2026/07/24: 🔥🔥🔥Release the evaluation code for VideoChat3.
• 2026/07/17: 🔥🔥🔥Release the VideoChat3 model weights and training data.

🚀 Highlights

• 🎬 Generalist video understanding: one model for motion, long video, temporal grounding, and online proactive response.
• ⚡ Token-efficient architecture: I3D-ViT compresses redundant visual tokens while preserving spatiotemporal evidence.
• 🔍 Adaptive streaming perception: frame resolution is increased only when closer visual inspection is needed.
• 🔓 Open resources: model weights and the complete training datasets are publicly available.

🛠️ Training

The training implementation is available in xtuner-videochat3. It is built on XTuner V1 and includes the VideoChat3 model and data pipeline, staged training configurations, initial checkpoint construction, and Slurm/non-Slurm distributed launchers.

Install the training project:

cd xtuner-videochat3
pip install -e ".[video]"

After preparing the model and dataset paths in the selected configuration, start a launcher from the same directory. For example, a single-node stage1-1 run is:

NNODES=1 NPROC_PER_NODE=8 \
  bash training_scripts/stage1/VideoChat3_4B_train_stage1-1.sh

See the complete training guide for initial checkpoint construction, all stage configurations, Slurm launch commands, and manual multi-node setup.

🏆 Evaluation

The evaluation implementation is available in vlmevalkit-videochat3. It is built on VLMEvalKit and supports all benchmarks for both offline and online video understanding evaluated in the paper.

Please refer to the evaluation guide for detailed instructions on environment setup, model downloading, and evaluation.

🌟 Quick Start

Please follow the instructions in VideoChat3-4B to set up the environment and download the model weights.

pip install torch transformers accelerate qwen-vl-utils
pip install decord opencv-python-headless
# optional
pip install flash-attn --no-build-isolation

We provide two inference implementations here—one for standard model inference and the other for online proactive-response inference—to help you get started quickly.

Citation

@misc{videochat3,
      title={VideoChat3: Fully Open Video MLLM for Efficient and Generalist Video Understanding}, 
      author={Xinhao Li and Yuhan Zhu and Xiangyu Zeng and Yuhao Dong and Haoning Wu and Zhiqiu Zhang and Yuandong Yang and Changlian Ma and Qingyu Zhang and Yansong Shi and Xinyu Chen and Haoran Chen and Zizheng Huang and Jun Zhang and Kun Ouyang and Lin Sui and Ziang Yan and Yicheng Xu and Chenting Wang and Yinan He and Hongjie Zhang and Yi Wang and Yu Qiao and Yali Wang and Ziwei Liu and Kai Chen and Limin Wang},
      year={2026},
      eprint={2607.14935},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2607.14935}, 
}