metadata

title: 'VideoChat: Chat-Centric Video Understanding'
emoji: 👀
colorFrom: green
colorTo: blue
sdk: gradio
python_version: 3.8.16
app_file: app.py
pinned: false
license: mit

🦜 VideoChat [paper]

In this study, we initiate an exploration into video understanding by introducing VideoChat, an end-to-end chat-centric video understanding system. It integrates video foundation models and large language models via a learnable neural interface, excelling in spatiotemporal reasoning, event localization, and causal relationship inference. To instructively tune this system, we propose a video-centric instruction dataset, composed of thousands of videos matched with detailed descriptions and conversations. This dataset emphasizes spatiotemporal reasoning and causal relationships, providing a valuable asset for training chat-centric video understanding systems. Preliminary qualitative experiments reveal our system’s potential across a broad spectrum of video applications and set the standard for future research.

:fire: Updates

2023/05/11: Release the 🦜VideoChat V1, which can handle both image and video understanding!
- Model and Data.
- 🧑‍💻 Online demo is Preparing.
- 🧑‍🔧 Tuning scripts are cleaning.

:hourglass_flowing_sand: Schedule

Small-scale video instuction data and tuning
Instruction tuning on BLIP+UniFormerV2+Vicuna
Large-scale and complex video instuction data
Instruction tuning on strong video foundation model
User-friendly interactions with longer videos
...

:speech_balloon: Example

Comparison with ChatGPT, MiniGPT-4, LLaVA and mPLUG-Owl.
Our VideoChat can handle both image and video understanding well!

[Video] Why the video is funny?

[Video] Spatial perception

[Video] Temporal perception

[Video] Multi-turn conversation

Image understanding

:running: Usage

Prepare the envirment.
```
pip install -r requirements.txt
```
Download BLIP2 model:
- ViT: wget https://storage.googleapis.com/sfr-vision-language-research/LAVIS/models/BLIP2/eva_vit_g.pth
- QFormer: wget https://storage.googleapis.com/sfr-vision-language-research/LAVIS/models/BLIP2/blip2_pretrained_flant5xxl.pth
- Change the vit_model_path and q_former_model_path in config.json.

Download StabelVicuna model:

LLAMA: Download it from the original repo or hugging face.
If you download LLAMA from the original repo, please process it via the following command:

# convert_llama_weights_to_hf is copied from transformers
python src/transformers/models/llama/convert_llama_weights_to_hf.py \
  --input_dir /path/to/downloaded/llama/weights \
  --model_size 7B --output_dir /output/path

Download StableVicuna-13b-deelta and process it:

# fastchat v0.1.10
python3 apply_delta.py \
  --base /path/to/model_weights/llama-13b \
  --target stable-vicuna-13b \
  --delta CarperAI/stable-vicuna-13b-delta

Change the llama_model_path in config.json.

Download VideoChat model:
- Change the videochat_model_path in config.json.
Running demo with Gradio:
```
python demo.py
```
Another demo on Jupyter Notebook can found in demo.ipynb

:page_facing_up: Citation

If you find this project useful in your research, please consider cite:

@article{2023videochat,
  title={VideoChat: Chat-Centric Video Understanding},
  author={KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao},
  journal={arXiv preprint arXiv:2305.06355},
  year={2023}
}

:thumbsup: Acknowledgement

Thanks to the open source of the following projects:

InternVideo, UniFormerV2, MiniGPT-4, LLaVA, BLIP2, StableLM.