metadata

license: apache-2.0
language:
  - en

MotionCtrl Model Card

Project Page | Paper (ArXiv) | Code

🤗 Gradio demo (MotionCtrl+VideoCrafter) | 🤗 Gradio demo (MotionCtrl+SVD)

Introduction

MotionCtrl, when provided with specific camera movements indicated by camera poses or several trajectories, can independently control both the camera and object motion in a generated video. It can be deployed on both Text-to-Video generation models, such as LVDM/VideoCrafter and AnimateDiff, as well as Image-to-Video generation models like SVD.

MotionCtrl + SVD

Model: motionctrl.pth
Results:

MotionCtrl + LVDM/VideoCrafter

Model:
- motionctrl_svd.ckpt
- motionctrl_videocrafter2_cmcm.ckpt
Results:

MotionCtrl + AnimateDiff

Model: motionctrl_animatediff_cmcm.ckpt
Results:

Usage

Download directly in this repository OR Download with python script

from huggingface_hub import hf_hub_download
motionctrl_lvdm_ckpt = hf_hub_download(repo_id="TencentARC/MotionCtrl", filename="motionctrl.ckpt", repo_type="model")
motionctrl_svd_ckpt = hf_hub_download(repo_id="TencentARC/MotionCtrl", filename="motionctrl_svd.ckpt", repo_type="model")

Then generate the controllable video following the instructions in our GitHub repository.

Citation

@inproceedings{wang2023motionctrl,
  title={MotionCtrl: A Unified and Flexible Motion Controller for Video Generation},
  author={Wang, Zhouxia and Yuan, Ziyang and Wang, Xintao and Chen, Tianshui and Xia, Menghan and Luo, Ping and Shan, Yin},
  booktitle={arXiv preprint arXiv:2312.03641},
  year={2023}
}