EchoMimicV2: Towards Striking, Simplified, and Semi-Body Human Animation Conditioning

Rang Meng Xingyu Zhang Yuming Li Chenguang Ma

Terminal Technology Department, Alipay, Ant Group.

🚀 EchoMimic Series

EchoMimicV1: Lifelike Audio-Driven Portrait Animations through Editable Landmark Conditioning. GitHub
EchoMimicV2: Towards Striking, Simplified, and Semi-Body Human Animation. GitHub

📣 Updates

[2024.11.19] 🔥 We release the EMTD dataset list and processing scripts.
[2024.11.19] 🔥 We release our EchoMimicV2 codes and models.
[2024.11.15] 🔥 Our paper is in public on arxiv.

🌅 Gallery

Introduction

English Driven Audio

Chinese Driven Audio

⚒️ Installation

Download the Codes

  git clone https://github.com/antgroup/echomimic_v2
  cd echomimic_v2

Python Environment Setup

Tested System Environment: Centos 7.2/Ubuntu 22.04, Cuda >= 11.7
Tested GPUs: A100(80G) / RTX4090D (24G) / V100(16G)
Tested Python Version: 3.8 / 3.10 / 3.11

Create conda environment (Recommended):

  conda create -n echomimic python=3.8
  conda activate echomimic

Install packages with pip

  pip install -r requirements.txt

Download ffmpeg-static

Download and decompress ffmpeg-static, then

export FFMPEG_PATH=/path/to/ffmpeg-4.4-amd64-static

Download pretrained weights

git lfs install
git clone https://huggingface.co/BadToBest/EchoMimic pretrained_weights

The pretrained_weights is organized as follows.

./pretrained_weights/
├── denoising_unet.pth
├── reference_unet.pth
├── motion_module.pth
├── face_locator.pth
├── sd-vae-ft-mse
│   └── ...
├── sd-image-variations-diffusers
│   └── ...
└── audio_processor
    └── whisper_tiny.pt

In which denoising_unet.pth / reference_unet.pth / motion_module.pth / face_locator.pth are the main checkpoints of EchoMimic. Other models in this hub can be also downloaded from it's original hub, thanks to their brilliant works:

Inference on Demo

Run the python inference script:

python infer.py --config='./configs/prompts/infer.yaml'

Inference on Your Own Case

xxxx.ipynb is a complete demo to generate animation video using the custom reference image, audio, and hand pose driven video.

EMTD Dataset

Download dataset:

python ./EMTD_dataset/download.py

Slice dataset:

bash ./EMTD_dataset/slice.sh

Process dataset:

python ./EMTD_dataset/preprocess.py

📝 Release Plans

Status	Milestone	ETA
✅	The inference source code of EchoMimicV2 meet everyone on GitHub	21st Nov, 2024
✅	Pretrained models trained on English and Mandarin Chinese on HuggingFace	21st Nov, 2024
✅	Pretrained models trained on English and Mandarin Chinese on ModelScope	21st Nov, 2024
✅	EMTD dataset list and processing scripts	21st Nov, 2024
🚀	Accelerated models to be released	TBD
🚀	Online Demo on ModelScope to be released	TBD
🚀	Online Demo on HuggingFace to be released	TBD

⚖️ Disclaimer

This project is intended for academic research, and we explicitly disclaim any responsibility for user-generated content. Users are solely liable for their actions while using the generative model. The project contributors have no legal affiliation with, nor accountability for, users' behaviors. It is imperative to use the generative model responsibly, adhering to both ethical and legal standards.

🙏🏻 Acknowledgements

We would like to thank the contributors to the MimicMotion and Moore-AnimateAnyone repositories, for their open research and exploration.

We are also grateful to CyberHost and Vlogger for their outstanding work in the area of audio-driven human animation.

If we missed any open-source projects or related articles, we would like to complement the acknowledgement of this specific work immediately.

📒 Citation

If you find our work useful for your research, please consider citing the paper :

@misc{meng2024echomimic,
  title={EchoMimicV2: Towards Striking, Simplified, and Semi-Body Human Animation},
  author={Rang Meng, Xingyu Zhang, Yuming Li, Chenguang Ma},
  year={2024},
  eprint={2411.10061},
  archivePrefix={arXiv},
  primaryClass={cs.CV}
}

BadToBest
/

EchoMimicV2