Update README.md

README.md

@@ -1,3 +1,129 @@
----
-license: mit
----
+---
+license: mit
+language:
+- en
+---
+
+<p align="center">
+
+  <h2 align="center">X-Dyna: Expressive Dynamic Human Image Animation</h2>
+  <p align="center">
+      <a href="https://boese0601.github.io/">Di Chang</a><sup>1,2</sup>
+      ·
+      <a href="https://hongyixu37.github.io/homepage/">Hongyi Xu</a><sup>2*</sup>
+    ·  
+      <a href="https://youxie.github.io/">You Xie</a><sup>2*</sup>
+    ·  
+      <a href="https://hlings.github.io/">Yipeng Gao</a><sup>1*</sup>
+    ·  
+      <a href="https://zhengfeikuang.com/">Zhengfei Kuang</a><sup>3*</sup>
+    ·  
+      <a href="https://primecai.github.io/">Shengqu Cai</a><sup>3*</sup>
+    ·  
+      <a href="https://zhangchenxu528.github.io/">Chenxu Zhang</a><sup>2*</sup>
+    <br>
+      <a href="https://guoxiansong.github.io/homepage/index.html">Guoxian Song</a><sup>2</sup>
+    ·  
+      <a href="https://chaowang.info/">Chao Wang</a><sup>2</sup>
+    ·  
+      <a href="https://seasonsh.github.io/">Yichun Shi</a><sup>2</sup>
+    ·  
+      <a href="https://zeyuan-chen.com/">Zeyuan Chen</a><sup>2,5</sup>
+    ·  
+      <a href="https://shijiezhou-ucla.github.io/">Shijie Zhou</a><sup>4</sup>
+    ·  
+      <a href="https://scholar.google.com/citations?user=fqubyX0AAAAJ&hl=en">Linjie Luo</a><sup>2</sup>
+    <br>
+      <a href="https://web.stanford.edu/~gordonwz/">Gordon Wetzstein</a><sup>3</sup>
+    ·  
+      <a href="https://www.ihp-lab.org/">Mohammad Soleymani</a><sup>1</sup>
+    <br>
+    <sup>1</sup>Unviersity of Southern California &nbsp;<sup>2</sup>ByteDance Inc. &nbsp; <sup>3</sup>Stanford University &nbsp; 
+    <br>
+    <sup>4</sup>University of California Los Angeles&nbsp; <sup>5</sup>University of California San Diego
+    <br>
+    <br>
+    <sup>*</sup> denotes equally contribution
+</p>
+
+
+-----
+
+This huggingface repo contains the pretrained models of X-Dyna.
+
+
+## **Abstract**
+We introduce X-Dyna, a novel zero-shot, diffusion-based pipeline for animating a single human image using facial expressions and body movements derived from a driving video, that generates realistic, context-aware dynamics for both the subject and the surrounding environment. Building on prior approaches centered on human pose control, X-Dyna addresses key factors underlying the loss of dynamic details, enhancing the lifelike qualities of human video animations. At the core of our approach is the Dynamics-Adapter, a lightweight module that effectively integrates reference appearance context into the spatial attentions of the diffusion backbone while preserving the capacity of motion modules in synthesizing fluid and intricate dynamic details. Beyond body pose control, we connect a local control module with our model to capture identity-disentangled facial expressions, facilitating accurate expression transfer for enhanced realism in animated scenes. Together, these components form a unified framework capable of learning physical human motion and natural scene dynamics from a diverse blend of human and scene videos. Comprehensive qualitative and quantitative evaluations demonstrate that X-Dyna outperforms state-of-the-art methods, creating highly lifelike and expressive animations.
+
+## **Architecture**
+
+We leverage a pretrained diffusion UNet backbone for controlled human image animation, enabling expressive dynamic details and precise motion control. Specifically, we introduce a dynamics adapter that seamlessly integrates the reference image context as a trainable residual to the spatial attention, in parallel with the denoising process, while preserving the original spatial and temporal attention mechanisms within the UNet. In addition to body pose control via a ControlNet, we introduce a local face control module that implicitly learns facial expression control from a synthesized cross-identity face patch. We train our model on a diverse dataset of human motion videos and natural scene videos simultaneously.
+
+<p align="center">
+  <img src="https://github.com/Boese0601/X-Dyna/blob/main/assets/figures/pipeline.png"  height=400>
+</p>
+
+## **Dynamics Adapter**
+### **Archtecture Designs for Human Video Animation**
+a) IP-Adapter encodes the reference image as an image CLIP embedding and injects the information into the cross-attention layers in SD as the residual. b) ReferenceNet is a trainable parallel UNet and feeds the semantic information into SD via concatenation of self-attention features. c) Dynamics-Adapter encodes the reference image with a partially shared-weight UNet. The appearance control is realized by learning a residual in the self-attention with trainable query and output linear layers. All other components share the same frozen weight with SD.
+
+<p align="center">
+  <img src="https://github.com/Boese0601/X-Dyna/blob/main/assets/figures/Arch_Design.png"  height=250>
+</p>
+
+
+
+
+
+## 📜 Requirements
+* An NVIDIA GPU with CUDA support is required. 
+  * We have tested on a single A100 GPU.
+  * **Minimum**: The minimum GPU memory required is 20GB for generating a single video (batch_size=1) of 16 frames.
+  * **Recommended**: We recommend using a GPU with 80GB of memory.
+* Operating system: Linux
+
+
+## 🧱 Download Pretrained Models
+Due to restrictions we are not able to release the model pretrained with in-house data. Instead, we re-train our model on public datasets, e.g. [TikTok](https://www.yasamin.page/hdnet_tiktok), and [HumanVid](https://github.com/zhenzhiwang/HumanVid), and other human video data for research use, e.g.[Pexels](https://www.pexels.com/). We follow the implementation details in our paper and release pretrained weights and other necessary network modules in this huggingface repository. After downloading, please put them under the pretrained_weights folder. Your file structure should look like this:
+
+```bash
+X-Dyna
+|----...
+|----pretrained_weights
+  |----controlnet
+    |----controlnet-checkpoint-epoch-5.ckpt
+  |----controlnet_face
+    |----controlnet-face-checkpoint-epoch-2.ckpt
+  |----unet 
+    |----unet-checkpoint-epoch-5.ckpt
+  
+  |----initialization
+    |----controlnets_initialization
+      |----controlnet
+        |----control_v11p_sd15_openpose
+      |----controlnet_face
+        |----controlnet2
+    |----unet_initialization
+      |----IP-Adapter
+        |----models
+      |----SD
+        |----stable-diffusion-v1-5
+|----...
+``` 
+
+
+## 🔗 BibTeX
+If you find [X-Dyna](https://arxiv.org) useful for your research and applications, please cite X-Dyna using this BibTeX:
+
+```BibTeX
+@misc{
+}
+```
+
+
+## Acknowledgements
+
+We appreciate the contributions from [AnimateDiff](https://github.com/guoyww/AnimateDiff), [MagicPose](https://github.com/Boese0601/MagicDance), [MimicMotion](https://github.com/tencent/MimicMotion), [Moore-AnimateAnyone](https://github.com/MooreThreads/Moore-AnimateAnyone), [MagicAnimate](https://github.com/magic-research/magic-animate), [IP-Adapter](https://github.com/tencent-ailab/IP-Adapter), [ControlNet](https://arxiv.org/abs/2302.05543), [I2V-Adapter](https://arxiv.org/abs/2312.16693) for their open-sourced research. We appreciate the support from <a href="https://zerg-overmind.github.io/">Quankai Gao</a>, <a href="https://xharlie.github.io/">Qiangeng Xu</a>, <a href="https://ssangx.github.io/">Shen Sang</a>, and <a href="https://tiancheng-zhi.github.io/">Tiancheng Zhi</a> for their suggestions and discussions.
+
+
+