| # AMT: All-Pairs Multi-Field Transforms for Efficient Frame Interpolation | |
| This repository contains the official implementation of the following paper: | |
| > **AMT: All-Pairs Multi-Field Transforms for Efficient Frame Interpolation**<br> | |
| > [Zhen Li](https://paper99.github.io/)<sup>\*</sup>, [Zuo-Liang Zhu](https://nk-cs-zzl.github.io/)<sup>\*</sup>, [Ling-Hao Han](https://scholar.google.com/citations?user=0ooNdgUAAAAJ&hl=en), [Qibin Hou](https://scholar.google.com/citations?hl=en&user=fF8OFV8AAAAJ&view_op=list_works), [Chun-Le Guo](https://scholar.google.com/citations?hl=en&user=RZLYwR0AAAAJ), [Ming-Ming Cheng](https://mmcheng.net/cmm)<br> | |
| > (\* denotes equal contribution) <br> | |
| > Nankai University <br> | |
| > In CVPR 2023<br> | |
| [[Paper](https://arxiv.org/abs/2304.09790)] | |
| [[Project Page](https://nk-cs-zzl.github.io/projects/amt/index.html)] | |
| [[Web demos](#web-demos)] | |
| [Video] | |
| AMT is a **lightweight, fast, and accurate** algorithm for Frame Interpolation. | |
| It aims to provide practical solutions for **video generation** from **a few given frames (at least two frames)**. | |
|  | |
| * More examples can be found in our [project page](https://nk-cs-zzl.github.io/projects/amt/index.html). | |
| ## Web demos | |
| Integrated into [Hugging Face Spaces 🤗](https://huggingface.co/spaces) using [Gradio](https://github.com/gradio-app/gradio). Try out the Web Demo: [](https://huggingface.co/spaces/NKU-AMT/AMT) | |
| Try AMT to interpolate between two or more images at [](https://colab.research.google.com/drive/1IeVO5BmLouhRh6fL2z_y18kgubotoaBq?usp=sharing) | |
| ## Change Log | |
| - **Apr 20, 2023**: Our code is publicly available. | |
| ## Method Overview | |
|  | |
| For technical details, please refer to the [method.md](docs/method.md) file, or read the full report on [arXiv](https://arxiv.org/abs/2304.09790). | |
| ## Dependencies and Installation | |
| 1. Clone Repo | |
| ```bash | |
| git clone https://github.com/MCG-NKU/AMT.git | |
| ``` | |
| 2. Create Conda Environment and Install Dependencies | |
| ```bash | |
| conda env create -f environment.yaml | |
| conda activate amt | |
| ``` | |
| 3. Download pretrained models for demos from [Pretrained Models](#pretrained-models) and place them to the `pretrained` folder | |
| ## Quick Demo | |
| **Note that the selected pretrained model (`[CKPT_PATH]`) needs to match the config file (`[CFG]`).** | |
| > Creating a video demo, increasing $n$ will slow down the motion in the video. (With $m$ input frames, `[N_ITER]` $=n$ corresponds to $2^n\times (m-1)+1$ output frames.) | |
| ```bash | |
| python demos/demo_2x.py -c [CFG] -p [CKPT] -n [N_ITER] -i [INPUT] -o [OUT_PATH] -r [FRAME_RATE] | |
| # e.g. [INPUT] | |
| # -i could be a video / a regular expression / a folder contains multiple images | |
| # -i demo.mp4 (video)/img_*.png (regular expression)/img0.png img1.png (images)/demo_input (folder) | |
| # e.g. a simple usage | |
| python demos/demo_2x.py -c cfgs/AMT-S.yaml -p pretrained/amt-s.pth -n 6 -i assets/quick_demo/img0.png assets/quick_demo/img1.png | |
| ``` | |
| + Note: Please enable `--save_images` for saving the output images (Save speed will be slowed down if there are too many output images) | |
| + Input type supported: `a video` / `a regular expression` / `multiple images` / `a folder containing input frames`. | |
| + Results are in the `[OUT_PATH]` (default is `results/2x`) folder. | |
| ## Pretrained Models | |
| <p id="Pretrained"></p> | |
| <table> | |
| <thead> | |
| <tr> | |
| <th> Dataset </th> | |
| <th> :link: Download Links </th> | |
| <th> Config file </th> | |
| <th> Trained on </th> | |
| <th> Arbitrary/Fixed </th> | |
| </tr> | |
| </thead> | |
| <tbody> | |
| <tr> | |
| <td>AMT-S</td> | |
| <th> [<a href="https://drive.google.com/file/d/1WmOKmQmd6pnLpID8EpUe-TddFpJuavrL/view?usp=share_link">Google Driver</a>][<a href="https://pan.baidu.com/s/1yGaNLeb9TG5-81t0skrOUA?pwd=f66n">Baidu Cloud</a>][<a href="https://huggingface.co/lalala125/AMT/resolve/main/amt-s.pth">Hugging Face</a>] </th> | |
| <th> [<a href="cfgs/AMT-S.yaml">cfgs/AMT-S</a>] </th> | |
| <th>Vimeo90k</th> | |
| <th>Fixed</th> | |
| </tr> | |
| <tr> | |
| <td>AMT-L</td> | |
| <th>[<a href="https://drive.google.com/file/d/1UyhYpAQLXMjFA55rlFZ0kdiSVTL7oU-z/view?usp=share_link">Google Driver</a>][<a href="https://pan.baidu.com/s/1qI4fBgS405Bd4Wn1R3Gbeg?pwd=nbne">Baidu Cloud</a>][<a href="https://huggingface.co/lalala125/AMT/resolve/main/amt-l.pth">Hugging Face</a>]</th> | |
| <th> [<a href="cfgs/AMT-L.yaml">cfgs/AMT-L</a>] </th> | |
| <th>Vimeo90k</th> | |
| <th>Fixed</th> | |
| </tr> | |
| <tr> | |
| <td>AMT-G</td> | |
| <th>[<a href="https://drive.google.com/file/d/1yieLtKh4ei3gOrLN1LhKSP_9157Q-mtP/view?usp=share_link">Google Driver</a>][<a href="https://pan.baidu.com/s/1AjmQVziQut1bXgQnDcDKvA?pwd=caf6">Baidu Cloud</a>][<a href="https://huggingface.co/lalala125/AMT/resolve/main/amt-g.pth">Hugging Face</a>] </th> | |
| <th> [<a href="cfgs/AMT-G.yaml">cfgs/AMT-G</a>] </th> | |
| <th>Vimeo90k</th> | |
| <th>Fixed</th> | |
| </tr> | |
| <tr> | |
| <td>AMT-S</td> | |
| <th>[<a href="https://drive.google.com/file/d/1f1xAF0EDm-rjDdny8_aLyeedfM0QL4-C/view?usp=share_link">Google Driver</a>][<a href="https://pan.baidu.com/s/1eZtoULyduQM8AkXeYEBOEw?pwd=8hy3">Baidu Cloud</a>][<a href="https://huggingface.co/lalala125/AMT/resolve/main/gopro_amt-s.pth">Hugging Face</a>] </th> | |
| <th> [<a href="cfgs/AMT-S_gopro.yaml">cfgs/AMT-S_gopro</a>] </th> | |
| <th>GoPro</th> | |
| <th>Arbitrary</th> | |
| </tr> | |
| </tbody> | |
| </table> | |
| ## Training and Evaluation | |
| Please refer to [develop.md](docs/develop.md) to learn how to benchmark the AMT and how to train a new AMT model from scratch. | |
| ## Citation | |
| If you find our repo useful for your research, please consider citing our paper: | |
| ```bibtex | |
| @inproceedings{licvpr23amt, | |
| title={AMT: All-Pairs Multi-Field Transforms for Efficient Frame Interpolation}, | |
| author={Li, Zhen and Zhu, Zuo-Liang and Han, Ling-Hao and Hou, Qibin and Guo, Chun-Le and Cheng, Ming-Ming}, | |
| booktitle={IEEE Conference on Computer Vision and Pattern Recognition (CVPR)}, | |
| year={2023} | |
| } | |
| ``` | |
| ## License | |
| This code is licensed under the [Creative Commons Attribution-NonCommercial 4.0 International](https://creativecommons.org/licenses/by-nc/4.0/) for non-commercial use only. | |
| Please note that any commercial use of this code requires formal permission prior to use. | |
| ## Contact | |
| For technical questions, please contact `zhenli1031[AT]gmail.com` and `nkuzhuzl[AT]gmail.com`. | |
| For commercial licensing, please contact `cmm[AT]nankai.edu.cn` | |
| ## Acknowledgement | |
| We thank Jia-Wen Xiao, Zheng-Peng Duan, Rui-Qi Wu, and Xin Jin for proof reading. | |
| We thank [Zhewei Huang](https://github.com/hzwer) for his suggestions. | |
| Here are some great resources we benefit from: | |
| - [IFRNet](https://github.com/ltkong218/IFRNet) and [RIFE](https://github.com/megvii-research/ECCV2022-RIFE) for data processing, benchmarking, and loss designs. | |
| - [RAFT](https://github.com/princeton-vl/RAFT), [M2M-VFI](https://github.com/feinanshan/M2M_VFI), and [GMFlow](https://github.com/haofeixu/gmflow) for inspirations. | |
| - [FILM](https://github.com/google-research/frame-interpolation) for Web demo reference. | |
| **If you develop/use AMT in your projects, welcome to let us know. We will list your projects in this repository.** | |
| We also thank all of our contributors. | |
| <a href="https://github.com/MCG-NKU/AMT/graphs/contributors"> | |
| <img src="https://contrib.rocks/image?repo=MCG-NKU/AMT" /> | |
| </a> | |