# AMT: All-Pairs Multi-Field Transforms for Efficient Frame Interpolation


This repository contains the official implementation of the following paper:
> **AMT: All-Pairs Multi-Field Transforms for Efficient Frame Interpolation**<br>
> [Zhen Li](https://paper99.github.io/)<sup>\*</sup>, [Zuo-Liang Zhu](https://nk-cs-zzl.github.io/)<sup>\*</sup>, [Ling-Hao Han](https://scholar.google.com/citations?user=0ooNdgUAAAAJ&hl=en), [Qibin Hou](https://scholar.google.com/citations?hl=en&user=fF8OFV8AAAAJ&view_op=list_works), [Chun-Le Guo](https://scholar.google.com/citations?hl=en&user=RZLYwR0AAAAJ),  [Ming-Ming Cheng](https://mmcheng.net/cmm)<br>
> (\* denotes equal contribution) <br>
> Nankai University <br>
> In CVPR 2023<br>

[[Paper](https://arxiv.org/abs/2304.09790)]
[[Project Page](https://nk-cs-zzl.github.io/projects/amt/index.html)]
[[Web demos](#web-demos)]
[Video]

AMT is a **lightweight, fast, and accurate** algorithm for Frame Interpolation. 
It aims to provide practical solutions for **video generation** from **a few given frames (at least two frames)**.

![Demo gif](assets/amt_demo.gif)
* More examples can be found in our [project page](https://nk-cs-zzl.github.io/projects/amt/index.html).

## Web demos
Integrated into [Hugging Face Spaces 🤗](https://huggingface.co/spaces) using [Gradio](https://github.com/gradio-app/gradio). Try out the Web Demo: [![Hugging Face Spaces](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue)](https://huggingface.co/spaces/NKU-AMT/AMT)

Try AMT to interpolate between two or more images at [![PyTTI-Tools:FILM](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1IeVO5BmLouhRh6fL2z_y18kgubotoaBq?usp=sharing)


## Change Log
- **Apr 20, 2023**: Our code is publicly available.


## Method Overview
![pipeline](https://user-images.githubusercontent.com/21050959/229420451-65951bd0-732c-4f09-9121-f291a3862d6e.png)

For technical details, please refer to the [method.md](docs/method.md) file, or read the full report on [arXiv](https://arxiv.org/abs/2304.09790).

## Dependencies and Installation
1. Clone Repo

   ```bash
   git clone https://github.com/MCG-NKU/AMT.git
   ```

2. Create Conda Environment and Install Dependencies

   ```bash
   conda env create -f environment.yaml
   conda activate amt
   ```
3. Download pretrained models for demos from [Pretrained Models](#pretrained-models) and place them to the `pretrained` folder

## Quick Demo

**Note that the selected pretrained model (`[CKPT_PATH]`) needs to match the config file (`[CFG]`).**

 > Creating a video demo, increasing $n$ will slow down the motion in the video. (With $m$ input frames, `[N_ITER]` $=n$ corresponds to $2^n\times (m-1)+1$ output frames.)


 ```bash
 python demos/demo_2x.py -c [CFG] -p [CKPT] -n [N_ITER] -i [INPUT] -o [OUT_PATH] -r [FRAME_RATE]
 # e.g. [INPUT]
 # -i could be a video / a regular expression / a folder contains multiple images
 # -i demo.mp4 (video)/img_*.png (regular expression)/img0.png img1.png (images)/demo_input (folder)

 # e.g. a simple usage
 python demos/demo_2x.py -c cfgs/AMT-S.yaml -p pretrained/amt-s.pth -n 6 -i assets/quick_demo/img0.png assets/quick_demo/img1.png

 ```

 + Note: Please enable `--save_images` for saving the output images (Save speed will be slowed down if there are too many output images)
 + Input type supported: `a video` / `a regular expression` / `multiple images` / `a folder containing input frames`.
 + Results are in the `[OUT_PATH]` (default is `results/2x`) folder.

## Pretrained Models

<p id="Pretrained"></p>

<table>
<thead>
  <tr>
    <th> Dataset </th>
    <th> :link: Download Links </th>
    <th> Config file </th>
    <th> Trained on </th>
    <th> Arbitrary/Fixed </th>
  </tr>
</thead>
<tbody>
  <tr>
    <td>AMT-S</td>
    <th> [<a href="https://drive.google.com/file/d/1WmOKmQmd6pnLpID8EpUe-TddFpJuavrL/view?usp=share_link">Google Driver</a>][<a href="https://pan.baidu.com/s/1yGaNLeb9TG5-81t0skrOUA?pwd=f66n">Baidu Cloud</a>][<a href="https://huggingface.co/lalala125/AMT/resolve/main/amt-s.pth">Hugging Face</a>] </th>
    <th> [<a href="cfgs/AMT-S.yaml">cfgs/AMT-S</a>] </th>
    <th>Vimeo90k</th>
    <th>Fixed</th>
  </tr>
  <tr>
    <td>AMT-L</td>
    <th>[<a href="https://drive.google.com/file/d/1UyhYpAQLXMjFA55rlFZ0kdiSVTL7oU-z/view?usp=share_link">Google Driver</a>][<a href="https://pan.baidu.com/s/1qI4fBgS405Bd4Wn1R3Gbeg?pwd=nbne">Baidu Cloud</a>][<a href="https://huggingface.co/lalala125/AMT/resolve/main/amt-l.pth">Hugging Face</a>]</th>
    <th> [<a href="cfgs/AMT-L.yaml">cfgs/AMT-L</a>] </th>
    <th>Vimeo90k</th>
    <th>Fixed</th>
  </tr>
  <tr>
    <td>AMT-G</td>
    <th>[<a href="https://drive.google.com/file/d/1yieLtKh4ei3gOrLN1LhKSP_9157Q-mtP/view?usp=share_link">Google Driver</a>][<a href="https://pan.baidu.com/s/1AjmQVziQut1bXgQnDcDKvA?pwd=caf6">Baidu Cloud</a>][<a href="https://huggingface.co/lalala125/AMT/resolve/main/amt-g.pth">Hugging Face</a>] </th>
    <th> [<a href="cfgs/AMT-G.yaml">cfgs/AMT-G</a>] </th>
    <th>Vimeo90k</th>
    <th>Fixed</th>
  </tr>
  <tr>
    <td>AMT-S</td>
    <th>[<a href="https://drive.google.com/file/d/1f1xAF0EDm-rjDdny8_aLyeedfM0QL4-C/view?usp=share_link">Google Driver</a>][<a href="https://pan.baidu.com/s/1eZtoULyduQM8AkXeYEBOEw?pwd=8hy3">Baidu Cloud</a>][<a href="https://huggingface.co/lalala125/AMT/resolve/main/gopro_amt-s.pth">Hugging Face</a>] </th>
    <th> [<a href="cfgs/AMT-S_gopro.yaml">cfgs/AMT-S_gopro</a>] </th>
    <th>GoPro</th>
    <th>Arbitrary</th>
  </tr>
</tbody>
</table>

## Training and Evaluation

Please refer to [develop.md](docs/develop.md) to learn how to benchmark the AMT and how to train a new AMT model from scratch.


## Citation
   If you find our repo useful for your research, please consider citing our paper:

   ```bibtex
   @inproceedings{licvpr23amt,
      title={AMT: All-Pairs Multi-Field Transforms for Efficient Frame Interpolation},
      author={Li, Zhen and Zhu, Zuo-Liang and Han, Ling-Hao and Hou, Qibin and Guo, Chun-Le and Cheng, Ming-Ming},
      booktitle={IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
      year={2023}
   }
   ```


## License
This code is licensed under the [Creative Commons Attribution-NonCommercial 4.0 International](https://creativecommons.org/licenses/by-nc/4.0/) for non-commercial use only.
Please note that any commercial use of this code requires formal permission prior to use.

## Contact

For technical questions, please contact `zhenli1031[AT]gmail.com` and `nkuzhuzl[AT]gmail.com`.

For commercial licensing, please contact `cmm[AT]nankai.edu.cn`

## Acknowledgement

We thank Jia-Wen Xiao, Zheng-Peng Duan, Rui-Qi Wu, and Xin Jin for proof reading.
We thank [Zhewei Huang](https://github.com/hzwer) for his suggestions.

Here are some great resources we benefit from:

- [IFRNet](https://github.com/ltkong218/IFRNet) and [RIFE](https://github.com/megvii-research/ECCV2022-RIFE) for data processing, benchmarking, and loss designs.
- [RAFT](https://github.com/princeton-vl/RAFT), [M2M-VFI](https://github.com/feinanshan/M2M_VFI), and [GMFlow](https://github.com/haofeixu/gmflow) for inspirations.
- [FILM](https://github.com/google-research/frame-interpolation) for Web demo reference.


**If you develop/use AMT in your projects, welcome to let us know. We will list your projects in this repository.**

We also thank all of our contributors.

<a href="https://github.com/MCG-NKU/AMT/graphs/contributors">
  <img src="https://contrib.rocks/image?repo=MCG-NKU/AMT" />
</a>