|
--- |
|
license: apache-2.0 |
|
language: |
|
- en |
|
tags: |
|
- magicdrive |
|
- image-generation |
|
- controllable-generation |
|
- autonomous-driving |
|
--- |
|
|
|
# MagicDrive-t |
|
|
|
[ArXiv](https://arxiv.org/abs/2310.02601) | [Web](https://gaoruiyuan.com/magicdrive/) | [License](https://github.com/cure-lab/MagicDrive/blob/video/LICENSE) | [GitHub](https://github.com/cure-lab/MagicDrive/tree/video) (`video`-branch) |
|
|
|
This repository contains 60-frame driving-view video generation model checkpoint. |
|
|
|
- We train this model by loading from the 16-frame 224x400 checkpoint (we interpolate the temporal positional embeddings from 16 to 60), and train for 80k steps. |
|
- Please check `hydra/overrides.yaml` for command overrides in the training config. |
|
|
|
> MagicDrive: Street View Generation with Diverse 3D Geometry Control <br> |
|
> [Ruiyuan Gao](https://gaoruiyuan.com/)<sup>1\*</sup>, [Kai Chen](https://kaichen1998.github.io/)<sup>2\*</sup>, [Enze Xie](https://xieenze.github.io/)<sup>3^</sup>, [Lanqing Hong](https://scholar.google.com.sg/citations?user=2p7x6OUAAAAJ&hl=en)<sup>3</sup>, [Zhenguo Li](https://scholar.google.com/citations?user=XboZC1AAAAAJ&hl=en)<sup>3</sup>, [Dit-Yan Yeung](https://sites.google.com/view/dyyeung)<sup>2</sup>, [Qiang Xu](https://cure-lab.github.io/)<sup>1^</sup><br> |
|
> <sup>1</sup>CUHK <sup>2</sup>HKUST <sup>3</sup>Huawei Noah's Ark Lab <br> |
|
> <sup>\*</sup>Equal Contribution <sup>^</sup>Corresponding Authors |
|
|
|
<div> |
|
<p class="top_p">Generated video A.</p> |
|
<video id="4470_rm2" muted preload controls playsinline autoplay loop height="100%"> |
|
<source src="https://gaoruiyuan.com/magicdrive/static/60f/4470_rm2.mp4" type="video/mp4" /> |
|
</video> |
|
<p class="top_p">More cars in A on the right and front.</p> |
|
<video id="4470_rm15" muted preload controls playsinline autoplay loop height="100%"> |
|
<source src="https://gaoruiyuan.com/magicdrive/static/60f/4470_rm15.mp4" type="video/mp4" /> |
|
</video> |
|
<p class="top_p"><br></p> |
|
|
|
<p class="top_p">Generated video B</p> |
|
<video id="5664_sunny" muted preload controls playsinline autoplay loop height="100%"> |
|
<source src="https://gaoruiyuan.com/magicdrive/static/60f/5664_sunny.mp4" type="video/mp4" /> |
|
</video> |
|
<p class="top_p">Change B to night.</p> |
|
<video id="5664_night" muted preload controls playsinline autoplay loop height="100%"> |
|
<source src="https://gaoruiyuan.com/magicdrive/static/60f/5664_night.mp4" type="video/mp4" /> |
|
</video> |
|
<p class="top_p">Remove the car on the left in B.</p> |
|
<video id="5664_sunny_n1" muted preload controls playsinline autoplay loop height="100%"> |
|
<source src="https://gaoruiyuan.com/magicdrive/static/60f/5664_sunny_n1.mp4" type="video/mp4" /> |
|
</video> |
|
<p class="top_p"><br></p> |
|
|
|
<p class="top_p">Generated video C.</p> |
|
<video id="9357_sunny_n1" muted preload controls playsinline autoplay loop height="100%"> |
|
<source src="https://gaoruiyuan.com/magicdrive/static/60f/9357_sunny_n1.mp4" type="video/mp4" /> |
|
</video> |
|
<p class="top_p">Remove the car on the right in C.</p> |
|
<video id="9357_sunny_n2" muted preload controls playsinline autoplay loop height="100%"> |
|
<source src="https://gaoruiyuan.com/magicdrive/static/60f/9357_sunny_n2.mp4" type="video/mp4" /> |
|
</video> |
|
<p class="top_p">Change C to a rainy day.</p> |
|
<video id="9357_rain_n1" muted preload controls playsinline autoplay loop height="100%"> |
|
<source src="https://gaoruiyuan.com/magicdrive/static/60f/9357_rain_n1.mp4" type="video/mp4" /> |
|
</video> |
|
<p class="top_p">Change C to a rainy day and remove the car on the right.</p> |
|
<video id="9357_rain_n2" muted preload controls playsinline autoplay loop height="100%"> |
|
<source src="https://gaoruiyuan.com/magicdrive/static/60f/9357_rain_n2.mp4" type="video/mp4" /> |
|
</video> |
|
</div> |
|
|
|
For more information, please refer to our GitHub: https://github.com/cure-lab/MagicDrive/tree/video (i.e., the video branch of MagicDrive repo) |
