# Marigold: Repurposing Diffusion-Based Image Generators for Monocular Depth Estimation
**CVPR 2024 (Oral, Best Paper Award Candidate)**
This repository represents the official implementation of the paper titled "Repurposing Diffusion-Based Image Generators for Monocular Depth Estimation".
[](https://marigoldmonodepth.github.io)
[](https://arxiv.org/abs/2312.02145)
[-Space-yellow)](https://huggingface.co/spaces/prs-eth/marigold-lcm)
[-Model-green)](https://huggingface.co/prs-eth/marigold-lcm-v1-0)
[](https://colab.research.google.com/drive/12G8reD13DdpMie5ZQlaFNo2WCGeNUH-u?usp=sharing)
[](https://www.apache.org/licenses/LICENSE-2.0)
[Bingxin Ke](http://www.kebingxin.com/),
[Anton Obukhov](https://www.obukhov.ai/),
[Shengyu Huang](https://shengyuh.github.io/),
[Nando Metzger](https://nandometzger.github.io/),
[Rodrigo Caye Daudt](https://rcdaudt.github.io/),
[Konrad Schindler](https://scholar.google.com/citations?user=FZuNgqIAAAAJ&hl=en )
We present Marigold, a diffusion model, and associated fine-tuning protocol for monocular depth estimation. Its core principle is to leverage the rich visual knowledge stored in modern generative image models. Our model, derived from Stable Diffusion and fine-tuned with synthetic data, can zero-shot transfer to unseen data, offering state-of-the-art monocular depth estimation results.
## ๐ข News
2024-05-28: Training code is released.
2024-03-23: Added [LCM v1.0](https://huggingface.co/prs-eth/marigold-lcm-v1-0) for faster inference - try it out at -Space-yellow)
2024-03-04: Accepted to CVPR 2024.
2023-12-22: Contributed to Diffusers [community pipeline](https://github.com/huggingface/diffusers/tree/main/examples/community#marigold-depth-estimation).
2023-12-19: Updated [license](LICENSE.txt) to Apache License, Version 2.0.
2023-12-08: Added
- try it out with your images for free!
2023-12-05: Added 
- dive deeper into our inference pipeline!
2023-12-04: Added 
paper and inference code (this repository).
## ๐ Usage
**We offer several ways to interact with Marigold**:
1. We integrated [Marigold Pipelines into diffusers ๐งจ](https://huggingface.co/docs/diffusers/api/pipelines/marigold). Check out many exciting usage scenarios in [this diffusers tutorial](https://huggingface.co/docs/diffusers/using-diffusers/marigold_usage).
1. A free online interactive demo is available here: (kudos to the HF team for the GPU grant)
1. Run the demo locally (requires a GPU and an `nvidia-docker2`, see [Installation Guide](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html)):
1. Paper version: `docker run -it -p 7860:7860 --platform=linux/amd64 --gpus all registry.hf.space/toshas-marigold:latest python app.py`
1. LCM version: `docker run -it -p 7860:7860 --platform=linux/amd64 --gpus all registry.hf.space/prs-eth-marigold-lcm:latest python app.py`
1. Extended demo on a Google Colab:
1. If you just want to see the examples, visit our gallery: 
1. Finally, local development instructions with this codebase are given below.
## ๐ ๏ธ Setup
The inference code was tested on:
- Ubuntu 22.04 LTS, Python 3.10.12, CUDA 11.7, GeForce RTX 3090 (pip, Mamba)
- CentOS Linux 7, Python 3.10.4, CUDA 11.7, GeForce RTX 4090 (pip)
- Windows 11 22H2, Python 3.10.12, CUDA 12.3, GeForce RTX 3080 (Mamba)
- MacOS 14.2, Python 3.10.12, M1 16G (pip)
### ๐ชง A Note for Windows users
We recommend running the code in WSL2:
1. Install WSL following [installation guide](https://learn.microsoft.com/en-us/windows/wsl/install#install-wsl-command).
1. Install CUDA support for WSL following [installation guide](https://docs.nvidia.com/cuda/wsl-user-guide/index.html#cuda-support-for-wsl-2).
1. Find your drives in `/mnt//`; check [WSL FAQ](https://learn.microsoft.com/en-us/windows/wsl/faq#how-do-i-access-my-c--drive-) for more details. Navigate to the working directory of choice.
### ๐ฆ Repository
Clone the repository (requires git):
```bash
git clone https://github.com/prs-eth/Marigold.git
cd Marigold
```
### ๐ป Dependencies
We provide several ways to install the dependencies.
1. **Using [Mamba](https://github.com/mamba-org/mamba)**, which can installed together with [Miniforge3](https://github.com/conda-forge/miniforge?tab=readme-ov-file#miniforge3).
Windows users: Install the Linux version into the WSL.
After the installation, Miniforge needs to be activated first: `source /home/$USER/miniforge3/bin/activate`.
Create the environment and install dependencies into it:
```bash
mamba env create -n marigold --file environment.yaml
conda activate marigold
```
2. **Using pip:**
Alternatively, create a Python native virtual environment and install dependencies into it:
```bash
python -m venv venv/marigold
source venv/marigold/bin/activate
pip install -r requirements.txt
```
Keep the environment activated before running the inference script.
Activate the environment again after restarting the terminal session.
## ๐ Testing on your images
### ๐ท Prepare images
1. Use selected images from our paper:
```bash
bash script/download_sample_data.sh
```
1. Or place your images in a directory, for example, under `input/in-the-wild_example`, and run the following inference command.
### ๐ Run inference with LCM (faster)
The [LCM checkpoint](https://huggingface.co/prs-eth/marigold-lcm-v1-0) is distilled from our original checkpoint towards faster inference speed (by reducing inference steps). The inference steps can be as few as 1 (default) to 4. Run with default LCM setting:
```bash
python run.py \
--input_rgb_dir input/in-the-wild_example \
--output_dir output/in-the-wild_example_lcm
```
### ๐ฎ Run inference with DDIM (paper setting)
This setting corresponds to our paper. For academic comparison, please run with this setting.
```bash
python run.py \
--checkpoint prs-eth/marigold-v1-0 \
--denoise_steps 50 \
--ensemble_size 10 \
--input_rgb_dir input/in-the-wild_example \
--output_dir output/in-the-wild_example
```
You can find all results in `output/in-the-wild_example`. Enjoy!
### โ๏ธ Inference settings
The default settings are optimized for the best result. However, the behavior of the code can be customized:
- Trade-offs between the **accuracy** and **speed** (for both options, larger values result in better accuracy at the cost of slower inference.)
- `--ensemble_size`: Number of inference passes in the ensemble. For LCM `ensemble_size` is more important than `denoise_steps`. Default: ~~10~~ 5 (for LCM).
- `--denoise_steps`: Number of denoising steps of each inference pass. For the original (DDIM) version, it's recommended to use 10-50 steps, while for LCM 1-4 steps. When unassigned (`None`), will read default setting from model config. Default: ~~10 4 (for LCM)~~ `None`.
- By default, the inference script resizes input images to the *processing resolution*, and then resizes the prediction back to the original resolution. This gives the best quality, as Stable Diffusion, from which Marigold is derived, performs best at 768x768 resolution.
- `--processing_res`: the processing resolution; set as 0 to process the input resolution directly. When unassigned (`None`), will read default setting from model config. Default: ~~768~~ `None`.
- `--output_processing_res`: produce output at the processing resolution instead of upsampling it to the input resolution. Default: False.
- `--resample_method`: the resampling method used to resize images and depth predictions. This can be one of `bilinear`, `bicubic`, or `nearest`. Default: `bilinear`.
- `--half_precision` or `--fp16`: Run with half-precision (16-bit float) to have faster speed and reduced VRAM usage, but might lead to suboptimal results.
- `--seed`: Random seed can be set to ensure additional reproducibility. Default: None (unseeded). Note: forcing `--batch_size 1` helps to increase reproducibility. To ensure full reproducibility, [deterministic mode](https://pytorch.org/docs/stable/notes/randomness.html#avoiding-nondeterministic-algorithms) needs to be used.
- `--batch_size`: Batch size of repeated inference. Default: 0 (best value determined automatically).
- `--color_map`: [Colormap](https://matplotlib.org/stable/users/explain/colors/colormaps.html) used to colorize the depth prediction. Default: Spectral. Set to `None` to skip colored depth map generation.
- `--apple_silicon`: Use Apple Silicon MPS acceleration.
### โฌ Checkpoint cache
By default, the [checkpoint](https://huggingface.co/prs-eth/marigold-v1-0) is stored in the Hugging Face cache.
The `HF_HOME` environment variable defines its location and can be overridden, e.g.:
```bash
export HF_HOME=$(pwd)/cache
```
Alternatively, use the following script to download the checkpoint weights locally:
```bash
bash script/download_weights.sh marigold-v1-0
# or LCM checkpoint
bash script/download_weights.sh marigold-lcm-v1-0
```
At inference, specify the checkpoint path:
```bash
python run.py \
--checkpoint checkpoint/marigold-v1-0 \
--denoise_steps 50 \
--ensemble_size 10 \
--input_rgb_dir input/in-the-wild_example\
--output_dir output/in-the-wild_example
```
## ๐ฆฟ Evaluation on test datasets
Install additional dependencies:
```bash
pip install -r requirements+.txt -r requirements.txt
```
Set data directory variable (also needed in evaluation scripts) and download [evaluation datasets](https://share.phys.ethz.ch/~pf/bingkedata/marigold/evaluation_dataset) into corresponding subfolders:
```bash
export BASE_DATA_DIR= # Set target data directory
wget -r -np -nH --cut-dirs=4 -R "index.html*" -P ${BASE_DATA_DIR} https://share.phys.ethz.ch/~pf/bingkedata/marigold/evaluation_dataset/
```
Run inference and evaluation scripts, for example:
```bash
# Run inference
bash script/eval/11_infer_nyu.sh
# Evaluate predictions
bash script/eval/12_eval_nyu.sh
```
Note: although the seed has been set, the results might still be slightly different on different hardware.
## ๐๏ธ Training
Based on the previously created environment, install extended requirements:
```bash
pip install -r requirements++.txt -r requirements+.txt -r requirements.txt
```
Set environment parameters for the data directory:
```bash
export BASE_DATA_DIR=YOUR_DATA_DIR # directory of training data
export BASE_CKPT_DIR=YOUR_CHECKPOINT_DIR # directory of pretrained checkpoint
```
Download Stable Diffusion v2 [checkpoint](https://huggingface.co/stabilityai/stable-diffusion-2) into `${BASE_CKPT_DIR}`
Prepare for [Hypersim](https://github.com/apple/ml-hypersim) and [Virtual KITTI 2](https://europe.naverlabs.com/research/computer-vision/proxy-virtual-worlds-vkitti-2/) datasets and save into `${BASE_DATA_DIR}`. Please refer to [this README](script/dataset_preprocess/hypersim/README.md) for Hypersim preprocessing.
Run training script
```bash
python train.py --config config/train_marigold.yaml
```
Resume from a checkpoint, e.g.
```bash
python train.py --resume_run output/marigold_base/checkpoint/latest
```
Evaluating results
Only the U-Net is updated and saved during training. To use the inference pipeline with your training result, replace `unet` folder in Marigold checkpoints with that in the `checkpoint` output folder. Then refer to [this section](#evaluation) for evaluation.
**Note**: Although random seeds have been set, the training result might be slightly different on different hardwares. It's recommended to train without interruption.
## โ๏ธ Contributing
Please refer to [this](CONTRIBUTING.md) instruction.
## ๐ค Troubleshooting
| Problem | Solution |
|----------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------|
| (Windows) Invalid DOS bash script on WSL | Run `dos2unix ` to convert script format |
| (Windows) error on WSL: `Could not load library libcudnn_cnn_infer.so.8. Error: libcuda.so: cannot open shared object file: No such file or directory` | Run `export LD_LIBRARY_PATH=/usr/lib/wsl/lib:$LD_LIBRARY_PATH` |
## ๐ Citation
Please cite our paper:
```bibtex
@InProceedings{ke2023repurposing,
title={Repurposing Diffusion-Based Image Generators for Monocular Depth Estimation},
author={Bingxin Ke and Anton Obukhov and Shengyu Huang and Nando Metzger and Rodrigo Caye Daudt and Konrad Schindler},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
year={2024}
}
```
## ๐ซ License
This work is licensed under the Apache License, Version 2.0 (as defined in the [LICENSE](LICENSE.txt)).
By downloading and using the code and model you agree to the terms in the [LICENSE](LICENSE.txt).
[](https://www.apache.org/licenses/LICENSE-2.0)