| ## Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer | |
| This repository contains code to compute depth from a single image. It accompanies our [paper](https://arxiv.org/abs/1907.01341v3): | |
| >Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer | |
| René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, Vladlen Koltun | |
| and our [preprint](https://arxiv.org/abs/2103.13413): | |
| > Vision Transformers for Dense Prediction | |
| > René Ranftl, Alexey Bochkovskiy, Vladlen Koltun | |
| For the latest release MiDaS 3.1, a [technical report](https://arxiv.org/pdf/2307.14460.pdf) and [video](https://www.youtube.com/watch?v=UjaeNNFf9sE&t=3s) are available. | |
| MiDaS was trained on up to 12 datasets (ReDWeb, DIML, Movies, MegaDepth, WSVD, TartanAir, HRWSI, ApolloScape, BlendedMVS, IRS, KITTI, NYU Depth V2) with | |
| multi-objective optimization. | |
| The original model that was trained on 5 datasets (`MIX 5` in the paper) can be found [here](https://github.com/isl-org/MiDaS/releases/tag/v2). | |
| The figure below shows an overview of the different MiDaS models; the bubble size scales with number of parameters. | |
|  | |
| ### Setup | |
| 1) Pick one or more models and download the corresponding weights to the `weights` folder: | |
| MiDaS 3.1 | |
| - For highest quality: [dpt_beit_large_512](https://github.com/isl-org/MiDaS/releases/download/v3_1/dpt_beit_large_512.pt) | |
| - For moderately less quality, but better speed-performance trade-off: [dpt_swin2_large_384](https://github.com/isl-org/MiDaS/releases/download/v3_1/dpt_swin2_large_384.pt) | |
| - For embedded devices: [dpt_swin2_tiny_256](https://github.com/isl-org/MiDaS/releases/download/v3_1/dpt_swin2_tiny_256.pt), [dpt_levit_224](https://github.com/isl-org/MiDaS/releases/download/v3_1/dpt_levit_224.pt) | |
| - For inference on Intel CPUs, OpenVINO may be used for the small legacy model: openvino_midas_v21_small [.xml](https://github.com/isl-org/MiDaS/releases/download/v3_1/openvino_midas_v21_small_256.xml), [.bin](https://github.com/isl-org/MiDaS/releases/download/v3_1/openvino_midas_v21_small_256.bin) | |
| MiDaS 3.0: Legacy transformer models [dpt_large_384](https://github.com/isl-org/MiDaS/releases/download/v3/dpt_large_384.pt) and [dpt_hybrid_384](https://github.com/isl-org/MiDaS/releases/download/v3/dpt_hybrid_384.pt) | |
| MiDaS 2.1: Legacy convolutional models [midas_v21_384](https://github.com/isl-org/MiDaS/releases/download/v2_1/midas_v21_384.pt) and [midas_v21_small_256](https://github.com/isl-org/MiDaS/releases/download/v2_1/midas_v21_small_256.pt) | |
| 1) Set up dependencies: | |
| ```shell | |
| conda env create -f environment.yaml | |
| conda activate midas-py310 | |
| ``` | |
| #### optional | |
| For the Next-ViT model, execute | |
| ```shell | |
| git submodule add https://github.com/isl-org/Next-ViT midas/external/next_vit | |
| ``` | |
| For the OpenVINO model, install | |
| ```shell | |
| pip install openvino | |
| ``` | |
| ### Usage | |
| 1) Place one or more input images in the folder `input`. | |
| 2) Run the model with | |
| ```shell | |
| python run.py --model_type <model_type> --input_path input --output_path output | |
| ``` | |
| where ```<model_type>``` is chosen from [dpt_beit_large_512](#model_type), [dpt_beit_large_384](#model_type), | |
| [dpt_beit_base_384](#model_type), [dpt_swin2_large_384](#model_type), [dpt_swin2_base_384](#model_type), | |
| [dpt_swin2_tiny_256](#model_type), [dpt_swin_large_384](#model_type), [dpt_next_vit_large_384](#model_type), | |
| [dpt_levit_224](#model_type), [dpt_large_384](#model_type), [dpt_hybrid_384](#model_type), | |
| [midas_v21_384](#model_type), [midas_v21_small_256](#model_type), [openvino_midas_v21_small_256](#model_type). | |
| 3) The resulting depth maps are written to the `output` folder. | |
| #### optional | |
| 1) By default, the inference resizes the height of input images to the size of a model to fit into the encoder. This | |
| size is given by the numbers in the model names of the [accuracy table](#accuracy). Some models do not only support a single | |
| inference height but a range of different heights. Feel free to explore different heights by appending the extra | |
| command line argument `--height`. Unsupported height values will throw an error. Note that using this argument may | |
| decrease the model accuracy. | |
| 2) By default, the inference keeps the aspect ratio of input images when feeding them into the encoder if this is | |
| supported by a model (all models except for Swin, Swin2, LeViT). In order to resize to a square resolution, | |
| disregarding the aspect ratio while preserving the height, use the command line argument `--square`. | |
| #### via Camera | |
| If you want the input images to be grabbed from the camera and shown in a window, leave the input and output paths | |
| away and choose a model type as shown above: | |
| ```shell | |
| python run.py --model_type <model_type> --side | |
| ``` | |
| The argument `--side` is optional and causes both the input RGB image and the output depth map to be shown | |
| side-by-side for comparison. | |
| #### via Docker | |
| 1) Make sure you have installed Docker and the | |
| [NVIDIA Docker runtime](https://github.com/NVIDIA/nvidia-docker/wiki/Installation-\(Native-GPU-Support\)). | |
| 2) Build the Docker image: | |
| ```shell | |
| docker build -t midas . | |
| ``` | |
| 3) Run inference: | |
| ```shell | |
| docker run --rm --gpus all -v $PWD/input:/opt/MiDaS/input -v $PWD/output:/opt/MiDaS/output -v $PWD/weights:/opt/MiDaS/weights midas | |
| ``` | |
| This command passes through all of your NVIDIA GPUs to the container, mounts the | |
| `input` and `output` directories and then runs the inference. | |
| #### via PyTorch Hub | |
| The pretrained model is also available on [PyTorch Hub](https://pytorch.org/hub/intelisl_midas_v2/) | |
| #### via TensorFlow or ONNX | |
| See [README](https://github.com/isl-org/MiDaS/tree/master/tf) in the `tf` subdirectory. | |
| Currently only supports MiDaS v2.1. | |
| #### via Mobile (iOS / Android) | |
| See [README](https://github.com/isl-org/MiDaS/tree/master/mobile) in the `mobile` subdirectory. | |
| #### via ROS1 (Robot Operating System) | |
| See [README](https://github.com/isl-org/MiDaS/tree/master/ros) in the `ros` subdirectory. | |
| Currently only supports MiDaS v2.1. DPT-based models to be added. | |
| ### Accuracy | |
| We provide a **zero-shot error** $\epsilon_d$ which is evaluated for 6 different datasets | |
| (see [paper](https://arxiv.org/abs/1907.01341v3)). **Lower error values are better**. | |
| $\color{green}{\textsf{Overall model quality is represented by the improvement}}$ ([Imp.](#improvement)) with respect to | |
| MiDaS 3.0 DPT<sub>L-384</sub>. The models are grouped by the height used for inference, whereas the square training resolution is given by | |
| the numbers in the model names. The table also shows the **number of parameters** (in millions) and the | |
| **frames per second** for inference at the training resolution (for GPU RTX 3090): | |
| | MiDaS Model | DIW </br><sup>WHDR</sup> | Eth3d </br><sup>AbsRel</sup> | Sintel </br><sup>AbsRel</sup> | TUM </br><sup>δ1</sup> | KITTI </br><sup>δ1</sup> | NYUv2 </br><sup>δ1</sup> | $\color{green}{\textsf{Imp.}}$ </br><sup>%</sup> | Par.</br><sup>M</sup> | FPS</br><sup> </sup> | | |
| |-----------------------------------------------------------------------------------------------------------------------|-------------------------:|-----------------------------:|------------------------------:|-------------------------:|-------------------------:|-------------------------:|-------------------------------------------------:|----------------------:|--------------------------:| | |
| | **Inference height 512** | | | | | | | | | | | |
| | [v3.1 BEiT<sub>L-512</sub>](https://github.com/isl-org/MiDaS/releases/download/v3_1/dpt_beit_large_512.pt) | 0.1137 | 0.0659 | 0.2366 | **6.13** | 11.56* | **1.86*** | $\color{green}{\textsf{19}}$ | **345** | **5.7** | | |
| | [v3.1 BEiT<sub>L-512</sub>](https://github.com/isl-org/MiDaS/releases/download/v3_1/dpt_beit_large_512.pt)$\tiny{\square}$ | **0.1121** | **0.0614** | **0.2090** | 6.46 | **5.00*** | 1.90* | $\color{green}{\textsf{34}}$ | **345** | **5.7** | | |
| | | | | | | | | | | | | |
| | **Inference height 384** | | | | | | | | | | | |
| | [v3.1 BEiT<sub>L-512</sub>](https://github.com/isl-org/MiDaS/releases/download/v3_1/dpt_beit_large_512.pt) | 0.1245 | 0.0681 | **0.2176** | **6.13** | 6.28* | **2.16*** | $\color{green}{\textsf{28}}$ | 345 | 12 | | |
| | [v3.1 Swin2<sub>L-384</sub>](https://github.com/isl-org/MiDaS/releases/download/v3_1/dpt_swin2_large_384.pt)$\tiny{\square}$ | 0.1106 | 0.0732 | 0.2442 | 8.87 | **5.84*** | 2.92* | $\color{green}{\textsf{22}}$ | 213 | 41 | | |
| | [v3.1 Swin2<sub>B-384</sub>](https://github.com/isl-org/MiDaS/releases/download/v3_1/dpt_swin2_base_384.pt)$\tiny{\square}$ | 0.1095 | 0.0790 | 0.2404 | 8.93 | 5.97* | 3.28* | $\color{green}{\textsf{22}}$ | 102 | 39 | | |
| | [v3.1 Swin<sub>L-384</sub>](https://github.com/isl-org/MiDaS/releases/download/v3_1/dpt_swin_large_384.pt)$\tiny{\square}$ | 0.1126 | 0.0853 | 0.2428 | 8.74 | 6.60* | 3.34* | $\color{green}{\textsf{17}}$ | 213 | 49 | | |
| | [v3.1 BEiT<sub>L-384</sub>](https://github.com/isl-org/MiDaS/releases/download/v3_1/dpt_beit_large_384.pt) | 0.1239 | **0.0667** | 0.2545 | 7.17 | 9.84* | 2.21* | $\color{green}{\textsf{17}}$ | 344 | 13 | | |
| | [v3.1 Next-ViT<sub>L-384</sub>](https://github.com/isl-org/MiDaS/releases/download/v3_1/dpt_next_vit_large_384.pt) | **0.1031** | 0.0954 | 0.2295 | 9.21 | 6.89* | 3.47* | $\color{green}{\textsf{16}}$ | **72** | 30 | | |
| | [v3.1 BEiT<sub>B-384</sub>](https://github.com/isl-org/MiDaS/releases/download/v3_1/dpt_beit_base_384.pt) | 0.1159 | 0.0967 | 0.2901 | 9.88 | 26.60* | 3.91* | $\color{green}{\textsf{-31}}$ | 112 | 31 | | |
| | [v3.0 DPT<sub>L-384</sub>](https://github.com/isl-org/MiDaS/releases/download/v3/dpt_large_384.pt) | 0.1082 | 0.0888 | 0.2697 | 9.97 | 8.46 | 8.32 | $\color{green}{\textsf{0}}$ | 344 | **61** | | |
| | [v3.0 DPT<sub>H-384</sub>](https://github.com/isl-org/MiDaS/releases/download/v3/dpt_hybrid_384.pt) | 0.1106 | 0.0934 | 0.2741 | 10.89 | 11.56 | 8.69 | $\color{green}{\textsf{-10}}$ | 123 | 50 | | |
| | [v2.1 Large<sub>384</sub>](https://github.com/isl-org/MiDaS/releases/download/v2_1/midas_v21_384.pt) | 0.1295 | 0.1155 | 0.3285 | 12.51 | 16.08 | 8.71 | $\color{green}{\textsf{-32}}$ | 105 | 47 | | |
| | | | | | | | | | | | | |
| | **Inference height 256** | | | | | | | | | | | |
| | [v3.1 Swin2<sub>T-256</sub>](https://github.com/isl-org/MiDaS/releases/download/v3_1/dpt_swin2_tiny_256.pt)$\tiny{\square}$ | **0.1211** | **0.1106** | **0.2868** | **13.43** | **10.13*** | **5.55*** | $\color{green}{\textsf{-11}}$ | 42 | 64 | | |
| | [v2.1 Small<sub>256</sub>](https://github.com/isl-org/MiDaS/releases/download/v2_1/midas_v21_small_256.pt) | 0.1344 | 0.1344 | 0.3370 | 14.53 | 29.27 | 13.43 | $\color{green}{\textsf{-76}}$ | **21** | **90** | | |
| | | | | | | | | | | | | |
| | **Inference height 224** | | | | | | | | | | | |
| | [v3.1 LeViT<sub>224</sub>](https://github.com/isl-org/MiDaS/releases/download/v3_1/dpt_levit_224.pt)$\tiny{\square}$ | **0.1314** | **0.1206** | **0.3148** | **18.21** | **15.27*** | **8.64*** | $\color{green}{\textsf{-40}}$ | **51** | **73** | | |
| * No zero-shot error, because models are also trained on KITTI and NYU Depth V2\ | |
| $\square$ Validation performed at **square resolution**, either because the transformer encoder backbone of a model | |
| does not support non-square resolutions (Swin, Swin2, LeViT) or for comparison with these models. All other | |
| validations keep the aspect ratio. A difference in resolution limits the comparability of the zero-shot error and the | |
| improvement, because these quantities are averages over the pixels of an image and do not take into account the | |
| advantage of more details due to a higher resolution.\ | |
| Best values per column and same validation height in bold | |
| #### Improvement | |
| The improvement in the above table is defined as the relative zero-shot error with respect to MiDaS v3.0 | |
| DPT<sub>L-384</sub> and averaging over the datasets. So, if $\epsilon_d$ is the zero-shot error for dataset $d$, then | |
| the $\color{green}{\textsf{improvement}}$ is given by $100(1-(1/6)\sum_d\epsilon_d/\epsilon_{d,\rm{DPT_{L-384}}})$%. | |
| Note that the improvements of 10% for MiDaS v2.0 → v2.1 and 21% for MiDaS v2.1 → v3.0 are not visible from the | |
| improvement column (Imp.) in the table but would require an evaluation with respect to MiDaS v2.1 Large<sub>384</sub> | |
| and v2.0 Large<sub>384</sub> respectively instead of v3.0 DPT<sub>L-384</sub>. | |
| ### Depth map comparison | |
| Zoom in for better visibility | |
|  | |
| ### Speed on Camera Feed | |
| Test configuration | |
| - Windows 10 | |
| - 11th Gen Intel Core i7-1185G7 3.00GHz | |
| - 16GB RAM | |
| - Camera resolution 640x480 | |
| - openvino_midas_v21_small_256 | |
| Speed: 22 FPS | |
| ### Applications | |
| MiDaS is used in the following other projects from Intel Labs: | |
| - [ZoeDepth](https://arxiv.org/pdf/2302.12288.pdf) (code available [here](https://github.com/isl-org/ZoeDepth)): MiDaS computes the relative depth map given an image. For metric depth estimation, ZoeDepth can be used, which combines MiDaS with a metric depth binning module appended to the decoder. | |
| - [LDM3D](https://arxiv.org/pdf/2305.10853.pdf) (Hugging Face model available [here](https://huggingface.co/Intel/ldm3d-4c)): LDM3D is an extension of vanilla stable diffusion designed to generate joint image and depth data from a text prompt. The depth maps used for supervision when training LDM3D have been computed using MiDaS. | |
| ### Changelog | |
| * [Dec 2022] Released [MiDaS v3.1](https://arxiv.org/pdf/2307.14460.pdf): | |
| - New models based on 5 different types of transformers ([BEiT](https://arxiv.org/pdf/2106.08254.pdf), [Swin2](https://arxiv.org/pdf/2111.09883.pdf), [Swin](https://arxiv.org/pdf/2103.14030.pdf), [Next-ViT](https://arxiv.org/pdf/2207.05501.pdf), [LeViT](https://arxiv.org/pdf/2104.01136.pdf)) | |
| - Training datasets extended from 10 to 12, including also KITTI and NYU Depth V2 using [BTS](https://github.com/cleinc/bts) split | |
| - Best model, BEiT<sub>Large 512</sub>, with resolution 512x512, is on average about [28% more accurate](#Accuracy) than MiDaS v3.0 | |
| - Integrated live depth estimation from camera feed | |
| * [Sep 2021] Integrated to [Huggingface Spaces](https://huggingface.co/spaces) with [Gradio](https://github.com/gradio-app/gradio). See [Gradio Web Demo](https://huggingface.co/spaces/akhaliq/DPT-Large). | |
| * [Apr 2021] Released MiDaS v3.0: | |
| - New models based on [Dense Prediction Transformers](https://arxiv.org/abs/2103.13413) are on average [21% more accurate](#Accuracy) than MiDaS v2.1 | |
| - Additional models can be found [here](https://github.com/isl-org/DPT) | |
| * [Nov 2020] Released MiDaS v2.1: | |
| - New model that was trained on 10 datasets and is on average about [10% more accurate](#Accuracy) than [MiDaS v2.0](https://github.com/isl-org/MiDaS/releases/tag/v2) | |
| - New light-weight model that achieves [real-time performance](https://github.com/isl-org/MiDaS/tree/master/mobile) on mobile platforms. | |
| - Sample applications for [iOS](https://github.com/isl-org/MiDaS/tree/master/mobile/ios) and [Android](https://github.com/isl-org/MiDaS/tree/master/mobile/android) | |
| - [ROS package](https://github.com/isl-org/MiDaS/tree/master/ros) for easy deployment on robots | |
| * [Jul 2020] Added TensorFlow and ONNX code. Added [online demo](http://35.202.76.57/). | |
| * [Dec 2019] Released new version of MiDaS - the new model is significantly more accurate and robust | |
| * [Jul 2019] Initial release of MiDaS ([Link](https://github.com/isl-org/MiDaS/releases/tag/v1)) | |
| ### Citation | |
| Please cite our paper if you use this code or any of the models: | |
| ``` | |
| @ARTICLE {Ranftl2022, | |
| author = "Ren\'{e} Ranftl and Katrin Lasinger and David Hafner and Konrad Schindler and Vladlen Koltun", | |
| title = "Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-Shot Cross-Dataset Transfer", | |
| journal = "IEEE Transactions on Pattern Analysis and Machine Intelligence", | |
| year = "2022", | |
| volume = "44", | |
| number = "3" | |
| } | |
| ``` | |
| If you use a DPT-based model, please also cite: | |
| ``` | |
| @article{Ranftl2021, | |
| author = {Ren\'{e} Ranftl and Alexey Bochkovskiy and Vladlen Koltun}, | |
| title = {Vision Transformers for Dense Prediction}, | |
| journal = {ICCV}, | |
| year = {2021}, | |
| } | |
| ``` | |
| Please cite the technical report for MiDaS 3.1 models: | |
| ``` | |
| @article{birkl2023midas, | |
| title={MiDaS v3.1 -- A Model Zoo for Robust Monocular Relative Depth Estimation}, | |
| author={Reiner Birkl and Diana Wofk and Matthias M{\"u}ller}, | |
| journal={arXiv preprint arXiv:2307.14460}, | |
| year={2023} | |
| } | |
| ``` | |
| For ZoeDepth, please use | |
| ``` | |
| @article{bhat2023zoedepth, | |
| title={Zoedepth: Zero-shot transfer by combining relative and metric depth}, | |
| author={Bhat, Shariq Farooq and Birkl, Reiner and Wofk, Diana and Wonka, Peter and M{\"u}ller, Matthias}, | |
| journal={arXiv preprint arXiv:2302.12288}, | |
| year={2023} | |
| } | |
| ``` | |
| and for LDM3D | |
| ``` | |
| @article{stan2023ldm3d, | |
| title={LDM3D: Latent Diffusion Model for 3D}, | |
| author={Stan, Gabriela Ben Melech and Wofk, Diana and Fox, Scottie and Redden, Alex and Saxton, Will and Yu, Jean and Aflalo, Estelle and Tseng, Shao-Yen and Nonato, Fabio and Muller, Matthias and others}, | |
| journal={arXiv preprint arXiv:2305.10853}, | |
| year={2023} | |
| } | |
| ``` | |
| ### Acknowledgements | |
| Our work builds on and uses code from [timm](https://github.com/rwightman/pytorch-image-models) and [Next-ViT](https://github.com/bytedance/Next-ViT). | |
| We'd like to thank the authors for making these libraries available. | |
| ### License | |
| MIT License | |