## Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer This repository contains code to compute depth from a single image. It accompanies our [paper](https://arxiv.org/abs/1907.01341v3): >Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, Vladlen Koltun and our [preprint](https://arxiv.org/abs/2103.13413): > Vision Transformers for Dense Prediction > René Ranftl, Alexey Bochkovskiy, Vladlen Koltun MiDaS was trained on 10 datasets (ReDWeb, DIML, Movies, MegaDepth, WSVD, TartanAir, HRWSI, ApolloScape, BlendedMVS, IRS) with multi-objective optimization. The original model that was trained on 5 datasets (`MIX 5` in the paper) can be found [here](https://github.com/intel-isl/MiDaS/releases/tag/v2). ### Changelog * [Sep 2021] Integrated to [Huggingface Spaces](https://huggingface.co/spaces) with [Gradio](https://github.com/gradio-app/gradio). See [Gradio Web Demo](https://huggingface.co/spaces/akhaliq/DPT-Large). * [Apr 2021] Released MiDaS v3.0: - New models based on [Dense Prediction Transformers](https://arxiv.org/abs/2103.13413) are on average [21% more accurate](#Accuracy) than MiDaS v2.1 - Additional models can be found [here](https://github.com/intel-isl/DPT) * [Nov 2020] Released MiDaS v2.1: - New model that was trained on 10 datasets and is on average about [10% more accurate](#Accuracy) than [MiDaS v2.0](https://github.com/intel-isl/MiDaS/releases/tag/v2) - New light-weight model that achieves [real-time performance](https://github.com/intel-isl/MiDaS/tree/master/mobile) on mobile platforms. - Sample applications for [iOS](https://github.com/intel-isl/MiDaS/tree/master/mobile/ios) and [Android](https://github.com/intel-isl/MiDaS/tree/master/mobile/android) - [ROS package](https://github.com/intel-isl/MiDaS/tree/master/ros) for easy deployment on robots * [Jul 2020] Added TensorFlow and ONNX code. Added [online demo](http://35.202.76.57/). * [Dec 2019] Released new version of MiDaS - the new model is significantly more accurate and robust * [Jul 2019] Initial release of MiDaS ([Link](https://github.com/intel-isl/MiDaS/releases/tag/v1)) ### Setup 1) Pick one or more models and download corresponding weights to the `weights` folder: - For highest quality: [dpt_large](https://github.com/intel-isl/DPT/releases/download/1_0/dpt_large-midas-2f21e586.pt) - For moderately less quality, but better speed on CPU and slower GPUs: [dpt_hybrid](https://github.com/intel-isl/DPT/releases/download/1_0/dpt_hybrid-midas-501f0c75.pt) - For real-time applications on resource-constrained devices: [midas_v21_small](https://github.com/AlexeyAB/MiDaS/releases/download/midas_dpt/midas_v21_small-70d6b9c8.pt) - Legacy convolutional model: [midas_v21](https://github.com/AlexeyAB/MiDaS/releases/download/midas_dpt/midas_v21-f6b98070.pt) 2) Set up dependencies: ```shell conda install pytorch torchvision opencv pip install timm ``` The code was tested with Python 3.7, PyTorch 1.8.0, OpenCV 4.5.1, and timm 0.4.5. ### Usage 1) Place one or more input images in the folder `input`. 2) Run the model: ```shell python run.py --model_type dpt_large python run.py --model_type dpt_hybrid python run.py --model_type midas_v21_small python run.py --model_type midas_v21 ``` 3) The resulting inverse depth maps are written to the `output` folder. #### via Docker 1) Make sure you have installed Docker and the [NVIDIA Docker runtime](https://github.com/NVIDIA/nvidia-docker/wiki/Installation-\(Native-GPU-Support\)). 2) Build the Docker image: ```shell docker build -t midas . ``` 3) Run inference: ```shell docker run --rm --gpus all -v $PWD/input:/opt/MiDaS/input -v $PWD/output:/opt/MiDaS/output midas ``` This command passes through all of your NVIDIA GPUs to the container, mounts the `input` and `output` directories and then runs the inference. #### via PyTorch Hub The pretrained model is also available on [PyTorch Hub](https://pytorch.org/hub/intelisl_midas_v2/) #### via TensorFlow or ONNX See [README](https://github.com/intel-isl/MiDaS/tree/master/tf) in the `tf` subdirectory. Currently only supports MiDaS v2.1. DPT-based models to be added. #### via Mobile (iOS / Android) See [README](https://github.com/intel-isl/MiDaS/tree/master/mobile) in the `mobile` subdirectory. #### via ROS1 (Robot Operating System) See [README](https://github.com/intel-isl/MiDaS/tree/master/ros) in the `ros` subdirectory. Currently only supports MiDaS v2.1. DPT-based models to be added. ### Accuracy Zero-shot error (the lower - the better) and speed (FPS): | Model | DIW, WHDR | Eth3d, AbsRel | Sintel, AbsRel | Kitti, δ>1.25 | NyuDepthV2, δ>1.25 | TUM, δ>1.25 | Speed, FPS | |---|---|---|---|---|---|---|---| | **Small models:** | | | | | | | iPhone 11 | | MiDaS v2 small | **0.1248** | 0.1550 | **0.3300** | **21.81** | 15.73 | 17.00 | 0.6 | | MiDaS v2.1 small [URL]() | 0.1344 | **0.1344** | 0.3370 | 29.27 | **13.43** | **14.53** | 30 | | | | | | | | | | **Big models:** | | | | | | | GPU RTX 3090 | | MiDaS v2 large [URL](https://github.com/intel-isl/MiDaS/releases/download/v2/model-f46da743.pt) | 0.1246 | 0.1290 | 0.3270 | 23.90 | 9.55 | 14.29 | 51 | | MiDaS v2.1 large [URL](https://github.com/AlexeyAB/MiDaS/releases/download/midas_dpt/midas_v21-f6b98070.pt) | 0.1295 | 0.1155 | 0.3285 | 16.08 | 8.71 | 12.51 | 51 | | MiDaS v3.0 DPT-Hybrid [URL](https://github.com/intel-isl/DPT/releases/download/1_0/dpt_hybrid-midas-501f0c75.pt) | 0.1106 | 0.0934 | 0.2741 | 11.56 | 8.69 | 10.89 | 46 | | MiDaS v3.0 DPT-Large [URL](https://github.com/intel-isl/DPT/releases/download/1_0/dpt_large-midas-2f21e586.pt) | **0.1082** | **0.0888** | **0.2697** | **8.46** | **8.32** | **9.97** | 47 | ### Citation Please cite our paper if you use this code or any of the models: ``` @ARTICLE {Ranftl2022, author = "Ren\'{e} Ranftl and Katrin Lasinger and David Hafner and Konrad Schindler and Vladlen Koltun", title = "Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-Shot Cross-Dataset Transfer", journal = "IEEE Transactions on Pattern Analysis and Machine Intelligence", year = "2022", volume = "44", number = "3" } ``` If you use a DPT-based model, please also cite: ``` @article{Ranftl2021, author = {Ren\'{e} Ranftl and Alexey Bochkovskiy and Vladlen Koltun}, title = {Vision Transformers for Dense Prediction}, journal = {ICCV}, year = {2021}, } ``` ### Acknowledgements Our work builds on and uses code from [timm](https://github.com/rwightman/pytorch-image-models). We'd like to thank the author for making these libraries available. ### License MIT License