---
datasets:
- laion/laion400m
language:
- en
pipeline_tag: text-to-image
---



# LDM3D model

The LDM3D model was proposed in "LDM3D: Latent Diffusion Model for 3D" by Gabriela Ben Melech Stan, Diana Wofk, Scottie Fox, Alex Redden, Will Saxton, Jean Yu, Estelle Aflalo, Shao-Yen Tseng, Fabio Nonato, Matthias Muller, Vasudev Lal. 
It was introduced in [this paper](https://arxiv.org/abs/2305.10853.pdf) 

LDM3D got accepted to [CVPRW'23](https://cvpr2023.thecvf.com/).

## Model description

The abstract from the paper is the following:
This research paper proposes a Latent Diffusion Model for 3D (LDM3D) that generates both image and depth map data from a given text prompt, allowing users to generate RGBD images from text prompts. The LDM3D model is fine-tuned on a dataset of tuples containing an RGB image, depth map and caption, and validated through extensive experiments. We also develop an application called DepthFusion, which uses the generated RGB images and depth maps to create immersive and interactive 360-degree-view experiences using TouchDesigner. This technology has the potential to transform a wide range of industries, from entertainment and gaming to architecture and design. Overall, this paper presents a significant contribution to the field of generative AI and computer vision, and showcases the potential of LDM3D and DepthFusion to revolutionize content creation and digital experiences

## Intended uses & limitations(TODO)

You can use the model to generate RGB and depth images from text prompt.
A short video summarizing the approach can be found at this [URL](https://t.ly/tdi2) and a VR demo can be found [here](https://www.youtube.com/watch?v=3hbUo-hwAs0)

### How to use

Here is how to use this model:

```python

from diffusers import StableDiffusionLDM3DPipeline

pipe_ldm3d = StableDiffusionLDM3DPipeline.from_pretrained("LDM3D/ldm3d-v1")
pipe_ldm3d.to("cuda")

prompt ="A picture of some lemons on a table"
name = "lemons"

rgb_image, depth_image = pipe_ldm3d(prompt).images
rgb_image[0].save(name+"_ldm3d_rgb.jpg")
depth_image[0].save(name+"_ldm3d_depth.png")
```
### Limitations and bias

TODO

## Training data

The LDM3D model was finetuned on a subset of the LAION-400M dataset, a large-scale image-caption dataset that contains over 400 million image-caption pairs


## Training procedure

The fine-tuning process comprises two stages. In the first stage, we train an autoencoder to generate a lower-dimensional, perceptually equivalent data representation. Subsequently, we fine-tune the diffusion model using the frozen autoencoder.

### Preprocessing

TODO

### Compute Infrastructure

All training runs reported in this work are conducted on an Intel AI supercomputing cluster comprising of Intel Xeon processors and Intel Habana Gaudi AI accelerators. The LDM3D model training run is scaled out to 16 accelerators (Gaudis) on the corpus of 9,600 tupples (text caption, RGB image, depth map). The KL-autoencoder used in our LDM3D model was trained on Nvidia A6000 GPUs.

## Evaluation results

Please refer to Table 1 and 2 of the [paper](https://arxiv.org/pdf/2305.10853.pdf)

### BibTeX entry and citation info
```bibtex
@misc{stan2023ldm3d,
      title={LDM3D: Latent Diffusion Model for 3D}, 
      author={Gabriela Ben Melech Stan and Diana Wofk and Scottie Fox and Alex Redden and Will Saxton and Jean Yu and Estelle Aflalo and Shao-Yen Tseng and Fabio Nonato and Matthias Muller and Vasudev Lal},
      year={2023},
      eprint={2305.10853},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}
```