metadata

license: apache-2.0
tags:
  - text-to-image
  - ultra-realistic
  - text-to-image
  - stable-diffusion
  - distilled-model
  - knowledge-distillation
pinned: true
datasets:
  - zzliang/GRIT
  - wanng/midjourney-v5-202304-clean
library_name: diffusers

SSD-Tiny Model Card

🔥🔥Join our Discord to give feedback on our models and get early access🔥🔥

Demo

Try out the SSD-Tiny model at Segmind SSD-Tiny for ⚡ fastest inference. You can also explore it on 🤗 Spaces

Model Description

The SSD-Tiny Model is a distilled version of the Stable Diffusion XL (SDXL), offering a remarkable 70% reduction in size and an impressive 80% speedup while retaining high-quality text-to-image generation capabilities. Trained on diverse datasets, including Grit and Midjourney scrape data, it excels at creating a wide range of visual content based on textual prompts.

Employing a knowledge distillation strategy, SSD-Tiny leverages the teachings of several expert models, including SDXL, ZavyChromaXL, and JuggernautXL, to combine their strengths and produce compelling visual outputs.

Special thanks to the HF team 🤗, especially Sayak, Patrick, and Poli, for their collaboration and guidance on this work.

Image Comparison (SDXL-1.0 vs SSD-Tiny)

Usage:

This model can be used via the 🧨 Diffusers library.

Make sure to install diffusers by running

pip install diffusers

In addition, please install transformers, safetensors, and accelerate:

pip install transformers accelerate safetensors

To use the model, you can run the following:

from diffusers import StableDiffusionXLPipeline
import torch

pipe = StableDiffusionXLPipeline.from_pretrained("segmind/SSD-Tiny", torch_dtype=torch.float16, use_safetensors=True, variant="fp16")
pipe.to("cuda")
# if using torch < 2.0
# pipe.enable_xformers_memory_efficient_attention()
prompt = "An astronaut riding a green horse"  # Your prompt here
neg_prompt = "ugly, blurry, poor quality"  # Negative prompt here
image = pipe(prompt=prompt, negative_prompt=neg_prompt).images[0]

Please do use negative prompting and a CFG around 9.0 for the best quality!

Model Description

Developed by: Segmind
Developers: Yatharth Gupta and Vishnu Jaddipal.
Model type: Diffusion-based text-to-image generative model
License: Apache 2.0
Distilled From: stabilityai/stable-diffusion-xl-base-1.0

Key Features

Text-to-Image Generation: The SSD-Tiny model excels at generating images from text prompts, enabling a wide range of creative applications.
Distilled for Speed: Designed for efficiency, this model offers an impressive 80% speedup, making it suitable for real-time applications and scenarios where rapid image generation is essential.
Diverse Training Data: Trained on diverse datasets, the model can handle a variety of textual prompts and generate corresponding images effectively.
Knowledge Distillation: By distilling knowledge from multiple expert models, the SSD-Tiny Model combines their strengths and minimizes their limitations, resulting in improved performance.

Model Architecture

The SSD-Tiny Model is a compact version with a remarkable 70% reduction in size compared to the Base SDXL Model.

Training Info

These are the key hyperparameters used during training:

Steps: 540,000
Learning rate: 1e-5
Batch size: 16
Gradient accumulation steps: 8
Image resolution: 1024
Mixed-precision: fp16

Speed Comparison

SSD-Tiny has demonstrated an impressive 80% speedup compared to the Base SDXL Model. Below is a comparison on an A100 80GB.

Below are the speed-up metrics on an RTX 4090 GPU.

Model Sources

For research and development purposes, the SSD-Tiny Model can be accessed via the Segmind AI platform. For more information and access details, please visit Segmind.

Uses

Direct Use

The SSD-Tiny Model is suitable for research and practical applications in various domains, including:

Art and Design: It can be used to generate artworks, designs, and other creative content, providing inspiration and enhancing the creative process.
Education: The model can be applied in educational tools to create visual content for teaching and learning purposes.
Research: Researchers can use the model to explore generative models, evaluate its performance, and push the boundaries of text-to-image generation.
Safe Content Generation: It offers a safe and controlled way to generate content, reducing the risk of harmful or inappropriate outputs.
Bias and Limitation Analysis: Researchers and developers can use the model to probe its limitations and biases, contributing to a better understanding of generative models' behavior.

Downstream Use

The SSD-Tiny Model can also be used directly with the 🧨 Diffusers library training scripts for further training, including:

LoRA:

export MODEL_NAME="segmind/SSD-Tiny"
export VAE_NAME="madebyollin/sdxl-vae-fp16-fix"
export DATASET_NAME="lambdalabs/pokemon-blip-captions"

accelerate launch train_text_to_image_lora_sdxl.py \
  --pretrained_model_name_or_path=$MODEL_NAME \
  --pretrained_vae_model_name_or_path=$VAE_NAME \
  --dataset_name=$DATASET_NAME --caption_column="text" \
  --resolution=1024 --random_flip \
  --train_batch_size=1 \
  --num_train_epochs=2 --checkpointing_steps=500 \
  --learning_rate=1e-04 --lr_scheduler="constant" --lr_warmup_steps=0 \
  --mixed_precision="fp16" \
  --seed=42 \
  --output_dir="sd-pokemon-model-lora-tiny" \
  --validation_prompt="cute dragon creature" --report_to="wandb" \
  --push_to_hub

Fine-Tune:

export MODEL_NAME="segmind/SSD-Tiny"
export VAE_NAME="madebyollin/sdxl-vae-fp16-fix"
export DATASET_NAME="lambdalabs/pokemon-blip-captions"

accelerate launch train_text_to_image_sdxl.py \
  --pretrained_model_name_or_path=$MODEL_NAME \
  --pretrained_vae_model_name_or_path=$VAE_NAME \
  --dataset_name=$DATASET_NAME \
  --enable_xformers_memory_efficient_attention \
  --resolution=1024 --center_crop --random_flip \
  --proportion_empty_prompts=0.2 \
  --train_batch_size=1 \
  --gradient_accumulation_steps=4 --gradient_checkpointing \
  --max_train_steps=10000 \
  --use_8bit_adam \
  --learning_rate=1e-06 --lr_scheduler="constant" --lr_warmup_steps=0 \
  --mixed_precision="fp16" \
  --report_to="wandb" \
  --validation_prompt="a cute Sundar Pichai creature" --validation_epochs 5 \
  --checkpointing_steps=5000 \
  --output_dir="ssd-pokemon-model-tiny" \
  --push_to_hub

Dreambooth LoRA:

export MODEL_NAME="segmind/SSD-Tiny"
export INSTANCE_DIR="dog"
export OUTPUT_DIR="lora-trained-tiny"
export VAE_PATH="madebyollin/sdxl-vae-fp16-fix"

accelerate launch train_dreambooth_lora_sdxl.py \
  --pretrained_model_name_or_path=$MODEL_NAME  \
  --instance_data_dir=$INSTANCE_DIR \
  --pretrained_vae_model_name_or_path=$VAE_PATH \
  --output_dir=$OUTPUT_DIR \
  --mixed_precision="fp16" \
  --instance_prompt="a photo of sks dog" \
  --resolution=1024 \
  --train_batch_size=1 \
  --gradient_accumulation_steps=4 \
  --learning_rate=1e-5 \
  --report_to="wandb" \
  --lr_scheduler="constant" \
  --lr_warmup_steps=0 \
  --max_train_steps=500 \
  --validation_prompt="A photo of sks dog in a bucket" \
  --validation_epochs=25 \
  --seed="0" \
  --push_to_hub

Out-of-Scope Use

The SSD-Tiny Model is not suitable for creating factual or accurate representations of people, events, or real-world information. It is not intended for tasks requiring high precision and accuracy.

Limitations and Bias

Limitations & Bias: The SSD-Tiny Model faces challenges in achieving absolute photorealism, especially in human depictions. While it may encounter difficulties in incorporating clear text and maintaining the fidelity of complex compositions due to its autoencoding approach, these challenges present opportunities for future enhancements. Importantly, the model's exposure to a diverse dataset, though not a cure-all for ingrained societal and digital biases, represents a foundational step toward more equitable technology. Users are encouraged to interact with this pioneering tool with an understanding of its current limitations, fostering an environment of conscious engagement and anticipation for its continued evolution.