|
--- |
|
tags: |
|
- text-to-image |
|
- stable-diffusion |
|
--- |
|
|
|
# Control-LoRA Model Card |
|
|
|
|
|
## Introduction |
|
What's better than ControlNets for SDXL? ControlNet... but, more efficient. |
|
|
|
By introducing low-rank parameter efficient fine tuning, we introduce ***Control-LoRAs***. |
|
|
|
Integrating the strengths of both ControlNet and PEFT, this approach offers a more efficient and compact method to bring model control for a wider variety of consumer GPUs. |
|
|
|
For each model below, you'll find `Rank 256` files (reducing the `~4.7GB` ControlNets to `~738MB`) and experimental, ultra-pruned `Rank 128` files (reducing to `~377MB`). |
|
|
|
### MiDaS and ClipDrop Depth |
|
 |
|
|
|
Depth estimation is an image processing technique that determines the distance of objects in a scene, providing a depth map that highlights variations in proximity. |
|
|
|
In the example above, we compare the depth results of MiDaS dpt_beit_large_512 with ClipDrop Depth for portraits, and their subsequent use in Depth Control-LoRa. |
|
|
|
The Control-LoRA utilizes a grayscale depth map for guided generation and has been trained on a diverse range of image concepts and aspect ratios. |
|
|
|
### Canny Edge |
|
 |
|
Canny Edge Detection is an image processing technique that identifies abrupt changes in intensity to highlight edges in an image. |
|
|
|
This Control-LoRA uses the edges from an image to guide the generation. |
|
|
|
### Photograph and Sketch Colorizer |
|
 |
|
These two Control-LoRAs can be used to colorize images. |
|
|
|
The first is designed to colorize black and white photographs. |
|
|
|
The second is designed to color in sketches input as a white-on-black image (either hand-drawn, or created with a `SoftEdge_PIDI` model). |
|
|
|
### Revision |
|
 |
|
Revision is a novel approach of using images to prompt SDXL. |
|
|
|
It uses pooled CLIP embeddings to produce images conceptually similar to the input. It can be used either in addition, or to replace text prompts. |
|
|
|
Revision also includes a blending function for combining multiple image or text concepts, as either positive or negative prompts. |
