|
--- |
|
license: mit |
|
--- |
|
|
|
# Smoothed Energy Guidance for SDXL |
|
|
|
https://arxiv.org/abs/2408.00760 | https://colab.research.google.com/github/SusungHong/SEG-SDXL/blob/master/sdxl_seg.ipynb |
|
|
|
Identical to https://github.com/SusungHong/SEG-SDXL/blob/8d3b2007a5f0660f9dba110a5e83556395f7535f/pipeline_seg.py |
|
|
|
Implementation of [Smoothed Energy Guidance: Guiding Diffusion Models with Reduced Energy Curvature of Attention](https://arxiv.org/abs/2408.00760) by [Susung Hong](https://susunghong.github.io). |
|
|
|
<img src="./teaser-2.jpg" width="90%"> |
|
|
|
## 🏔️ What is Smoothed Energy Guidance? How does it work? |
|
|
|
Smoothed Energy Guidance (SEG) is a training- and condition-free approach that leverages the energy-based perspective of the self-attention mechanism to improve image generation. |
|
|
|
Key points: |
|
- Does not rely on the guidance scale parameter that causes side effects when the value becomes large |
|
- Allows continuous control of the original and maximally attenuated curvature of the energy landscape behind self-attention |
|
- Introduces a query blurring method, equivalent to blurring the entire attention weights without significant computational cost |
|
|
|
## 🔍 Comparison with other works |
|
|
|
SEG does not severely suffer from side effects such as making the overall image grayish or significantly changing the original structure, while improving generation quality even without prompts. |
|
|
|
Unconditional generation without prompts |
|
<img src="./seg_comparison-2.jpg" width="90%"> |
|
|
|
|
|
|
