|
--- |
|
license: creativeml-openrail-m |
|
tags: |
|
- computer vision |
|
- stable-diffusion |
|
- stable-diffusion-2-1 |
|
- photography |
|
- photoreal |
|
--- |
|
|
|
# Deprecation notice |
|
|
|
This model was a research project that is deprecated in favour of ptx0/pseudo-flex-base |
|
|
|
# Capabilities |
|
|
|
This model is capable of producing photorealistic images of people. |
|
|
|
It retains much of the base 2.1-v model knowledge, as its text encoder is minimally tuned. |
|
|
|
# Limitations |
|
|
|
This model does not produce perfect results every time. |
|
|
|
This model cannot reproduce most real people. Instead, it makes "Derp-a-Like" equivalents to real people, which I prefer. |
|
|
|
This model is not great at abstract imagery or digital art, though it certainly can produce a variety of amazing art styles. |
|
|
|
# Dataset |
|
|
|
* cushman (8000 kodachrome slides from 1939 to 1969) |
|
* midjourney v5.1-filtered (about 22,000 upscaled v5.1 images) |
|
* national geographic (about 3-4,000 >1024x768 images of animals, wildlife, landscapes, history) |
|
* a small dataset of stock images of people vaping / smoking |
|
|
|
# Training parameters |
|
|
|
* polynomial learning rate scheduler shared between TE and Unet starting at 4e-8 and decaying to 1e-8 |
|
* batch size 15, gradient accumulations 10 => effective BS=150 |
|
* target is 30,000 steps but will likely stop sooner |
|
* terminal SNR enforced betas |
|
|
|
# Training goals |
|
|
|
* explore the effects of terminal SNR scheduling |
|
* improve faces, especially "at a distance" |
|
* improve composition, eg. completeness of resulting image |
|
* improve prompt comprehension, eg. "do what i want, even if it is weird" |
|
* retain / introduce a slightly colourful flavour due to the midjourney data |
|
* enhance understanding of the past, through the Cushman collection |
|
* retain the ability to produce natural landscapes and animals via National Geographic |
|
|
|
# Observations |
|
|
|
* at 1650 steps, we still haven't cracked the code on faces. |
|
* at 250 steps, we had amazing photoreal Mars landscapes that have carried forward mostly to 1650 steps |
|
* lighting and composition are at their best |
|
|
|
# Future work |
|
|
|
This model inspired the search for a solution to the proliferation issue that led me to ttj/flex-diffusion-2-1, which led to the creation of ptx0/pseudo-flex-base, another photoreal model with multiple aspect support. |
|
|
|
This model was trained **purely** on 768x768 square images, which were randomly resized and cropped. It can produce some higher resolution landscapes, but it cannot reliably do higher resolution subjects without deformities. |
