LibreVAE
LibreVAE is a Variational Autoencoder designed to serve as a component for future generative modelling projects. It has 8 latent channels, and reduces images dimensionally by a factor of 8. It was trained using HuggingFace Diffusers, and can be loaded with the AutoencoderKL
class.
Example Usage
from diffusers import AutoencoderKL
from PIL import Image
from torchvision import transforms
transform_image = transforms.Compose([
transforms.Lambda(lambda x: x.convert("RGB")),
transforms.ToTensor(),
transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))
])
untransform_image = transforms.Compose([
transforms.Normalize((-1, -1, -1), (2, 2, 2)),
transforms.ToPILImage()
])
model = AutoencoderKL.from_pretrained("scrumptious/librevae-f8-d8").to("cuda").eval()
model.requires_grad_(False)
# For a 512x512 image, image_tensor will be (1, 3, 512, 512)
image_tensor = transform_image(Image.open("sample.png")).unsqueeze(0).to("cuda")
# latent will be (1, 8, 64, 64)
# we multiply it by the scaling factor so it has an approximate mean of 0 and variance of 1
latent = model.encode(image_tensor).latent_dist.sample() * model.config.scaling_factor
# output will be (1, 3, 512, 512)
output = model.decode(latent / model.config.scaling_factor).sample
output_image = untransform_image(output.squeeze(0).clamp(-1, 1).cpu())
output_image.save('sample_decoded.png')
Training Details
Training Datasets
LibreVAE was trained on the e621-2024 and danbooru2023 datasets, both of which are large, curated collections of artwork. While the model was trained primarily on artwork, our testing showed that it was capable of working with other types of images.
Dataset Preprocessing
We applied a modified version of NovelAI's Aspect Ratio Bucketing to the images, where we dynamically selected aspect ratio buckets using K-Means over our training dataset instead of predetermining them based on fixed sizes. We then set sizes for these buckets to be around 256x256.
Loss Function
The loss function for this model was MSE_lab + (0.5 * MSE_rgb) + (0.1 * LPIPS) + (1e-4 * KL), where MSE_lab was the mean squared error calculated in CIELAB color space, MSE_rgb was the mean squared error calculated in RGB color space, LPIPS was the LPIPS loss, and KL was the KL divergence.
Other Details
- Precision: BF16 mixed precision
- Learning Rate: 1e-4 with a 50% decay per epoch
- Epochs: 2
- Optimizer: AdamW
- Batch Size: 2 (per-GPU batch) * 2 (GPUs) * 128 (gradient accumulation steps) = 512
Validation Performance
The model achieved the following scores in its final validation run.
- MSE in CIELAB space: 0.002154
- MSE in RGB space: 0.0062
- LPIPS: 0.0555
Uses
LibreVAE is intended to be used by researchers or developers as a component for generative models, such as text-to-image models. The developers don't forsee any direct uses that would not be better served by an existing image compression solution.
License
The weights for LibreVAE are released under the CC0 1.0 license.
Citation
Under the CC0 1.0 license, you are not required to provide any attribution when using or redistributing LibreVAE. If you use LibreVAE in your research or projects and would like to provide attribution, you can cite it as:
@misc{LibreVAE2024,
title={LibreVAE},
author={Scrumptious AI Labs},
year={2024},
note={https://huggingface.co/scrumptious/librevae-f8-d8}
}
Acknowledgments
Special thanks to the contributors of the e621-2024 and danbooru2023 datasets, the HuggingFace Diffusers team, and the PyTorch Lightning team.
- Downloads last month
- 9