Open Preference Dataset for Text-to-Image Generation by the π€ Community
β’
49
Sayak Paul
sayakpaul
AI & ML interests
Diffusion models, representation learning
Recent Activity
new activity
about 16 hours ago
hunyuanvideo-community/HunyuanVideo:Black output
posted
an
update
1 day ago
Commits speak louder than words π€ͺ
* 4 new video models
* Multiple image models, including SANA & Flux Control
* New quantizers -> GGUF & TorchAO
* New training scripts
Enjoy this holiday-special Diffusers release π€
Notes: https://github.com/huggingface/diffusers/releases/tag/v0.32.0
new activity
5 days ago
black-forest-labs/FLUX.1-Fill-dev:Can't use latest diffusers ?
Articles
Organizations
sayakpaul's activity
posted
an
update
1 day ago
Post
1519
Commits speak louder than words π€ͺ
* 4 new video models
* Multiple image models, including SANA & Flux Control
* New quantizers -> GGUF & TorchAO
* New training scripts
Enjoy this holiday-special Diffusers release π€
Notes: https://github.com/huggingface/diffusers/releases/tag/v0.32.0
* 4 new video models
* Multiple image models, including SANA & Flux Control
* New quantizers -> GGUF & TorchAO
* New training scripts
Enjoy this holiday-special Diffusers release π€
Notes: https://github.com/huggingface/diffusers/releases/tag/v0.32.0
posted
an
update
7 days ago
Post
1537
In the past seven days, the Diffusers team has shipped:
1. Two new video models
2. One new image model
3. Two new quantization backends
4. Three new fine-tuning scripts
5. Multiple fixes and library QoL improvements
Coffee on me if someone can guess 1 - 4 correctly.
1. Two new video models
2. One new image model
3. Two new quantization backends
4. Three new fine-tuning scripts
5. Multiple fixes and library QoL improvements
Coffee on me if someone can guess 1 - 4 correctly.
- 1 reply
posted
an
update
15 days ago
Post
2039
Introducing a high-quality open-preference dataset to further this line of research for image generation.
Despite being such an inseparable component for modern image generation, open preference datasets are a rarity!
So, we decided to work on one with the community!
Check it out here:
https://huggingface.co/blog/image-preferences
Despite being such an inseparable component for modern image generation, open preference datasets are a rarity!
So, we decided to work on one with the community!
Check it out here:
https://huggingface.co/blog/image-preferences
- 7 replies
posted
an
update
16 days ago
Post
2095
The Control family of Flux from
@black-forest-labs
should be discussed more!
It enables structural controls like ControlNets while being significantly less expensive to run!
So, we're working on a Control LoRA training script π€
It's still WIP, so go easy:
https://github.com/huggingface/diffusers/pull/10130
It enables structural controls like ControlNets while being significantly less expensive to run!
So, we're working on a Control LoRA training script π€
It's still WIP, so go easy:
https://github.com/huggingface/diffusers/pull/10130
posted
an
update
26 days ago
Post
1466
Let 2024 be the year of video model fine-tunes!
Check it out here:
https://github.com/a-r-r-o-w/cogvideox-factory/tree/main/training/mochi-1
Check it out here:
https://github.com/a-r-r-o-w/cogvideox-factory/tree/main/training/mochi-1
posted
an
update
about 1 month ago
Post
2598
It's been a while we shipped native quantization support in
We currently support
This post is just a reminder of what's possible:
1. Loading a model with a quantization config
2. Saving a model with quantization config
3. Loading a pre-quantized model
4.
5. Training and loading LoRAs into quantized checkpoints
Docs:
https://huggingface.co/docs/diffusers/main/en/quantization/bitsandbytes
diffusers π§¨We currently support
bistandbytes as the official backend but using others like torchao is already very simple. This post is just a reminder of what's possible:
1. Loading a model with a quantization config
2. Saving a model with quantization config
3. Loading a pre-quantized model
4.
enable_model_cpu_offload()5. Training and loading LoRAs into quantized checkpoints
Docs:
https://huggingface.co/docs/diffusers/main/en/quantization/bitsandbytes
- 1 reply
posted
an
update
3 months ago
Post
2752
Did some little experimentation to resize pre-trained LoRAs on Flux. I explored two themes:
* Decrease the rank of a LoRA
* Increase the rank of a LoRA
The first one is helpful in reducing memory requirements if the LoRA is of a high rank, while the second one is merely an experiment. Another implication of this study is in the unification of LoRA ranks when you would like to
Check it out here:
sayakpaul/flux-lora-resizing
* Decrease the rank of a LoRA
* Increase the rank of a LoRA
The first one is helpful in reducing memory requirements if the LoRA is of a high rank, while the second one is merely an experiment. Another implication of this study is in the unification of LoRA ranks when you would like to
torch.compile() them. Check it out here:
sayakpaul/flux-lora-resizing
- 1 reply
reacted to
dn6's
post with β€οΈ
4 months ago
Post
2635
Sharing for anyone using Diffusers
If you have
If you do not have the model repo saved in your cache, then automatically inferring the pipeline config will not work since the reference repo
You can use an alternative SD1.5 repo id to still configure your pipeline.
We're working on resolving the issue ASAP.
from_single_file loading and affected by the Runway SD 1.5 issue.If you have
runwayml/stable-diffusion-v1-5 saved locally in your HF cache then loading single file checkpoints in the following way should still work. from diffusers import StableDiffusionPipeline
pipe = StableDiffusionPipeline.from_single_file("<url or path to single file checkpoint>")If you do not have the model repo saved in your cache, then automatically inferring the pipeline config will not work since the reference repo
runwayml/stable-diffusion-v1-5 doesn't exist anymore. You can use an alternative SD1.5 repo id to still configure your pipeline.
from diffusers import StableDiffusionPipeline
pipe = StableDiffusionPipeline.from_single_file("<url or path to single file checkpoint>", config="Lykon/DreamShaper")We're working on resolving the issue ASAP.
-
- 2 replies
posted
an
update
4 months ago
Post
2945
Here is a hackable and minimal implementation showing how to perform distributed text-to-image generation with Diffusers and Accelerate.
Full snippet is here: https://gist.github.com/sayakpaul/cfaebd221820d7b43fae638b4dfa01ba
With @JW17
Full snippet is here: https://gist.github.com/sayakpaul/cfaebd221820d7b43fae638b4dfa01ba
With @JW17
posted
an
update
5 months ago
Post
4478
Flux.1-Dev like images but in fewer steps.
Merging code (very simple), inference code, merged params: sayakpaul/FLUX.1-merged
Enjoy the Monday π€
Merging code (very simple), inference code, merged params: sayakpaul/FLUX.1-merged
Enjoy the Monday π€
- 4 replies
posted
an
update
5 months ago
Post
3793
With larger and larger diffusion transformers coming up, it's becoming increasingly important to have some good quantization tools for them.
We present our findings from a series of experiments on quantizing different diffusion pipelines based on diffusion transformers.
We demonstrate excellent memory savings with a bit of sacrifice on inference latency which is expected to improve in the coming days.
Diffusers π€ Quanto β€οΈ
This was a juicy collaboration between @dacorvo and myself.
Check out the post to learn all about it
https://huggingface.co/blog/quanto-diffusers
We present our findings from a series of experiments on quantizing different diffusion pipelines based on diffusion transformers.
We demonstrate excellent memory savings with a bit of sacrifice on inference latency which is expected to improve in the coming days.
Diffusers π€ Quanto β€οΈ
This was a juicy collaboration between @dacorvo and myself.
Check out the post to learn all about it
https://huggingface.co/blog/quanto-diffusers
- 3 replies
reacted to
alex-abb's
post with π₯
6 months ago
Post
4813
Hi everyone!
I'm Alex, I'm 16, I've been an internship at Hugging Face for a little over a week and I've already learned a lot about using and prompting LLM models. With @victor as tutor I've just finished a space that analyzes your feelings by prompting an LLM chat model. The aim is to extend it so that it can categorize hugging face posts.
alex-abb/LLM_Feeling_Analyzer
I'm Alex, I'm 16, I've been an internship at Hugging Face for a little over a week and I've already learned a lot about using and prompting LLM models. With @victor as tutor I've just finished a space that analyzes your feelings by prompting an LLM chat model. The aim is to extend it so that it can categorize hugging face posts.
alex-abb/LLM_Feeling_Analyzer
- 4 replies
posted
an
update
6 months ago
Post
2207
Were you aware that we have a dedicated guide on different prompting mechanisms to improve the image generation quality? π§¨
Takes you through simple prompt engineering, prompt weighting, prompt enhancement using GPT-2, and more.
Check out the guide here π¦―
https://huggingface.co/docs/diffusers/main/en/using-diffusers/weighted_prompts
Takes you through simple prompt engineering, prompt weighting, prompt enhancement using GPT-2, and more.
Check out the guide here π¦―
https://huggingface.co/docs/diffusers/main/en/using-diffusers/weighted_prompts
-
- 1 reply
posted
an
update
6 months ago
Post
3131
What is your favorite part of our Diffusers integration of Stable Diffusion 3?
My personal favorite is the ability to run it on a variety of different GPUs with minimal code changes.
Learn more about them here:
https://huggingface.co/blog/sd3
My personal favorite is the ability to run it on a variety of different GPUs with minimal code changes.
Learn more about them here:
https://huggingface.co/blog/sd3
I think you definitely missed out on another big release:
https://huggingface.co/posts/sayakpaul/557387472547604
posted
an
update
7 months ago
Post
1866
𧨠Diffusers 0.28.0 is out π₯
It features the first non-generative pipeline of the library -- Marigold π₯
Marigold shines at performing Depth Estimation and Surface Normal Estimation. It was contributed by @toshas , one of the authors of Marigold.
This release also features a massive refactor (led by @DN6 ) of the
Check out the release notes here:
https://github.com/huggingface/diffusers/releases/tag/v0.28.0
It features the first non-generative pipeline of the library -- Marigold π₯
Marigold shines at performing Depth Estimation and Surface Normal Estimation. It was contributed by @toshas , one of the authors of Marigold.
This release also features a massive refactor (led by @DN6 ) of the
from_single_file() method, highlighting our efforts for making our library more amenable to community features π€Check out the release notes here:
https://github.com/huggingface/diffusers/releases/tag/v0.28.0
reacted to
lunarflu's
post with β€οΈ
7 months ago
Post
1934
cooking up something....anyone interested in a daily activity tracker for HF?
- 12 replies
posted
an
update
8 months ago
Post
2022
Custom pipelines and components in Diffusers πΈ
Wanted to use customized pipelines and other components (schedulers, unets, text encoders, etc.) in Diffusers?
Found it inflexible?
Since the first dawn on earth, we have supported loading custom pipelines via a
These pipelines are inference-only, i.e., the assumption is that we're leveraging an existing checkpoint (e.g., runwayml/stable-diffusion-v1-5) and ONLY modifying the pipeline implementation.
We have many cool pipelines, implemented that way. They all share the same benefits available to a
Check them here:
https://github.com/huggingface/diffusers/tree/main/examples/community
Then we might have a requirement of everything customized i.e., custom components along with a custom pipeline. Sure, that's all possible.
All you have to do is keep the implementations of those custom components on the Hub repository you're loading your pipeline checkpoint from.
SDXL Japanese was implemented like this π₯
stabilityai/japanese-stable-diffusion-xl
Full guide is available here β¬οΈ
https://huggingface.co/docs/diffusers/main/en/using-diffusers/custom_pipeline_overview
And, of course, these share all the benefits that come with
Wanted to use customized pipelines and other components (schedulers, unets, text encoders, etc.) in Diffusers?
Found it inflexible?
Since the first dawn on earth, we have supported loading custom pipelines via a
custom_pipeline argument πThese pipelines are inference-only, i.e., the assumption is that we're leveraging an existing checkpoint (e.g., runwayml/stable-diffusion-v1-5) and ONLY modifying the pipeline implementation.
We have many cool pipelines, implemented that way. They all share the same benefits available to a
DiffusionPipeline, no compromise there π€Check them here:
https://github.com/huggingface/diffusers/tree/main/examples/community
Then we might have a requirement of everything customized i.e., custom components along with a custom pipeline. Sure, that's all possible.
All you have to do is keep the implementations of those custom components on the Hub repository you're loading your pipeline checkpoint from.
SDXL Japanese was implemented like this π₯
stabilityai/japanese-stable-diffusion-xl
Full guide is available here β¬οΈ
https://huggingface.co/docs/diffusers/main/en/using-diffusers/custom_pipeline_overview
And, of course, these share all the benefits that come with
DiffusionPipeline.
reacted to
chansung's
post with β€οΈπ€
8 months ago
Post
4398
π» Smoothing the Transition from Service LLM to Local LLM
Imagine your go-to LLM service is down, or you need to use it offline β yikes! This project is all about having that "Plan B" ready to go. Here's LLaMA Duo I've been building with @sayakpaul :
β¨ Fine-tune a smaller LLM: We used Hugging Face's alignment-handbook to teach a smaller-sized LLM to mimic my favorite large language model. Think of it as that super-smart AI assistant getting a capable understudy.
π€ Batch Inference: Let's get that fine-tuned LLM working! My scripts generate lots of text like a champ, and we've made sure things run smoothly even with bigger workloads.
π§ Evaluation: How well is my small LLM doing? We integrated with the Gemini API to use it as an expert judge β it compares my model's work to the original. Talk about a tough critic!
πͺ Synthetic Data Generation: Need to boost that model's performance? Using Gemini's feedback, we can create even more training data, custom-made to make the LLM better.
𧱠Building Blocks: This isn't just a one-time thing β it's a toolkit for all kinds of LLMOps work. Want to change your evaluation metrics? Bring in models trained differently? Absolutely, let's make it happen.
Why this project is awesome:
πͺ Reliability: Keep things running no matter what happens to your main LLM source.
π Privacy: Process sensitive information on your own terms.
πΊοΈ Offline capable: No internet connection? No problem!
π°οΈ Version Control: Lock in your favorite LLM's behavior, even if the service model changes.
We'm excited to share the code on GitHub. Curious to see what you all think! ππ» https://github.com/deep-diver/llamaduo
Imagine your go-to LLM service is down, or you need to use it offline β yikes! This project is all about having that "Plan B" ready to go. Here's LLaMA Duo I've been building with @sayakpaul :
β¨ Fine-tune a smaller LLM: We used Hugging Face's alignment-handbook to teach a smaller-sized LLM to mimic my favorite large language model. Think of it as that super-smart AI assistant getting a capable understudy.
π€ Batch Inference: Let's get that fine-tuned LLM working! My scripts generate lots of text like a champ, and we've made sure things run smoothly even with bigger workloads.
π§ Evaluation: How well is my small LLM doing? We integrated with the Gemini API to use it as an expert judge β it compares my model's work to the original. Talk about a tough critic!
πͺ Synthetic Data Generation: Need to boost that model's performance? Using Gemini's feedback, we can create even more training data, custom-made to make the LLM better.
𧱠Building Blocks: This isn't just a one-time thing β it's a toolkit for all kinds of LLMOps work. Want to change your evaluation metrics? Bring in models trained differently? Absolutely, let's make it happen.
Why this project is awesome:
πͺ Reliability: Keep things running no matter what happens to your main LLM source.
π Privacy: Process sensitive information on your own terms.
πΊοΈ Offline capable: No internet connection? No problem!
π°οΈ Version Control: Lock in your favorite LLM's behavior, even if the service model changes.
We'm excited to share the code on GitHub. Curious to see what you all think! ππ» https://github.com/deep-diver/llamaduo