| Quantization made by Richard Erkhov. | |
| [Github](https://github.com/RichardErkhov) | |
| [Discord](https://discord.gg/pvy7H8DZMG) | |
| [Request more models](https://github.com/RichardErkhov/quant_request) | |
| bloom-560m-RLHF-SD2-prompter - bnb 4bits | |
| - Model creator: https://huggingface.co/crumb/ | |
| - Original model: https://huggingface.co/crumb/bloom-560m-RLHF-SD2-prompter/ | |
| Original model description: | |
| --- | |
| license: bigscience-bloom-rail-1.0 | |
| tags: | |
| - stable-diffusion | |
| - diffusion | |
| model-index: | |
| - name: bloom-560m-RLHF-SD2-prompter | |
| results: [] | |
| datasets: | |
| - Gustavosta/Stable-Diffusion-Prompts | |
| widget: | |
| - text: "<s>Prompt: " | |
| inference: | |
| parameters: | |
| eos_token_id: 2 | |
| max_length: 128 | |
| do_sample: true | |
| --- | |
| # BLOOM-560m RLHF SD2 Prompter | |
| **COLAB DEMO INCLUDING STABLE DIFFUSION: https://colab.research.google.com/github/aicrumb/doohickey/blob/main/rlhf_prompt_tuner.ipynb** | |
| Using RLHF (Reinforcement Learning from Human Feedback) to finetune [mrm8488/bloom-560m-finetuned-sd-prompts](https://hf.co/mrm8488/bloom-560m-finetuned-sd-prompts) further for SD2.0 | |
| ``` | |
| batch_size = 16 | |
| learning_rate = 0.001 # this is why I didn't have to spend _forever_ on it | |
| ``` | |
| Generate extension with "\<s>Prompt: " and whatever your normal prompt is. | |
| I did this myself. I sat down and just ranked images for so long. It's gone through a couple iterations. Only the biases and layernorm weights were trained. The commit messages are a MESS. **First iteration of this project** | |
| donate so i can do this on real hardware : https://github.com/aicrumb/aicrumb/blob/main/README.md | |
| ## Example usage | |
| ```python | |
| # Install libraries needed to run the models | |
| !pip install transformers diffusers accelerate -qq | |
| # Import the libraries | |
| from diffusers import StableDiffusionPipeline, EulerDiscreteScheduler | |
| from transformers import pipeline | |
| import torch | |
| # This is the model that the transformer was finetuned to generate prompts for | |
| model_id = "stabilityai/stable-diffusion-2-base" | |
| # Use the Euler scheduler here | |
| scheduler = EulerDiscreteScheduler.from_pretrained(model_id, subfolder="scheduler") | |
| pipe = StableDiffusionPipeline.from_pretrained(model_id, scheduler=scheduler, revision="fp16", torch_dtype=torch.float16) | |
| pipe = pipe.to("cuda") | |
| # Load the transformer model | |
| prompt_pipe = pipeline("text-generation", model="crumb/bloom-560m-RLHF-SD2-prompter") | |
| prompt = "cool landscape" | |
| # Auto-complete prompt | |
| prompt = "<s>Prompt: " + prompt + "," | |
| extended_prompt = prompt_pipe(prompt, do_sample=True, max_length=42)[0]['generated_text'] | |
| extended_prompt = extended_prompt[10:] | |
| print("Prompt is now: ", extended_prompt) | |
| # Generate image | |
| image = pipe(extended_prompt).images[0] | |
| image.save("output.png") | |
| image | |
| ``` | |
| *Prompt is now: cool landscape, concept art* | |
|  | |
| *Prompt is now: cool landscape, concept art, sharp focus, digital painting* | |
|  | |
| short additions, they work though I guess (results vary) | |
| It's also very good at generating prompts by itself, with just the "Prompt:" prompt. | |
| *\<s>Prompt: 1 0 th century, highly detailed, concept art, cinematic lighting, unreal engine, trending on artstation, artstation hd, artstation hq, very very detailed* | |
|  | |
| Further testing to be done in this area (automated training with aesthetic predicting models, larger data collection about prompt scores, better training in general) | |
| Also, enjoy this graphic I had to make myself because I kept being indecisive of the reward methodology  | |