ReNeg: Learning Negative Embedding with Reward Guidance

   

We present ReNeg, a Reward-guided approach that directly learns Negative embeddings through gradient descent. The global negative embeddings learned using ReNeg exhibit strong generalization capabilities and can be seamlessly adaptable to text-to-image and even text-to-video models. Strikingly simple yet highly effective, ReNeg amplifies the visual appeal of outputs from base Stable Diffusion models.

Examples

Using the 🤗's Diffusers library to run ReNeg in a simple and efficient manner.

pip install diffusers transformers accelerate
git clone https://github.com/AMD-AIG-AIMA/ReNeg.git

We provide three negative embeddings, including SD1.4, SD1.5, and SD2.1-base. Running ReNeg with a specific SD version as follows:

import os
from pathlib import Path
import torch
from diffusers import (
    StableDiffusionPipeline,
    DDIMScheduler,
)
from safetensors.torch import load_file

model_path = "stable-diffusion-v1-5"
neg_embeddings_path = "checkpoints/sd1.5_reneg_emb.safetensors"
pipe = StableDiffusionPipeline.from_pretrained(
    model_path,
    safety_checker=None,
)
pipe.scheduler = DDIMScheduler.from_pretrained(
    model_path, subfolder="scheduler"
)
device = "cuda"
pipe.to(device)

neg_embeddings = load_file(neg_embeddings_path)["embedding"].to(device)  # Assuming the key is "embedding"
output = pipe(
    "A girl in a school uniform playing an electric guitar.",
    negative_prompt_embeds=neg_embeddings,
)

image = output.images[0]
# TextToImageModel is the model you want to evaluate
image.save("output.png")

To compare with the inference results using neg_emb, you can perform inference using only positive prompt.

• To perform inference using only the pos_prompt, you need to run inference.py with args.prompt_type = only_pos.

python inference.py --model_path "your_sd1.5_path" --prompt_type "only_pos" --prompt "A girl in a school uniform playing an electric guitar."

Citation

@misc{li2024reneg,
      title={ReNeg: Learning Negative Embedding with Reward Guidance},
      author={Xiaomin Li, Yixuan Liu, Takashi Isobe, Xu Jia, Qinpeng Cui, Dong Zhou, Dong Li, You He, Huchuan Lu, Zhongdao Wang, Emad Barsoum},
      year={2024},
      eprint={2412.19637},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}