Rename evoukiyoe_v1.py to evo_ukiyoe_v1.py

README.md

@@ -9,13 +9,21 @@ tags:
 ---
 # 🐟 Evo-Ukiyoe-v1
 
-🤗 [Models](https://huggingface.co/SakanaAI/Evo-Ukiyoe-v1/) | 📝 [Blog](https://sakana.ai/evo-ukiyoe/) | 🐦 [Twitter](https://twitter.com/SakanaAILabs)
+🤗 [Models](https://huggingface.co/SakanaAI) | 📝 [Blog](TODO) | 🐦 [Twitter](https://twitter.com/SakanaAILabs)
 
 
-**Evo-Ukiyoe-v1** is an experimental education-purpose Japanese woodblock print Ukiyoe style image generation model. The model was train based on Sakana AI's [Evo-SDXL-JP](https://huggingface.co/SakanaAI/EvoSDXL-JP-v1).
-All the dataset used to train Evo-Ukiyoe comes from Ukiyoe images belonged to [Ritsumeikan University, Art Research Center](https://www.arc.ritsumei.ac.jp/).
+**EvoSDXL-JP-v1** is an experimental education-purpose Japanese SDXL Lightning. 
+This model was created using the Evolutionary Model Merge method. 
+Please refer to our [report](https://arxiv.org/abs/2403.13187) and [blog](https://sakana.ai/evosdxl-jp/) for more details. 
+This model was produced by merging the following models. 
+We are grateful to the developers of the source models.
+
+- [SDXL](https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0)
+- [Juggernaut-XL-v9](https://huggingface.co/RunDiffusion/Juggernaut-XL-v9)
+- [SDXL-DPO](https://huggingface.co/mhdang/dpo-sdxl-text2image-v1)
+- [JSDXL](https://huggingface.co/stabilityai/japanese-stable-diffusion-xl)
+- [SDXL-Lightning](https://huggingface.co/ByteDance/SDXL-Lightning)
 
-Please refer to our [blog](https://sakana.ai/evo-ukiyoe/) for more details. 
 
 ## Usage
 
@@ -29,28 +37,18 @@ Use the code below to get started with the model.
    ```
    git clone https://huggingface.co/SakanaAI/Evo-Ukiyoe-v1
    ```
-2. Install git-lfs if you don't have it yet.
-   ```
-   sudo apt install git-lfs
-   git lfs install
-   ```
-3. Create conda env
-   ```
-   conda create -n evo-ukiyoe python=3.11
-   conda activate evo-ukiyoe
-   ```
-4. Install packages
+2. Install packages
    ```
    cd Evo-Ukiyoe-v1
    pip install -r requirements.txt
    ```
-5. Run
+3. Run
    ```python
-   from evo_ukiyoe_v1 import load_evo_ukiyoe
+   from evoukiyoe_v1 import load_evoukiyoe
     
-   prompt = "着物を着ている猫が庭でお茶を飲んでいる。"
-   pipe = load_evo_ukiyoe(device="cuda")
-   images = pipe(prompt + "輻の浮世絵。超詳細。", negative_prompt='', guidance_scale=8.0, num_inference_steps=40).images
+   prompt = "魚が泳いでいる。"
+   pipe = load_evoukiyoe(device="cuda")
+   images = pipe(prompt + "輻の浮世絵。", guidance_scale=8.0, num_inference_steps=40).images
    images[0].save("image.png")
    ```
 
@@ -65,11 +63,11 @@ Use the code below to get started with the model.
 - **Developed by:** [Sakana AI](https://sakana.ai/)
 - **Model type:** Diffusion-based text-to-image generative model
 - **Language(s):** Japanese
-- **Blog:** https://sakana.ai/evo-ukiyoe/
+- **Blog:** https://sakana.ai/TODO
 
 
 ## License
-The Python script included in this repository and Lora weight are licensed under the [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0).
+The Python script included in this repository is licensed under the [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0).
 Please note that the license for the model/pipeline generated by this script is inherited from the source models.
 
 ## Uses
@@ -82,16 +80,11 @@ Users must fully understand the risks associated with the use of this model and
 
 ## Acknowledgement
 
-Evo-Ukiyoe was trained based on Evo-SDXL-JP. We would like to thank the developers of Evo-SDXL-JP source models for their contributions and for making their work available. 
-- [SDXL](https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0)
-- [Juggernaut-XL-v9](https://huggingface.co/RunDiffusion/Juggernaut-XL-v9)
-- [SDXL-DPO](https://huggingface.co/mhdang/dpo-sdxl-text2image-v1)
-- [JSDXL](https://huggingface.co/stabilityai/japanese-stable-diffusion-xl)
+We would like to thank the developers of the source models for their contributions and for making their work available. 
+
 
 ## Citation
 
-    @misc{Evo-Ukiyoe, 
-    url    = {[https://huggingface.co/SakanaAI/Evo-Nishikie-v1](https://huggingface.co/SakanaAI/Evo-Nishikie-v1)}, 
-    title  = {Evo-Ukiyoe}, 
-    author = {Clanuwat, Tarin and Shing, Makoto and Imajuku, Yuki and Kitamoto, Asanobu and Akama, Ryo}
-    }
+```bibtex
+TODO
+```

evo_ukiyoe_v1.py

@@ -12,8 +12,7 @@ import torch
 from tqdm import tqdm
 from transformers import AutoTokenizer, CLIPTextModelWithProjection
 
-
-# Base models
+# Base models (fine-tuned from SDXL-1.0)
 SDXL_REPO = "stabilityai/stable-diffusion-xl-base-1.0"
 DPO_REPO = "mhdang/dpo-sdxl-text2image-v1"
 JN_REPO = "RunDiffusion/Juggernaut-XL-v9"
@@ -117,7 +116,6 @@ def load_evo_ukiyoe(device="cuda") -> StableDiffusionXLPipeline:
     )
     jn_weights = split_conv_attn(load_from_pretrained(JN_REPO, device=device))
     jsdxl_weights = split_conv_attn(load_from_pretrained(JSDXL_REPO, device=device))
-
     # Merge base models
     tensors = [sdxl_weights, dpo_weights, jn_weights, jsdxl_weights]
     new_conv = merge_models(
@@ -138,14 +136,11 @@ def load_evo_ukiyoe(device="cuda") -> StableDiffusionXLPipeline:
             0.2198623756106564,
         ],
     )
-
-    # Delete no longer needed variables to free
     del sdxl_weights, dpo_weights, jn_weights, jsdxl_weights
     gc.collect()
     if "cuda" in device:
         torch.cuda.empty_cache()
 
-    # Instantiate UNet
     unet_config = UNet2DConditionModel.load_config(SDXL_REPO, subfolder="unet")
     unet = UNet2DConditionModel.from_config(unet_config).to(device=device)
     unet.load_state_dict({**new_conv, **new_attn})
@@ -167,10 +162,25 @@ def load_evo_ukiyoe(device="cuda") -> StableDiffusionXLPipeline:
         torch_dtype=torch.float16,
         variant="fp16",
     )
+    pipe = pipe.to(device, dtype=torch.float16)
 
     # Load Evo-Ukiyoe weights
     pipe.load_lora_weights(UKIYOE_REPO)
     pipe.fuse_lora(lora_scale=1.0)
-
-    pipe = pipe.to(device=torch.device(device), dtype=torch.float16)
     return pipe
+
+
+if __name__ == "__main__":
+    pipe: StableDiffusionXLPipeline = load_evo_ukiyoe()
+    images = pipe(
+        prompt="鶴が庭に立っている。雪が降っている。最高品質の輻の浮世絵。",
+        negative_prompt="",
+        width=1024,
+        height=1024,
+        guidance_scale=8.0,
+        num_inference_steps=50,
+        generator=torch.Generator().manual_seed(0),
+        num_images_per_prompt=1,
+        output_type="pil",
+    ).images
+    images[0].save("out.png")

requirements.txt

@@ -1,8 +0,0 @@
-torch
-torchvision
-
-accelerate==0.32.0
-diffusers==0.29.2
-sentencepiece==0.2.0
-transformers==4.42.3
-peft==0.11.1