--- base_model: stabilityai/stable-diffusion-3-medium-diffusers library_name: diffusers license: openrail++ tags: - text-to-image - diffusers-training - diffusers - lora - template:sd-lora - stable-diffusion-3 - stable-diffusion-3-diffusers - adapters - LoRA - biological structures - science - materiomics - bio-inspired - materials science instance_prompt: widget: [] --- # Stable Diffusion 3 Medium Fine-tuned with Leaf Microstructure Images DreamBooth is an advanced technique designed for fine-tuning text-to-image diffusion models to generate personalized images of specific subjects. By leveraging a few reference images (around 5 or so), DreamBooth integrates unique visual features of the subject into the model's output domain. This is achieved by binding a unique identifier "\<..IDENTIFIER..\>", such as \ in this work, to the subject. An optional class-specific prior preservation loss can be used to maintain high fidelity and contextual diversity. The result is a model capable of synthesizing novel, photorealistic images of the subject in various scenes, poses, and lighting conditions, guided by text prompts. In this project, DreamBooth has been applied to render images with specific biological patterns, making it ideal for applications in materials science and engineering where accurate representation of biological material microstructures is crucial. For example, an original prompt might be: "a vase with intricate patterns, high quality." With the fine-tuned model, using the unique identifier, the prompt becomes: "a vase that resembles a \, high quality." This allows the model to generate images that specifically incorporate the desired biological pattern. ## Model description These are LoRA adaption weights for stabilityai/stable-diffusion-3-medium-diffusers. ## Trigger keywords The following image were used during fine-tuning using the keyword \: ![image/png](https://cdn-uploads.huggingface.co/production/uploads/623ce1c6b66fedf374859fe7/sI_exTnLy6AtOFDX1-7eq.png) You should use \ to trigger the image generation. [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/#fileId=https://huggingface.co/lamm-mit/stable-diffusion-3-medium-leaf-inspired/blob/main/SD3_leaf_inspired_inference.ipynb) ## How to use Defining some helper functions: ```python from diffusers import DiffusionPipeline import torch import os from datetime import datetime from PIL import Image def generate_filename(base_name, extension=".png"): timestamp = datetime.now().strftime("%Y%m%d_%H%M%S") return f"{base_name}_{timestamp}{extension}" def save_image(image, directory, base_name="image_grid"): filename = generate_filename(base_name) file_path = os.path.join(directory, filename) image.save(file_path) print(f"Image saved as {file_path}") def image_grid(imgs, rows, cols, save=True, save_dir='generated_images', base_name="image_grid", save_individual_files=False): if not os.path.exists(save_dir): os.makedirs(save_dir) assert len(imgs) == rows * cols w, h = imgs[0].size grid = Image.new('RGB', size=(cols * w, rows * h)) grid_w, grid_h = grid.size for i, img in enumerate(imgs): grid.paste(img, box=(i % cols * w, i // cols * h)) if save_individual_files: save_image(img, save_dir, base_name=base_name+f'_{i}-of-{len(imgs)}_') if save and save_dir: save_image(grid, save_dir, base_name) return grid ``` ### Text-to-image Model loading and generation pipeline: ```python repo_id_load='lamm-mit/stable-diffusion-3-medium-leaf-inspired' pipeline = DiffusionPipeline.from_pretrained ("stabilityai/stable-diffusion-3-medium-diffusers", torch_dtype=torch.float16 ) pipeline.load_lora_weights(repo_id_load) pipeline=pipeline.to('cuda') prompt = "a cube in the shape of a " negative_prompt = "" num_samples = 3 num_rows = 3 n_steps=75 guidance_scale=15 all_images = [] for _ in range(num_rows): image = pipeline(prompt,num_inference_steps=n_steps,num_images_per_prompt=num_samples, guidance_scale=guidance_scale,negative_prompt=negative_prompt).images all_images.extend(image) grid = image_grid(all_images, num_rows, num_samples, save_individual_files=True, save_dir='generated_images', base_name="image_grid", ) grid ``` ![image/png](https://cdn-uploads.huggingface.co/production/uploads/623ce1c6b66fedf374859fe7/qk5kRJJmetvhZ0ctltc3z.png) ### Image-to-image We start with this image generated earlier: ![image/png](https://cdn-uploads.huggingface.co/production/uploads/623ce1c6b66fedf374859fe7/JYVEhq6yqVtG_MHup3rDb.png) ```python from diffusers import StableDiffusion3Img2ImgPipeline from diffusers.utils import load_image pipeline = StableDiffusion3Img2ImgPipeline.from_pretrained("stabilityai/stable-diffusion-3-medium-diffusers", torch_dtype=torch.float16) pipeline=pipeline.to('cuda') init_image = load_image("https://huggingface.co/lamm-mit/stable-diffusion-3-medium-leaf-inspired/resolve/main/image_20240721_212111.png") prompt = "Turn this image into a spider web." negative_prompt="" n_steps=20 guidance_scale=25 image = pipeline(prompt, num_inference_steps=n_steps, guidance_scale=guidance_scale, negative_prompt=negative_prompt, image=init_image, ).images[0] save_image(image, directory='generated_images', base_name="image_grid", ) image ``` ![image/png](https://cdn-uploads.huggingface.co/production/uploads/623ce1c6b66fedf374859fe7/kI-lx0UCFBErbdUIMn-cG.png) ## More examples ![image/png](https://cdn-uploads.huggingface.co/production/uploads/623ce1c6b66fedf374859fe7/XxOb6nKuYl4H2pYO-jVNi.png) ![image/png](https://cdn-uploads.huggingface.co/production/uploads/623ce1c6b66fedf374859fe7/TH1IZsPRMQssYIDHzIsYI.png) ![image/png](https://cdn-uploads.huggingface.co/production/uploads/623ce1c6b66fedf374859fe7/4EvwVf4l2-CvCKO8Ldg1N.png) ![image/png](https://cdn-uploads.huggingface.co/production/uploads/623ce1c6b66fedf374859fe7/qFBQG-smW5W75MBaNwZcH.png) ## Fine-tuning script Download this script: [SD3 DreamBooth-LoRA_Fine-Tune.ipynb](https://huggingface.co/lamm-mit/stable-diffusion-3-medium-leaf-inspired/resolve/main/SD3_DreamBooth-LoRA_Fine-Tune.ipynb) You need to create a local folder ```leaf_concept_dir_SD3_12``` and add the leaf images (provided in this repository, see subfolder). The code will automatically download the training script. The training script can handle custom prompts associated with each image, which are generated using BLIP. For instance, for the images used here, they are: ```raw [', a close up of a green plant with a lot of small holes', ', a close up of a leaf with a small insect on it', ', a close up of a plant with a lot of green leaves', ', a close up of a green plant with a yellow light', ', a close up of a green plant with a white center', ', arafed leaf with a white line on the center', ', a close up of a leaf with a yellow light shining through it', ', arafed image of a green plant with a yellow cross'] ``` The Parquet dataset generated during pre-calculation of embeddings is stored in the folder ```{data_df_path}```. It includes the image paths, embeddings, and a few other columns that are used by the training script. Training then proceeds as: ```raw accelerate launch train_dreambooth_lora_sd3_miniature.py \ --pretrained_model_name_or_path="{pretrained_model_name_or_path}" \ --instance_data_dir="{instance_data_dir}" \ --data_df_path="{instance_output_dir_embed}" \ --output_dir="{instance_output_dir}" \ --mixed_precision="fp16" \ --instance_prompt="{instance_prompt}" \ --resolution=1024 \ --train_batch_size=1 \ --gradient_accumulation_steps=4 \ --gradient_checkpointing \ --learning_rate=1e-4 \ --lr_scheduler="constant" \ --weighting_scheme="logit_normal" \ --lr_warmup_steps=0 \ --use_8bit_adam \ --max_train_steps=500 \ --checkpointing_steps=500 \ --seed="3234290" ### With prior preservation and a more flexible training script Training notebook with prior preservation, using more flexible framework: [SD3_DreamBooth-LoRA_Fine-Tune-with-prior-preservation.ipynb](https://huggingface.co/lamm-mit/stable-diffusion-3-medium-leaf-inspired/resolve/main/SD3_DreamBooth-LoRA_Fine-Tune-with-prior-preservation.ipynb) The notebook automatically downloads the training code ```launch train_dreambooth_lora_sd3.py```. ```raw accelerate launch train_dreambooth_lora_sd3.py \ --pretrained_model_name_or_path="{pretrained_model_name_or_path}" \ --dataset_name="lamm-mit/{instance_output_dir}_data" \ --caption_column='caption' \ --image_column='image' \ --instance_prompt="{instance_prompt}" \ --with_prior_preservation \ --prior_loss_weight=1.0 \ --output_dir="{instance_output_dir}" \ --class_data_dir="{class_data_dir}" \ --class_prompt="{class_prompt}" \ --num_class_images={num_class_images} \ --mixed_precision="fp16" \ --resolution=1024 \ --train_batch_size=1 \ --gradient_accumulation_steps=4 \ --gradient_checkpointing \ --learning_rate=1e-4 \ --lr_scheduler="constant" \ --weighting_scheme="logit_normal" \ --lr_warmup_steps=0 \ --use_8bit_adam \ --max_train_steps=500 \ --checkpointing_steps=500 \ --seed="3234290" ``` ![image/png](https://cdn-uploads.huggingface.co/production/uploads/623ce1c6b66fedf374859fe7/PQrUWTt7S0l5S62zgjeNo.png) ![image/png](https://cdn-uploads.huggingface.co/production/uploads/623ce1c6b66fedf374859fe7/7oRedflOmvxOTgXbuRBrJ.png) ![image/png](https://cdn-uploads.huggingface.co/production/uploads/623ce1c6b66fedf374859fe7/GdQlPVZ2NKoPbO69O95wU.png) ![image/png](https://cdn-uploads.huggingface.co/production/uploads/623ce1c6b66fedf374859fe7/uSXtkG1CSfkhq9JHHeWvV.png) ![image/png](https://cdn-uploads.huggingface.co/production/uploads/623ce1c6b66fedf374859fe7/CLeDOghDw9q5WNNmAg2h9.png)