LoRA text2image fine-tuning - spockren/naruto-lora-xl
These are LoRA adaption weights for stabilityai/stable-diffusion-xl-base-1.0. The weights were fine-tuned on the lambdalabs/naruto-blip-captions dataset. You can find some example images in the following.
LoRA for the text encoder was enabled: False.
Special VAE used for training: None.
Intended uses & limitations
How to use
from diffusers import AutoPipelineForText2Image
import torch
pipeline = AutoPipelineForText2Image.from_pretrained("stabilityai/stable-diffusion-xl-base-1.0", torch_dtype=torch.float16).to("cuda")
pipeline.load_lora_weights("spockren/naruto_lora_xl", weight_name="pytorch_lora_weights.safetensors")
image = pipeline("A naruto with blue eyes").images[0]
image
Limitations and bias
[TODO: provide examples of latent issues and potential remediations]
Training details
export MODEL_NAME="stabilityai/stable-diffusion-xl-base-1.0"
export OUTPUT_DIR="./xl_lora/naruto"
export HUB_MODEL_ID="naruto-lora-xl"
export DATASET_NAME="lambdalabs/naruto-blip-captions"
accelerate launch --mixed_precision="fp16" train_text_to_image_lora_sdxl.py \
--pretrained_model_name_or_path=$MODEL_NAME \
--dataset_name=$DATASET_NAME \
--dataloader_num_workers=8 \
--resolution=512 \
--center_crop \
--random_flip \
--train_batch_size=1 \
--gradient_accumulation_steps=4 \
--max_train_steps=15000 \
--learning_rate=1e-04 \
--max_grad_norm=1 \
--lr_scheduler="cosine" \
--lr_warmup_steps=0 \
--output_dir=${OUTPUT_DIR} \
--push_to_hub \
--hub_model_id=${HUB_MODEL_ID} \
--checkpointing_steps=500 \
--validation_prompt="A naruto with blue eyes." \
--checkpoints_total_limit=6 \
--seed=1337
- Downloads last month
- 3
Model tree for spockren/naruto_lora_xl
Base model
stabilityai/stable-diffusion-xl-base-1.0