ToonCrafter_with_SketchGuidance
This repository is an implementation that recreates the SketchGuidance feature of "ToonCrafter".
https://github.com/user-attachments/assets/f72f287d-f848-4982-8f91-43c49d037007
ToonCrafter with Sketch Guidance
Overview
This repository contains the ToonCrafter model with additional sketch guidance functionality. The model is hosted on GitHub and can be accessed via the following link:
Installation
Clone the Repository
To clone the repository and set up the environment, follow these steps:
git clone https://github.com/svjack/ToonCrafter_with_SketchGuidance && cd ToonCrafter_with_SketchGuidance
pip install -r requirements.txt
pip install --upgrade einops
List Control Models
After cloning the repository, you can list the control models available:
ls control_models
Download Sketch Encoder
Download the sketch_encoder.ckpt file from the Hugging Face repository:
wget https://huggingface.co/Doubiiu/ToonCrafter/resolve/main/sketch_encoder.ckpt
Copy Sketch Encoder to Control Models
Move the downloaded sketch_encoder.ckpt file to the control_models directory:
cp sketch_encoder.ckpt control_models
Usage
Run the Gradio App
Once the repository is cloned and the necessary files are in place, you can run the Gradio app using the following command:
python gradio_app.py
This will start the Gradio interface, allowing you to interact with the ToonCrafter model with sketch guidance.
🧰 Models
| Model | Resolution | GPU Mem. & Inference Time (A100, ddim 50steps) | Checkpoint |
|---|---|---|---|
| ToonCrafter_512 | 320x512 | TBD (perframe_ae=True) |
Hugging Face |
| SketchEncoder | TBD | TBD | Hugging Face |
Currently, ToonCrafter can support generating videos of up to 16 frames with a resolution of 512x320. The inference time can be reduced by using fewer DDIM steps.
⚙️ Setup
Install Environment via Anaconda (Recommended)
conda create -n tooncrafter python=3.8.5
conda activate tooncrafter
pip install -r requirements.txt
💫 Inference
1. Local Gradio demo
- Download pretrained ToonCrafter_512 and put the model.ckpt in checkpoints/tooncrafter_512_interp_v1/model.ckpt.
- Download pretrained SketchEncoder and put the model.ckpt in control_models/sketch_encoder.ckpt.
python gradio_app.py