ToonCrafter_with_SketchGuidance

This repository is an implementation that recreates the SketchGuidance feature of "ToonCrafter".

https://github.com/user-attachments/assets/f72f287d-f848-4982-8f91-43c49d037007

ToonCrafter with Sketch Guidance

Overview

This repository contains the ToonCrafter model with additional sketch guidance functionality. The model is hosted on GitHub and can be accessed via the following link:

Repository: https://github.com/svjack/ToonCrafter_with_SketchGuidance

Installation

Clone the Repository

To clone the repository and set up the environment, follow these steps:

git clone https://github.com/svjack/ToonCrafter_with_SketchGuidance && cd ToonCrafter_with_SketchGuidance
pip install -r requirements.txt
pip install --upgrade einops
pip install pyav==13.1.0

List Control Models

After cloning the repository, you can list the control models available:

ls control_models

Download Sketch Encoder

Download the sketch_encoder.ckpt file from the Hugging Face repository:

wget https://huggingface.co/Doubiiu/ToonCrafter/resolve/main/sketch_encoder.ckpt

Copy Sketch Encoder to Control Models

Move the downloaded sketch_encoder.ckpt file to the control_models directory:

cp sketch_encoder.ckpt control_models

Usage

Run the Gradio App

Once the repository is cloned and the necessary files are in place, you can run the Gradio app using the following command:

python gradio_app.py

This will start the Gradio interface, allowing you to interact with the ToonCrafter model with sketch guidance.

🧰 Models

Model	Resolution	GPU Mem. & Inference Time (A100, ddim 50steps)	Checkpoint
ToonCrafter_512	320x512	TBD (`perframe_ae=True`)	Hugging Face
SketchEncoder	TBD	TBD	Hugging Face

Currently, ToonCrafter can support generating videos of up to 16 frames with a resolution of 512x320. The inference time can be reduced by using fewer DDIM steps.

⚙️ Setup

Install Environment via Anaconda (Recommended)

conda create -n tooncrafter python=3.8.5
conda activate tooncrafter
pip install -r requirements.txt

💫 Inference

1. Local Gradio demo

Download pretrained ToonCrafter_512 and put the model.ckpt in checkpoints/tooncrafter_512_interp_v1/model.ckpt.
Download pretrained SketchEncoder and put the model.ckpt in control_models/sketch_encoder.ckpt.

  python gradio_app.py