---
library_name: diffusers
license: other
base_model: BleachNick/SD3_UltraEdit_w_mask
---

# Model Card for UltraSketch
UltraSketch is a diffusion model that has been trained primarily to convert
scientific figures into sketches with a hand-drawn style. It is based on
[SD3_UltraEdit_w_mask](https://huggingface.co/BleachNick/SD3_UltraEdit_w_mask)
and was fine-tuned on
[SketchFig](https://huggingface.co/datasets/nllg/sketchfig). In addition,
figures from
[DaTi*k*Z<sub>v2</sub>](https://huggingface.co/datasets/nllg/datikz-v2)
rendered in a hand-drawn style using
[Rough.js](https://github.com/rough-stuff/rough), as well as the [Sketchy
Database](https://github.com/CDOTAD/SketchyDatabase) and [Photo
Sketching](https://github.com/mtli/PhotoSketch) datasets, have been used for
data augmentation. Check out the
[DeTi*k*Zify](https://github.com/potamides/DeTikZify) project for more
information.

## Usage
```python
from PIL import Image
from datasets import load_dataset
from diffusers import DiffusionPipeline
import torch

figure = load_dataset("nllg/datikz-v2", split="train")['image'][0]

pipe = DiffusionPipeline.from_pretrained(
    pretrained_model_name_or_path="nllg/ultrasketch",
    custom_pipeline="nllg/ultrasketch",
    trust_remote_code=True,
    torch_dtype=torch.float16,
    device_map="balanced"
)

sketch = pipe(
    prompt="Turn it into a hand-drawn sketch",
    image=figure,
    mask_img=Image.new("RGB", figure.size, "white"),
    num_inference_steps=50,
    image_guidance_scale=1.7,
    guidance_scale=1.5,
    strength=0.9
).images[0]

sketch.save("sketch.png")
```