File size: 1,596 Bytes
6d7e7ce 000df51 6d7e7ce 000df51 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 |
---
library_name: diffusers
license: other
base_model: BleachNick/SD3_UltraEdit_w_mask
---
# Model Card for UltraSketch
UltraSketch is a diffusion model that has been trained primarily to convert
scientific figures into sketches with a hand-drawn style. It is based on
[SD3_UltraEdit_w_mask](https://huggingface.co/BleachNick/SD3_UltraEdit_w_mask)
and was fine-tuned on
[SketchFig](https://huggingface.co/datasets/nllg/sketchfig). In addition,
figures from
[DaTi*k*Z<sub>v2</sub>](https://huggingface.co/datasets/nllg/datikz-v2)
rendered in a hand-drawn style using
[Rough.js](https://github.com/rough-stuff/rough), as well as the [Sketchy
Database](https://github.com/CDOTAD/SketchyDatabase) and [Photo
Sketching](https://github.com/mtli/PhotoSketch) datasets, have been used for
data augmentation. Check out the
[DeTi*k*Zify](https://github.com/potamides/DeTikZify) project for more
information.
## Usage
```python
from PIL import Image
from datasets import load_dataset
from diffusers import DiffusionPipeline
import torch
figure = load_dataset("nllg/datikz-v2", split="train")['image'][0]
pipe = DiffusionPipeline.from_pretrained(
pretrained_model_name_or_path="nllg/ultrasketch",
custom_pipeline="nllg/ultrasketch",
trust_remote_code=True,
torch_dtype=torch.float16,
device_map="balanced"
)
sketch = pipe(
prompt="Turn it into a hand-drawn sketch",
image=figure,
mask_img=Image.new("RGB", figure.size, "white"),
num_inference_steps=50,
image_guidance_scale=1.7,
guidance_scale=1.5,
strength=0.9
).images[0]
sketch.save("sketch.png")
```
|