|
--- |
|
library_name: diffusers |
|
license: other |
|
base_model: BleachNick/SD3_UltraEdit_w_mask |
|
--- |
|
|
|
# Model Card for UltraSketch |
|
UltraSketch is a diffusion model that has been trained primarily to convert |
|
scientific figures into sketches with a hand-drawn style. It is based on |
|
[SD3_UltraEdit_w_mask](https://huggingface.co/BleachNick/SD3_UltraEdit_w_mask) |
|
and was fine-tuned on |
|
[SketchFig](https://huggingface.co/datasets/nllg/sketchfig). In addition, |
|
figures from |
|
[DaTi*k*Z<sub>v2</sub>](https://huggingface.co/datasets/nllg/datikz-v2) |
|
rendered in a hand-drawn style using |
|
[Rough.js](https://github.com/rough-stuff/rough), as well as the [Sketchy |
|
Database](https://github.com/CDOTAD/SketchyDatabase) and [Photo |
|
Sketching](https://github.com/mtli/PhotoSketch) datasets, have been used for |
|
data augmentation. Check out the |
|
[DeTi*k*Zify](https://github.com/potamides/DeTikZify) project for more |
|
information. |
|
|
|
## Usage |
|
```python |
|
from PIL import Image |
|
from datasets import load_dataset |
|
from diffusers import DiffusionPipeline |
|
import torch |
|
|
|
figure = load_dataset("nllg/datikz-v2", split="train")['image'][0] |
|
|
|
pipe = DiffusionPipeline.from_pretrained( |
|
pretrained_model_name_or_path="nllg/ultrasketch", |
|
custom_pipeline="nllg/ultrasketch", |
|
trust_remote_code=True, |
|
torch_dtype=torch.float16, |
|
device_map="balanced" |
|
) |
|
|
|
sketch = pipe( |
|
prompt="Turn it into a hand-drawn sketch", |
|
image=figure, |
|
mask_img=Image.new("RGB", figure.size, "white"), |
|
num_inference_steps=50, |
|
image_guidance_scale=1.7, |
|
guidance_scale=1.5, |
|
strength=0.9 |
|
).images[0] |
|
|
|
sketch.save("sketch.png") |
|
``` |
|
|
