|
--- |
|
library_name: keras |
|
license: mit |
|
language: |
|
- en |
|
pipeline_tag: image-to-image |
|
tags: |
|
- art |
|
- pixel_art |
|
- character_sprite |
|
- missing_data_imputation |
|
- image_to_image |
|
--- |
|
|
|
## Model description |
|
|
|
The MDIGAN-Characters model was proposed in SBGames 2024 ([paper on ArXiv][paper-arxiv], [page][paper-page] [demo][paper-demo]) |
|
It is a model trained for the task of generating characters in a missing pose: for instance, |
|
given images of a character facing back, left, and right, it can generate the character facing front (missing data imputation task). |
|
 |
|
|
|
The model's architecture is based on [CollaGAN][paper-collagan]'s, a model trained to impute images in missing domains |
|
in a multi-domain scenario. In our case, the domains are the sides a character might face, i.e., back, left, front, and right. |
|
|
|
We tested providing 3 images to the model, to generate the missing one. But we also evaluated the quality of the generated |
|
images when the model receives 2 or 1 input image. |
|
|
|
The inputs to the model are the target (missing) domain and 4 image-like tensors with size 64x64x4 in the order |
|
back, left, front, and right. The input images should be floating point tensors in the range of [-1, 1]. |
|
In place of the missing image(s), we must provide a tensor with shape 64x64x4 filled with zeros. |
|
|
|
|
|
[paper-collagan]: https://www.computer.org/csdl/proceedings-article/cvpr/2019/329300c482/1gys5gg67QY |
|
[paper-arxiv]: https://arxiv.org/abs/2409.10721 |
|
[paper-page]: https://fegemo.github.io/mdigan-characters |
|
[paper-demo]: https://fegemo.github.io/interactive-generator |
|
|
|
## Intended uses & limitations |
|
|
|
This can be used for research purposes only. The quality of the generated images vary a lot, and a |
|
post-processing step to quantize the colors of the generated image to the intended palette is benefitial. |
|
|
|
|
|
## Training and evaluation data |
|
|
|
The model was trained with the [PAC dataset][pac], which features 12,074 paired images of pixel art characters |
|
in 4 directions: back, left, front, and right. Compared to StarGAN and Pix2Pix-based baselines, the MDIGAN-Characters |
|
model yielded much better images when it received 3 images, and still good images when only 2 are provided. |
|
|
|
[pac]: https://github.com/plucksquire/pac/ |
