README.md

---
license: apache-2.0
language:
- ko
- en
pipeline_tag: visual-question-answering
tags:
- text2text-generation
base_model: google/deplot
---
# **Ko-Deplot**

Ko-Deplot is a korean Visual-QA model based on the Google's Pix2Struct architecture. It was fine-tuned from [Deplot](https://huggingface.co/google/deplot), using korean chart image-text pairs.

Ko-Deplot은 Google의 Pix2Struct 구조를 기반으로 한 한국어 Visual-QA 모델입니다. [Deplot](https://huggingface.co/google/deplot) 모델을 한국어 차트 이미지-텍스트 쌍 데이터셋을 이용하여 파인튜닝하였습니다.

- **Developed by:** [NUUA](https://www.nuua.ai/en/): Yohan Kim, Jiyou Shin, Robin Lee
- **Model type:** Visual Question Answering
- **License:** apache-2.0
- **Finetuned from model:** [google/deplot](https://huggingface.co/google/deplot)

# **Model Usage**
You can run a prediction by querying an input image together with a question as follows:

```python
from transformers import Pix2StructProcessor, Pix2StructForConditionalGeneration
from PIL import Image

processor = Pix2StructProcessor.from_pretrained('nuua/Ko-Deplot')
model = Pix2StructForConditionalGeneration.from_pretrained('nuua/Ko-Deplot')

IMAGE_PATH = "LOCAL_PATH_TO_IMAGE"
image = Image.open(IMAGE_PATH)

inputs = processor(images=image, text="Generate underlying data table of the figure below:", return_tensors="pt")
predictions = model.generate(**inputs, max_new_tokens=512)
print(processor.decode(predictions[0], skip_special_tokens=True))
```

# **Training Details**

## Training Data

Synthetic chart data from three libraries were used:

- [GenPlot](https://github.com/brendanartley/genplot)
- [Chart.js](https://github.com/chartjs/Chart.js)
- [Plotly](https://github.com/plotly/plotly.py)

## Training Procedure 

The model was first exposed to a short warmup stage, following its [original paper](https://arxiv.org/pdf/2210.03347.pdf). It was then trained using the chart data for 50,000 steps.

# **Technical Specifications**

## Hardware

Ko-Deplot was trained by using A100 80G. 

# **Contact**

Any questions and suggestions, please use the discussion tab. If you want to contact us directly, email [email protected].