README.md

---
license: cc-by-nc-sa-4.0
datasets:
- Cartinoe5930/KoRAE_filtered_12k
language:
- ko
library_name: transformers
---

## KoRAE

<p align="center"><img src="https://cdn-uploads.huggingface.co/production/uploads/63e087b6a98d931aa90c1b9c/XQ-pNzRDRccd7UFgYDOrx.png", width='300', height='300'></p>

We introduce **KoRAE** which finetuned with filtered high-quality Korean dataset.

The **KoRAE** is output of combination of high-quality data which filtered by special data filtering method and Korean Llama-2 that Korean vocabularis were added. 
We utilized special data filtering methods which introduced in [AlpaGasus](https://arxiv.org/abs/2307.08701) to filter high-quality data from mixture of several Korean datasets(OpenOrca-KO, KOpen-Platypus, KoCoT_2000, databricks-dolly-15k-ko). 
We finetuned [Korean Llama-2](https://huggingface.co/beomi/llama-2-koen-13b) that introduced by [@beomi](https://huggingface.co/beomi) on the filtered dataset.
The Flash-Attention2 and LoRA were utilized for efficient finetuning.

The KoRAE will be uploaded in [Open Ko-LLM Leaderboard](https://huggingface.co/spaces/upstage/open-ko-llm-leaderboard)!
In addition, the DPO trained version of KoRAE will be uploaded soon too!
Stay tuned for the update of KoRAE!

## Model Details

- **Developed by:** [Cartinoe5930](https://huggingface.co/Cartinoe5930)
- **Base model:** [beomi/llama-2-koen-13b](https://huggingface.co/beomi/llama-2-koen-13b)
- **Repository:** [gauss5930/KoRAE](https://github.com/gauss5930/KoRAE)

For more details, please check the GitHub Repository!

## Training Details

- **Hardward:** We utilized A100 80G for finetuning
- **Training factors:** The [Transformers Trainer](https://huggingface.co/docs/transformers/main_classes/trainer) and [Huggingface PEFT](https://huggingface.co/docs/peft/index) were utilized for finetuning.

For more details, please check the GitHub Repository!

## Training Dataset

The KoRAE was finetuned with KoRAE dataset filtered high-quality dataset.
This dataset is a combination of the publicly available Koraen dataset and a filtering method was applied to the result of the combination dataset.
For more information, please refer to the [dataset card](https://huggingface.co/datasets/Cartinoe5930/KoRAE_filtered_12k) of KoRAE.

## Open Ko-LLM Leaderboard

## Prompt Template

```
### System:
{system_prompt}

### User:
{instruction + input}

### Assistant:
{output}
```

## Usage example

```python
# Use a pipeline as a high-level helper
from transformers import pipeline
import torch

pipe = pipeline("text-generation", model="Cartinoe5930/KoRAE-13b", torch_dtype=torch.bfloat16, device_map="auto")
messages = [
    {
        "role": "system",
        "content": "당신은 유용한 인공지능 비서입니다. 사용자가 몇 가지 지시가 포함된 작업을 제공합니다. 요청을 적절히 완료하는 응답을 작성하세요.",
    },
    {"role": "user", "content": "스트레스를 해소하는 5가지 방법에 대해서 설명해줘."}
]

prompt = pipe.tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
outputs = pipe(prompt, max_new_tokens=256, do_sample=True, temperature=0.7, top_k=50, top_p=0.95)
print(outputs[0]["generated_text"])
```

## Citation

- [KO-Platypus](https://github.com/Marker-Inc-Korea/KO-Platypus)
- [Korean-OpenOrca](https://github.com/Marker-Inc-Korea/Korean-OpenOrca)

```
@inproceedings{lee2023kullm,
  title={KULLM: Learning to Construct Korean Instruction-following Large Language Models},
  author={Lee, SeungJun and Lee, Taemin and Lee, Jeongwoo and Jang, Yoona and Lim, Heuiseok},
  booktitle={Annual Conference on Human and Language Technology},
  pages={196--202},
  year={2023},
  organization={Human and Language Technology}
}
```

```
@misc{chen2023alpagasus,
      title={AlpaGasus: Training A Better Alpaca with Fewer Data}, 
      author={Lichang Chen and Shiyang Li and Jun Yan and Hai Wang and Kalpa Gunaratna and Vikas Yadav and Zheng Tang and Vijay Srinivasan and Tianyi Zhou and Heng Huang and Hongxia Jin},
      year={2023},
      eprint={2307.08701},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
```

```
@misc {l._junbum_2023,
    author       = { {L. Junbum, Taekyoon Choi} },
    title        = { llama-2-koen-13b },
    year         = 2023,
    url          = { https://huggingface.co/beomi/llama-2-koen-13b },
    doi          = { 10.57967/hf/1280 },
    publisher    = { Hugging Face }
}
```