|
--- |
|
license: llama3 |
|
language: |
|
- ko |
|
tags: |
|
- korean |
|
- llama3 |
|
- instruction-tuning |
|
- dora |
|
datasets: |
|
- Acyrl |
|
- llm-kr-eval |
|
- Counter-MT-bench |
|
base_model: |
|
- meta-llama/Meta-Llama-3-8B |
|
pipeline_tag: text-generation |
|
--- |
|
|
|
# A-LLM: Korean Language Model based on Llama-3 |
|
|
|
## Introduction |
|
A-LLM is a Korean language model built on Meta's Llama-3-8B architecture, specifically optimized for Korean language understanding and generation. The model was trained using the DoRA (Weight-Decomposed Low-Rank Adaptation) methodology on a comprehensive Korean dataset, achieving state-of-the-art performance among open-source Korean language models. |
|
|
|
|
|
## Performance Benchmarks |
|
### Horangi Korean LLM Leaderboard |
|
The model's performance was evaluated using the Horangi Korean LLM Leaderboard |
|
, which combines two major evaluation frameworks normalized to a 1.0 scale and averages their scores. |
|
|
|
#### 1. LLM-KR-EVAL |
|
A comprehensive benchmark that measures fundamental NLP capabilities across 5 core tasks: |
|
- Natural Language Inference (NLI) |
|
- Question Answering (QA) |
|
- Reading Comprehension (RC) |
|
- Entity Linking (EL) |
|
- Fundamental Analysis (FA) |
|
|
|
The benchmark comprises 10 different datasets distributed across these tasks, providing a thorough assessment of Korean language understanding and processing capabilities. |
|
|
|
#### 2. MT-Bench |
|
A diverse evaluation framework consisting of 80 questions (10 questions each from 8 categories), evaluated using GPT-4 as the judge. Categories include: |
|
- Writing |
|
- Roleplay |
|
- Extraction |
|
- Reasoning |
|
- Math |
|
- Coding |
|
- Knowledge (STEM) |
|
- Knowledge (Humanities/social science) |
|
|
|
### Performance Results |
|
|
|
| Model | Total Score | AVG_llm_kr_eval | AVG_mtbench | |
|
|-------|-------------|-----------------|-------------| |
|
| A-LLM (Ours) | 0.6675 | 0.5937 | 7.413 | |
|
| GPT-4 | 0.7363 | 0.6158 | 8.569 | |
|
| Mixtral-8x7B | 0.5843 | 0.4304 | 7.381 | |
|
| KULLM3 | 0.5764 | 0.5204 | 6.325 | |
|
| SOLAR-1-mini | 0.5173 | 0.37 | 6.647 | |
|
|
|
Our model achieves state-of-the-art performance among open-source Korean language models, demonstrating strong capabilities across both general language understanding (LLM-KR-EVAL) and diverse task-specific applications (MT-Bench). |
|
|
|
### Model Components |
|
This repository provides: |
|
- Tokenizer configuration |
|
- Model weights in safetensor format |
|
|
|
## Usage Instructions |
|
|
|
### Prerequisites |
|
- Python 3.8 or higher |
|
- PyTorch 2.0 or higher |
|
- Transformers library |
|
|
