---
language:
- ko
- en
pipeline_tag: text-generation
inference: false
tags:
- solar
- mistral
- pytorch
- solar-ko
library_name: transformers
license: apache-2.0
---

<img src="https://cdn-uploads.huggingface.co/production/uploads/5e56829137cb5b49818287ea/WuiaS45EAWDurGTOtjR_d.png" style="max-width:250px;margin:0 auto;" />

**Update Log**

- 2024.07.01: Released Solar-Ko-Recovery & Uploaded Benchmark scores
- 2024.05.16: Preview Released Solar-Ko-Recovery

# **Solar-Ko-Recovery-11B** 🌟❤️‍🩹

Solar-Ko-Recovery-11B aimed to recover Solar's capability on Korean with re-arrange of Embeddings and LM head, featuring an expanded vocabulary and the inclusion of a Korean+English corpus for enhanced representation. 

## Model Details

**Model Developers:** Junbum Lee (Beomi)

**Variations:** Solar-Ko-Recovery is available with one parameter sizes — 11B(10.99B🤣).

**Input:** The model accepts only text input.

**Output:** The model produces text output exclusively.

**Model Architecture:** 

Solar-Ko-Recovery is an auto-regressive language model that leverages an optimized transformer architecture derived from Llama-2.

| |Training Data|Parameters|Content Length|GQA|Tokens|Learning Rate|
|---|---|---|---|---|---|---|
|Solar-Ko-Recovery|*A curated mix of Korean+English Corpora*|11B(10.99B)|4k|O|>100B*|5e<sup>-5</sup>|

> NOTE: 2-step training processed
>
> 1) Only Embedding layer and LM Head layer are trained
> 2) Full params trained

**Vocab Expansion**

Vocab expansion is conducted on edited [upstage/solar-1-mini-tokenizer](https://huggingface.co/upstage/solar-1-mini-tokenizer), which is superset of Solar tokenizer.

| Model Name | Vocabulary Size | Description | 
| --- | --- | --- |
| Original Solar | 32000 | Sentencepiece BPE |
| **solar-1-mini-tokenizer** | 64000 | Sentencepiece BPE. Added Ko/JP vocabs |

**Tokenizing "안녕하세요, 오늘은 날씨가 좋네요."**

- SOLAR-10.7B: 26 tokens
- Solar-Ko-Recovery: 7 tokens

| Model | Tokens |
| --- | --- |
| SOLAR-10.7B | `['▁', '안', '<0xEB>', '<0x85>', '<0x95>', '하', '세', '요', ',', '▁', '오', '<0xEB>', '<0x8A>', '<0x98>', '은', '▁', '날', '<0xEC>', '<0x94>', '<0xA8>', '가', '▁', '좋', '네', '요', '.']` |
| Solar-Ko-Recovery | `['▁안녕하세요', ',', '▁오늘은', '▁날씨가', '▁좋', '네요', '.']` |

**Tokenizing "Meet 10.7B Solar: Elevating Performance with Upstage Depth UP Scaling!"**

- SOLAR-10.7B: 22 tokens
- Solar-Ko-Recovery: 22 tokens

| Model | Tokens |
| --- | --- |
| SOLAR-10.7B | `['▁Meet', '▁', '1', '0', '.', '7', 'B', '▁Solar', ':', '▁E', 'lev', 'ating', '▁Performance', '▁with', '▁Up', 'stage', '▁Dep', 'th', '▁UP', '▁Scal', 'ing', '!']` |
| Solar-Ko-Recovery | `['▁Meet', '▁', '1', '0', '.', '7', 'B', '▁Solar', ':', '▁E', 'lev', 'ating', '▁Performance', '▁with', '▁Up', 'stage', '▁Dep', 'th', '▁UP', '▁Scal', 'ing', '!']` |

# LICENSE

Apache 2.0

# **Model Benchmark**

## LM Eval Harness - Korean

- Used EleutherAI's [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness)
- 5-shot scores

|                          Tasks                           |  Metric   |  Value  |   |  Stderr |
|----------------------------------------------------------|-----------|--------:|---|--------:|
|haerae                                                    |acc_norm   | 0.7874  |±  | 0.0118  |
| - haerae_general_knowledge                               |acc        | 0.5000  |±  | 0.0378  |
| - haerae_history                                         |acc        | 0.8723  |±  | 0.0244  |
| - haerae_loan_word                                       |acc        | 0.8402  |±  | 0.0283  |
| - haerae_rare_word                                       |acc        | 0.8346  |±  | 0.0185  |
| - haerae_standard_nomenclature                           |acc        | 0.8301  |±  | 0.0305  |
|kmmlu_direct                                              |exact_match| 0.4205  |±  | 0.0026  |
| - kmmlu_direct_accounting                                |exact_match| 0.3700  |±  | 0.0485  |
| - kmmlu_direct_agricultural_sciences                     |exact_match| 0.3140  |±  | 0.0147  |
| - kmmlu_direct_aviation_engineering_and_maintenance      |exact_match| 0.3870  |±  | 0.0154  |
| - kmmlu_direct_biology                                   |exact_match| 0.3510  |±  | 0.0151  |
| - kmmlu_direct_chemical_engineering                      |exact_match| 0.3910  |±  | 0.0154  |
| - kmmlu_direct_chemistry                                 |exact_match| 0.4000  |±  | 0.0200  |
| - kmmlu_direct_civil_engineering                         |exact_match| 0.4010  |±  | 0.0155  |
| - kmmlu_direct_computer_science                          |exact_match| 0.6520  |±  | 0.0151  |
| - kmmlu_direct_construction                              |exact_match| 0.3080  |±  | 0.0146  |
| - kmmlu_direct_criminal_law                              |exact_match| 0.3100  |±  | 0.0328  |
| - kmmlu_direct_ecology                                   |exact_match| 0.4660  |±  | 0.0158  |
| - kmmlu_direct_economics                                 |exact_match| 0.5385  |±  | 0.0439  |
| - kmmlu_direct_education                                 |exact_match| 0.6200  |±  | 0.0488  |
| - kmmlu_direct_electrical_engineering                    |exact_match| 0.3000  |±  | 0.0145  |
| - kmmlu_direct_electronics_engineering                   |exact_match| 0.4740  |±  | 0.0158  |
| - kmmlu_direct_energy_management                         |exact_match| 0.3560  |±  | 0.0151  |
| - kmmlu_direct_environmental_science                     |exact_match| 0.2980  |±  | 0.0145  |
| - kmmlu_direct_fashion                                   |exact_match| 0.4470  |±  | 0.0157  |
| - kmmlu_direct_food_processing                           |exact_match| 0.3690  |±  | 0.0153  |
| - kmmlu_direct_gas_technology_and_engineering            |exact_match| 0.3000  |±  | 0.0145  |
| - kmmlu_direct_geomatics                                 |exact_match| 0.3820  |±  | 0.0154  |
| - kmmlu_direct_health                                    |exact_match| 0.5700  |±  | 0.0498  |
| - kmmlu_direct_industrial_engineer                       |exact_match| 0.3830  |±  | 0.0154  |
| - kmmlu_direct_information_technology                    |exact_match| 0.6090  |±  | 0.0154  |
| - kmmlu_direct_interior_architecture_and_design          |exact_match| 0.5440  |±  | 0.0158  |
| - kmmlu_direct_korean_history                            |exact_match| 0.3800  |±  | 0.0488  |
| - kmmlu_direct_law                                       |exact_match| 0.4670  |±  | 0.0158  |
| - kmmlu_direct_machine_design_and_manufacturing          |exact_match| 0.3960  |±  | 0.0155  |
| - kmmlu_direct_management                                |exact_match| 0.5030  |±  | 0.0158  |
| - kmmlu_direct_maritime_engineering                      |exact_match| 0.4283  |±  | 0.0202  |
| - kmmlu_direct_marketing                                 |exact_match| 0.7460  |±  | 0.0138  |
| - kmmlu_direct_materials_engineering                     |exact_match| 0.4020  |±  | 0.0155  |
| - kmmlu_direct_math                                      |exact_match| 0.2867  |±  | 0.0262  |
| - kmmlu_direct_mechanical_engineering                    |exact_match| 0.3490  |±  | 0.0151  |
| - kmmlu_direct_nondestructive_testing                    |exact_match| 0.3760  |±  | 0.0153  |
| - kmmlu_direct_patent                                    |exact_match| 0.3700  |±  | 0.0485  |
| - kmmlu_direct_political_science_and_sociology           |exact_match| 0.5300  |±  | 0.0289  |
| - kmmlu_direct_psychology                                |exact_match| 0.4470  |±  | 0.0157  |
| - kmmlu_direct_public_safety                             |exact_match| 0.3520  |±  | 0.0151  |
| - kmmlu_direct_railway_and_automotive_engineering        |exact_match| 0.3220  |±  | 0.0148  |
| - kmmlu_direct_real_estate                               |exact_match| 0.4350  |±  | 0.0351  |
| - kmmlu_direct_refrigerating_machinery                   |exact_match| 0.3240  |±  | 0.0148  |
| - kmmlu_direct_social_welfare                            |exact_match| 0.4970  |±  | 0.0158  |
| - kmmlu_direct_taxation                                  |exact_match| 0.3800  |±  | 0.0344  |
| - kmmlu_direct_telecommunications_and_wireless_technology|exact_match| 0.5480  |±  | 0.0157  |
|kobest_boolq                                              |acc        | 0.9202  |±  | 0.0072  |
|                                                          |f1         | 0.9202  |±  |N/A      |
|kobest_copa                                               |acc        | 0.8680  |±  | 0.0107  |
|                                                          |f1         | 0.8678  |±  |N/A      |
|kobest_hellaswag                                          |acc        | 0.5560  |±  | 0.0222  |
|                                                          |f1         | 0.5520  |±  |N/A      |
|                                                          |acc_norm   | 0.6540  |±  | 0.0213  |
|kobest_sentineg                                           |acc        | 0.9824  |±  | 0.0066  |
|                                                          |f1         | 0.9824  |±  |N/A      |



## Citation

TBD

## Acknowledgements

- Training support was provided by the [TPU Research Cloud](https://sites.research.google/trc/) program.