File size: 4,693 Bytes
dfcc484 dce1a7b dfcc484 dce1a7b dfcc484 dce1a7b dfcc484 dce1a7b dfcc484 47df0a8 dfcc484 dce1a7b |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 |
---
license: llama3.1
datasets:
- survivi/Llama-3-SynE-Dataset
- hfl/stem_zh_instruction
- llamafactory/alpaca_zh
- llamafactory/alpaca_gpt4_zh
- hfl/ruozhiba_gpt4
- codingsteven/Llama-3-8B-chat
language:
- zh
metrics:
- accuracy
base_model:
- meta-llama/Llama-3.1-8B
pipeline_tag: text-generation
library_name: transformers
model-index:
- name: Control-LLM-Llama3.1-8B-SynE-Concat16-Dlerp
results:
- task:
type: pretraining-evaluation
dataset:
type: mixed
name: Pretraining Evaluation Dataset
metrics:
- name: exact_match,strict-match (meta_pretrain)
type: exact_match
value: 0.48514264142803215
stderr: 0.003513307445696379
verified: false
- name: exact_match,strict-match (meta_bbh_3shot_cot_pretrain)
type: exact_match
value: 0.6817693134695131
stderr: 0.0057729694388110805
verified: false
- name: acc,none (meta_mmlu_5shot_pretrain)
type: accuracy
value: 0.65596068936049
stderr: 0.0040090726054856874
verified: false
- name: exact_match,strict-match (meta_mmlu_pro_5shot_pretrain)
type: exact_match
value: 0.3787400265957447
stderr: 0.004422383756050139
verified: false
- task:
type: chinese-evaluation
dataset:
type: mixed
name: Chinese Evaluation Dataset
metrics:
- name: exact_match,strict-match (zh_pretrain_multishot)
type: exact_match
value: 0.44848391089108913
stderr: 0.004255614019851072
verified: false
- name: acc,none (ceval-valid)
type: accuracy
value: 0.5698365527488856
stderr: 0.012893833892221353
verified: false
- name: exact_match,strict-match (ceval-valid-pretrain-cot_zh)
type: exact_match
value: 0.4472511144130758
stderr: 0.013203606600472227
verified: false
- name: acc,none (cmmlu)
type: accuracy
value: 0.5602659298912105
stderr: 0.0044928840587441605
verified: false
- name: exact_match,strict-match (cmmlu_pretrain_cot_zh)
type: exact_match
value: 0.4486271801070627
stderr: 0.00449553418468653
verified: false
---
# Control-LLM-Llama3.1-8B-SynE-Concat16-Dlerp
This is a fine-tuned model of Llama-3.1-8B for muliligual-Chinese tasks on SynE dataset by Control LLM-Concat16-Dlerp, as described in [Control LLM: Controlled Evolution for Intelligence Retention in LLM](https://huggingface.co/papers/2501.10979).
## Linked Paper
This model is associated with the paper: [Control-LLM](https://arxiv.org/abs/2410.14745).
## Linked Open Source code - training, eval and benchmark
This model is associated with the github: [Control-LLM](https://github.com/linkedin/ControlLLM).
## Evaluation Results
Here is an overview of the evaluation results and findings:
### Benchmark Results Table
The table below summarizes evaluation results across Chinese tasks and original capabilities.
| **Model** | **CEval** | **CEvalC** | **CMMLU** | **CMMLUC** | **C-Avg** | **BBH** | **MLU** | **MLUP** | **O-Avg** | **Overall** |
|--------------------|-----------|------------|-----------|------------|-----------|---------|---------|----------|-----------|-------------|
| Llama3.1-8B | 48.3 | 12.8 | 51.1 | 14.1 | 13.9 | 65.2 | 65.4 | 35.5 | 45.9 | 29.9 |
| Llama-3-SynE | 57.7 | 22.3 | 57.1 | 22.8 | 22.8 | 61.9 | 64.0 | 32.6 | 42.9 | 32.9 |
| Full Param Tune | 59.0 | 40.2 | **60.2** | 44.3 | 43.8 | 64.8 | 64.9 | 35.0 | 45.4 | 44.6 |
| Stack Expansion | 56.0 | 32.7 | 55.2 | 33.4 | 33.3 | 62.3 | 65.6 | 35.3 | 44.8 | 39.1 |
| Concat-Lerp | 57.1 | 34.8 | 57.0 | 37.4 | 37.1 | 64.4 | 64.6 | 35.8 | 45.9 | 41.5 |
| Hybrid Expansion | **58.9** | 44.7 | 57.9 | 44.3 | 44.4 | 65.1 | **65.7**| 36.9 | 46.8 | 45.6 |
| **Control LLM*** | 57.0 | **44.7** | 56.0 | **44.9** | **44.8** | **68.2**| 65.6 | **37.9** | **48.5** | **46.7** |
---
### Explanation:
- **CEval**: Chinese Evaluation
- **CEvalC**: Chinese Evaluation (CoT - Chain of Thought)
- **CMMLU**: Chinese MMLU
- **CMMLUC**: Chinese MMLU (CoT)
- **C-Avg**: Chinese - Size Weighted Average across CEval, CEvalC, CMMLU, and CMMLUC
- **BBH**: BigBench Hard
- **MLU**: MMLU (Massive Multitask Language Understanding)
- **MLUP**: MMLU Pro
- **O-Avg**: Original Capability - Size Weighted Average across BBH, MLU, and MLUP
- **Overall**: Combined average across all tasks |