Heng666 leaderboard-pr-bot commited on
Commit
05e6b3a
1 Parent(s): c5777a6

Adding Evaluation Results (#1)

Browse files

- Adding Evaluation Results (6b2f83fb28827feee755c5541bfefc74f5c192c1)


Co-authored-by: Open LLM Leaderboard PR Bot <[email protected]>

Files changed (1) hide show
  1. README.md +122 -6
README.md CHANGED
@@ -1,4 +1,9 @@
1
  ---
 
 
 
 
 
2
  license: apache-2.0
3
  tags:
4
  - moe
@@ -8,11 +13,109 @@ tags:
8
  - MediaTek-Research/Breeze-7B-Instruct-v0.1
9
  - augmxnt/shisa-7b-v1
10
  - beomi/OPEN-SOLAR-KO-10.7B
11
- language:
12
- - zh
13
- - ja
14
- - ko
15
- - tw
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
16
  ---
17
 
18
  # EastAsia-4x7B-Moe-experiment
@@ -62,4 +165,17 @@ messages = [{"role": "user", "content": "Explain what a Mixture of Experts is in
62
  prompt = pipeline.tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
63
  outputs = pipeline(prompt, max_new_tokens=256, do_sample=True, temperature=0.7, top_k=50, top_p=0.95)
64
  print(outputs[0]["generated_text"])
65
- ```
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
+ language:
3
+ - zh
4
+ - ja
5
+ - ko
6
+ - tw
7
  license: apache-2.0
8
  tags:
9
  - moe
 
13
  - MediaTek-Research/Breeze-7B-Instruct-v0.1
14
  - augmxnt/shisa-7b-v1
15
  - beomi/OPEN-SOLAR-KO-10.7B
16
+ model-index:
17
+ - name: EastAsia-4x7B-Moe-experiment
18
+ results:
19
+ - task:
20
+ type: text-generation
21
+ name: Text Generation
22
+ dataset:
23
+ name: AI2 Reasoning Challenge (25-Shot)
24
+ type: ai2_arc
25
+ config: ARC-Challenge
26
+ split: test
27
+ args:
28
+ num_few_shot: 25
29
+ metrics:
30
+ - type: acc_norm
31
+ value: 39.51
32
+ name: normalized accuracy
33
+ source:
34
+ url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Heng666/EastAsia-4x7B-Moe-experiment
35
+ name: Open LLM Leaderboard
36
+ - task:
37
+ type: text-generation
38
+ name: Text Generation
39
+ dataset:
40
+ name: HellaSwag (10-Shot)
41
+ type: hellaswag
42
+ split: validation
43
+ args:
44
+ num_few_shot: 10
45
+ metrics:
46
+ - type: acc_norm
47
+ value: 48.92
48
+ name: normalized accuracy
49
+ source:
50
+ url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Heng666/EastAsia-4x7B-Moe-experiment
51
+ name: Open LLM Leaderboard
52
+ - task:
53
+ type: text-generation
54
+ name: Text Generation
55
+ dataset:
56
+ name: MMLU (5-Shot)
57
+ type: cais/mmlu
58
+ config: all
59
+ split: test
60
+ args:
61
+ num_few_shot: 5
62
+ metrics:
63
+ - type: acc
64
+ value: 56.2
65
+ name: accuracy
66
+ source:
67
+ url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Heng666/EastAsia-4x7B-Moe-experiment
68
+ name: Open LLM Leaderboard
69
+ - task:
70
+ type: text-generation
71
+ name: Text Generation
72
+ dataset:
73
+ name: TruthfulQA (0-shot)
74
+ type: truthful_qa
75
+ config: multiple_choice
76
+ split: validation
77
+ args:
78
+ num_few_shot: 0
79
+ metrics:
80
+ - type: mc2
81
+ value: 49.83
82
+ source:
83
+ url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Heng666/EastAsia-4x7B-Moe-experiment
84
+ name: Open LLM Leaderboard
85
+ - task:
86
+ type: text-generation
87
+ name: Text Generation
88
+ dataset:
89
+ name: Winogrande (5-shot)
90
+ type: winogrande
91
+ config: winogrande_xl
92
+ split: validation
93
+ args:
94
+ num_few_shot: 5
95
+ metrics:
96
+ - type: acc
97
+ value: 58.09
98
+ name: accuracy
99
+ source:
100
+ url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Heng666/EastAsia-4x7B-Moe-experiment
101
+ name: Open LLM Leaderboard
102
+ - task:
103
+ type: text-generation
104
+ name: Text Generation
105
+ dataset:
106
+ name: GSM8k (5-shot)
107
+ type: gsm8k
108
+ config: main
109
+ split: test
110
+ args:
111
+ num_few_shot: 5
112
+ metrics:
113
+ - type: acc
114
+ value: 0.15
115
+ name: accuracy
116
+ source:
117
+ url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Heng666/EastAsia-4x7B-Moe-experiment
118
+ name: Open LLM Leaderboard
119
  ---
120
 
121
  # EastAsia-4x7B-Moe-experiment
 
165
  prompt = pipeline.tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
166
  outputs = pipeline(prompt, max_new_tokens=256, do_sample=True, temperature=0.7, top_k=50, top_p=0.95)
167
  print(outputs[0]["generated_text"])
168
+ ```
169
+ # [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
170
+ Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_Heng666__EastAsia-4x7B-Moe-experiment)
171
+
172
+ | Metric |Value|
173
+ |---------------------------------|----:|
174
+ |Avg. |42.12|
175
+ |AI2 Reasoning Challenge (25-Shot)|39.51|
176
+ |HellaSwag (10-Shot) |48.92|
177
+ |MMLU (5-Shot) |56.20|
178
+ |TruthfulQA (0-shot) |49.83|
179
+ |Winogrande (5-shot) |58.09|
180
+ |GSM8k (5-shot) | 0.15|
181
+