Update README.md
Browse files
README.md
CHANGED
|
@@ -2,4 +2,31 @@
|
|
| 2 |
license: apache-2.0
|
| 3 |
---
|
| 4 |
|
| 5 |
-
WIP
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2 |
license: apache-2.0
|
| 3 |
---
|
| 4 |
|
| 5 |
+
WIP
|
| 6 |
+
|
| 7 |
+
|
| 8 |
+
This is an early experiment on laser and its influence on language understanding.
|
| 9 |
+
|
| 10 |
+
Will keep you updated..
|
| 11 |
+
|
| 12 |
+
### German benchmarks
|
| 13 |
+
|
| 14 |
+
| **German tasks:** | **MMLU-DE** | **Hellaswag-DE** | **ARC-DE** |**Average** |
|
| 15 |
+
|-------------------------------|-------------|---------------|--------------|--------------|
|
| 16 |
+
| **Models / Few-shots:** | _(5 shots)_ | _(10 shots)_ | _(24 shots)_ | |
|
| 17 |
+
| _7B parameters_ | | | | |
|
| 18 |
+
| llama-2-7b | 0.400 | 0.513 | 0.381 | 0.431 |
|
| 19 |
+
| leo-hessianai-7b | 0.400 | 0.609 | 0.429 | 0.479 |
|
| 20 |
+
| bloom-6b4-clp-german | 0.274 | 0.550 | 0.351 | 0.392 |
|
| 21 |
+
| mistral-7b | **0.524** | 0.588 | 0.473 | 0.528 |
|
| 22 |
+
| leo-mistral-hessianai-7b | 0.481 | 0.663 | 0.485 | 0.543 |
|
| 23 |
+
| leo-mistral-hessianai-7b-chat | 0.458 | 0.617 | 0.465 | 0.513 |
|
| 24 |
+
| DPOpenHermes-7B-v2 | 0.517 | 0.603 | 0.515 | 0.545 |
|
| 25 |
+
| hermeo-7b | 0.511 | **0.668** | **0.528** | **0.569** |
|
| 26 |
+
| **germeo-7b-laser (this model)**| ? | ? | ? | ? |
|
| 27 |
+
| _13B parameters_ | | | | |
|
| 28 |
+
| llama-2-13b | 0.469 | 0.581 | 0.468 | 0.506 |
|
| 29 |
+
| leo-hessianai-13b | **0.486** | **0.658** | **0.509** | **0.551** |
|
| 30 |
+
| _70B parameters_ | | | | |
|
| 31 |
+
| llama-2-70b | 0.597 | 0.674 | 0.561 | 0.611 |
|
| 32 |
+
| leo-hessianai-70b | **0.653** | **0.721** | **0.600** | **0.658** |
|