Adding Evaluation Results (#8)

- Adding Evaluation Results (ba4cb820ce7b0e3dfc7b99575aff11d07b843b1d)


Co-authored-by: Yağız Çalık <[email protected]>

README.md

@@ -1,5 +1,108 @@
 ---
 license: mit
+model-index:
+- name: miqu-1-70b-sf
+  results:
+  - task:
+      type: text-generation
+      name: Text Generation
+    dataset:
+      name: AI2 Reasoning Challenge (25-Shot)
+      type: ai2_arc
+      config: ARC-Challenge
+      split: test
+      args:
+        num_few_shot: 25
+    metrics:
+    - type: acc_norm
+      value: 73.04
+      name: normalized accuracy
+    source:
+      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=152334H/miqu-1-70b-sf
+      name: Open LLM Leaderboard
+  - task:
+      type: text-generation
+      name: Text Generation
+    dataset:
+      name: HellaSwag (10-Shot)
+      type: hellaswag
+      split: validation
+      args:
+        num_few_shot: 10
+    metrics:
+    - type: acc_norm
+      value: 88.61
+      name: normalized accuracy
+    source:
+      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=152334H/miqu-1-70b-sf
+      name: Open LLM Leaderboard
+  - task:
+      type: text-generation
+      name: Text Generation
+    dataset:
+      name: MMLU (5-Shot)
+      type: cais/mmlu
+      config: all
+      split: test
+      args:
+        num_few_shot: 5
+    metrics:
+    - type: acc
+      value: 75.49
+      name: accuracy
+    source:
+      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=152334H/miqu-1-70b-sf
+      name: Open LLM Leaderboard
+  - task:
+      type: text-generation
+      name: Text Generation
+    dataset:
+      name: TruthfulQA (0-shot)
+      type: truthful_qa
+      config: multiple_choice
+      split: validation
+      args:
+        num_few_shot: 0
+    metrics:
+    - type: mc2
+      value: 69.38
+    source:
+      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=152334H/miqu-1-70b-sf
+      name: Open LLM Leaderboard
+  - task:
+      type: text-generation
+      name: Text Generation
+    dataset:
+      name: Winogrande (5-shot)
+      type: winogrande
+      config: winogrande_xl
+      split: validation
+      args:
+        num_few_shot: 5
+    metrics:
+    - type: acc
+      value: 85.32
+      name: accuracy
+    source:
+      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=152334H/miqu-1-70b-sf
+      name: Open LLM Leaderboard
+  - task:
+      type: text-generation
+      name: Text Generation
+    dataset:
+      name: GSM8k (5-shot)
+      type: gsm8k
+      config: main
+      split: test
+      args:
+        num_few_shot: 5
+    metrics:
+    - type: acc
+      value: 67.7
+      name: accuracy
+    source:
+      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=152334H/miqu-1-70b-sf
+      name: Open LLM Leaderboard
 ---
 this is [miqu-1-70b](https://huggingface.co/miqudev/miqu-1-70b), dequantised from q5 to f16 && transposed to pytorch. shapes have been rotated less wrongly than in [alpindale/miqu-1-70b-pytorch](https://huggingface.co/alpindale/miqu-1-70b-pytorch/tree/main)
 
@@ -130,4 +233,17 @@ some benchmarks
 ```
 no i do not know why the stderr is high. plausibly it is due to the vllm backend used. this is my lm-eval command in most cases (works on h100):
 
-`lm_eval --model vllm --model_args pretrained=./miqu-1-70b-sf,tensor_parallel_size=4,dtype=auto,gpu_memory_utilization=0.88,data_parallel_size=2 --tasks mmlu --batch_size 20`
+`lm_eval --model vllm --model_args pretrained=./miqu-1-70b-sf,tensor_parallel_size=4,dtype=auto,gpu_memory_utilization=0.88,data_parallel_size=2 --tasks mmlu --batch_size 20`
+# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
+Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_152334H__miqu-1-70b-sf)
+
+|             Metric              |Value|
+|---------------------------------|----:|
+|Avg.                             |76.59|
+|AI2 Reasoning Challenge (25-Shot)|73.04|
+|HellaSwag (10-Shot)              |88.61|
+|MMLU (5-Shot)                    |75.49|
+|TruthfulQA (0-shot)              |69.38|
+|Winogrande (5-shot)              |85.32|
+|GSM8k (5-shot)                   |67.70|
+