abhinavnmagic
commited on
Update README.md
Browse files
README.md
CHANGED
@@ -124,14 +124,10 @@ model.save_pretrained("Meta-Llama-3.1-405B-Instruct-quantized.w4a16")
|
|
124 |
|
125 |
## Evaluation
|
126 |
|
127 |
-
The model was evaluated on
|
128 |
-
|
129 |
-
|
130 |
-
|
131 |
-
--model_args pretrained="neuralmagic/Meta-Llama-3.1-405B-Instruct-quantized.w4a16",dtype=auto,gpu_memory_utilization=0.4,add_bos_token=True,max_model_len=4096,tensor_parallel_size=1 \
|
132 |
-
--tasks openllm \
|
133 |
-
--batch_size auto
|
134 |
-
```
|
135 |
|
136 |
### Accuracy
|
137 |
|
|
|
124 |
|
125 |
## Evaluation
|
126 |
|
127 |
+
The model was evaluated on MMLU, ARC-Challenge, GSM-8K, Hellaswag, Winogrande and TruthfulQA.
|
128 |
+
Evaluation was conducted using the Neural Magic fork of [lm-evaluation-harness](https://github.com/neuralmagic/lm-evaluation-harness/tree/llama_3.1_instruct) (branch llama_3.1_instruct) and the [vLLM](https://docs.vllm.ai/en/stable/) engine.
|
129 |
+
This version of the lm-evaluation-harness includes versions of ARC-Challenge, GSM-8K, and MMLU that match the prompting style of [Meta-Llama-3.1-Instruct-evals](https://huggingface.co/datasets/meta-llama/Meta-Llama-3.1-8B-Instruct-evals).
|
130 |
+
|
|
|
|
|
|
|
|
|
131 |
|
132 |
### Accuracy
|
133 |
|