updated Evaluation WiP

README.md

@@ -633,7 +633,8 @@ This instructed-tuned variant has been fine-tuned with a collection of 273k inst
 ## Evaluation
 
 ### Gold-standard benchmarks
-
+WiP
+<!--
 Evaluation is done using the Language Model Evaluation Harness (Gao et al., 2024). We evaluate on a set of tasks taken from [SpanishBench](https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/spanish_bench), [CatalanBench](https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/catalan_bench), [BasqueBench](https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/basque_bench) and [GalicianBench](https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/galician_bench). These benchmarks include both new and existing tasks and datasets. Given that this is an instructed model, we add LM Evaluation Harness's native feature of `chat-template` to the setup. In the tables below, we include the results in a selection of evaluation datasets that represent model's performance across a variety of tasks within these benchmarks. 
 
 We only use tasks that are either human generated, human translated, or with a strong human-in-the-loop (i.e., machine translation followed by professional revision or machine generation followed by human revision and annotation). This is the reason behind the variety in number of tasks reported across languages. As more tasks that fulfill these requirements are published, we will update the presented results. We also intend to expand the evaluation to other languages, as long as the datasets meet our quality standards.
@@ -660,36 +661,36 @@ All results reported below are on a 0-shot setting.
     <td>Commonsense Reasoning</td>
     <td>xstorycloze_es</td>
     <td>acc</td>
-    <td>69.29</td>
+    <td>73.13</td>
   </tr>
   <tr>
     <td rowspan="2">NLI</td>
     <td>wnli_es</td>
     <td>acc</td>
-    <td>45.07</td>
+    <td>60.56</td>
   </tr>
   <tr>
     <td>xnli_es</td>
     <td>acc</td>
-    <td>51.49</td>
+    <td>50.84</td>
   </tr>
   <tr>
     <td>Paraphrasing</td>
     <td>paws_es</td>
     <td>acc</td>
-    <td>59.4</td>
+    <td>60.75</td>
   </tr>
   <tr>
     <td>QA</td>
     <td>xquad_es</td>
     <td>acc</td>
-    <td>43.82</td>
+    <td>63.20/td>
   </tr>
   <tr>
     <td>Translation</td>
     <td>flores_es</td>
     <td>bleu</td>
-    <td>22.98</td>
+    <td>14.95</td>
   </tr>
 </tbody>
 </table>
@@ -708,66 +709,66 @@ All results reported below are on a 0-shot setting.
     <td rowspan="2">Commonsense Reasoning</td>
     <td>copa_ca</td>
     <td>acc</td>
-    <td>81.2</td>
+    <td>82.80</td>
   </tr>
   <tr>
     <td>xstorycloze_ca</td>
     <td>acc</td>
-    <td>70.68</td>
+    <td>73.73</td>
   </tr>
   <tr>
     <td rowspan="2">NLI</td>
     <td>wnli_ca</td>
     <td>acc</td>
-    <td>50.7</td>
+    <td>64.79</td>
   </tr>
   <tr>
     <td>xnli_ca</td>
     <td>acc</td>
-    <td>55.14</td>
+    <td>53.45</td>
   </tr>
   <tr>
     <td rowspan="2">Paraphrasing</td>
     <td>parafraseja</td>
     <td>acc</td>
-    <td>65.18</td>
+    <td>64.15</td>
   </tr>
   <tr>
     <td>paws_ca</td>
     <td>acc</td>
-    <td>62.95</td>
+    <td>64.35</td>
   </tr>
   <tr>
     <td rowspan="5">QA</td>
     <td>arc_ca_easy</td>
     <td>acc</td>
-    <td>64.98</td>
+    <td>73.57</td>
   </tr>
   <tr>
     <td>arc_ca_challenge</td>
     <td>acc</td>
-    <td>41.89</td>
+    <td>45.90</td>
   </tr>
   <tr>
     <td>openbookqa_ca</td>
     <td>acc</td>
-    <td>35.2</td>
+    <td>40.60</td>
   </tr>
   <tr>
     <td>piqa_ca</td>
     <td>acc</td>
-    <td>69.53</td>
+    <td>73.39</td>
   </tr>
   <tr>
     <td>siqa_ca</td>
     <td>acc</td>
-    <td>48.62</td>
+    <td>51.84</td>
   </tr>
   <tr>
     <td>Translation</td>
     <td>flores_ca</td>
     <td>bleu</td>
-    <td>28.65</td>
+    <td>20.49</td>
   </tr>
 </tbody></table>
 
@@ -785,51 +786,51 @@ All results reported below are on a 0-shot setting.
     <td rowspan="2">Commonsense Reasoning</td>
     <td>xcopa_eu</td>
     <td>acc</td>
-    <td>61.6</td>
+    <td>67.80</td>
   </tr>
   <tr>
     <td>xstorycloze_eu</td>
     <td>acc</td>
-    <td>61.15</td>
+    <td>65.06</td>
   </tr>
   <tr>
     <td rowspan="2">NLI</td>
     <td>wnli_eu</td>
     <td>acc</td>
-    <td>45.07</td>
+    <td>56.34</td>
   </tr>
   <tr>
     <td>xnli_eu</td>
     <td>acc</td>
-    <td>46.81</td>
+    <td>47.34</td>
   </tr>
   <tr>
     <td rowspan="3">QA</td>
     <td>eus_exams</td>
     <td>acc</td>
-    <td>39.09</td>
+    <td>45.98</td>
   </tr>
   <tr>
     <td>eus_proficiency</td>
     <td>acc</td>
-    <td>36.93</td>
+    <td>43.92</td>
   </tr>
   <tr>
     <td>eus_trivia</td>
     <td>acc</td>
-    <td>46.94</td>
+    <td>50.38</td>
   </tr>
   <tr>
     <td>Reading Comprehension</td>
     <td>eus_reading</td>
     <td>acc</td>
-    <td>45.45</td>
+    <td>48.01</td>
   </tr>
   <tr>
     <td>Translation</td>
     <td>flores_eu</td>
     <td>bleu</td>
-    <td>14.89</td>
+    <td>10.99</td>
   </tr>
 </tbody></table>
 
@@ -847,27 +848,28 @@ All results reported below are on a 0-shot setting.
     <td rowspan="2">Paraphrasing</td>
     <td>parafrases_gl</td>
     <td>acc</td>
-    <td>55.44</td>
+    <td>58.50</td>
   </tr>
   <tr>
     <td>paws_gl</td>
     <td>acc</td>
-    <td>56.55</td>
+    <td>62.45</td>
   </tr>
   <tr>
     <td>QA</td>
     <td>openbookqa_gl</td>
     <td>acc</td>
-    <td>38.4</td>
+    <td>37.20</td>
   </tr>
   <tr>
     <td>Translation</td>
     <td>flores_gl</td>
     <td>bleu</td>
-    <td>27.03</td>
+    <td>18.81</td>
   </tr>
 </tbody>
 </table>
+-->
 
 ### LLM-as-a-judge