jarodrigues
commited on
Update README.md
Browse files
README.md
CHANGED
@@ -35,7 +35,7 @@ datasets:
|
|
35 |
**Gervásio PT-*** is a **fully open** decoder for the **Portuguese language**.
|
36 |
|
37 |
|
38 |
-
It is a **decoder** of the LLaMA family, based on the neural architecture Transformer and developed over the LLaMA
|
39 |
Its further improvement through additional training was done over language resources that include new instruction data sets of Portuguese prepared for this purpose.
|
40 |
|
41 |
It has different versions that were trained for different variants of Portuguese (PT),
|
@@ -126,8 +126,8 @@ This involves repurposing the tasks in various ways, such as generation of answe
|
|
126 |
| Model | MRPC (F1) | RTE (F1) | COPA (F1) |
|
127 |
|--------------------------|----------------|----------------|-----------|
|
128 |
| **Gervásio 7B PT-BR** | **0.7822** | **0.8321** | 0.2134 |
|
129 |
-
| **LLaMA
|
130 |
-
| **LLaMA
|
131 |
<br>
|
132 |
|
133 |
For further testing our decoder, in addition to the testing data described above, we also reused some of the datasets that had been resorted for American Portuguese to test the state-of-the-art Sabiá model and that were originally developed with materials from Portuguese: ASSIN2 RTE (entailment) and ASSIN2 STS (similarity), BLUEX (question answering), ENEM 2022 (question answering) and FaQuAD (extractive question-answering).
|
@@ -142,8 +142,8 @@ To evaluate Gervásio, the examples were randomly selected to be included in the
|
|
142 |
| Model | ENEM 2022 (Accuracy) | BLUEX (Accuracy)| RTE (F1) | STS (Pearson) |
|
143 |
|--------------------------|----------------------|-----------------|-----------|---------------|
|
144 |
| **Gervásio 7B PT-BR** | 0.1977 | 0.2640 | **0.7469**| **0.2136** |
|
145 |
-
| **LLaMA
|
146 |
-
| **LLaMA
|
147 |
||||||
|
148 |
| **Sabiá-7B** | **0.6017** | **0.7743** | 0.6847 | 0.1363 |
|
149 |
|
|
|
35 |
**Gervásio PT-*** is a **fully open** decoder for the **Portuguese language**.
|
36 |
|
37 |
|
38 |
+
It is a **decoder** of the LLaMA family, based on the neural architecture Transformer and developed over the LLaMA-2 7B model.
|
39 |
Its further improvement through additional training was done over language resources that include new instruction data sets of Portuguese prepared for this purpose.
|
40 |
|
41 |
It has different versions that were trained for different variants of Portuguese (PT),
|
|
|
126 |
| Model | MRPC (F1) | RTE (F1) | COPA (F1) |
|
127 |
|--------------------------|----------------|----------------|-----------|
|
128 |
| **Gervásio 7B PT-BR** | **0.7822** | **0.8321** | 0.2134 |
|
129 |
+
| **LLaMA-2** | 0.0369 | 0.0516 | 0.4867 |
|
130 |
+
| **LLaMA-2 Chat** | 0.5432 | 0.3807 | **0.5493**|
|
131 |
<br>
|
132 |
|
133 |
For further testing our decoder, in addition to the testing data described above, we also reused some of the datasets that had been resorted for American Portuguese to test the state-of-the-art Sabiá model and that were originally developed with materials from Portuguese: ASSIN2 RTE (entailment) and ASSIN2 STS (similarity), BLUEX (question answering), ENEM 2022 (question answering) and FaQuAD (extractive question-answering).
|
|
|
142 |
| Model | ENEM 2022 (Accuracy) | BLUEX (Accuracy)| RTE (F1) | STS (Pearson) |
|
143 |
|--------------------------|----------------------|-----------------|-----------|---------------|
|
144 |
| **Gervásio 7B PT-BR** | 0.1977 | 0.2640 | **0.7469**| **0.2136** |
|
145 |
+
| **LLaMA-2** | 0.2458 | 0.2903 | 0.0913 | 0.1034 |
|
146 |
+
| **LLaMA-2 Chat** | 0.2231 | 0.2959 | 0.5546 | 0.1750 |
|
147 |
||||||
|
148 |
| **Sabiá-7B** | **0.6017** | **0.7743** | 0.6847 | 0.1363 |
|
149 |
|