Commit
·
aba60a4
1
Parent(s):
ee45ad0
Update README.md
Browse files
README.md
CHANGED
@@ -30,7 +30,7 @@ co2_eq_emissions: 100
|
|
30 |
|
31 |
## Model Description
|
32 |
|
33 |
-
We present **QAmemBERT**, which is a [CamemBERT base](https://huggingface.co/camembert-base) fine-tuned for the Question-Answering task for the French language on four French Q&A datasets composed of contexts and questions with their answers inside the context (= SQuAD
|
34 |
All these datasets were concatenated into a single dataset that we called [frenchQA](https://huggingface.co/datasets/CATIE-AQ/frenchQA).
|
35 |
This represents a total of over **221,348 questions/answers pairs used to finetune this model and 6,376 to test it**.
|
36 |
|
@@ -39,14 +39,14 @@ This represents a total of over **221,348 questions/answers pairs used to finetu
|
|
39 |
|
40 |
| Dataset | Format | Train split | Dev split | Test split |
|
41 |
| ----------- | ----------- | ----------- | ----------- | ----------- |
|
42 |
-
| [piaf](https://www.data.gouv.fr/en/datasets/piaf-le-dataset-francophone-de-questions-reponses/)| SQuAD
|
43 |
-
| piaf_v2| SQuAD
|
44 |
-
| [fquad](https://fquad.illuin.tech/)| SQuAD
|
45 |
-
| fquad_v2 | SQuAD
|
46 |
-
| [lincoln/newsquadfr](https://huggingface.co/datasets/lincoln/newsquadfr) | SQuAD
|
47 |
-
| lincoln/newsquadfr_v2 | SQuAD
|
48 |
-
| [pragnakalp/squad_v2_french_translated](https://huggingface.co/datasets/pragnakalp/squad_v2_french_translated)| SQuAD
|
49 |
-
| pragnakalp/squad_v2_french_translated_v2| SQuAD
|
50 |
|
51 |
All these datasets were concatenated into a single dataset that we called [frenchQA](https://huggingface.co/datasets/CATIE-AQ/frenchQA).
|
52 |
|
@@ -58,44 +58,38 @@ The evaluation was carried out using the [**evaluate**](https://pypi.org/project
|
|
58 |
|
59 |
### FQuaD 1.0 (validation)
|
60 |
|
61 |
-
The metric used is
|
62 |
|
63 |
| Model | Exact_match | F1-score |
|
64 |
| ----------- | ----------- | ----------- |
|
65 |
| [etalab-ia/camembert-base-squadFR-fquad-piaf](https://huggingface.co/etalab-ia/camembert-base-squadFR-fquad-piaf) | 53.60 | 78.09 |
|
66 |
| QAmembert (previous version) | 54.26 | 77.87 |
|
67 |
| QAmembert (**this version**) | 53.98 | 78.00 |
|
68 |
-
| QAmembert-large
|
69 |
-
| [fT0](https://huggingface.co/CATIE-AQ/frenchT0) | 41.15 | 65.79 |
|
70 |
|
71 |
-
♪ this model is available on request only
|
72 |
|
73 |
### qwant/squad_fr (validation)
|
74 |
|
75 |
-
The metric used is
|
76 |
|
77 |
| Model | Exact_match | F1-score |
|
78 |
| ----------- | ----------- | ----------- |
|
79 |
| [etalab-ia/camembert-base-squadFR-fquad-piaf](https://huggingface.co/etalab-ia/camembert-base-squadFR-fquad-piaf) | 60.17 | 78.27 |
|
80 |
| QAmembert (previous version) | 60.40 | 77.27 |
|
81 |
| QAmembert (**this version**) | 60.95 | 77.30 |
|
82 |
-
| QAmembert-large
|
83 |
-
| [fT0](https://huggingface.co/CATIE-AQ/frenchT0) | 41.05 | 56.14 |
|
84 |
|
85 |
-
♪ this model is available on request only.
|
86 |
|
87 |
### frenchQA
|
88 |
|
89 |
-
This dataset includes question with no answers in the context. The metric used is
|
90 |
|
91 |
| Model | Exact_match | F1-score | Answer_f1 | NoAnswer_f1 |
|
92 |
| ----------- | ----------- | ----------- | ----------- | ----------- |
|
93 |
| [etalab-ia/camembert-base-squadFR-fquad-piaf](https://huggingface.co/etalab-ia/camembert-base-squadFR-fquad-piaf) | n/a | n/a | n/a | n/a |
|
94 |
| QAmembert (previous version) | 60.28 | 71.29 | 75.92 | 66.65
|
95 |
| QAmembert (**this version**) | **77.14** | 86.88 | 75.66 | 98.11
|
96 |
-
| QAmembert-large
|
97 |
-
|
98 |
-
♪ this model is available on request only.
|
99 |
|
100 |
|
101 |
|
|
|
30 |
|
31 |
## Model Description
|
32 |
|
33 |
+
We present **QAmemBERT**, which is a [CamemBERT base](https://huggingface.co/camembert-base) fine-tuned for the Question-Answering task for the French language on four French Q&A datasets composed of contexts and questions with their answers inside the context (= SQuAD 1.0 format) but also contexts and questions with their answers not inside the context (= SQuAD 2.0 format).
|
34 |
All these datasets were concatenated into a single dataset that we called [frenchQA](https://huggingface.co/datasets/CATIE-AQ/frenchQA).
|
35 |
This represents a total of over **221,348 questions/answers pairs used to finetune this model and 6,376 to test it**.
|
36 |
|
|
|
39 |
|
40 |
| Dataset | Format | Train split | Dev split | Test split |
|
41 |
| ----------- | ----------- | ----------- | ----------- | ----------- |
|
42 |
+
| [piaf](https://www.data.gouv.fr/en/datasets/piaf-le-dataset-francophone-de-questions-reponses/)| SQuAD 1.0 | 9 224 Q & A | X | X |
|
43 |
+
| piaf_v2| SQuAD 2.0 | 9 224 Q & A | X | X |
|
44 |
+
| [fquad](https://fquad.illuin.tech/)| SQuAD 1.0 | 20 731 Q & A | 3 188 Q & A (not used in training because it serves as a test dataset) | 2 189 Q & A (not used in our work because not freely available)|
|
45 |
+
| fquad_v2 | SQuAD 2.0 | 20 731 Q & A | 3 188 Q & A (not used in training because it serves as a test dataset) | X |
|
46 |
+
| [lincoln/newsquadfr](https://huggingface.co/datasets/lincoln/newsquadfr) | SQuAD 1.0 | 1 650 Q & A | 455 Q & A (not used in our work) | X |
|
47 |
+
| lincoln/newsquadfr_v2 | SQuAD 2.0 | 1 650 Q & A | 455 Q & A (not used in our work) | X |
|
48 |
+
| [pragnakalp/squad_v2_french_translated](https://huggingface.co/datasets/pragnakalp/squad_v2_french_translated)| SQuAD 2.0 | 79 069 Q & A | X | X |
|
49 |
+
| pragnakalp/squad_v2_french_translated_v2| SQuAD 2.0 | 79 069 Q & A | X | X |
|
50 |
|
51 |
All these datasets were concatenated into a single dataset that we called [frenchQA](https://huggingface.co/datasets/CATIE-AQ/frenchQA).
|
52 |
|
|
|
58 |
|
59 |
### FQuaD 1.0 (validation)
|
60 |
|
61 |
+
The metric used is SQuAD 1.0.
|
62 |
|
63 |
| Model | Exact_match | F1-score |
|
64 |
| ----------- | ----------- | ----------- |
|
65 |
| [etalab-ia/camembert-base-squadFR-fquad-piaf](https://huggingface.co/etalab-ia/camembert-base-squadFR-fquad-piaf) | 53.60 | 78.09 |
|
66 |
| QAmembert (previous version) | 54.26 | 77.87 |
|
67 |
| QAmembert (**this version**) | 53.98 | 78.00 |
|
68 |
+
| [QAmembert-large](https://huggingface.co/CATIE-AQ/QAmembert-large) | **55.95** | **81.05** |
|
|
|
69 |
|
|
|
70 |
|
71 |
### qwant/squad_fr (validation)
|
72 |
|
73 |
+
The metric used is SQuAD 1.0.
|
74 |
|
75 |
| Model | Exact_match | F1-score |
|
76 |
| ----------- | ----------- | ----------- |
|
77 |
| [etalab-ia/camembert-base-squadFR-fquad-piaf](https://huggingface.co/etalab-ia/camembert-base-squadFR-fquad-piaf) | 60.17 | 78.27 |
|
78 |
| QAmembert (previous version) | 60.40 | 77.27 |
|
79 |
| QAmembert (**this version**) | 60.95 | 77.30 |
|
80 |
+
| [QAmembert-large](https://huggingface.co/CATIE-AQ/QAmembert-large) | **65.58** | **81.74** |
|
|
|
81 |
|
|
|
82 |
|
83 |
### frenchQA
|
84 |
|
85 |
+
This dataset includes question with no answers in the context. The metric used is SQuAD 2.0.
|
86 |
|
87 |
| Model | Exact_match | F1-score | Answer_f1 | NoAnswer_f1 |
|
88 |
| ----------- | ----------- | ----------- | ----------- | ----------- |
|
89 |
| [etalab-ia/camembert-base-squadFR-fquad-piaf](https://huggingface.co/etalab-ia/camembert-base-squadFR-fquad-piaf) | n/a | n/a | n/a | n/a |
|
90 |
| QAmembert (previous version) | 60.28 | 71.29 | 75.92 | 66.65
|
91 |
| QAmembert (**this version**) | **77.14** | 86.88 | 75.66 | 98.11
|
92 |
+
| [QAmembert-large](https://huggingface.co/CATIE-AQ/QAmembert-large) | **77.14** | **88.74** | **78.83** | **98.65**
|
|
|
|
|
93 |
|
94 |
|
95 |
|