Hugging Face's logo

saattrupdan
/
verdict-classifier

Text Classification
Transformers
PyTorch
TensorBoard
Safetensors
xlm-roberta
Generated from Trainer
Inference Endpoints
Model card Files Metrics Community
1
saattrupdan commited on
Commit
c8ddabe
·
1 Parent(s): c6878df

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +57 -14
README.md CHANGED
@@ -82,15 +82,16 @@ widget:
82
 
83
  This model is a fine-tuned version of [xlm-roberta-base](https://huggingface.co/xlm-roberta-base) on 1,500 deduplicated multilingual verdicts from [Google Fact Check Tools API](https://developers.google.com/fact-check/tools/api/reference/rest/v1alpha1/claims/search), translated into 65 languages with the [Google Cloud Translation API](https://cloud.google.com/translate/docs/reference/rest/).
84
  It achieves the following results on the evaluation set, being 1,000 such verdicts, but here including duplicates to represent the true distribution:
85
- - Loss: 0.1856
86
- - F1 Macro: 0.8148
87
- - F1 Misinformation: 0.9764
88
- - F1 Factual: 0.9375
89
- - F1 Other: 0.5306
90
- - Precision Macro: 0.8117
91
- - Precision Misinformation: 0.9775
92
- - Precision Factual: 0.9375
93
- - Precision Other: 0.52
 
94
 
95
  ## Training procedure
96
 
@@ -110,12 +111,54 @@ The following hyperparameters were used during training:
110
 
111
  ### Training results
112
 
113
- | Training Loss | Epoch | Step | Validation Loss | F1 Macro | F1 Misinformation | F1 Factual | F1 Other | Precision Macro | Precision Misinformation | Precision Factual | Precision Other |
114
  |:-------------:|:-----:|:-----:|:---------------:|:--------:|:-----------------:|:----------:|:--------:|:----------:|:-------------------:|:------------:|:----------:|
115
- | 0.8707 | 1.0 | 3758 | 0.2414 | 0.7832 | 0.9639 | 0.7857 | 0.6 | 0.7950 | 0.9683 | 0.9167 | 0.5 |
116
- | 0.3918 | 2.0 | 7516 | 0.1856 | 0.8148 | 0.9764 | 0.9375 | 0.5306 | 0.8117 | 0.9775 | 0.9375 | 0.52 |
117
- | 0.1766 | 3.0 | 11274 | 0.1942 | 0.8394 | 0.9809 | 0.9538 | 0.5833 | 0.8349 | 0.9820 | 0.9394 | 0.5833 |
118
- | 0.1071 | 4.0 | 15032 | 0.2078 | 0.8676 | 0.9786 | 0.9841 | 0.64 | 0.8650 | 0.9797 | 1.0 | 0.6154 |
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
119
 
120
 
121
  ### Framework versions
 
82
 
83
  This model is a fine-tuned version of [xlm-roberta-base](https://huggingface.co/xlm-roberta-base) on 1,500 deduplicated multilingual verdicts from [Google Fact Check Tools API](https://developers.google.com/fact-check/tools/api/reference/rest/v1alpha1/claims/search), translated into 65 languages with the [Google Cloud Translation API](https://cloud.google.com/translate/docs/reference/rest/).
84
  It achieves the following results on the evaluation set, being 1,000 such verdicts, but here including duplicates to represent the true distribution:
85
+ - Loss: 0.2245
86
+ - F1 Macro: 0.8818
87
+ - F1 Misinformation: 0.9842
88
+ - F1 Factual: 0.9688
89
+ - F1 Other: 0.6923
90
+ - Prec Macro: 0.8668
91
+ - Prec Misinformation: 0.9887
92
+ - Prec Factual: 0.9688
93
+ - Prec Other: 0.6429
94
+
95
 
96
  ## Training procedure
97
 
 
111
 
112
  ### Training results
113
 
114
+ | Training Loss | Epoch | Step | Validation Loss | F1 Macro | F1 Misinformation | F1 Factual | F1 Other | Prec Macro | Prec Misinformation | Prec Factual | Prec Other |
115
  |:-------------:|:-----:|:-----:|:---------------:|:--------:|:-----------------:|:----------:|:--------:|:----------:|:-------------------:|:------------:|:----------:|
116
+ | 1.0833 | 0.13 | 500 | 0.9331 | 0.3136 | 0.9408 | 0.0 | 0.0 | 0.2961 | 0.8882 | 0.0 | 0.0 |
117
+ | 0.9697 | 0.27 | 1000 | 0.8839 | 0.3136 | 0.9408 | 0.0 | 0.0 | 0.2961 | 0.8882 | 0.0 | 0.0 |
118
+ | 0.9406 | 0.4 | 1500 | 0.7238 | 0.3591 | 0.9394 | 0.0 | 0.1379 | 0.4304 | 0.8911 | 0.0 | 0.4 |
119
+ | 0.9183 | 0.53 | 2000 | 0.6494 | 0.3482 | 0.9336 | 0.0 | 0.1111 | 0.3528 | 0.8916 | 0.0 | 0.1667 |
120
+ | 0.9103 | 0.67 | 2500 | 0.5686 | 0.3621 | 0.9323 | 0.0 | 0.1538 | 0.3643 | 0.8930 | 0.0 | 0.2 |
121
+ | 0.8637 | 0.8 | 3000 | 0.5720 | 0.4233 | 0.9367 | 0.0 | 0.3333 | 0.4036 | 0.9108 | 0.0 | 0.3 |
122
+ | 0.7905 | 0.93 | 3500 | 0.3918 | 0.4721 | 0.9418 | 0.0 | 0.4746 | 0.4402 | 0.9206 | 0.0 | 0.4 |
123
+ | 0.6625 | 1.06 | 4000 | 0.2771 | 0.8225 | 0.9666 | 0.9375 | 0.5634 | 0.7853 | 0.9929 | 0.9375 | 0.4255 |
124
+ | 0.5403 | 1.2 | 4500 | 0.2187 | 0.8083 | 0.9703 | 0.9062 | 0.5484 | 0.7799 | 0.9861 | 0.9062 | 0.4474 |
125
+ | 0.4837 | 1.33 | 5000 | 0.1708 | 0.8138 | 0.9728 | 0.9231 | 0.5455 | 0.7916 | 0.9817 | 0.9091 | 0.4839 |
126
+ | 0.4411 | 1.46 | 5500 | 0.1734 | 0.7873 | 0.9707 | 0.9206 | 0.4706 | 0.7843 | 0.9729 | 0.9355 | 0.4444 |
127
+ | 0.3991 | 1.6 | 6000 | 0.1671 | 0.8131 | 0.9705 | 0.9355 | 0.5333 | 0.7976 | 0.9816 | 0.9667 | 0.4444 |
128
+ | 0.344 | 1.73 | 6500 | 0.1719 | 0.7989 | 0.9749 | 0.8955 | 0.5263 | 0.7667 | 0.9885 | 0.8571 | 0.4545 |
129
+ | 0.3005 | 1.86 | 7000 | 0.1855 | 0.8052 | 0.9704 | 0.9206 | 0.5246 | 0.7839 | 0.9838 | 0.9355 | 0.4324 |
130
+ | 0.2638 | 2.0 | 7500 | 0.1802 | 0.7896 | 0.9752 | 0.9231 | 0.4706 | 0.7777 | 0.9796 | 0.9091 | 0.4444 |
131
+ | 0.2362 | 2.13 | 8000 | 0.1752 | 0.7762 | 0.9718 | 0.8986 | 0.4583 | 0.7578 | 0.9773 | 0.8378 | 0.4583 |
132
+ | 0.2077 | 2.26 | 8500 | 0.1739 | 0.8101 | 0.9740 | 0.9206 | 0.5357 | 0.7953 | 0.9817 | 0.9355 | 0.4688 |
133
+ | 0.1858 | 2.39 | 9000 | 0.1986 | 0.8073 | 0.9716 | 0.9412 | 0.5091 | 0.7748 | 0.9839 | 0.8889 | 0.4516 |
134
+ | 0.1755 | 2.53 | 9500 | 0.1945 | 0.7872 | 0.9754 | 0.9180 | 0.4681 | 0.8049 | 0.9710 | 0.9655 | 0.4783 |
135
+ | 0.1591 | 2.66 | 10000 | 0.2366 | 0.7880 | 0.9692 | 0.8857 | 0.5091 | 0.7504 | 0.9838 | 0.8158 | 0.4516 |
136
+ | 0.1457 | 2.79 | 10500 | 0.2346 | 0.7614 | 0.9671 | 0.8857 | 0.4314 | 0.7334 | 0.9771 | 0.8158 | 0.4074 |
137
+ | 0.1376 | 2.93 | 11000 | 0.2361 | 0.8015 | 0.9729 | 0.9118 | 0.52 | 0.7802 | 0.9795 | 0.8611 | 0.5 |
138
+ | 0.126 | 3.06 | 11500 | 0.2276 | 0.8331 | 0.9751 | 0.9688 | 0.5556 | 0.8168 | 0.9818 | 0.9688 | 0.5 |
139
+ | 0.1133 | 3.19 | 12000 | 0.2972 | 0.8014 | 0.9727 | 0.9231 | 0.5085 | 0.7746 | 0.9861 | 0.9091 | 0.4286 |
140
+ | 0.1114 | 3.33 | 12500 | 0.2600 | 0.8038 | 0.9705 | 0.8955 | 0.5455 | 0.7742 | 0.9816 | 0.8571 | 0.4839 |
141
+ | 0.1099 | 3.46 | 13000 | 0.3221 | 0.8273 | 0.9738 | 0.9118 | 0.5965 | 0.7882 | 0.9884 | 0.8611 | 0.5152 |
142
+ | 0.1116 | 3.59 | 13500 | 0.2277 | 0.8376 | 0.9775 | 0.9231 | 0.6122 | 0.8296 | 0.9797 | 0.9091 | 0.6 |
143
+ | 0.106 | 3.73 | 14000 | 0.2347 | 0.8148 | 0.9774 | 0.8955 | 0.5714 | 0.7997 | 0.9819 | 0.8571 | 0.56 |
144
+ | 0.098 | 3.86 | 14500 | 0.2337 | 0.8487 | 0.9775 | 0.9688 | 0.6 | 0.8418 | 0.9797 | 0.9688 | 0.5769 |
145
+ | 0.0899 | 3.99 | 15000 | 0.2072 | 0.8636 | 0.9820 | 0.9688 | 0.64 | 0.8561 | 0.9842 | 0.9688 | 0.6154 |
146
+ | 0.0855 | 4.12 | 15500 | 0.2385 | 0.8409 | 0.9762 | 0.9538 | 0.5926 | 0.8189 | 0.9840 | 0.9394 | 0.5333 |
147
+ | 0.0864 | 4.26 | 16000 | 0.2780 | 0.8462 | 0.9774 | 0.9688 | 0.5926 | 0.8287 | 0.9841 | 0.9688 | 0.5333 |
148
+ | 0.0784 | 4.39 | 16500 | 0.2668 | 0.8277 | 0.9776 | 0.9524 | 0.5532 | 0.8361 | 0.9754 | 0.9677 | 0.5652 |
149
+ | 0.0923 | 4.52 | 17000 | 0.2893 | 0.8399 | 0.9738 | 0.9254 | 0.6207 | 0.8012 | 0.9884 | 0.8857 | 0.5294 |
150
+ | 0.0794 | 4.66 | 17500 | 0.3101 | 0.8556 | 0.9773 | 0.9688 | 0.6207 | 0.8289 | 0.9885 | 0.9688 | 0.5294 |
151
+ | 0.082 | 4.79 | 18000 | 0.2245 | 0.8818 | 0.9842 | 0.9688 | 0.6923 | 0.8668 | 0.9887 | 0.9688 | 0.6429 |
152
+ | 0.084 | 4.92 | 18500 | 0.2771 | 0.8247 | 0.9797 | 0.8986 | 0.5957 | 0.8102 | 0.9841 | 0.8378 | 0.6087 |
153
+ | 0.0757 | 5.06 | 19000 | 0.2971 | 0.8594 | 0.9773 | 0.9677 | 0.6333 | 0.8388 | 0.9885 | 1.0 | 0.5278 |
154
+ | 0.0709 | 5.19 | 19500 | 0.3601 | 0.8410 | 0.9774 | 0.9688 | 0.5769 | 0.8288 | 0.9819 | 0.9688 | 0.5357 |
155
+ | 0.0698 | 5.32 | 20000 | 0.2772 | 0.8333 | 0.9762 | 0.9524 | 0.5714 | 0.8173 | 0.9840 | 0.9677 | 0.5 |
156
+ | 0.0652 | 5.45 | 20500 | 0.3397 | 0.8186 | 0.9752 | 0.9524 | 0.5283 | 0.8100 | 0.9796 | 0.9677 | 0.4828 |
157
+ | 0.0735 | 5.59 | 21000 | 0.3027 | 0.8412 | 0.9785 | 0.9524 | 0.5926 | 0.8284 | 0.9841 | 0.9677 | 0.5333 |
158
+ | 0.0746 | 5.72 | 21500 | 0.3122 | 0.8384 | 0.9751 | 0.9688 | 0.5714 | 0.8176 | 0.9840 | 0.9688 | 0.5 |
159
+ | 0.0714 | 5.85 | 22000 | 0.2683 | 0.8381 | 0.9787 | 0.9524 | 0.5833 | 0.8429 | 0.9776 | 0.9677 | 0.5833 |
160
+ | 0.073 | 5.99 | 22500 | 0.2436 | 0.8676 | 0.9786 | 0.9841 | 0.64 | 0.8650 | 0.9797 | 1.0 | 0.6154 |
161
+ | 0.0653 | 6.12 | 23000 | 0.3380 | 0.8559 | 0.9761 | 0.9688 | 0.6230 | 0.8243 | 0.9907 | 0.9688 | 0.5135 |
162
 
163
 
164
  ### Framework versions