Hugging Face's logo

saattrupdan
/
verdict-classifier

Text Classification
Transformers
PyTorch
TensorBoard
Safetensors
xlm-roberta
Generated from Trainer
Inference Endpoints
Model card Files Metrics Community
1
verdict-classifier / README.md
saattrupdan's picture
saattrupdan
Update README.md
c8ddabe
preview code
|
raw
history blame
10.1 kB
metadata
license: mit
language:
  - am
  - ar
  - hy
  - eu
  - bn
  - bs
  - bg
  - my
  - hr
  - ca
  - cs
  - da
  - nl
  - en
  - et
  - fi
  - fr
  - ka
  - de
  - el
  - gu
  - ht
  - iw
  - hi
  - hu
  - is
  - in
  - it
  - ja
  - kn
  - km
  - ko
  - lo
  - lv
  - lt
  - ml
  - mr
  - ne
  - 'no'
  - or
  - pa
  - ps
  - fa
  - pl
  - pt
  - ro
  - ru
  - sr
  - zh
  - sd
  - si
  - sk
  - sl
  - es
  - sv
  - tl
  - ta
  - te
  - th
  - tr
  - uk
  - ur
  - ug
  - vi
  - cy
tags:
  - generated_from_trainer
model-index:
  - name: verdict-classifier-en
    results:
      - task:
          type: text-classification
          name: Verdict Classification
widget:
  - One might think that this is true, but it's taken out of context.

Multilingual Verdict Classifier

This model is a fine-tuned version of xlm-roberta-base on 1,500 deduplicated multilingual verdicts from Google Fact Check Tools API, translated into 65 languages with the Google Cloud Translation API. It achieves the following results on the evaluation set, being 1,000 such verdicts, but here including duplicates to represent the true distribution:

  • Loss: 0.2245
  • F1 Macro: 0.8818
  • F1 Misinformation: 0.9842
  • F1 Factual: 0.9688
  • F1 Other: 0.6923
  • Prec Macro: 0.8668
  • Prec Misinformation: 0.9887
  • Prec Factual: 0.9688
  • Prec Other: 0.6429

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 2e-05
  • train_batch_size: 4
  • eval_batch_size: 4
  • seed: 42
  • gradient_accumulation_steps: 8
  • total_train_batch_size: 32
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: linear
  • lr_scheduler_warmup_steps: 30066
  • num_epochs: 1000

Training results

Training Loss Epoch Step Validation Loss F1 Macro F1 Misinformation F1 Factual F1 Other Prec Macro Prec Misinformation Prec Factual Prec Other
1.0833 0.13 500 0.9331 0.3136 0.9408 0.0 0.0 0.2961 0.8882 0.0 0.0
0.9697 0.27 1000 0.8839 0.3136 0.9408 0.0 0.0 0.2961 0.8882 0.0 0.0
0.9406 0.4 1500 0.7238 0.3591 0.9394 0.0 0.1379 0.4304 0.8911 0.0 0.4
0.9183 0.53 2000 0.6494 0.3482 0.9336 0.0 0.1111 0.3528 0.8916 0.0 0.1667
0.9103 0.67 2500 0.5686 0.3621 0.9323 0.0 0.1538 0.3643 0.8930 0.0 0.2
0.8637 0.8 3000 0.5720 0.4233 0.9367 0.0 0.3333 0.4036 0.9108 0.0 0.3
0.7905 0.93 3500 0.3918 0.4721 0.9418 0.0 0.4746 0.4402 0.9206 0.0 0.4
0.6625 1.06 4000 0.2771 0.8225 0.9666 0.9375 0.5634 0.7853 0.9929 0.9375 0.4255
0.5403 1.2 4500 0.2187 0.8083 0.9703 0.9062 0.5484 0.7799 0.9861 0.9062 0.4474
0.4837 1.33 5000 0.1708 0.8138 0.9728 0.9231 0.5455 0.7916 0.9817 0.9091 0.4839
0.4411 1.46 5500 0.1734 0.7873 0.9707 0.9206 0.4706 0.7843 0.9729 0.9355 0.4444
0.3991 1.6 6000 0.1671 0.8131 0.9705 0.9355 0.5333 0.7976 0.9816 0.9667 0.4444
0.344 1.73 6500 0.1719 0.7989 0.9749 0.8955 0.5263 0.7667 0.9885 0.8571 0.4545
0.3005 1.86 7000 0.1855 0.8052 0.9704 0.9206 0.5246 0.7839 0.9838 0.9355 0.4324
0.2638 2.0 7500 0.1802 0.7896 0.9752 0.9231 0.4706 0.7777 0.9796 0.9091 0.4444
0.2362 2.13 8000 0.1752 0.7762 0.9718 0.8986 0.4583 0.7578 0.9773 0.8378 0.4583
0.2077 2.26 8500 0.1739 0.8101 0.9740 0.9206 0.5357 0.7953 0.9817 0.9355 0.4688
0.1858 2.39 9000 0.1986 0.8073 0.9716 0.9412 0.5091 0.7748 0.9839 0.8889 0.4516
0.1755 2.53 9500 0.1945 0.7872 0.9754 0.9180 0.4681 0.8049 0.9710 0.9655 0.4783
0.1591 2.66 10000 0.2366 0.7880 0.9692 0.8857 0.5091 0.7504 0.9838 0.8158 0.4516
0.1457 2.79 10500 0.2346 0.7614 0.9671 0.8857 0.4314 0.7334 0.9771 0.8158 0.4074
0.1376 2.93 11000 0.2361 0.8015 0.9729 0.9118 0.52 0.7802 0.9795 0.8611 0.5
0.126 3.06 11500 0.2276 0.8331 0.9751 0.9688 0.5556 0.8168 0.9818 0.9688 0.5
0.1133 3.19 12000 0.2972 0.8014 0.9727 0.9231 0.5085 0.7746 0.9861 0.9091 0.4286
0.1114 3.33 12500 0.2600 0.8038 0.9705 0.8955 0.5455 0.7742 0.9816 0.8571 0.4839
0.1099 3.46 13000 0.3221 0.8273 0.9738 0.9118 0.5965 0.7882 0.9884 0.8611 0.5152
0.1116 3.59 13500 0.2277 0.8376 0.9775 0.9231 0.6122 0.8296 0.9797 0.9091 0.6
0.106 3.73 14000 0.2347 0.8148 0.9774 0.8955 0.5714 0.7997 0.9819 0.8571 0.56
0.098 3.86 14500 0.2337 0.8487 0.9775 0.9688 0.6 0.8418 0.9797 0.9688 0.5769
0.0899 3.99 15000 0.2072 0.8636 0.9820 0.9688 0.64 0.8561 0.9842 0.9688 0.6154
0.0855 4.12 15500 0.2385 0.8409 0.9762 0.9538 0.5926 0.8189 0.9840 0.9394 0.5333
0.0864 4.26 16000 0.2780 0.8462 0.9774 0.9688 0.5926 0.8287 0.9841 0.9688 0.5333
0.0784 4.39 16500 0.2668 0.8277 0.9776 0.9524 0.5532 0.8361 0.9754 0.9677 0.5652
0.0923 4.52 17000 0.2893 0.8399 0.9738 0.9254 0.6207 0.8012 0.9884 0.8857 0.5294
0.0794 4.66 17500 0.3101 0.8556 0.9773 0.9688 0.6207 0.8289 0.9885 0.9688 0.5294
0.082 4.79 18000 0.2245 0.8818 0.9842 0.9688 0.6923 0.8668 0.9887 0.9688 0.6429
0.084 4.92 18500 0.2771 0.8247 0.9797 0.8986 0.5957 0.8102 0.9841 0.8378 0.6087
0.0757 5.06 19000 0.2971 0.8594 0.9773 0.9677 0.6333 0.8388 0.9885 1.0 0.5278
0.0709 5.19 19500 0.3601 0.8410 0.9774 0.9688 0.5769 0.8288 0.9819 0.9688 0.5357
0.0698 5.32 20000 0.2772 0.8333 0.9762 0.9524 0.5714 0.8173 0.9840 0.9677 0.5
0.0652 5.45 20500 0.3397 0.8186 0.9752 0.9524 0.5283 0.8100 0.9796 0.9677 0.4828
0.0735 5.59 21000 0.3027 0.8412 0.9785 0.9524 0.5926 0.8284 0.9841 0.9677 0.5333
0.0746 5.72 21500 0.3122 0.8384 0.9751 0.9688 0.5714 0.8176 0.9840 0.9688 0.5
0.0714 5.85 22000 0.2683 0.8381 0.9787 0.9524 0.5833 0.8429 0.9776 0.9677 0.5833
0.073 5.99 22500 0.2436 0.8676 0.9786 0.9841 0.64 0.8650 0.9797 1.0 0.6154
0.0653 6.12 23000 0.3380 0.8559 0.9761 0.9688 0.6230 0.8243 0.9907 0.9688 0.5135

Framework versions

  • Transformers 4.11.3
  • Pytorch 1.9.0+cu102
  • Datasets 1.9.0
  • Tokenizers 0.10.2