maximuspowers commited on
Commit
329eb21
·
verified ·
1 Parent(s): ef58703

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +57 -4
README.md CHANGED
@@ -1,4 +1,57 @@
1
- Accuracy: 0.7597
2
- Precision: 0.9223
3
- Recall: 0.7407
4
- F1 Score: 0.8216
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ library_name: transformers
3
+ datasets:
4
+ - vector-institute/newsmediabias-plus
5
+ language:
6
+ - en
7
+ metrics:
8
+ - accuracy
9
+ - precision
10
+ - recall
11
+ - f1
12
+ base_model:
13
+ - google-bert/bert-base-uncased
14
+ pipeline_tag: text-classification
15
+ ---
16
+
17
+ # BERT NMB+ (Disinformation Sequence Classification):
18
+
19
+ Classifies 512 chunks of a news article as "Likely" or "Unlikely" biased/disinformation.
20
+
21
+ Fine-tuned BERT ([bert-base-uncased](https://huggingface.co/google-bert/bert-base-uncased)) on the `headline`, `aritcle_text` and `text_label` fields in the [News Media Bias Plus Dataset](https://huggingface.co/datasets/vector-institute/newsmediabias-plus).
22
+
23
+ **This model was trained with weighted sampling so that each batch contains 50% 'Likely' examples and 50% 'Unlikely' examples.** The same model trained without weighted sampling is here, and got slightly better taining eval metrics. However, this model preformed better when predictions were evaluated by gpt-4o as a judge.
24
+
25
+ ### Metics
26
+
27
+ *Evaluated on a 0.1 random sample of the NMB+ dataset, unseen during training*
28
+
29
+ - Accuracy: 0.7597
30
+ - Precision: 0.9223
31
+ - Recall: 0.7407
32
+ - F1 Score: 0.8216
33
+
34
+ ## How to Use:
35
+
36
+ *Keep in mind, this model was trained on full 512 token chunks (tends to over-predict Unlikely for standalone sentences). If you're planning on processing stand alone sentences, you may find better results with this NMB+ model, which was trained on biased headlines.*
37
+
38
+ ```
39
+ from transformers import pipeline
40
+
41
+ classifier = pipeline("text-classification", model="maximuspowers/nmbp-bert-full-articles-balanced")
42
+ result = classifier("He was a terrible politician.", top_k=2)
43
+ ```
44
+
45
+ ### Example Response:
46
+ ```
47
+ [
48
+ {
49
+ 'label': 'Likely',
50
+ 'score': 0.9967995882034302
51
+ },
52
+ {
53
+ 'label': 'Unlikely',
54
+ 'score': 0.003200419945642352
55
+ }
56
+ ]
57
+ ```