maximuspowers
commited on
Update README.md
Browse files
README.md
CHANGED
@@ -1,4 +1,57 @@
|
|
1 |
-
|
2 |
-
|
3 |
-
|
4 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
---
|
2 |
+
library_name: transformers
|
3 |
+
datasets:
|
4 |
+
- vector-institute/newsmediabias-plus
|
5 |
+
language:
|
6 |
+
- en
|
7 |
+
metrics:
|
8 |
+
- accuracy
|
9 |
+
- precision
|
10 |
+
- recall
|
11 |
+
- f1
|
12 |
+
base_model:
|
13 |
+
- google-bert/bert-base-uncased
|
14 |
+
pipeline_tag: text-classification
|
15 |
+
---
|
16 |
+
|
17 |
+
# BERT NMB+ (Disinformation Sequence Classification):
|
18 |
+
|
19 |
+
Classifies 512 chunks of a news article as "Likely" or "Unlikely" biased/disinformation.
|
20 |
+
|
21 |
+
Fine-tuned BERT ([bert-base-uncased](https://huggingface.co/google-bert/bert-base-uncased)) on the `headline`, `aritcle_text` and `text_label` fields in the [News Media Bias Plus Dataset](https://huggingface.co/datasets/vector-institute/newsmediabias-plus).
|
22 |
+
|
23 |
+
**This model was trained with weighted sampling so that each batch contains 50% 'Likely' examples and 50% 'Unlikely' examples.** The same model trained without weighted sampling is here, and got slightly better taining eval metrics. However, this model preformed better when predictions were evaluated by gpt-4o as a judge.
|
24 |
+
|
25 |
+
### Metics
|
26 |
+
|
27 |
+
*Evaluated on a 0.1 random sample of the NMB+ dataset, unseen during training*
|
28 |
+
|
29 |
+
- Accuracy: 0.7597
|
30 |
+
- Precision: 0.9223
|
31 |
+
- Recall: 0.7407
|
32 |
+
- F1 Score: 0.8216
|
33 |
+
|
34 |
+
## How to Use:
|
35 |
+
|
36 |
+
*Keep in mind, this model was trained on full 512 token chunks (tends to over-predict Unlikely for standalone sentences). If you're planning on processing stand alone sentences, you may find better results with this NMB+ model, which was trained on biased headlines.*
|
37 |
+
|
38 |
+
```
|
39 |
+
from transformers import pipeline
|
40 |
+
|
41 |
+
classifier = pipeline("text-classification", model="maximuspowers/nmbp-bert-full-articles-balanced")
|
42 |
+
result = classifier("He was a terrible politician.", top_k=2)
|
43 |
+
```
|
44 |
+
|
45 |
+
### Example Response:
|
46 |
+
```
|
47 |
+
[
|
48 |
+
{
|
49 |
+
'label': 'Likely',
|
50 |
+
'score': 0.9967995882034302
|
51 |
+
},
|
52 |
+
{
|
53 |
+
'label': 'Unlikely',
|
54 |
+
'score': 0.003200419945642352
|
55 |
+
}
|
56 |
+
]
|
57 |
+
```
|