metadata

license: apache-2.0
tags:
  - generated_from_trainer
metrics:
  - f1
  - auc
model-index:
  - name: pretrained_model
    results:
      - task:
          name: Text Classification
          type: text-classification
        metrics:
          - name: F1
            type: f1
            value: 0.6356
          - name: AUC
            type: auc
            value: 0.7643
widget:
  - text: >-
      I have trouble understanding what other people think or feel. I also like
      numbers, and finding patterns in numbers.

This model is a hybrid fine-tuned version of distilbert-base-uncased on Reddit dataset contains text related to mental health reports of users. it predicts mental health disorders from textual content.

It achieves the following results on the validation set:

Loss: 0.1873
F1: 0.6356
AUC: 0.7643
Precision: 0.7671

Description

This model is based on an existing lighter variation of BERT (distilBERT), in order to predict different mental disorders.

It is using combinded features of sentiments and emotions (distilbert-base-uncased-finetuned-sst-2-english and roberta-base-go_emotions).
It is trained on a costume dataset of texts or posts (from Reddit) about general experiences of users with mental health problems.
All direct mentions of the disorder names in the texts were removed.

It includes the following classes:

Borderline
Anxiety
Depression
Bipolar
OCD
ADHD
Schizophrenia
Asperger
PTSD

Training

Train size: 90%
Val size: 10%

Training set class counts (text samples) after balancing:
Borderline: 10398
Anxiety: 10393
Depression: 10400
Bipolar: 10359
OCD: 10413
ADHD: 10412
Schizophrenia: 10447
Asperger: 10470
PTSD: 10489

Validation set class counts after balancing:
Borderline: 1180
Anxiety: 1185
Depression: 1178
Bipolar: 1219
OCD: 1165
ADHD: 1166
Schizophrenia: 1131
Asperger: 1108
PTSD: 1089

model-finetuning: distilbert/distilbert-base-uncased

additional features (GoEmotions - SamLowe/roberta-base-go_emotions + SST2 - distilbert/distilbert-base-uncased-finetuned-sst-2-english):
negative, positive, admiration, amusement, anger, annoyance, approval, caring, confusion, curiosity,
desire, disappointment, disapproval, disgust, embarrassment, excitement, fear, gratitude, grief,
joy, love, nervousness, optimism, pride, realization, relief, remorse, sadness, surprise, neutral

The following hyperparameters were used during training:

learning_rate: 1e-5
train_batch_size: 64
val_batch_size: 64
weight_decay: 0.01
optimizer: AdamW
num_epochs: 2-3

Training results

Epoch	Training Loss	Validation Loss
1.0	0.2660	0.2031
2.0	0.1891	0.1872

F1 Score: 0.6355
AUC Score: 0.7642

Classification Report

Borderline:
Precision: 0.7606
Recall: 0.4525
F1-score: 0.5674

Anxiety:
Precision: 0.7063
Recall: 0.5459
F1-score: 0.6158

Depression:
Precision: 0.7286
Recall: 0.4626
F1-score: 0.5659

Bipolar:
Precision: 0.7997
Recall: 0.4487
F1-score: 0.5748

OCD:
Precision: 0.8222
Recall: 0.5957
F1-score: 0.6908

ADHD:
Precision: 0.8856
Recall: 0.5711
F1-score: 0.6944

Schizophrenia:
Precision: 0.7540
Recall: 0.6153
F1-score: 0.6777

Asperger:
Precision: 0.6743
Recall: 0.6335
F1-score: 0.6533

PTSD: Precision: 0.7724
Recall: 0.6235
F1-score: 0.6900