File size: 4,441 Bytes
0d576bb 6b290b1 f31107f 6b290b1 f31107f 6b290b1 f31107f 6b290b1 f31107f 6b290b1 8ead19d 9d49652 e3e6403 8ead19d 9d49652 4e481ff f31107f 8ead19d e3e6403 8ead19d 9d49652 8ead19d e3e6403 8ead19d e3e6403 8ead19d |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 |
---
license: mit
language:
- en
metrics:
- accuracy
- mse
- f1
base_model:
- dmis-lab/biobert-base-cased-v1.2
- google-bert/bert-base-cased
pipeline_tag: text-classification
model-index:
- name: bert-causation-rating-dr1
results:
- task:
type: text-classification
dataset:
name: rating_dr1
type: dataset
metrics:
- name: off by 1 accuracy
type: accuracy
value: 71.1864406779661
- name: mean squared error for ordinal data
type: mse
value: 0.7796610169491526
- name: weighted F1 score
type: f1
value: 0.7164302155262606
- name: Kendall's tau coefficient
type: Kendall's tau
value: 0.8013637826548397
source:
name: Keling Wang
url: https://github.com/Keling-Wang
datasets:
- kelingwang/causation_strength_rating
---
# Model description
This `bert-causation-rating-dr1` model is a fine-tuned [biobert-base-cased-v1.2](https://huggingface.co/dmis-lab/biobert-base-cased-v1.2) model on a small set of manually annotated texts with causation labels. This model is tasked with classifying a sentence into different levels of strength of causation expressed in this sentence.
Before tuning on this dataset, the `biobert-base-cased-v1.2` model is fine-tuned on a dataset containing causation labels from a published paper. This model starts from pre-trained [`kelingwang/bert-causation-rating-pubmed`](https://huggingface.co/kelingwang/bert-causation-rating-pubmed). For more information please view the link and my [GitHub page](https://github.com/Keling-Wang/causation_rating).
The sentences in the dataset were rated independently by two researchers. This `dr1` version is tuned on the set of sentences with labels rated by Rater 1.
# Intended use and limitations
This model is primarily used to rate for the strength of expressed causation in a sentence extracted from a clinical guideline in the field of diabetes mellitus management.
This model predicts strength of causation (SoC) labels based on the text inputs as:
* -1: No correlation or variable relationships mentioned in the sentence.
* 0: There is correlational relationships but not causation in the sentence.
* 1: The sentence expresses weak causation.
* 2: The sentence expresses moderate causation.
* 3: The sentence expresses strong causation.
*NOTE:* The model output is five one-hot logits and will be 0-index based, and the labels will be 0 to 4. It is good to use [this `python` module](https://github.com/Keling-Wang/causation_rating/blob/main/tests/prediction_from_pretrained.py) if one wants to make predictions.
# Performance and hyperparameters
## Test metrics
This model achieves the following results on the test dataset. The test dataset is a 25% held-out stratified split of the entire dataset with `SEED=114514`.
* Loss: 5.2014
* Off-by-1 accuracy: 71.1864
* Off-by-2 accuracy: 90.6780
* MSE for ordinal data: 0.7797
* Weighted F1: 0.7164
* Kendall's Tau: 0.8014
This performance is achieved with the following hyperparameters:
* Learning rate: 7.94278e-05
* Weight decay: 0.111616
* Warmup ratio: 0.301057
* Power of polynomial learning rate scheduler: 2.619975
* Power to the distance measure used in the loss function \alpha: 2.0
## Hyperparameter tuning metrics
During the Bayesian optimization procedure for hyperparameter tuning, this model achieves the best target metric (Off-by-1 accuracy) of *99.1147*, as the result from 4-fold cross-validation procedure based on best hyperparameters.
# Training settings
The following training configurations apply:
* Pre-trained model: `kelingwang/bert-causation-rating-pubmed`
* `seed`: 114514
* `batch_size`: 128
* `epoch`: 8
* `max_length` in `torch.utils.data.Dataset`: 128
* Loss function: the [OLL loss](https://aclanthology.org/2022.coling-1.407/) with a tunable hyperparameter \alpha (Power to the distance measure used in the loss function).
* `lr`: 7.94278e-05
* `weight_decay`: 0.111616
* `warmup_ratio`: 0.301057
* `lr_scheduler_type`: polynomial
* `lr_scheduler_kwargs`: `{"power": 2.619975, "lr_end": 1e-8}`
* Power to the distance measure used in the loss function \alpha: 2.0
# Framework versions and devices
This model is run on a NVIDIA P100 CPU provided by Kaggle.
Framework versions are:
* python==3.10.14
* cuda==12.4
* NVIDIA-SMI==550.90.07
* torch=2.4.0
* transformers==4.45.1
* scikit-learn==1.2.2
* optuna==4.0.0
* nlpaug==1.1.11 |