Zamoranesis's picture
Update README.md
30b1c91
metadata
base_model: allenai/biomed_roberta_base
tags:
  - medical transcriptions
  - healthcare
model-index:
  - name: clinical_transcripts_roberta
    results: []
widget:
  - text: >-
      General endotracheal <mask> was induced without incident. Preoperative
      antibiotics were given for prophylaxis against surgical infection.

clinical_transcripts_roberta

This model is a fine-tuned version of allenai/biomed_roberta_base on medical transcriptions dataset. It achieves the following results on the evaluation set:

  • Loss: 1.0331

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 0.0005
  • train_batch_size: 16
  • eval_batch_size: 16
  • seed: 42
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: linear
  • lr_scheduler_warmup_ratio: 0.1
  • lr_scheduler_warmup_steps: 100
  • training_steps: 4000

Training results

Training Loss Epoch Step Validation Loss
1.405 0.51 100 1.2925
1.316 1.01 200 1.2107
1.2781 1.52 300 1.1704
1.2911 2.02 400 1.1745
1.2241 2.53 500 1.1730
1.2063 3.03 600 1.1248
1.174 3.54 700 1.1416
1.1588 4.04 800 1.1495
1.1513 4.55 900 1.1145
1.1541 5.05 1000 1.1402
1.1266 5.56 1100 1.1156
1.1205 6.06 1200 1.1075
1.1141 6.57 1300 1.1157
1.0956 7.07 1400 1.1047
1.0809 7.58 1500 1.0921
1.0755 8.08 1600 1.0891
1.044 8.59 1700 1.0758
1.1103 9.09 1800 1.0881
1.0578 9.6 1900 1.0578
1.0462 10.1 2000 1.1043
1.0302 10.61 2100 1.0787
1.0236 11.11 2200 1.0841
1.0371 11.62 2300 1.0904
1.0178 12.12 2400 1.0593
0.999 12.63 2500 1.0661
0.9867 13.13 2600 1.0670
0.9986 13.64 2700 1.0470
0.9867 14.14 2800 1.0347
0.9848 14.65 2900 1.0274
0.9627 15.15 3000 1.0550
0.9659 15.66 3100 1.0499
0.9743 16.16 3200 1.0419
0.9507 16.67 3300 1.0679
0.941 17.17 3400 1.0142
0.9548 17.68 3500 1.0422
0.9378 18.18 3600 1.0471
0.9339 18.69 3700 1.0473
0.9195 19.19 3800 1.0248
0.9254 19.7 3900 1.0235
0.9393 20.2 4000 1.0331

Framework versions

  • Transformers 4.34.0
  • Pytorch 2.0.1+cu118
  • Datasets 2.14.5
  • Tokenizers 0.14.0