---
base_model: allenai/biomed_roberta_base
tags:
- medical transcriptions
- healthcare
model-index:
- name: clinical_transcripts_roberta
  results: []

widget:
- fill-mask:
    mask_token: <mask>
    prompt: "General endotracheal <mask> was induced without incident. Preoperative antibiotics were given for prophylaxis  against surgical infection."
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# clinical_transcripts_roberta

This model is a fine-tuned version of [allenai/biomed_roberta_base](https://huggingface.co/allenai/biomed_roberta_base) on [medical transcriptions dataset](https://www.kaggle.com/datasets/tboyle10/medicaltranscriptions).
It achieves the following results on the evaluation set:
- Loss: 1.0331

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 0.0005
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_ratio: 0.1
- lr_scheduler_warmup_steps: 100
- training_steps: 4000

### Training results

| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:-----:|:----:|:---------------:|
| 1.405         | 0.51  | 100  | 1.2925          |
| 1.316         | 1.01  | 200  | 1.2107          |
| 1.2781        | 1.52  | 300  | 1.1704          |
| 1.2911        | 2.02  | 400  | 1.1745          |
| 1.2241        | 2.53  | 500  | 1.1730          |
| 1.2063        | 3.03  | 600  | 1.1248          |
| 1.174         | 3.54  | 700  | 1.1416          |
| 1.1588        | 4.04  | 800  | 1.1495          |
| 1.1513        | 4.55  | 900  | 1.1145          |
| 1.1541        | 5.05  | 1000 | 1.1402          |
| 1.1266        | 5.56  | 1100 | 1.1156          |
| 1.1205        | 6.06  | 1200 | 1.1075          |
| 1.1141        | 6.57  | 1300 | 1.1157          |
| 1.0956        | 7.07  | 1400 | 1.1047          |
| 1.0809        | 7.58  | 1500 | 1.0921          |
| 1.0755        | 8.08  | 1600 | 1.0891          |
| 1.044         | 8.59  | 1700 | 1.0758          |
| 1.1103        | 9.09  | 1800 | 1.0881          |
| 1.0578        | 9.6   | 1900 | 1.0578          |
| 1.0462        | 10.1  | 2000 | 1.1043          |
| 1.0302        | 10.61 | 2100 | 1.0787          |
| 1.0236        | 11.11 | 2200 | 1.0841          |
| 1.0371        | 11.62 | 2300 | 1.0904          |
| 1.0178        | 12.12 | 2400 | 1.0593          |
| 0.999         | 12.63 | 2500 | 1.0661          |
| 0.9867        | 13.13 | 2600 | 1.0670          |
| 0.9986        | 13.64 | 2700 | 1.0470          |
| 0.9867        | 14.14 | 2800 | 1.0347          |
| 0.9848        | 14.65 | 2900 | 1.0274          |
| 0.9627        | 15.15 | 3000 | 1.0550          |
| 0.9659        | 15.66 | 3100 | 1.0499          |
| 0.9743        | 16.16 | 3200 | 1.0419          |
| 0.9507        | 16.67 | 3300 | 1.0679          |
| 0.941         | 17.17 | 3400 | 1.0142          |
| 0.9548        | 17.68 | 3500 | 1.0422          |
| 0.9378        | 18.18 | 3600 | 1.0471          |
| 0.9339        | 18.69 | 3700 | 1.0473          |
| 0.9195        | 19.19 | 3800 | 1.0248          |
| 0.9254        | 19.7  | 3900 | 1.0235          |
| 0.9393        | 20.2  | 4000 | 1.0331          |


### Framework versions

- Transformers 4.34.0
- Pytorch 2.0.1+cu118
- Datasets 2.14.5
- Tokenizers 0.14.0