vihealthbert-w_unsup-SynPD

This model is a fine-tuned version of demdecuong/vihealthbert-base-word on the tmnam20/pretrained-vn-med-nli all dataset. It achieves the following results on the evaluation set:

  • Loss: 1.5579
  • Accuracy: 0.6891

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 3e-05
  • train_batch_size: 32
  • eval_batch_size: 16
  • seed: 19144
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: linear
  • lr_scheduler_warmup_ratio: 0.1
  • num_epochs: 10.0

Training results

Training Loss Epoch Step Validation Loss Accuracy
5.8543 0.3446 2000 3.8967 0.3950
3.4544 0.6893 4000 2.8119 0.5306
2.8312 1.0339 6000 2.4040 0.5771
2.5914 1.3786 8000 2.6482 0.5350
2.5649 1.7232 10000 2.1335 0.6087
2.2749 2.0679 12000 1.9895 0.6282
2.1572 2.4125 14000 1.9313 0.6353
2.1009 2.7572 16000 1.8646 0.6429
2.0609 3.1018 18000 1.8572 0.6450
2.0885 3.4465 20000 1.9489 0.6285
1.9891 3.7911 22000 1.7700 0.6583
1.9368 4.1358 24000 1.7398 0.6609
1.9003 4.4804 26000 1.7165 0.6664
1.9058 4.8251 28000 1.7032 0.6670
1.859 5.1697 30000 1.6771 0.6718
1.8401 5.5144 32000 1.6652 0.6710
1.8181 5.8590 34000 1.6417 0.6754
1.8133 6.2037 36000 1.6431 0.6748
1.7888 6.5483 38000 1.6363 0.6755
1.7811 6.8930 40000 1.6205 0.6793
1.7481 7.2376 42000 1.6190 0.6807
1.7509 7.5823 44000 1.6142 0.6794
1.7517 7.9269 46000 1.5949 0.6819
1.7358 8.2716 48000 1.5909 0.6843
1.7287 8.6162 50000 1.5757 0.6851
1.7132 8.9609 52000 1.5671 0.6885
1.7042 9.3055 54000 1.5685 0.6867
1.7051 9.6502 56000 1.5609 0.6876
1.7051 9.9948 58000 1.5576 0.6883

Framework versions

  • Transformers 4.40.2
  • Pytorch 2.2.0+cu121
  • Datasets 2.19.1
  • Tokenizers 0.19.1
Downloads last month
13
Safetensors
Model size
135M params
Tensor type
F32
·
Inference Examples
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social visibility and check back later, or deploy to Inference Endpoints (dedicated) instead.

Model tree for tmnam/vihealthbert-w_unsup-SynPD

Finetuned
(3)
this model

Dataset used to train tmnam/vihealthbert-w_unsup-SynPD

Evaluation results