Edit model card

training_bert

This model is a fine-tuned version of Bert Base Uncased on dataset composed of different jobs posted in several job platforms and thousands of resumes. It achieves the following results on the evaluation set:

  • Loss: 4.0495

Model description

Pretraining done on bert base architecture.

Intended uses & limitations

This model can be used to generate contextual embeddings for textual data used in Applicant Tracking Systems such as resumes, jobs and cover letters. The embeddings can be further used to perform other NLP downstream tasks such as classification, Named Entity Recognition and so on.

Training and evaluation data

THe training corpus is developed using about 40000 resumes and 2000 jobs posted scrapped from different job portals. This is a preliminary dataset for the experimentation. THe corpus size is about 2.35 GB of textual data. Similary evaluation data contains few resumes and jobs making about 12 mb of textual data.

Training procedure

For the pretraining of masked language model, Trainer API from Huggingface is used. The pretraining took about 6 hrs 40 mins.

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 5e-05
  • train_batch_size: 128
  • eval_batch_size: 128
  • seed: 42
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: linear
  • num_epochs: 5

Training results

Training Loss Epoch Step Validation Loss
6.7862 0.11 500 6.9461
5.9428 0.22 1000 6.4640
5.5463 0.33 1500 6.2736
5.1871 0.44 2000 5.8517
4.896 0.55 2500 5.6070
4.6557 0.66 3000 5.4669
4.4832 0.77 3500 5.3318
4.3368 0.88 4000 5.2414
4.1887 0.99 4500 5.0666
4.053 1.1 5000 4.9532
3.9653 1.21 5500 4.8288
3.8865 1.33 6000 4.6741
3.8294 1.44 6500 4.7943
3.7565 1.55 7000 4.7336
3.673 1.66 7500 4.4760
3.6447 1.77 8000 4.5856
3.5808 1.88 8500 4.6133
3.5329 1.99 9000 4.4766
3.4916 2.1 9500 4.5085
3.4392 2.21 10000 4.5306
3.4333 2.32 10500 4.5433
3.3905 2.43 11000 4.1829
3.3701 2.54 11500 4.2976
3.3345 2.65 12000 4.2817
3.2815 2.76 12500 4.3146
3.2689 2.87 13000 4.2634
3.2401 2.98 13500 4.0907
3.2068 3.09 14000 4.1130
3.2097 3.2 14500 4.2001
3.1627 3.31 15000 4.0852
3.1647 3.42 15500 4.0383
3.1294 3.53 16000 3.9377
3.1166 3.64 16500 4.0733
3.1028 3.75 17000 3.8429
3.0903 3.86 17500 4.1127
3.0877 3.98 18000 3.8605
3.0407 4.09 18500 3.8482
3.0452 4.2 19000 4.0345
3.0496 4.31 19500 3.8602
3.0229 4.42 20000 4.2268
3.0157 4.53 20500 3.8028
3.0037 4.64 21000 3.8668
2.9992 4.75 21500 3.9542
3.016 4.86 22000 3.9090
2.9804 4.97 22500 4.0495

Framework versions

  • Transformers 4.25.1
  • Pytorch 1.8.0+cu111
  • Datasets 2.7.1
  • Tokenizers 0.13.2
Downloads last month
20
Inference Examples
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social visibility and check back later, or deploy to Inference Endpoints (dedicated) instead.