training_bert

This model is a fine-tuned version of Bert Base Uncased on dataset composed of different jobs posted in several job platforms and thousands of resumes. It achieves the following results on the evaluation set:

Loss: 4.0495

Model description

Pretraining done on bert base architecture.

Intended uses & limitations

This model can be used to generate contextual embeddings for textual data used in Applicant Tracking Systems such as resumes, jobs and cover letters. The embeddings can be further used to perform other NLP downstream tasks such as classification, Named Entity Recognition and so on.

Training and evaluation data

THe training corpus is developed using about 40000 resumes and 2000 jobs posted scrapped from different job portals. This is a preliminary dataset for the experimentation. THe corpus size is about 2.35 GB of textual data. Similary evaluation data contains few resumes and jobs making about 12 mb of textual data.

Training procedure

For the pretraining of masked language model, Trainer API from Huggingface is used. The pretraining took about 6 hrs 40 mins.

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 5e-05
train_batch_size: 128
eval_batch_size: 128
seed: 42
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: linear
num_epochs: 5

Training results

Training Loss	Epoch	Step	Validation Loss
6.7862	0.11	500	6.9461
5.9428	0.22	1000	6.4640
5.5463	0.33	1500	6.2736
5.1871	0.44	2000	5.8517
4.896	0.55	2500	5.6070
4.6557	0.66	3000	5.4669
4.4832	0.77	3500	5.3318
4.3368	0.88	4000	5.2414
4.1887	0.99	4500	5.0666
4.053	1.1	5000	4.9532
3.9653	1.21	5500	4.8288
3.8865	1.33	6000	4.6741
3.8294	1.44	6500	4.7943
3.7565	1.55	7000	4.7336
3.673	1.66	7500	4.4760
3.6447	1.77	8000	4.5856
3.5808	1.88	8500	4.6133
3.5329	1.99	9000	4.4766
3.4916	2.1	9500	4.5085
3.4392	2.21	10000	4.5306
3.4333	2.32	10500	4.5433
3.3905	2.43	11000	4.1829
3.3701	2.54	11500	4.2976
3.3345	2.65	12000	4.2817
3.2815	2.76	12500	4.3146
3.2689	2.87	13000	4.2634
3.2401	2.98	13500	4.0907
3.2068	3.09	14000	4.1130
3.2097	3.2	14500	4.2001
3.1627	3.31	15000	4.0852
3.1647	3.42	15500	4.0383
3.1294	3.53	16000	3.9377
3.1166	3.64	16500	4.0733
3.1028	3.75	17000	3.8429
3.0903	3.86	17500	4.1127
3.0877	3.98	18000	3.8605
3.0407	4.09	18500	3.8482
3.0452	4.2	19000	4.0345
3.0496	4.31	19500	3.8602
3.0229	4.42	20000	4.2268
3.0157	4.53	20500	3.8028
3.0037	4.64	21000	3.8668
2.9992	4.75	21500	3.9542
3.016	4.86	22000	3.9090
2.9804	4.97	22500	4.0495

Framework versions

Transformers 4.25.1
Pytorch 1.8.0+cu111
Datasets 2.7.1
Tokenizers 0.13.2

Shushant
/

ApplicantTrackingSystemBERT