w2v-bert-2.0-bemgen-male-model

This model is a fine-tuned version of facebook/w2v-bert-2.0 on the BEMGEN - BEM dataset. It achieves the following results on the evaluation set:

• Loss: 0.3124
• Wer: 0.4801

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0003
• train_batch_size: 4
• eval_batch_size: 4
• seed: 42
• optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 500
• training_steps: 3000
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Wer
No log	0.1034	100	0.9843	0.9557
No log	0.2068	200	0.9218	0.9166
No log	0.3102	300	0.9675	0.9435
No log	0.4137	400	1.1171	0.9413
1.1866	0.5171	500	1.0130	0.9430
1.1866	0.6205	600	0.9595	1.0711
1.1866	0.7239	700	0.8301	0.8833
1.1866	0.8273	800	0.8072	0.8745
1.1866	0.9307	900	0.7413	0.8407
0.7779	1.0341	1000	0.6572	0.7763
0.7779	1.1375	1100	0.6588	0.7513
0.7779	1.2410	1200	0.5933	0.7642
0.7779	1.3444	1300	0.5910	0.7305
0.7779	1.4478	1400	0.5967	0.7584
0.5649	1.5512	1500	0.5757	0.7299
0.5649	1.6546	1600	0.5121	0.6682
0.5649	1.7580	1700	0.5339	0.6576
0.5649	1.8614	1800	0.4539	0.6172
0.5649	1.9648	1900	0.4375	0.5837
0.4839	2.0683	2000	0.4384	0.6114
0.4839	2.1717	2100	0.3993	0.5685
0.4839	2.2751	2200	0.4016	0.5947
0.4839	2.3785	2300	0.3897	0.5646
0.4839	2.4819	2400	0.3798	0.5631
0.3288	2.5853	2500	0.3543	0.5203
0.3288	2.6887	2600	0.3427	0.5156
0.3288	2.7921	2700	0.3382	0.5109
0.3288	2.8956	2800	0.3257	0.4982
0.3288	2.9990	2900	0.3133	0.4790
0.2666	3.1024	3000	0.3124	0.4797

Framework versions

• Transformers 4.47.1
• Pytorch 2.5.1+cu124
• Datasets 3.2.0
• Tokenizers 0.21.0