asr-africa/W2V2-Bert_nchlt_speech_corpus_ZULU_50hr_v1

facebook/w2v-bert-2.0

This model is a fine-tuned version of facebook/w2v-bert-2.0 on the NCHLT dataset. It achieves the following results on the evaluation set:

• Loss: 0.3728
• Wer: 0.6044
• Cer: 0.1317

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 16
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 2
• total_train_batch_size: 32
• optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.01
• num_epochs: 100

Training results

Training Loss	Epoch	Step	Validation Loss	Wer	Cer
0.8037	0.9996	1261	0.2249	0.3222	0.0456
0.1618	2.0	2523	0.1750	0.2472	0.0350
0.1247	2.9996	3784	0.1514	0.2119	0.0314
0.1021	4.0	5046	0.1542	0.2276	0.0324
0.0858	4.9996	6307	0.1577	0.2153	0.0314
0.0731	6.0	7569	0.1516	0.2115	0.0311
0.0617	6.9996	8830	0.1433	0.2012	0.0293
0.0512	8.0	10092	0.1524	0.2037	0.0292
0.0441	8.9996	11353	0.1482	0.1954	0.0295
0.0383	10.0	12615	0.1726	0.2028	0.0300
0.0322	10.9996	13876	0.1684	0.1887	0.0286
0.0272	12.0	15138	0.1719	0.1958	0.0286
0.0219	12.9996	16399	0.1656	0.2057	0.0309
0.0199	14.0	17661	0.1762	0.2008	0.0298
0.0167	14.9996	18922	0.2003	0.1999	0.0305
0.0149	16.0	20184	0.1960	0.1979	0.0299
0.0139	16.9996	21445	0.1783	0.1974	0.0302
0.0118	18.0	22707	0.1992	0.1938	0.0286
0.0113	18.9996	23968	0.1859	0.2117	0.0320
0.0102	20.0	25230	0.1983	0.1867	0.0280
0.0093	20.9996	26491	0.2128	0.1938	0.0293
0.0095	22.0	27753	0.1851	0.1833	0.0274
0.0082	22.9996	29014	0.1972	0.1889	0.0281
0.0078	24.0	30276	0.2096	0.1865	0.0276
0.007	24.9996	31537	0.2165	0.1921	0.0292
0.0071	26.0	32799	0.2164	0.1887	0.0282
0.0069	26.9996	34060	0.2070	0.1898	0.0289
0.006	28.0	35322	0.2114	0.1742	0.0266
0.0059	28.9996	36583	0.2126	0.1950	0.0291
0.0052	30.0	37845	0.2325	0.1923	0.0291
0.0055	30.9996	39106	0.2167	0.1760	0.0259
0.0047	32.0	40368	0.2173	0.1865	0.0278
0.005	32.9996	41629	0.2063	0.2032	0.0302
0.0042	34.0	42891	0.2312	0.1811	0.0273
0.0043	34.9996	44152	0.2115	0.1798	0.0275
0.0038	36.0	45414	0.2277	0.1883	0.0287
0.0036	36.9996	46675	0.2314	0.1802	0.0275
0.0038	38.0	47937	0.2356	0.1849	0.0278

Framework versions

• Transformers 4.46.3
• Pytorch 2.1.0+cu118
• Datasets 3.1.0
• Tokenizers 0.20.3