mms-1b-swagen-combined-5hrs-model

This model is a fine-tuned version of facebook/mms-1b-all on the SWAGEN - SWA dataset. It achieves the following results on the evaluation set:

• Loss: 0.2261
• Wer: 0.1896

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0003
• train_batch_size: 4
• eval_batch_size: 4
• seed: 42
• gradient_accumulation_steps: 2
• total_train_batch_size: 8
• optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 100
• num_epochs: 30.0
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Wer
15.7298	0.2392	100	0.9582	0.6895
0.6897	0.4785	200	0.2673	0.1978
0.5013	0.7177	300	0.2438	0.1978
0.5306	0.9569	400	0.2348	0.1948
0.4662	1.1962	500	0.2367	0.1990
0.486	1.4354	600	0.2354	0.1951
0.447	1.6746	700	0.2343	0.1942
0.4513	1.9139	800	0.2311	0.1955
0.474	2.1531	900	0.2302	0.1940
0.451	2.3923	1000	0.2307	0.1924
0.4616	2.6316	1100	0.2279	0.1918
0.4489	2.8708	1200	0.2275	0.1927
0.4452	3.1100	1300	0.2297	0.1927
0.4023	3.3493	1400	0.2287	0.1934
0.4471	3.5885	1500	0.2271	0.1906
0.4471	3.8278	1600	0.2274	0.1893
0.4386	4.0670	1700	0.2293	0.1925
0.4281	4.3062	1800	0.2269	0.1930
0.4058	4.5455	1900	0.2279	0.1922
0.4246	4.7847	2000	0.2285	0.1939
0.4457	5.0239	2100	0.2246	0.1900
0.424	5.2632	2200	0.2258	0.1953
0.4463	5.5024	2300	0.2237	0.1921
0.4095	5.7416	2400	0.2279	0.1896
0.3959	5.9809	2500	0.2265	0.1884
0.3916	6.2201	2600	0.2275	0.1912
0.435	6.4593	2700	0.2260	0.1896

Framework versions

• Transformers 4.47.1
• Pytorch 2.5.1+cu124
• Datasets 3.2.0
• Tokenizers 0.21.0