w2v2-bert-wolof-kallaama-fleurs-51-hours

This model is a fine-tuned version of facebook/w2v-bert-2.0 on the fleurs dataset. It achieves the following results on the evaluation set:

• Cer: 0.9547
• Loss: 3.0591
• Wer: 0.9973

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0003
• train_batch_size: 4
• eval_batch_size: 4
• seed: 42
• gradient_accumulation_steps: 4
• total_train_batch_size: 16
• optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 500
• num_epochs: 50

Training results

Training Loss	Epoch	Step	Cer	Validation Loss	Wer
11.365	0.1575	200	0.3285	1.3757	0.8596
8.1173	0.3150	400	0.3903	1.6151	0.9262
8.6222	0.4725	600	0.4416	1.6832	1.0897
8.5075	0.6300	800	0.4690	1.9821	1.1733
8.2242	0.7876	1000	0.4320	1.5638	0.9101
8.0292	0.9451	1200	0.5443	2.2441	1.3338
8.324	1.1028	1400	0.3711	1.3820	0.9277
7.788	1.2603	1600	0.3300	1.3004	0.8186
7.9105	1.4178	1800	0.3188	1.2392	0.8271
7.8063	1.5753	2000	0.4218	1.7494	1.0251
7.46	1.7328	2200	0.3314	1.2549	0.8616
7.6575	1.8903	2400	0.3287	1.2104	0.8541
7.5972	2.0480	2600	0.3319	1.2033	0.8645
7.4658	2.2056	2800	0.3006	1.1858	0.8008
7.221	2.3631	3000	0.3109	1.1342	0.7934
7.1349	2.5206	3200	0.3179	1.1102	0.7883
7.3413	2.6781	3400	0.3013	1.0917	0.7674
6.953	2.8356	3600	0.2940	1.1294	0.7573
7.1135	2.9931	3800	0.2927	1.0893	0.7646
6.8008	3.1508	4000	0.3085	1.1878	0.7744
6.8463	3.3083	4200	0.3228	1.2166	0.8680
7.1754	3.4658	4400	0.3241	1.1947	0.7983
6.9404	3.6234	4600	0.2952	1.1283	0.7607
6.8082	3.7809	4800	0.3496	1.2277	0.8241
6.8116	3.9384	5000	0.2760	1.0372	0.7161
6.3426	4.0961	5200	0.2851	1.0466	0.7291
6.6638	4.2536	5400	0.2931	1.0963	0.7538
6.5524	4.4111	5600	0.2743	1.0331	0.7057
6.5238	4.5686	5800	0.2915	1.0792	0.7476
6.4341	4.7261	6000	0.2663	1.0037	0.6886
6.5839	4.8836	6200	0.2677	0.9936	0.6844
6.3873	5.0413	6400	0.3212	1.1431	0.7763
6.1866	5.1989	6600	0.2948	1.0253	0.7325
6.3827	5.3564	6800	0.2897	1.0631	0.7383
6.2649	5.5139	7000	0.2907	1.0646	0.7435
6.4511	5.6714	7200	0.2827	1.0214	0.6989
6.0703	5.8289	7400	0.2903	1.0758	0.7679
6.0693	5.9864	7600	0.2665	0.9777	0.7000
5.983	6.1441	7800	0.2771	1.0404	0.7173
6.0049	6.3016	8000	0.3001	1.1549	0.7450
6.1467	6.4591	8200	0.3097	1.1164	0.7518
6.115	6.6167	8400	0.2933	1.0852	0.7684
6.0742	6.7742	8600	0.2609	0.9603	0.6832
5.9315	6.9317	8800	0.2752	1.0255	0.7091
5.9475	7.0894	9000	0.2613	0.9664	0.6975
5.7488	7.2469	9200	0.2568	0.9330	0.6638
5.7661	7.4044	9400	0.2520	0.9191	0.6527
5.7212	7.5619	9600	0.2659	0.9423	0.6759
5.7596	7.7194	9800	0.2636	0.9645	0.6682
5.977	7.8769	10000	0.2451	0.9276	0.6381
5.7647	8.0347	10200	0.2988	1.0626	0.7295
5.5935	8.1922	10400	0.2721	1.0407	0.6989
5.6624	8.3497	10600	0.2699	1.0167	0.6748
5.3758	8.5072	10800	0.2556	0.9655	0.6675
5.6656	8.6647	11000	0.2571	0.9696	0.6989
5.5387	8.8222	11200	0.2704	0.9757	0.6866
5.46	8.9829	11400	0.2885	1.0217	0.7150
5.5785	9.1404	11600	0.2587	0.9573	0.6678
5.3339	9.2979	11800	0.2601	0.9564	0.6861
5.4872	9.4554	12000	0.2514	0.9417	0.6539
5.4436	9.6129	12200	0.2676	0.9710	0.6946
5.3492	9.7704	12400	0.2526	0.9393	0.6669
5.294	9.9279	12600	0.2661	0.9635	0.6942
5.1977	10.0856	12800	0.2493	0.9161	0.6575
5.2717	10.2432	13000	0.2469	0.8901	0.6491
5.1552	10.4007	13200	0.2614	0.9210	0.6580
5.1994	10.5582	13400	0.2504	0.9304	0.6372
5.2514	10.7157	13600	0.2428	0.9013	0.6356
5.1896	10.8732	13800	0.2404	0.8774	0.6270
5.1585	11.0309	14000	0.2730	0.9703	0.6900
4.9355	11.1884	14200	0.2415	0.9128	0.6359
5.0276	11.3459	14400	0.2650	0.9904	0.6739
5.0548	11.5034	14600	0.2467	0.9101	0.6384
5.0494	11.6610	14800	0.2429	0.9135	0.6354
4.9313	11.8185	15000	0.2542	0.9873	0.6525
4.9135	11.9760	15200	0.2676	1.0139	0.6916
4.7921	12.1337	15400	0.2384	0.8734	0.6091
4.8773	12.2912	15600	0.2362	0.8873	0.6340
4.8873	12.4487	15800	0.2297	0.8668	0.6026
4.8673	12.6062	16000	0.2462	0.9202	0.6450
4.9313	12.7637	16200	0.2245	0.8581	0.5927
4.7984	12.9212	16400	0.2210	0.8378	0.5938
4.6018	13.0790	16600	0.2453	0.9018	0.6329
4.6903	13.2365	16800	0.2643	0.9834	0.6656
4.6313	13.3940	17000	0.2894	1.0031	0.7200
4.7024	13.5515	17200	0.2523	0.8988	0.6632
4.7703	13.7090	17400	0.2307	0.9010	0.6132
4.694	13.8665	17600	0.2391	0.9059	0.6300
4.8065	14.0242	17800	0.2424	0.8890	0.6288
4.5063	14.1817	18000	0.2440	0.9116	0.6444
4.5276	14.3392	18200	0.2474	0.8997	0.6477
4.5014	14.4968	18400	0.2555	0.9188	0.6549
4.6184	14.6543	18600	0.2392	0.9068	0.6236
4.5953	14.8118	18800	0.2258	0.8868	0.5955
4.61	14.9693	19000	0.2362	0.8585	0.6276
4.3269	15.1270	19200	0.2235	0.8278	0.5904
4.4577	15.2845	19400	0.2324	0.8325	0.6242
4.3569	15.4420	19600	0.2270	0.8431	0.6095
4.3704	15.5995	19800	0.2247	0.8658	0.5888
4.4042	15.7570	20000	0.2220	0.8382	0.5901
6.5475	15.9146	20200	0.9918	3.0998	0.9995
11.4462	16.0723	20400	0.9918	3.5271	0.9995
11.4217	16.2298	20600	0.9918	3.5480	0.9995
11.4528	16.3873	20800	0.9918	3.5337	0.9995
11.4605	16.5448	21000	0.9918	3.4432	0.9995
11.5022	16.7023	21200	0.9918	3.6739	0.9995
11.2903	16.8598	21400	0.9918	3.4050	0.9995
11.4459	17.0175	21600	0.9918	3.0868	0.9995
11.4196	17.1750	21800	0.9918	3.0837	0.9995
11.5229	17.3325	22000	0.9918	3.0956	0.9995
11.3011	17.4901	22200	0.9918	3.1089	0.9995
11.5656	17.6476	22400	0.9918	3.1083	0.9995
11.3655	17.8051	22600	0.9918	3.0943	0.9995
11.4166	17.9626	22800	1.0000	3.0819	1.0
11.2684	18.1203	23000	0.9918	3.5478	0.9995
11.4198	18.2778	23200	0.9918	3.6414	0.9995
11.4383	18.4353	23400	0.9918	3.4188	0.9995
11.3783	18.5928	23600	0.9918	3.3586	0.9995
11.3902	18.7503	23800	0.9918	3.4406	0.9995
11.4154	18.9079	24000	0.9918	3.4694	0.9995
11.3858	19.0656	24200	0.9918	3.2139	0.9995
11.3676	19.2231	24400	0.9918	3.0753	0.9995
11.3732	19.3806	24600	0.9918	3.2396	0.9995
11.3314	19.5381	24800	0.9918	3.2999	0.9995
11.3988	19.6956	25000	0.9918	3.2368	0.9995
11.3691	19.8531	25200	0.9918	3.1855	0.9995
11.3116	20.0108	25400	0.9918	3.0588	0.9995
11.3058	20.1683	25600	0.9918	3.0848	0.9995
11.3146	20.3259	25800	0.9918	3.0528	0.9995
11.3532	20.4834	26000	0.9918	3.0744	0.9995
11.2811	20.6409	26200	0.9918	3.0667	0.9995
11.2941	20.7984	26400	0.9918	3.0652	0.9995
11.3823	20.9559	26600	0.9918	3.0776	0.9995
11.2676	21.1136	26800	0.9918	3.2268	0.9995
11.2981	21.2711	27000	0.9918	3.2142	0.9995
11.2612	21.4286	27200	0.9918	3.1933	0.9995
11.3154	21.5861	27400	0.9578	3.1970	0.9995
11.2117	21.7437	27600	0.9918	3.0928	0.9995
11.2915	21.9012	27800	0.9918	3.1445	0.9995
11.2099	22.0589	28000	0.9918	3.0328	0.9995
11.2094	22.2164	28200	0.9918	3.0387	0.9995
11.1962	22.3739	28400	0.9918	3.0567	0.9995
11.2243	22.5314	28600	0.9833	3.0652	1.0
11.2216	22.6889	28800	0.9918	3.0638	0.9995
11.2719	22.8464	29000	0.9918	3.0451	0.9995
11.1563	23.0041	29200	0.9918	3.0807	0.9995
11.2882	23.1616	29400	0.9907	3.1393	0.9995
11.302	23.3192	29600	0.9917	3.0528	0.9995
11.2164	23.4767	29800	0.9913	3.0546	0.9995
11.1795	23.6342	30000	0.9903	3.1129	0.9995
11.0366	23.7917	30200	0.9916	3.0650	0.9995
11.1686	23.9492	30400	0.9913	3.1736	0.9995
11.208	24.1069	30600	0.9906	3.0996	0.9995
11.3055	24.2644	30800	0.9912	3.0992	0.9995
11.0145	24.4219	31000	0.9881	3.0860	0.9998
11.1562	24.5794	31200	0.9912	3.1058	0.9995
11.0759	24.7370	31400	0.9905	3.0840	0.9995
11.2897	24.8945	31600	0.9830	3.0686	1.0
11.1279	25.0522	31800	0.9914	3.0696	0.9995
11.1537	25.2097	32000	0.9903	3.1248	0.9995
11.1116	25.3672	32200	0.9917	3.1970	0.9995
11.1538	25.5247	32400	0.9906	3.0686	0.9995
11.127	25.6822	32600	0.9818	3.0850	0.9995
11.2396	25.8397	32800	0.9906	3.2026	0.9995
11.0745	25.9972	33000	0.9900	3.0807	0.9995
11.1073	26.1550	33200	0.9908	3.1192	0.9995
11.1427	26.3125	33400	0.9898	3.2036	0.9997
11.1308	26.4700	33600	0.9822	3.0885	1.0
11.242	26.6275	33800	0.9890	3.0951	0.9998
11.0914	26.7850	34000	0.9906	3.0998	0.9995
11.1505	26.9425	34200	0.9826	3.1465	1.0
11.204	27.1002	34400	0.9831	3.1053	1.0
11.1805	27.2577	34600	0.9826	3.1879	1.0
11.0551	27.4152	34800	0.9828	3.2913	1.0
11.0709	27.5728	35000	0.9752	3.2019	0.9982
11.1597	27.7303	35200	0.9819	3.0945	1.0
11.121	27.8878	35400	0.9817	3.1601	0.9998
11.1171	28.0455	35600	0.9813	3.1293	0.9998
11.1643	28.2030	35800	0.9782	3.1577	0.9995
11.0813	28.3605	36000	0.9828	3.1471	1.0
11.1462	28.5180	36200	0.9814	3.1006	0.9998
11.1104	28.6755	36400	0.9815	3.0795	0.9998
11.0728	28.8330	36600	0.9826	3.1310	1.0
11.0569	28.9905	36800	0.9825	3.1427	1.0
11.1292	29.1483	37000	0.9802	3.1240	0.9998
11.1124	29.3058	37200	0.9829	3.1054	1.0
11.1312	29.4633	37400	0.9802	3.1249	0.9998
11.0397	29.6208	37600	0.9810	3.3649	0.9998
11.1638	29.7783	37800	0.9659	3.1539	0.9995
10.9487	29.9358	38000	0.9788	3.1141	0.9998
11.2128	30.0935	38200	0.9817	3.1292	0.9998
10.9408	30.2510	38400	0.9829	3.2014	1.0
11.1553	30.4085	38600	0.9825	3.2633	1.0
10.9881	30.5661	38800	0.9829	3.2195	1.0
11.2454	30.7236	39000	0.9829	3.0897	1.0
11.0626	30.8811	39200	0.9824	3.1417	1.0
10.9765	31.0388	39400	0.9792	3.2526	0.9998
10.9974	31.1963	39600	0.9751	3.1686	0.9998
11.052	31.3538	39800	0.9809	3.1762	1.0
11.0791	31.5113	40000	0.9824	3.1593	1.0
11.2061	31.6688	40200	0.9802	3.1605	1.0
11.1431	31.8263	40400	0.9845	3.1205	1.0
11.0067	31.9839	40600	0.9732	3.1436	1.0
11.1241	32.1416	40800	0.9810	3.2679	1.0
11.0069	32.2991	41000	0.9793	3.1982	1.0
11.0413	32.4566	41200	0.9823	3.2032	1.0
11.1059	32.6141	41400	0.9830	3.1603	1.0
11.1694	32.7716	41600	0.9874	3.1735	0.9998
11.0381	32.9291	41800	0.9828	3.1048	1.0
11.0083	33.0868	42000	0.9876	3.3171	0.9998
11.0924	33.2443	42200	0.9822	3.1374	1.0
11.0602	33.4019	42400	0.9705	3.1099	0.9998
11.049	33.5594	42600	0.9759	3.1430	1.0
11.0629	33.7169	42800	0.9666	3.1308	0.9993
11.0722	33.8744	43000	0.9716	3.2294	0.9998
10.9753	34.0321	43200	0.9652	3.1188	0.9995
11.0705	34.1896	43400	0.9643	3.1276	0.9994
11.1455	34.3471	43600	0.9610	3.2275	0.9992
11.078	34.5046	43800	0.9573	3.1349	0.9977
11.0564	34.6621	44000	0.9806	3.1319	0.9984
10.9806	34.8196	44200	0.9740	3.0662	0.9998
11.0345	34.9772	44400	0.9771	3.1630	0.9998
10.9724	35.1349	44600	0.9830	3.2076	1.0
11.0362	35.2924	44800	0.9820	3.2517	0.9999
11.0334	35.4499	45000	0.9806	3.3696	0.9998
11.0538	35.6074	45200	0.9826	3.1394	1.0
11.069	35.7649	45400	0.9764	3.1198	0.9993
11.0797	35.9224	45600	0.9824	3.2666	0.9969
10.9769	36.0801	45800	0.9690	3.1051	0.9931
11.0627	36.2376	46000	0.9809	3.1442	1.0
11.0745	36.3952	46200	0.9786	3.3483	0.9997
10.925	36.5527	46400	0.9743	3.2175	0.9998
10.9807	36.7102	46600	0.9611	3.1274	0.9994
11.1879	36.8677	46800	0.9714	3.2428	0.9998
11.0968	37.0254	47000	0.9601	3.0837	0.9986
11.1022	37.1829	47200	0.9611	3.1820	0.9939
11.0787	37.3404	47400	0.9578	3.1182	0.9983
10.949	37.4979	47600	0.9661	3.0898	0.9994
10.9539	37.6554	47800	0.9646	3.0690	0.9992
11.0789	37.8130	48000	0.9698	3.1843	0.9968
11.07	37.9705	48200	0.9747	3.0750	0.9993
11.1234	38.1282	48400	0.9625	3.0746	0.9986
11.0366	38.2857	48600	0.9644	3.0992	0.9992
11.0859	38.4432	48800	0.9658	3.1111	0.9991
10.9869	38.6007	49000	0.9754	3.0759	0.9995
11.0709	38.7582	49200	0.9658	3.0700	0.9993
10.9567	38.9157	49400	0.9631	3.1137	0.9992
11.0935	39.0734	49600	0.9618	3.1144	0.9990
11.0608	39.2310	49800	0.9661	3.1945	0.9994
11.076	39.3885	50000	0.9667	3.0982	0.9997
10.9764	39.5460	50200	0.9553	3.1247	0.9982
11.0652	39.7035	50400	0.9543	3.0895	0.9976
10.9708	39.8610	50600	0.9571	3.0799	0.9983
11.0508	40.0187	50800	0.9635	3.1092	0.9989
11.1156	40.1762	51000	0.9688	3.1279	0.9998
11.0016	40.3337	51200	0.9640	3.1014	0.9992
10.9556	40.4912	51400	0.9617	3.1109	0.9989
11.0919	40.6487	51600	0.9631	3.0996	0.9991
11.0243	40.8063	51800	0.9633	3.0884	0.9991
11.0418	40.9638	52000	0.9620	3.1216	0.9990
11.035	41.1215	52200	0.9541	3.1574	0.9973
11.0619	41.2790	52400	0.9593	3.1282	0.9985
11.0296	41.4365	52600	0.9544	3.0862	0.9974
11.0412	41.5940	52800	0.9533	3.1396	0.9979
11.0199	41.7515	53000	0.9754	3.0858	0.9993
11.0276	41.9090	53200	0.9760	3.0934	0.9995
10.998	42.0667	53400	0.9749	3.1018	0.9995
11.0202	42.2243	53600	0.9684	3.1158	0.9991
11.0136	42.3818	53800	0.9665	3.1606	0.9995
10.9226	42.5393	54000	0.9621	3.1147	0.9990
11.0931	42.6968	54200	0.9619	3.0862	0.9991
11.0501	42.8543	54400	0.9644	3.0925	0.9992
11.0863	43.0120	54600	0.9714	3.1181	0.9993
10.9404	43.1695	54800	0.9556	3.1436	0.9981
11.0515	43.3270	55000	0.9584	3.0902	0.9982
11.0546	43.4845	55200	0.9625	3.0884	0.9991
11.0463	43.6421	55400	0.9617	3.0910	0.9990
11.0697	43.7996	55600	0.9593	3.0587	0.9988
11.0233	43.9571	55800	0.9622	3.0703	0.9991
10.9304	44.1148	56000	0.9605	3.0434	0.9990
11.0742	44.2723	56200	0.9647	3.0841	0.9954
10.9191	44.4298	56400	0.9545	3.0844	0.9973
11.0861	44.5873	56600	0.9546	3.0726	0.9973
11.1076	44.7448	56800	0.9541	3.0976	0.9975
10.984	44.9023	57000	0.9539	3.0956	0.9972
11.0133	45.0601	57200	0.9539	3.1178	0.9972
11.0717	45.2176	57400	0.9551	3.0933	0.9974
11.1244	45.3751	57600	0.9556	3.1024	0.9974
10.9426	45.5326	57800	0.9554	3.1071	0.9974
11.044	45.6901	58000	0.9537	3.0805	0.9930
11.073	45.8476	58200	0.9548	3.0562	0.9974
10.9002	46.0053	58400	0.9550	3.0600	0.9974
11.0946	46.1628	58600	0.9535	3.0709	0.9973
10.9461	46.3203	58800	0.9535	3.1048	0.9972
10.9388	46.4778	59000	0.9549	3.1046	0.9973
11.0717	46.6354	59200	0.9547	3.0571	0.9973
11.0434	46.7929	59400	0.9547	3.1111	0.9973
10.9955	46.9504	59600	0.9540	3.0607	0.9972
11.0413	47.1081	59800	0.9544	3.0791	0.9973
11.1388	47.2656	60000	0.9539	3.0903	0.9972
10.9972	47.4231	60200	0.9544	3.0904	0.9972
10.9828	47.5806	60400	0.9547	3.0882	0.9973
10.9937	47.7381	60600	0.9541	3.0521	0.9973
11.079	47.8956	60800	0.9542	3.0727	0.9972
10.9176	48.0534	61000	0.9537	3.0500	0.9972
10.9927	48.2109	61200	0.9542	3.0996	0.9972
11.0922	48.3684	61400	0.9549	3.0880	0.9974
10.9493	48.5259	61600	0.9547	3.0791	0.9973
11.0304	48.6834	61800	0.9545	3.0761	0.9967
11.0789	48.8409	62000	0.9540	3.0634	0.9972
10.9328	48.9984	62200	0.9539	3.0851	0.9972
11.0392	49.1561	62400	0.9543	3.0718	0.9972
10.9172	49.3136	62600	0.9545	3.0757	0.9973
11.0505	49.4712	62800	0.9549	3.0718	0.9974
11.035	49.6287	63000	0.9549	3.0840	0.9973
10.9821	49.7862	63200	0.9547	3.0783	0.9973
11.0789	49.9437	63400	0.9547	3.0591	0.9973

Framework versions

• Transformers 4.46.1
• Pytorch 2.1.0+cu118
• Datasets 2.17.0
• Tokenizers 0.20.3