mistral-rand-300k

This model is a fine-tuned version of TheBloke/Mistral-7B-v0.1-GPTQ on the None dataset. It achieves the following results on the evaluation set:

• Loss: 0.3545

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 16
• eval_batch_size: 64
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• training_steps: 12451
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss
1.6373	0.01	50	0.8699
0.7891	0.02	100	0.7454
0.7135	0.02	150	0.6984
0.6931	0.03	200	0.6722
0.6592	0.08	250	0.6407
0.634	0.1	300	0.6212
0.61	0.11	350	0.6015
0.5961	0.13	400	0.5892
0.5795	0.14	450	0.5773
0.5765	0.16	500	0.5683
0.5619	0.18	550	0.5612
0.5558	0.19	600	0.5519
0.5572	0.21	650	0.5438
0.5467	0.22	700	0.5348
0.5303	0.24	750	0.5248
0.5307	0.26	800	0.5135
0.508	0.27	850	0.5036
0.5087	0.29	900	0.5009
0.5073	0.3	950	0.4945
0.5078	0.32	1000	0.4916
0.5089	0.34	1050	0.4893
0.4829	0.35	1100	0.4862
0.4872	0.37	1150	0.4832
0.4845	0.39	1200	0.4803
0.4993	0.4	1250	0.4774
0.475	0.42	1300	0.4746
0.4836	0.43	1350	0.4735
0.4748	0.45	1400	0.4708
0.4809	0.47	1450	0.4693
0.4755	0.48	1500	0.4668
0.4679	0.5	1550	0.4644
0.4685	0.51	1600	0.4622
0.4706	0.53	1650	0.4611
0.4673	0.55	1700	0.4605
0.4539	0.56	1750	0.4577
0.4501	0.58	1800	0.4560
0.4638	0.59	1850	0.4542
0.4663	0.61	1900	0.4521
0.4638	0.63	1950	0.4509
0.4562	0.64	2000	0.4501
0.4535	0.66	2050	0.4496
0.4548	0.67	2100	0.4470
0.442	0.69	2150	0.4453
0.4543	0.71	2200	0.4449
0.4435	0.72	2250	0.4428
0.4633	0.74	2300	0.4418
0.4438	0.75	2350	0.4416
0.4443	0.77	2400	0.4392
0.4424	0.79	2450	0.4386
0.4341	0.8	2500	0.4367
0.4329	0.82	2550	0.4353
0.4356	0.83	2600	0.4349
0.4384	0.85	2650	0.4351
0.4327	0.87	2700	0.4321
0.4356	0.88	2750	0.4323
0.4428	0.9	2800	0.4310
0.4358	0.91	2850	0.4304
0.4322	0.93	2900	0.4293
0.4336	0.95	2950	0.4280
0.4296	0.96	3000	0.4269
0.4365	0.98	3050	0.4267
0.4313	0.99	3100	0.4250
0.4256	1.01	3150	0.4251
0.4258	1.03	3200	0.4241
0.4245	1.04	3250	0.4225
0.4161	1.06	3300	0.4223
0.4228	1.07	3350	0.4215
0.4194	1.09	3400	0.4205
0.4331	1.11	3450	0.4193
0.4246	1.12	3500	0.4191
0.4246	1.14	3550	0.4173
0.4229	1.16	3600	0.4175
0.4128	1.17	3650	0.4160
0.4189	1.19	3700	0.4156
0.4154	1.2	3750	0.4148
0.4263	1.22	3800	0.4140
0.4124	1.24	3850	0.4139
0.4201	1.25	3900	0.4132
0.4224	1.27	3950	0.4122
0.4114	1.28	4000	0.4122
0.4169	1.3	4050	0.4112
0.4167	1.32	4100	0.4107
0.4019	1.33	4150	0.4095
0.4142	1.35	4200	0.4087
0.4086	1.36	4250	0.4080
0.406	1.38	4300	0.4075
0.4091	1.4	4350	0.4069
0.4149	1.41	4400	0.4062
0.4078	1.43	4450	0.4054
0.3997	1.44	4500	0.4052
0.3985	1.46	4550	0.4040
0.4035	1.48	4600	0.4035
0.3982	1.49	4650	0.4025
0.4018	1.51	4700	0.4030
0.4078	1.52	4750	0.4021
0.3991	1.54	4800	0.4010
0.4033	1.56	4850	0.4003
0.3964	1.57	4900	0.4005
0.3965	1.59	4950	0.3993
0.407	1.6	5000	0.3994
0.4036	1.62	5050	0.3983
0.4063	1.64	5100	0.3980
0.3857	1.65	5150	0.3973
0.3949	1.67	5200	0.3973
0.3872	1.68	5250	0.3965
0.393	1.7	5300	0.3959
0.3891	1.72	5350	0.3955
0.3903	1.73	5400	0.3950
0.3941	1.75	5450	0.3947
0.3879	1.76	5500	0.3941
0.4016	1.78	5550	0.3937
0.3936	1.8	5600	0.3929
0.4005	1.81	5650	0.3932
0.3939	1.83	5700	0.3923
0.4032	1.85	5750	0.3921
0.3921	1.86	5800	0.3921
0.3903	1.88	5850	0.3905
0.3983	1.89	5900	0.3910
0.3806	1.91	5950	0.3897
0.3964	1.93	6000	0.3906
0.3866	1.94	6050	0.3890
0.3882	1.96	6100	0.3888
0.3835	1.97	6150	0.3885
0.3921	1.99	6200	0.3875
0.388	2.01	6250	0.3878
0.3829	2.02	6300	0.3872
0.3814	2.04	6350	0.3867
0.3818	2.05	6400	0.3862
0.3802	2.07	6450	0.3860
0.3739	2.09	6500	0.3853
0.3771	2.1	6550	0.3852
0.3732	2.12	6600	0.3846
0.385	2.13	6650	0.3849
0.3767	2.15	6700	0.3833
0.3802	2.17	6750	0.3836
0.3844	2.18	6800	0.3828
0.3761	2.2	6850	0.3826
0.3765	2.21	6900	0.3826
0.3787	2.23	6950	0.3825
0.378	2.25	7000	0.3815
0.3792	2.26	7050	0.3815
0.3908	2.28	7100	0.3811
0.3757	2.29	7150	0.3810
0.376	2.31	7200	0.3804
0.3785	2.33	7250	0.3805
0.3744	2.34	7300	0.3797
0.3984	2.36	7350	0.3791
0.3833	2.37	7400	0.3792
0.3808	2.39	7450	0.3785
0.3803	2.41	7500	0.3786
0.3828	2.42	7550	0.3778
0.3697	2.44	7600	0.3780
0.3692	2.45	7650	0.3763
0.3808	2.47	7700	0.3769
0.3764	2.49	7750	0.3763
0.3865	2.5	7800	0.3763
0.375	2.52	7850	0.3760
0.368	2.53	7900	0.3755
0.3632	2.55	7950	0.3757
0.3792	2.57	8000	0.3758
0.374	2.58	8050	0.3748
0.3689	2.6	8100	0.3741
0.3843	2.62	8150	0.3741
0.3669	2.63	8200	0.3739
0.368	2.65	8250	0.3732
0.3726	2.66	8300	0.3734
0.3653	2.68	8350	0.3728
0.3777	2.7	8400	0.3732
0.3625	2.71	8450	0.3724
0.3749	2.73	8500	0.3716
0.3708	2.74	8550	0.3725
0.3618	2.76	8600	0.3711
0.3659	2.78	8650	0.3714
0.3661	2.79	8700	0.3711
0.3771	2.81	8750	0.3714
0.3637	2.82	8800	0.3704
0.3768	2.84	8850	0.3700
0.3722	2.86	8900	0.3701
0.366	2.87	8950	0.3693
0.3716	2.89	9000	0.3690
0.3622	2.9	9050	0.3688
0.3594	2.92	9100	0.3682
0.368	2.94	9150	0.3680
0.3538	2.95	9200	0.3678
0.3578	2.97	9250	0.3676
0.3685	2.98	9300	0.3679
0.3631	3.0	9350	0.3674
0.3645	3.02	9400	0.3665
0.3654	3.03	9450	0.3671
0.3502	3.05	9500	0.3662
0.356	3.06	9550	0.3665
0.3642	3.08	9600	0.3662
0.3688	3.1	9650	0.3659
0.3514	3.11	9700	0.3655
0.3463	3.13	9750	0.3656
0.3517	3.14	9800	0.3651
0.3666	3.16	9850	0.3650
0.3617	3.18	9900	0.3660
0.3452	3.19	9950	0.3649
0.3591	3.21	10000	0.3647
0.3509	3.22	10050	0.3643
0.3618	3.24	10100	0.3641
0.3571	3.26	10150	0.3640
0.3587	3.27	10200	0.3633
0.3664	3.29	10250	0.3637
0.3502	3.3	10300	0.3633
0.373	3.32	10350	0.3626
0.3623	3.34	10400	0.3622
0.3554	3.35	10450	0.3624
0.3511	3.37	10500	0.3622
0.3534	3.39	10550	0.3626
0.3473	3.4	10600	0.3620
0.3563	3.42	10650	0.3618
0.3612	3.43	10700	0.3614
0.3587	3.45	10750	0.3610
0.3521	3.47	10800	0.3609
0.3443	3.48	10850	0.3610
0.3615	3.5	10900	0.3608
0.3589	3.51	10950	0.3609
0.364	3.53	11000	0.3598
0.3498	3.55	11050	0.3600
0.3541	3.56	11100	0.3597
0.3555	3.58	11150	0.3594
0.3491	3.59	11200	0.3596
0.3498	3.61	11250	0.3589
0.3484	3.63	11300	0.3590
0.3483	3.64	11350	0.3586
0.3533	3.66	11400	0.3580
0.3479	3.67	11450	0.3589
0.3539	3.69	11500	0.3580
0.3507	3.71	11550	0.3582
0.3534	3.72	11600	0.3579
0.3559	3.74	11650	0.3575
0.3477	3.75	11700	0.3577
0.3501	3.77	11750	0.3574
0.3491	3.79	11800	0.3569
0.3661	3.8	11850	0.3569
0.3455	3.82	11900	0.3568
0.3522	3.83	11950	0.3564
0.3532	3.85	12000	0.3562
0.3513	3.87	12050	0.3559
0.3527	3.88	12100	0.3561
0.3575	3.9	12150	0.3556
0.3403	3.92	12200	0.3550
0.3495	3.93	12250	0.3554
0.3514	3.95	12300	0.3548
0.3556	3.96	12350	0.3547
0.3549	3.98	12400	0.3545
0.3454	2.0	12450	0.3545

Framework versions

• PEFT 0.7.1
• Transformers 4.36.2
• Pytorch 2.1.2+cu121
• Datasets 2.16.1
• Tokenizers 0.15.0