zephyr-7b-sft-lora-accum8-lr1e_6

This model is a fine-tuned version of mistralai/Mistral-7B-v0.1 on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 1.8034

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1e-06
• train_batch_size: 4
• eval_batch_size: 8
• seed: 42
• distributed_type: multi-GPU
• num_devices: 2
• gradient_accumulation_steps: 8
• total_train_batch_size: 64
• total_eval_batch_size: 16
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• num_epochs: 50.0

Training results

Training Loss	Epoch	Step	Validation Loss
2.0856	0.51	6	2.0622
2.0731	1.53	13	2.0591
2.0751	2.55	20	2.0578
2.0677	3.57	27	2.0556
2.0744	4.51	33	2.0476
2.0611	5.53	40	2.0437
2.051	6.55	47	2.0360
2.0559	7.57	54	2.0278
2.0429	8.51	60	2.0241
2.0373	9.53	67	2.0169
2.027	10.55	74	2.0083
2.0231	11.57	81	1.9966
2.0053	12.51	87	1.9906
2.0102	13.53	94	1.9815
1.989	14.55	101	1.9756
1.9901	15.57	108	1.9675
1.9893	16.51	114	1.9577
1.969	17.53	121	1.9478
1.9607	18.55	128	1.9387
1.9522	19.57	135	1.9304
1.9454	20.51	141	1.9281
1.9416	21.53	148	1.9205
1.9265	22.55	155	1.9154
1.9286	23.57	162	1.9079
1.921	24.51	168	1.9001
1.9082	25.53	175	1.8921
1.9102	26.55	182	1.8856
1.9072	27.57	189	1.8808
1.898	28.51	195	1.8763
1.885	29.53	202	1.8710
1.896	30.55	209	1.8660
1.8813	31.57	216	1.8625
1.8819	32.51	222	1.8561
1.8627	33.53	229	1.8486
1.8683	34.55	236	1.8457
1.8671	35.57	243	1.8423
1.86	36.51	249	1.8396
1.8573	37.53	256	1.8334
1.847	38.55	263	1.8330
1.842	39.57	270	1.8292
1.8437	40.51	276	1.8254
1.8406	41.53	283	1.8224
1.8369	42.55	290	1.8169
1.8345	43.57	297	1.8179
1.8418	44.51	303	1.8128
1.8315	45.53	310	1.8125
1.8243	46.55	317	1.8103
1.8312	47.57	324	1.8057
1.8247	48.51	330	1.8047
1.8189	49.53	337	1.8032

Framework versions

• Transformers 4.35.0
• Pytorch 2.1.0
• Datasets 2.14.6
• Tokenizers 0.14.1