zephyr-7b-dpo-qlora-pairrm

This model is a fine-tuned version of mistralai/Mistral-7B-Instruct-v0.2 on the snorkelai/Snorkel-Mistral-PairRM-DPO-Dataset dataset. It achieves the following results on the evaluation set:

• Loss: 0.6749
• Rewards/chosen: -1.5494
• Rewards/rejected: -1.6972
• Rewards/accuracies: 0.5740
• Rewards/margins: 0.1478
• Logps/rejected: -396.0607
• Logps/chosen: -375.6489
• Logits/rejected: -4.3831
• Logits/chosen: -4.3963

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-06
• train_batch_size: 4
• eval_batch_size: 8
• seed: 42
• distributed_type: multi-GPU
• gradient_accumulation_steps: 4
• total_train_batch_size: 16
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 1

Training results

Training Loss	Epoch	Step	Validation Loss	Rewards/chosen	Rewards/rejected	Rewards/accuracies	Rewards/margins	Logps/rejected	Logps/chosen	Logits/rejected	Logits/chosen
0.6909	0.08	100	0.6922	-0.0180	-0.0200	0.5237	0.0020	-228.3365	-222.5081	-2.6230	-2.6258
0.6843	0.16	200	0.6873	-0.0792	-0.0932	0.5547	0.0140	-235.6526	-228.6281	-2.7602	-2.7632
0.68	0.24	300	0.6840	-0.5133	-0.5505	0.5473	0.0372	-281.3831	-272.0374	-3.5011	-3.5067
0.657	0.32	400	0.6808	-0.8624	-0.9278	0.5583	0.0654	-319.1186	-306.9489	-3.7427	-3.7523
0.6342	0.4	500	0.6784	-0.9237	-1.0040	0.5580	0.0803	-326.7317	-313.0756	-4.1065	-4.1163
0.6341	0.48	600	0.6767	-1.2010	-1.3087	0.5630	0.1077	-357.2099	-340.8084	-4.4445	-4.4555
0.573	0.56	700	0.6808	-1.5618	-1.6922	0.5633	0.1304	-395.5549	-376.8888	-4.4479	-4.4607
0.6384	0.64	800	0.6753	-1.3648	-1.4912	0.5660	0.1264	-375.4593	-357.1873	-4.1713	-4.1824
0.6728	0.72	900	0.6791	-1.6856	-1.8328	0.5713	0.1471	-409.6128	-389.2689	-4.5250	-4.5392
0.603	0.8	1000	0.6767	-1.6183	-1.7689	0.5770	0.1506	-403.2299	-382.5415	-4.4899	-4.5034
0.6732	0.88	1100	0.6753	-1.5670	-1.7162	0.5777	0.1493	-397.9568	-377.4016	-4.4104	-4.4236
0.6431	0.96	1200	0.6750	-1.5516	-1.6995	0.5773	0.1479	-396.2836	-375.8671	-4.3848	-4.3979

Framework versions

• PEFT 0.7.1
• Transformers 4.36.2
• Pytorch 2.1.2
• Datasets 2.14.6
• Tokenizers 0.15.0