phi-2-dpo-ultrachat-lora

This model is a fine-tuned version of lole25/phi-2-sft-ultrachat-lora on the HuggingFaceH4/ultrafeedback_binarized dataset. It achieves the following results on the evaluation set:

• Loss: 0.6872
• Rewards/chosen: -0.0312
• Rewards/rejected: -0.0436
• Rewards/accuracies: 0.3340
• Rewards/margins: 0.0124
• Logps/rejected: -98.5542
• Logps/chosen: -94.8435
• Logits/rejected: 0.7532
• Logits/chosen: 0.7326

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-06
• train_batch_size: 4
• eval_batch_size: 4
• seed: 42
• distributed_type: multi-GPU
• num_devices: 4
• gradient_accumulation_steps: 4
• total_train_batch_size: 64
• total_eval_batch_size: 16
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 2

Training results

Training Loss	Epoch	Step	Logits/chosen	Logits/rejected	Logps/chosen	Logps/rejected	Validation Loss	Rewards/accuracies	Rewards/chosen	Rewards/margins	Rewards/rejected
0.693	0.21	100	0.7998	0.8176	-91.7748	-94.2804	0.6931	0.2680	-0.0005	0.0004	-0.0008
0.6922	0.42	200	0.7941	0.8121	-91.9068	-94.5141	0.6924	0.3020	-0.0018	0.0014	-0.0032
0.6917	0.63	300	0.7870	0.8057	-92.2189	-94.9659	0.6917	0.3100	-0.0049	0.0028	-0.0077
0.6905	0.84	400	0.7827	0.8012	-92.4247	-95.2509	0.6913	0.3280	-0.0070	0.0036	-0.0105
0.6898	1.05	500	0.6900	-0.0142	-0.0205	0.3360	0.0064	-96.2490	-93.1429	0.7903	0.7711
0.6882	1.26	600	0.6887	-0.0217	-0.0306	0.3340	0.0089	-97.2594	-93.8981	0.7722	0.7527
0.6858	1.47	700	0.6879	-0.0274	-0.0383	0.3280	0.0108	-98.0249	-94.4717	0.7600	0.7395
0.6857	1.67	800	0.6874	-0.0303	-0.0423	0.3340	0.0120	-98.4270	-94.7618	0.7548	0.7341
0.6866	1.88	900	0.6872	-0.0313	-0.0437	0.3420	0.0124	-98.5655	-94.8550	0.7528	0.7321

Framework versions

• PEFT 0.7.1
• Transformers 4.36.2
• Pytorch 2.1.2+cu118
• Datasets 2.14.6
• Tokenizers 0.15.2