phi-2-gpo-renew2-b0.001-extra-i1

This model is a fine-tuned version of DUAL-GPO/phi-2-gpo-renew2-b0.001-i0 on the HuggingFaceH4/ultrafeedback_binarized dataset. It achieves the following results on the evaluation set:

• Loss: 0.0421
• Rewards/chosen: -0.0100
• Rewards/rejected: -0.0414
• Rewards/accuracies: 0.6015
• Rewards/margins: 0.0314
• Logps/rejected: -408.6569
• Logps/chosen: -406.3272
• Logits/rejected: -1.0065
• Logits/chosen: -1.0521

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-06
• train_batch_size: 4
• eval_batch_size: 4
• seed: 42
• distributed_type: multi-GPU
• gradient_accumulation_steps: 4
• total_train_batch_size: 16
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 1

Training results

Training Loss	Epoch	Step	Validation Loss	Rewards/chosen	Rewards/rejected	Rewards/accuracies	Rewards/margins	Logps/rejected	Logps/chosen	Logits/rejected	Logits/chosen
0.0982	0.11	100	0.0526	-0.0081	-0.0101	0.5190	0.0020	-377.3773	-404.4459	-0.7816	-0.8697
0.0846	0.21	200	0.0485	-0.0265	-0.0402	0.5530	0.0137	-407.4654	-422.8199	-0.9792	-1.0427
0.0859	0.32	300	0.0464	-0.0257	-0.0460	0.5725	0.0203	-413.2813	-422.0490	-1.0612	-1.1154
0.0957	0.43	400	0.0443	-0.0207	-0.0481	0.5780	0.0274	-415.3487	-417.0023	-1.0450	-1.0984
0.068	0.53	500	0.0432	-0.0067	-0.0318	0.5955	0.0252	-399.0811	-402.9732	-0.9791	-1.0329
0.0847	0.64	600	0.0427	-0.0050	-0.0312	0.5945	0.0263	-398.4744	-401.2879	-0.9837	-1.0364
0.0519	0.75	700	0.0423	-0.0082	-0.0377	0.5905	0.0295	-404.9791	-404.5331	-0.9872	-1.0360
0.0742	0.85	800	0.0422	-0.0105	-0.0420	0.6000	0.0315	-409.2462	-406.8035	-1.0109	-1.0556
0.0768	0.96	900	0.0421	-0.0100	-0.0415	0.5930	0.0315	-408.7397	-406.3475	-1.0050	-1.0502

Framework versions

• PEFT 0.7.1
• Transformers 4.36.2
• Pytorch 2.1.2
• Datasets 2.14.6
• Tokenizers 0.15.2