zephyr-7b-dpo-full-prometheus-reward-scale-05

This model is a fine-tuned version of alignment-handbook/zephyr-7b-sft-full on the None dataset. It achieves the following results on the evaluation set:

• Loss: 0.5286
• Rewards/chosen: -1.4143
• Rewards/rejected: -2.7417
• Rewards/accuracies: 0.7629
• Rewards/margins: 1.3275
• Logps/rejected: -493.2510
• Logps/chosen: -417.0316
• Logits/rejected: 1.9856
• Logits/chosen: 0.4911

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-07
• train_batch_size: 8
• eval_batch_size: 8
• seed: 55
• distributed_type: multi-GPU
• num_devices: 8
• gradient_accumulation_steps: 2
• total_train_batch_size: 128
• total_eval_batch_size: 64
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 1

Training results

Training Loss	Epoch	Step	Validation Loss	Rewards/chosen	Rewards/rejected	Rewards/accuracies	Rewards/margins	Logps/rejected	Logps/chosen	Logits/rejected	Logits/chosen
0.6696	0.1143	50	0.6584	-0.0084	-0.1643	0.6853	0.1559	-235.5072	-276.4426	-2.4382	-2.5406
0.6122	0.2286	100	0.6111	-0.4070	-0.8953	0.6767	0.4883	-308.6058	-316.3019	-2.5533	-2.6512
0.5476	0.3429	150	0.5583	-1.3343	-2.3426	0.7371	1.0083	-453.3369	-409.0355	0.9770	0.1441
0.5582	0.4571	200	0.5499	-1.0345	-2.1424	0.7328	1.1079	-433.3173	-379.0511	0.5624	-0.4976
0.5503	0.5714	250	0.5393	-1.1701	-2.3108	0.7371	1.1406	-450.1522	-392.6152	0.7719	-0.3725
0.5224	0.6857	300	0.5312	-1.2228	-2.5102	0.7543	1.2874	-470.0949	-397.8840	1.7088	0.1892
0.5396	0.8	350	0.5290	-1.4462	-2.7485	0.75	1.3024	-493.9275	-420.2202	1.9215	0.4365
0.55	0.9143	400	0.5286	-1.4143	-2.7417	0.7629	1.3275	-493.2510	-417.0316	1.9856	0.4911

Framework versions

• Transformers 4.44.0.dev0
• Pytorch 2.1.2
• Datasets 2.20.0
• Tokenizers 0.19.1