--- license: apache-2.0 library_name: peft tags: - alignment-handbook - generated_from_trainer - trl - dpo - generated_from_trainer datasets: - HuggingFaceH4/ultrafeedback_binarized_fixed base_model: mistralai/Mistral-7B-v0.1 model-index: - name: zephyr-7b-dpo-lora results: [] --- # zephyr-7b-dpo-lora This model is a fine-tuned version of [lewtun/zephyr-7b-sft-qlora](https://huggingface.co/lewtun/zephyr-7b-sft-qlora) on the HuggingFaceH4/ultrafeedback_binarized_fixed dataset. It achieves the following results on the evaluation set: - Loss: 0.5133 - Rewards/chosen: -1.2447 - Rewards/rejected: -2.1118 - Rewards/accuracies: 0.7539 - Rewards/margins: 0.8671 - Logps/rejected: -457.0128 - Logps/chosen: -385.9082 - Logits/rejected: 1.2523 - Logits/chosen: 0.7989 ## Model description More information needed ## Intended uses & limitations More information needed ## Training and evaluation data More information needed ## Training procedure ### Training hyperparameters The following hyperparameters were used during training: - learning_rate: 5e-06 - train_batch_size: 4 - eval_batch_size: 8 - seed: 42 - distributed_type: multi-GPU - num_devices: 8 - total_train_batch_size: 32 - total_eval_batch_size: 64 - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08 - lr_scheduler_type: cosine - lr_scheduler_warmup_ratio: 0.1 - num_epochs: 1 ### Training results | Training Loss | Epoch | Step | Validation Loss | Rewards/chosen | Rewards/rejected | Rewards/accuracies | Rewards/margins | Logps/rejected | Logps/chosen | Logits/rejected | Logits/chosen | |:-------------:|:-----:|:----:|:---------------:|:--------------:|:----------------:|:------------------:|:---------------:|:--------------:|:------------:|:---------------:|:-------------:| | 0.6918 | 0.05 | 100 | 0.6914 | 0.0059 | 0.0018 | 0.7109 | 0.0041 | -245.6464 | -260.8458 | -2.1364 | -2.2285 | | 0.6619 | 0.1 | 200 | 0.6497 | -0.0263 | -0.1318 | 0.7070 | 0.1056 | -259.0110 | -264.0628 | -2.0537 | -2.1558 | | 0.6077 | 0.16 | 300 | 0.6083 | -0.2610 | -0.5505 | 0.7188 | 0.2895 | -300.8820 | -287.5379 | -1.8505 | -1.9870 | | 0.5813 | 0.21 | 400 | 0.5857 | -0.5019 | -0.9224 | 0.7344 | 0.4205 | -338.0691 | -311.6292 | -1.7834 | -1.9347 | | 0.6033 | 0.26 | 500 | 0.5684 | -0.6480 | -1.1327 | 0.7578 | 0.4847 | -359.0957 | -326.2360 | -1.0646 | -1.2844 | | 0.5338 | 0.31 | 600 | 0.5431 | -0.9068 | -1.6081 | 0.7539 | 0.7013 | -406.6367 | -352.1152 | -0.0058 | -0.3463 | | 0.5235 | 0.37 | 700 | 0.5304 | -1.0331 | -1.8281 | 0.7461 | 0.7951 | -428.6434 | -364.7436 | 0.2246 | -0.1374 | | 0.5241 | 0.42 | 800 | 0.5276 | -0.9760 | -1.7110 | 0.7578 | 0.7350 | -416.9325 | -359.0362 | 0.3361 | -0.0432 | | 0.5332 | 0.47 | 900 | 0.5257 | -1.2407 | -2.0657 | 0.75 | 0.8250 | -452.3993 | -385.5118 | 0.8926 | 0.4681 | | 0.531 | 0.52 | 1000 | 0.5232 | -1.1277 | -1.8553 | 0.7461 | 0.7276 | -431.3623 | -374.2120 | 0.2825 | -0.0766 | | 0.4864 | 0.58 | 1100 | 0.5172 | -1.1670 | -1.9894 | 0.75 | 0.8224 | -444.7675 | -378.1358 | 1.1814 | 0.7409 | | 0.5467 | 0.63 | 1200 | 0.5196 | -1.3633 | -2.1690 | 0.7383 | 0.8058 | -462.7306 | -397.7628 | 1.3020 | 0.8593 | | 0.5125 | 0.68 | 1300 | 0.5179 | -1.2033 | -2.0041 | 0.7422 | 0.8009 | -446.2437 | -381.7657 | 1.1045 | 0.6639 | | 0.4881 | 0.73 | 1400 | 0.5158 | -1.2792 | -2.1334 | 0.7539 | 0.8543 | -459.1728 | -389.3554 | 1.1891 | 0.7445 | | 0.5273 | 0.78 | 1500 | 0.5135 | -1.2081 | -2.0746 | 0.7539 | 0.8664 | -453.2860 | -382.2505 | 1.2533 | 0.7973 | | 0.5317 | 0.84 | 1600 | 0.5140 | -1.2815 | -2.1592 | 0.75 | 0.8777 | -461.7518 | -389.5859 | 1.2752 | 0.8202 | | 0.5384 | 0.89 | 1700 | 0.5134 | -1.2549 | -2.1287 | 0.7539 | 0.8738 | -458.7038 | -386.9291 | 1.2938 | 0.8384 | | 0.5619 | 0.94 | 1800 | 0.5135 | -1.2438 | -2.1108 | 0.7578 | 0.8670 | -456.9133 | -385.8195 | 1.2532 | 0.7986 | | 0.5169 | 0.99 | 1900 | 0.5133 | -1.2447 | -2.1118 | 0.7539 | 0.8671 | -457.0128 | -385.9082 | 1.2523 | 0.7989 | ### Framework versions - PEFT 0.7.1 - Transformers 4.36.2 - Pytorch 2.1.2+cu121 - Datasets 2.14.6 - Tokenizers 0.15.0