metadata
license: llama3
base_model: RyanYr/openchat-3.6-8b-20240522_iter1
tags:
- trl
- dpo
- generated_from_trainer
model-index:
- name: openchat-3.6-8b-20240522_iter2
results: []
openchat-3.6-8b-20240522_iter2
This model is a fine-tuned version of RyanYr/openchat-3.6-8b-20240522_iter1 on the None dataset. It achieves the following results on the evaluation set:
- Loss: 0.5377
- Rewards/chosen: -0.2459
- Rewards/rejected: -0.7375
- Rewards/accuracies: 0.7200
- Rewards/margins: 0.4916
- Logps/rejected: -139.4380
- Logps/chosen: -132.3574
- Logits/rejected: -1.3194
- Logits/chosen: -1.3369
Model description
More information needed
Intended uses & limitations
More information needed
Training and evaluation data
More information needed
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 2e-07
- train_batch_size: 1
- eval_batch_size: 1
- seed: 42
- distributed_type: multi-GPU
- num_devices: 4
- gradient_accumulation_steps: 16
- total_train_batch_size: 64
- total_eval_batch_size: 4
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- lr_scheduler_warmup_steps: 100
- num_epochs: 2
Training results
| Training Loss | Epoch | Step | Validation Loss | Rewards/chosen | Rewards/rejected | Rewards/accuracies | Rewards/margins | Logps/rejected | Logps/chosen | Logits/rejected | Logits/chosen |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 0.7344 | 0.1140 | 100 | 0.7121 | 0.3107 | 0.3572 | 0.4000 | -0.0465 | -128.4905 | -126.7911 | -1.4523 | -1.4659 |
| 0.6553 | 0.2281 | 200 | 0.6966 | 0.2237 | 0.2513 | 0.5200 | -0.0276 | -129.5494 | -127.6609 | -1.4428 | -1.4571 |
| 0.669 | 0.3421 | 300 | 0.6798 | 0.1031 | 0.0581 | 0.5200 | 0.0450 | -131.4815 | -128.8666 | -1.4122 | -1.4281 |
| 0.6402 | 0.4561 | 400 | 0.6595 | 0.0694 | -0.0114 | 0.6400 | 0.0808 | -132.1772 | -129.2041 | -1.4254 | -1.4406 |
| 0.6716 | 0.5702 | 500 | 0.6351 | 0.1022 | -0.0221 | 0.6400 | 0.1243 | -132.2838 | -128.8764 | -1.4550 | -1.4689 |
| 0.655 | 0.6842 | 600 | 0.6278 | 0.1039 | -0.0286 | 0.6000 | 0.1325 | -132.3487 | -128.8587 | -1.4625 | -1.4766 |
| 0.5943 | 0.7982 | 700 | 0.6084 | 0.0643 | -0.1073 | 0.6400 | 0.1716 | -133.1360 | -129.2548 | -1.4485 | -1.4622 |
| 0.6048 | 0.9123 | 800 | 0.6002 | 0.0902 | -0.1175 | 0.6800 | 0.2077 | -133.2379 | -128.9962 | -1.4607 | -1.4735 |
| 0.4934 | 1.0263 | 900 | 0.5798 | 0.0298 | -0.2745 | 0.7200 | 0.3043 | -134.8078 | -129.5996 | -1.4349 | -1.4491 |
| 0.4284 | 1.1403 | 1000 | 0.5724 | -0.1252 | -0.4897 | 0.6800 | 0.3645 | -136.9601 | -131.1501 | -1.3824 | -1.3981 |
| 0.4132 | 1.2544 | 1100 | 0.5563 | -0.1930 | -0.5928 | 0.7600 | 0.3998 | -137.9906 | -131.8278 | -1.3545 | -1.3715 |
| 0.3957 | 1.3684 | 1200 | 0.5543 | -0.2162 | -0.6427 | 0.7600 | 0.4264 | -138.4894 | -132.0604 | -1.3412 | -1.3583 |
| 0.4893 | 1.4824 | 1300 | 0.5476 | -0.2078 | -0.6782 | 0.7200 | 0.4704 | -138.8445 | -131.9757 | -1.3340 | -1.3521 |
| 0.4361 | 1.5965 | 1400 | 0.5413 | -0.2007 | -0.6908 | 0.7200 | 0.4901 | -138.9703 | -131.9046 | -1.3316 | -1.3490 |
| 0.4406 | 1.7105 | 1500 | 0.5477 | -0.2466 | -0.6913 | 0.7200 | 0.4448 | -138.9762 | -132.3638 | -1.3242 | -1.3421 |
| 0.3988 | 1.8245 | 1600 | 0.5449 | -0.2388 | -0.7225 | 0.7200 | 0.4838 | -139.2881 | -132.2855 | -1.3254 | -1.3431 |
| 0.4044 | 1.9386 | 1700 | 0.5377 | -0.2459 | -0.7375 | 0.7200 | 0.4916 | -139.4380 | -132.3574 | -1.3194 | -1.3369 |
Framework versions
- Transformers 4.43.4
- Pytorch 2.1.2+cu121
- Datasets 2.14.6
- Tokenizers 0.19.1