---
license: llama3
base_model: RyanYr/openchat-3.6-8b-20240522_iter2
tags:
- trl
- dpo
- generated_from_trainer
model-index:
- name: openchat-3.6-8b-20240522_iter3
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# openchat-3.6-8b-20240522_iter3

This model is a fine-tuned version of [RyanYr/openchat-3.6-8b-20240522_iter2](https://huggingface.co/RyanYr/openchat-3.6-8b-20240522_iter2) on the None dataset.
It achieves the following results on the evaluation set:
- Loss: 0.5950
- Rewards/chosen: -0.8463
- Rewards/rejected: -1.5198
- Rewards/accuracies: 0.7600
- Rewards/margins: 0.6735
- Logps/rejected: -149.1518
- Logps/chosen: -144.0943
- Logits/rejected: -1.2099
- Logits/chosen: -1.2151

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 2e-07
- train_batch_size: 1
- eval_batch_size: 1
- seed: 42
- distributed_type: multi-GPU
- num_devices: 4
- gradient_accumulation_steps: 16
- total_train_batch_size: 64
- total_eval_batch_size: 4
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- lr_scheduler_warmup_steps: 100
- num_epochs: 2

### Training results

| Training Loss | Epoch  | Step | Validation Loss | Rewards/chosen | Rewards/rejected | Rewards/accuracies | Rewards/margins | Logps/rejected | Logps/chosen | Logits/rejected | Logits/chosen |
|:-------------:|:------:|:----:|:---------------:|:--------------:|:----------------:|:------------------:|:---------------:|:--------------:|:------------:|:---------------:|:-------------:|
| 0.6665        | 0.1158 | 100  | 0.6985          | -0.1138        | -0.2545          | 0.6000             | 0.1407          | -136.4984      | -136.7689    | -1.2600         | -1.2619       |
| 0.6393        | 0.2316 | 200  | 0.6824          | -0.1312        | -0.2764          | 0.5600             | 0.1452          | -136.7173      | -136.9431    | -1.3003         | -1.3045       |
| 0.5871        | 0.3474 | 300  | 0.6834          | -0.2655        | -0.3888          | 0.6000             | 0.1233          | -137.8412      | -138.2859    | -1.3107         | -1.3155       |
| 0.6151        | 0.4633 | 400  | 0.6799          | -0.4578        | -0.5623          | 0.6000             | 0.1045          | -139.5763      | -140.2087    | -1.3024         | -1.3069       |
| 0.5577        | 0.5791 | 500  | 0.6544          | -0.3815        | -0.5536          | 0.6000             | 0.1722          | -139.4899      | -139.4459    | -1.3045         | -1.3100       |
| 0.6366        | 0.6949 | 600  | 0.6261          | -0.1856        | -0.4357          | 0.6400             | 0.2500          | -138.3102      | -137.4874    | -1.3360         | -1.3430       |
| 0.53          | 0.8107 | 700  | 0.6434          | -0.4043        | -0.6780          | 0.6400             | 0.2737          | -140.7333      | -139.6738    | -1.2803         | -1.2844       |
| 0.5761        | 0.9265 | 800  | 0.6186          | -0.3762        | -0.6989          | 0.6400             | 0.3227          | -140.9429      | -139.3935    | -1.3125         | -1.3198       |
| 0.4286        | 1.0423 | 900  | 0.6368          | -0.8084        | -1.1996          | 0.6800             | 0.3913          | -145.9498      | -143.7149    | -1.2632         | -1.2671       |
| 0.407         | 1.1582 | 1000 | 0.6345          | -0.8524        | -1.3574          | 0.7200             | 0.5049          | -147.5273      | -144.1555    | -1.2234         | -1.2269       |
| 0.4758        | 1.2740 | 1100 | 0.6022          | -0.6198        | -1.1935          | 0.6800             | 0.5738          | -145.8886      | -141.8288    | -1.2307         | -1.2366       |
| 0.4415        | 1.3898 | 1200 | 0.5959          | -0.7170        | -1.3440          | 0.7200             | 0.6270          | -147.3939      | -142.8015    | -1.2248         | -1.2305       |
| 0.4228        | 1.5056 | 1300 | 0.5890          | -0.6584        | -1.3069          | 0.7600             | 0.6485          | -147.0226      | -142.2149    | -1.2222         | -1.2276       |
| 0.4199        | 1.6214 | 1400 | 0.6033          | -0.9116        | -1.5633          | 0.7200             | 0.6517          | -149.5865      | -144.7474    | -1.2084         | -1.2133       |
| 0.4188        | 1.7372 | 1500 | 0.5948          | -0.8277        | -1.5036          | 0.7200             | 0.6759          | -148.9892      | -143.9083    | -1.2126         | -1.2175       |
| 0.4185        | 1.8531 | 1600 | 0.5908          | -0.8393        | -1.5404          | 0.7600             | 0.7011          | -149.3580      | -144.0246    | -1.2042         | -1.2096       |
| 0.3986        | 1.9689 | 1700 | 0.5950          | -0.8463        | -1.5198          | 0.7600             | 0.6735          | -149.1518      | -144.0943    | -1.2099         | -1.2151       |


### Framework versions

- Transformers 4.43.4
- Pytorch 2.1.2+cu121
- Datasets 2.14.6
- Tokenizers 0.19.1