Model save

README.md

@@ -0,0 +1,89 @@
+---
+license: llama3
+base_model: RyanYr/openchat-3.6-8b-20240522_iter2
+tags:
+- trl
+- dpo
+- generated_from_trainer
+model-index:
+- name: openchat-3.6-8b-20240522_iter3
+  results: []
+---
+
+<!-- This model card has been generated automatically according to the information the Trainer had access to. You
+should probably proofread and complete it, then remove this comment. -->
+
+# openchat-3.6-8b-20240522_iter3
+
+This model is a fine-tuned version of [RyanYr/openchat-3.6-8b-20240522_iter2](https://huggingface.co/RyanYr/openchat-3.6-8b-20240522_iter2) on the None dataset.
+It achieves the following results on the evaluation set:
+- Loss: 0.5950
+- Rewards/chosen: -0.8463
+- Rewards/rejected: -1.5198
+- Rewards/accuracies: 0.7600
+- Rewards/margins: 0.6735
+- Logps/rejected: -149.1518
+- Logps/chosen: -144.0943
+- Logits/rejected: -1.2099
+- Logits/chosen: -1.2151
+
+## Model description
+
+More information needed
+
+## Intended uses & limitations
+
+More information needed
+
+## Training and evaluation data
+
+More information needed
+
+## Training procedure
+
+### Training hyperparameters
+
+The following hyperparameters were used during training:
+- learning_rate: 2e-07
+- train_batch_size: 1
+- eval_batch_size: 1
+- seed: 42
+- distributed_type: multi-GPU
+- num_devices: 4
+- gradient_accumulation_steps: 16
+- total_train_batch_size: 64
+- total_eval_batch_size: 4
+- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
+- lr_scheduler_type: cosine
+- lr_scheduler_warmup_steps: 100
+- num_epochs: 2
+
+### Training results
+
+| Training Loss | Epoch  | Step | Validation Loss | Rewards/chosen | Rewards/rejected | Rewards/accuracies | Rewards/margins | Logps/rejected | Logps/chosen | Logits/rejected | Logits/chosen |
+|:-------------:|:------:|:----:|:---------------:|:--------------:|:----------------:|:------------------:|:---------------:|:--------------:|:------------:|:---------------:|:-------------:|
+| 0.6665        | 0.1158 | 100  | 0.6985          | -0.1138        | -0.2545          | 0.6000             | 0.1407          | -136.4984      | -136.7689    | -1.2600         | -1.2619       |
+| 0.6393        | 0.2316 | 200  | 0.6824          | -0.1312        | -0.2764          | 0.5600             | 0.1452          | -136.7173      | -136.9431    | -1.3003         | -1.3045       |
+| 0.5871        | 0.3474 | 300  | 0.6834          | -0.2655        | -0.3888          | 0.6000             | 0.1233          | -137.8412      | -138.2859    | -1.3107         | -1.3155       |
+| 0.6151        | 0.4633 | 400  | 0.6799          | -0.4578        | -0.5623          | 0.6000             | 0.1045          | -139.5763      | -140.2087    | -1.3024         | -1.3069       |
+| 0.5577        | 0.5791 | 500  | 0.6544          | -0.3815        | -0.5536          | 0.6000             | 0.1722          | -139.4899      | -139.4459    | -1.3045         | -1.3100       |
+| 0.6366        | 0.6949 | 600  | 0.6261          | -0.1856        | -0.4357          | 0.6400             | 0.2500          | -138.3102      | -137.4874    | -1.3360         | -1.3430       |
+| 0.53          | 0.8107 | 700  | 0.6434          | -0.4043        | -0.6780          | 0.6400             | 0.2737          | -140.7333      | -139.6738    | -1.2803         | -1.2844       |
+| 0.5761        | 0.9265 | 800  | 0.6186          | -0.3762        | -0.6989          | 0.6400             | 0.3227          | -140.9429      | -139.3935    | -1.3125         | -1.3198       |
+| 0.4286        | 1.0423 | 900  | 0.6368          | -0.8084        | -1.1996          | 0.6800             | 0.3913          | -145.9498      | -143.7149    | -1.2632         | -1.2671       |
+| 0.407         | 1.1582 | 1000 | 0.6345          | -0.8524        | -1.3574          | 0.7200             | 0.5049          | -147.5273      | -144.1555    | -1.2234         | -1.2269       |
+| 0.4758        | 1.2740 | 1100 | 0.6022          | -0.6198        | -1.1935          | 0.6800             | 0.5738          | -145.8886      | -141.8288    | -1.2307         | -1.2366       |
+| 0.4415        | 1.3898 | 1200 | 0.5959          | -0.7170        | -1.3440          | 0.7200             | 0.6270          | -147.3939      | -142.8015    | -1.2248         | -1.2305       |
+| 0.4228        | 1.5056 | 1300 | 0.5890          | -0.6584        | -1.3069          | 0.7600             | 0.6485          | -147.0226      | -142.2149    | -1.2222         | -1.2276       |
+| 0.4199        | 1.6214 | 1400 | 0.6033          | -0.9116        | -1.5633          | 0.7200             | 0.6517          | -149.5865      | -144.7474    | -1.2084         | -1.2133       |
+| 0.4188        | 1.7372 | 1500 | 0.5948          | -0.8277        | -1.5036          | 0.7200             | 0.6759          | -148.9892      | -143.9083    | -1.2126         | -1.2175       |
+| 0.4185        | 1.8531 | 1600 | 0.5908          | -0.8393        | -1.5404          | 0.7600             | 0.7011          | -149.3580      | -144.0246    | -1.2042         | -1.2096       |
+| 0.3986        | 1.9689 | 1700 | 0.5950          | -0.8463        | -1.5198          | 0.7600             | 0.6735          | -149.1518      | -144.0943    | -1.2099         | -1.2151       |
+
+
+### Framework versions
+
+- Transformers 4.43.4
+- Pytorch 2.1.2+cu121
+- Datasets 2.14.6
+- Tokenizers 0.19.1

generation_config.json

@@ -0,0 +1,6 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 128000,
+  "eos_token_id": 128001,
+  "transformers_version": "4.43.4"
+}