Model save

README.md

@@ -0,0 +1,93 @@
+---
+license: apache-2.0
+base_model: hZzy/qwen2.5-0.5b-sft-news-IFT
+tags:
+- trl
+- expo
+- generated_from_trainer
+model-index:
+- name: qwen2.5-0.5b-expo-DPO-ES-1
+  results: []
+---
+
+<!-- This model card has been generated automatically according to the information the Trainer had access to. You
+should probably proofread and complete it, then remove this comment. -->
+
+[<img src="https://raw.githubusercontent.com/wandb/assets/main/wandb-github-badge-28.svg" alt="Visualize in Weights & Biases" width="200" height="32"/>](https://wandb.ai/zhiyuzha-university-of-florida/huggingface/runs/qhqu3qtk)
+# qwen2.5-0.5b-expo-DPO-ES-1
+
+This model is a fine-tuned version of [hZzy/qwen2.5-0.5b-sft-news-IFT](https://huggingface.co/hZzy/qwen2.5-0.5b-sft-news-IFT) on an unknown dataset.
+It achieves the following results on the evaluation set:
+- Loss: 2.3403
+- Logps: -83.2967
+- Logits: -0.7158
+- Objective: 2.2649
+- Dpo Loss: 2.2649
+- Regularize: 2.2649
+- Ranking Simple: 0.5331
+- Ranking Idealized: 0.5295
+- Ranking Idealized Expo: 0.5212
+- Wo Beta: 6.8864
+
+## Model description
+
+More information needed
+
+## Intended uses & limitations
+
+More information needed
+
+## Training and evaluation data
+
+More information needed
+
+## Training procedure
+
+### Training hyperparameters
+
+The following hyperparameters were used during training:
+- learning_rate: 5e-06
+- train_batch_size: 4
+- eval_batch_size: 4
+- seed: 42
+- distributed_type: multi-GPU
+- num_devices: 3
+- gradient_accumulation_steps: 12
+- total_train_batch_size: 144
+- total_eval_batch_size: 12
+- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
+- lr_scheduler_type: cosine
+- lr_scheduler_warmup_ratio: 0.1
+- num_epochs: 5
+
+### Training results
+
+| Training Loss | Epoch  | Step | Validation Loss | Logps    | Logits  | Objective | Dpo Loss | Regularize | Ranking Simple | Ranking Idealized | Ranking Idealized Expo | Wo Beta |
+|:-------------:|:------:|:----:|:---------------:|:--------:|:-------:|:---------:|:--------:|:----------:|:--------------:|:-----------------:|:----------------------:|:-------:|
+| 0.7017        | 0.1417 | 50   | 0.8470          | -93.0243 | -1.4582 | 0.8570    | 0.8570   | 0.8570     | 0.5238         | 0.5295            | 0.5212                 | 7.8507  |
+| 0.8112        | 0.2834 | 100  | 1.0529          | -86.6835 | -1.4382 | 1.0273    | 1.0273   | 1.0273     | 0.5285         | 0.5295            | 0.5212                 | 7.4982  |
+| 1.0895        | 0.4251 | 150  | 1.4497          | -84.4337 | -1.2965 | 1.4010    | 1.4010   | 1.4010     | 0.5321         | 0.5295            | 0.5212                 | 7.2692  |
+| 1.2363        | 0.5668 | 200  | 1.7035          | -77.7201 | -1.2956 | 1.6116    | 1.6116   | 1.6116     | 0.5321         | 0.5295            | 0.5212                 | 7.2264  |
+| 1.3152        | 0.7085 | 250  | 1.9222          | -92.7241 | -1.2565 | 1.8319    | 1.8319   | 1.8319     | 0.5311         | 0.5295            | 0.5212                 | 7.1856  |
+| 1.1899        | 0.8503 | 300  | 2.0298          | -90.9373 | -0.9785 | 1.9588    | 1.9588   | 1.9588     | 0.5367         | 0.5295            | 0.5212                 | 6.9336  |
+| 1.1443        | 0.9920 | 350  | 2.1654          | -82.1414 | -1.0214 | 2.0541    | 2.0541   | 2.0541     | 0.5435         | 0.5295            | 0.5212                 | 7.0024  |
+| 0.725         | 1.1337 | 400  | 2.2884          | -84.2526 | -0.7535 | 2.2360    | 2.2360   | 2.2360     | 0.5336         | 0.5295            | 0.5212                 | 7.1525  |
+| 0.7629        | 1.2754 | 450  | 2.1606          | -80.4165 | -0.8866 | 2.0671    | 2.0671   | 2.0671     | 0.5321         | 0.5295            | 0.5212                 | 6.7949  |
+| 0.8044        | 1.4171 | 500  | 2.2094          | -82.3927 | -0.7503 | 2.0981    | 2.0981   | 2.0981     | 0.5347         | 0.5295            | 0.5212                 | 6.8050  |
+| 0.7105        | 1.5588 | 550  | 2.1697          | -84.9780 | -0.6734 | 2.0733    | 2.0733   | 2.0733     | 0.5321         | 0.5295            | 0.5212                 | 6.8722  |
+| 0.6925        | 1.7005 | 600  | 2.1957          | -81.5342 | -0.7411 | 2.0558    | 2.0558   | 2.0558     | 0.5357         | 0.5295            | 0.5212                 | 6.7186  |
+| 0.6883        | 1.8422 | 650  | 2.2080          | -82.7303 | -0.6908 | 2.1330    | 2.1330   | 2.1330     | 0.5383         | 0.5295            | 0.5212                 | 6.8081  |
+| 0.6486        | 1.9839 | 700  | 2.3243          | -83.2882 | -0.6651 | 2.2471    | 2.2471   | 2.2471     | 0.5378         | 0.5295            | 0.5212                 | 6.6815  |
+| 0.3793        | 2.1256 | 750  | 2.2675          | -84.2296 | -0.7879 | 2.1825    | 2.1825   | 2.1825     | 0.5409         | 0.5295            | 0.5212                 | 6.8794  |
+| 0.3314        | 2.2674 | 800  | 2.2106          | -84.3675 | -0.6651 | 2.1041    | 2.1041   | 2.1041     | 0.5414         | 0.5295            | 0.5212                 | 6.7463  |
+| 0.3301        | 2.4091 | 850  | 2.2964          | -84.8913 | -0.6177 | 2.2221    | 2.2221   | 2.2221     | 0.5388         | 0.5295            | 0.5212                 | 6.8020  |
+| 0.3509        | 2.5508 | 900  | 2.2796          | -84.3833 | -0.6097 | 2.2099    | 2.2099   | 2.2099     | 0.5393         | 0.5295            | 0.5212                 | 6.7934  |
+| 0.321         | 2.6925 | 950  | 2.3403          | -83.2967 | -0.7158 | 2.2649    | 2.2649   | 2.2649     | 0.5331         | 0.5295            | 0.5212                 | 6.8864  |
+
+
+### Framework versions
+
+- Transformers 4.42.0
+- Pytorch 2.3.0+cu121
+- Datasets 2.19.1
+- Tokenizers 0.19.1

all_results.json

@@ -0,0 +1,9 @@
+{
+    "epoch": 2.69248937175248,
+    "total_flos": 0.0,
+    "train_loss": 0.7490891541932758,
+    "train_runtime": 28216.8085,
+    "train_samples": 50802,
+    "train_samples_per_second": 9.002,
+    "train_steps_per_second": 0.062
+}

generation_config.json

@@ -0,0 +1,7 @@
+{
+  "bos_token_id": 151644,
+  "eos_token_id": 151645,
+  "max_new_tokens": 2048,
+  "pad_token_id": 151645,
+  "transformers_version": "4.42.0"
+}

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:67ea4f637539ff138e995b360f68b133c612f4b6e652f6b3c9366920b7a336c9
+oid sha256:298052dd057c2cd532a59aca77a07211f0fb5c7cd9cfae8f6f2297e3de96205c
 size 1975192208

train_results.json

@@ -0,0 +1,9 @@
+{
+    "epoch": 2.69248937175248,
+    "total_flos": 0.0,
+    "train_loss": 0.7490891541932758,
+    "train_runtime": 28216.8085,
+    "train_samples": 50802,
+    "train_samples_per_second": 9.002,
+    "train_steps_per_second": 0.062
+}

trainer_state.json

@@ -0,0 +1,694 @@
+{
+  "best_metric": 6.681451797485352,
+  "best_model_checkpoint": "./qwen2.5-0.5b/qwen2.5-0.5b-expo-DPO-ES-1/checkpoint-700",
+  "epoch": 2.69248937175248,
+  "eval_steps": 50,
+  "global_step": 950,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "dpo_loss": 0.6931471824645996,
+      "epoch": 0.002834199338686821,
+      "grad_norm": 184.4249864582654,
+      "learning_rate": 2.840909090909091e-08,
+      "logits": -1.359458565711975,
+      "logps": -84.69721221923828,
+      "loss": 0.6931,
+      "objective": 0.6931471824645996,
+      "ranking_idealized": 0.5833333134651184,
+      "ranking_idealized_expo": 0.5833333134651184,
+      "ranking_simple": 0.5833333134651184,
+      "regularize": 0.6931471824645996,
+      "step": 1,
+      "wo_beta": 5.271125316619873
+    },
+    {
+      "dpo_loss": 0.7105385065078735,
+      "epoch": 0.14170996693434104,
+      "grad_norm": 184.86918480832395,
+      "learning_rate": 1.4204545454545458e-06,
+      "logits": -1.4640510082244873,
+      "logps": -84.16496276855469,
+      "loss": 0.7017,
+      "objective": 0.7105385065078735,
+      "ranking_idealized": 0.5289115905761719,
+      "ranking_idealized_expo": 0.5221088528633118,
+      "ranking_simple": 0.5225340127944946,
+      "regularize": 0.7105385065078735,
+      "step": 50,
+      "wo_beta": 7.0693583488464355
+    },
+    {
+      "epoch": 0.14170996693434104,
+      "eval_dpo_loss": 0.8569971919059753,
+      "eval_logits": -1.4581880569458008,
+      "eval_logps": -93.02434539794922,
+      "eval_loss": 0.8469977974891663,
+      "eval_objective": 0.8569971919059753,
+      "eval_ranking_idealized": 0.5295031070709229,
+      "eval_ranking_idealized_expo": 0.5212215185165405,
+      "eval_ranking_simple": 0.523809552192688,
+      "eval_regularize": 0.8569971919059753,
+      "eval_runtime": 308.8499,
+      "eval_samples_per_second": 18.747,
+      "eval_steps_per_second": 1.564,
+      "eval_wo_beta": 7.850653171539307,
+      "step": 50
+    },
+    {
+      "dpo_loss": 0.804880678653717,
+      "epoch": 0.2834199338686821,
+      "grad_norm": 165.0032665851806,
+      "learning_rate": 2.8409090909090916e-06,
+      "logits": -1.454908847808838,
+      "logps": -84.21910095214844,
+      "loss": 0.8112,
+      "objective": 0.804880678653717,
+      "ranking_idealized": 0.5241666436195374,
+      "ranking_idealized_expo": 0.5137500166893005,
+      "ranking_simple": 0.543749988079071,
+      "regularize": 0.804880678653717,
+      "step": 100,
+      "wo_beta": 6.3764801025390625
+    },
+    {
+      "epoch": 0.2834199338686821,
+      "eval_dpo_loss": 1.0273478031158447,
+      "eval_logits": -1.4382472038269043,
+      "eval_logps": -86.68347930908203,
+      "eval_loss": 1.0528956651687622,
+      "eval_objective": 1.0273478031158447,
+      "eval_ranking_idealized": 0.5295031070709229,
+      "eval_ranking_idealized_expo": 0.5212215185165405,
+      "eval_ranking_simple": 0.5284678936004639,
+      "eval_regularize": 1.0273478031158447,
+      "eval_runtime": 325.8896,
+      "eval_samples_per_second": 17.767,
+      "eval_steps_per_second": 1.482,
+      "eval_wo_beta": 7.498155117034912,
+      "step": 100
+    },
+    {
+      "dpo_loss": 1.0717346668243408,
+      "epoch": 0.42512990080302315,
+      "grad_norm": 132.12956442526567,
+      "learning_rate": 4.2613636363636365e-06,
+      "logits": -1.3764687776565552,
+      "logps": -75.39175415039062,
+      "loss": 1.0895,
+      "objective": 1.0717346668243408,
+      "ranking_idealized": 0.5333333611488342,
+      "ranking_idealized_expo": 0.527916669845581,
+      "ranking_simple": 0.5645833611488342,
+      "regularize": 1.0717346668243408,
+      "step": 150,
+      "wo_beta": 6.200186252593994
+    },
+    {
+      "epoch": 0.42512990080302315,
+      "eval_dpo_loss": 1.4009853601455688,
+      "eval_logits": -1.2964659929275513,
+      "eval_logps": -84.43374633789062,
+      "eval_loss": 1.4497475624084473,
+      "eval_objective": 1.4009853601455688,
+      "eval_ranking_idealized": 0.5295031070709229,
+      "eval_ranking_idealized_expo": 0.5212215185165405,
+      "eval_ranking_simple": 0.5320910811424255,
+      "eval_regularize": 1.4009853601455688,
+      "eval_runtime": 308.2116,
+      "eval_samples_per_second": 18.786,
+      "eval_steps_per_second": 1.567,
+      "eval_wo_beta": 7.269238471984863,
+      "step": 150
+    },
+    {
+      "dpo_loss": 1.2378712892532349,
+      "epoch": 0.5668398677373642,
+      "grad_norm": 120.89571295715444,
+      "learning_rate": 4.997168347957521e-06,
+      "logits": -1.3378713130950928,
+      "logps": -75.85318756103516,
+      "loss": 1.2363,
+      "objective": 1.2378712892532349,
+      "ranking_idealized": 0.5204166769981384,
+      "ranking_idealized_expo": 0.51541668176651,
+      "ranking_simple": 0.5699999928474426,
+      "regularize": 1.2378712892532349,
+      "step": 200,
+      "wo_beta": 6.13140869140625
+    },
+    {
+      "epoch": 0.5668398677373642,
+      "eval_dpo_loss": 1.6115626096725464,
+      "eval_logits": -1.295562505722046,
+      "eval_logps": -77.72007751464844,
+      "eval_loss": 1.7034852504730225,
+      "eval_objective": 1.6115626096725464,
+      "eval_ranking_idealized": 0.5295031070709229,
+      "eval_ranking_idealized_expo": 0.5212215185165405,
+      "eval_ranking_simple": 0.5320910811424255,
+      "eval_regularize": 1.6115626096725464,
+      "eval_runtime": 334.6734,
+      "eval_samples_per_second": 17.3,
+      "eval_steps_per_second": 1.443,
+      "eval_wo_beta": 7.226394176483154,
+      "step": 200
+    },
+    {
+      "dpo_loss": 1.2734622955322266,
+      "epoch": 0.7085498346717053,
+      "grad_norm": 126.97038898052826,
+      "learning_rate": 4.973122855144066e-06,
+      "logits": -1.1745208501815796,
+      "logps": -80.53473663330078,
+      "loss": 1.3152,
+      "objective": 1.2734622955322266,
+      "ranking_idealized": 0.5249999761581421,
+      "ranking_idealized_expo": 0.5162500143051147,
+      "ranking_simple": 0.5929166674613953,
+      "regularize": 1.2734622955322266,
+      "step": 250,
+      "wo_beta": 6.077574253082275
+    },
+    {
+      "epoch": 0.7085498346717053,
+      "eval_dpo_loss": 1.8319419622421265,
+      "eval_logits": -1.2564637660980225,
+      "eval_logps": -92.72406005859375,
+      "eval_loss": 1.9222145080566406,
+      "eval_objective": 1.8319419622421265,
+      "eval_ranking_idealized": 0.5295031070709229,
+      "eval_ranking_idealized_expo": 0.5212215185165405,
+      "eval_ranking_simple": 0.5310559272766113,
+      "eval_regularize": 1.8319419622421265,
+      "eval_runtime": 308.15,
+      "eval_samples_per_second": 18.79,
+      "eval_steps_per_second": 1.567,
+      "eval_wo_beta": 7.185611248016357,
+      "step": 250
+    },
+    {
+      "dpo_loss": 1.1058255434036255,
+      "epoch": 0.8502598016060463,
+      "grad_norm": 93.62858320092595,
+      "learning_rate": 4.924776641419513e-06,
+      "logits": -1.0275437831878662,
+      "logps": -86.21664428710938,
+      "loss": 1.1899,
+      "objective": 1.1058255434036255,
+      "ranking_idealized": 0.5062500238418579,
+      "ranking_idealized_expo": 0.4950000047683716,
+      "ranking_simple": 0.5833333134651184,
+      "regularize": 1.1058255434036255,
+      "step": 300,
+      "wo_beta": 5.932236194610596
+    },
+    {
+      "epoch": 0.8502598016060463,
+      "eval_dpo_loss": 1.958790898323059,
+      "eval_logits": -0.9785082340240479,
+      "eval_logps": -90.93729400634766,
+      "eval_loss": 2.0297622680664062,
+      "eval_objective": 1.958790898323059,
+      "eval_ranking_idealized": 0.5295031070709229,
+      "eval_ranking_idealized_expo": 0.5212215185165405,
+      "eval_ranking_simple": 0.5367494821548462,
+      "eval_regularize": 1.958790898323059,
+      "eval_runtime": 320.477,
+      "eval_samples_per_second": 18.067,
+      "eval_steps_per_second": 1.507,
+      "eval_wo_beta": 6.933594703674316,
+      "step": 300
+    },
+    {
+      "dpo_loss": 1.104241967201233,
+      "epoch": 0.9919697685403873,
+      "grad_norm": 81.69582378962005,
+      "learning_rate": 4.8526047530778175e-06,
+      "logits": -0.8600361943244934,
+      "logps": -83.0993423461914,
+      "loss": 1.1443,
+      "objective": 1.104241967201233,
+      "ranking_idealized": 0.5354166626930237,
+      "ranking_idealized_expo": 0.5254166722297668,
+      "ranking_simple": 0.6141666769981384,
+      "regularize": 1.104241967201233,
+      "step": 350,
+      "wo_beta": 5.234530925750732
+    },
+    {
+      "epoch": 0.9919697685403873,
+      "eval_dpo_loss": 2.0541160106658936,
+      "eval_logits": -1.0213720798492432,
+      "eval_logps": -82.14143371582031,
+      "eval_loss": 2.165400743484497,
+      "eval_objective": 2.0541160106658936,
+      "eval_ranking_idealized": 0.5295031070709229,
+      "eval_ranking_idealized_expo": 0.5212215185165405,
+      "eval_ranking_simple": 0.54347825050354,
+      "eval_regularize": 2.0541160106658936,
+      "eval_runtime": 352.7203,
+      "eval_samples_per_second": 16.415,
+      "eval_steps_per_second": 1.369,
+      "eval_wo_beta": 7.002377033233643,
+      "step": 350
+    },
+    {
+      "dpo_loss": 0.7012434005737305,
+      "epoch": 1.1336797354747283,
+      "grad_norm": 72.02321161366883,
+      "learning_rate": 4.757316345716554e-06,
+      "logits": -0.8228326439857483,
+      "logps": -77.82636260986328,
+      "loss": 0.725,
+      "objective": 0.7012434005737305,
+      "ranking_idealized": 0.5412499904632568,
+      "ranking_idealized_expo": 0.5320833325386047,
+      "ranking_simple": 0.6487500071525574,
+      "regularize": 0.7012434005737305,
+      "step": 400,
+      "wo_beta": 4.493627548217773
+    },
+    {
+      "epoch": 1.1336797354747283,
+      "eval_dpo_loss": 2.2360119819641113,
+      "eval_logits": -0.7534947991371155,
+      "eval_logps": -84.25261688232422,
+      "eval_loss": 2.288405179977417,
+      "eval_objective": 2.2360119819641113,
+      "eval_ranking_idealized": 0.5295031070709229,
+      "eval_ranking_idealized_expo": 0.5212215185165405,
+      "eval_ranking_simple": 0.533643901348114,
+      "eval_regularize": 2.2360119819641113,
+      "eval_runtime": 343.9192,
+      "eval_samples_per_second": 16.835,
+      "eval_steps_per_second": 1.404,
+      "eval_wo_beta": 7.152450084686279,
+      "step": 400
+    },
+    {
+      "dpo_loss": 0.787284255027771,
+      "epoch": 1.2753897024090695,
+      "grad_norm": 67.24645157140542,
+      "learning_rate": 4.639847716126855e-06,
+      "logits": -0.7304993867874146,
+      "logps": -79.94586181640625,
+      "loss": 0.7629,
+      "objective": 0.787284255027771,
+      "ranking_idealized": 0.5245833396911621,
+      "ranking_idealized_expo": 0.5191666483879089,
+      "ranking_simple": 0.6483333110809326,
+      "regularize": 0.787284255027771,
+      "step": 450,
+      "wo_beta": 5.019043922424316
+    },
+    {
+      "epoch": 1.2753897024090695,
+      "eval_dpo_loss": 2.0671327114105225,
+      "eval_logits": -0.8865557909011841,
+      "eval_logps": -80.41649627685547,
+      "eval_loss": 2.160627841949463,
+      "eval_objective": 2.0671327114105225,
+      "eval_ranking_idealized": 0.5295031070709229,
+      "eval_ranking_idealized_expo": 0.5212215185165405,
+      "eval_ranking_simple": 0.5320910811424255,
+      "eval_regularize": 2.0671327114105225,
+      "eval_runtime": 361.4999,
+      "eval_samples_per_second": 16.017,
+      "eval_steps_per_second": 1.336,
+      "eval_wo_beta": 6.794886589050293,
+      "step": 450
+    },
+    {
+      "dpo_loss": 0.7948352098464966,
+      "epoch": 1.4170996693434104,
+      "grad_norm": 80.7228445203749,
+      "learning_rate": 4.501353102310901e-06,
+      "logits": -0.7204355001449585,
+      "logps": -76.1670150756836,
+      "loss": 0.8044,
+      "objective": 0.7948352098464966,
+      "ranking_idealized": 0.5054166913032532,
+      "ranking_idealized_expo": 0.4970833361148834,
+      "ranking_simple": 0.6416666507720947,
+      "regularize": 0.7948352098464966,
+      "step": 500,
+      "wo_beta": 4.868573188781738
+    },
+    {
+      "epoch": 1.4170996693434104,
+      "eval_dpo_loss": 2.0981180667877197,
+      "eval_logits": -0.7502567172050476,
+      "eval_logps": -82.39266967773438,
+      "eval_loss": 2.2094054222106934,
+      "eval_objective": 2.0981180667877197,
+      "eval_ranking_idealized": 0.5295031070709229,
+      "eval_ranking_idealized_expo": 0.5212215185165405,
+      "eval_ranking_simple": 0.534679114818573,
+      "eval_regularize": 2.0981180667877197,
+      "eval_runtime": 308.1869,
+      "eval_samples_per_second": 18.787,
+      "eval_steps_per_second": 1.567,
+      "eval_wo_beta": 6.8049798011779785,
+      "step": 500
+    },
+    {
+      "dpo_loss": 0.7342749834060669,
+      "epoch": 1.5588096362777515,
+      "grad_norm": 72.59655396244632,
+      "learning_rate": 4.34319334202531e-06,
+      "logits": -0.6162157654762268,
+      "logps": -79.39734649658203,
+      "loss": 0.7105,
+      "objective": 0.7342749834060669,
+      "ranking_idealized": 0.5199999809265137,
+      "ranking_idealized_expo": 0.5108333230018616,
+      "ranking_simple": 0.6587499976158142,
+      "regularize": 0.7342749834060669,
+      "step": 550,
+      "wo_beta": 4.5973711013793945
+    },
+    {
+      "epoch": 1.5588096362777515,
+      "eval_dpo_loss": 2.073348045349121,
+      "eval_logits": -0.6734257340431213,
+      "eval_logps": -84.9780044555664,
+      "eval_loss": 2.16965651512146,
+      "eval_objective": 2.073348045349121,
+      "eval_ranking_idealized": 0.5295031070709229,
+      "eval_ranking_idealized_expo": 0.5212215185165405,
+      "eval_ranking_simple": 0.5320910811424255,
+      "eval_regularize": 2.073348045349121,
+      "eval_runtime": 374.3868,
+      "eval_samples_per_second": 15.465,
+      "eval_steps_per_second": 1.29,
+      "eval_wo_beta": 6.8721699714660645,
+      "step": 550
+    },
+    {
+      "dpo_loss": 0.6709804534912109,
+      "epoch": 1.7005196032120926,
+      "grad_norm": 58.045592434636404,
+      "learning_rate": 4.16692250129073e-06,
+      "logits": -0.5785077810287476,
+      "logps": -79.41590118408203,
+      "loss": 0.6925,
+      "objective": 0.6709804534912109,
+      "ranking_idealized": 0.5220833420753479,
+      "ranking_idealized_expo": 0.5149999856948853,
+      "ranking_simple": 0.6520833373069763,
+      "regularize": 0.6709804534912109,
+      "step": 600,
+      "wo_beta": 4.63525390625
+    },
+    {
+      "epoch": 1.7005196032120926,
+      "eval_dpo_loss": 2.055753707885742,
+      "eval_logits": -0.7411422729492188,
+      "eval_logps": -81.53422546386719,
+      "eval_loss": 2.195674419403076,
+      "eval_objective": 2.055753707885742,
+      "eval_ranking_idealized": 0.5295031070709229,
+      "eval_ranking_idealized_expo": 0.5212215185165405,
+      "eval_ranking_simple": 0.5357142686843872,
+      "eval_regularize": 2.055753707885742,
+      "eval_runtime": 308.1508,
+      "eval_samples_per_second": 18.79,
+      "eval_steps_per_second": 1.567,
+      "eval_wo_beta": 6.718571662902832,
+      "step": 600
+    },
+    {
+      "dpo_loss": 0.6708642244338989,
+      "epoch": 1.8422295701464337,
+      "grad_norm": 71.07668633244941,
+      "learning_rate": 3.974272604254906e-06,
+      "logits": -0.6567726135253906,
+      "logps": -78.93568420410156,
+      "loss": 0.6883,
+      "objective": 0.6708642244338989,
+      "ranking_idealized": 0.5408333539962769,
+      "ranking_idealized_expo": 0.527916669845581,
+      "ranking_simple": 0.6583333611488342,
+      "regularize": 0.6708642244338989,
+      "step": 650,
+      "wo_beta": 4.770144939422607
+    },
+    {
+      "epoch": 1.8422295701464337,
+      "eval_dpo_loss": 2.1330411434173584,
+      "eval_logits": -0.6908087134361267,
+      "eval_logps": -82.73030853271484,
+      "eval_loss": 2.208021640777588,
+      "eval_objective": 2.1330411434173584,
+      "eval_ranking_idealized": 0.5295031070709229,
+      "eval_ranking_idealized_expo": 0.5212215185165405,
+      "eval_ranking_simple": 0.5383023023605347,
+      "eval_regularize": 2.1330411434173584,
+      "eval_runtime": 351.6869,
+      "eval_samples_per_second": 16.464,
+      "eval_steps_per_second": 1.373,
+      "eval_wo_beta": 6.808131694793701,
+      "step": 650
+    },
+    {
+      "dpo_loss": 0.6912000179290771,
+      "epoch": 1.9839395370807746,
+      "grad_norm": 70.15640375480324,
+      "learning_rate": 3.767136614452458e-06,
+      "logits": -0.5081126093864441,
+      "logps": -78.9532699584961,
+      "loss": 0.6486,
+      "objective": 0.6912000179290771,
+      "ranking_idealized": 0.5170833468437195,
+      "ranking_idealized_expo": 0.5112500190734863,
+      "ranking_simple": 0.6445833444595337,
+      "regularize": 0.6912000179290771,
+      "step": 700,
+      "wo_beta": 4.81070613861084
+    },
+    {
+      "epoch": 1.9839395370807746,
+      "eval_dpo_loss": 2.2471413612365723,
+      "eval_logits": -0.6650525331497192,
+      "eval_logps": -83.28822326660156,
+      "eval_loss": 2.3242855072021484,
+      "eval_objective": 2.2471413612365723,
+      "eval_ranking_idealized": 0.5295031070709229,
+      "eval_ranking_idealized_expo": 0.5212215185165405,
+      "eval_ranking_simple": 0.5377846956253052,
+      "eval_regularize": 2.2471413612365723,
+      "eval_runtime": 308.3677,
+      "eval_samples_per_second": 18.776,
+      "eval_steps_per_second": 1.566,
+      "eval_wo_beta": 6.681451797485352,
+      "step": 700
+    },
+    {
+      "dpo_loss": 0.3827283978462219,
+      "epoch": 2.1256495040151155,
+      "grad_norm": 45.41487893439101,
+      "learning_rate": 3.547549834686222e-06,
+      "logits": -0.5765664577484131,
+      "logps": -79.59234619140625,
+      "loss": 0.3793,
+      "objective": 0.3827283978462219,
+      "ranking_idealized": 0.5216666460037231,
+      "ranking_idealized_expo": 0.5112500190734863,
+      "ranking_simple": 0.6779166460037231,
+      "regularize": 0.3827283978462219,
+      "step": 750,
+      "wo_beta": 4.318643569946289
+    },
+    {
+      "epoch": 2.1256495040151155,
+      "eval_dpo_loss": 2.182473659515381,
+      "eval_logits": -0.7878842353820801,
+      "eval_logps": -84.22957611083984,
+      "eval_loss": 2.2674732208251953,
+      "eval_objective": 2.182473659515381,
+      "eval_ranking_idealized": 0.5295031070709229,
+      "eval_ranking_idealized_expo": 0.5212215185165405,
+      "eval_ranking_simple": 0.5408902764320374,
+      "eval_regularize": 2.182473659515381,
+      "eval_runtime": 325.9016,
+      "eval_samples_per_second": 17.766,
+      "eval_steps_per_second": 1.482,
+      "eval_wo_beta": 6.879360675811768,
+      "step": 750
+    },
+    {
+      "dpo_loss": 0.3352104723453522,
+      "epoch": 2.2673594709494567,
+      "grad_norm": 40.69824942472133,
+      "learning_rate": 3.3176699082935546e-06,
+      "logits": -0.6111759543418884,
+      "logps": -80.46817779541016,
+      "loss": 0.3314,
+      "objective": 0.3352104723453522,
+      "ranking_idealized": 0.5195833444595337,
+      "ranking_idealized_expo": 0.512499988079071,
+      "ranking_simple": 0.6787499785423279,
+      "regularize": 0.3352104723453522,
+      "step": 800,
+      "wo_beta": 3.9888339042663574
+    },
+    {
+      "epoch": 2.2673594709494567,
+      "eval_dpo_loss": 2.104069471359253,
+      "eval_logits": -0.665122926235199,
+      "eval_logps": -84.3675308227539,
+      "eval_loss": 2.2106120586395264,
+      "eval_objective": 2.104069471359253,
+      "eval_ranking_idealized": 0.5295031070709229,
+      "eval_ranking_idealized_expo": 0.5212215185165405,
+      "eval_ranking_simple": 0.5414078831672668,
+      "eval_regularize": 2.104069471359253,
+      "eval_runtime": 341.6693,
+      "eval_samples_per_second": 16.946,
+      "eval_steps_per_second": 1.414,
+      "eval_wo_beta": 6.746298313140869,
+      "step": 800
+    },
+    {
+      "dpo_loss": 0.3527311086654663,
+      "epoch": 2.409069437883798,
+      "grad_norm": 34.29885501684047,
+      "learning_rate": 3.0797556183036582e-06,
+      "logits": -0.582695722579956,
+      "logps": -80.7869873046875,
+      "loss": 0.3301,
+      "objective": 0.3527311086654663,
+      "ranking_idealized": 0.5254166722297668,
+      "ranking_idealized_expo": 0.5141666531562805,
+      "ranking_simple": 0.6770833134651184,
+      "regularize": 0.3527311086654663,
+      "step": 850,
+      "wo_beta": 4.281794548034668
+    },
+    {
+      "epoch": 2.409069437883798,
+      "eval_dpo_loss": 2.222111940383911,
+      "eval_logits": -0.6176895499229431,
+      "eval_logps": -84.89134979248047,
+      "eval_loss": 2.296431303024292,
+      "eval_objective": 2.222111940383911,
+      "eval_ranking_idealized": 0.5295031070709229,
+      "eval_ranking_idealized_expo": 0.5212215185165405,
+      "eval_ranking_simple": 0.5388198494911194,
+      "eval_regularize": 2.222111940383911,
+      "eval_runtime": 308.4106,
+      "eval_samples_per_second": 18.774,
+      "eval_steps_per_second": 1.566,
+      "eval_wo_beta": 6.801980495452881,
+      "step": 850
+    },
+    {
+      "dpo_loss": 0.3560827672481537,
+      "epoch": 2.550779404818139,
+      "grad_norm": 59.77914010101811,
+      "learning_rate": 2.8361446928038298e-06,
+      "logits": -0.5723668336868286,
+      "logps": -79.74351501464844,
+      "loss": 0.3509,
+      "objective": 0.3560827672481537,
+      "ranking_idealized": 0.5249999761581421,
+      "ranking_idealized_expo": 0.5183333158493042,
+      "ranking_simple": 0.675000011920929,
+      "regularize": 0.3560827672481537,
+      "step": 900,
+      "wo_beta": 4.2809739112854
+    },
+    {
+      "epoch": 2.550779404818139,
+      "eval_dpo_loss": 2.2098634243011475,
+      "eval_logits": -0.6096944808959961,
+      "eval_logps": -84.38330841064453,
+      "eval_loss": 2.2795627117156982,
+      "eval_objective": 2.2098634243011475,
+      "eval_ranking_idealized": 0.5295031070709229,
+      "eval_ranking_idealized_expo": 0.5212215185165405,
+      "eval_ranking_simple": 0.5393374562263489,
+      "eval_regularize": 2.2098634243011475,
+      "eval_runtime": 342.057,
+      "eval_samples_per_second": 16.927,
+      "eval_steps_per_second": 1.412,
+      "eval_wo_beta": 6.793368339538574,
+      "step": 900
+    },
+    {
+      "dpo_loss": 0.3693172037601471,
+      "epoch": 2.69248937175248,
+      "grad_norm": 43.046650555153526,
+      "learning_rate": 2.5892308345974517e-06,
+      "logits": -0.5750654339790344,
+      "logps": -79.79653930664062,
+      "loss": 0.321,
+      "objective": 0.3693172037601471,
+      "ranking_idealized": 0.5162500143051147,
+      "ranking_idealized_expo": 0.5058333277702332,
+      "ranking_simple": 0.6879166960716248,
+      "regularize": 0.3693172037601471,
+      "step": 950,
+      "wo_beta": 4.201274871826172
+    },
+    {
+      "epoch": 2.69248937175248,
+      "eval_dpo_loss": 2.264862298965454,
+      "eval_logits": -0.7158052921295166,
+      "eval_logps": -83.29668426513672,
+      "eval_loss": 2.3403165340423584,
+      "eval_objective": 2.264862298965454,
+      "eval_ranking_idealized": 0.5295031070709229,
+      "eval_ranking_idealized_expo": 0.5212215185165405,
+      "eval_ranking_simple": 0.5331262946128845,
+      "eval_regularize": 2.264862298965454,
+      "eval_runtime": 315.8555,
+      "eval_samples_per_second": 18.331,
+      "eval_steps_per_second": 1.529,
+      "eval_wo_beta": 6.886437892913818,
+      "step": 950
+    },
+    {
+      "epoch": 2.69248937175248,
+      "step": 950,
+      "total_flos": 0.0,
+      "train_loss": 0.7490891541932758,
+      "train_runtime": 28216.8085,
+      "train_samples_per_second": 9.002,
+      "train_steps_per_second": 0.062
+    }
+  ],
+  "logging_steps": 50,
+  "max_steps": 1760,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 5,
+  "save_steps": 50,
+  "stateful_callbacks": {
+    "EarlyStoppingCallback": {
+      "args": {
+        "early_stopping_patience": 5,
+        "early_stopping_threshold": 0.0
+      },
+      "attributes": {
+        "early_stopping_patience_counter": 0
+      }
+    },
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": true
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 0.0,
+  "train_batch_size": 4,
+  "trial_name": null,
+  "trial_params": null
+}