Hugging Face's logo

hZzy
/
qwen2.5-0.5b-expo-L2EXPO-ES-1000

Safetensors
qwen2
alignment-handbook
ndcg
trl
expo
Generated from Trainer
Model card Files Community
hZzy commited on
Commit
a52a8a0
•
1 Parent(s): 0e30399

Model save

Browse files
Files changed (6) hide show
  1. README.md +100 -0
  2. all_results.json +9 -0
  3. generation_config.json +7 -0
  4. model.safetensors +1 -1
  5. train_results.json +9 -0
  6. trainer_state.json +925 -0
README.md ADDED
@@ -0,0 +1,100 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ base_model: hZzy/qwen2.5-0.5b-sft-news-IFT
4
+ tags:
5
+ - trl
6
+ - expo
7
+ - generated_from_trainer
8
+ model-index:
9
+ - name: qwen2.5-0.5b-expo-L2EXPO-ES-1000
10
+ results: []
11
+ ---
12
+
13
+ <!-- This model card has been generated automatically according to the information the Trainer had access to. You
14
+ should probably proofread and complete it, then remove this comment. -->
15
+
16
+ [<img src="https://raw.githubusercontent.com/wandb/assets/main/wandb-github-badge-28.svg" alt="Visualize in Weights & Biases" width="200" height="32"/>](https://wandb.ai/zhiyuzha-university-of-florida/huggingface/runs/dy5f34x3)
17
+ # qwen2.5-0.5b-expo-L2EXPO-ES-1000
18
+
19
+ This model is a fine-tuned version of [hZzy/qwen2.5-0.5b-sft-news-IFT](https://huggingface.co/hZzy/qwen2.5-0.5b-sft-news-IFT) on an unknown dataset.
20
+ It achieves the following results on the evaluation set:
21
+ - Loss: 5143.4609
22
+ - Logps: -84.2929
23
+ - Logits: -0.5245
24
+ - Objective: 5173.7549
25
+ - Dpo Loss: 2621.0728
26
+ - Regularize: 5173.7549
27
+ - Ranking Simple: 0.5259
28
+ - Ranking Idealized: 0.5212
29
+ - Ranking Idealized Expo: 0.5212
30
+ - Wo Beta: 14.1584
31
+
32
+ ## Model description
33
+
34
+ More information needed
35
+
36
+ ## Intended uses & limitations
37
+
38
+ More information needed
39
+
40
+ ## Training and evaluation data
41
+
42
+ More information needed
43
+
44
+ ## Training procedure
45
+
46
+ ### Training hyperparameters
47
+
48
+ The following hyperparameters were used during training:
49
+ - learning_rate: 5e-06
50
+ - train_batch_size: 4
51
+ - eval_batch_size: 4
52
+ - seed: 42
53
+ - distributed_type: multi-GPU
54
+ - num_devices: 3
55
+ - gradient_accumulation_steps: 12
56
+ - total_train_batch_size: 144
57
+ - total_eval_batch_size: 12
58
+ - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
59
+ - lr_scheduler_type: cosine
60
+ - lr_scheduler_warmup_ratio: 0.1
61
+ - num_epochs: 5
62
+
63
+ ### Training results
64
+
65
+ | Training Loss | Epoch | Step | Dpo Loss | Logits | Logps | Validation Loss | Objective | Ranking Idealized | Ranking Idealized Expo | Ranking Simple | Regularize | Wo Beta |
66
+ |:-------------:|:------:|:----:|:---------:|:-------:|:--------:|:---------------:|:---------:|:-----------------:|:----------------------:|:--------------:|:----------:|:-------:|
67
+ | 423.0596 | 0.1417 | 50 | 269.6528 | -1.3995 | -90.7221 | 547.3513 | 547.0782 | 0.5212 | 0.5212 | 0.5238 | 547.0782 | 16.2386 |
68
+ | 1707.6314 | 0.2834 | 100 | 848.0440 | -1.3188 | -86.9761 | 1694.1810 | 1675.6274 | 0.5212 | 0.5212 | 0.5202 | 1675.6274 | 15.6407 |
69
+ | 2824.1562 | 0.4251 | 150 | 1511.3630 | -1.2862 | -82.2178 | 3014.1465 | 2968.7720 | 0.5212 | 0.5212 | 0.5274 | 2968.7720 | 15.0323 |
70
+ | 3551.5363 | 0.5668 | 200 | 1970.6587 | -0.7799 | -81.0056 | 3928.5837 | 3925.4968 | 0.5212 | 0.5212 | 0.5248 | 3925.4968 | 14.6132 |
71
+ | 3769.9247 | 0.7085 | 250 | 2167.5466 | -0.7280 | -80.5808 | 4317.6050 | 4303.1143 | 0.5212 | 0.5212 | 0.5269 | 4303.1143 | 14.5829 |
72
+ | 3591.3281 | 0.8503 | 300 | 2308.4351 | -0.5846 | -82.7713 | 4553.7632 | 4559.6348 | 0.5212 | 0.5212 | 0.5248 | 4559.6348 | 14.5914 |
73
+ | 3315.5613 | 0.9920 | 350 | 2326.0144 | -0.7541 | -80.8051 | 4667.9404 | 4670.2617 | 0.5212 | 0.5212 | 0.5331 | 4670.2617 | 14.3052 |
74
+ | 3140.2284 | 1.1337 | 400 | 2524.6191 | -0.6474 | -81.5771 | 4876.3184 | 4879.0815 | 0.5212 | 0.5212 | 0.5228 | 4879.0815 | 14.3271 |
75
+ | 2984.025 | 1.2754 | 450 | 2466.7131 | -0.7908 | -84.2705 | 4773.4326 | 4785.7534 | 0.5212 | 0.5212 | 0.5248 | 4785.7534 | 14.3213 |
76
+ | 2769.3719 | 1.4171 | 500 | 2513.8191 | -0.7098 | -81.4917 | 4863.7148 | 4866.6235 | 0.5212 | 0.5212 | 0.5192 | 4866.6235 | 14.1934 |
77
+ | 2620.0086 | 1.5588 | 550 | 2463.1169 | -0.5653 | -81.8307 | 4887.2939 | 4877.4683 | 0.5212 | 0.5212 | 0.5248 | 4877.4683 | 14.1757 |
78
+ | 2530.9462 | 1.7005 | 600 | 2522.0715 | -0.4886 | -82.8727 | 4965.4233 | 5013.2871 | 0.5212 | 0.5212 | 0.5233 | 5013.2871 | 14.2573 |
79
+ | 2445.0009 | 1.8422 | 650 | 2509.7644 | -0.5173 | -81.8303 | 4964.3994 | 4986.9541 | 0.5212 | 0.5212 | 0.5243 | 4986.9541 | 14.2557 |
80
+ | 2287.7192 | 1.9839 | 700 | 2561.1602 | -0.5354 | -83.8738 | 5034.0654 | 5065.8521 | 0.5212 | 0.5212 | 0.5217 | 5065.8521 | 14.0847 |
81
+ | 2066.9519 | 2.1256 | 750 | 2654.1794 | -0.4949 | -82.1944 | 5229.8853 | 5264.4932 | 0.5212 | 0.5212 | 0.5254 | 5264.4932 | 14.0981 |
82
+ | 1963.7713 | 2.2674 | 800 | 2636.3833 | -0.4790 | -82.2307 | 5180.2388 | 5235.7695 | 0.5212 | 0.5212 | 0.5243 | 5235.7695 | 14.0378 |
83
+ | 1854.7628 | 2.4091 | 850 | 2612.3875 | -0.4900 | -82.9664 | 5130.6069 | 5171.9189 | 0.5212 | 0.5212 | 0.5269 | 5171.9189 | 14.1142 |
84
+ | 1711.9678 | 2.5508 | 900 | 2703.0312 | -0.4645 | -85.1920 | 5280.8613 | 5329.0571 | 0.5212 | 0.5212 | 0.5264 | 5329.0571 | 14.0257 |
85
+ | 1682.3781 | 2.6925 | 950 | 2644.8484 | -0.4320 | -83.9376 | 5177.0815 | 5195.8457 | 0.5212 | 0.5212 | 0.5254 | 5195.8457 | 14.1691 |
86
+ | 1508.6941 | 2.8342 | 1000 | 2632.4006 | -0.5014 | -83.4235 | 5124.7144 | 5131.5728 | 0.5212 | 0.5212 | 0.5243 | 5131.5728 | 14.1501 |
87
+ | 1432.2169 | 2.9759 | 1050 | 2638.4963 | -0.4687 | -83.8074 | 5215.6191 | 5232.5947 | 0.5212 | 0.5212 | 0.5295 | 5232.5947 | 14.2389 |
88
+ | 1247.6562 | 3.1223 | 1100 | 5184.8696 | -84.1614| -0.5461 | 5190.6357 | 2631.7661 | 5190.6357 | 0.5264 | 0.5212 | 0.5212 | 14.1529 |
89
+ | 1136.2859 | 3.2641 | 1150 | 5110.2056 | -83.8852| -0.5632 | 5112.0278 | 2590.1838 | 5112.0278 | 0.5280 | 0.5212 | 0.5212 | 14.0933 |
90
+ | 1042.7762 | 3.4058 | 1200 | 5146.4077 | -83.9630| -0.5505 | 5162.1665 | 2612.2661 | 5162.1665 | 0.5274 | 0.5212 | 0.5212 | 14.1122 |
91
+ | 978.7787 | 3.5475 | 1250 | 5115.5093 | -83.8987| -0.4993 | 5140.9258 | 2605.3420 | 5140.9258 | 0.5280 | 0.5212 | 0.5212 | 14.1279 |
92
+ | 864.8715 | 3.6892 | 1300 | 5143.4609 | -84.2929| -0.5245 | 5173.7549 | 2621.0728 | 5173.7549 | 0.5259 | 0.5212 | 0.5212 | 14.1584 |
93
+
94
+
95
+ ### Framework versions
96
+
97
+ - Transformers 4.42.0
98
+ - Pytorch 2.3.0+cu121
99
+ - Datasets 2.19.1
100
+ - Tokenizers 0.19.1
all_results.json ADDED
@@ -0,0 +1,9 @@
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "epoch": 3.689182805857345,
3
+ "total_flos": 0.0,
4
+ "train_loss": 202.7064813701923,
5
+ "train_runtime": 6887.9795,
6
+ "train_samples": 50802,
7
+ "train_samples_per_second": 36.877,
8
+ "train_steps_per_second": 0.256
9
+ }
generation_config.json ADDED
@@ -0,0 +1,7 @@
 
 
 
 
 
 
 
 
1
+ {
2
+ "bos_token_id": 151644,
3
+ "eos_token_id": 151645,
4
+ "max_new_tokens": 2048,
5
+ "pad_token_id": 151645,
6
+ "transformers_version": "4.42.0"
7
+ }
model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:133737915baa3fdd16939ace92dd96ee2e4e50bd30967fe970ddd1760b6d6a24
3
  size 1975192208
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ae5d539285f7fb848ff4eb2e473a9553694bbc73a90cdcd28e55c47a487ab549
3
  size 1975192208
train_results.json ADDED
@@ -0,0 +1,9 @@
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "epoch": 3.689182805857345,
3
+ "total_flos": 0.0,
4
+ "train_loss": 202.7064813701923,
5
+ "train_runtime": 6887.9795,
6
+ "train_samples": 50802,
7
+ "train_samples_per_second": 36.877,
8
+ "train_steps_per_second": 0.256
9
+ }
trainer_state.json ADDED
@@ -0,0 +1,925 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "best_metric": 14.025749206542969,
3
+ "best_model_checkpoint": "./qwen2.5-0.5b/qwen2.5-0.5b-expo-L2EXPO-ES-1000/checkpoint-900",
4
+ "epoch": 3.689182805857345,
5
+ "eval_steps": 50,
6
+ "global_step": 1300,
7
+ "is_hyper_param_search": false,
8
+ "is_local_process_zero": true,
9
+ "is_world_process_zero": true,
10
+ "log_history": [
11
+ {
12
+ "dpo_loss": 0.6931471824645996,
13
+ "epoch": 0.002834199338686821,
14
+ "grad_norm": 368848.22474206856,
15
+ "learning_rate": 2.840909090909091e-08,
16
+ "logits": -1.359458565711975,
17
+ "logps": -84.69721221923828,
18
+ "loss": 0.3913,
19
+ "objective": 0.3618059456348419,
20
+ "ranking_idealized": 0.5833333134651184,
21
+ "ranking_idealized_expo": 0.5833333134651184,
22
+ "ranking_simple": 0.5833333134651184,
23
+ "regularize": 0.3618059456348419,
24
+ "step": 1,
25
+ "wo_beta": 14.830931663513184
26
+ },
27
+ {
28
+ "dpo_loss": 200.59906005859375,
29
+ "epoch": 0.14170996693434104,
30
+ "grad_norm": 364227.60279874475,
31
+ "learning_rate": 1.4204545454545458e-06,
32
+ "logits": -1.4551467895507812,
33
+ "logps": -84.4092788696289,
34
+ "loss": 423.0596,
35
+ "objective": 404.94305419921875,
36
+ "ranking_idealized": 0.5216836929321289,
37
+ "ranking_idealized_expo": 0.5216836929321289,
38
+ "ranking_simple": 0.5204081535339355,
39
+ "regularize": 404.94305419921875,
40
+ "step": 50,
41
+ "wo_beta": 15.620062828063965
42
+ },
43
+ {
44
+ "epoch": 0.14170996693434104,
45
+ "eval_dpo_loss": 269.6528015136719,
46
+ "eval_logits": -1.399541974067688,
47
+ "eval_logps": -90.7221450805664,
48
+ "eval_loss": 547.3512573242188,
49
+ "eval_objective": 547.0781860351562,
50
+ "eval_ranking_idealized": 0.5212215185165405,
51
+ "eval_ranking_idealized_expo": 0.5212215185165405,
52
+ "eval_ranking_simple": 0.523809552192688,
53
+ "eval_regularize": 547.0781860351562,
54
+ "eval_runtime": 309.3398,
55
+ "eval_samples_per_second": 18.717,
56
+ "eval_steps_per_second": 1.561,
57
+ "eval_wo_beta": 16.238561630249023,
58
+ "step": 50
59
+ },
60
+ {
61
+ "dpo_loss": 905.3652954101562,
62
+ "epoch": 0.2834199338686821,
63
+ "grad_norm": 272683.76538272557,
64
+ "learning_rate": 2.8409090909090916e-06,
65
+ "logits": -1.356760025024414,
66
+ "logps": -82.99842071533203,
67
+ "loss": 1707.6314,
68
+ "objective": 1747.880615234375,
69
+ "ranking_idealized": 0.5137500166893005,
70
+ "ranking_idealized_expo": 0.5137500166893005,
71
+ "ranking_simple": 0.5120833516120911,
72
+ "regularize": 1747.880615234375,
73
+ "step": 100,
74
+ "wo_beta": 15.263524055480957
75
+ },
76
+ {
77
+ "epoch": 0.2834199338686821,
78
+ "eval_dpo_loss": 848.0440063476562,
79
+ "eval_logits": -1.3187600374221802,
80
+ "eval_logps": -86.97607421875,
81
+ "eval_loss": 1694.1810302734375,
82
+ "eval_objective": 1675.62744140625,
83
+ "eval_ranking_idealized": 0.5212215185165405,
84
+ "eval_ranking_idealized_expo": 0.5212215185165405,
85
+ "eval_ranking_simple": 0.5201863646507263,
86
+ "eval_regularize": 1675.62744140625,
87
+ "eval_runtime": 309.6783,
88
+ "eval_samples_per_second": 18.697,
89
+ "eval_steps_per_second": 1.56,
90
+ "eval_wo_beta": 15.640727996826172,
91
+ "step": 100
92
+ },
93
+ {
94
+ "dpo_loss": 1518.7205810546875,
95
+ "epoch": 0.42512990080302315,
96
+ "grad_norm": 216991.1383206667,
97
+ "learning_rate": 4.2613636363636365e-06,
98
+ "logits": -1.2785295248031616,
99
+ "logps": -78.66228485107422,
100
+ "loss": 2824.1562,
101
+ "objective": 2852.929443359375,
102
+ "ranking_idealized": 0.527916669845581,
103
+ "ranking_idealized_expo": 0.527916669845581,
104
+ "ranking_simple": 0.5241666436195374,
105
+ "regularize": 2852.929443359375,
106
+ "step": 150,
107
+ "wo_beta": 15.12118148803711
108
+ },
109
+ {
110
+ "epoch": 0.42512990080302315,
111
+ "eval_dpo_loss": 1511.363037109375,
112
+ "eval_logits": -1.2861689329147339,
113
+ "eval_logps": -82.21775817871094,
114
+ "eval_loss": 3014.146484375,
115
+ "eval_objective": 2968.77197265625,
116
+ "eval_ranking_idealized": 0.5212215185165405,
117
+ "eval_ranking_idealized_expo": 0.5212215185165405,
118
+ "eval_ranking_simple": 0.5274327397346497,
119
+ "eval_regularize": 2968.77197265625,
120
+ "eval_runtime": 313.2906,
121
+ "eval_samples_per_second": 18.481,
122
+ "eval_steps_per_second": 1.542,
123
+ "eval_wo_beta": 15.032276153564453,
124
+ "step": 150
125
+ },
126
+ {
127
+ "dpo_loss": 1896.1614990234375,
128
+ "epoch": 0.5668398677373642,
129
+ "grad_norm": 205665.2147772606,
130
+ "learning_rate": 4.997168347957521e-06,
131
+ "logits": -0.987150251865387,
132
+ "logps": -77.89788055419922,
133
+ "loss": 3551.5363,
134
+ "objective": 3598.224609375,
135
+ "ranking_idealized": 0.51541668176651,
136
+ "ranking_idealized_expo": 0.51541668176651,
137
+ "ranking_simple": 0.5062500238418579,
138
+ "regularize": 3598.224609375,
139
+ "step": 200,
140
+ "wo_beta": 15.392863273620605
141
+ },
142
+ {
143
+ "epoch": 0.5668398677373642,
144
+ "eval_dpo_loss": 1970.65869140625,
145
+ "eval_logits": -0.7799269556999207,
146
+ "eval_logps": -81.00556945800781,
147
+ "eval_loss": 3928.583740234375,
148
+ "eval_objective": 3925.496826171875,
149
+ "eval_ranking_idealized": 0.5212215185165405,
150
+ "eval_ranking_idealized_expo": 0.5212215185165405,
151
+ "eval_ranking_simple": 0.5248447060585022,
152
+ "eval_regularize": 3925.496826171875,
153
+ "eval_runtime": 308.6708,
154
+ "eval_samples_per_second": 18.758,
155
+ "eval_steps_per_second": 1.565,
156
+ "eval_wo_beta": 14.613198280334473,
157
+ "step": 200
158
+ },
159
+ {
160
+ "dpo_loss": 1941.4073486328125,
161
+ "epoch": 0.7085498346717053,
162
+ "grad_norm": 204108.9756559366,
163
+ "learning_rate": 4.973122855144066e-06,
164
+ "logits": -0.7714519500732422,
165
+ "logps": -75.8036117553711,
166
+ "loss": 3769.9247,
167
+ "objective": 3796.307861328125,
168
+ "ranking_idealized": 0.5162500143051147,
169
+ "ranking_idealized_expo": 0.5162500143051147,
170
+ "ranking_simple": 0.5104166865348816,
171
+ "regularize": 3796.307861328125,
172
+ "step": 250,
173
+ "wo_beta": 15.552165985107422
174
+ },
175
+ {
176
+ "epoch": 0.7085498346717053,
177
+ "eval_dpo_loss": 2167.546630859375,
178
+ "eval_logits": -0.7279743552207947,
179
+ "eval_logps": -80.5808334350586,
180
+ "eval_loss": 4317.60498046875,
181
+ "eval_objective": 4303.1142578125,
182
+ "eval_ranking_idealized": 0.5212215185165405,
183
+ "eval_ranking_idealized_expo": 0.5212215185165405,
184
+ "eval_ranking_simple": 0.5269151329994202,
185
+ "eval_regularize": 4303.1142578125,
186
+ "eval_runtime": 309.0835,
187
+ "eval_samples_per_second": 18.733,
188
+ "eval_steps_per_second": 1.563,
189
+ "eval_wo_beta": 14.582874298095703,
190
+ "step": 250
191
+ },
192
+ {
193
+ "dpo_loss": 1839.3994140625,
194
+ "epoch": 0.8502598016060463,
195
+ "grad_norm": 180485.23157207714,
196
+ "learning_rate": 4.924776641419513e-06,
197
+ "logits": -0.5737716555595398,
198
+ "logps": -78.0151138305664,
199
+ "loss": 3591.3281,
200
+ "objective": 3690.973388671875,
201
+ "ranking_idealized": 0.4950000047683716,
202
+ "ranking_idealized_expo": 0.4950000047683716,
203
+ "ranking_simple": 0.5079166889190674,
204
+ "regularize": 3690.973388671875,
205
+ "step": 300,
206
+ "wo_beta": 15.1497163772583
207
+ },
208
+ {
209
+ "epoch": 0.8502598016060463,
210
+ "eval_dpo_loss": 2308.43505859375,
211
+ "eval_logits": -0.584596574306488,
212
+ "eval_logps": -82.77127075195312,
213
+ "eval_loss": 4553.76318359375,
214
+ "eval_objective": 4559.634765625,
215
+ "eval_ranking_idealized": 0.5212215185165405,
216
+ "eval_ranking_idealized_expo": 0.5212215185165405,
217
+ "eval_ranking_simple": 0.5248447060585022,
218
+ "eval_regularize": 4559.634765625,
219
+ "eval_runtime": 308.87,
220
+ "eval_samples_per_second": 18.746,
221
+ "eval_steps_per_second": 1.564,
222
+ "eval_wo_beta": 14.591386795043945,
223
+ "step": 300
224
+ },
225
+ {
226
+ "dpo_loss": 1713.5703125,
227
+ "epoch": 0.9919697685403873,
228
+ "grad_norm": 169039.33062429514,
229
+ "learning_rate": 4.8526047530778175e-06,
230
+ "logits": -0.629047155380249,
231
+ "logps": -77.71513366699219,
232
+ "loss": 3315.5613,
233
+ "objective": 3303.522705078125,
234
+ "ranking_idealized": 0.5254166722297668,
235
+ "ranking_idealized_expo": 0.5254166722297668,
236
+ "ranking_simple": 0.5233333110809326,
237
+ "regularize": 3303.522705078125,
238
+ "step": 350,
239
+ "wo_beta": 15.061549186706543
240
+ },
241
+ {
242
+ "epoch": 0.9919697685403873,
243
+ "eval_dpo_loss": 2326.014404296875,
244
+ "eval_logits": -0.7541128396987915,
245
+ "eval_logps": -80.80506896972656,
246
+ "eval_loss": 4667.9404296875,
247
+ "eval_objective": 4670.26171875,
248
+ "eval_ranking_idealized": 0.5212215185165405,
249
+ "eval_ranking_idealized_expo": 0.5212215185165405,
250
+ "eval_ranking_simple": 0.5331262946128845,
251
+ "eval_regularize": 4670.26171875,
252
+ "eval_runtime": 308.7398,
253
+ "eval_samples_per_second": 18.754,
254
+ "eval_steps_per_second": 1.564,
255
+ "eval_wo_beta": 14.305211067199707,
256
+ "step": 350
257
+ },
258
+ {
259
+ "dpo_loss": 1624.2042236328125,
260
+ "epoch": 1.1336797354747283,
261
+ "grad_norm": 165220.9620293089,
262
+ "learning_rate": 4.757316345716554e-06,
263
+ "logits": -0.6056586503982544,
264
+ "logps": -78.14913940429688,
265
+ "loss": 3140.2284,
266
+ "objective": 3177.530029296875,
267
+ "ranking_idealized": 0.5320833325386047,
268
+ "ranking_idealized_expo": 0.5320833325386047,
269
+ "ranking_simple": 0.5299999713897705,
270
+ "regularize": 3177.530029296875,
271
+ "step": 400,
272
+ "wo_beta": 15.318251609802246
273
+ },
274
+ {
275
+ "epoch": 1.1336797354747283,
276
+ "eval_dpo_loss": 2524.619140625,
277
+ "eval_logits": -0.6474378108978271,
278
+ "eval_logps": -81.57711029052734,
279
+ "eval_loss": 4876.318359375,
280
+ "eval_objective": 4879.08154296875,
281
+ "eval_ranking_idealized": 0.5212215185165405,
282
+ "eval_ranking_idealized_expo": 0.5212215185165405,
283
+ "eval_ranking_simple": 0.522774338722229,
284
+ "eval_regularize": 4879.08154296875,
285
+ "eval_runtime": 308.5826,
286
+ "eval_samples_per_second": 18.763,
287
+ "eval_steps_per_second": 1.565,
288
+ "eval_wo_beta": 14.327096939086914,
289
+ "step": 400
290
+ },
291
+ {
292
+ "dpo_loss": 1540.3419189453125,
293
+ "epoch": 1.2753897024090695,
294
+ "grad_norm": 171392.04304109752,
295
+ "learning_rate": 4.639847716126855e-06,
296
+ "logits": -0.636415421962738,
297
+ "logps": -79.45496368408203,
298
+ "loss": 2984.025,
299
+ "objective": 3080.5654296875,
300
+ "ranking_idealized": 0.5191666483879089,
301
+ "ranking_idealized_expo": 0.5191666483879089,
302
+ "ranking_simple": 0.5229166746139526,
303
+ "regularize": 3080.5654296875,
304
+ "step": 450,
305
+ "wo_beta": 15.8497314453125
306
+ },
307
+ {
308
+ "epoch": 1.2753897024090695,
309
+ "eval_dpo_loss": 2466.713134765625,
310
+ "eval_logits": -0.79082190990448,
311
+ "eval_logps": -84.27051544189453,
312
+ "eval_loss": 4773.4326171875,
313
+ "eval_objective": 4785.75341796875,
314
+ "eval_ranking_idealized": 0.5212215185165405,
315
+ "eval_ranking_idealized_expo": 0.5212215185165405,
316
+ "eval_ranking_simple": 0.5248447060585022,
317
+ "eval_regularize": 4785.75341796875,
318
+ "eval_runtime": 309.0671,
319
+ "eval_samples_per_second": 18.734,
320
+ "eval_steps_per_second": 1.563,
321
+ "eval_wo_beta": 14.321349143981934,
322
+ "step": 450
323
+ },
324
+ {
325
+ "dpo_loss": 1396.68310546875,
326
+ "epoch": 1.4170996693434104,
327
+ "grad_norm": 149255.8927126336,
328
+ "learning_rate": 4.501353102310901e-06,
329
+ "logits": -0.6039130687713623,
330
+ "logps": -79.58264923095703,
331
+ "loss": 2769.3719,
332
+ "objective": 2762.444580078125,
333
+ "ranking_idealized": 0.4970833361148834,
334
+ "ranking_idealized_expo": 0.4970833361148834,
335
+ "ranking_simple": 0.49916666746139526,
336
+ "regularize": 2762.444580078125,
337
+ "step": 500,
338
+ "wo_beta": 15.438196182250977
339
+ },
340
+ {
341
+ "epoch": 1.4170996693434104,
342
+ "eval_dpo_loss": 2513.819091796875,
343
+ "eval_logits": -0.7098349332809448,
344
+ "eval_logps": -81.49170684814453,
345
+ "eval_loss": 4863.71484375,
346
+ "eval_objective": 4866.62353515625,
347
+ "eval_ranking_idealized": 0.5212215185165405,
348
+ "eval_ranking_idealized_expo": 0.5212215185165405,
349
+ "eval_ranking_simple": 0.5191511511802673,
350
+ "eval_regularize": 4866.62353515625,
351
+ "eval_runtime": 308.6488,
352
+ "eval_samples_per_second": 18.759,
353
+ "eval_steps_per_second": 1.565,
354
+ "eval_wo_beta": 14.19344425201416,
355
+ "step": 500
356
+ },
357
+ {
358
+ "dpo_loss": 1364.8533935546875,
359
+ "epoch": 1.5588096362777515,
360
+ "grad_norm": 159666.559697962,
361
+ "learning_rate": 4.34319334202531e-06,
362
+ "logits": -0.5462002158164978,
363
+ "logps": -79.02467346191406,
364
+ "loss": 2620.0086,
365
+ "objective": 2664.316162109375,
366
+ "ranking_idealized": 0.5104166865348816,
367
+ "ranking_idealized_expo": 0.5104166865348816,
368
+ "ranking_simple": 0.5095833539962769,
369
+ "regularize": 2664.316162109375,
370
+ "step": 550,
371
+ "wo_beta": 15.136716842651367
372
+ },
373
+ {
374
+ "epoch": 1.5588096362777515,
375
+ "eval_dpo_loss": 2463.116943359375,
376
+ "eval_logits": -0.5653336644172668,
377
+ "eval_logps": -81.83069610595703,
378
+ "eval_loss": 4887.2939453125,
379
+ "eval_objective": 4877.46826171875,
380
+ "eval_ranking_idealized": 0.5212215185165405,
381
+ "eval_ranking_idealized_expo": 0.5212215185165405,
382
+ "eval_ranking_simple": 0.5248447060585022,
383
+ "eval_regularize": 4877.46826171875,
384
+ "eval_runtime": 308.8536,
385
+ "eval_samples_per_second": 18.747,
386
+ "eval_steps_per_second": 1.564,
387
+ "eval_wo_beta": 14.175724029541016,
388
+ "step": 550
389
+ },
390
+ {
391
+ "dpo_loss": 1253.067138671875,
392
+ "epoch": 1.7005196032120926,
393
+ "grad_norm": 158366.9559774092,
394
+ "learning_rate": 4.16692250129073e-06,
395
+ "logits": -0.4213114380836487,
396
+ "logps": -78.87406921386719,
397
+ "loss": 2530.9462,
398
+ "objective": 2507.11328125,
399
+ "ranking_idealized": 0.5149999856948853,
400
+ "ranking_idealized_expo": 0.5149999856948853,
401
+ "ranking_simple": 0.5095833539962769,
402
+ "regularize": 2507.11328125,
403
+ "step": 600,
404
+ "wo_beta": 15.036109924316406
405
+ },
406
+ {
407
+ "epoch": 1.7005196032120926,
408
+ "eval_dpo_loss": 2522.071533203125,
409
+ "eval_logits": -0.48861420154571533,
410
+ "eval_logps": -82.87271881103516,
411
+ "eval_loss": 4965.42333984375,
412
+ "eval_objective": 5013.287109375,
413
+ "eval_ranking_idealized": 0.5212215185165405,
414
+ "eval_ranking_idealized_expo": 0.5212215185165405,
415
+ "eval_ranking_simple": 0.5232919454574585,
416
+ "eval_regularize": 5013.287109375,
417
+ "eval_runtime": 308.7234,
418
+ "eval_samples_per_second": 18.755,
419
+ "eval_steps_per_second": 1.565,
420
+ "eval_wo_beta": 14.257344245910645,
421
+ "step": 600
422
+ },
423
+ {
424
+ "dpo_loss": 1245.341064453125,
425
+ "epoch": 1.8422295701464337,
426
+ "grad_norm": 159825.03654087533,
427
+ "learning_rate": 3.974272604254906e-06,
428
+ "logits": -0.44398489594459534,
429
+ "logps": -79.5468521118164,
430
+ "loss": 2445.0009,
431
+ "objective": 2442.403076171875,
432
+ "ranking_idealized": 0.527916669845581,
433
+ "ranking_idealized_expo": 0.527916669845581,
434
+ "ranking_simple": 0.5249999761581421,
435
+ "regularize": 2442.403076171875,
436
+ "step": 650,
437
+ "wo_beta": 15.771297454833984
438
+ },
439
+ {
440
+ "epoch": 1.8422295701464337,
441
+ "eval_dpo_loss": 2509.764404296875,
442
+ "eval_logits": -0.5172779560089111,
443
+ "eval_logps": -81.83026123046875,
444
+ "eval_loss": 4964.3994140625,
445
+ "eval_objective": 4986.9541015625,
446
+ "eval_ranking_idealized": 0.5212215185165405,
447
+ "eval_ranking_idealized_expo": 0.5212215185165405,
448
+ "eval_ranking_simple": 0.5243270993232727,
449
+ "eval_regularize": 4986.9541015625,
450
+ "eval_runtime": 309.186,
451
+ "eval_samples_per_second": 18.727,
452
+ "eval_steps_per_second": 1.562,
453
+ "eval_wo_beta": 14.255688667297363,
454
+ "step": 650
455
+ },
456
+ {
457
+ "dpo_loss": 1206.533203125,
458
+ "epoch": 1.9839395370807746,
459
+ "grad_norm": 156012.46785149653,
460
+ "learning_rate": 3.767136614452458e-06,
461
+ "logits": -0.4590357840061188,
462
+ "logps": -80.01580047607422,
463
+ "loss": 2287.7192,
464
+ "objective": 2330.523193359375,
465
+ "ranking_idealized": 0.5108333230018616,
466
+ "ranking_idealized_expo": 0.5108333230018616,
467
+ "ranking_simple": 0.5108333230018616,
468
+ "regularize": 2330.523193359375,
469
+ "step": 700,
470
+ "wo_beta": 15.533180236816406
471
+ },
472
+ {
473
+ "epoch": 1.9839395370807746,
474
+ "eval_dpo_loss": 2561.16015625,
475
+ "eval_logits": -0.5353549718856812,
476
+ "eval_logps": -83.87384033203125,
477
+ "eval_loss": 5034.0654296875,
478
+ "eval_objective": 5065.85205078125,
479
+ "eval_ranking_idealized": 0.5212215185165405,
480
+ "eval_ranking_idealized_expo": 0.5212215185165405,
481
+ "eval_ranking_simple": 0.52173912525177,
482
+ "eval_regularize": 5065.85205078125,
483
+ "eval_runtime": 309.3567,
484
+ "eval_samples_per_second": 18.716,
485
+ "eval_steps_per_second": 1.561,
486
+ "eval_wo_beta": 14.084709167480469,
487
+ "step": 700
488
+ },
489
+ {
490
+ "dpo_loss": 1036.373779296875,
491
+ "epoch": 2.1256495040151155,
492
+ "grad_norm": 147121.88795994341,
493
+ "learning_rate": 3.547549834686222e-06,
494
+ "logits": -0.47968342900276184,
495
+ "logps": -79.23291015625,
496
+ "loss": 2066.9519,
497
+ "objective": 2005.8623046875,
498
+ "ranking_idealized": 0.5112500190734863,
499
+ "ranking_idealized_expo": 0.5112500190734863,
500
+ "ranking_simple": 0.5066666603088379,
501
+ "regularize": 2005.8623046875,
502
+ "step": 750,
503
+ "wo_beta": 15.400221824645996
504
+ },
505
+ {
506
+ "epoch": 2.1256495040151155,
507
+ "eval_dpo_loss": 2654.179443359375,
508
+ "eval_logits": -0.49486637115478516,
509
+ "eval_logps": -82.19441223144531,
510
+ "eval_loss": 5229.88525390625,
511
+ "eval_objective": 5264.4931640625,
512
+ "eval_ranking_idealized": 0.5212215185165405,
513
+ "eval_ranking_idealized_expo": 0.5212215185165405,
514
+ "eval_ranking_simple": 0.5253623127937317,
515
+ "eval_regularize": 5264.4931640625,
516
+ "eval_runtime": 308.7479,
517
+ "eval_samples_per_second": 18.753,
518
+ "eval_steps_per_second": 1.564,
519
+ "eval_wo_beta": 14.098138809204102,
520
+ "step": 750
521
+ },
522
+ {
523
+ "dpo_loss": 994.4859619140625,
524
+ "epoch": 2.2673594709494567,
525
+ "grad_norm": 143685.9375211257,
526
+ "learning_rate": 3.3176699082935546e-06,
527
+ "logits": -0.466074675321579,
528
+ "logps": -79.23563385009766,
529
+ "loss": 1963.7713,
530
+ "objective": 1942.085205078125,
531
+ "ranking_idealized": 0.512499988079071,
532
+ "ranking_idealized_expo": 0.512499988079071,
533
+ "ranking_simple": 0.5108333230018616,
534
+ "regularize": 1942.085205078125,
535
+ "step": 800,
536
+ "wo_beta": 14.980566024780273
537
+ },
538
+ {
539
+ "epoch": 2.2673594709494567,
540
+ "eval_dpo_loss": 2636.38330078125,
541
+ "eval_logits": -0.47902175784111023,
542
+ "eval_logps": -82.23072052001953,
543
+ "eval_loss": 5180.23876953125,
544
+ "eval_objective": 5235.76953125,
545
+ "eval_ranking_idealized": 0.5212215185165405,
546
+ "eval_ranking_idealized_expo": 0.5212215185165405,
547
+ "eval_ranking_simple": 0.5243270993232727,
548
+ "eval_regularize": 5235.76953125,
549
+ "eval_runtime": 308.7297,
550
+ "eval_samples_per_second": 18.754,
551
+ "eval_steps_per_second": 1.564,
552
+ "eval_wo_beta": 14.037761688232422,
553
+ "step": 800
554
+ },
555
+ {
556
+ "dpo_loss": 922.8798828125,
557
+ "epoch": 2.409069437883798,
558
+ "grad_norm": 142497.46137454137,
559
+ "learning_rate": 3.0797556183036582e-06,
560
+ "logits": -0.435017853975296,
561
+ "logps": -80.0512466430664,
562
+ "loss": 1854.7628,
563
+ "objective": 1856.9925537109375,
564
+ "ranking_idealized": 0.5133333206176758,
565
+ "ranking_idealized_expo": 0.5133333206176758,
566
+ "ranking_simple": 0.5149999856948853,
567
+ "regularize": 1856.9925537109375,
568
+ "step": 850,
569
+ "wo_beta": 15.263174057006836
570
+ },
571
+ {
572
+ "epoch": 2.409069437883798,
573
+ "eval_dpo_loss": 2612.387451171875,
574
+ "eval_logits": -0.4899817407131195,
575
+ "eval_logps": -82.96641540527344,
576
+ "eval_loss": 5130.60693359375,
577
+ "eval_objective": 5171.9189453125,
578
+ "eval_ranking_idealized": 0.5212215185165405,
579
+ "eval_ranking_idealized_expo": 0.5212215185165405,
580
+ "eval_ranking_simple": 0.5269151329994202,
581
+ "eval_regularize": 5171.9189453125,
582
+ "eval_runtime": 313.09,
583
+ "eval_samples_per_second": 18.493,
584
+ "eval_steps_per_second": 1.543,
585
+ "eval_wo_beta": 14.114248275756836,
586
+ "step": 850
587
+ },
588
+ {
589
+ "dpo_loss": 880.90771484375,
590
+ "epoch": 2.550779404818139,
591
+ "grad_norm": 149283.81145436864,
592
+ "learning_rate": 2.8361446928038298e-06,
593
+ "logits": -0.4352048635482788,
594
+ "logps": -80.65638732910156,
595
+ "loss": 1711.9678,
596
+ "objective": 1688.95556640625,
597
+ "ranking_idealized": 0.5183333158493042,
598
+ "ranking_idealized_expo": 0.5179166793823242,
599
+ "ranking_simple": 0.5204166769981384,
600
+ "regularize": 1688.95556640625,
601
+ "step": 900,
602
+ "wo_beta": 15.470394134521484
603
+ },
604
+ {
605
+ "epoch": 2.550779404818139,
606
+ "eval_dpo_loss": 2703.03125,
607
+ "eval_logits": -0.4645061194896698,
608
+ "eval_logps": -85.19200134277344,
609
+ "eval_loss": 5280.861328125,
610
+ "eval_objective": 5329.05712890625,
611
+ "eval_ranking_idealized": 0.5212215185165405,
612
+ "eval_ranking_idealized_expo": 0.5212215185165405,
613
+ "eval_ranking_simple": 0.5263975262641907,
614
+ "eval_regularize": 5329.05712890625,
615
+ "eval_runtime": 309.3219,
616
+ "eval_samples_per_second": 18.718,
617
+ "eval_steps_per_second": 1.561,
618
+ "eval_wo_beta": 14.025749206542969,
619
+ "step": 900
620
+ },
621
+ {
622
+ "dpo_loss": 851.3322143554688,
623
+ "epoch": 2.69248937175248,
624
+ "grad_norm": 149343.91741290496,
625
+ "learning_rate": 2.5892308345974517e-06,
626
+ "logits": -0.4298136830329895,
627
+ "logps": -81.61705017089844,
628
+ "loss": 1682.3781,
629
+ "objective": 1668.57568359375,
630
+ "ranking_idealized": 0.5058333277702332,
631
+ "ranking_idealized_expo": 0.5058333277702332,
632
+ "ranking_simple": 0.5095833539962769,
633
+ "regularize": 1668.57568359375,
634
+ "step": 950,
635
+ "wo_beta": 15.541181564331055
636
+ },
637
+ {
638
+ "epoch": 2.69248937175248,
639
+ "eval_dpo_loss": 2644.848388671875,
640
+ "eval_logits": -0.4320021867752075,
641
+ "eval_logps": -83.93761444091797,
642
+ "eval_loss": 5177.08154296875,
643
+ "eval_objective": 5195.845703125,
644
+ "eval_ranking_idealized": 0.5212215185165405,
645
+ "eval_ranking_idealized_expo": 0.5212215185165405,
646
+ "eval_ranking_simple": 0.5253623127937317,
647
+ "eval_regularize": 5195.845703125,
648
+ "eval_runtime": 308.7152,
649
+ "eval_samples_per_second": 18.755,
650
+ "eval_steps_per_second": 1.565,
651
+ "eval_wo_beta": 14.169083595275879,
652
+ "step": 950
653
+ },
654
+ {
655
+ "dpo_loss": 776.9803466796875,
656
+ "epoch": 2.8341993386868207,
657
+ "grad_norm": 137448.18146621465,
658
+ "learning_rate": 2.341440200858589e-06,
659
+ "logits": -0.4145285189151764,
660
+ "logps": -79.74474334716797,
661
+ "loss": 1508.6941,
662
+ "objective": 1524.09228515625,
663
+ "ranking_idealized": 0.5095833539962769,
664
+ "ranking_idealized_expo": 0.5095833539962769,
665
+ "ranking_simple": 0.5054166913032532,
666
+ "regularize": 1524.09228515625,
667
+ "step": 1000,
668
+ "wo_beta": 15.0802640914917
669
+ },
670
+ {
671
+ "epoch": 2.8341993386868207,
672
+ "eval_dpo_loss": 2632.400634765625,
673
+ "eval_logits": -0.5013601779937744,
674
+ "eval_logps": -83.42354583740234,
675
+ "eval_loss": 5124.71435546875,
676
+ "eval_objective": 5131.57275390625,
677
+ "eval_ranking_idealized": 0.5212215185165405,
678
+ "eval_ranking_idealized_expo": 0.5212215185165405,
679
+ "eval_ranking_simple": 0.5243270993232727,
680
+ "eval_regularize": 5131.57275390625,
681
+ "eval_runtime": 311.7146,
682
+ "eval_samples_per_second": 18.575,
683
+ "eval_steps_per_second": 1.549,
684
+ "eval_wo_beta": 14.1500883102417,
685
+ "step": 1000
686
+ },
687
+ {
688
+ "dpo_loss": 708.8336181640625,
689
+ "epoch": 2.975909305621162,
690
+ "grad_norm": 147550.24239683093,
691
+ "learning_rate": 2.0952075638923656e-06,
692
+ "logits": -0.49126318097114563,
693
+ "logps": -79.99681854248047,
694
+ "loss": 1432.2169,
695
+ "objective": 1416.6346435546875,
696
+ "ranking_idealized": 0.5170833468437195,
697
+ "ranking_idealized_expo": 0.5170833468437195,
698
+ "ranking_simple": 0.5212500095367432,
699
+ "regularize": 1416.6346435546875,
700
+ "step": 1050,
701
+ "wo_beta": 15.470003128051758
702
+ },
703
+ {
704
+ "epoch": 2.975909305621162,
705
+ "eval_dpo_loss": 2638.496337890625,
706
+ "eval_logits": -0.46873465180397034,
707
+ "eval_logps": -83.80735778808594,
708
+ "eval_loss": 5215.619140625,
709
+ "eval_objective": 5232.5947265625,
710
+ "eval_ranking_idealized": 0.5212215185165405,
711
+ "eval_ranking_idealized_expo": 0.5212215185165405,
712
+ "eval_ranking_simple": 0.5295031070709229,
713
+ "eval_regularize": 5232.5947265625,
714
+ "eval_runtime": 308.5567,
715
+ "eval_samples_per_second": 18.765,
716
+ "eval_steps_per_second": 1.565,
717
+ "eval_wo_beta": 14.238861083984375,
718
+ "step": 1050
719
+ },
720
+ {
721
+ "dpo_loss": 630.0592651367188,
722
+ "epoch": 3.122342938119981,
723
+ "grad_norm": 146194.44633521218,
724
+ "learning_rate": 1.852952387243698e-06,
725
+ "logits": -0.47443822026252747,
726
+ "logps": -80.19267272949219,
727
+ "loss": 1247.6562,
728
+ "objective": 1267.6502685546875,
729
+ "ranking_idealized": 0.528333306312561,
730
+ "ranking_idealized_expo": 0.528333306312561,
731
+ "ranking_simple": 0.5299999713897705,
732
+ "regularize": 1267.6502685546875,
733
+ "step": 1100,
734
+ "wo_beta": 15.557552337646484
735
+ },
736
+ {
737
+ "epoch": 3.122342938119981,
738
+ "eval_dpo_loss": 2631.76611328125,
739
+ "eval_logits": -0.5460559129714966,
740
+ "eval_logps": -84.16140747070312,
741
+ "eval_loss": 5184.86962890625,
742
+ "eval_objective": 5190.6357421875,
743
+ "eval_ranking_idealized": 0.5212215185165405,
744
+ "eval_ranking_idealized_expo": 0.5212215185165405,
745
+ "eval_ranking_simple": 0.5263975262641907,
746
+ "eval_regularize": 5190.6357421875,
747
+ "eval_runtime": 310.23,
748
+ "eval_samples_per_second": 18.664,
749
+ "eval_steps_per_second": 1.557,
750
+ "eval_wo_beta": 14.152945518493652,
751
+ "step": 1100
752
+ },
753
+ {
754
+ "dpo_loss": 565.9061889648438,
755
+ "epoch": 3.264052905054322,
756
+ "grad_norm": 138842.10059998793,
757
+ "learning_rate": 1.617055052228768e-06,
758
+ "logits": -0.4821857213973999,
759
+ "logps": -80.27399444580078,
760
+ "loss": 1136.2859,
761
+ "objective": 1102.266845703125,
762
+ "ranking_idealized": 0.5083333253860474,
763
+ "ranking_idealized_expo": 0.5083333253860474,
764
+ "ranking_simple": 0.5137500166893005,
765
+ "regularize": 1102.266845703125,
766
+ "step": 1150,
767
+ "wo_beta": 15.317670822143555
768
+ },
769
+ {
770
+ "epoch": 3.264052905054322,
771
+ "eval_dpo_loss": 2590.183837890625,
772
+ "eval_logits": -0.5632494688034058,
773
+ "eval_logps": -83.88517761230469,
774
+ "eval_loss": 5110.20556640625,
775
+ "eval_objective": 5112.02783203125,
776
+ "eval_ranking_idealized": 0.5212215185165405,
777
+ "eval_ranking_idealized_expo": 0.5212215185165405,
778
+ "eval_ranking_simple": 0.5279502868652344,
779
+ "eval_regularize": 5112.02783203125,
780
+ "eval_runtime": 308.5625,
781
+ "eval_samples_per_second": 18.764,
782
+ "eval_steps_per_second": 1.565,
783
+ "eval_wo_beta": 14.093263626098633,
784
+ "step": 1150
785
+ },
786
+ {
787
+ "dpo_loss": 526.139404296875,
788
+ "epoch": 3.4057628719886632,
789
+ "grad_norm": 142396.22516348853,
790
+ "learning_rate": 1.3898334684855647e-06,
791
+ "logits": -0.49703070521354675,
792
+ "logps": -81.49961853027344,
793
+ "loss": 1042.7762,
794
+ "objective": 1025.806884765625,
795
+ "ranking_idealized": 0.5079166889190674,
796
+ "ranking_idealized_expo": 0.5079166889190674,
797
+ "ranking_simple": 0.5054166913032532,
798
+ "regularize": 1025.806884765625,
799
+ "step": 1200,
800
+ "wo_beta": 15.595675468444824
801
+ },
802
+ {
803
+ "epoch": 3.4057628719886632,
804
+ "eval_dpo_loss": 2612.26611328125,
805
+ "eval_logits": -0.5505208969116211,
806
+ "eval_logps": -83.96298217773438,
807
+ "eval_loss": 5146.40771484375,
808
+ "eval_objective": 5162.16650390625,
809
+ "eval_ranking_idealized": 0.5212215185165405,
810
+ "eval_ranking_idealized_expo": 0.5212215185165405,
811
+ "eval_ranking_simple": 0.5274327397346497,
812
+ "eval_regularize": 5162.16650390625,
813
+ "eval_runtime": 313.9682,
814
+ "eval_samples_per_second": 18.441,
815
+ "eval_steps_per_second": 1.538,
816
+ "eval_wo_beta": 14.112215042114258,
817
+ "step": 1200
818
+ },
819
+ {
820
+ "dpo_loss": 476.4833984375,
821
+ "epoch": 3.5474728389230044,
822
+ "grad_norm": 138317.9821042345,
823
+ "learning_rate": 1.1735202983664803e-06,
824
+ "logits": -0.45970824360847473,
825
+ "logps": -81.46965789794922,
826
+ "loss": 978.7787,
827
+ "objective": 968.6390380859375,
828
+ "ranking_idealized": 0.5141666531562805,
829
+ "ranking_idealized_expo": 0.5141666531562805,
830
+ "ranking_simple": 0.5137500166893005,
831
+ "regularize": 968.6390380859375,
832
+ "step": 1250,
833
+ "wo_beta": 15.359482765197754
834
+ },
835
+ {
836
+ "epoch": 3.5474728389230044,
837
+ "eval_dpo_loss": 2605.342041015625,
838
+ "eval_logits": -0.4993049204349518,
839
+ "eval_logps": -83.89867401123047,
840
+ "eval_loss": 5115.50927734375,
841
+ "eval_objective": 5140.92578125,
842
+ "eval_ranking_idealized": 0.5212215185165405,
843
+ "eval_ranking_idealized_expo": 0.5212215185165405,
844
+ "eval_ranking_simple": 0.5279502868652344,
845
+ "eval_regularize": 5140.92578125,
846
+ "eval_runtime": 308.6628,
847
+ "eval_samples_per_second": 18.758,
848
+ "eval_steps_per_second": 1.565,
849
+ "eval_wo_beta": 14.127862930297852,
850
+ "step": 1250
851
+ },
852
+ {
853
+ "dpo_loss": 455.1197814941406,
854
+ "epoch": 3.689182805857345,
855
+ "grad_norm": 136461.83557945368,
856
+ "learning_rate": 9.702410189643838e-07,
857
+ "logits": -0.4638132154941559,
858
+ "logps": -80.26090240478516,
859
+ "loss": 864.8715,
860
+ "objective": 869.080810546875,
861
+ "ranking_idealized": 0.49958333373069763,
862
+ "ranking_idealized_expo": 0.49958333373069763,
863
+ "ranking_simple": 0.4983333349227905,
864
+ "regularize": 869.080810546875,
865
+ "step": 1300,
866
+ "wo_beta": 15.24673843383789
867
+ },
868
+ {
869
+ "epoch": 3.689182805857345,
870
+ "eval_dpo_loss": 2621.07275390625,
871
+ "eval_logits": -0.5245055556297302,
872
+ "eval_logps": -84.29293060302734,
873
+ "eval_loss": 5143.4609375,
874
+ "eval_objective": 5173.7548828125,
875
+ "eval_ranking_idealized": 0.5212215185165405,
876
+ "eval_ranking_idealized_expo": 0.5212215185165405,
877
+ "eval_ranking_simple": 0.5258799195289612,
878
+ "eval_regularize": 5173.7548828125,
879
+ "eval_runtime": 308.7148,
880
+ "eval_samples_per_second": 18.755,
881
+ "eval_steps_per_second": 1.565,
882
+ "eval_wo_beta": 14.158381462097168,
883
+ "step": 1300
884
+ },
885
+ {
886
+ "epoch": 3.689182805857345,
887
+ "step": 1300,
888
+ "total_flos": 0.0,
889
+ "train_loss": 202.7064813701923,
890
+ "train_runtime": 6887.9795,
891
+ "train_samples_per_second": 36.877,
892
+ "train_steps_per_second": 0.256
893
+ }
894
+ ],
895
+ "logging_steps": 50,
896
+ "max_steps": 1760,
897
+ "num_input_tokens_seen": 0,
898
+ "num_train_epochs": 5,
899
+ "save_steps": 50,
900
+ "stateful_callbacks": {
901
+ "EarlyStoppingCallback": {
902
+ "args": {
903
+ "early_stopping_patience": 5,
904
+ "early_stopping_threshold": 0.0
905
+ },
906
+ "attributes": {
907
+ "early_stopping_patience_counter": 0
908
+ }
909
+ },
910
+ "TrainerControl": {
911
+ "args": {
912
+ "should_epoch_stop": false,
913
+ "should_evaluate": false,
914
+ "should_log": false,
915
+ "should_save": true,
916
+ "should_training_stop": true
917
+ },
918
+ "attributes": {}
919
+ }
920
+ },
921
+ "total_flos": 0.0,
922
+ "train_batch_size": 4,
923
+ "trial_name": null,
924
+ "trial_params": null
925
+ }