Hugging Face's logo

fenguhao
/
zephyr-7b-dpo-full

Text Generation
Transformers
TensorBoard
Safetensors
mistral
alignment-handbook
Generated from Trainer
trl
dpo
conversational
text-generation-inference
Inference Endpoints
Model card Files Metrics Community
fenguhao commited on
Commit
b0434e9
·
verified ·
1 Parent(s): 26f651b

Model save

Browse files
Files changed (15) hide show
  1. README.md +14 -24
  2. all_results.json +16 -16
  3. eval_results.json +12 -12
  4. model-00001-of-00003.safetensors +1 -1
  5. model-00002-of-00003.safetensors +1 -1
  6. model-00003-of-00003.safetensors +1 -1
  7. runs/May29_05-14-12_temp1a22200000C/events.out.tfevents.1716959703.temp1a22200000C.268304.0 +3 -0
  8. runs/May29_05-22-02_temp1a22200000C/events.out.tfevents.1716960155.temp1a22200000C.269479.0 +3 -0
  9. runs/May29_05-29-50_temp1a22200000C/events.out.tfevents.1716960622.temp1a22200000C.271997.0 +3 -0
  10. runs/May29_05-33-59_temp1a22200000C/events.out.tfevents.1716960890.temp1a22200000C.273921.0 +3 -0
  11. runs/May29_08-08-25_temp1a22200000C/events.out.tfevents.1716970147.temp1a22200000C.287909.0 +3 -0
  12. runs/May29_08-08-25_temp1a22200000C/events.out.tfevents.1716988192.temp1a22200000C.287909.1 +3 -0
  13. train_results.json +4 -4
  14. trainer_state.json +0 -0
  15. training_args.bin +2 -2
README.md CHANGED
@@ -2,13 +2,9 @@
2
  license: apache-2.0
3
  base_model: alignment-handbook/zephyr-7b-sft-full
4
  tags:
5
- - alignment-handbook
6
- - generated_from_trainer
7
  - trl
8
  - dpo
9
  - generated_from_trainer
10
- datasets:
11
- - HuggingFaceH4/ultrafeedback_binarized
12
  model-index:
13
  - name: zephyr-7b-dpo-full
14
  results: []
@@ -19,17 +15,17 @@ should probably proofread and complete it, then remove this comment. -->
19
 
20
  # zephyr-7b-dpo-full
21
 
22
- This model is a fine-tuned version of [alignment-handbook/zephyr-7b-sft-full](https://huggingface.co/alignment-handbook/zephyr-7b-sft-full) on the HuggingFaceH4/ultrafeedback_binarized dataset.
23
  It achieves the following results on the evaluation set:
24
- - Loss: 0.5168
25
- - Rewards/chosen: 0.1142
26
- - Rewards/rejected: -0.9329
27
- - Rewards/accuracies: 0.7695
28
- - Rewards/margins: 1.0471
29
- - Logps/rejected: -272.7659
30
- - Logps/chosen: -262.3270
31
- - Logits/rejected: -2.5421
32
- - Logits/chosen: -2.5184
33
 
34
  ## Model description
35
 
@@ -53,10 +49,10 @@ The following hyperparameters were used during training:
53
  - eval_batch_size: 8
54
  - seed: 42
55
  - distributed_type: multi-GPU
56
- - num_devices: 8
57
- - gradient_accumulation_steps: 2
58
  - total_train_batch_size: 128
59
- - total_eval_batch_size: 64
60
  - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
61
  - lr_scheduler_type: cosine
62
  - lr_scheduler_warmup_ratio: 0.1
@@ -64,17 +60,11 @@ The following hyperparameters were used during training:
64
 
65
  ### Training results
66
 
67
- | Training Loss | Epoch | Step | Validation Loss | Rewards/chosen | Rewards/rejected | Rewards/accuracies | Rewards/margins | Logps/rejected | Logps/chosen | Logits/rejected | Logits/chosen |
68
- |:-------------:|:-----:|:----:|:---------------:|:--------------:|:----------------:|:------------------:|:---------------:|:--------------:|:------------:|:---------------:|:-------------:|
69
- | 0.5155 | 0.21 | 100 | 0.5466 | 0.2422 | -0.6585 | 0.7656 | 0.9007 | -270.0222 | -261.0468 | -2.6271 | -2.6117 |
70
- | 0.4988 | 0.42 | 200 | 0.5294 | 0.0326 | -0.9993 | 0.7617 | 1.0320 | -273.4304 | -263.1426 | -2.5977 | -2.5791 |
71
- | 0.5354 | 0.63 | 300 | 0.5235 | 0.5272 | -0.4961 | 0.7656 | 1.0232 | -268.3980 | -258.1974 | -2.5449 | -2.5231 |
72
- | 0.5043 | 0.84 | 400 | 0.5181 | 0.1494 | -0.9114 | 0.7734 | 1.0608 | -272.5513 | -261.9752 | -2.5355 | -2.5122 |
73
 
74
 
75
  ### Framework versions
76
 
77
  - Transformers 4.36.2
78
- - Pytorch 2.1.2
79
  - Datasets 2.14.6
80
  - Tokenizers 0.15.2
 
2
  license: apache-2.0
3
  base_model: alignment-handbook/zephyr-7b-sft-full
4
  tags:
 
 
5
  - trl
6
  - dpo
7
  - generated_from_trainer
 
 
8
  model-index:
9
  - name: zephyr-7b-dpo-full
10
  results: []
 
15
 
16
  # zephyr-7b-dpo-full
17
 
18
+ This model is a fine-tuned version of [alignment-handbook/zephyr-7b-sft-full](https://huggingface.co/alignment-handbook/zephyr-7b-sft-full) on the None dataset.
19
  It achieves the following results on the evaluation set:
20
+ - Loss: 0.6488
21
+ - Rewards/chosen: -0.1752
22
+ - Rewards/rejected: -0.8876
23
+ - Rewards/accuracies: 0.7778
24
+ - Rewards/margins: 0.7123
25
+ - Logps/rejected: -350.5372
26
+ - Logps/chosen: -301.6346
27
+ - Logits/rejected: 0.8510
28
+ - Logits/chosen: -0.0436
29
 
30
  ## Model description
31
 
 
49
  - eval_batch_size: 8
50
  - seed: 42
51
  - distributed_type: multi-GPU
52
+ - num_devices: 4
53
+ - gradient_accumulation_steps: 4
54
  - total_train_batch_size: 128
55
+ - total_eval_batch_size: 32
56
  - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
57
  - lr_scheduler_type: cosine
58
  - lr_scheduler_warmup_ratio: 0.1
 
60
 
61
  ### Training results
62
 
 
 
 
 
 
 
63
 
64
 
65
  ### Framework versions
66
 
67
  - Transformers 4.36.2
68
+ - Pytorch 2.1.2+cu121
69
  - Datasets 2.14.6
70
  - Tokenizers 0.15.2
all_results.json CHANGED
@@ -1,21 +1,21 @@
1
  {
2
  "epoch": 1.0,
3
- "eval_logits/chosen": -2.5183939933776855,
4
- "eval_logits/rejected": -2.542128086090088,
5
- "eval_logps/chosen": -262.3270263671875,
6
- "eval_logps/rejected": -272.76593017578125,
7
- "eval_loss": 0.5167638063430786,
8
- "eval_rewards/accuracies": 0.76953125,
9
- "eval_rewards/chosen": 0.11419209837913513,
10
- "eval_rewards/margins": 1.047075629234314,
11
- "eval_rewards/rejected": -0.932883620262146,
12
- "eval_runtime": 110.44,
13
  "eval_samples": 2000,
14
- "eval_samples_per_second": 18.109,
15
- "eval_steps_per_second": 0.29,
16
- "train_loss": 0.5342033233602675,
17
- "train_runtime": 8549.6251,
18
  "train_samples": 61135,
19
- "train_samples_per_second": 7.151,
20
- "train_steps_per_second": 0.056
21
  }
 
1
  {
2
  "epoch": 1.0,
3
+ "eval_logits/chosen": -0.043629273772239685,
4
+ "eval_logits/rejected": 0.8509737253189087,
5
+ "eval_logps/chosen": -301.6346435546875,
6
+ "eval_logps/rejected": -350.5372314453125,
7
+ "eval_loss": 0.6487784385681152,
8
+ "eval_rewards/accuracies": 0.7777777910232544,
9
+ "eval_rewards/chosen": -0.17524881660938263,
10
+ "eval_rewards/margins": 0.7123382687568665,
11
+ "eval_rewards/rejected": -0.8875871300697327,
12
+ "eval_runtime": 237.6574,
13
  "eval_samples": 2000,
14
+ "eval_samples_per_second": 8.415,
15
+ "eval_steps_per_second": 0.265,
16
+ "train_loss": 0.6847315603082285,
17
+ "train_runtime": 17806.8146,
18
  "train_samples": 61135,
19
+ "train_samples_per_second": 3.433,
20
+ "train_steps_per_second": 0.027
21
  }
eval_results.json CHANGED
@@ -1,16 +1,16 @@
1
  {
2
  "epoch": 1.0,
3
- "eval_logits/chosen": -2.5183939933776855,
4
- "eval_logits/rejected": -2.542128086090088,
5
- "eval_logps/chosen": -262.3270263671875,
6
- "eval_logps/rejected": -272.76593017578125,
7
- "eval_loss": 0.5167638063430786,
8
- "eval_rewards/accuracies": 0.76953125,
9
- "eval_rewards/chosen": 0.11419209837913513,
10
- "eval_rewards/margins": 1.047075629234314,
11
- "eval_rewards/rejected": -0.932883620262146,
12
- "eval_runtime": 110.44,
13
  "eval_samples": 2000,
14
- "eval_samples_per_second": 18.109,
15
- "eval_steps_per_second": 0.29
16
  }
 
1
  {
2
  "epoch": 1.0,
3
+ "eval_logits/chosen": -0.043629273772239685,
4
+ "eval_logits/rejected": 0.8509737253189087,
5
+ "eval_logps/chosen": -301.6346435546875,
6
+ "eval_logps/rejected": -350.5372314453125,
7
+ "eval_loss": 0.6487784385681152,
8
+ "eval_rewards/accuracies": 0.7777777910232544,
9
+ "eval_rewards/chosen": -0.17524881660938263,
10
+ "eval_rewards/margins": 0.7123382687568665,
11
+ "eval_rewards/rejected": -0.8875871300697327,
12
+ "eval_runtime": 237.6574,
13
  "eval_samples": 2000,
14
+ "eval_samples_per_second": 8.415,
15
+ "eval_steps_per_second": 0.265
16
  }
model-00001-of-00003.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:bb781c5a1590b1b2cff3f9ae568198877a4545ed55fe160ed4826ae81c933ab8
3
  size 4943162336
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ad8d86de5f3f1e6967a35753473c27374145bbea35955d48d8602c4f04cd3623
3
  size 4943162336
model-00002-of-00003.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:6ec574e01e07b0d368cc7b0b4495c16999b1353f7da36e1c608480aea8d5989d
3
  size 4999819336
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:76db58a6dd2f78f5fde4981c0d3ab115bb1e7b449de56635b7bdb0e9eeb6cf3a
3
  size 4999819336
model-00003-of-00003.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:dcd449ad535ecf5d109e1e308f8930738c139f57eb492b202edb552d8990a6d9
3
  size 4540516344
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:5495a2c7078d56c15a3760772508f33e616b66fb9bb34fac5f1103b27f3b293a
3
  size 4540516344
runs/May29_05-14-12_temp1a22200000C/events.out.tfevents.1716959703.temp1a22200000C.268304.0 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7697a9c56e4f6f992917bf4c611d506596c3cf9f30b18a04be5a04924777f7d1
3
+ size 5151
runs/May29_05-22-02_temp1a22200000C/events.out.tfevents.1716960155.temp1a22200000C.269479.0 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:2029b4e854e7fdff512ff03b6327802d08ef4447a489e971e5b9e35484a48807
3
+ size 5774
runs/May29_05-29-50_temp1a22200000C/events.out.tfevents.1716960622.temp1a22200000C.271997.0 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:dd15d51e4cc80fa1f263ded02affcfdc3bef151f5317e21f7070c9bddb37dd66
3
+ size 5151
runs/May29_05-33-59_temp1a22200000C/events.out.tfevents.1716960890.temp1a22200000C.273921.0 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d762d75745ba8150d3245243ea8025550fe5db3999291ab4f65d879ae4f5fafb
3
+ size 82439
runs/May29_08-08-25_temp1a22200000C/events.out.tfevents.1716970147.temp1a22200000C.287909.0 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f0493a9484ccc6b4cf027bde74b9fc05dd53d2d8a59b135e7410411c4cb3ee63
3
+ size 155703
runs/May29_08-08-25_temp1a22200000C/events.out.tfevents.1716988192.temp1a22200000C.287909.1 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:cd81dde34afa68b46298d88122a1a6a5befc849452a952ae195451f4e1178ae9
3
+ size 828
train_results.json CHANGED
@@ -1,8 +1,8 @@
1
  {
2
  "epoch": 1.0,
3
- "train_loss": 0.5342033233602675,
4
- "train_runtime": 8549.6251,
5
  "train_samples": 61135,
6
- "train_samples_per_second": 7.151,
7
- "train_steps_per_second": 0.056
8
  }
 
1
  {
2
  "epoch": 1.0,
3
+ "train_loss": 0.6847315603082285,
4
+ "train_runtime": 17806.8146,
5
  "train_samples": 61135,
6
+ "train_samples_per_second": 3.433,
7
+ "train_steps_per_second": 0.027
8
  }
trainer_state.json CHANGED
The diff for this file is too large to render. See raw diff
 
training_args.bin CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:82cb00b2f20dd4226222faa5c46c0799aa8473e25919ccf7e6a0f57e5b530962
3
- size 6008
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:fbb28b6afd7530c6646f571fefbb34d4a66b7a9b801c985436aa3755dfb76e42
3
+ size 5944