Hugging Face's logo

qingyangzhang
/
Qwen-2.5-7B-Simple-RL

Text Generation
Transformers
Safetensors
qwen2
Generated from Trainer
open-r1
trl
grpo
conversational
text-generation-inference
Inference Endpoints
Model card Files Community
qingyangzhang commited on
Commit
0821c78
·
verified ·
1 Parent(s): 0fc2cc8

Model save

Browse files
Files changed (5) hide show
  1. README.md +67 -0
  2. all_results.json +8 -0
  3. generation_config.json +6 -0
  4. train_results.json +8 -0
  5. trainer_state.json +386 -0
README.md ADDED
@@ -0,0 +1,67 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ library_name: transformers
3
+ model_name: Qwen-2.5-7B-Simple-RL
4
+ tags:
5
+ - generated_from_trainer
6
+ - trl
7
+ - grpo
8
+ licence: license
9
+ ---
10
+
11
+ # Model Card for Qwen-2.5-7B-Simple-RL
12
+
13
+ This model is a fine-tuned version of [None](https://huggingface.co/None).
14
+ It has been trained using [TRL](https://github.com/huggingface/trl).
15
+
16
+ ## Quick start
17
+
18
+ ```python
19
+ from transformers import pipeline
20
+
21
+ question = "If you had a time machine, but could only go to the past or the future once and never return, which would you choose and why?"
22
+ generator = pipeline("text-generation", model="qingyangzhang/Qwen-2.5-7B-Simple-RL", device="cuda")
23
+ output = generator([{"role": "user", "content": question}], max_new_tokens=128, return_full_text=False)[0]
24
+ print(output["generated_text"])
25
+ ```
26
+
27
+ ## Training procedure
28
+
29
+ [<img src="https://raw.githubusercontent.com/wandb/assets/main/wandb-github-badge-28.svg" alt="Visualize in Weights & Biases" width="150" height="24"/>](https://wandb.ai/zqyoung1127-tianjin-university/huggingface/runs/h8qxagbw)
30
+
31
+
32
+ This model was trained with GRPO, a method introduced in [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://huggingface.co/papers/2402.03300).
33
+
34
+ ### Framework versions
35
+
36
+ - TRL: 0.14.0
37
+ - Transformers: 4.48.3
38
+ - Pytorch: 2.5.1
39
+ - Datasets: 3.1.0
40
+ - Tokenizers: 0.21.0
41
+
42
+ ## Citations
43
+
44
+ Cite GRPO as:
45
+
46
+ ```bibtex
47
+ @article{zhihong2024deepseekmath,
48
+ title = {{DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models}},
49
+ author = {Zhihong Shao and Peiyi Wang and Qihao Zhu and Runxin Xu and Junxiao Song and Mingchuan Zhang and Y. K. Li and Y. Wu and Daya Guo},
50
+ year = 2024,
51
+ eprint = {arXiv:2402.03300},
52
+ }
53
+
54
+ ```
55
+
56
+ Cite TRL as:
57
+
58
+ ```bibtex
59
+ @misc{vonwerra2022trl,
60
+ title = {{TRL: Transformer Reinforcement Learning}},
61
+ author = {Leandro von Werra and Younes Belkada and Lewis Tunstall and Edward Beeching and Tristan Thrush and Nathan Lambert and Shengyi Huang and Kashif Rasul and Quentin Gallouédec},
62
+ year = 2020,
63
+ journal = {GitHub repository},
64
+ publisher = {GitHub},
65
+ howpublished = {\url{https://github.com/huggingface/trl}}
66
+ }
67
+ ```
all_results.json ADDED
@@ -0,0 +1,8 @@
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "total_flos": 0.0,
3
+ "train_loss": 0.00029843695958687075,
4
+ "train_runtime": 20680.6754,
5
+ "train_samples": 7500,
6
+ "train_samples_per_second": 0.363,
7
+ "train_steps_per_second": 0.006
8
+ }
generation_config.json ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ {
2
+ "bos_token_id": 151643,
3
+ "eos_token_id": 151643,
4
+ "max_new_tokens": 2048,
5
+ "transformers_version": "4.48.3"
6
+ }
train_results.json ADDED
@@ -0,0 +1,8 @@
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "total_flos": 0.0,
3
+ "train_loss": 0.00029843695958687075,
4
+ "train_runtime": 20680.6754,
5
+ "train_samples": 7500,
6
+ "train_samples_per_second": 0.363,
7
+ "train_steps_per_second": 0.006
8
+ }
trainer_state.json ADDED
@@ -0,0 +1,386 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "best_metric": null,
3
+ "best_model_checkpoint": null,
4
+ "epoch": 1.0,
5
+ "eval_steps": 100,
6
+ "global_step": 134,
7
+ "is_hyper_param_search": false,
8
+ "is_local_process_zero": true,
9
+ "is_world_process_zero": true,
10
+ "log_history": [
11
+ {
12
+ "completion_length": 594.4356994628906,
13
+ "epoch": 0.03731343283582089,
14
+ "grad_norm": 0.1971098631620407,
15
+ "kl": 0.00011806488037109375,
16
+ "learning_rate": 1.0714285714285716e-06,
17
+ "loss": 0.0,
18
+ "reward": 0.6617346815764904,
19
+ "reward_std": 0.3125086955726147,
20
+ "rewards/accuracy_reward": 0.6617346815764904,
21
+ "rewards/format_reward": 0.0,
22
+ "step": 5
23
+ },
24
+ {
25
+ "completion_length": 630.1530487060547,
26
+ "epoch": 0.07462686567164178,
27
+ "grad_norm": 0.18470077216625214,
28
+ "kl": 0.0004942655563354492,
29
+ "learning_rate": 2.142857142857143e-06,
30
+ "loss": 0.0,
31
+ "reward": 0.6520408019423485,
32
+ "reward_std": 0.2983996603637934,
33
+ "rewards/accuracy_reward": 0.651530598104,
34
+ "rewards/format_reward": 0.0005102040711790323,
35
+ "step": 10
36
+ },
37
+ {
38
+ "completion_length": 610.2959053039551,
39
+ "epoch": 0.11194029850746269,
40
+ "grad_norm": 0.052106305956840515,
41
+ "kl": 0.0017307758331298827,
42
+ "learning_rate": 2.999485987463336e-06,
43
+ "loss": 0.0001,
44
+ "reward": 0.7372448824346065,
45
+ "reward_std": 0.2600953433662653,
46
+ "rewards/accuracy_reward": 0.7372448824346065,
47
+ "rewards/format_reward": 0.0,
48
+ "step": 15
49
+ },
50
+ {
51
+ "completion_length": 611.4326400756836,
52
+ "epoch": 0.14925373134328357,
53
+ "grad_norm": 0.025398777797818184,
54
+ "kl": 0.0024486541748046874,
55
+ "learning_rate": 2.981532510892707e-06,
56
+ "loss": 0.0001,
57
+ "reward": 0.7372448846697808,
58
+ "reward_std": 0.2582925198599696,
59
+ "rewards/accuracy_reward": 0.7372448846697808,
60
+ "rewards/format_reward": 0.0,
61
+ "step": 20
62
+ },
63
+ {
64
+ "completion_length": 613.5106994628907,
65
+ "epoch": 0.1865671641791045,
66
+ "grad_norm": 0.012594266794621944,
67
+ "kl": 0.002337074279785156,
68
+ "learning_rate": 2.9382296023022897e-06,
69
+ "loss": 0.0001,
70
+ "reward": 0.735204067081213,
71
+ "reward_std": 0.20146227926015853,
72
+ "rewards/accuracy_reward": 0.735204067081213,
73
+ "rewards/format_reward": 0.0,
74
+ "step": 25
75
+ },
76
+ {
77
+ "completion_length": 565.3295829772949,
78
+ "epoch": 0.22388059701492538,
79
+ "grad_norm": 0.013191360048949718,
80
+ "kl": 0.0028142929077148438,
81
+ "learning_rate": 2.8703181864639013e-06,
82
+ "loss": 0.0001,
83
+ "reward": 0.7913265138864517,
84
+ "reward_std": 0.18436302840709687,
85
+ "rewards/accuracy_reward": 0.7913265138864517,
86
+ "rewards/format_reward": 0.0,
87
+ "step": 30
88
+ },
89
+ {
90
+ "completion_length": 572.9270294189453,
91
+ "epoch": 0.26119402985074625,
92
+ "grad_norm": 0.011080138385295868,
93
+ "kl": 0.0028873443603515624,
94
+ "learning_rate": 2.7789602465311384e-06,
95
+ "loss": 0.0001,
96
+ "reward": 0.7826530456542968,
97
+ "reward_std": 0.17215402722358703,
98
+ "rewards/accuracy_reward": 0.7826530456542968,
99
+ "rewards/format_reward": 0.0,
100
+ "step": 35
101
+ },
102
+ {
103
+ "completion_length": 585.7760055541992,
104
+ "epoch": 0.29850746268656714,
105
+ "grad_norm": 0.015958011150360107,
106
+ "kl": 0.003158760070800781,
107
+ "learning_rate": 2.6657189421854562e-06,
108
+ "loss": 0.0001,
109
+ "reward": 0.7872448831796646,
110
+ "reward_std": 0.1621189709752798,
111
+ "rewards/accuracy_reward": 0.7872448831796646,
112
+ "rewards/format_reward": 0.0,
113
+ "step": 40
114
+ },
115
+ {
116
+ "completion_length": 570.9520301818848,
117
+ "epoch": 0.3358208955223881,
118
+ "grad_norm": 0.02447451651096344,
119
+ "kl": 0.00324554443359375,
120
+ "learning_rate": 2.532531863540631e-06,
121
+ "loss": 0.0001,
122
+ "reward": 0.7755101852118969,
123
+ "reward_std": 0.16486475374549628,
124
+ "rewards/accuracy_reward": 0.7755101852118969,
125
+ "rewards/format_reward": 0.0,
126
+ "step": 45
127
+ },
128
+ {
129
+ "completion_length": 578.0943778991699,
130
+ "epoch": 0.373134328358209,
131
+ "grad_norm": 0.013620010577142239,
132
+ "kl": 0.003280830383300781,
133
+ "learning_rate": 2.3816778784387097e-06,
134
+ "loss": 0.0001,
135
+ "reward": 0.8040816187858582,
136
+ "reward_std": 0.1476132795214653,
137
+ "rewards/accuracy_reward": 0.8040816187858582,
138
+ "rewards/format_reward": 0.0,
139
+ "step": 50
140
+ },
141
+ {
142
+ "completion_length": 578.3994766235352,
143
+ "epoch": 0.41044776119402987,
144
+ "grad_norm": 0.016489438712596893,
145
+ "kl": 0.003299522399902344,
146
+ "learning_rate": 2.2157381403894125e-06,
147
+ "loss": 0.0001,
148
+ "reward": 0.7852040603756905,
149
+ "reward_std": 0.15987344700843095,
150
+ "rewards/accuracy_reward": 0.7852040603756905,
151
+ "rewards/format_reward": 0.0,
152
+ "step": 55
153
+ },
154
+ {
155
+ "completion_length": 570.6836616516114,
156
+ "epoch": 0.44776119402985076,
157
+ "grad_norm": 0.014856858178973198,
158
+ "kl": 0.003414154052734375,
159
+ "learning_rate": 2.03755192431795e-06,
160
+ "loss": 0.0001,
161
+ "reward": 0.7525510065257549,
162
+ "reward_std": 0.16357502806931734,
163
+ "rewards/accuracy_reward": 0.7525510065257549,
164
+ "rewards/format_reward": 0.0,
165
+ "step": 60
166
+ },
167
+ {
168
+ "completion_length": 557.9943771362305,
169
+ "epoch": 0.48507462686567165,
170
+ "grad_norm": 0.013422299176454544,
171
+ "kl": 0.00412750244140625,
172
+ "learning_rate": 1.8501680457838584e-06,
173
+ "loss": 0.0002,
174
+ "reward": 0.7938775330781936,
175
+ "reward_std": 0.15019273720681667,
176
+ "rewards/accuracy_reward": 0.7938775330781936,
177
+ "rewards/format_reward": 0.0,
178
+ "step": 65
179
+ },
180
+ {
181
+ "completion_length": 597.3453926086426,
182
+ "epoch": 0.5223880597014925,
183
+ "grad_norm": 0.014206411316990852,
184
+ "kl": 0.0033687591552734376,
185
+ "learning_rate": 1.6567926949014804e-06,
186
+ "loss": 0.0001,
187
+ "reward": 0.7632652923464776,
188
+ "reward_std": 0.17450573313981294,
189
+ "rewards/accuracy_reward": 0.7632652923464776,
190
+ "rewards/format_reward": 0.0,
191
+ "step": 70
192
+ },
193
+ {
194
+ "completion_length": 553.4872329711914,
195
+ "epoch": 0.5597014925373134,
196
+ "grad_norm": 0.01648498699069023,
197
+ "kl": 0.0038028717041015624,
198
+ "learning_rate": 1.4607345775381906e-06,
199
+ "loss": 0.0002,
200
+ "reward": 0.7770408011972905,
201
+ "reward_std": 0.158019458130002,
202
+ "rewards/accuracy_reward": 0.7770408011972905,
203
+ "rewards/format_reward": 0.0,
204
+ "step": 75
205
+ },
206
+ {
207
+ "completion_length": 561.41937789917,
208
+ "epoch": 0.5970149253731343,
209
+ "grad_norm": 0.014115195721387863,
210
+ "kl": 0.0034923553466796875,
211
+ "learning_rate": 1.2653483024396534e-06,
212
+ "loss": 0.0001,
213
+ "reward": 0.7872448809444904,
214
+ "reward_std": 0.14782564975321294,
215
+ "rewards/accuracy_reward": 0.7872448809444904,
216
+ "rewards/format_reward": 0.0,
217
+ "step": 80
218
+ },
219
+ {
220
+ "completion_length": 531.1652923583985,
221
+ "epoch": 0.6343283582089553,
222
+ "grad_norm": 0.01905824802815914,
223
+ "kl": 0.004101181030273437,
224
+ "learning_rate": 1.073976982944116e-06,
225
+ "loss": 0.0002,
226
+ "reward": 0.7877550825476647,
227
+ "reward_std": 0.16267810724675655,
228
+ "rewards/accuracy_reward": 0.7877550825476647,
229
+ "rewards/format_reward": 0.0,
230
+ "step": 85
231
+ },
232
+ {
233
+ "completion_length": 545.3045799255372,
234
+ "epoch": 0.6716417910447762,
235
+ "grad_norm": 0.021177947521209717,
236
+ "kl": 0.004157257080078125,
237
+ "learning_rate": 8.898950353863e-07,
238
+ "loss": 0.0002,
239
+ "reward": 0.7816326349973679,
240
+ "reward_std": 0.1574961107224226,
241
+ "rewards/accuracy_reward": 0.7816326349973679,
242
+ "rewards/format_reward": 0.0,
243
+ "step": 90
244
+ },
245
+ {
246
+ "completion_length": 568.895905303955,
247
+ "epoch": 0.7089552238805971,
248
+ "grad_norm": 0.02039457857608795,
249
+ "kl": 0.0038570404052734376,
250
+ "learning_rate": 7.162521529260768e-07,
251
+ "loss": 0.0002,
252
+ "reward": 0.7908163070678711,
253
+ "reward_std": 0.17509947922080754,
254
+ "rewards/accuracy_reward": 0.7903061032295227,
255
+ "rewards/format_reward": 0.0005102040711790323,
256
+ "step": 95
257
+ },
258
+ {
259
+ "completion_length": 588.9494773864747,
260
+ "epoch": 0.746268656716418,
261
+ "grad_norm": 0.013423638418316841,
262
+ "kl": 0.003804779052734375,
263
+ "learning_rate": 5.560194134252441e-07,
264
+ "loss": 0.0002,
265
+ "reward": 0.7403061062097549,
266
+ "reward_std": 0.17368433568626643,
267
+ "rewards/accuracy_reward": 0.7403061062097549,
268
+ "rewards/format_reward": 0.0,
269
+ "step": 100
270
+ },
271
+ {
272
+ "completion_length": 560.371418762207,
273
+ "epoch": 0.7835820895522388,
274
+ "grad_norm": 0.0290207602083683,
275
+ "kl": 0.0039691925048828125,
276
+ "learning_rate": 4.1193844348156887e-07,
277
+ "loss": 0.0002,
278
+ "reward": 0.7642856985330582,
279
+ "reward_std": 0.18369458429515362,
280
+ "rewards/accuracy_reward": 0.7637754946947097,
281
+ "rewards/format_reward": 0.0005102040711790323,
282
+ "step": 105
283
+ },
284
+ {
285
+ "completion_length": 560.294888305664,
286
+ "epoch": 0.8208955223880597,
287
+ "grad_norm": 0.02158834971487522,
288
+ "kl": 0.004729461669921875,
289
+ "learning_rate": 2.86474508437579e-07,
290
+ "loss": 0.0002,
291
+ "reward": 0.7780612073838711,
292
+ "reward_std": 0.1638113146647811,
293
+ "rewards/accuracy_reward": 0.7724489636719227,
294
+ "rewards/format_reward": 0.005612244782969356,
295
+ "step": 110
296
+ },
297
+ {
298
+ "completion_length": 550.3765190124511,
299
+ "epoch": 0.8582089552238806,
300
+ "grad_norm": 0.1028282567858696,
301
+ "kl": 0.01652069091796875,
302
+ "learning_rate": 1.8177433100705209e-07,
303
+ "loss": 0.0007,
304
+ "reward": 1.107142834365368,
305
+ "reward_std": 0.38318478502333164,
306
+ "rewards/accuracy_reward": 0.7530612118542195,
307
+ "rewards/format_reward": 0.3540816267952323,
308
+ "step": 115
309
+ },
310
+ {
311
+ "completion_length": 556.8137657165528,
312
+ "epoch": 0.8955223880597015,
313
+ "grad_norm": 0.05659456178545952,
314
+ "kl": 0.0318084716796875,
315
+ "learning_rate": 9.962936025419756e-08,
316
+ "loss": 0.0013,
317
+ "reward": 1.6938775151968002,
318
+ "reward_std": 0.296724752895534,
319
+ "rewards/accuracy_reward": 0.7999999821186066,
320
+ "rewards/format_reward": 0.8938775390386582,
321
+ "step": 120
322
+ },
323
+ {
324
+ "completion_length": 569.5673370361328,
325
+ "epoch": 0.9328358208955224,
326
+ "grad_norm": 0.03255928307771683,
327
+ "kl": 0.0380767822265625,
328
+ "learning_rate": 4.144511940348516e-08,
329
+ "loss": 0.0015,
330
+ "reward": 1.7311224043369293,
331
+ "reward_std": 0.26802414935082197,
332
+ "rewards/accuracy_reward": 0.7811224296689033,
333
+ "rewards/format_reward": 0.9499999895691872,
334
+ "step": 125
335
+ },
336
+ {
337
+ "completion_length": 524.7413139343262,
338
+ "epoch": 0.9701492537313433,
339
+ "grad_norm": 0.22196683287620544,
340
+ "kl": 0.0370025634765625,
341
+ "learning_rate": 8.217156947590065e-09,
342
+ "loss": 0.0015,
343
+ "reward": 1.7397958785295486,
344
+ "reward_std": 0.26053950041532514,
345
+ "rewards/accuracy_reward": 0.7872448846697807,
346
+ "rewards/format_reward": 0.9525510028004647,
347
+ "step": 130
348
+ },
349
+ {
350
+ "completion_length": 524.1900415420532,
351
+ "epoch": 1.0,
352
+ "kl": 0.03165435791015625,
353
+ "reward": 1.7072703689336777,
354
+ "reward_std": 0.254364542895928,
355
+ "rewards/accuracy_reward": 0.7646683501079679,
356
+ "rewards/format_reward": 0.9426020290702581,
357
+ "step": 134,
358
+ "total_flos": 0.0,
359
+ "train_loss": 0.00029843695958687075,
360
+ "train_runtime": 20680.6754,
361
+ "train_samples_per_second": 0.363,
362
+ "train_steps_per_second": 0.006
363
+ }
364
+ ],
365
+ "logging_steps": 5,
366
+ "max_steps": 134,
367
+ "num_input_tokens_seen": 0,
368
+ "num_train_epochs": 1,
369
+ "save_steps": 10,
370
+ "stateful_callbacks": {
371
+ "TrainerControl": {
372
+ "args": {
373
+ "should_epoch_stop": false,
374
+ "should_evaluate": false,
375
+ "should_log": false,
376
+ "should_save": true,
377
+ "should_training_stop": true
378
+ },
379
+ "attributes": {}
380
+ }
381
+ },
382
+ "total_flos": 0.0,
383
+ "train_batch_size": 1,
384
+ "trial_name": null,
385
+ "trial_params": null
386
+ }