Hugging Face's logo

sanchit-gandhi
/
distil-zephyr-1.5b-dpo-ultrafeedback

Text Generation
Transformers
TensorBoard
Safetensors
mistral
alignment-handbook
trl
dpo
Generated from Trainer
conversational
text-generation-inference
Inference Endpoints
Model card Files Metrics Community
distil-zephyr-1.5b-dpo-ultrafeedback
File size: 41,486 Bytes 
409a825
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
 
2024-04-26 15:57:15,698 INFO    StreamThr :713738 [internal.py:wandb_internal():86] W&B internal server running at pid: 713738, started at: 2024-04-26 15:57:15.697145
2024-04-26 15:57:15,700 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status
2024-04-26 15:57:15,705 INFO    WriterThread:713738 [datastore.py:open_for_write():87] open: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/run-vljs8xzo.wandb
2024-04-26 15:57:15,706 DEBUG   SenderThread:713738 [sender.py:send():379] send: header
2024-04-26 15:57:15,754 DEBUG   SenderThread:713738 [sender.py:send():379] send: run
2024-04-26 15:57:16,079 INFO    SenderThread:713738 [dir_watcher.py:__init__():211] watching files in: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files
2024-04-26 15:57:16,079 INFO    SenderThread:713738 [sender.py:_start_run_threads():1124] run started: vljs8xzo with start time 1714147035.700869
2024-04-26 15:57:16,085 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: check_version
2024-04-26 15:57:16,086 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: check_version
2024-04-26 15:57:16,171 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: run_start
2024-04-26 15:57:16,229 DEBUG   HandlerThread:713738 [system_info.py:__init__():26] System info init
2024-04-26 15:57:16,229 DEBUG   HandlerThread:713738 [system_info.py:__init__():41] System info init done
2024-04-26 15:57:16,229 INFO    HandlerThread:713738 [system_monitor.py:start():194] Starting system monitor
2024-04-26 15:57:16,229 INFO    SystemMonitor:713738 [system_monitor.py:_start():158] Starting system asset monitoring threads
2024-04-26 15:57:16,229 INFO    HandlerThread:713738 [system_monitor.py:probe():214] Collecting system info
2024-04-26 15:57:16,230 INFO    SystemMonitor:713738 [interfaces.py:start():190] Started cpu monitoring
2024-04-26 15:57:16,230 INFO    SystemMonitor:713738 [interfaces.py:start():190] Started disk monitoring
2024-04-26 15:57:16,230 INFO    SystemMonitor:713738 [interfaces.py:start():190] Started gpu monitoring
2024-04-26 15:57:16,231 INFO    SystemMonitor:713738 [interfaces.py:start():190] Started memory monitoring
2024-04-26 15:57:16,231 INFO    SystemMonitor:713738 [interfaces.py:start():190] Started network monitoring
2024-04-26 15:57:16,277 DEBUG   HandlerThread:713738 [system_info.py:probe():150] Probing system
2024-04-26 15:57:16,279 DEBUG   HandlerThread:713738 [system_info.py:_probe_git():135] Probing git
2024-04-26 15:57:16,299 DEBUG   HandlerThread:713738 [system_info.py:_probe_git():143] Probing git done
2024-04-26 15:57:16,300 DEBUG   HandlerThread:713738 [system_info.py:probe():198] Probing system done
2024-04-26 15:57:16,300 DEBUG   HandlerThread:713738 [system_monitor.py:probe():223] {'os': 'Linux-5.15.0-1048-aws-x86_64-with-glibc2.31', 'python': '3.11.9', 'heartbeatAt': '2024-04-26T15:57:16.277357', 'startedAt': '2024-04-26T15:57:15.677140', 'docker': None, 'cuda': None, 'args': ('./config_full.yaml',), 'state': 'running', 'program': '/fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/./run_dpo.py', 'codePathLocal': 'run_dpo.py', 'codePath': 'run_dpo.py', 'git': {'remote': 'https://huggingface.co/sanchit-gandhi/distil-zephyr-1.5b-dpo-ultrafeedback', 'commit': '087b22a9416f56b8862e6f5259874a73b30a9c0b'}, 'email': None, 'root': '/fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback', 'host': 'ip-26-0-160-225', 'username': 'sanchit', 'executable': '/fsx/sanchit/miniconda3/envs/alignment/bin/python', 'cpu_count': 96, 'cpu_count_logical': 96, 'cpu_freq': {'current': 2728.6554687499997, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 3569.387, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 3597.689, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 3598.118, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 3597.248, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 3597.735, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 3598.023, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 3594.821, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 3597.904, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}], 'disk': {'/': {'total': 290.7472343444824, 'used': 59.15196990966797}}, 'gpu': 'NVIDIA H100 80GB HBM3', 'gpu_count': 8, 'gpu_devices': [{'name': 'NVIDIA H100 80GB HBM3', 'memory_total': 85520809984}, {'name': 'NVIDIA H100 80GB HBM3', 'memory_total': 85520809984}, {'name': 'NVIDIA H100 80GB HBM3', 'memory_total': 85520809984}, {'name': 'NVIDIA H100 80GB HBM3', 'memory_total': 85520809984}, {'name': 'NVIDIA H100 80GB HBM3', 'memory_total': 85520809984}, {'name': 'NVIDIA H100 80GB HBM3', 'memory_total': 85520809984}, {'name': 'NVIDIA H100 80GB HBM3', 'memory_total': 85520809984}, {'name': 'NVIDIA H100 80GB HBM3', 'memory_total': 85520809984}], 'memory': {'total': 1999.985538482666}}
2024-04-26 15:57:16,300 INFO    HandlerThread:713738 [system_monitor.py:probe():224] Finished collecting system info
2024-04-26 15:57:16,300 INFO    HandlerThread:713738 [system_monitor.py:probe():227] Publishing system info
2024-04-26 15:57:16,300 DEBUG   HandlerThread:713738 [system_info.py:_save_conda():207] Saving list of conda packages installed into the current environment
2024-04-26 15:57:17,081 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_created():271] file/dir created: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/conda-environment.yaml
2024-04-26 15:57:20,083 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/conda-environment.yaml
2024-04-26 15:57:20,172 DEBUG   HandlerThread:713738 [system_info.py:_save_conda():222] Saving conda packages done
2024-04-26 15:57:20,174 INFO    HandlerThread:713738 [system_monitor.py:probe():229] Finished publishing system info
2024-04-26 15:57:20,199 DEBUG   SenderThread:713738 [sender.py:send():379] send: files
2024-04-26 15:57:20,199 INFO    SenderThread:713738 [sender.py:_save_file():1390] saving file wandb-metadata.json with policy now
2024-04-26 15:57:20,348 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: python_packages
2024-04-26 15:57:20,349 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: python_packages
2024-04-26 15:57:20,349 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
2024-04-26 15:57:20,351 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
2024-04-26 15:57:20,352 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-26 15:57:20,453 DEBUG   SenderThread:713738 [sender.py:send():379] send: telemetry
2024-04-26 15:57:20,453 DEBUG   SenderThread:713738 [sender.py:send():379] send: config
2024-04-26 15:57:20,454 DEBUG   SenderThread:713738 [sender.py:send():379] send: metric
2024-04-26 15:57:20,454 DEBUG   SenderThread:713738 [sender.py:send():379] send: telemetry
2024-04-26 15:57:20,454 DEBUG   SenderThread:713738 [sender.py:send():379] send: metric
2024-04-26 15:57:20,454 WARNING SenderThread:713738 [sender.py:send_metric():1341] Seen metric with glob (shouldn't happen)
2024-04-26 15:57:20,454 DEBUG   SenderThread:713738 [sender.py:send():379] send: telemetry
2024-04-26 15:57:20,491 INFO    wandb-upload_0:713738 [upload_job.py:push():131] Uploaded file /tmp/tmptr8aibf2wandb/74m4dddf-wandb-metadata.json
2024-04-26 15:57:21,085 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_created():271] file/dir created: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:57:21,085 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_created():271] file/dir created: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/requirements.txt
2024-04-26 15:57:21,086 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_created():271] file/dir created: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/wandb-metadata.json
2024-04-26 15:57:21,456 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 15:57:22,512 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: partial_history
2024-04-26 15:57:22,515 DEBUG   SenderThread:713738 [sender.py:send():379] send: metric
2024-04-26 15:57:22,516 DEBUG   SenderThread:713738 [sender.py:send():379] send: metric
2024-04-26 15:57:22,516 DEBUG   SenderThread:713738 [sender.py:send():379] send: metric
2024-04-26 15:57:22,516 DEBUG   SenderThread:713738 [sender.py:send():379] send: metric
2024-04-26 15:57:22,516 DEBUG   SenderThread:713738 [sender.py:send():379] send: metric
2024-04-26 15:57:22,516 DEBUG   SenderThread:713738 [sender.py:send():379] send: metric
2024-04-26 15:57:22,516 DEBUG   SenderThread:713738 [sender.py:send():379] send: metric
2024-04-26 15:57:22,516 DEBUG   SenderThread:713738 [sender.py:send():379] send: metric
2024-04-26 15:57:22,517 DEBUG   SenderThread:713738 [sender.py:send():379] send: metric
2024-04-26 15:57:22,517 DEBUG   SenderThread:713738 [sender.py:send():379] send: metric
2024-04-26 15:57:22,517 DEBUG   SenderThread:713738 [sender.py:send():379] send: metric
2024-04-26 15:57:22,517 DEBUG   SenderThread:713738 [sender.py:send():379] send: metric
2024-04-26 15:57:22,517 DEBUG   SenderThread:713738 [sender.py:send():379] send: history
2024-04-26 15:57:22,517 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: summary_record
2024-04-26 15:57:22,519 INFO    SenderThread:713738 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-26 15:57:23,088 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:57:23,089 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_created():271] file/dir created: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/wandb-summary.json
2024-04-26 15:57:25,091 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:57:27,096 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:57:27,303 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 15:57:29,100 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:57:31,102 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:57:32,406 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 15:57:33,103 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:57:35,106 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:57:35,349 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
2024-04-26 15:57:35,350 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
2024-04-26 15:57:35,351 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-26 15:57:37,110 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:57:37,525 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 15:57:39,114 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:57:41,117 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:57:42,595 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 15:57:43,121 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:57:45,126 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:57:47,130 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:57:47,734 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 15:57:48,135 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/config.yaml
2024-04-26 15:57:49,137 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:57:50,349 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
2024-04-26 15:57:50,350 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
2024-04-26 15:57:50,351 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-26 15:57:51,141 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:57:52,879 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 15:57:53,145 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:57:53,204 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: partial_history
2024-04-26 15:57:53,206 DEBUG   SenderThread:713738 [sender.py:send():379] send: history
2024-04-26 15:57:53,206 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: summary_record
2024-04-26 15:57:53,208 INFO    SenderThread:713738 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-26 15:57:54,151 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/wandb-summary.json
2024-04-26 15:57:55,153 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:57:57,155 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:57:58,052 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 15:57:59,157 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:58:01,161 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:58:03,163 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 15:58:03,166 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:58:05,169 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:58:05,349 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
2024-04-26 15:58:05,349 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
2024-04-26 15:58:05,352 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-26 15:58:07,172 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:58:08,289 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 15:58:09,174 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:58:11,176 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:58:13,178 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:58:13,439 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 15:58:15,180 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:58:16,231 DEBUG   SystemMonitor:713738 [system_monitor.py:_start():172] Starting system metrics aggregation loop
2024-04-26 15:58:16,235 DEBUG   SenderThread:713738 [sender.py:send():379] send: stats
2024-04-26 15:58:17,182 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:58:18,586 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 15:58:19,184 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:58:20,349 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
2024-04-26 15:58:20,349 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
2024-04-26 15:58:20,352 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-26 15:58:21,187 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:58:23,190 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:58:23,691 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 15:58:25,192 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:58:25,356 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: partial_history
2024-04-26 15:58:25,358 DEBUG   SenderThread:713738 [sender.py:send():379] send: history
2024-04-26 15:58:25,359 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: summary_record
2024-04-26 15:58:25,360 INFO    SenderThread:713738 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-26 15:58:26,195 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/wandb-summary.json
2024-04-26 15:58:27,197 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:58:28,906 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 15:58:29,201 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:58:31,205 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:58:33,210 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:58:34,031 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 15:58:35,214 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:58:35,349 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
2024-04-26 15:58:35,349 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
2024-04-26 15:58:35,352 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-26 15:58:37,219 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:58:39,148 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 15:58:39,223 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:58:41,228 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:58:43,232 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:58:44,292 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 15:58:45,237 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:58:46,237 DEBUG   SenderThread:713738 [sender.py:send():379] send: stats
2024-04-26 15:58:47,239 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:58:49,241 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:58:49,372 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 15:58:50,350 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
2024-04-26 15:58:50,350 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
2024-04-26 15:58:50,351 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-26 15:58:51,243 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:58:53,247 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:58:54,523 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 15:58:55,252 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:58:57,254 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:58:57,437 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: partial_history
2024-04-26 15:58:57,439 DEBUG   SenderThread:713738 [sender.py:send():379] send: history
2024-04-26 15:58:57,440 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: summary_record
2024-04-26 15:58:57,441 INFO    SenderThread:713738 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-26 15:58:58,256 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/wandb-summary.json
2024-04-26 15:58:59,257 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:58:59,663 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 15:59:01,259 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:59:03,261 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:59:04,824 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 15:59:05,263 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:59:05,349 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
2024-04-26 15:59:05,350 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
2024-04-26 15:59:05,351 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-26 15:59:07,265 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:59:09,267 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:59:09,968 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 15:59:11,269 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:59:13,271 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:59:15,260 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 15:59:15,273 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:59:16,238 DEBUG   SenderThread:713738 [sender.py:send():379] send: stats
2024-04-26 15:59:17,285 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:59:19,288 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:59:20,350 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
2024-04-26 15:59:20,350 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
2024-04-26 15:59:20,352 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-26 15:59:20,496 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 15:59:21,290 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:59:23,292 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:59:25,294 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:59:25,498 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 15:59:27,296 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:59:29,297 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:59:29,411 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: partial_history
2024-04-26 15:59:29,413 DEBUG   SenderThread:713738 [sender.py:send():379] send: history
2024-04-26 15:59:29,413 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: summary_record
2024-04-26 15:59:29,415 INFO    SenderThread:713738 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-26 15:59:30,299 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/wandb-summary.json
2024-04-26 15:59:30,708 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 15:59:31,300 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:59:33,302 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:59:35,304 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:59:35,350 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
2024-04-26 15:59:35,350 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
2024-04-26 15:59:35,352 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-26 15:59:35,807 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 15:59:37,306 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:59:37,698 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: partial_history
2024-04-26 15:59:37,700 DEBUG   SenderThread:713738 [sender.py:send():379] send: metric
2024-04-26 15:59:37,701 DEBUG   SenderThread:713738 [sender.py:send():379] send: metric
2024-04-26 15:59:37,701 DEBUG   SenderThread:713738 [sender.py:send():379] send: metric
2024-04-26 15:59:37,702 DEBUG   SenderThread:713738 [sender.py:send():379] send: metric
2024-04-26 15:59:37,702 DEBUG   SenderThread:713738 [sender.py:send():379] send: metric
2024-04-26 15:59:37,702 DEBUG   SenderThread:713738 [sender.py:send():379] send: metric
2024-04-26 15:59:37,702 DEBUG   SenderThread:713738 [sender.py:send():379] send: metric
2024-04-26 15:59:37,702 DEBUG   SenderThread:713738 [sender.py:send():379] send: metric
2024-04-26 15:59:37,702 DEBUG   SenderThread:713738 [sender.py:send():379] send: metric
2024-04-26 15:59:37,703 DEBUG   SenderThread:713738 [sender.py:send():379] send: metric
2024-04-26 15:59:37,703 DEBUG   SenderThread:713738 [sender.py:send():379] send: metric
2024-04-26 15:59:37,703 DEBUG   SenderThread:713738 [sender.py:send():379] send: metric
2024-04-26 15:59:37,703 DEBUG   SenderThread:713738 [sender.py:send():379] send: history
2024-04-26 15:59:37,703 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: summary_record
2024-04-26 15:59:37,705 INFO    SenderThread:713738 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-26 15:59:38,308 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/wandb-summary.json
2024-04-26 15:59:39,309 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:59:41,714 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 15:59:46,241 DEBUG   SenderThread:713738 [sender.py:send():379] send: stats
2024-04-26 15:59:47,242 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 15:59:49,320 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:59:51,474 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
2024-04-26 15:59:51,474 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
2024-04-26 15:59:51,734 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-26 15:59:52,625 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 15:59:53,325 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/config.yaml
2024-04-26 15:59:57,725 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:00:02,726 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:00:05,958 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-26 16:00:06,058 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
2024-04-26 16:00:06,059 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: stop_status