Hugging Face's logo

Spaces:
NewBreaker
/
chatglm-6b-int4
Runtime error

App Files Community
chatglm-6b-int4
File size: 29,463 Bytes 
23aa310
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
 
2023-04-21 16:52:08,224 INFO    StreamThr :24428 [internal.py:wandb_internal():86] W&B internal server running at pid: 24428, started at: 2023-04-21 16:52:08.224660
2023-04-21 16:52:08,226 DEBUG   HandlerThread:24428 [handler.py:handle_request():144] handle_request: status
2023-04-21 16:52:08,228 INFO    WriterThread:24428 [datastore.py:open_for_write():85] open: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_165208-onaovmt4\run-onaovmt4.wandb
2023-04-21 16:52:08,232 DEBUG   SenderThread:24428 [sender.py:send():375] send: header
2023-04-21 16:52:08,320 DEBUG   SenderThread:24428 [sender.py:send():375] send: run
2023-04-21 16:52:09,776 DEBUG   HandlerThread:24428 [handler.py:handle_request():144] handle_request: check_version
2023-04-21 16:52:09,776 INFO    SenderThread:24428 [dir_watcher.py:__init__():219] watching files in: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_165208-onaovmt4\files
2023-04-21 16:52:09,777 INFO    SenderThread:24428 [sender.py:_start_run_threads():1124] run started: onaovmt4 with start time 1682067128.226776
2023-04-21 16:52:09,777 DEBUG   SenderThread:24428 [sender.py:send_request():402] send_request: summary_record
2023-04-21 16:52:09,778 INFO    SenderThread:24428 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-04-21 16:52:09,778 DEBUG   SenderThread:24428 [sender.py:send_request():402] send_request: check_version
2023-04-21 16:52:10,793 INFO    Thread-16 :24428 [dir_watcher.py:_on_file_created():278] file/dir created: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_165208-onaovmt4\files\wandb-summary.json
2023-04-21 16:52:10,858 DEBUG   HandlerThread:24428 [handler.py:handle_request():144] handle_request: run_start
2023-04-21 16:52:10,916 DEBUG   HandlerThread:24428 [system_info.py:__init__():31] System info init
2023-04-21 16:52:10,916 DEBUG   HandlerThread:24428 [system_info.py:__init__():46] System info init done
2023-04-21 16:52:10,916 INFO    HandlerThread:24428 [system_monitor.py:start():181] Starting system monitor
2023-04-21 16:52:10,916 INFO    SystemMonitor:24428 [system_monitor.py:_start():145] Starting system asset monitoring threads
2023-04-21 16:52:10,916 INFO    HandlerThread:24428 [system_monitor.py:probe():201] Collecting system info
2023-04-21 16:52:10,924 INFO    SystemMonitor:24428 [interfaces.py:start():190] Started cpu monitoring
2023-04-21 16:52:10,925 INFO    SystemMonitor:24428 [interfaces.py:start():190] Started disk monitoring
2023-04-21 16:52:10,926 INFO    SystemMonitor:24428 [interfaces.py:start():190] Started gpu monitoring
2023-04-21 16:52:10,939 INFO    SystemMonitor:24428 [interfaces.py:start():190] Started memory monitoring
2023-04-21 16:52:10,957 INFO    SystemMonitor:24428 [interfaces.py:start():190] Started network monitoring
2023-04-21 16:52:10,981 ERROR   gpu       :24428 [interfaces.py:monitor():144] Failed to sample metric: Not Supported
2023-04-21 16:52:10,989 DEBUG   HandlerThread:24428 [system_info.py:probe():195] Probing system
2023-04-21 16:52:10,992 DEBUG   HandlerThread:24428 [system_info.py:_probe_git():180] Probing git
2023-04-21 16:52:11,095 DEBUG   HandlerThread:24428 [system_info.py:_probe_git():188] Probing git done
2023-04-21 16:52:11,095 DEBUG   HandlerThread:24428 [system_info.py:probe():240] Probing system done
2023-04-21 16:52:11,095 DEBUG   HandlerThread:24428 [system_monitor.py:probe():210] {'os': 'Windows-10-10.0.19041-SP0', 'python': '3.8.10', 'heartbeatAt': '2023-04-21T08:52:10.989425', 'startedAt': '2023-04-21T08:52:08.204662', 'docker': None, 'cuda': None, 'args': ('--do_train', '--train_file', '.\\datasets\\chat\\train.json', '--validation_file', '.\\datasets\\chat\\dev.json', '--prompt_column', 'content', '--response_column', 'summary', '--overwrite_cache', '--model_name_or_path', '..\\models\\chatglm-6b-int4', '--output_dir', 'output\\adgen-chatglm-6b-pt-128-2e-2', '--overwrite_output_dir', '--max_source_length', '64', '--max_target_length', '64', '--per_device_train_batch_size', '1', '--per_device_eval_batch_size', '1', '--gradient_accumulation_steps', '16', '--predict_with_generate', '--max_steps', '1000', '--logging_steps', '10', '--save_steps', '100', '--learning_rate', '2e-2', '--pre_seq_len', '128', '--quantization_bit', '4'), 'state': 'running', 'program': 'main.py', 'codePath': 'ptuning\\main.py', 'git': {'remote': 'https://github.com/THUDM/ChatGLM-6B', 'commit': '01e6313abf4122d789d6e68128856af52847b355'}, 'cpu_count': 6, 'cpu_count_logical': 12, 'cpu_freq': {'current': 2592.0, 'min': 0.0, 'max': 2592.0}, 'cpu_freq_per_core': [{'current': 2592.0, 'min': 0.0, 'max': 2592.0}], 'disk': {'total': 500.32030868530273, 'used': 236.01997756958008}, 'gpu': 'NVIDIA GeForce RTX 2060', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 2060', 'memory_total': 6442450944}], 'memory': {'total': 63.87089538574219}}
2023-04-21 16:52:11,095 INFO    HandlerThread:24428 [system_monitor.py:probe():211] Finished collecting system info
2023-04-21 16:52:11,095 INFO    HandlerThread:24428 [system_monitor.py:probe():214] Publishing system info
2023-04-21 16:52:11,095 DEBUG   HandlerThread:24428 [system_info.py:_save_pip():51] Saving list of pip packages installed into the current environment
2023-04-21 16:52:11,096 DEBUG   HandlerThread:24428 [system_info.py:_save_pip():67] Saving pip packages done
2023-04-21 16:52:11,099 INFO    HandlerThread:24428 [system_monitor.py:probe():216] Finished publishing system info
2023-04-21 16:52:11,143 DEBUG   SenderThread:24428 [sender.py:send():375] send: files
2023-04-21 16:52:11,143 INFO    SenderThread:24428 [sender.py:_save_file():1378] saving file wandb-metadata.json with policy now
2023-04-21 16:52:11,156 DEBUG   HandlerThread:24428 [handler.py:handle_request():144] handle_request: stop_status
2023-04-21 16:52:11,157 DEBUG   SenderThread:24428 [sender.py:send_request():402] send_request: stop_status
2023-04-21 16:52:11,577 DEBUG   SenderThread:24428 [sender.py:send():375] send: telemetry
2023-04-21 16:52:11,577 DEBUG   SenderThread:24428 [sender.py:send():375] send: config
2023-04-21 16:52:11,578 DEBUG   SenderThread:24428 [sender.py:send():375] send: metric
2023-04-21 16:52:11,578 DEBUG   SenderThread:24428 [sender.py:send():375] send: telemetry
2023-04-21 16:52:11,578 DEBUG   SenderThread:24428 [sender.py:send():375] send: metric
2023-04-21 16:52:11,579 WARNING SenderThread:24428 [sender.py:send_metric():1329] Seen metric with glob (shouldn't happen)
2023-04-21 16:52:11,800 INFO    Thread-16 :24428 [dir_watcher.py:_on_file_created():278] file/dir created: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_165208-onaovmt4\files\output.log
2023-04-21 16:52:11,800 INFO    Thread-16 :24428 [dir_watcher.py:_on_file_created():278] file/dir created: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_165208-onaovmt4\files\wandb-metadata.json
2023-04-21 16:52:11,800 INFO    Thread-16 :24428 [dir_watcher.py:_on_file_created():278] file/dir created: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_165208-onaovmt4\files\requirements.txt
2023-04-21 16:52:12,178 INFO    wandb-upload_0:24428 [upload_job.py:push():137] Uploaded file C:\Users\Lenovo\AppData\Local\Temp\tmpedo5nrgdwandb\1xtewqbc-wandb-metadata.json
2023-04-21 16:52:13,036 ERROR   gpu       :24428 [interfaces.py:monitor():144] Failed to sample metric: Not Supported
2023-04-21 16:52:13,600 DEBUG   HandlerThread:24428 [handler.py:handle_request():144] handle_request: status_report
2023-04-21 16:52:13,818 INFO    Thread-16 :24428 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_165208-onaovmt4\files\output.log
2023-04-21 16:52:15,078 ERROR   gpu       :24428 [interfaces.py:monitor():144] Failed to sample metric: Not Supported
2023-04-21 16:52:15,829 INFO    Thread-16 :24428 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_165208-onaovmt4\files\output.log
2023-04-21 16:52:17,130 ERROR   gpu       :24428 [interfaces.py:monitor():144] Failed to sample metric: Not Supported
2023-04-21 16:52:18,669 DEBUG   HandlerThread:24428 [handler.py:handle_request():144] handle_request: status_report
2023-04-21 16:52:19,182 ERROR   gpu       :24428 [interfaces.py:monitor():144] Failed to sample metric: Not Supported
2023-04-21 16:52:21,226 ERROR   gpu       :24428 [interfaces.py:monitor():144] Failed to sample metric: Not Supported
2023-04-21 16:52:23,265 ERROR   gpu       :24428 [interfaces.py:monitor():144] Failed to sample metric: Not Supported
2023-04-21 16:52:23,721 DEBUG   HandlerThread:24428 [handler.py:handle_request():144] handle_request: status_report
2023-04-21 16:52:25,323 ERROR   gpu       :24428 [interfaces.py:monitor():144] Failed to sample metric: Not Supported
2023-04-21 16:52:26,169 DEBUG   HandlerThread:24428 [handler.py:handle_request():144] handle_request: stop_status
2023-04-21 16:52:26,169 DEBUG   SenderThread:24428 [sender.py:send_request():402] send_request: stop_status
2023-04-21 16:52:27,371 ERROR   gpu       :24428 [interfaces.py:monitor():144] Failed to sample metric: Not Supported
2023-04-21 16:52:28,978 DEBUG   HandlerThread:24428 [handler.py:handle_request():144] handle_request: status_report
2023-04-21 16:52:29,406 ERROR   gpu       :24428 [interfaces.py:monitor():144] Failed to sample metric: Not Supported
2023-04-21 16:52:29,958 INFO    Thread-16 :24428 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_165208-onaovmt4\files\output.log
2023-04-21 16:52:31,451 ERROR   gpu       :24428 [interfaces.py:monitor():144] Failed to sample metric: Not Supported
2023-04-21 16:52:33,485 ERROR   gpu       :24428 [interfaces.py:monitor():144] Failed to sample metric: Not Supported
2023-04-21 16:52:34,631 DEBUG   HandlerThread:24428 [handler.py:handle_request():144] handle_request: status_report
2023-04-21 16:52:35,541 ERROR   gpu       :24428 [interfaces.py:monitor():144] Failed to sample metric: Not Supported
2023-04-21 16:52:36,010 INFO    Thread-16 :24428 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_165208-onaovmt4\files\output.log
2023-04-21 16:52:37,580 ERROR   gpu       :24428 [interfaces.py:monitor():144] Failed to sample metric: Not Supported
2023-04-21 16:52:39,617 ERROR   gpu       :24428 [interfaces.py:monitor():144] Failed to sample metric: Not Supported
2023-04-21 16:52:39,647 DEBUG   HandlerThread:24428 [handler.py:handle_request():144] handle_request: status_report
2023-04-21 16:52:41,072 INFO    Thread-16 :24428 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_165208-onaovmt4\files\config.yaml
2023-04-21 16:52:41,179 DEBUG   HandlerThread:24428 [handler.py:handle_request():144] handle_request: stop_status
2023-04-21 16:52:41,180 DEBUG   SenderThread:24428 [sender.py:send_request():402] send_request: stop_status
2023-04-21 16:52:41,735 ERROR   gpu       :24428 [interfaces.py:monitor():144] Failed to sample metric: Not Supported
2023-04-21 16:52:43,767 ERROR   gpu       :24428 [interfaces.py:monitor():144] Failed to sample metric: Not Supported
2023-04-21 16:52:44,091 INFO    Thread-16 :24428 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_165208-onaovmt4\files\output.log
2023-04-21 16:52:45,799 ERROR   gpu       :24428 [interfaces.py:monitor():144] Failed to sample metric: Not Supported
2023-04-21 16:52:46,152 DEBUG   HandlerThread:24428 [handler.py:handle_request():144] handle_request: status_report
2023-04-21 16:52:47,856 ERROR   gpu       :24428 [interfaces.py:monitor():144] Failed to sample metric: Not Supported
2023-04-21 16:52:49,876 ERROR   gpu       :24428 [interfaces.py:monitor():144] Failed to sample metric: Not Supported
2023-04-21 16:52:50,188 INFO    Thread-16 :24428 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_165208-onaovmt4\files\output.log
2023-04-21 16:52:51,402 DEBUG   HandlerThread:24428 [handler.py:handle_request():144] handle_request: status_report
2023-04-21 16:52:51,892 ERROR   gpu       :24428 [interfaces.py:monitor():144] Failed to sample metric: Not Supported
2023-04-21 16:52:53,916 ERROR   gpu       :24428 [interfaces.py:monitor():144] Failed to sample metric: Not Supported
2023-04-21 16:52:54,266 INFO    Thread-16 :24428 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_165208-onaovmt4\files\output.log
2023-04-21 16:52:55,929 ERROR   gpu       :24428 [interfaces.py:monitor():144] Failed to sample metric: Not Supported
2023-04-21 16:52:56,195 DEBUG   HandlerThread:24428 [handler.py:handle_request():144] handle_request: stop_status
2023-04-21 16:52:56,196 DEBUG   SenderThread:24428 [sender.py:send_request():402] send_request: stop_status
2023-04-21 16:52:56,454 DEBUG   HandlerThread:24428 [handler.py:handle_request():144] handle_request: status_report
2023-04-21 16:52:57,959 ERROR   gpu       :24428 [interfaces.py:monitor():144] Failed to sample metric: Not Supported
2023-04-21 16:52:59,979 ERROR   gpu       :24428 [interfaces.py:monitor():144] Failed to sample metric: Not Supported
2023-04-21 16:53:00,438 INFO    Thread-16 :24428 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_165208-onaovmt4\files\output.log
2023-04-21 16:53:01,207 DEBUG   SenderThread:24428 [sender.py:send():375] send: exit
2023-04-21 16:53:01,207 INFO    SenderThread:24428 [sender.py:send_exit():598] handling exit code: 255
2023-04-21 16:53:01,207 INFO    SenderThread:24428 [sender.py:send_exit():600] handling runtime: 50
2023-04-21 16:53:01,207 INFO    SenderThread:24428 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-04-21 16:53:01,208 INFO    SenderThread:24428 [sender.py:send_exit():606] send defer
2023-04-21 16:53:01,208 DEBUG   HandlerThread:24428 [handler.py:handle_request():144] handle_request: defer
2023-04-21 16:53:01,208 INFO    HandlerThread:24428 [handler.py:handle_request_defer():170] handle defer: 0
2023-04-21 16:53:01,208 DEBUG   SenderThread:24428 [sender.py:send_request():402] send_request: defer
2023-04-21 16:53:01,208 INFO    SenderThread:24428 [sender.py:send_request_defer():622] handle sender defer: 0
2023-04-21 16:53:01,209 INFO    SenderThread:24428 [sender.py:transition_state():626] send defer: 1
2023-04-21 16:53:01,209 DEBUG   HandlerThread:24428 [handler.py:handle_request():144] handle_request: defer
2023-04-21 16:53:01,209 INFO    HandlerThread:24428 [handler.py:handle_request_defer():170] handle defer: 1
2023-04-21 16:53:01,209 DEBUG   SenderThread:24428 [sender.py:send_request():402] send_request: defer
2023-04-21 16:53:01,209 INFO    SenderThread:24428 [sender.py:send_request_defer():622] handle sender defer: 1
2023-04-21 16:53:01,209 INFO    SenderThread:24428 [sender.py:transition_state():626] send defer: 2
2023-04-21 16:53:01,210 DEBUG   HandlerThread:24428 [handler.py:handle_request():144] handle_request: defer
2023-04-21 16:53:01,210 INFO    HandlerThread:24428 [handler.py:handle_request_defer():170] handle defer: 2
2023-04-21 16:53:01,210 INFO    HandlerThread:24428 [system_monitor.py:finish():190] Stopping system monitor
2023-04-21 16:53:01,210 INFO    HandlerThread:24428 [interfaces.py:finish():202] Joined cpu monitor
2023-04-21 16:53:01,210 DEBUG   SystemMonitor:24428 [system_monitor.py:_start():159] Starting system metrics aggregation loop
2023-04-21 16:53:01,225 INFO    HandlerThread:24428 [interfaces.py:finish():202] Joined disk monitor
2023-04-21 16:53:01,225 DEBUG   SystemMonitor:24428 [system_monitor.py:_start():166] Finished system metrics aggregation loop
2023-04-21 16:53:01,226 DEBUG   SystemMonitor:24428 [system_monitor.py:_start():170] Publishing last batch of metrics
2023-04-21 16:53:01,274 INFO    HandlerThread:24428 [interfaces.py:finish():202] Joined gpu monitor
2023-04-21 16:53:01,274 INFO    HandlerThread:24428 [interfaces.py:finish():202] Joined memory monitor
2023-04-21 16:53:01,275 INFO    HandlerThread:24428 [interfaces.py:finish():202] Joined network monitor
2023-04-21 16:53:01,276 DEBUG   SenderThread:24428 [sender.py:send_request():402] send_request: defer
2023-04-21 16:53:01,277 INFO    SenderThread:24428 [sender.py:send_request_defer():622] handle sender defer: 2
2023-04-21 16:53:01,277 INFO    SenderThread:24428 [sender.py:transition_state():626] send defer: 3
2023-04-21 16:53:01,277 DEBUG   SenderThread:24428 [sender.py:send():375] send: stats
2023-04-21 16:53:01,277 DEBUG   HandlerThread:24428 [handler.py:handle_request():144] handle_request: defer
2023-04-21 16:53:01,278 INFO    HandlerThread:24428 [handler.py:handle_request_defer():170] handle defer: 3
2023-04-21 16:53:01,278 DEBUG   SenderThread:24428 [sender.py:send_request():402] send_request: defer
2023-04-21 16:53:01,278 INFO    SenderThread:24428 [sender.py:send_request_defer():622] handle sender defer: 3
2023-04-21 16:53:01,278 INFO    SenderThread:24428 [sender.py:transition_state():626] send defer: 4
2023-04-21 16:53:01,278 DEBUG   HandlerThread:24428 [handler.py:handle_request():144] handle_request: defer
2023-04-21 16:53:01,278 INFO    HandlerThread:24428 [handler.py:handle_request_defer():170] handle defer: 4
2023-04-21 16:53:01,279 DEBUG   SenderThread:24428 [sender.py:send_request():402] send_request: defer
2023-04-21 16:53:01,279 INFO    SenderThread:24428 [sender.py:send_request_defer():622] handle sender defer: 4
2023-04-21 16:53:01,279 INFO    SenderThread:24428 [sender.py:transition_state():626] send defer: 5
2023-04-21 16:53:01,279 DEBUG   HandlerThread:24428 [handler.py:handle_request():144] handle_request: defer
2023-04-21 16:53:01,279 INFO    HandlerThread:24428 [handler.py:handle_request_defer():170] handle defer: 5
2023-04-21 16:53:01,279 DEBUG   SenderThread:24428 [sender.py:send():375] send: summary
2023-04-21 16:53:01,280 INFO    SenderThread:24428 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-04-21 16:53:01,280 DEBUG   SenderThread:24428 [sender.py:send_request():402] send_request: defer
2023-04-21 16:53:01,280 INFO    SenderThread:24428 [sender.py:send_request_defer():622] handle sender defer: 5
2023-04-21 16:53:01,280 INFO    SenderThread:24428 [sender.py:transition_state():626] send defer: 6
2023-04-21 16:53:01,280 DEBUG   HandlerThread:24428 [handler.py:handle_request():144] handle_request: defer
2023-04-21 16:53:01,281 INFO    HandlerThread:24428 [handler.py:handle_request_defer():170] handle defer: 6
2023-04-21 16:53:01,281 DEBUG   SenderThread:24428 [sender.py:send_request():402] send_request: defer
2023-04-21 16:53:01,281 INFO    SenderThread:24428 [sender.py:send_request_defer():622] handle sender defer: 6
2023-04-21 16:53:01,281 INFO    SenderThread:24428 [sender.py:transition_state():626] send defer: 7
2023-04-21 16:53:01,281 DEBUG   HandlerThread:24428 [handler.py:handle_request():144] handle_request: status_report
2023-04-21 16:53:01,281 DEBUG   HandlerThread:24428 [handler.py:handle_request():144] handle_request: defer
2023-04-21 16:53:01,281 INFO    HandlerThread:24428 [handler.py:handle_request_defer():170] handle defer: 7
2023-04-21 16:53:01,282 DEBUG   SenderThread:24428 [sender.py:send_request():402] send_request: defer
2023-04-21 16:53:01,282 INFO    SenderThread:24428 [sender.py:send_request_defer():622] handle sender defer: 7
2023-04-21 16:53:01,442 INFO    Thread-16 :24428 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_165208-onaovmt4\files\wandb-summary.json
2023-04-21 16:53:02,283 DEBUG   HandlerThread:24428 [handler.py:handle_request():144] handle_request: poll_exit
2023-04-21 16:53:02,395 INFO    SenderThread:24428 [sender.py:transition_state():626] send defer: 8
2023-04-21 16:53:02,395 DEBUG   SenderThread:24428 [sender.py:send_request():402] send_request: poll_exit
2023-04-21 16:53:02,395 DEBUG   HandlerThread:24428 [handler.py:handle_request():144] handle_request: defer
2023-04-21 16:53:02,395 INFO    HandlerThread:24428 [handler.py:handle_request_defer():170] handle defer: 8
2023-04-21 16:53:02,396 DEBUG   SenderThread:24428 [sender.py:send_request():402] send_request: defer
2023-04-21 16:53:02,396 INFO    SenderThread:24428 [sender.py:send_request_defer():622] handle sender defer: 8
2023-04-21 16:53:02,429 INFO    SenderThread:24428 [sender.py:transition_state():626] send defer: 9
2023-04-21 16:53:02,429 DEBUG   HandlerThread:24428 [handler.py:handle_request():144] handle_request: defer
2023-04-21 16:53:02,429 INFO    HandlerThread:24428 [handler.py:handle_request_defer():170] handle defer: 9
2023-04-21 16:53:02,429 DEBUG   SenderThread:24428 [sender.py:send():375] send: artifact
2023-04-21 16:53:02,456 INFO    Thread-16 :24428 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_165208-onaovmt4\files\output.log
2023-04-21 16:53:03,292 DEBUG   HandlerThread:24428 [handler.py:handle_request():144] handle_request: poll_exit
2023-04-21 16:53:04,709 INFO    wandb-upload_0:24428 [upload_job.py:push():95] Uploaded file C:\Users\Lenovo\AppData\Local\wandb\wandb\artifacts\staging\tmppu32xnxj
2023-04-21 16:53:04,789 INFO    wandb-upload_1:24428 [upload_job.py:push():95] Uploaded file C:\Users\Lenovo\AppData\Local\wandb\wandb\artifacts\staging\tmpztqfj66q
2023-04-21 16:53:06,544 INFO    SenderThread:24428 [sender.py:send_artifact():1474] sent artifact job-https___github.com_THUDM_ChatGLM-6B_ptuning_main.py - {'id': 'QXJ0aWZhY3Q6NDI5NzAxNTEw', 'digest': '1077e319ad39e537c1ccc8f9a5c233bc', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjYzNDcyMzc3', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6NDI5NjczOTYx', 'versionIndex': 3}}, 'version': 'latest'}
2023-04-21 16:53:06,544 DEBUG   SenderThread:24428 [sender.py:send_request():402] send_request: defer
2023-04-21 16:53:06,544 INFO    SenderThread:24428 [sender.py:send_request_defer():622] handle sender defer: 9
2023-04-21 16:53:06,544 INFO    SenderThread:24428 [dir_watcher.py:finish():365] shutting down directory watcher
2023-04-21 16:53:07,513 INFO    SenderThread:24428 [dir_watcher.py:finish():395] scan: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_165208-onaovmt4\files
2023-04-21 16:53:07,514 INFO    SenderThread:24428 [dir_watcher.py:finish():409] scan save: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_165208-onaovmt4\files\config.yaml config.yaml
2023-04-21 16:53:07,514 INFO    SenderThread:24428 [dir_watcher.py:finish():409] scan save: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_165208-onaovmt4\files\output.log output.log
2023-04-21 16:53:07,517 INFO    SenderThread:24428 [dir_watcher.py:finish():409] scan save: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_165208-onaovmt4\files\requirements.txt requirements.txt
2023-04-21 16:53:07,519 INFO    SenderThread:24428 [dir_watcher.py:finish():409] scan save: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_165208-onaovmt4\files\wandb-metadata.json wandb-metadata.json
2023-04-21 16:53:07,519 INFO    SenderThread:24428 [dir_watcher.py:finish():409] scan save: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_165208-onaovmt4\files\wandb-summary.json wandb-summary.json
2023-04-21 16:53:07,522 INFO    SenderThread:24428 [sender.py:transition_state():626] send defer: 10
2023-04-21 16:53:07,522 DEBUG   SenderThread:24428 [sender.py:send_request():402] send_request: poll_exit
2023-04-21 16:53:07,522 DEBUG   HandlerThread:24428 [handler.py:handle_request():144] handle_request: defer
2023-04-21 16:53:07,522 INFO    HandlerThread:24428 [handler.py:handle_request_defer():170] handle defer: 10
2023-04-21 16:53:07,523 DEBUG   HandlerThread:24428 [handler.py:handle_request():144] handle_request: status_report
2023-04-21 16:53:07,525 DEBUG   SenderThread:24428 [sender.py:send_request():402] send_request: defer
2023-04-21 16:53:07,525 INFO    SenderThread:24428 [sender.py:send_request_defer():622] handle sender defer: 10
2023-04-21 16:53:07,525 INFO    SenderThread:24428 [file_pusher.py:finish():168] shutting down file pusher
2023-04-21 16:53:08,278 INFO    wandb-upload_0:24428 [upload_job.py:push():137] Uploaded file E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_165208-onaovmt4\files\config.yaml
2023-04-21 16:53:08,332 DEBUG   HandlerThread:24428 [handler.py:handle_request():144] handle_request: keepalive
2023-04-21 16:53:08,566 INFO    wandb-upload_1:24428 [upload_job.py:push():137] Uploaded file E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_165208-onaovmt4\files\output.log
2023-04-21 16:53:08,756 INFO    wandb-upload_3:24428 [upload_job.py:push():137] Uploaded file E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_165208-onaovmt4\files\wandb-summary.json
2023-04-21 16:53:08,813 INFO    wandb-upload_2:24428 [upload_job.py:push():137] Uploaded file E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_165208-onaovmt4\files\requirements.txt
2023-04-21 16:53:09,028 INFO    Thread-15 :24428 [sender.py:transition_state():626] send defer: 11
2023-04-21 16:53:09,028 DEBUG   HandlerThread:24428 [handler.py:handle_request():144] handle_request: defer
2023-04-21 16:53:09,028 INFO    HandlerThread:24428 [handler.py:handle_request_defer():170] handle defer: 11
2023-04-21 16:53:09,028 DEBUG   SenderThread:24428 [sender.py:send_request():402] send_request: defer
2023-04-21 16:53:09,029 INFO    SenderThread:24428 [sender.py:send_request_defer():622] handle sender defer: 11
2023-04-21 16:53:09,029 INFO    SenderThread:24428 [file_pusher.py:join():173] waiting for file pusher
2023-04-21 16:53:09,029 INFO    SenderThread:24428 [sender.py:transition_state():626] send defer: 12
2023-04-21 16:53:09,029 DEBUG   HandlerThread:24428 [handler.py:handle_request():144] handle_request: defer
2023-04-21 16:53:09,029 INFO    HandlerThread:24428 [handler.py:handle_request_defer():170] handle defer: 12
2023-04-21 16:53:09,029 DEBUG   SenderThread:24428 [sender.py:send_request():402] send_request: defer
2023-04-21 16:53:09,029 INFO    SenderThread:24428 [sender.py:send_request_defer():622] handle sender defer: 12
2023-04-21 16:53:09,905 INFO    SenderThread:24428 [sender.py:transition_state():626] send defer: 13
2023-04-21 16:53:09,905 DEBUG   HandlerThread:24428 [handler.py:handle_request():144] handle_request: defer
2023-04-21 16:53:09,905 INFO    HandlerThread:24428 [handler.py:handle_request_defer():170] handle defer: 13
2023-04-21 16:53:09,905 DEBUG   SenderThread:24428 [sender.py:send_request():402] send_request: defer
2023-04-21 16:53:09,905 INFO    SenderThread:24428 [sender.py:send_request_defer():622] handle sender defer: 13
2023-04-21 16:53:09,905 INFO    SenderThread:24428 [sender.py:transition_state():626] send defer: 14
2023-04-21 16:53:09,906 DEBUG   HandlerThread:24428 [handler.py:handle_request():144] handle_request: defer
2023-04-21 16:53:09,906 INFO    HandlerThread:24428 [handler.py:handle_request_defer():170] handle defer: 14
2023-04-21 16:53:09,906 DEBUG   SenderThread:24428 [sender.py:send():375] send: final
2023-04-21 16:53:09,906 DEBUG   SenderThread:24428 [sender.py:send():375] send: footer
2023-04-21 16:53:09,906 DEBUG   SenderThread:24428 [sender.py:send_request():402] send_request: defer
2023-04-21 16:53:09,906 INFO    SenderThread:24428 [sender.py:send_request_defer():622] handle sender defer: 14
2023-04-21 16:53:09,907 DEBUG   HandlerThread:24428 [handler.py:handle_request():144] handle_request: poll_exit
2023-04-21 16:53:09,907 DEBUG   HandlerThread:24428 [handler.py:handle_request():144] handle_request: server_info
2023-04-21 16:53:09,907 DEBUG   HandlerThread:24428 [handler.py:handle_request():144] handle_request: get_summary
2023-04-21 16:53:09,907 DEBUG   SenderThread:24428 [sender.py:send_request():402] send_request: poll_exit
2023-04-21 16:53:09,907 DEBUG   HandlerThread:24428 [handler.py:handle_request():144] handle_request: sampled_history
2023-04-21 16:53:09,908 DEBUG   SenderThread:24428 [sender.py:send_request():402] send_request: server_info
2023-04-21 16:53:10,163 INFO    MainThread:24428 [wandb_run.py:_footer_history_summary_info():3476] rendering history
2023-04-21 16:53:10,163 INFO    MainThread:24428 [wandb_run.py:_footer_history_summary_info():3508] rendering summary
2023-04-21 16:53:10,163 INFO    MainThread:24428 [wandb_run.py:_footer_sync_info():3434] logging synced files
2023-04-21 16:53:10,164 DEBUG   HandlerThread:24428 [handler.py:handle_request():144] handle_request: shutdown
2023-04-21 16:53:10,165 INFO    HandlerThread:24428 [handler.py:finish():845] shutting down handler
2023-04-21 16:53:10,910 INFO    WriterThread:24428 [datastore.py:close():298] close: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_165208-onaovmt4\run-onaovmt4.wandb
2023-04-21 16:53:11,171 INFO    SenderThread:24428 [sender.py:finish():1550] shutting down sender
2023-04-21 16:53:11,171 INFO    SenderThread:24428 [file_pusher.py:finish():168] shutting down file pusher
2023-04-21 16:53:11,171 INFO    SenderThread:24428 [file_pusher.py:join():173] waiting for file pusher