Spaces:
Runtime error
Runtime error
File size: 24,856 Bytes
23aa310 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 |
2023-04-22 15:52:55,534 INFO StreamThr :33684 [internal.py:wandb_internal():86] W&B internal server running at pid: 33684, started at: 2023-04-22 15:52:55.534078
2023-04-22 15:52:55,536 DEBUG HandlerThread:33684 [handler.py:handle_request():144] handle_request: status
2023-04-22 15:52:55,537 INFO WriterThread:33684 [datastore.py:open_for_write():85] open: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230422_155255-30hvteby\run-30hvteby.wandb
2023-04-22 15:52:55,543 DEBUG SenderThread:33684 [sender.py:send():375] send: header
2023-04-22 15:52:55,606 DEBUG SenderThread:33684 [sender.py:send():375] send: run
2023-04-22 15:52:56,299 DEBUG HandlerThread:33684 [handler.py:handle_request():144] handle_request: check_version
2023-04-22 15:52:56,301 INFO SenderThread:33684 [dir_watcher.py:__init__():219] watching files in: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230422_155255-30hvteby\files
2023-04-22 15:52:56,301 INFO SenderThread:33684 [sender.py:_start_run_threads():1124] run started: 30hvteby with start time 1682149975.535061
2023-04-22 15:52:56,301 DEBUG SenderThread:33684 [sender.py:send_request():402] send_request: summary_record
2023-04-22 15:52:56,302 INFO SenderThread:33684 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-04-22 15:52:56,302 DEBUG SenderThread:33684 [sender.py:send_request():402] send_request: check_version
2023-04-22 15:52:56,959 DEBUG HandlerThread:33684 [handler.py:handle_request():144] handle_request: run_start
2023-04-22 15:52:57,024 DEBUG HandlerThread:33684 [system_info.py:__init__():31] System info init
2023-04-22 15:52:57,024 DEBUG HandlerThread:33684 [system_info.py:__init__():46] System info init done
2023-04-22 15:52:57,024 INFO HandlerThread:33684 [system_monitor.py:start():181] Starting system monitor
2023-04-22 15:52:57,025 INFO SystemMonitor:33684 [system_monitor.py:_start():145] Starting system asset monitoring threads
2023-04-22 15:52:57,025 INFO HandlerThread:33684 [system_monitor.py:probe():201] Collecting system info
2023-04-22 15:52:57,032 INFO SystemMonitor:33684 [interfaces.py:start():190] Started cpu monitoring
2023-04-22 15:52:57,033 INFO SystemMonitor:33684 [interfaces.py:start():190] Started disk monitoring
2023-04-22 15:52:57,033 INFO SystemMonitor:33684 [interfaces.py:start():190] Started gpu monitoring
2023-04-22 15:52:57,045 INFO SystemMonitor:33684 [interfaces.py:start():190] Started memory monitoring
2023-04-22 15:52:57,079 INFO SystemMonitor:33684 [interfaces.py:start():190] Started network monitoring
2023-04-22 15:52:57,094 DEBUG HandlerThread:33684 [system_info.py:probe():195] Probing system
2023-04-22 15:52:57,097 DEBUG HandlerThread:33684 [system_info.py:_probe_git():180] Probing git
2023-04-22 15:52:57,110 ERROR gpu :33684 [interfaces.py:monitor():144] Failed to sample metric: Not Supported
2023-04-22 15:52:57,198 DEBUG HandlerThread:33684 [system_info.py:_probe_git():188] Probing git done
2023-04-22 15:52:57,198 DEBUG HandlerThread:33684 [system_info.py:probe():240] Probing system done
2023-04-22 15:52:57,198 DEBUG HandlerThread:33684 [system_monitor.py:probe():210] {'os': 'Windows-10-10.0.19041-SP0', 'python': '3.8.10', 'heartbeatAt': '2023-04-22T07:52:57.094978', 'startedAt': '2023-04-22T07:52:55.495061', 'docker': None, 'cuda': None, 'args': ('--do_train', '--train_file', '.\\datasets\\Zettels\\train.json', '--validation_file', '.\\datasets\\Zettels\\dev.json', '--prompt_column', 'content', '--response_column', 'summary', '--overwrite_cache', '--model_name_or_path', '..\\models\\chatglm-6b-int4', '--output_dir', 'output\\adgen-chatglm-6b-pt-128-2e-2', '--overwrite_output_dir', '--max_source_length', '64', '--max_target_length', '64', '--per_device_train_batch_size', '1', '--per_device_eval_batch_size', '1', '--gradient_accumulation_steps', '16', '--predict_with_generate', '--max_steps', '1000', '--logging_steps', '10', '--save_steps', '10', '--learning_rate', '2e-2', '--pre_seq_len', '128', '--quantization_bit', '4'), 'state': 'running', 'program': 'main.py', 'codePath': 'ptuning\\main.py', 'git': {'remote': 'https://github.com/THUDM/ChatGLM-6B', 'commit': '01e6313abf4122d789d6e68128856af52847b355'}, 'cpu_count': 6, 'cpu_count_logical': 12, 'cpu_freq': {'current': 2592.0, 'min': 0.0, 'max': 2592.0}, 'cpu_freq_per_core': [{'current': 2592.0, 'min': 0.0, 'max': 2592.0}], 'disk': {'total': 500.32030868530273, 'used': 239.2071533203125}, 'gpu': 'NVIDIA GeForce RTX 2060', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 2060', 'memory_total': 6442450944}], 'memory': {'total': 63.87089538574219}}
2023-04-22 15:52:57,199 INFO HandlerThread:33684 [system_monitor.py:probe():211] Finished collecting system info
2023-04-22 15:52:57,199 INFO HandlerThread:33684 [system_monitor.py:probe():214] Publishing system info
2023-04-22 15:52:57,199 DEBUG HandlerThread:33684 [system_info.py:_save_pip():51] Saving list of pip packages installed into the current environment
2023-04-22 15:52:57,200 DEBUG HandlerThread:33684 [system_info.py:_save_pip():67] Saving pip packages done
2023-04-22 15:52:57,201 INFO HandlerThread:33684 [system_monitor.py:probe():216] Finished publishing system info
2023-04-22 15:52:57,216 DEBUG SenderThread:33684 [sender.py:send():375] send: files
2023-04-22 15:52:57,216 INFO SenderThread:33684 [sender.py:_save_file():1378] saving file wandb-metadata.json with policy now
2023-04-22 15:52:57,231 DEBUG HandlerThread:33684 [handler.py:handle_request():144] handle_request: stop_status
2023-04-22 15:52:57,231 DEBUG SenderThread:33684 [sender.py:send_request():402] send_request: stop_status
2023-04-22 15:52:57,317 INFO Thread-16 :33684 [dir_watcher.py:_on_file_created():278] file/dir created: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230422_155255-30hvteby\files\requirements.txt
2023-04-22 15:52:57,317 INFO Thread-16 :33684 [dir_watcher.py:_on_file_created():278] file/dir created: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230422_155255-30hvteby\files\wandb-summary.json
2023-04-22 15:52:57,318 INFO Thread-16 :33684 [dir_watcher.py:_on_file_created():278] file/dir created: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230422_155255-30hvteby\files\wandb-metadata.json
2023-04-22 15:52:57,611 DEBUG SenderThread:33684 [sender.py:send():375] send: telemetry
2023-04-22 15:52:57,611 DEBUG SenderThread:33684 [sender.py:send():375] send: config
2023-04-22 15:52:57,613 DEBUG SenderThread:33684 [sender.py:send():375] send: metric
2023-04-22 15:52:57,613 DEBUG SenderThread:33684 [sender.py:send():375] send: telemetry
2023-04-22 15:52:57,613 DEBUG SenderThread:33684 [sender.py:send():375] send: metric
2023-04-22 15:52:57,613 WARNING SenderThread:33684 [sender.py:send_metric():1329] Seen metric with glob (shouldn't happen)
2023-04-22 15:52:58,319 INFO Thread-16 :33684 [dir_watcher.py:_on_file_created():278] file/dir created: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230422_155255-30hvteby\files\output.log
2023-04-22 15:52:58,483 INFO wandb-upload_0:33684 [upload_job.py:push():137] Uploaded file C:\Users\Lenovo\AppData\Local\Temp\tmpskkixiipwandb\wfodv4i9-wandb-metadata.json
2023-04-22 15:52:59,131 ERROR gpu :33684 [interfaces.py:monitor():144] Failed to sample metric: Not Supported
2023-04-22 15:53:00,331 INFO Thread-16 :33684 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230422_155255-30hvteby\files\output.log
2023-04-22 15:53:00,960 DEBUG HandlerThread:33684 [handler.py:handle_request():144] handle_request: status_report
2023-04-22 15:53:01,185 ERROR gpu :33684 [interfaces.py:monitor():144] Failed to sample metric: Not Supported
2023-04-22 15:53:02,355 INFO Thread-16 :33684 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230422_155255-30hvteby\files\output.log
2023-04-22 15:53:03,221 ERROR gpu :33684 [interfaces.py:monitor():144] Failed to sample metric: Not Supported
2023-04-22 15:53:05,272 ERROR gpu :33684 [interfaces.py:monitor():144] Failed to sample metric: Not Supported
2023-04-22 15:53:06,005 DEBUG HandlerThread:33684 [handler.py:handle_request():144] handle_request: status_report
2023-04-22 15:53:07,319 ERROR gpu :33684 [interfaces.py:monitor():144] Failed to sample metric: Not Supported
2023-04-22 15:53:09,369 ERROR gpu :33684 [interfaces.py:monitor():144] Failed to sample metric: Not Supported
2023-04-22 15:53:11,046 DEBUG HandlerThread:33684 [handler.py:handle_request():144] handle_request: status_report
2023-04-22 15:53:11,424 ERROR gpu :33684 [interfaces.py:monitor():144] Failed to sample metric: Not Supported
2023-04-22 15:53:12,250 DEBUG HandlerThread:33684 [handler.py:handle_request():144] handle_request: stop_status
2023-04-22 15:53:12,251 DEBUG SenderThread:33684 [sender.py:send_request():402] send_request: stop_status
2023-04-22 15:53:12,989 DEBUG SenderThread:33684 [sender.py:send():375] send: exit
2023-04-22 15:53:12,989 INFO SenderThread:33684 [sender.py:send_exit():598] handling exit code: 255
2023-04-22 15:53:12,989 INFO SenderThread:33684 [sender.py:send_exit():600] handling runtime: 16
2023-04-22 15:53:12,990 INFO SenderThread:33684 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-04-22 15:53:12,991 INFO SenderThread:33684 [sender.py:send_exit():606] send defer
2023-04-22 15:53:12,991 DEBUG HandlerThread:33684 [handler.py:handle_request():144] handle_request: defer
2023-04-22 15:53:12,991 INFO HandlerThread:33684 [handler.py:handle_request_defer():170] handle defer: 0
2023-04-22 15:53:12,992 DEBUG SenderThread:33684 [sender.py:send_request():402] send_request: defer
2023-04-22 15:53:12,992 INFO SenderThread:33684 [sender.py:send_request_defer():622] handle sender defer: 0
2023-04-22 15:53:12,992 INFO SenderThread:33684 [sender.py:transition_state():626] send defer: 1
2023-04-22 15:53:12,992 DEBUG HandlerThread:33684 [handler.py:handle_request():144] handle_request: defer
2023-04-22 15:53:12,992 INFO HandlerThread:33684 [handler.py:handle_request_defer():170] handle defer: 1
2023-04-22 15:53:12,992 DEBUG SenderThread:33684 [sender.py:send_request():402] send_request: defer
2023-04-22 15:53:12,992 INFO SenderThread:33684 [sender.py:send_request_defer():622] handle sender defer: 1
2023-04-22 15:53:12,992 INFO SenderThread:33684 [sender.py:transition_state():626] send defer: 2
2023-04-22 15:53:12,992 DEBUG HandlerThread:33684 [handler.py:handle_request():144] handle_request: defer
2023-04-22 15:53:12,993 INFO HandlerThread:33684 [handler.py:handle_request_defer():170] handle defer: 2
2023-04-22 15:53:12,993 INFO HandlerThread:33684 [system_monitor.py:finish():190] Stopping system monitor
2023-04-22 15:53:12,993 DEBUG SystemMonitor:33684 [system_monitor.py:_start():159] Starting system metrics aggregation loop
2023-04-22 15:53:12,993 INFO HandlerThread:33684 [interfaces.py:finish():202] Joined cpu monitor
2023-04-22 15:53:12,993 DEBUG SystemMonitor:33684 [system_monitor.py:_start():166] Finished system metrics aggregation loop
2023-04-22 15:53:12,993 INFO HandlerThread:33684 [interfaces.py:finish():202] Joined disk monitor
2023-04-22 15:53:12,993 DEBUG SystemMonitor:33684 [system_monitor.py:_start():170] Publishing last batch of metrics
2023-04-22 15:53:13,070 INFO HandlerThread:33684 [interfaces.py:finish():202] Joined gpu monitor
2023-04-22 15:53:13,070 INFO HandlerThread:33684 [interfaces.py:finish():202] Joined memory monitor
2023-04-22 15:53:13,070 INFO HandlerThread:33684 [interfaces.py:finish():202] Joined network monitor
2023-04-22 15:53:13,071 DEBUG SenderThread:33684 [sender.py:send_request():402] send_request: defer
2023-04-22 15:53:13,071 INFO SenderThread:33684 [sender.py:send_request_defer():622] handle sender defer: 2
2023-04-22 15:53:13,071 INFO SenderThread:33684 [sender.py:transition_state():626] send defer: 3
2023-04-22 15:53:13,071 DEBUG HandlerThread:33684 [handler.py:handle_request():144] handle_request: defer
2023-04-22 15:53:13,071 INFO HandlerThread:33684 [handler.py:handle_request_defer():170] handle defer: 3
2023-04-22 15:53:13,073 DEBUG SenderThread:33684 [sender.py:send():375] send: stats
2023-04-22 15:53:13,074 DEBUG SenderThread:33684 [sender.py:send_request():402] send_request: defer
2023-04-22 15:53:13,074 INFO SenderThread:33684 [sender.py:send_request_defer():622] handle sender defer: 3
2023-04-22 15:53:13,074 INFO SenderThread:33684 [sender.py:transition_state():626] send defer: 4
2023-04-22 15:53:13,074 DEBUG HandlerThread:33684 [handler.py:handle_request():144] handle_request: defer
2023-04-22 15:53:13,074 INFO HandlerThread:33684 [handler.py:handle_request_defer():170] handle defer: 4
2023-04-22 15:53:13,074 DEBUG SenderThread:33684 [sender.py:send_request():402] send_request: defer
2023-04-22 15:53:13,074 INFO SenderThread:33684 [sender.py:send_request_defer():622] handle sender defer: 4
2023-04-22 15:53:13,074 INFO SenderThread:33684 [sender.py:transition_state():626] send defer: 5
2023-04-22 15:53:13,074 DEBUG HandlerThread:33684 [handler.py:handle_request():144] handle_request: defer
2023-04-22 15:53:13,074 INFO HandlerThread:33684 [handler.py:handle_request_defer():170] handle defer: 5
2023-04-22 15:53:13,075 DEBUG SenderThread:33684 [sender.py:send():375] send: summary
2023-04-22 15:53:13,075 INFO SenderThread:33684 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-04-22 15:53:13,076 DEBUG SenderThread:33684 [sender.py:send_request():402] send_request: defer
2023-04-22 15:53:13,076 INFO SenderThread:33684 [sender.py:send_request_defer():622] handle sender defer: 5
2023-04-22 15:53:13,076 INFO SenderThread:33684 [sender.py:transition_state():626] send defer: 6
2023-04-22 15:53:13,076 DEBUG HandlerThread:33684 [handler.py:handle_request():144] handle_request: defer
2023-04-22 15:53:13,076 INFO HandlerThread:33684 [handler.py:handle_request_defer():170] handle defer: 6
2023-04-22 15:53:13,076 DEBUG SenderThread:33684 [sender.py:send_request():402] send_request: defer
2023-04-22 15:53:13,076 INFO SenderThread:33684 [sender.py:send_request_defer():622] handle sender defer: 6
2023-04-22 15:53:13,080 DEBUG HandlerThread:33684 [handler.py:handle_request():144] handle_request: status_report
2023-04-22 15:53:13,427 INFO Thread-16 :33684 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230422_155255-30hvteby\files\wandb-summary.json
2023-04-22 15:53:13,706 INFO SenderThread:33684 [sender.py:transition_state():626] send defer: 7
2023-04-22 15:53:13,706 DEBUG HandlerThread:33684 [handler.py:handle_request():144] handle_request: defer
2023-04-22 15:53:13,706 INFO HandlerThread:33684 [handler.py:handle_request_defer():170] handle defer: 7
2023-04-22 15:53:13,706 DEBUG SenderThread:33684 [sender.py:send_request():402] send_request: defer
2023-04-22 15:53:13,706 INFO SenderThread:33684 [sender.py:send_request_defer():622] handle sender defer: 7
2023-04-22 15:53:14,056 DEBUG HandlerThread:33684 [handler.py:handle_request():144] handle_request: poll_exit
2023-04-22 15:53:14,433 INFO Thread-16 :33684 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230422_155255-30hvteby\files\config.yaml
2023-04-22 15:53:14,433 INFO Thread-16 :33684 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230422_155255-30hvteby\files\output.log
2023-04-22 15:53:16,166 INFO SenderThread:33684 [sender.py:transition_state():626] send defer: 8
2023-04-22 15:53:16,166 DEBUG SenderThread:33684 [sender.py:send_request():402] send_request: poll_exit
2023-04-22 15:53:16,166 DEBUG HandlerThread:33684 [handler.py:handle_request():144] handle_request: defer
2023-04-22 15:53:16,166 INFO HandlerThread:33684 [handler.py:handle_request_defer():170] handle defer: 8
2023-04-22 15:53:16,167 DEBUG SenderThread:33684 [sender.py:send_request():402] send_request: defer
2023-04-22 15:53:16,167 INFO SenderThread:33684 [sender.py:send_request_defer():622] handle sender defer: 8
2023-04-22 15:53:16,180 INFO SenderThread:33684 [sender.py:transition_state():626] send defer: 9
2023-04-22 15:53:16,181 DEBUG HandlerThread:33684 [handler.py:handle_request():144] handle_request: defer
2023-04-22 15:53:16,181 DEBUG SenderThread:33684 [sender.py:send():375] send: artifact
2023-04-22 15:53:16,181 INFO HandlerThread:33684 [handler.py:handle_request_defer():170] handle defer: 9
2023-04-22 15:53:16,461 INFO Thread-16 :33684 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230422_155255-30hvteby\files\output.log
2023-04-22 15:53:17,602 INFO SenderThread:33684 [sender.py:send_artifact():1474] sent artifact job-https___github.com_THUDM_ChatGLM-6B_ptuning_main.py - {'id': 'QXJ0aWZhY3Q6NDI5NzAxNTEw', 'digest': '1077e319ad39e537c1ccc8f9a5c233bc', 'state': 'COMMITTED', 'aliases': [{'artifactCollectionName': 'job-https___github.com_THUDM_ChatGLM-6B_ptuning_main.py', 'alias': 'latest'}, {'artifactCollectionName': 'job-https___github.com_THUDM_ChatGLM-6B_ptuning_main.py', 'alias': 'v4'}], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjYzNDcyMzc3', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6NDI5OTE5NTc5', 'versionIndex': 5}}, 'version': 'v4'}
2023-04-22 15:53:17,602 DEBUG SenderThread:33684 [sender.py:send_request():402] send_request: defer
2023-04-22 15:53:17,602 INFO SenderThread:33684 [sender.py:send_request_defer():622] handle sender defer: 9
2023-04-22 15:53:17,603 INFO SenderThread:33684 [dir_watcher.py:finish():365] shutting down directory watcher
2023-04-22 15:53:18,483 INFO SenderThread:33684 [dir_watcher.py:finish():395] scan: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230422_155255-30hvteby\files
2023-04-22 15:53:18,483 INFO SenderThread:33684 [dir_watcher.py:finish():409] scan save: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230422_155255-30hvteby\files\config.yaml config.yaml
2023-04-22 15:53:18,484 INFO SenderThread:33684 [dir_watcher.py:finish():409] scan save: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230422_155255-30hvteby\files\output.log output.log
2023-04-22 15:53:18,486 INFO SenderThread:33684 [dir_watcher.py:finish():409] scan save: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230422_155255-30hvteby\files\requirements.txt requirements.txt
2023-04-22 15:53:18,489 INFO SenderThread:33684 [dir_watcher.py:finish():409] scan save: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230422_155255-30hvteby\files\wandb-metadata.json wandb-metadata.json
2023-04-22 15:53:18,489 INFO SenderThread:33684 [dir_watcher.py:finish():409] scan save: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230422_155255-30hvteby\files\wandb-summary.json wandb-summary.json
2023-04-22 15:53:18,492 INFO SenderThread:33684 [sender.py:transition_state():626] send defer: 10
2023-04-22 15:53:18,492 DEBUG HandlerThread:33684 [handler.py:handle_request():144] handle_request: defer
2023-04-22 15:53:18,492 INFO HandlerThread:33684 [handler.py:handle_request_defer():170] handle defer: 10
2023-04-22 15:53:18,494 DEBUG SenderThread:33684 [sender.py:send_request():402] send_request: defer
2023-04-22 15:53:18,494 INFO SenderThread:33684 [sender.py:send_request_defer():622] handle sender defer: 10
2023-04-22 15:53:18,495 INFO SenderThread:33684 [file_pusher.py:finish():168] shutting down file pusher
2023-04-22 15:53:19,107 DEBUG HandlerThread:33684 [handler.py:handle_request():144] handle_request: keepalive
2023-04-22 15:53:19,301 INFO wandb-upload_1:33684 [upload_job.py:push():137] Uploaded file E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230422_155255-30hvteby\files\output.log
2023-04-22 15:53:19,632 INFO wandb-upload_0:33684 [upload_job.py:push():137] Uploaded file E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230422_155255-30hvteby\files\config.yaml
2023-04-22 15:53:19,821 INFO wandb-upload_2:33684 [upload_job.py:push():137] Uploaded file E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230422_155255-30hvteby\files\requirements.txt
2023-04-22 15:53:20,243 INFO wandb-upload_3:33684 [upload_job.py:push():137] Uploaded file E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230422_155255-30hvteby\files\wandb-summary.json
2023-04-22 15:53:20,445 INFO Thread-15 :33684 [sender.py:transition_state():626] send defer: 11
2023-04-22 15:53:20,445 DEBUG HandlerThread:33684 [handler.py:handle_request():144] handle_request: defer
2023-04-22 15:53:20,445 INFO HandlerThread:33684 [handler.py:handle_request_defer():170] handle defer: 11
2023-04-22 15:53:20,446 DEBUG SenderThread:33684 [sender.py:send_request():402] send_request: defer
2023-04-22 15:53:20,446 INFO SenderThread:33684 [sender.py:send_request_defer():622] handle sender defer: 11
2023-04-22 15:53:20,446 INFO SenderThread:33684 [file_pusher.py:join():173] waiting for file pusher
2023-04-22 15:53:20,446 INFO SenderThread:33684 [sender.py:transition_state():626] send defer: 12
2023-04-22 15:53:20,446 DEBUG HandlerThread:33684 [handler.py:handle_request():144] handle_request: defer
2023-04-22 15:53:20,446 INFO HandlerThread:33684 [handler.py:handle_request_defer():170] handle defer: 12
2023-04-22 15:53:20,446 DEBUG SenderThread:33684 [sender.py:send_request():402] send_request: defer
2023-04-22 15:53:20,446 INFO SenderThread:33684 [sender.py:send_request_defer():622] handle sender defer: 12
2023-04-22 15:53:20,872 INFO SenderThread:33684 [sender.py:transition_state():626] send defer: 13
2023-04-22 15:53:20,872 DEBUG HandlerThread:33684 [handler.py:handle_request():144] handle_request: defer
2023-04-22 15:53:20,872 INFO HandlerThread:33684 [handler.py:handle_request_defer():170] handle defer: 13
2023-04-22 15:53:20,872 DEBUG SenderThread:33684 [sender.py:send_request():402] send_request: defer
2023-04-22 15:53:20,872 INFO SenderThread:33684 [sender.py:send_request_defer():622] handle sender defer: 13
2023-04-22 15:53:20,872 INFO SenderThread:33684 [sender.py:transition_state():626] send defer: 14
2023-04-22 15:53:20,873 DEBUG HandlerThread:33684 [handler.py:handle_request():144] handle_request: defer
2023-04-22 15:53:20,873 DEBUG SenderThread:33684 [sender.py:send():375] send: final
2023-04-22 15:53:20,873 INFO HandlerThread:33684 [handler.py:handle_request_defer():170] handle defer: 14
2023-04-22 15:53:20,873 DEBUG SenderThread:33684 [sender.py:send():375] send: footer
2023-04-22 15:53:20,873 DEBUG SenderThread:33684 [sender.py:send_request():402] send_request: defer
2023-04-22 15:53:20,873 INFO SenderThread:33684 [sender.py:send_request_defer():622] handle sender defer: 14
2023-04-22 15:53:20,874 DEBUG HandlerThread:33684 [handler.py:handle_request():144] handle_request: poll_exit
2023-04-22 15:53:20,874 DEBUG HandlerThread:33684 [handler.py:handle_request():144] handle_request: server_info
2023-04-22 15:53:20,874 DEBUG HandlerThread:33684 [handler.py:handle_request():144] handle_request: get_summary
2023-04-22 15:53:20,875 DEBUG SenderThread:33684 [sender.py:send_request():402] send_request: poll_exit
2023-04-22 15:53:20,875 DEBUG HandlerThread:33684 [handler.py:handle_request():144] handle_request: sampled_history
2023-04-22 15:53:20,875 DEBUG SenderThread:33684 [sender.py:send_request():402] send_request: server_info
2023-04-22 15:53:21,122 INFO MainThread:33684 [wandb_run.py:_footer_history_summary_info():3476] rendering history
2023-04-22 15:53:21,123 INFO MainThread:33684 [wandb_run.py:_footer_history_summary_info():3508] rendering summary
2023-04-22 15:53:21,123 INFO MainThread:33684 [wandb_run.py:_footer_sync_info():3434] logging synced files
2023-04-22 15:53:21,125 DEBUG HandlerThread:33684 [handler.py:handle_request():144] handle_request: shutdown
2023-04-22 15:53:21,125 INFO HandlerThread:33684 [handler.py:finish():845] shutting down handler
2023-04-22 15:53:21,881 INFO WriterThread:33684 [datastore.py:close():298] close: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230422_155255-30hvteby\run-30hvteby.wandb
2023-04-22 15:53:22,129 INFO SenderThread:33684 [sender.py:finish():1550] shutting down sender
2023-04-22 15:53:22,129 INFO SenderThread:33684 [file_pusher.py:finish():168] shutting down file pusher
2023-04-22 15:53:22,129 INFO SenderThread:33684 [file_pusher.py:join():173] waiting for file pusher
|