2023-04-21 04:05:03,268 INFO StreamThr :44364 [internal.py:wandb_internal():86] W&B internal server running at pid: 44364, started at: 2023-04-21 04:05:03.268342 2023-04-21 04:05:03,270 DEBUG HandlerThread:44364 [handler.py:handle_request():144] handle_request: status 2023-04-21 04:05:03,271 INFO WriterThread:44364 [datastore.py:open_for_write():85] open: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_040503-ofh2ksim\run-ofh2ksim.wandb 2023-04-21 04:05:03,275 DEBUG SenderThread:44364 [sender.py:send():375] send: header 2023-04-21 04:05:03,345 DEBUG SenderThread:44364 [sender.py:send():375] send: run 2023-04-21 04:05:04,150 INFO SenderThread:44364 [dir_watcher.py:__init__():219] watching files in: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_040503-ofh2ksim\files 2023-04-21 04:05:04,150 INFO SenderThread:44364 [sender.py:_start_run_threads():1124] run started: ofh2ksim with start time 1682021103.269339 2023-04-21 04:05:04,150 DEBUG SenderThread:44364 [sender.py:send_request():402] send_request: summary_record 2023-04-21 04:05:04,151 INFO SenderThread:44364 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-21 04:05:04,151 DEBUG HandlerThread:44364 [handler.py:handle_request():144] handle_request: check_version 2023-04-21 04:05:04,152 DEBUG SenderThread:44364 [sender.py:send_request():402] send_request: check_version 2023-04-21 04:05:04,728 DEBUG HandlerThread:44364 [handler.py:handle_request():144] handle_request: run_start 2023-04-21 04:05:04,781 DEBUG HandlerThread:44364 [system_info.py:__init__():31] System info init 2023-04-21 04:05:04,781 DEBUG HandlerThread:44364 [system_info.py:__init__():46] System info init done 2023-04-21 04:05:04,781 INFO HandlerThread:44364 [system_monitor.py:start():181] Starting system monitor 2023-04-21 04:05:04,782 INFO SystemMonitor:44364 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-04-21 04:05:04,782 INFO HandlerThread:44364 [system_monitor.py:probe():201] Collecting system info 2023-04-21 04:05:04,790 INFO SystemMonitor:44364 [interfaces.py:start():190] Started cpu monitoring 2023-04-21 04:05:04,790 INFO SystemMonitor:44364 [interfaces.py:start():190] Started disk monitoring 2023-04-21 04:05:04,791 INFO SystemMonitor:44364 [interfaces.py:start():190] Started gpu monitoring 2023-04-21 04:05:04,808 INFO SystemMonitor:44364 [interfaces.py:start():190] Started memory monitoring 2023-04-21 04:05:04,828 INFO SystemMonitor:44364 [interfaces.py:start():190] Started network monitoring 2023-04-21 04:05:04,854 ERROR gpu :44364 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:05:04,863 DEBUG HandlerThread:44364 [system_info.py:probe():195] Probing system 2023-04-21 04:05:04,866 DEBUG HandlerThread:44364 [system_info.py:_probe_git():180] Probing git 2023-04-21 04:05:04,971 DEBUG HandlerThread:44364 [system_info.py:_probe_git():188] Probing git done 2023-04-21 04:05:04,972 DEBUG HandlerThread:44364 [system_info.py:probe():240] Probing system done 2023-04-21 04:05:04,972 DEBUG HandlerThread:44364 [system_monitor.py:probe():210] {'os': 'Windows-10-10.0.19041-SP0', 'python': '3.8.10', 'heartbeatAt': '2023-04-20T20:05:04.864612', 'startedAt': '2023-04-20T20:05:03.256341', 'docker': None, 'cuda': None, 'args': ('--do_train', '--train_file', '..\\AdvertiseGen\\train.json', '--validation_file', '..\\AdvertiseGen\\dev.json', '--prompt_column', 'content', '--response_column', 'summary', '--overwrite_cache', '--model_name_or_path', '..\\models\\chatglm-6b-int4-1\\models--THUDM--chatglm-6b-int4\\snapshots\\e02ba894cf18f3fd9b2526c795f983683c4ec732', '--output_dir', 'output\\adgen-chatglm-6b-pt-128-2e-2', '--overwrite_output_dir', '--max_source_length', '64', '--max_target_length', '64', '--per_device_train_batch_size', '1', '--per_device_eval_batch_size', '1', '--gradient_accumulation_steps', '16', '--predict_with_generate', '--max_steps', '10', '--logging_steps', '10', '--save_steps', '5', '--learning_rate', '2e-2', '--pre_seq_len', '128', '--quantization_bit', '4'), 'state': 'running', 'program': 'main.py', 'codePath': 'ptuning\\main.py', 'git': {'remote': 'https://github.com/THUDM/ChatGLM-6B', 'commit': '01e6313abf4122d789d6e68128856af52847b355'}, 'cpu_count': 6, 'cpu_count_logical': 12, 'cpu_freq': {'current': 2592.0, 'min': 0.0, 'max': 2592.0}, 'cpu_freq_per_core': [{'current': 2592.0, 'min': 0.0, 'max': 2592.0}], 'disk': {'total': 500.32030868530273, 'used': 495.093204498291}, 'gpu': 'NVIDIA GeForce RTX 2060', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 2060', 'memory_total': 6442450944}], 'memory': {'total': 63.87089538574219}} 2023-04-21 04:05:04,972 INFO HandlerThread:44364 [system_monitor.py:probe():211] Finished collecting system info 2023-04-21 04:05:04,972 INFO HandlerThread:44364 [system_monitor.py:probe():214] Publishing system info 2023-04-21 04:05:04,972 DEBUG HandlerThread:44364 [system_info.py:_save_pip():51] Saving list of pip packages installed into the current environment 2023-04-21 04:05:04,973 DEBUG HandlerThread:44364 [system_info.py:_save_pip():67] Saving pip packages done 2023-04-21 04:05:04,974 INFO HandlerThread:44364 [system_monitor.py:probe():216] Finished publishing system info 2023-04-21 04:05:04,986 DEBUG SenderThread:44364 [sender.py:send():375] send: files 2023-04-21 04:05:04,986 INFO SenderThread:44364 [sender.py:_save_file():1378] saving file wandb-metadata.json with policy now 2023-04-21 04:05:04,998 DEBUG HandlerThread:44364 [handler.py:handle_request():144] handle_request: stop_status 2023-04-21 04:05:04,999 DEBUG SenderThread:44364 [sender.py:send_request():402] send_request: stop_status 2023-04-21 04:05:05,154 INFO Thread-16 :44364 [dir_watcher.py:_on_file_created():278] file/dir created: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_040503-ofh2ksim\files\requirements.txt 2023-04-21 04:05:05,156 INFO Thread-16 :44364 [dir_watcher.py:_on_file_created():278] file/dir created: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_040503-ofh2ksim\files\wandb-metadata.json 2023-04-21 04:05:05,156 INFO Thread-16 :44364 [dir_watcher.py:_on_file_created():278] file/dir created: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_040503-ofh2ksim\files\wandb-summary.json 2023-04-21 04:05:05,397 DEBUG SenderThread:44364 [sender.py:send():375] send: telemetry 2023-04-21 04:05:05,397 DEBUG SenderThread:44364 [sender.py:send():375] send: config 2023-04-21 04:05:05,398 DEBUG SenderThread:44364 [sender.py:send():375] send: metric 2023-04-21 04:05:05,398 DEBUG SenderThread:44364 [sender.py:send():375] send: telemetry 2023-04-21 04:05:05,399 DEBUG SenderThread:44364 [sender.py:send():375] send: metric 2023-04-21 04:05:05,399 WARNING SenderThread:44364 [sender.py:send_metric():1329] Seen metric with glob (shouldn't happen) 2023-04-21 04:05:05,941 INFO wandb-upload_0:44364 [upload_job.py:push():137] Uploaded file C:\Users\Lenovo\AppData\Local\Temp\tmpttogkq_swandb\3u4h3e8i-wandb-metadata.json 2023-04-21 04:05:06,166 INFO Thread-16 :44364 [dir_watcher.py:_on_file_created():278] file/dir created: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_040503-ofh2ksim\files\output.log 2023-04-21 04:05:06,908 ERROR gpu :44364 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:05:08,176 INFO Thread-16 :44364 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_040503-ofh2ksim\files\output.log 2023-04-21 04:05:08,689 DEBUG HandlerThread:44364 [handler.py:handle_request():144] handle_request: status_report 2023-04-21 04:05:08,965 ERROR gpu :44364 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:05:09,177 INFO Thread-16 :44364 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_040503-ofh2ksim\files\output.log 2023-04-21 04:05:11,007 ERROR gpu :44364 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:05:13,046 ERROR gpu :44364 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:05:13,751 DEBUG HandlerThread:44364 [handler.py:handle_request():144] handle_request: status_report 2023-04-21 04:05:15,087 ERROR gpu :44364 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:05:17,132 ERROR gpu :44364 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:05:18,785 DEBUG HandlerThread:44364 [handler.py:handle_request():144] handle_request: status_report 2023-04-21 04:05:19,178 ERROR gpu :44364 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:05:19,994 DEBUG HandlerThread:44364 [handler.py:handle_request():144] handle_request: stop_status 2023-04-21 04:05:19,995 DEBUG SenderThread:44364 [sender.py:send_request():402] send_request: stop_status 2023-04-21 04:05:21,226 ERROR gpu :44364 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:05:23,274 ERROR gpu :44364 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:05:24,280 DEBUG HandlerThread:44364 [handler.py:handle_request():144] handle_request: status_report 2023-04-21 04:05:25,322 ERROR gpu :44364 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:05:27,404 ERROR gpu :44364 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:05:29,337 DEBUG HandlerThread:44364 [handler.py:handle_request():144] handle_request: status_report 2023-04-21 04:05:29,452 ERROR gpu :44364 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:05:31,505 ERROR gpu :44364 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:05:33,543 ERROR gpu :44364 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:05:34,384 DEBUG HandlerThread:44364 [handler.py:handle_request():144] handle_request: status_report 2023-04-21 04:05:35,001 DEBUG HandlerThread:44364 [handler.py:handle_request():144] handle_request: stop_status 2023-04-21 04:05:35,001 DEBUG SenderThread:44364 [sender.py:send_request():402] send_request: stop_status 2023-04-21 04:05:35,609 INFO Thread-16 :44364 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_040503-ofh2ksim\files\config.yaml 2023-04-21 04:05:35,654 ERROR gpu :44364 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:05:37,684 ERROR gpu :44364 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:05:39,734 ERROR gpu :44364 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:05:40,296 DEBUG HandlerThread:44364 [handler.py:handle_request():144] handle_request: status_report 2023-04-21 04:05:41,771 ERROR gpu :44364 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:05:43,813 ERROR gpu :44364 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:05:45,321 DEBUG HandlerThread:44364 [handler.py:handle_request():144] handle_request: status_report 2023-04-21 04:05:45,862 ERROR gpu :44364 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:05:47,944 ERROR gpu :44364 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:05:49,991 ERROR gpu :44364 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:05:50,006 DEBUG HandlerThread:44364 [handler.py:handle_request():144] handle_request: stop_status 2023-04-21 04:05:50,007 DEBUG SenderThread:44364 [sender.py:send_request():402] send_request: stop_status 2023-04-21 04:05:51,257 DEBUG HandlerThread:44364 [handler.py:handle_request():144] handle_request: status_report 2023-04-21 04:05:52,035 INFO Thread-16 :44364 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_040503-ofh2ksim\files\output.log 2023-04-21 04:05:52,042 ERROR gpu :44364 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:05:54,087 ERROR gpu :44364 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:05:56,126 ERROR gpu :44364 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:05:56,310 DEBUG HandlerThread:44364 [handler.py:handle_request():144] handle_request: status_report 2023-04-21 04:05:58,165 ERROR gpu :44364 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:06:00,232 ERROR gpu :44364 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:06:01,351 DEBUG HandlerThread:44364 [handler.py:handle_request():144] handle_request: status_report 2023-04-21 04:06:02,296 ERROR gpu :44364 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:06:04,353 ERROR gpu :44364 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:06:04,839 DEBUG SystemMonitor:44364 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-04-21 04:06:04,840 DEBUG SenderThread:44364 [sender.py:send():375] send: stats 2023-04-21 04:06:05,014 DEBUG HandlerThread:44364 [handler.py:handle_request():144] handle_request: stop_status 2023-04-21 04:06:05,014 DEBUG SenderThread:44364 [sender.py:send_request():402] send_request: stop_status 2023-04-21 04:06:06,429 INFO Thread-16 :44364 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_040503-ofh2ksim\files\output.log 2023-04-21 04:06:06,467 ERROR gpu :44364 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:06:07,292 DEBUG HandlerThread:44364 [handler.py:handle_request():144] handle_request: status_report 2023-04-21 04:06:08,479 ERROR gpu :44364 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:06:10,502 ERROR gpu :44364 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:06:12,327 DEBUG HandlerThread:44364 [handler.py:handle_request():144] handle_request: status_report 2023-04-21 04:06:12,555 ERROR gpu :44364 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:06:14,602 ERROR gpu :44364 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:06:16,651 ERROR gpu :44364 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:06:17,366 DEBUG HandlerThread:44364 [handler.py:handle_request():144] handle_request: status_report 2023-04-21 04:06:18,708 ERROR gpu :44364 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:06:20,019 DEBUG HandlerThread:44364 [handler.py:handle_request():144] handle_request: stop_status 2023-04-21 04:06:20,019 DEBUG SenderThread:44364 [sender.py:send_request():402] send_request: stop_status 2023-04-21 04:06:20,751 INFO Thread-16 :44364 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_040503-ofh2ksim\files\output.log 2023-04-21 04:06:20,764 ERROR gpu :44364 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:06:22,818 ERROR gpu :44364 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:06:23,289 DEBUG HandlerThread:44364 [handler.py:handle_request():144] handle_request: status_report 2023-04-21 04:06:24,854 ERROR gpu :44364 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:06:26,905 ERROR gpu :44364 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:06:28,330 DEBUG HandlerThread:44364 [handler.py:handle_request():144] handle_request: status_report 2023-04-21 04:06:28,949 ERROR gpu :44364 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:06:31,011 ERROR gpu :44364 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:06:33,058 ERROR gpu :44364 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:06:33,528 DEBUG HandlerThread:44364 [handler.py:handle_request():144] handle_request: status_report 2023-04-21 04:06:34,052 INFO Thread-16 :44364 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_040503-ofh2ksim\files\output.log 2023-04-21 04:06:34,843 DEBUG SenderThread:44364 [sender.py:send():375] send: stats 2023-04-21 04:06:35,028 DEBUG HandlerThread:44364 [handler.py:handle_request():144] handle_request: stop_status 2023-04-21 04:06:35,029 DEBUG SenderThread:44364 [sender.py:send_request():402] send_request: stop_status 2023-04-21 04:06:35,108 ERROR gpu :44364 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:06:37,225 ERROR gpu :44364 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:06:39,247 ERROR gpu :44364 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:06:39,316 DEBUG HandlerThread:44364 [handler.py:handle_request():144] handle_request: status_report 2023-04-21 04:06:41,277 ERROR gpu :44364 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:06:43,328 ERROR gpu :44364 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:06:44,369 DEBUG HandlerThread:44364 [handler.py:handle_request():144] handle_request: status_report 2023-04-21 04:06:45,378 ERROR gpu :44364 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:06:47,431 INFO Thread-16 :44364 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_040503-ofh2ksim\files\output.log 2023-04-21 04:06:47,445 ERROR gpu :44364 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:06:48,439 INFO Thread-16 :44364 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_040503-ofh2ksim\files\output.log 2023-04-21 04:06:49,501 ERROR gpu :44364 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:06:49,912 DEBUG HandlerThread:44364 [handler.py:handle_request():144] handle_request: status_report 2023-04-21 04:06:50,037 DEBUG HandlerThread:44364 [handler.py:handle_request():144] handle_request: stop_status 2023-04-21 04:06:50,037 DEBUG SenderThread:44364 [sender.py:send_request():402] send_request: stop_status 2023-04-21 04:06:51,566 ERROR gpu :44364 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:06:53,615 ERROR gpu :44364 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:06:55,327 DEBUG HandlerThread:44364 [handler.py:handle_request():144] handle_request: status_report 2023-04-21 04:06:55,677 ERROR gpu :44364 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:06:57,738 ERROR gpu :44364 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:06:59,789 ERROR gpu :44364 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:07:00,363 DEBUG HandlerThread:44364 [handler.py:handle_request():144] handle_request: status_report 2023-04-21 04:07:01,838 ERROR gpu :44364 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:07:03,881 ERROR gpu :44364 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:07:04,857 DEBUG SenderThread:44364 [sender.py:send():375] send: stats 2023-04-21 04:07:04,872 INFO Thread-16 :44364 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_040503-ofh2ksim\files\output.log 2023-04-21 04:07:05,039 DEBUG HandlerThread:44364 [handler.py:handle_request():144] handle_request: stop_status 2023-04-21 04:07:05,040 DEBUG SenderThread:44364 [sender.py:send_request():402] send_request: stop_status 2023-04-21 04:07:05,918 ERROR gpu :44364 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:07:06,301 DEBUG HandlerThread:44364 [handler.py:handle_request():144] handle_request: status_report 2023-04-21 04:07:08,065 ERROR gpu :44364 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:07:10,076 ERROR gpu :44364 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:07:11,325 DEBUG HandlerThread:44364 [handler.py:handle_request():144] handle_request: status_report 2023-04-21 04:07:12,118 ERROR gpu :44364 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:07:14,171 ERROR gpu :44364 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:07:16,214 ERROR gpu :44364 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:07:16,387 DEBUG HandlerThread:44364 [handler.py:handle_request():144] handle_request: status_report 2023-04-21 04:07:18,275 INFO Thread-16 :44364 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_040503-ofh2ksim\files\output.log 2023-04-21 04:07:18,284 ERROR gpu :44364 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:07:20,046 DEBUG HandlerThread:44364 [handler.py:handle_request():144] handle_request: stop_status 2023-04-21 04:07:20,047 DEBUG SenderThread:44364 [sender.py:send_request():402] send_request: stop_status 2023-04-21 04:07:20,339 ERROR gpu :44364 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-21 04:07:20,714 DEBUG SenderThread:44364 [sender.py:send():375] send: exit 2023-04-21 04:07:20,714 INFO SenderThread:44364 [sender.py:send_exit():598] handling exit code: 255 2023-04-21 04:07:20,714 INFO SenderThread:44364 [sender.py:send_exit():600] handling runtime: 135 2023-04-21 04:07:20,715 INFO SenderThread:44364 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-21 04:07:20,715 INFO SenderThread:44364 [sender.py:send_exit():606] send defer 2023-04-21 04:07:20,715 DEBUG HandlerThread:44364 [handler.py:handle_request():144] handle_request: defer 2023-04-21 04:07:20,716 INFO HandlerThread:44364 [handler.py:handle_request_defer():170] handle defer: 0 2023-04-21 04:07:20,716 DEBUG SenderThread:44364 [sender.py:send_request():402] send_request: defer 2023-04-21 04:07:20,716 INFO SenderThread:44364 [sender.py:send_request_defer():622] handle sender defer: 0 2023-04-21 04:07:20,716 INFO SenderThread:44364 [sender.py:transition_state():626] send defer: 1 2023-04-21 04:07:20,716 DEBUG HandlerThread:44364 [handler.py:handle_request():144] handle_request: defer 2023-04-21 04:07:20,716 INFO HandlerThread:44364 [handler.py:handle_request_defer():170] handle defer: 1 2023-04-21 04:07:20,716 DEBUG SenderThread:44364 [sender.py:send_request():402] send_request: defer 2023-04-21 04:07:20,716 INFO SenderThread:44364 [sender.py:send_request_defer():622] handle sender defer: 1 2023-04-21 04:07:20,717 INFO SenderThread:44364 [sender.py:transition_state():626] send defer: 2 2023-04-21 04:07:20,717 DEBUG HandlerThread:44364 [handler.py:handle_request():144] handle_request: defer 2023-04-21 04:07:20,717 INFO HandlerThread:44364 [handler.py:handle_request_defer():170] handle defer: 2 2023-04-21 04:07:20,717 INFO HandlerThread:44364 [system_monitor.py:finish():190] Stopping system monitor 2023-04-21 04:07:20,717 DEBUG SystemMonitor:44364 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-04-21 04:07:20,717 DEBUG SystemMonitor:44364 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-04-21 04:07:20,717 INFO HandlerThread:44364 [interfaces.py:finish():202] Joined cpu monitor 2023-04-21 04:07:20,730 INFO HandlerThread:44364 [interfaces.py:finish():202] Joined disk monitor 2023-04-21 04:07:20,787 INFO HandlerThread:44364 [interfaces.py:finish():202] Joined gpu monitor 2023-04-21 04:07:20,787 INFO HandlerThread:44364 [interfaces.py:finish():202] Joined memory monitor 2023-04-21 04:07:20,787 INFO HandlerThread:44364 [interfaces.py:finish():202] Joined network monitor 2023-04-21 04:07:20,788 DEBUG SenderThread:44364 [sender.py:send_request():402] send_request: defer 2023-04-21 04:07:20,788 INFO SenderThread:44364 [sender.py:send_request_defer():622] handle sender defer: 2 2023-04-21 04:07:20,788 INFO SenderThread:44364 [sender.py:transition_state():626] send defer: 3 2023-04-21 04:07:20,789 DEBUG SenderThread:44364 [sender.py:send():375] send: stats 2023-04-21 04:07:20,789 DEBUG HandlerThread:44364 [handler.py:handle_request():144] handle_request: defer 2023-04-21 04:07:20,789 INFO HandlerThread:44364 [handler.py:handle_request_defer():170] handle defer: 3 2023-04-21 04:07:20,790 DEBUG SenderThread:44364 [sender.py:send_request():402] send_request: defer 2023-04-21 04:07:20,790 INFO SenderThread:44364 [sender.py:send_request_defer():622] handle sender defer: 3 2023-04-21 04:07:20,790 INFO SenderThread:44364 [sender.py:transition_state():626] send defer: 4 2023-04-21 04:07:20,790 DEBUG HandlerThread:44364 [handler.py:handle_request():144] handle_request: defer 2023-04-21 04:07:20,791 INFO HandlerThread:44364 [handler.py:handle_request_defer():170] handle defer: 4 2023-04-21 04:07:20,792 DEBUG SenderThread:44364 [sender.py:send_request():402] send_request: defer 2023-04-21 04:07:20,792 INFO SenderThread:44364 [sender.py:send_request_defer():622] handle sender defer: 4 2023-04-21 04:07:20,792 INFO SenderThread:44364 [sender.py:transition_state():626] send defer: 5 2023-04-21 04:07:20,792 DEBUG HandlerThread:44364 [handler.py:handle_request():144] handle_request: defer 2023-04-21 04:07:20,792 INFO HandlerThread:44364 [handler.py:handle_request_defer():170] handle defer: 5 2023-04-21 04:07:20,793 DEBUG SenderThread:44364 [sender.py:send():375] send: summary 2023-04-21 04:07:20,793 INFO SenderThread:44364 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-21 04:07:20,794 DEBUG SenderThread:44364 [sender.py:send_request():402] send_request: defer 2023-04-21 04:07:20,794 INFO SenderThread:44364 [sender.py:send_request_defer():622] handle sender defer: 5 2023-04-21 04:07:20,794 INFO SenderThread:44364 [sender.py:transition_state():626] send defer: 6 2023-04-21 04:07:20,794 DEBUG HandlerThread:44364 [handler.py:handle_request():144] handle_request: defer 2023-04-21 04:07:20,794 INFO HandlerThread:44364 [handler.py:handle_request_defer():170] handle defer: 6 2023-04-21 04:07:20,795 DEBUG SenderThread:44364 [sender.py:send_request():402] send_request: defer 2023-04-21 04:07:20,795 INFO SenderThread:44364 [sender.py:send_request_defer():622] handle sender defer: 6 2023-04-21 04:07:20,795 INFO SenderThread:44364 [sender.py:transition_state():626] send defer: 7 2023-04-21 04:07:20,795 DEBUG HandlerThread:44364 [handler.py:handle_request():144] handle_request: status_report 2023-04-21 04:07:20,795 DEBUG HandlerThread:44364 [handler.py:handle_request():144] handle_request: defer 2023-04-21 04:07:20,795 INFO HandlerThread:44364 [handler.py:handle_request_defer():170] handle defer: 7 2023-04-21 04:07:20,795 DEBUG SenderThread:44364 [sender.py:send_request():402] send_request: defer 2023-04-21 04:07:20,795 INFO SenderThread:44364 [sender.py:send_request_defer():622] handle sender defer: 7 2023-04-21 04:07:21,345 INFO Thread-16 :44364 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_040503-ofh2ksim\files\wandb-summary.json 2023-04-21 04:07:21,803 DEBUG HandlerThread:44364 [handler.py:handle_request():144] handle_request: poll_exit 2023-04-21 04:07:22,356 INFO Thread-16 :44364 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_040503-ofh2ksim\files\output.log 2023-04-21 04:07:23,275 INFO SenderThread:44364 [sender.py:transition_state():626] send defer: 8 2023-04-21 04:07:23,275 DEBUG SenderThread:44364 [sender.py:send_request():402] send_request: poll_exit 2023-04-21 04:07:23,275 DEBUG HandlerThread:44364 [handler.py:handle_request():144] handle_request: defer 2023-04-21 04:07:23,276 INFO HandlerThread:44364 [handler.py:handle_request_defer():170] handle defer: 8 2023-04-21 04:07:23,276 DEBUG SenderThread:44364 [sender.py:send_request():402] send_request: defer 2023-04-21 04:07:23,276 INFO SenderThread:44364 [sender.py:send_request_defer():622] handle sender defer: 8 2023-04-21 04:07:23,294 INFO SenderThread:44364 [sender.py:transition_state():626] send defer: 9 2023-04-21 04:07:23,295 DEBUG HandlerThread:44364 [handler.py:handle_request():144] handle_request: defer 2023-04-21 04:07:23,295 DEBUG SenderThread:44364 [sender.py:send():375] send: artifact 2023-04-21 04:07:23,295 INFO HandlerThread:44364 [handler.py:handle_request_defer():170] handle defer: 9 2023-04-21 04:07:23,370 INFO Thread-16 :44364 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_040503-ofh2ksim\files\output.log 2023-04-21 04:07:24,678 INFO SenderThread:44364 [sender.py:send_artifact():1474] sent artifact job-https___github.com_THUDM_ChatGLM-6B_ptuning_main.py - {'id': 'QXJ0aWZhY3Q6NDI5MjY1Nzg0', 'digest': '53476254d59d98151858729a5d45f45c', 'state': 'COMMITTED', 'aliases': [{'artifactCollectionName': 'job-https___github.com_THUDM_ChatGLM-6B_ptuning_main.py', 'alias': 'latest'}, {'artifactCollectionName': 'job-https___github.com_THUDM_ChatGLM-6B_ptuning_main.py', 'alias': 'v2'}], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjYzNDcyMzc3', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6NDI5MjY1Nzg0', 'versionIndex': 2}}, 'version': 'v2'} 2023-04-21 04:07:24,679 DEBUG SenderThread:44364 [sender.py:send_request():402] send_request: defer 2023-04-21 04:07:24,679 INFO SenderThread:44364 [sender.py:send_request_defer():622] handle sender defer: 9 2023-04-21 04:07:24,679 INFO SenderThread:44364 [dir_watcher.py:finish():365] shutting down directory watcher 2023-04-21 04:07:25,388 INFO SenderThread:44364 [dir_watcher.py:finish():395] scan: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_040503-ofh2ksim\files 2023-04-21 04:07:25,388 INFO SenderThread:44364 [dir_watcher.py:finish():409] scan save: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_040503-ofh2ksim\files\config.yaml config.yaml 2023-04-21 04:07:25,389 INFO SenderThread:44364 [dir_watcher.py:finish():409] scan save: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_040503-ofh2ksim\files\output.log output.log 2023-04-21 04:07:25,392 INFO SenderThread:44364 [dir_watcher.py:finish():409] scan save: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_040503-ofh2ksim\files\requirements.txt requirements.txt 2023-04-21 04:07:25,395 INFO SenderThread:44364 [dir_watcher.py:finish():409] scan save: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_040503-ofh2ksim\files\wandb-metadata.json wandb-metadata.json 2023-04-21 04:07:25,395 INFO SenderThread:44364 [dir_watcher.py:finish():409] scan save: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_040503-ofh2ksim\files\wandb-summary.json wandb-summary.json 2023-04-21 04:07:25,397 INFO SenderThread:44364 [sender.py:transition_state():626] send defer: 10 2023-04-21 04:07:25,398 DEBUG HandlerThread:44364 [handler.py:handle_request():144] handle_request: defer 2023-04-21 04:07:25,398 INFO HandlerThread:44364 [handler.py:handle_request_defer():170] handle defer: 10 2023-04-21 04:07:25,400 DEBUG SenderThread:44364 [sender.py:send_request():402] send_request: defer 2023-04-21 04:07:25,401 INFO SenderThread:44364 [sender.py:send_request_defer():622] handle sender defer: 10 2023-04-21 04:07:25,401 INFO SenderThread:44364 [file_pusher.py:finish():168] shutting down file pusher 2023-04-21 04:07:26,054 INFO wandb-upload_0:44364 [upload_job.py:push():137] Uploaded file E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_040503-ofh2ksim\files\config.yaml 2023-04-21 04:07:26,501 INFO wandb-upload_1:44364 [upload_job.py:push():137] Uploaded file E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_040503-ofh2ksim\files\output.log 2023-04-21 04:07:26,601 INFO wandb-upload_3:44364 [upload_job.py:push():137] Uploaded file E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_040503-ofh2ksim\files\wandb-summary.json 2023-04-21 04:07:26,682 INFO wandb-upload_2:44364 [upload_job.py:push():137] Uploaded file E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_040503-ofh2ksim\files\requirements.txt 2023-04-21 04:07:26,840 DEBUG HandlerThread:44364 [handler.py:handle_request():144] handle_request: keepalive 2023-04-21 04:07:26,886 INFO Thread-15 :44364 [sender.py:transition_state():626] send defer: 11 2023-04-21 04:07:26,886 DEBUG HandlerThread:44364 [handler.py:handle_request():144] handle_request: defer 2023-04-21 04:07:26,886 INFO HandlerThread:44364 [handler.py:handle_request_defer():170] handle defer: 11 2023-04-21 04:07:26,886 DEBUG SenderThread:44364 [sender.py:send_request():402] send_request: defer 2023-04-21 04:07:26,887 INFO SenderThread:44364 [sender.py:send_request_defer():622] handle sender defer: 11 2023-04-21 04:07:26,887 INFO SenderThread:44364 [file_pusher.py:join():173] waiting for file pusher 2023-04-21 04:07:26,887 INFO SenderThread:44364 [sender.py:transition_state():626] send defer: 12 2023-04-21 04:07:26,887 DEBUG HandlerThread:44364 [handler.py:handle_request():144] handle_request: defer 2023-04-21 04:07:26,887 INFO HandlerThread:44364 [handler.py:handle_request_defer():170] handle defer: 12 2023-04-21 04:07:26,887 DEBUG SenderThread:44364 [sender.py:send_request():402] send_request: defer 2023-04-21 04:07:26,887 INFO SenderThread:44364 [sender.py:send_request_defer():622] handle sender defer: 12 2023-04-21 04:07:27,497 INFO SenderThread:44364 [sender.py:transition_state():626] send defer: 13 2023-04-21 04:07:27,497 DEBUG HandlerThread:44364 [handler.py:handle_request():144] handle_request: defer 2023-04-21 04:07:27,497 INFO HandlerThread:44364 [handler.py:handle_request_defer():170] handle defer: 13 2023-04-21 04:07:27,497 DEBUG SenderThread:44364 [sender.py:send_request():402] send_request: defer 2023-04-21 04:07:27,497 INFO SenderThread:44364 [sender.py:send_request_defer():622] handle sender defer: 13 2023-04-21 04:07:27,497 INFO SenderThread:44364 [sender.py:transition_state():626] send defer: 14 2023-04-21 04:07:27,498 DEBUG HandlerThread:44364 [handler.py:handle_request():144] handle_request: defer 2023-04-21 04:07:27,498 DEBUG SenderThread:44364 [sender.py:send():375] send: final 2023-04-21 04:07:27,498 INFO HandlerThread:44364 [handler.py:handle_request_defer():170] handle defer: 14 2023-04-21 04:07:27,498 DEBUG SenderThread:44364 [sender.py:send():375] send: footer 2023-04-21 04:07:27,498 DEBUG SenderThread:44364 [sender.py:send_request():402] send_request: defer 2023-04-21 04:07:27,498 INFO SenderThread:44364 [sender.py:send_request_defer():622] handle sender defer: 14 2023-04-21 04:07:27,499 DEBUG HandlerThread:44364 [handler.py:handle_request():144] handle_request: poll_exit 2023-04-21 04:07:27,499 DEBUG HandlerThread:44364 [handler.py:handle_request():144] handle_request: server_info 2023-04-21 04:07:27,499 DEBUG HandlerThread:44364 [handler.py:handle_request():144] handle_request: get_summary 2023-04-21 04:07:27,499 DEBUG SenderThread:44364 [sender.py:send_request():402] send_request: poll_exit 2023-04-21 04:07:27,499 DEBUG HandlerThread:44364 [handler.py:handle_request():144] handle_request: sampled_history 2023-04-21 04:07:27,499 DEBUG SenderThread:44364 [sender.py:send_request():402] send_request: server_info 2023-04-21 04:07:27,738 INFO MainThread:44364 [wandb_run.py:_footer_history_summary_info():3476] rendering history 2023-04-21 04:07:27,738 INFO MainThread:44364 [wandb_run.py:_footer_history_summary_info():3508] rendering summary 2023-04-21 04:07:27,738 INFO MainThread:44364 [wandb_run.py:_footer_sync_info():3434] logging synced files 2023-04-21 04:07:27,739 DEBUG HandlerThread:44364 [handler.py:handle_request():144] handle_request: shutdown 2023-04-21 04:07:27,739 INFO HandlerThread:44364 [handler.py:finish():845] shutting down handler 2023-04-21 04:07:28,504 INFO WriterThread:44364 [datastore.py:close():298] close: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_040503-ofh2ksim\run-ofh2ksim.wandb 2023-04-21 04:07:28,739 INFO SenderThread:44364 [sender.py:finish():1550] shutting down sender 2023-04-21 04:07:28,739 INFO SenderThread:44364 [file_pusher.py:finish():168] shutting down file pusher 2023-04-21 04:07:28,739 INFO SenderThread:44364 [file_pusher.py:join():173] waiting for file pusher