ptuning/wandb/run-20230421_120047-67c5qu7s/logs/debug-internal.log

2023-04-21 12:00:47,163 INFO StreamThr :4128 [internal.py:wandb_internal():86] W&B internal server running at pid: 4128, started at: 2023-04-21 12:00:47.162997
2023-04-21 12:00:47,164 DEBUG HandlerThread:4128 [handler.py:handle_request():144] handle_request: status
2023-04-21 12:00:47,165 INFO WriterThread:4128 [datastore.py:open_for_write():85] open: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_120047-67c5qu7s\run-67c5qu7s.wandb
2023-04-21 12:00:47,169 DEBUG SenderThread:4128 [sender.py:send():375] send: header
2023-04-21 12:00:47,234 DEBUG SenderThread:4128 [sender.py:send():375] send: run
2023-04-21 12:00:47,952 INFO SenderThread:4128 [dir_watcher.py:__init__():219] watching files in: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_120047-67c5qu7s\files
2023-04-21 12:00:47,952 INFO SenderThread:4128 [sender.py:_start_run_threads():1124] run started: 67c5qu7s with start time 1682049647.163995
2023-04-21 12:00:47,952 DEBUG SenderThread:4128 [sender.py:send_request():402] send_request: summary_record
2023-04-21 12:00:47,953 INFO SenderThread:4128 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-04-21 12:00:47,955 DEBUG HandlerThread:4128 [handler.py:handle_request():144] handle_request: check_version
2023-04-21 12:00:47,956 DEBUG SenderThread:4128 [sender.py:send_request():402] send_request: check_version
2023-04-21 12:00:48,555 DEBUG HandlerThread:4128 [handler.py:handle_request():144] handle_request: run_start
2023-04-21 12:00:48,628 DEBUG HandlerThread:4128 [system_info.py:__init__():31] System info init
2023-04-21 12:00:48,628 DEBUG HandlerThread:4128 [system_info.py:__init__():46] System info init done
2023-04-21 12:00:48,628 INFO HandlerThread:4128 [system_monitor.py:start():181] Starting system monitor
2023-04-21 12:00:48,628 INFO SystemMonitor:4128 [system_monitor.py:_start():145] Starting system asset monitoring threads
2023-04-21 12:00:48,628 INFO HandlerThread:4128 [system_monitor.py:probe():201] Collecting system info
2023-04-21 12:00:48,637 INFO SystemMonitor:4128 [interfaces.py:start():190] Started cpu monitoring
2023-04-21 12:00:48,637 INFO SystemMonitor:4128 [interfaces.py:start():190] Started disk monitoring
2023-04-21 12:00:48,638 INFO SystemMonitor:4128 [interfaces.py:start():190] Started gpu monitoring
2023-04-21 12:00:48,654 INFO SystemMonitor:4128 [interfaces.py:start():190] Started memory monitoring
2023-04-21 12:00:48,687 INFO SystemMonitor:4128 [interfaces.py:start():190] Started network monitoring
2023-04-21 12:00:48,730 DEBUG HandlerThread:4128 [system_info.py:probe():195] Probing system
2023-04-21 12:00:48,733 DEBUG HandlerThread:4128 [system_info.py:_probe_git():180] Probing git
2023-04-21 12:00:48,737 ERROR gpu :4128 [interfaces.py:monitor():144] Failed to sample metric: Not Supported
2023-04-21 12:00:48,843 DEBUG HandlerThread:4128 [system_info.py:_probe_git():188] Probing git done
2023-04-21 12:00:48,844 DEBUG HandlerThread:4128 [system_info.py:probe():240] Probing system done
2023-04-21 12:00:48,844 DEBUG HandlerThread:4128 [system_monitor.py:probe():210] {'os': 'Windows-10-10.0.19041-SP0', 'python': '3.8.10', 'heartbeatAt': '2023-04-21T04:00:48.730598', 'startedAt': '2023-04-21T04:00:47.144996', 'docker': None, 'cuda': None, 'args': ('--do_train', '--train_file', '..\\AdvertiseGen\\train.json', '--validation_file', '..\\AdvertiseGen\\dev.json', '--prompt_column', 'content', '--response_column', 'summary', '--overwrite_cache', '--model_name_or_path', '..\\models\\chatglm-6b-int4-1\\models--THUDM--chatglm-6b-int4\\snapshots\\e02ba894cf18f3fd9b2526c795f983683c4ec732', '--output_dir', 'output\\adgen-chatglm-6b-pt-128-2e-2', '--overwrite_output_dir', '--max_source_length', '64', '--max_target_length', '64', '--per_device_train_batch_size', '1', '--per_device_eval_batch_size', '1', '--gradient_accumulation_steps', '16', '--predict_with_generate', '--max_steps', '1000', '--logging_steps', '10', '--save_steps', '100', '--learning_rate', '2e-2', '--pre_seq_len', '128', '--quantization_bit', '4'), 'state': 'running', 'program': 'main.py', 'codePath': 'ptuning\\main.py', 'git': {'remote': 'https://github.com/THUDM/ChatGLM-6B', 'commit': '01e6313abf4122d789d6e68128856af52847b355'}, 'cpu_count': 6, 'cpu_count_logical': 12, 'cpu_freq': {'current': 2592.0, 'min': 0.0, 'max': 2592.0}, 'cpu_freq_per_core': [{'current': 2592.0, 'min': 0.0, 'max': 2592.0}], 'disk': {'total': 500.32030868530273, 'used': 486.5024871826172}, 'gpu': 'NVIDIA GeForce RTX 2060', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 2060', 'memory_total': 6442450944}], 'memory': {'total': 63.87089538574219}}
2023-04-21 12:00:48,844 INFO HandlerThread:4128 [system_monitor.py:probe():211] Finished collecting system info
2023-04-21 12:00:48,844 INFO HandlerThread:4128 [system_monitor.py:probe():214] Publishing system info
2023-04-21 12:00:48,844 DEBUG HandlerThread:4128 [system_info.py:_save_pip():51] Saving list of pip packages installed into the current environment
2023-04-21 12:00:48,845 DEBUG HandlerThread:4128 [system_info.py:_save_pip():67] Saving pip packages done
2023-04-21 12:00:48,846 INFO HandlerThread:4128 [system_monitor.py:probe():216] Finished publishing system info
2023-04-21 12:00:48,858 DEBUG SenderThread:4128 [sender.py:send():375] send: files
2023-04-21 12:00:48,859 INFO SenderThread:4128 [sender.py:_save_file():1378] saving file wandb-metadata.json with policy now
2023-04-21 12:00:48,872 DEBUG HandlerThread:4128 [handler.py:handle_request():144] handle_request: stop_status
2023-04-21 12:00:48,873 DEBUG SenderThread:4128 [sender.py:send_request():402] send_request: stop_status
2023-04-21 12:00:48,972 INFO Thread-16 :4128 [dir_watcher.py:_on_file_created():278] file/dir created: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_120047-67c5qu7s\files\wandb-metadata.json
2023-04-21 12:00:48,973 INFO Thread-16 :4128 [dir_watcher.py:_on_file_created():278] file/dir created: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_120047-67c5qu7s\files\requirements.txt
2023-04-21 12:00:48,973 INFO Thread-16 :4128 [dir_watcher.py:_on_file_created():278] file/dir created: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_120047-67c5qu7s\files\wandb-summary.json
2023-04-21 12:00:49,258 DEBUG SenderThread:4128 [sender.py:send():375] send: telemetry
2023-04-21 12:00:49,259 DEBUG SenderThread:4128 [sender.py:send():375] send: config
2023-04-21 12:00:49,259 DEBUG SenderThread:4128 [sender.py:send():375] send: metric
2023-04-21 12:00:49,260 DEBUG SenderThread:4128 [sender.py:send():375] send: telemetry
2023-04-21 12:00:49,260 DEBUG SenderThread:4128 [sender.py:send():375] send: metric
2023-04-21 12:00:49,260 WARNING SenderThread:4128 [sender.py:send_metric():1329] Seen metric with glob (shouldn't happen)
2023-04-21 12:00:49,843 INFO wandb-upload_0:4128 [upload_job.py:push():137] Uploaded file C:\Users\Lenovo\AppData\Local\Temp\tmpanqo2sfqwandb\n8g764s2-wandb-metadata.json
2023-04-21 12:00:49,980 INFO Thread-16 :4128 [dir_watcher.py:_on_file_created():278] file/dir created: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_120047-67c5qu7s\files\output.log
2023-04-21 12:00:50,768 ERROR gpu :4128 [interfaces.py:monitor():144] Failed to sample metric: Not Supported
2023-04-21 12:00:51,996 INFO Thread-16 :4128 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_120047-67c5qu7s\files\output.log
2023-04-21 12:00:52,622 DEBUG HandlerThread:4128 [handler.py:handle_request():144] handle_request: status_report
2023-04-21 12:00:52,818 ERROR gpu :4128 [interfaces.py:monitor():144] Failed to sample metric: Not Supported
2023-04-21 12:00:53,003 INFO Thread-16 :4128 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_120047-67c5qu7s\files\output.log
2023-04-21 12:00:54,880 ERROR gpu :4128 [interfaces.py:monitor():144] Failed to sample metric: Not Supported
2023-04-21 12:00:56,906 ERROR gpu :4128 [interfaces.py:monitor():144] Failed to sample metric: Not Supported
2023-04-21 12:00:57,657 DEBUG HandlerThread:4128 [handler.py:handle_request():144] handle_request: status_report
2023-04-21 12:00:58,953 ERROR gpu :4128 [interfaces.py:monitor():144] Failed to sample metric: Not Supported
2023-04-21 12:01:00,996 ERROR gpu :4128 [interfaces.py:monitor():144] Failed to sample metric: Not Supported
2023-04-21 12:01:02,702 DEBUG HandlerThread:4128 [handler.py:handle_request():144] handle_request: status_report
2023-04-21 12:01:03,044 ERROR gpu :4128 [interfaces.py:monitor():144] Failed to sample metric: Not Supported
2023-04-21 12:01:03,889 DEBUG HandlerThread:4128 [handler.py:handle_request():144] handle_request: stop_status
2023-04-21 12:01:03,890 DEBUG SenderThread:4128 [sender.py:send_request():402] send_request: stop_status
2023-04-21 12:01:05,099 ERROR gpu :4128 [interfaces.py:monitor():144] Failed to sample metric: Not Supported
2023-04-21 12:01:06,122 INFO Thread-16 :4128 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_120047-67c5qu7s\files\output.log
2023-04-21 12:01:07,141 ERROR gpu :4128 [interfaces.py:monitor():144] Failed to sample metric: Not Supported
2023-04-21 12:01:08,682 DEBUG HandlerThread:4128 [handler.py:handle_request():144] handle_request: status_report
2023-04-21 12:01:09,183 ERROR gpu :4128 [interfaces.py:monitor():144] Failed to sample metric: Not Supported
2023-04-21 12:01:11,220 ERROR gpu :4128 [interfaces.py:monitor():144] Failed to sample metric: Not Supported
2023-04-21 12:01:13,273 ERROR gpu :4128 [interfaces.py:monitor():144] Failed to sample metric: Not Supported
2023-04-21 12:01:13,737 DEBUG HandlerThread:4128 [handler.py:handle_request():144] handle_request: status_report
2023-04-21 12:01:15,317 ERROR gpu :4128 [interfaces.py:monitor():144] Failed to sample metric: Not Supported
2023-04-21 12:01:17,365 ERROR gpu :4128 [interfaces.py:monitor():144] Failed to sample metric: Not Supported
2023-04-21 12:01:18,744 DEBUG HandlerThread:4128 [handler.py:handle_request():144] handle_request: status_report
2023-04-21 12:01:18,896 DEBUG HandlerThread:4128 [handler.py:handle_request():144] handle_request: stop_status
2023-04-21 12:01:19,165 DEBUG SenderThread:4128 [sender.py:send_request():402] send_request: stop_status
2023-04-21 12:01:19,421 INFO Thread-16 :4128 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_120047-67c5qu7s\files\config.yaml
2023-04-21 12:01:19,431 INFO Thread-16 :4128 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_120047-67c5qu7s\files\output.log
2023-04-21 12:01:19,477 ERROR gpu :4128 [interfaces.py:monitor():144] Failed to sample metric: Not Supported
2023-04-21 12:01:21,493 ERROR gpu :4128 [interfaces.py:monitor():144] Failed to sample metric: Not Supported
2023-04-21 12:01:23,518 ERROR gpu :4128 [interfaces.py:monitor():144] Failed to sample metric: Not Supported
2023-04-21 12:01:24,459 DEBUG HandlerThread:4128 [handler.py:handle_request():144] handle_request: status_report
2023-04-21 12:01:25,555 ERROR gpu :4128 [interfaces.py:monitor():144] Failed to sample metric: Not Supported
2023-04-21 12:01:27,600 ERROR gpu :4128 [interfaces.py:monitor():144] Failed to sample metric: Not Supported
2023-04-21 12:01:29,496 DEBUG HandlerThread:4128 [handler.py:handle_request():144] handle_request: status_report
2023-04-21 12:01:29,658 ERROR gpu :4128 [interfaces.py:monitor():144] Failed to sample metric: Not Supported
2023-04-21 12:01:31,708 ERROR gpu :4128 [interfaces.py:monitor():144] Failed to sample metric: Not Supported
2023-04-21 12:01:33,846 INFO Thread-16 :4128 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_120047-67c5qu7s\files\output.log
2023-04-21 12:01:33,848 ERROR gpu :4128 [interfaces.py:monitor():144] Failed to sample metric: Not Supported
2023-04-21 12:01:33,908 DEBUG HandlerThread:4128 [handler.py:handle_request():144] handle_request: stop_status
2023-04-21 12:01:33,909 DEBUG SenderThread:4128 [sender.py:send_request():402] send_request: stop_status
2023-04-21 12:01:35,173 DEBUG HandlerThread:4128 [handler.py:handle_request():144] handle_request: status_report
2023-04-21 12:01:35,917 ERROR gpu :4128 [interfaces.py:monitor():144] Failed to sample metric: Not Supported
2023-04-21 12:01:38,051 ERROR gpu :4128 [interfaces.py:monitor():144] Failed to sample metric: Not Supported
2023-04-21 12:01:40,113 ERROR gpu :4128 [interfaces.py:monitor():144] Failed to sample metric: Not Supported
2023-04-21 12:01:40,239 DEBUG HandlerThread:4128 [handler.py:handle_request():144] handle_request: status_report
2023-04-21 12:01:42,175 ERROR gpu :4128 [interfaces.py:monitor():144] Failed to sample metric: Not Supported
2023-04-21 12:01:44,219 ERROR gpu :4128 [interfaces.py:monitor():144] Failed to sample metric: Not Supported
2023-04-21 12:01:45,299 DEBUG HandlerThread:4128 [handler.py:handle_request():144] handle_request: status_report
2023-04-21 12:01:46,269 ERROR gpu :4128 [interfaces.py:monitor():144] Failed to sample metric: Not Supported
2023-04-21 12:01:48,306 INFO Thread-16 :4128 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_120047-67c5qu7s\files\output.log
2023-04-21 12:01:48,318 ERROR gpu :4128 [interfaces.py:monitor():144] Failed to sample metric: Not Supported
2023-04-21 12:01:48,725 DEBUG SystemMonitor:4128 [system_monitor.py:_start():159] Starting system metrics aggregation loop
2023-04-21 12:01:48,726 DEBUG SenderThread:4128 [sender.py:send():375] send: stats
2023-04-21 12:01:48,913 DEBUG HandlerThread:4128 [handler.py:handle_request():144] handle_request: stop_status
2023-04-21 12:01:48,914 DEBUG SenderThread:4128 [sender.py:send_request():402] send_request: stop_status
2023-04-21 12:01:50,456 ERROR gpu :4128 [interfaces.py:monitor():144] Failed to sample metric: Not Supported
2023-04-21 12:01:51,173 DEBUG HandlerThread:4128 [handler.py:handle_request():144] handle_request: status_report
2023-04-21 12:01:52,473 ERROR gpu :4128 [interfaces.py:monitor():144] Failed to sample metric: Not Supported
2023-04-21 12:01:54,491 ERROR gpu :4128 [interfaces.py:monitor():144] Failed to sample metric: Not Supported
2023-04-21 12:01:56,206 DEBUG HandlerThread:4128 [handler.py:handle_request():144] handle_request: status_report
2023-04-21 12:01:56,534 ERROR gpu :4128 [interfaces.py:monitor():144] Failed to sample metric: Not Supported
2023-04-21 12:01:58,584 ERROR gpu :4128 [interfaces.py:monitor():144] Failed to sample metric: Not Supported
2023-04-21 12:02:00,622 ERROR gpu :4128 [interfaces.py:monitor():144] Failed to sample metric: Not Supported
2023-04-21 12:02:01,887 DEBUG HandlerThread:4128 [handler.py:handle_request():144] handle_request: status_report
2023-04-21 12:02:02,679 INFO Thread-16 :4128 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_120047-67c5qu7s\files\output.log
2023-04-21 12:02:02,690 ERROR gpu :4128 [interfaces.py:monitor():144] Failed to sample metric: Not Supported
2023-04-21 12:02:03,917 DEBUG HandlerThread:4128 [handler.py:handle_request():144] handle_request: stop_status
2023-04-21 12:02:03,918 DEBUG SenderThread:4128 [sender.py:send_request():402] send_request: stop_status
2023-04-21 12:02:04,758 ERROR gpu :4128 [interfaces.py:monitor():144] Failed to sample metric: Not Supported
2023-04-21 12:02:06,816 ERROR gpu :4128 [interfaces.py:monitor():144] Failed to sample metric: Not Supported
2023-04-21 12:02:07,205 DEBUG HandlerThread:4128 [handler.py:handle_request():144] handle_request: status_report
2023-04-21 12:02:08,864 ERROR gpu :4128 [interfaces.py:monitor():144] Failed to sample metric: Not Supported
2023-04-21 12:02:10,908 ERROR gpu :4128 [interfaces.py:monitor():144] Failed to sample metric: Not Supported
2023-04-21 12:02:12,250 DEBUG HandlerThread:4128 [handler.py:handle_request():144] handle_request: status_report
2023-04-21 12:02:12,938 ERROR gpu :4128 [interfaces.py:monitor():144] Failed to sample metric: Not Supported
2023-04-21 12:02:15,060 ERROR gpu :4128 [interfaces.py:monitor():144] Failed to sample metric: Not Supported
2023-04-21 12:02:16,064 INFO Thread-16 :4128 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_120047-67c5qu7s\files\output.log
2023-04-21 12:02:17,121 ERROR gpu :4128 [interfaces.py:monitor():144] Failed to sample metric: Not Supported
2023-04-21 12:02:18,135 DEBUG HandlerThread:4128 [handler.py:handle_request():144] handle_request: status_report
2023-04-21 12:02:18,732 DEBUG SenderThread:4128 [sender.py:send():375] send: stats
2023-04-21 12:02:18,935 DEBUG HandlerThread:4128 [handler.py:handle_request():144] handle_request: stop_status
2023-04-21 12:02:18,935 DEBUG SenderThread:4128 [sender.py:send_request():402] send_request: stop_status
2023-04-21 12:02:19,175 ERROR gpu :4128 [interfaces.py:monitor():144] Failed to sample metric: Not Supported
2023-04-21 12:02:21,322 ERROR gpu :4128 [interfaces.py:monitor():144] Failed to sample metric: Not Supported
2023-04-21 12:02:23,216 DEBUG HandlerThread:4128 [handler.py:handle_request():144] handle_request: status_report
2023-04-21 12:02:23,333 ERROR gpu :4128 [interfaces.py:monitor():144] Failed to sample metric: Not Supported
2023-04-21 12:02:25,369 ERROR gpu :4128 [interfaces.py:monitor():144] Failed to sample metric: Not Supported
2023-04-21 12:02:27,439 ERROR gpu :4128 [interfaces.py:monitor():144] Failed to sample metric: Not Supported
2023-04-21 12:02:28,260 DEBUG HandlerThread:4128 [handler.py:handle_request():144] handle_request: status_report
2023-04-21 12:02:29,474 ERROR gpu :4128 [interfaces.py:monitor():144] Failed to sample metric: Not Supported
2023-04-21 12:02:30,477 INFO Thread-16 :4128 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_120047-67c5qu7s\files\output.log
2023-04-21 12:02:31,534 ERROR gpu :4128 [interfaces.py:monitor():144] Failed to sample metric: Not Supported
2023-04-21 12:02:33,452 DEBUG HandlerThread:4128 [handler.py:handle_request():144] handle_request: status_report
2023-04-21 12:02:33,629 ERROR gpu :4128 [interfaces.py:monitor():144] Failed to sample metric: Not Supported
2023-04-21 12:02:33,946 DEBUG HandlerThread:4128 [handler.py:handle_request():144] handle_request: stop_status
2023-04-21 12:02:33,947 DEBUG SenderThread:4128 [sender.py:send_request():402] send_request: stop_status
2023-04-21 12:02:35,767 ERROR gpu :4128 [interfaces.py:monitor():144] Failed to sample metric: Not Supported
2023-04-21 12:02:37,830 ERROR gpu :4128 [interfaces.py:monitor():144] Failed to sample metric: Not Supported
2023-04-21 12:02:39,321 DEBUG HandlerThread:4128 [handler.py:handle_request():144] handle_request: status_report
2023-04-21 12:02:39,874 ERROR gpu :4128 [interfaces.py:monitor():144] Failed to sample metric: Not Supported
2023-04-21 12:02:41,026 DEBUG SenderThread:4128 [sender.py:send():375] send: exit
2023-04-21 12:02:41,026 INFO SenderThread:4128 [sender.py:send_exit():598] handling exit code: 255
2023-04-21 12:02:41,026 INFO SenderThread:4128 [sender.py:send_exit():600] handling runtime: 112
2023-04-21 12:02:41,027 INFO SenderThread:4128 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-04-21 12:02:41,027 INFO SenderThread:4128 [sender.py:send_exit():606] send defer
2023-04-21 12:02:41,027 DEBUG HandlerThread:4128 [handler.py:handle_request():144] handle_request: defer
2023-04-21 12:02:41,028 INFO HandlerThread:4128 [handler.py:handle_request_defer():170] handle defer: 0
2023-04-21 12:02:41,028 DEBUG SenderThread:4128 [sender.py:send_request():402] send_request: defer
2023-04-21 12:02:41,028 INFO SenderThread:4128 [sender.py:send_request_defer():622] handle sender defer: 0
2023-04-21 12:02:41,028 INFO SenderThread:4128 [sender.py:transition_state():626] send defer: 1
2023-04-21 12:02:41,028 DEBUG HandlerThread:4128 [handler.py:handle_request():144] handle_request: defer
2023-04-21 12:02:41,029 INFO HandlerThread:4128 [handler.py:handle_request_defer():170] handle defer: 1
2023-04-21 12:02:41,029 DEBUG SenderThread:4128 [sender.py:send_request():402] send_request: defer
2023-04-21 12:02:41,029 INFO SenderThread:4128 [sender.py:send_request_defer():622] handle sender defer: 1
2023-04-21 12:02:41,029 INFO SenderThread:4128 [sender.py:transition_state():626] send defer: 2
2023-04-21 12:02:41,029 DEBUG HandlerThread:4128 [handler.py:handle_request():144] handle_request: defer
2023-04-21 12:02:41,030 INFO HandlerThread:4128 [handler.py:handle_request_defer():170] handle defer: 2
2023-04-21 12:02:41,030 INFO HandlerThread:4128 [system_monitor.py:finish():190] Stopping system monitor
2023-04-21 12:02:41,030 DEBUG SystemMonitor:4128 [system_monitor.py:_start():166] Finished system metrics aggregation loop
2023-04-21 12:02:41,030 INFO HandlerThread:4128 [interfaces.py:finish():202] Joined cpu monitor
2023-04-21 12:02:41,030 DEBUG SystemMonitor:4128 [system_monitor.py:_start():170] Publishing last batch of metrics
2023-04-21 12:02:41,043 INFO HandlerThread:4128 [interfaces.py:finish():202] Joined disk monitor
2023-04-21 12:02:41,103 INFO HandlerThread:4128 [interfaces.py:finish():202] Joined gpu monitor
2023-04-21 12:02:41,104 INFO HandlerThread:4128 [interfaces.py:finish():202] Joined memory monitor
2023-04-21 12:02:41,104 INFO HandlerThread:4128 [interfaces.py:finish():202] Joined network monitor
2023-04-21 12:02:41,104 DEBUG SenderThread:4128 [sender.py:send_request():402] send_request: defer
2023-04-21 12:02:41,104 INFO SenderThread:4128 [sender.py:send_request_defer():622] handle sender defer: 2
2023-04-21 12:02:41,105 INFO SenderThread:4128 [sender.py:transition_state():626] send defer: 3
2023-04-21 12:02:41,105 DEBUG SenderThread:4128 [sender.py:send():375] send: stats
2023-04-21 12:02:41,105 DEBUG HandlerThread:4128 [handler.py:handle_request():144] handle_request: defer
2023-04-21 12:02:41,106 INFO HandlerThread:4128 [handler.py:handle_request_defer():170] handle defer: 3
2023-04-21 12:02:41,106 DEBUG SenderThread:4128 [sender.py:send_request():402] send_request: defer
2023-04-21 12:02:41,106 INFO SenderThread:4128 [sender.py:send_request_defer():622] handle sender defer: 3
2023-04-21 12:02:41,106 INFO SenderThread:4128 [sender.py:transition_state():626] send defer: 4
2023-04-21 12:02:41,107 DEBUG HandlerThread:4128 [handler.py:handle_request():144] handle_request: defer
2023-04-21 12:02:41,107 INFO HandlerThread:4128 [handler.py:handle_request_defer():170] handle defer: 4
2023-04-21 12:02:41,107 DEBUG SenderThread:4128 [sender.py:send_request():402] send_request: defer
2023-04-21 12:02:41,108 INFO SenderThread:4128 [sender.py:send_request_defer():622] handle sender defer: 4
2023-04-21 12:02:41,108 INFO SenderThread:4128 [sender.py:transition_state():626] send defer: 5
2023-04-21 12:02:41,108 DEBUG HandlerThread:4128 [handler.py:handle_request():144] handle_request: defer
2023-04-21 12:02:41,108 INFO HandlerThread:4128 [handler.py:handle_request_defer():170] handle defer: 5
2023-04-21 12:02:41,109 DEBUG SenderThread:4128 [sender.py:send():375] send: summary
2023-04-21 12:02:41,110 INFO SenderThread:4128 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end
2023-04-21 12:02:41,110 DEBUG SenderThread:4128 [sender.py:send_request():402] send_request: defer
2023-04-21 12:02:41,110 INFO SenderThread:4128 [sender.py:send_request_defer():622] handle sender defer: 5
2023-04-21 12:02:41,110 INFO SenderThread:4128 [sender.py:transition_state():626] send defer: 6
2023-04-21 12:02:41,111 DEBUG HandlerThread:4128 [handler.py:handle_request():144] handle_request: defer
2023-04-21 12:02:41,111 INFO HandlerThread:4128 [handler.py:handle_request_defer():170] handle defer: 6
2023-04-21 12:02:41,111 DEBUG SenderThread:4128 [sender.py:send_request():402] send_request: defer
2023-04-21 12:02:41,111 INFO SenderThread:4128 [sender.py:send_request_defer():622] handle sender defer: 6
2023-04-21 12:02:41,111 INFO SenderThread:4128 [sender.py:transition_state():626] send defer: 7
2023-04-21 12:02:41,111 DEBUG HandlerThread:4128 [handler.py:handle_request():144] handle_request: status_report
2023-04-21 12:02:41,112 DEBUG HandlerThread:4128 [handler.py:handle_request():144] handle_request: defer
2023-04-21 12:02:41,112 INFO HandlerThread:4128 [handler.py:handle_request_defer():170] handle defer: 7
2023-04-21 12:02:41,112 DEBUG SenderThread:4128 [sender.py:send_request():402] send_request: defer
2023-04-21 12:02:41,112 INFO SenderThread:4128 [sender.py:send_request_defer():622] handle sender defer: 7
2023-04-21 12:02:41,656 INFO Thread-16 :4128 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_120047-67c5qu7s\files\wandb-summary.json
2023-04-21 12:02:42,109 DEBUG HandlerThread:4128 [handler.py:handle_request():144] handle_request: poll_exit
2023-04-21 12:02:42,659 INFO Thread-16 :4128 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_120047-67c5qu7s\files\output.log
2023-04-21 12:02:43,229 INFO SenderThread:4128 [sender.py:transition_state():626] send defer: 8
2023-04-21 12:02:43,229 DEBUG SenderThread:4128 [sender.py:send_request():402] send_request: poll_exit
2023-04-21 12:02:43,229 DEBUG HandlerThread:4128 [handler.py:handle_request():144] handle_request: defer
2023-04-21 12:02:43,229 INFO HandlerThread:4128 [handler.py:handle_request_defer():170] handle defer: 8
2023-04-21 12:02:43,229 DEBUG SenderThread:4128 [sender.py:send_request():402] send_request: defer
2023-04-21 12:02:43,230 INFO SenderThread:4128 [sender.py:send_request_defer():622] handle sender defer: 8
2023-04-21 12:02:43,255 INFO SenderThread:4128 [sender.py:transition_state():626] send defer: 9
2023-04-21 12:02:43,255 DEBUG HandlerThread:4128 [handler.py:handle_request():144] handle_request: defer
2023-04-21 12:02:43,255 INFO HandlerThread:4128 [handler.py:handle_request_defer():170] handle defer: 9
2023-04-21 12:02:43,255 DEBUG SenderThread:4128 [sender.py:send():375] send: artifact
2023-04-21 12:02:43,668 INFO Thread-16 :4128 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_120047-67c5qu7s\files\output.log
2023-04-21 12:02:44,590 INFO SenderThread:4128 [sender.py:send_artifact():1474] sent artifact job-https___github.com_THUDM_ChatGLM-6B_ptuning_main.py - {'id': 'QXJ0aWZhY3Q6NDI5MjY1Nzg0', 'digest': '53476254d59d98151858729a5d45f45c', 'state': 'COMMITTED', 'aliases': [{'artifactCollectionName': 'job-https___github.com_THUDM_ChatGLM-6B_ptuning_main.py', 'alias': 'latest'}, {'artifactCollectionName': 'job-https___github.com_THUDM_ChatGLM-6B_ptuning_main.py', 'alias': 'v2'}], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjYzNDcyMzc3', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6NDI5MjY1Nzg0', 'versionIndex': 2}}, 'version': 'v2'}
2023-04-21 12:02:44,590 DEBUG SenderThread:4128 [sender.py:send_request():402] send_request: defer
2023-04-21 12:02:44,590 INFO SenderThread:4128 [sender.py:send_request_defer():622] handle sender defer: 9
2023-04-21 12:02:44,590 INFO SenderThread:4128 [dir_watcher.py:finish():365] shutting down directory watcher
2023-04-21 12:02:44,672 INFO SenderThread:4128 [dir_watcher.py:finish():395] scan: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_120047-67c5qu7s\files
2023-04-21 12:02:44,672 INFO SenderThread:4128 [dir_watcher.py:finish():409] scan save: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_120047-67c5qu7s\files\config.yaml config.yaml
2023-04-21 12:02:44,673 INFO SenderThread:4128 [dir_watcher.py:finish():409] scan save: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_120047-67c5qu7s\files\output.log output.log
2023-04-21 12:02:44,675 INFO SenderThread:4128 [dir_watcher.py:finish():409] scan save: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_120047-67c5qu7s\files\requirements.txt requirements.txt
2023-04-21 12:02:44,677 INFO SenderThread:4128 [dir_watcher.py:finish():409] scan save: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_120047-67c5qu7s\files\wandb-metadata.json wandb-metadata.json
2023-04-21 12:02:44,678 INFO SenderThread:4128 [dir_watcher.py:finish():409] scan save: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_120047-67c5qu7s\files\wandb-summary.json wandb-summary.json
2023-04-21 12:02:44,681 INFO SenderThread:4128 [sender.py:transition_state():626] send defer: 10
2023-04-21 12:02:44,681 DEBUG HandlerThread:4128 [handler.py:handle_request():144] handle_request: defer
2023-04-21 12:02:44,681 INFO HandlerThread:4128 [handler.py:handle_request_defer():170] handle defer: 10
2023-04-21 12:02:44,686 DEBUG SenderThread:4128 [sender.py:send_request():402] send_request: defer
2023-04-21 12:02:44,686 INFO SenderThread:4128 [sender.py:send_request_defer():622] handle sender defer: 10
2023-04-21 12:02:44,686 INFO SenderThread:4128 [file_pusher.py:finish():168] shutting down file pusher
2023-04-21 12:02:45,331 INFO wandb-upload_1:4128 [upload_job.py:push():137] Uploaded file E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_120047-67c5qu7s\files\output.log
2023-04-21 12:02:45,742 INFO wandb-upload_0:4128 [upload_job.py:push():137] Uploaded file E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_120047-67c5qu7s\files\config.yaml
2023-04-21 12:02:45,763 INFO wandb-upload_3:4128 [upload_job.py:push():137] Uploaded file E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_120047-67c5qu7s\files\wandb-summary.json
2023-04-21 12:02:45,781 INFO wandb-upload_2:4128 [upload_job.py:push():137] Uploaded file E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_120047-67c5qu7s\files\requirements.txt
2023-04-21 12:02:45,981 INFO Thread-15 :4128 [sender.py:transition_state():626] send defer: 11
2023-04-21 12:02:45,981 DEBUG HandlerThread:4128 [handler.py:handle_request():144] handle_request: defer
2023-04-21 12:02:45,981 INFO HandlerThread:4128 [handler.py:handle_request_defer():170] handle defer: 11
2023-04-21 12:02:45,981 DEBUG SenderThread:4128 [sender.py:send_request():402] send_request: defer
2023-04-21 12:02:45,981 INFO SenderThread:4128 [sender.py:send_request_defer():622] handle sender defer: 11
2023-04-21 12:02:45,981 INFO SenderThread:4128 [file_pusher.py:join():173] waiting for file pusher
2023-04-21 12:02:45,982 INFO SenderThread:4128 [sender.py:transition_state():626] send defer: 12
2023-04-21 12:02:45,982 DEBUG HandlerThread:4128 [handler.py:handle_request():144] handle_request: defer
2023-04-21 12:02:45,982 INFO HandlerThread:4128 [handler.py:handle_request_defer():170] handle defer: 12
2023-04-21 12:02:45,982 DEBUG SenderThread:4128 [sender.py:send_request():402] send_request: defer
2023-04-21 12:02:45,982 INFO SenderThread:4128 [sender.py:send_request_defer():622] handle sender defer: 12
2023-04-21 12:02:47,138 DEBUG HandlerThread:4128 [handler.py:handle_request():144] handle_request: keepalive
2023-04-21 12:02:47,578 INFO SenderThread:4128 [sender.py:transition_state():626] send defer: 13
2023-04-21 12:02:47,578 DEBUG HandlerThread:4128 [handler.py:handle_request():144] handle_request: defer
2023-04-21 12:02:47,578 INFO HandlerThread:4128 [handler.py:handle_request_defer():170] handle defer: 13
2023-04-21 12:02:47,578 DEBUG SenderThread:4128 [sender.py:send_request():402] send_request: defer
2023-04-21 12:02:47,578 INFO SenderThread:4128 [sender.py:send_request_defer():622] handle sender defer: 13
2023-04-21 12:02:47,578 INFO SenderThread:4128 [sender.py:transition_state():626] send defer: 14
2023-04-21 12:02:47,579 DEBUG HandlerThread:4128 [handler.py:handle_request():144] handle_request: defer
2023-04-21 12:02:47,579 DEBUG SenderThread:4128 [sender.py:send():375] send: final
2023-04-21 12:02:47,579 INFO HandlerThread:4128 [handler.py:handle_request_defer():170] handle defer: 14
2023-04-21 12:02:47,579 DEBUG SenderThread:4128 [sender.py:send():375] send: footer
2023-04-21 12:02:47,579 DEBUG SenderThread:4128 [sender.py:send_request():402] send_request: defer
2023-04-21 12:02:47,579 INFO SenderThread:4128 [sender.py:send_request_defer():622] handle sender defer: 14
2023-04-21 12:02:47,580 DEBUG HandlerThread:4128 [handler.py:handle_request():144] handle_request: poll_exit
2023-04-21 12:02:47,580 DEBUG HandlerThread:4128 [handler.py:handle_request():144] handle_request: server_info
2023-04-21 12:02:47,580 DEBUG HandlerThread:4128 [handler.py:handle_request():144] handle_request: get_summary
2023-04-21 12:02:47,580 DEBUG SenderThread:4128 [sender.py:send_request():402] send_request: poll_exit
2023-04-21 12:02:47,580 DEBUG HandlerThread:4128 [handler.py:handle_request():144] handle_request: sampled_history
2023-04-21 12:02:47,580 DEBUG SenderThread:4128 [sender.py:send_request():402] send_request: server_info
2023-04-21 12:02:47,827 INFO MainThread:4128 [wandb_run.py:_footer_history_summary_info():3476] rendering history
2023-04-21 12:02:47,827 INFO MainThread:4128 [wandb_run.py:_footer_history_summary_info():3508] rendering summary
2023-04-21 12:02:47,827 INFO MainThread:4128 [wandb_run.py:_footer_sync_info():3434] logging synced files
2023-04-21 12:02:47,829 DEBUG HandlerThread:4128 [handler.py:handle_request():144] handle_request: shutdown
2023-04-21 12:02:47,829 INFO HandlerThread:4128 [handler.py:finish():845] shutting down handler
2023-04-21 12:02:48,594 INFO WriterThread:4128 [datastore.py:close():298] close: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230421_120047-67c5qu7s\run-67c5qu7s.wandb
2023-04-21 12:02:48,829 INFO SenderThread:4128 [sender.py:finish():1550] shutting down sender
2023-04-21 12:02:48,829 INFO SenderThread:4128 [file_pusher.py:finish():168] shutting down file pusher
2023-04-21 12:02:48,829 INFO SenderThread:4128 [file_pusher.py:join():173] waiting for file pusher