2023-04-23 02:25:03,135 INFO StreamThr :34348 [internal.py:wandb_internal():86] W&B internal server running at pid: 34348, started at: 2023-04-23 02:25:03.134408 2023-04-23 02:25:03,137 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status 2023-04-23 02:25:03,152 INFO WriterThread:34348 [datastore.py:open_for_write():85] open: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\run-g3y4djvd.wandb 2023-04-23 02:25:03,152 DEBUG SenderThread:34348 [sender.py:send():375] send: header 2023-04-23 02:25:03,216 DEBUG SenderThread:34348 [sender.py:send():375] send: run 2023-04-23 02:25:03,919 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: check_version 2023-04-23 02:25:03,920 INFO SenderThread:34348 [dir_watcher.py:__init__():219] watching files in: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files 2023-04-23 02:25:03,920 INFO SenderThread:34348 [sender.py:_start_run_threads():1124] run started: g3y4djvd with start time 1682187903.150408 2023-04-23 02:25:03,920 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 02:25:03,921 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 02:25:03,921 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: check_version 2023-04-23 02:25:04,568 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: run_start 2023-04-23 02:25:04,621 DEBUG HandlerThread:34348 [system_info.py:__init__():31] System info init 2023-04-23 02:25:04,621 DEBUG HandlerThread:34348 [system_info.py:__init__():46] System info init done 2023-04-23 02:25:04,621 INFO HandlerThread:34348 [system_monitor.py:start():181] Starting system monitor 2023-04-23 02:25:04,622 INFO SystemMonitor:34348 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-04-23 02:25:04,622 INFO HandlerThread:34348 [system_monitor.py:probe():201] Collecting system info 2023-04-23 02:25:04,631 INFO SystemMonitor:34348 [interfaces.py:start():190] Started cpu monitoring 2023-04-23 02:25:04,631 INFO SystemMonitor:34348 [interfaces.py:start():190] Started disk monitoring 2023-04-23 02:25:04,632 INFO SystemMonitor:34348 [interfaces.py:start():190] Started gpu monitoring 2023-04-23 02:25:04,645 INFO SystemMonitor:34348 [interfaces.py:start():190] Started memory monitoring 2023-04-23 02:25:04,664 INFO SystemMonitor:34348 [interfaces.py:start():190] Started network monitoring 2023-04-23 02:25:04,689 DEBUG HandlerThread:34348 [system_info.py:probe():195] Probing system 2023-04-23 02:25:04,692 DEBUG HandlerThread:34348 [system_info.py:_probe_git():180] Probing git 2023-04-23 02:25:04,693 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:25:04,790 DEBUG HandlerThread:34348 [system_info.py:_probe_git():188] Probing git done 2023-04-23 02:25:04,790 DEBUG HandlerThread:34348 [system_info.py:probe():240] Probing system done 2023-04-23 02:25:04,790 DEBUG HandlerThread:34348 [system_monitor.py:probe():210] {'os': 'Windows-10-10.0.19041-SP0', 'python': '3.8.10', 'heartbeatAt': '2023-04-22T18:25:04.689821', 'startedAt': '2023-04-22T18:25:03.115711', 'docker': None, 'cuda': None, 'args': ('--do_train', '--train_file', '.\\datasets\\Zettels\\train.json', '--validation_file', '.\\datasets\\Zettels\\dev.json', '--prompt_column', 'content', '--response_column', 'summary', '--overwrite_cache', '--model_name_or_path', '..\\models\\chatglm-6b-int4', '--output_dir', 'output\\adgen-chatglm-6b-pt-128-2e-2', '--overwrite_output_dir', '--max_source_length', '64', '--max_target_length', '64', '--per_device_train_batch_size', '1', '--per_device_eval_batch_size', '1', '--gradient_accumulation_steps', '16', '--predict_with_generate', '--max_steps', '1000', '--logging_steps', '10', '--save_steps', '10', '--learning_rate', '2e-2', '--pre_seq_len', '128', '--quantization_bit', '4'), 'state': 'running', 'program': 'main.py', 'codePath': 'ptuning\\main.py', 'git': {'remote': 'https://github.com/THUDM/ChatGLM-6B', 'commit': '01e6313abf4122d789d6e68128856af52847b355'}, 'cpu_count': 6, 'cpu_count_logical': 12, 'cpu_freq': {'current': 2592.0, 'min': 0.0, 'max': 2592.0}, 'cpu_freq_per_core': [{'current': 2592.0, 'min': 0.0, 'max': 2592.0}], 'disk': {'total': 500.32030868530273, 'used': 240.75128936767578}, 'gpu': 'NVIDIA GeForce RTX 2060', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 2060', 'memory_total': 6442450944}], 'memory': {'total': 63.87089538574219}} 2023-04-23 02:25:04,791 INFO HandlerThread:34348 [system_monitor.py:probe():211] Finished collecting system info 2023-04-23 02:25:04,791 INFO HandlerThread:34348 [system_monitor.py:probe():214] Publishing system info 2023-04-23 02:25:04,791 DEBUG HandlerThread:34348 [system_info.py:_save_pip():51] Saving list of pip packages installed into the current environment 2023-04-23 02:25:04,792 DEBUG HandlerThread:34348 [system_info.py:_save_pip():67] Saving pip packages done 2023-04-23 02:25:04,793 INFO HandlerThread:34348 [system_monitor.py:probe():216] Finished publishing system info 2023-04-23 02:25:04,807 DEBUG SenderThread:34348 [sender.py:send():375] send: files 2023-04-23 02:25:04,807 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-metadata.json with policy now 2023-04-23 02:25:04,820 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:25:04,820 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:25:04,936 INFO Thread-16 :34348 [dir_watcher.py:_on_file_created():278] file/dir created: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\requirements.txt 2023-04-23 02:25:04,936 INFO Thread-16 :34348 [dir_watcher.py:_on_file_created():278] file/dir created: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-metadata.json 2023-04-23 02:25:04,936 INFO Thread-16 :34348 [dir_watcher.py:_on_file_created():278] file/dir created: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 02:25:05,207 DEBUG SenderThread:34348 [sender.py:send():375] send: telemetry 2023-04-23 02:25:05,207 DEBUG SenderThread:34348 [sender.py:send():375] send: config 2023-04-23 02:25:05,208 DEBUG SenderThread:34348 [sender.py:send():375] send: metric 2023-04-23 02:25:05,208 DEBUG SenderThread:34348 [sender.py:send():375] send: telemetry 2023-04-23 02:25:05,208 DEBUG SenderThread:34348 [sender.py:send():375] send: metric 2023-04-23 02:25:05,208 WARNING SenderThread:34348 [sender.py:send_metric():1329] Seen metric with glob (shouldn't happen) 2023-04-23 02:25:05,938 INFO Thread-16 :34348 [dir_watcher.py:_on_file_created():278] file/dir created: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:25:06,740 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:25:07,959 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:25:08,301 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:25:08,798 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:25:08,973 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:25:10,847 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:25:11,682 INFO wandb-upload_0:34348 [upload_job.py:push():137] Uploaded file C:\Users\Lenovo\AppData\Local\Temp\tmpsvd1s5c8wandb\jrb6svnl-wandb-metadata.json 2023-04-23 02:25:12,896 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:25:13,327 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:25:14,928 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:25:16,976 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:25:18,371 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:25:19,027 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:25:19,834 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:25:19,835 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:25:21,071 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:25:23,123 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:25:23,709 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:25:25,157 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:25:26,160 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:25:27,213 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:25:28,746 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:25:29,252 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:25:31,309 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:25:33,350 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:25:33,791 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:25:34,852 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:25:34,852 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:25:35,403 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\config.yaml 2023-04-23 02:25:35,480 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:25:37,501 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:25:39,523 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:25:40,147 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:25:41,522 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:25:41,547 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:25:43,569 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:25:45,614 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:25:45,661 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:25:47,659 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:25:49,714 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:25:49,859 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:25:49,859 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:25:51,102 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:25:51,748 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:25:53,790 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:25:55,847 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:25:56,154 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:25:57,870 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:25:57,878 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:25:59,924 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:26:01,968 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:26:02,058 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:26:04,017 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:26:04,691 DEBUG SystemMonitor:34348 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-04-23 02:26:04,692 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 02:26:04,863 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:26:04,864 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:26:06,230 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:26:07,151 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:26:08,252 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:26:10,281 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:26:12,244 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:26:12,321 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:26:14,368 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:26:16,406 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:26:16,415 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:26:17,775 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:26:18,451 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:26:19,866 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:26:19,866 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:26:20,498 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:26:22,544 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:26:23,120 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:26:24,587 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:26:26,635 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:26:28,160 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:26:28,677 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:26:30,750 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:26:31,752 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:26:32,800 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:26:33,935 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:26:34,700 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 02:26:34,840 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:26:34,885 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:26:34,886 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:26:36,958 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:26:38,969 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:26:39,178 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:26:40,994 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:26:43,017 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:26:44,238 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:26:45,081 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:26:47,158 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:26:48,154 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:26:49,213 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:26:49,856 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:26:49,902 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:26:49,902 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:26:51,266 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:26:53,311 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:26:55,175 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:26:55,345 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:26:57,393 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:26:59,446 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:27:00,225 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:27:01,492 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:27:03,540 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:27:04,706 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 02:27:04,911 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:27:04,912 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:27:05,584 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:27:07,264 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:27:07,719 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:27:09,749 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:27:10,728 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:27:11,804 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:27:12,917 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:27:13,833 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:27:15,871 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:27:17,923 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:27:17,964 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:27:19,960 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:27:19,960 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:27:19,973 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:27:22,004 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:27:23,717 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:27:24,038 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:27:24,047 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:27:26,104 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:27:28,154 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:27:28,745 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:27:30,198 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:27:32,250 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:27:33,772 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:27:34,290 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:27:34,721 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 02:27:34,955 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:27:34,955 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:27:36,334 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:27:38,454 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:27:39,230 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:27:40,421 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:27:40,472 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:27:42,484 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:27:44,256 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:27:44,535 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:27:46,568 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:27:48,619 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:27:49,314 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:27:49,971 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:27:49,971 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:27:50,667 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:27:52,705 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:27:54,762 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:27:55,266 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:27:55,475 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 02:27:55,477 DEBUG SenderThread:34348 [sender.py:send():375] send: metric 2023-04-23 02:27:55,477 DEBUG SenderThread:34348 [sender.py:send():375] send: metric 2023-04-23 02:27:55,477 DEBUG SenderThread:34348 [sender.py:send():375] send: metric 2023-04-23 02:27:55,478 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 02:27:55,478 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 02:27:55,479 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 02:27:55,766 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 02:27:56,799 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:27:56,802 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:27:57,815 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:27:58,848 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:28:00,750 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:28:00,905 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:28:02,969 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:28:04,728 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 02:28:05,012 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:28:05,013 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:28:05,024 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:28:06,267 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:28:07,083 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:28:09,213 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:28:11,227 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:28:11,306 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:28:13,248 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:28:14,237 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:28:15,281 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:28:17,233 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:28:17,324 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:28:18,320 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\config.yaml 2023-04-23 02:28:19,378 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:28:19,997 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:28:19,997 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:28:21,420 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:28:23,278 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:28:23,475 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:28:25,511 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:28:27,553 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:28:28,327 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:28:29,598 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:28:31,654 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:28:32,650 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:28:33,694 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:28:34,100 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:28:34,736 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 02:28:35,008 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:28:35,009 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:28:35,733 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:28:37,780 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:28:39,291 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:28:39,893 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:28:41,910 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:28:43,930 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:28:44,336 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:28:45,976 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:28:48,021 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:28:49,008 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:28:49,919 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:28:50,058 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:28:50,058 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:28:50,078 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:28:52,143 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:28:54,182 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:28:55,396 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:28:56,469 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:28:58,523 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:29:00,470 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:29:00,580 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:29:02,651 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:29:04,711 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:29:04,748 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 02:29:05,014 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:29:05,015 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:29:06,239 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:29:06,811 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:29:08,566 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:29:08,848 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:29:10,967 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:29:11,296 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:29:12,987 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:29:15,008 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:29:16,345 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:29:17,032 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:29:19,065 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:29:20,021 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:29:20,022 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:29:20,730 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:29:21,113 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:29:22,283 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:29:23,160 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:29:25,199 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:29:27,249 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:29:27,343 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:29:29,293 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:29:31,351 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:29:32,874 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:29:33,257 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:29:33,405 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:29:34,755 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 02:29:35,033 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:29:35,034 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:29:35,451 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:29:37,499 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:29:38,305 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:29:39,545 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:29:41,672 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:29:43,350 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:29:43,693 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:29:45,714 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:29:47,034 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:29:47,743 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:29:49,276 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:29:49,793 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:29:50,044 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:29:50,045 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:29:51,844 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:29:53,888 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:29:54,319 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:29:55,930 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:29:57,968 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:29:59,865 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:30:00,010 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:30:01,219 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:30:02,061 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:30:04,099 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:30:04,758 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 02:30:04,965 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:30:05,040 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:30:05,040 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:30:06,141 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:30:08,195 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:30:10,240 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:30:10,350 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:30:12,403 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:30:13,375 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:30:14,416 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:30:15,780 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:30:16,451 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:30:18,493 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:30:20,041 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:30:20,041 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:30:20,539 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:30:21,292 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:30:22,584 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:30:24,475 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 02:30:24,476 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 02:30:24,477 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 02:30:24,478 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 02:30:24,629 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:30:24,629 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 02:30:24,642 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:30:26,680 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:30:26,691 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:30:26,729 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:30:28,735 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:30:30,793 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:30:31,781 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:30:32,834 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:30:34,764 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 02:30:34,873 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:30:35,060 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:30:35,061 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:30:36,929 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:30:37,312 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:30:38,975 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:30:38,977 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:30:41,021 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:30:42,741 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:30:43,134 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:30:45,150 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:30:47,165 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:30:47,792 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:30:49,202 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:30:50,054 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:30:50,055 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:30:51,229 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:30:51,240 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:30:53,281 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:30:53,449 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:30:55,324 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:30:57,389 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:30:58,488 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:30:59,419 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:31:01,467 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:31:03,527 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:31:04,215 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:31:04,766 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 02:31:05,064 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:31:05,064 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:31:05,555 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:31:05,564 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:31:07,609 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:31:09,339 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:31:09,654 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:31:11,705 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:31:13,815 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:31:14,375 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:31:15,835 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:31:17,810 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:31:17,848 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:31:19,867 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:31:20,015 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:31:20,063 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:31:20,063 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:31:21,911 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:31:23,953 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:31:25,351 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:31:25,992 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:31:28,040 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:31:30,080 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:31:30,841 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:31:31,088 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:31:32,141 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:31:34,182 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:31:34,782 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 02:31:35,065 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:31:35,066 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:31:36,228 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:31:36,330 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:31:38,261 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:31:40,315 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:31:41,389 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:31:42,357 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:31:43,368 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:31:44,470 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:31:46,485 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:31:46,660 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:31:48,512 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:31:50,077 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:31:50,077 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:31:50,547 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:31:52,353 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:31:52,595 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:31:54,643 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:31:55,650 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:31:56,682 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:31:57,474 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:31:58,723 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:32:00,786 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:32:02,522 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:32:02,833 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:32:04,790 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 02:32:04,882 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:32:05,089 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:32:05,089 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:32:06,918 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:32:08,169 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:32:08,963 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:32:09,968 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:32:11,024 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:32:13,097 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:32:13,214 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:32:15,201 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:32:17,223 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:32:18,251 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:32:19,238 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:32:20,099 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:32:20,099 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:32:21,269 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:32:21,278 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:32:23,358 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:32:23,365 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:32:25,411 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:32:27,457 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:32:28,462 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:32:29,516 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:32:31,551 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:32:32,966 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 02:32:32,968 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 02:32:32,968 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 02:32:32,969 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 02:32:33,590 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 02:32:33,607 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:32:34,197 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:32:34,603 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:32:34,802 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 02:32:35,100 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:32:35,100 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:32:35,636 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:32:35,645 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:32:37,694 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:32:39,375 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:32:39,727 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:32:41,773 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:32:43,805 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:32:44,434 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:32:45,919 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:32:47,889 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:32:47,939 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:32:49,949 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:32:50,067 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:32:50,113 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:32:50,113 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:32:52,001 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:32:54,052 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:32:55,414 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:32:56,096 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:32:58,147 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:33:00,183 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:33:00,191 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:33:01,261 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:33:02,230 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:33:04,267 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:33:04,818 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 02:33:05,117 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:33:05,117 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:33:06,312 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:33:06,377 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:33:08,348 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:33:10,393 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:33:11,416 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:33:12,481 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:33:13,443 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:33:14,507 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:33:16,623 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:33:17,140 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:33:18,644 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:33:20,125 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:33:20,125 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:33:20,676 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:33:22,384 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:33:22,735 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:33:24,763 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:33:25,771 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:33:26,810 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:33:28,091 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:33:28,862 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:33:30,931 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:33:32,972 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:33:33,113 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:33:34,832 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 02:33:35,021 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:33:35,127 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:33:35,128 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:33:37,065 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:33:38,933 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:33:39,116 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:33:40,120 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:33:41,163 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:33:43,197 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:33:43,953 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:33:45,237 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:33:47,350 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:33:48,991 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:33:49,369 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:33:50,126 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:33:50,126 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:33:51,396 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:33:52,359 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:33:53,420 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:33:54,865 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:33:55,447 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:33:57,495 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:33:59,532 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:33:59,913 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:34:01,573 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:34:03,613 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:34:04,844 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 02:34:05,098 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:34:05,144 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:34:05,144 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:34:05,661 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:34:06,657 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:34:07,704 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:34:09,752 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:34:10,433 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:34:11,798 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:34:13,848 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:34:15,475 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:34:15,893 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:34:17,958 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:34:17,992 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:34:20,010 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:34:20,153 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:34:20,154 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:34:21,410 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:34:22,033 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:34:24,066 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:34:26,114 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:34:26,451 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:34:28,161 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:34:30,205 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:34:31,636 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:34:32,242 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:34:32,257 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:34:34,306 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:34:34,851 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 02:34:35,149 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:34:35,150 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:34:36,351 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:34:37,404 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:34:38,383 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:34:40,427 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:34:42,459 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:34:42,464 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:34:42,479 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 02:34:42,481 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 02:34:42,481 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 02:34:42,482 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 02:34:43,454 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:34:43,454 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 02:34:44,500 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:34:44,508 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:34:46,553 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:34:47,739 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:34:48,662 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:34:50,169 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:34:50,169 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:34:50,677 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:34:52,693 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:34:53,431 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:34:54,742 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:34:56,783 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:34:58,815 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:34:58,825 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:34:58,952 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:35:00,893 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:35:02,945 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:35:03,981 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:35:04,866 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 02:35:04,986 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:35:05,164 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:35:05,164 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:35:07,031 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:35:09,078 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:35:09,765 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:35:10,082 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:35:11,125 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:35:13,173 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:35:14,814 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:35:15,221 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:35:17,251 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:35:19,363 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:35:19,852 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:35:20,169 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:35:20,170 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:35:21,380 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:35:22,360 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:35:23,389 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:35:25,419 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:35:25,669 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:35:27,460 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:35:29,519 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:35:30,721 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:35:31,566 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:35:33,596 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:35:34,878 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 02:35:35,192 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:35:35,193 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:35:35,643 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:35:36,454 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:35:36,646 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:35:37,687 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:35:39,733 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:35:41,513 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:35:41,779 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:35:43,807 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:35:45,853 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:35:46,547 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:35:47,894 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:35:48,891 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:35:50,005 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:35:50,210 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:35:50,211 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:35:52,039 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:35:52,491 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:35:54,061 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:35:56,103 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:35:57,542 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:35:58,137 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:36:00,179 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:36:02,212 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:36:02,220 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:36:03,237 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:36:04,263 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:36:04,885 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 02:36:05,216 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:36:05,216 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:36:06,309 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:36:08,353 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:36:08,484 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:36:10,400 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:36:12,450 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:36:14,010 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:36:14,488 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:36:14,489 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:36:16,539 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:36:18,570 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:36:19,035 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:36:20,218 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:36:20,219 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:36:20,686 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:36:22,711 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:36:24,514 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:36:24,728 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:36:26,738 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:36:26,749 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:36:28,783 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:36:29,843 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:36:30,849 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:36:32,889 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:36:34,928 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:36:34,928 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 02:36:34,939 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:36:35,230 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:36:35,230 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:36:36,998 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:36:39,036 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:36:40,705 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:36:41,084 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:36:41,093 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:36:43,145 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:36:45,205 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:36:45,722 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:36:47,251 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:36:49,296 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:36:50,246 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:36:50,247 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:36:51,405 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:36:51,497 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:36:51,613 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 02:36:51,614 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 02:36:51,614 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 02:36:51,615 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 02:36:52,377 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:36:52,377 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 02:36:53,423 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:36:54,388 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:36:55,448 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:36:56,896 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:36:57,475 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:36:59,523 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:37:01,564 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:37:01,944 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:37:03,608 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:37:04,934 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 02:37:05,261 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:37:05,261 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:37:05,649 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:37:06,657 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:37:07,528 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:37:07,706 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:37:09,751 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:37:11,796 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:37:12,571 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:37:13,843 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:37:15,891 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:37:17,613 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:37:17,930 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:37:18,923 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:37:19,965 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:37:20,278 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:37:20,278 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:37:22,084 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:37:23,552 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:37:24,096 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:37:26,120 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:37:28,151 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:37:28,592 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:37:30,195 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:37:32,248 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:37:33,240 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:37:34,088 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:37:34,290 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:37:34,939 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 02:37:35,287 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:37:35,288 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:37:36,334 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:37:38,382 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:37:39,572 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:37:40,451 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:37:42,488 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:37:44,514 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:37:44,522 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:37:44,875 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:37:46,567 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:37:48,633 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:37:49,922 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:37:50,297 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:37:50,298 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:37:50,684 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:37:52,790 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:37:54,799 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:37:55,577 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:37:56,842 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:37:58,873 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:37:58,886 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:38:00,805 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:38:00,952 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:38:02,997 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:38:04,939 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 02:38:05,042 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:38:05,302 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:38:05,302 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:38:06,556 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:38:07,086 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:38:09,128 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:38:11,156 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:38:11,164 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:38:11,671 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:38:13,219 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:38:15,268 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:38:16,704 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:38:17,309 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:38:19,351 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:38:20,314 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:38:20,314 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:38:21,393 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:38:22,577 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:38:23,475 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:38:23,497 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:38:25,518 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:38:27,538 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:38:27,667 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:38:29,575 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:38:31,619 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:38:32,718 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:38:33,660 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:38:34,944 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 02:38:35,319 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:38:35,320 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:38:35,712 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:38:37,754 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:38:37,762 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:38:38,581 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:38:39,819 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:38:41,865 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:38:43,627 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:38:43,912 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:38:45,954 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:38:47,998 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:38:49,006 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:38:49,370 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:38:50,046 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:38:50,345 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:38:50,346 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:38:52,099 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:38:54,206 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:38:54,620 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:38:56,231 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:38:58,264 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:38:59,671 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:39:00,308 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:39:01,193 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 02:39:01,194 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 02:39:01,194 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 02:39:01,195 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 02:39:01,310 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 02:39:02,350 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:39:02,351 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:39:03,360 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:39:04,413 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:39:04,951 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 02:39:04,952 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:39:05,354 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:39:05,354 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:39:06,481 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:39:08,515 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:39:10,563 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:39:10,618 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:39:12,616 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:39:14,661 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:39:15,665 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:39:15,883 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:39:16,720 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:39:18,762 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:39:20,359 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:39:20,359 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:39:20,814 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:39:21,622 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:39:22,862 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:39:24,992 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:39:26,661 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:39:27,009 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:39:28,997 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:39:29,035 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:39:31,080 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:39:31,790 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:39:33,129 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:39:34,960 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 02:39:35,186 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:39:35,371 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:39:35,371 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:39:37,244 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:39:37,635 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:39:39,280 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:39:41,321 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:39:41,334 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:39:42,701 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:39:43,374 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:39:45,417 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:39:47,470 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:39:47,736 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:39:49,509 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:39:50,378 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:39:50,378 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:39:51,552 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:39:53,448 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:39:53,599 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:39:55,678 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:39:55,731 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:39:57,748 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:39:58,480 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:39:59,767 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:40:01,824 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:40:03,538 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:40:03,869 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:40:04,973 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 02:40:05,380 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:40:05,380 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:40:05,917 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:40:07,939 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:40:07,947 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:40:08,555 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:40:09,986 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:40:12,055 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:40:13,598 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:40:14,089 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:40:16,119 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:40:18,165 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:40:18,653 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:40:20,195 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:40:20,406 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:40:20,406 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:40:22,252 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:40:22,258 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:40:23,701 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:40:24,305 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:40:26,417 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:40:28,436 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:40:28,743 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:40:30,461 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:40:32,509 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:40:33,517 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:40:34,547 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:40:34,555 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:40:34,976 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 02:40:35,417 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:40:35,418 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:40:36,604 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:40:38,645 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:40:39,707 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:40:40,682 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:40:42,743 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:40:44,764 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:40:44,773 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:40:46,817 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:40:47,819 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:40:48,882 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:40:50,287 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:40:50,427 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:40:50,428 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:40:50,911 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:40:52,946 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:40:54,987 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:40:55,733 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:40:57,108 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:40:59,125 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:41:00,098 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:41:01,149 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:41:01,252 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:41:03,177 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:41:04,978 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 02:41:05,214 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:41:05,437 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:41:05,437 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:41:06,692 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:41:07,254 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:41:09,303 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:41:11,137 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 02:41:11,138 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 02:41:11,138 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 02:41:11,139 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 02:41:11,360 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 02:41:11,370 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:41:12,364 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:41:12,409 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:41:13,400 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:41:13,408 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:41:14,409 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:41:15,444 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:41:17,459 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:41:17,485 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:41:19,532 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:41:20,451 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:41:20,452 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:41:21,582 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:41:22,738 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:41:23,626 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:41:25,658 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:41:25,671 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:41:27,779 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:41:28,603 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:41:29,804 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:41:31,822 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:41:33,664 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:41:33,849 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:41:34,986 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 02:41:35,469 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:41:35,470 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:41:35,890 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:41:37,930 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:41:39,506 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:41:39,957 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:41:39,966 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:41:42,014 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:41:44,054 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:41:44,550 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:41:46,095 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:41:48,165 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:41:49,610 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:41:50,204 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:41:50,462 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:41:50,463 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:41:52,252 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:41:52,259 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:41:54,315 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:41:54,774 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:41:56,369 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:41:58,481 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:41:59,835 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:42:00,498 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:42:02,521 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:42:04,564 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:42:04,569 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:42:04,986 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 02:42:04,987 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:42:05,483 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:42:05,483 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:42:06,611 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:42:08,646 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:42:10,684 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:42:10,794 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:42:12,728 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:42:14,776 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:42:15,833 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:42:16,837 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:42:17,838 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:42:18,883 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:42:20,493 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:42:20,494 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:42:20,929 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:42:21,753 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:42:22,972 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:42:25,002 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:42:26,799 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:42:27,047 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:42:29,153 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:42:30,117 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:42:31,180 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:42:31,871 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:42:33,193 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:42:34,992 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 02:42:35,221 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:42:35,505 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:42:35,506 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:42:37,283 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:42:37,762 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:42:39,328 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:42:41,381 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:42:43,444 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:42:43,678 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:42:44,436 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:42:45,472 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:42:47,519 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:42:48,718 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:42:49,570 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:42:50,518 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:42:50,518 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:42:51,612 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:42:53,664 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:42:53,798 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:42:55,710 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:42:56,707 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:42:57,754 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:42:59,607 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:42:59,863 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:43:01,889 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:43:03,903 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:43:04,653 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:43:05,000 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 02:43:05,515 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:43:05,516 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:43:05,943 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:43:07,984 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:43:08,977 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:43:10,031 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:43:10,482 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:43:12,080 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:43:14,131 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:43:15,533 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:43:16,169 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:43:18,203 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:43:20,244 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:43:20,313 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 02:43:20,314 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 02:43:20,314 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 02:43:20,316 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 02:43:20,553 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:43:20,566 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:43:20,566 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:43:21,244 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:43:21,244 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 02:43:22,286 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:43:22,303 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:43:24,347 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:43:25,861 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:43:26,387 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:43:28,432 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:43:30,552 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:43:30,872 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:43:32,565 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:43:34,549 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:43:34,588 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:43:35,012 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 02:43:35,565 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:43:35,565 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:43:36,612 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:43:36,814 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:43:38,643 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:43:40,685 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:43:41,848 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:43:42,736 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:43:44,780 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:43:46,823 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:43:47,555 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:43:48,863 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:43:48,872 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:43:50,560 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:43:50,560 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:43:50,929 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:43:52,829 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:43:52,964 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:43:54,997 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:43:57,045 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:43:57,877 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:43:59,086 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:44:01,166 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:44:01,186 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:44:03,200 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:44:03,413 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:44:05,016 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 02:44:05,225 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:44:05,570 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:44:05,571 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:44:07,267 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:44:08,845 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:44:09,299 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:44:11,341 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:44:13,382 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:44:14,207 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:44:14,382 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:44:15,428 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:44:17,462 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:44:19,247 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:44:19,518 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:44:20,580 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:44:20,580 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:44:21,548 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:44:23,594 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:44:24,858 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:44:25,639 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:44:26,632 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:44:27,680 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:44:29,735 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:44:30,024 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:44:31,857 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:44:33,876 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:44:35,020 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 02:44:35,422 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:44:35,578 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:44:35,578 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:44:35,904 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:44:37,936 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:44:38,933 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:44:39,977 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:44:40,840 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:44:42,021 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:44:44,069 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:44:45,886 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:44:46,104 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:44:48,152 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:44:50,189 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:44:50,584 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:44:50,585 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:44:51,859 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:44:52,249 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:44:53,243 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:44:54,293 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:44:56,337 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:44:56,912 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:44:58,373 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:45:00,415 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:45:01,971 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:45:02,517 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:45:04,545 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:45:05,033 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 02:45:05,508 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:45:05,601 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:45:05,601 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:45:06,567 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:45:07,875 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:45:08,580 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:45:10,628 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:45:12,664 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:45:12,901 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:45:14,709 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:45:16,749 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:45:16,750 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:45:18,285 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:45:18,811 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:45:20,615 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:45:20,615 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:45:20,850 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:45:22,892 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:45:23,899 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:45:24,950 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:45:26,990 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:45:28,939 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:45:28,951 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 02:45:28,952 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 02:45:28,953 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 02:45:28,954 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 02:45:29,025 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 02:45:29,038 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:45:30,037 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:45:31,085 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:45:31,102 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:45:33,211 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:45:34,240 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:45:35,047 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 02:45:35,229 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:45:35,625 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:45:35,626 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:45:37,257 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:45:39,306 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:45:39,922 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:45:41,351 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:45:43,380 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:45:43,396 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:45:45,208 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:45:45,442 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:45:47,495 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:45:49,552 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:45:50,245 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:45:50,641 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:45:50,641 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:45:51,602 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:45:53,648 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:45:55,696 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:45:55,827 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:45:57,738 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:45:57,746 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:45:59,786 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:46:00,884 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:46:01,834 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:46:03,947 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:46:05,058 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 02:46:05,654 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:46:05,654 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:46:05,915 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:46:05,965 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:46:07,985 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:46:08,968 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:46:10,016 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:46:11,511 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:46:12,063 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:46:14,106 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:46:16,146 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:46:16,555 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:46:18,192 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:46:20,237 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:46:20,660 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:46:20,661 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:46:21,228 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:46:21,920 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:46:22,270 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:46:24,316 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:46:26,353 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:46:26,955 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:46:28,406 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:46:30,447 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:46:31,998 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:46:32,503 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:46:33,498 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:46:34,610 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:46:35,072 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 02:46:35,667 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:46:35,668 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:46:36,633 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:46:37,934 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:46:38,656 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:46:40,676 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:46:42,729 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:46:42,979 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:46:44,771 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:46:46,810 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:46:47,818 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:46:48,426 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:46:48,850 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:46:50,673 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:46:50,674 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:46:50,890 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:46:52,942 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:46:53,939 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:46:54,988 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:46:57,036 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:46:59,077 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:46:59,095 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:47:00,076 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:47:01,126 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:47:03,172 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:47:04,151 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:47:05,086 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 02:47:05,292 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:47:05,682 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:47:05,683 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:47:07,305 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:47:09,328 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:47:09,966 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:47:11,355 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:47:11,364 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:47:13,399 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:47:15,437 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:47:15,767 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:47:17,475 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:47:19,517 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:47:20,693 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:47:20,694 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:47:20,936 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:47:21,555 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:47:23,596 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:47:25,640 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:47:25,655 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:47:26,467 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:47:27,690 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:47:29,733 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:47:31,494 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:47:31,782 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:47:33,833 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:47:35,093 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 02:47:35,712 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:47:35,713 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:47:35,946 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:47:36,176 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 02:47:36,177 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 02:47:36,177 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 02:47:36,179 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 02:47:36,923 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 02:47:37,430 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:47:37,932 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:47:37,970 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:47:38,933 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:47:39,987 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:47:42,019 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:47:42,477 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:47:44,067 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:47:46,112 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:47:47,520 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:47:48,151 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:47:50,191 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:47:50,205 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:47:50,724 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:47:50,724 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:47:52,250 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:47:52,993 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:47:54,304 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:47:56,359 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:47:58,051 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:47:58,409 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:48:00,452 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:48:02,503 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:48:03,356 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:48:03,501 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:48:04,566 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:48:05,110 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 02:48:05,733 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:48:05,734 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:48:06,704 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:48:08,729 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:48:09,002 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:48:10,750 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:48:12,766 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:48:14,044 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:48:14,811 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:48:16,872 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:48:17,856 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:48:18,902 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:48:19,552 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:48:20,733 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:48:20,734 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:48:20,961 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:48:23,011 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:48:25,064 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:48:25,076 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:48:27,128 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:48:29,165 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:48:30,171 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:48:30,407 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:48:31,209 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:48:33,255 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:48:35,125 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 02:48:35,293 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:48:35,585 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:48:35,741 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:48:35,741 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:48:37,409 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:48:39,426 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:48:41,029 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:48:41,449 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:48:42,445 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:48:43,499 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:48:45,541 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:48:46,093 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:48:47,596 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:48:49,640 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:48:50,761 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:48:50,761 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:48:51,687 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:48:52,021 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:48:53,732 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:48:55,761 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:48:55,770 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:48:57,808 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:48:57,837 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:48:59,883 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:49:01,915 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:49:02,829 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:49:03,964 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:49:05,128 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 02:49:05,767 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:49:05,768 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:49:06,009 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:49:07,863 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:49:08,099 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:49:08,141 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:49:10,169 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:49:12,223 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:49:12,891 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:49:14,251 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:49:16,293 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:49:17,925 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:49:18,348 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:49:20,396 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:49:20,775 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:49:20,775 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:49:22,437 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:49:22,438 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:49:23,043 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:49:24,483 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:49:26,533 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:49:28,077 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:49:28,582 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:49:30,640 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:49:32,687 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:49:33,624 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:49:34,718 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:49:34,727 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:49:35,141 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 02:49:35,799 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:49:35,799 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:49:36,778 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:49:38,907 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:49:39,077 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:49:40,926 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:49:42,955 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:49:44,138 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:49:45,003 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:49:45,314 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 02:49:45,315 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 02:49:45,316 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 02:49:45,316 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 02:49:46,011 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:49:46,012 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 02:49:47,042 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:49:47,056 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:49:49,104 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:49:49,592 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:49:50,808 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:49:50,809 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:49:51,150 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:49:53,192 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:49:55,099 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:49:55,236 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:49:57,279 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:49:59,324 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:50:00,325 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:50:00,575 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:50:01,359 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:50:03,401 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:50:05,151 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 02:50:05,446 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:50:05,621 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:50:05,828 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:50:05,828 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:50:07,506 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:50:09,612 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:50:11,128 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:50:11,643 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:50:12,608 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:50:13,654 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:50:15,672 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:50:16,410 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:50:17,702 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:50:19,744 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:50:20,835 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:50:20,836 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:50:21,792 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:50:22,096 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:50:23,834 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:50:24,829 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:50:25,872 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:50:27,924 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:50:28,069 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:50:29,969 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:50:32,019 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:50:33,107 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:50:34,063 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:50:35,163 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 02:50:35,841 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:50:35,842 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:50:36,102 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:50:38,166 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:50:38,743 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:50:39,169 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:50:40,265 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:50:42,287 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:50:43,786 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:50:44,304 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:50:46,329 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:50:48,382 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:50:48,821 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:50:50,428 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:50:50,436 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:50:50,861 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:50:50,861 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:50:52,471 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:50:54,117 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:50:54,526 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:50:56,572 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:50:58,606 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:50:59,147 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:51:00,667 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:51:02,712 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:51:02,723 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:51:04,181 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:51:04,773 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:51:05,170 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 02:51:05,854 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:51:05,854 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:51:06,820 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:51:08,852 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:51:10,140 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:51:10,965 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:51:12,982 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:51:15,004 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:51:15,902 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:51:17,036 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:51:17,043 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:51:19,085 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:51:20,864 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:51:20,865 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:51:21,123 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:51:21,128 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:51:23,165 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:51:25,201 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:51:26,184 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:51:27,251 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:51:29,301 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:51:29,306 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:51:31,354 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:51:31,569 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:51:33,401 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:51:35,179 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 02:51:35,455 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:51:35,882 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:51:35,882 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:51:37,142 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:51:37,508 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:51:39,547 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:51:40,547 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:51:41,654 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:51:42,236 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:51:43,665 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:51:45,694 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:51:47,267 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:51:47,717 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:51:49,752 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:51:50,902 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:51:50,902 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:51:51,794 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:51:52,785 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:51:52,807 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 02:51:52,808 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 02:51:52,809 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 02:51:52,810 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 02:51:53,837 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:51:53,837 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 02:51:53,849 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:51:54,852 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:51:55,896 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:51:57,954 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:51:58,093 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:52:00,001 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:52:02,037 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:52:03,150 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:52:04,080 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:52:05,188 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 02:52:05,917 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:52:05,917 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:52:06,117 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:52:08,159 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:52:08,178 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:52:09,158 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:52:10,218 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:52:12,367 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:52:13,235 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:52:14,380 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:52:16,408 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:52:18,284 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:52:18,437 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:52:20,487 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:52:20,935 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:52:20,936 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:52:21,487 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:52:22,529 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:52:24,207 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:52:24,567 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:52:26,603 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:52:28,651 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:52:29,263 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:52:30,724 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:52:32,749 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:52:32,757 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:52:34,796 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:52:35,189 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 02:52:35,190 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:52:35,933 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:52:35,934 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:52:36,839 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:52:38,896 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:52:40,207 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:52:40,930 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:52:43,036 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:52:45,010 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:52:45,047 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:52:45,796 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:52:47,061 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:52:49,114 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:52:50,947 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:52:50,947 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:52:51,144 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:52:51,205 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:52:53,197 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:52:55,245 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:52:56,251 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:52:57,281 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:52:59,317 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:52:59,326 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:53:01,368 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:53:01,465 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:53:03,421 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:53:05,198 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 02:53:05,458 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:53:05,959 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:53:05,959 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:53:07,224 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:53:07,500 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:53:09,551 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:53:11,581 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:53:11,593 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:53:13,134 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:53:13,703 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:53:15,715 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:53:17,732 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:53:18,179 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:53:19,776 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:53:20,963 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:53:20,964 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:53:21,824 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:53:23,790 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:53:23,856 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:53:23,866 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:53:25,909 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:53:27,954 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:53:28,830 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:53:29,999 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:53:32,047 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:53:33,867 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:53:34,079 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:53:35,083 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:53:35,208 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 02:53:35,977 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:53:35,978 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:53:36,136 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:53:38,186 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:53:39,275 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:53:40,237 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:53:42,291 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:53:44,326 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:53:44,400 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:53:46,424 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:53:48,449 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:53:49,400 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:53:50,019 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:53:50,474 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:53:50,990 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:53:50,990 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:53:52,487 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:53:54,532 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:53:55,271 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:53:56,575 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:53:58,622 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:53:59,655 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 02:53:59,657 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 02:53:59,657 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 02:53:59,658 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 02:54:00,690 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:54:00,690 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 02:54:00,701 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:54:00,878 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:54:01,694 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:54:02,745 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:54:04,801 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:54:05,213 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 02:54:06,012 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:54:06,012 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:54:06,266 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:54:06,841 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:54:08,890 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:54:10,933 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:54:11,321 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:54:12,982 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:54:13,977 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:54:15,080 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:54:16,840 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:54:17,102 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:54:19,130 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:54:21,023 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:54:21,023 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:54:21,162 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:54:22,277 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:54:23,205 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:54:25,251 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:54:27,291 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:54:27,299 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:54:27,415 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:54:29,356 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:54:31,392 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:54:32,465 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:54:33,427 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:54:35,216 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 02:54:35,465 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:54:36,036 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:54:36,037 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:54:37,509 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:54:38,091 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:54:39,557 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:54:39,567 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:54:41,612 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:54:43,141 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:54:43,662 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:54:45,779 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:54:47,789 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:54:48,192 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:54:49,809 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:54:51,043 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:54:51,043 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:54:51,852 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:54:51,854 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:54:53,308 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:54:53,906 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:54:55,957 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:54:58,000 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:54:58,346 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:55:00,057 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:55:02,089 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:55:04,128 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:55:04,348 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:55:05,220 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 02:55:06,052 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:55:06,053 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:55:06,168 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:55:06,177 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:55:08,226 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:55:10,277 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:55:10,345 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:55:12,339 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:55:14,423 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:55:15,413 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:55:16,548 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:55:17,515 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:55:18,567 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:55:20,593 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:55:20,832 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:55:21,068 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:55:21,069 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:55:22,618 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:55:24,663 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:55:26,375 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:55:26,708 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:55:28,757 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:55:29,749 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:55:30,814 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:55:31,595 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:55:32,854 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:55:34,898 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:55:35,232 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 02:55:36,083 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:55:36,084 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:55:36,936 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:55:37,344 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:55:38,983 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:55:41,031 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:55:42,024 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:55:43,079 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:55:43,225 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:55:45,127 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:55:47,233 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:55:48,286 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:55:49,243 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:55:51,088 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:55:51,088 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:55:51,267 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:55:53,309 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:55:53,361 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:55:55,350 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:55:56,342 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:55:57,399 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:55:58,825 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:55:59,445 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:56:01,494 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:56:03,538 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:56:03,868 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:56:05,237 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 02:56:05,573 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:56:06,097 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:56:06,097 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:56:06,492 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 02:56:06,494 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 02:56:06,494 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 02:56:06,495 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 02:56:06,566 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 02:56:07,626 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:56:08,634 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:56:09,716 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:56:09,763 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:56:11,755 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:56:13,792 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:56:14,809 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:56:15,832 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:56:17,946 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:56:19,963 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:56:20,734 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:56:21,097 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:56:21,098 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:56:21,945 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:56:21,979 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:56:24,014 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:56:26,079 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:56:26,400 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:56:28,112 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:56:30,161 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:56:31,442 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:56:32,218 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:56:34,257 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:56:34,266 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:56:35,240 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 02:56:36,092 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:56:36,092 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:56:36,317 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:56:37,358 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:56:38,368 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:56:40,407 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:56:42,412 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:56:42,453 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:56:44,507 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:56:46,531 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:56:46,539 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:56:47,968 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:56:48,645 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:56:50,662 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:56:51,102 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:56:51,102 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:56:52,684 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:56:53,376 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:56:54,722 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:56:56,763 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:56:58,569 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:56:58,804 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:57:00,868 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:57:00,880 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:57:02,920 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:57:03,614 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:57:04,968 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:57:05,245 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 02:57:06,111 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:57:06,112 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:57:07,008 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:57:09,053 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:57:09,367 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:57:11,096 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:57:12,089 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:57:13,133 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:57:15,175 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:57:15,241 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:57:17,225 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:57:19,331 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:57:20,289 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:57:21,110 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:57:21,111 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:57:21,355 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:57:23,368 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:57:24,358 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:57:25,406 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:57:25,910 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:57:27,449 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:57:29,491 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:57:30,938 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:57:31,540 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:57:33,589 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:57:35,249 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 02:57:35,641 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:57:36,119 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:57:36,120 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:57:36,361 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:57:36,638 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:57:37,679 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:57:39,722 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:57:41,408 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:57:41,783 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:57:43,831 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:57:45,875 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:57:46,464 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:57:47,938 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:57:50,043 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:57:51,029 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:57:51,139 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:57:51,139 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:57:52,062 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:57:52,401 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:57:54,084 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:57:56,122 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:57:57,449 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:57:58,170 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:58:00,215 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:58:02,257 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:58:02,265 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:58:02,759 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:58:04,310 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:58:05,255 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 02:58:06,150 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:58:06,150 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:58:06,366 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:58:08,421 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:58:08,427 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:58:10,478 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:58:12,523 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:58:13,363 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 02:58:13,364 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 02:58:13,365 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 02:58:13,366 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 02:58:13,524 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 02:58:13,619 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:58:14,548 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:58:14,557 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:58:15,556 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:58:16,631 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:58:18,665 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:58:18,668 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:58:20,785 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:58:21,157 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:58:21,157 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:58:22,810 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:58:24,427 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:58:24,832 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:58:26,862 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:58:28,898 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:58:28,906 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:58:29,699 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:58:30,971 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:58:33,015 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:58:34,726 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:58:35,063 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:58:35,257 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 02:58:36,152 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:58:36,153 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:58:37,121 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:58:39,160 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:58:40,353 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:58:41,195 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:58:41,204 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:58:43,246 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:58:45,306 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:58:45,394 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:58:47,357 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:58:49,393 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:58:50,430 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:58:51,165 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:58:51,165 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:58:51,496 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:58:53,523 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:58:54,479 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:58:55,546 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:58:56,010 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:58:57,577 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:58:59,613 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:59:01,046 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:59:01,657 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:59:03,703 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:59:05,271 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 02:59:05,741 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:59:06,188 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:59:06,189 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:59:06,450 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:59:06,739 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:59:07,808 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:59:09,852 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:59:11,499 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:59:11,903 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:59:13,945 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:59:15,992 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:59:16,545 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:59:18,041 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:59:19,041 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:59:20,087 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:59:21,194 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:59:21,194 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:59:22,194 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:59:22,450 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:59:24,211 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:59:26,238 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:59:27,498 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:59:28,283 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:59:30,333 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:59:31,335 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:59:32,398 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:59:32,881 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:59:34,431 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:59:35,282 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 02:59:36,224 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:59:36,225 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:59:36,476 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:59:38,518 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:59:38,531 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:59:40,565 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:59:42,613 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:59:44,522 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:59:44,655 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:59:44,665 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:59:46,715 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:59:48,763 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:59:49,581 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:59:50,810 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:59:51,239 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 02:59:51,239 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 02:59:52,930 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:59:54,954 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:59:55,135 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 02:59:56,947 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 02:59:56,969 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 02:59:59,005 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:00:00,156 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:00:01,056 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:00:03,109 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:00:05,149 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:00:05,211 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:00:05,290 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:00:06,257 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:00:06,257 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:00:07,193 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:00:09,229 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:00:09,237 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:00:10,829 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:00:11,284 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:00:13,336 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:00:15,380 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:00:15,861 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:00:17,420 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:00:19,461 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:00:20,453 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 03:00:20,454 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 03:00:20,455 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 03:00:20,456 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 03:00:20,463 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 03:00:21,043 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:00:21,268 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:00:21,268 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:00:21,497 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:00:21,504 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:00:22,511 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:00:23,574 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:00:23,607 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:00:25,617 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:00:26,571 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:00:27,640 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:00:29,684 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:00:31,623 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:00:31,739 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:00:33,797 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:00:35,292 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:00:35,830 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:00:35,840 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:00:36,277 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:00:36,277 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:00:37,524 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:00:37,885 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:00:39,938 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:00:41,984 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:00:42,568 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:00:44,022 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:00:46,063 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:00:47,062 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:00:48,114 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:00:48,342 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:00:50,163 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:00:51,280 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:00:51,280 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:00:52,222 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:00:53,547 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:00:54,346 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:00:56,376 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:00:58,392 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:00:58,589 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:01:00,425 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:01:01,428 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:01:02,474 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:01:04,020 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:01:04,526 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:01:05,301 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:01:06,291 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:01:06,292 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:01:06,573 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:01:08,625 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:01:09,563 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:01:10,664 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:01:12,712 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:01:13,717 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:01:14,609 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:01:14,756 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:01:16,806 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:01:18,855 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:01:19,650 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:01:20,895 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:01:21,309 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:01:21,310 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:01:22,939 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:01:25,046 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:01:25,313 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:01:26,008 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:01:27,072 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:01:29,096 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:01:30,353 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:01:31,139 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:01:33,182 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:01:35,235 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:01:35,312 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:01:36,085 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:01:36,322 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:01:36,322 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:01:37,264 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:01:37,273 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:01:39,327 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:01:41,377 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:01:42,004 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:01:43,420 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:01:45,455 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:01:47,050 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:01:47,498 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:01:49,544 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:01:51,344 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:01:51,344 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:01:51,586 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:01:51,594 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:01:52,603 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:01:53,644 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:01:55,744 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:01:57,663 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:01:57,766 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:01:59,782 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:02:01,825 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:02:03,218 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:02:03,870 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:02:03,881 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:02:05,316 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:02:05,922 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:02:06,342 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:02:06,343 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:02:07,973 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:02:08,596 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:02:10,020 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:02:12,070 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:02:13,626 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:02:14,131 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:02:16,162 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:02:16,172 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:02:18,217 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:02:18,955 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:02:20,263 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:02:21,337 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:02:21,337 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:02:22,307 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:02:24,376 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:02:24,614 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:02:26,518 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:02:27,664 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 03:02:27,665 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 03:02:27,665 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 03:02:27,666 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 03:02:28,491 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:02:28,491 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 03:02:28,538 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:02:29,505 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:02:29,926 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:02:30,591 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:02:32,632 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:02:34,665 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:02:34,946 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:02:35,324 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:02:36,346 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:02:36,346 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:02:36,717 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:02:38,766 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:02:40,608 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:02:40,825 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:02:41,826 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:02:42,870 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:02:44,917 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:02:45,899 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:02:46,964 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:02:49,015 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:02:50,927 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:02:51,077 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:02:51,349 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:02:51,349 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:02:53,112 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:02:55,170 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:02:56,163 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:02:56,534 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:02:57,278 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:02:59,296 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:03:01,328 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:03:01,603 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:03:03,368 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:03:05,324 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:03:05,400 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:03:06,373 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:03:06,373 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:03:06,616 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:03:07,447 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:03:08,451 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:03:09,491 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:03:11,535 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:03:11,665 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:03:13,592 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:03:15,641 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:03:16,682 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:03:17,691 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:03:19,737 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:03:19,749 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:03:21,381 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:03:21,382 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:03:21,789 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:03:22,639 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:03:23,823 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:03:25,876 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:03:27,674 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:03:27,993 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:03:30,002 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:03:32,031 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:03:32,043 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:03:33,300 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:03:34,078 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:03:35,334 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:03:36,128 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:03:36,396 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:03:36,396 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:03:38,169 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:03:38,654 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:03:40,231 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:03:42,282 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:03:43,707 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:03:44,321 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:03:46,370 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:03:46,371 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:03:48,433 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:03:48,953 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:03:50,471 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:03:51,399 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:03:51,400 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:03:52,518 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:03:54,551 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:03:54,667 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:03:56,595 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:03:58,674 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:03:58,713 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:04:00,536 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:04:00,739 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:04:02,755 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:04:04,787 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:04:05,347 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:04:06,138 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:04:06,405 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:04:06,406 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:04:06,848 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:04:08,893 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:04:09,883 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:04:10,940 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:04:11,138 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:04:12,976 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:04:15,033 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:04:16,180 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:04:17,081 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:04:19,125 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:04:21,172 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:04:21,413 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:04:21,413 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:04:21,667 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:04:23,222 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:04:24,221 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:04:25,279 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:04:26,812 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:04:27,325 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:04:29,446 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:04:31,458 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:04:31,842 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:04:33,471 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:04:34,414 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 03:04:34,415 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 03:04:34,415 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 03:04:34,416 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 03:04:34,468 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 03:04:35,362 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:04:35,520 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:04:36,424 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:04:36,425 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:04:36,518 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:04:37,554 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:04:37,567 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:04:37,689 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:04:39,612 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:04:41,661 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:04:42,738 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:04:43,709 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:04:45,745 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:04:47,803 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:04:48,687 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:04:48,798 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:04:49,845 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:04:51,438 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:04:51,439 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:04:51,893 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:04:53,713 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:04:53,937 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:04:55,974 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:04:58,020 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:04:58,741 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:05:00,139 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:05:02,127 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:05:02,154 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:05:04,177 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:05:04,322 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:05:05,364 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:05:06,220 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:05:06,441 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:05:06,442 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:05:08,269 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:05:09,710 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:05:10,330 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:05:12,379 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:05:14,411 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:05:14,420 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:05:14,977 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:05:16,459 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:05:18,497 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:05:20,019 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:05:20,558 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:05:21,452 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:05:21,452 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:05:22,605 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:05:24,650 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:05:25,570 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:05:26,683 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:05:26,691 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:05:28,743 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:05:30,619 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:05:30,897 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:05:32,917 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:05:34,932 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:05:35,365 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:05:36,184 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:05:36,452 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:05:36,452 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:05:36,949 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:05:38,997 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:05:41,031 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:05:41,044 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:05:41,222 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:05:43,089 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:05:45,137 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:05:46,261 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:05:47,185 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:05:49,227 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:05:51,269 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:05:51,453 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:05:51,454 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:05:51,700 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:05:52,270 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:05:53,301 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:05:55,341 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:05:56,745 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:05:57,386 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:05:59,424 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:06:01,536 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:06:01,786 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:06:03,555 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:06:04,534 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:06:05,371 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:06:05,569 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:06:06,458 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:06:06,458 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:06:07,607 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:06:07,722 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:06:09,644 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:06:11,687 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:06:12,780 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:06:13,740 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:06:15,780 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:06:17,830 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:06:18,155 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:06:18,822 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:06:19,878 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:06:21,465 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:06:21,466 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:06:21,923 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:06:23,740 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:06:23,957 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:06:25,993 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:06:28,038 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:06:28,776 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:06:30,070 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:06:31,069 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:06:32,214 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:06:33,820 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:06:34,237 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:06:35,383 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:06:36,259 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:06:36,469 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:06:36,470 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:06:38,273 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:06:39,763 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:06:40,315 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:06:41,385 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 03:06:41,387 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 03:06:41,388 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 03:06:41,388 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 03:06:42,343 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 03:06:42,353 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:06:43,359 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:06:44,394 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:06:45,692 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:06:46,444 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:06:48,479 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:06:50,521 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:06:50,720 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:06:51,484 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:06:51,485 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:06:52,567 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:06:54,611 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:06:56,671 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:06:56,672 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:06:56,676 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:06:58,719 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:07:00,784 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:07:01,732 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:07:02,890 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:07:04,913 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:07:05,392 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:07:06,503 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:07:06,503 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:07:06,755 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:07:06,939 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:07:08,918 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:07:08,955 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:07:10,982 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:07:12,331 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:07:13,024 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:07:15,058 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:07:17,105 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:07:17,359 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:07:19,148 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:07:21,174 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:07:21,187 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:07:21,519 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:07:21,519 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:07:22,767 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:07:23,221 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:07:25,263 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:07:27,305 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:07:27,807 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:07:29,351 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:07:31,395 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:07:33,515 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:07:33,625 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:07:35,394 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:07:35,481 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:07:35,528 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:07:36,525 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:07:36,526 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:07:37,542 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:07:38,771 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:07:39,591 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:07:41,647 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:07:43,693 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:07:43,807 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:07:45,735 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:07:47,771 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:07:47,780 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:07:49,249 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:07:49,824 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:07:51,552 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:07:51,553 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:07:51,876 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:07:53,910 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:07:54,830 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:07:55,949 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:07:57,993 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:07:59,005 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:07:59,871 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:08:00,040 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:08:02,083 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:08:04,198 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:08:04,926 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:08:05,400 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:08:06,207 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:08:06,565 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:08:06,566 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:08:08,241 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:08:10,273 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:08:10,517 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:08:12,324 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:08:13,317 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:08:14,362 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:08:15,559 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:08:16,409 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:08:18,462 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:08:20,497 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:08:20,611 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:08:21,571 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:08:21,572 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:08:22,541 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:08:24,588 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:08:25,602 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:08:26,199 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:08:26,632 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:08:28,682 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:08:30,748 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:08:31,235 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:08:32,797 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:08:34,913 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:08:35,411 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:08:36,292 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:08:36,589 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:08:36,589 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:08:36,939 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:08:37,874 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:08:38,965 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:08:40,987 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:08:41,885 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:08:43,002 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:08:45,048 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:08:46,941 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:08:47,094 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:08:48,467 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 03:08:48,468 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 03:08:48,469 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 03:08:48,469 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 03:08:49,128 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:08:49,129 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 03:08:49,137 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:08:50,132 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:08:51,170 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:08:51,177 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:08:51,619 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:08:51,620 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:08:52,877 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:08:53,218 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:08:55,258 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:08:57,309 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:08:57,915 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:08:59,344 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:09:01,385 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:09:03,418 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:09:03,432 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:09:03,698 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:09:05,460 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:09:05,539 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:09:06,623 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:09:06,624 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:09:07,563 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:09:08,874 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:09:09,587 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:09:11,607 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:09:13,654 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:09:13,921 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:09:15,684 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:09:15,692 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:09:17,744 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:09:19,393 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:09:19,782 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:09:21,625 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:09:21,625 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:09:21,818 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:09:23,866 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:09:24,905 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:09:25,911 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:09:27,952 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:09:29,985 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:09:29,992 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:09:30,073 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:09:32,058 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:09:34,115 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:09:35,129 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:09:35,431 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:09:36,223 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:09:36,639 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:09:36,640 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:09:38,235 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:09:40,264 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:09:40,687 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:09:41,268 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:09:42,315 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:09:44,366 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:09:45,724 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:09:46,394 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:09:48,437 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:09:50,480 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:09:50,762 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:09:51,649 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:09:51,649 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:09:52,529 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:09:53,533 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:09:54,576 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:09:56,312 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:09:56,615 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:09:58,664 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:10:00,730 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:10:01,367 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:10:02,789 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:10:04,841 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:10:05,446 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:10:06,653 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:10:06,654 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:10:06,918 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:10:06,951 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:10:07,932 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:10:08,968 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:10:10,987 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:10:11,984 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:10:13,037 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:10:15,074 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:10:17,097 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:10:17,123 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:10:19,170 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:10:20,167 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:10:21,216 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:10:21,666 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:10:21,667 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:10:22,934 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:10:23,274 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:10:25,326 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:10:27,374 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:10:27,974 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:10:29,419 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:10:31,465 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:10:31,472 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:10:33,368 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:10:33,524 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:10:35,456 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:10:35,566 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:10:36,682 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:10:36,683 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:10:37,674 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:10:38,935 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:10:39,683 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:10:41,724 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:10:43,764 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:10:43,773 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:10:44,007 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:10:45,817 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:10:47,866 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:10:49,050 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:10:49,901 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:10:51,686 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:10:51,686 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:10:51,964 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:10:54,007 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:10:55,000 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:10:55,657 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 03:10:55,659 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 03:10:55,659 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 03:10:55,661 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 03:10:56,051 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 03:10:56,065 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:10:57,063 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:10:58,087 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:10:58,096 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:11:00,133 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:11:00,955 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:11:02,166 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:11:04,206 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:11:05,459 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:11:06,248 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:11:06,375 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:11:06,693 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:11:06,693 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:11:08,338 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:11:10,336 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:11:10,362 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:11:11,906 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:11:12,404 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:11:14,441 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:11:16,478 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:11:16,949 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:11:18,529 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:11:20,581 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:11:21,695 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:11:21,695 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:11:22,617 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:11:22,954 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:11:24,644 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:11:24,654 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:11:26,684 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:11:28,006 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:11:28,749 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:11:30,811 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:11:32,845 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:11:33,025 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:11:34,886 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:11:35,474 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:11:35,885 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:11:36,703 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:11:36,703 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:11:36,929 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:11:38,976 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:11:39,034 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:11:41,042 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:11:43,073 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:11:44,034 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:11:45,128 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:11:47,169 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:11:48,163 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:11:49,220 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:11:49,679 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:11:51,256 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:11:51,701 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:11:51,701 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:11:53,306 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:11:54,981 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:11:55,352 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:11:57,400 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:11:59,447 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:12:00,362 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:12:00,441 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:12:01,488 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:12:03,526 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:12:05,402 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:12:05,481 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:12:05,565 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:12:06,709 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:12:06,709 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:12:07,614 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:12:09,732 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:12:10,996 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:12:11,748 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:12:13,770 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:12:14,756 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:12:15,809 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:12:16,001 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:12:17,877 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:12:19,942 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:12:21,034 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:12:21,728 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:12:21,729 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:12:22,003 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:12:24,041 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:12:26,089 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:12:26,098 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:12:26,652 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:12:28,138 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:12:30,176 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:12:31,705 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:12:32,235 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:12:34,284 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:12:35,493 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:12:36,335 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:12:36,753 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:12:36,754 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:12:37,010 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:12:38,372 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:12:38,382 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:12:40,497 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:12:42,262 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:12:42,524 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:12:44,549 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:12:46,558 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:12:47,333 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:12:48,586 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:12:50,616 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:12:50,630 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:12:51,766 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:12:51,767 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:12:52,674 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:12:53,031 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:12:54,707 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:12:56,748 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:12:58,066 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:12:58,789 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:13:00,859 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:13:02,698 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 03:13:02,699 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 03:13:02,699 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 03:13:02,701 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 03:13:02,910 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 03:13:02,924 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:13:03,954 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:13:04,954 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:13:04,966 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:13:05,496 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:13:06,757 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:13:06,758 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:13:07,000 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:13:09,029 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:13:09,037 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:13:11,143 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:13:13,153 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:13:14,102 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:13:15,175 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:13:17,223 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:13:18,233 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:13:19,266 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:13:20,030 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:13:21,308 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:13:21,769 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:13:21,769 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:13:23,358 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:13:25,039 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:13:25,394 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:13:27,435 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:13:29,482 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:13:30,481 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:13:30,633 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:13:31,519 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:13:33,559 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:13:35,507 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:13:35,598 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:13:36,467 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:13:36,765 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:13:36,765 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:13:37,647 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:13:39,697 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:13:41,798 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:13:42,210 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:13:42,777 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:13:43,819 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:13:45,840 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:13:47,252 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:13:47,864 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:13:49,919 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:13:51,761 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:13:51,761 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:13:51,978 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:13:53,002 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:13:54,029 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:13:55,027 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:13:56,066 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:13:58,111 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:13:58,848 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:14:00,149 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:14:02,193 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:14:03,887 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:14:04,251 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:14:05,508 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:14:06,296 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:14:06,781 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:14:06,781 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:14:08,336 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:14:09,048 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:14:09,335 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:14:10,375 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:14:12,483 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:14:14,088 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:14:14,506 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:14:16,516 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:14:18,543 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:14:20,086 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:14:20,576 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:14:20,584 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:14:21,789 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:14:21,790 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:14:22,621 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:14:24,656 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:14:26,079 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:14:26,700 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:14:28,735 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:14:30,808 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:14:31,126 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:14:32,848 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:14:32,856 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:14:34,896 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:14:35,524 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:14:36,501 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:14:36,801 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:14:36,801 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:14:36,935 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:14:38,977 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:14:41,022 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:14:42,092 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:14:43,132 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:14:45,120 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:14:45,151 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:14:47,164 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:14:47,358 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:14:49,193 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:14:51,236 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:14:51,798 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:14:51,798 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:14:53,047 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:14:53,276 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:14:55,341 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:14:57,390 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:14:58,966 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:14:59,415 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:14:59,422 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:15:01,462 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:15:03,504 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:15:03,993 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:15:05,529 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:15:05,537 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:15:06,810 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:15:06,811 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:15:07,577 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:15:09,086 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:15:09,525 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 03:15:09,526 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 03:15:09,527 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 03:15:09,528 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 03:15:09,617 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 03:15:09,625 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:15:11,663 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:15:11,671 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:15:13,791 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:15:14,803 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:15:15,806 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:15:17,821 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:15:19,845 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:15:19,852 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:15:21,817 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:15:21,817 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:15:21,900 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:15:23,931 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:15:24,937 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:15:25,849 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:15:25,970 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:15:28,016 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:15:30,050 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:15:30,910 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:15:32,129 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:15:34,171 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:15:35,529 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:15:36,235 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:15:36,532 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:15:36,829 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:15:36,829 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:15:37,225 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:15:38,288 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:15:40,351 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:15:42,129 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:15:42,394 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:15:44,505 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:15:46,529 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:15:47,174 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:15:48,553 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:15:49,532 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:15:50,587 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:15:51,834 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:15:51,834 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:15:52,633 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:15:53,087 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:15:54,684 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:15:56,725 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:15:58,134 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:15:58,776 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:16:00,855 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:16:01,848 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:16:02,885 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:16:03,661 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:16:04,925 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:16:05,532 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:16:06,843 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:16:06,843 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:16:06,967 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:16:09,017 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:16:09,114 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:16:11,073 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:16:13,106 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:16:14,264 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:16:15,174 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:16:15,212 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:16:17,239 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:16:19,248 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:16:19,303 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:16:21,287 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:16:21,844 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:16:21,844 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:16:23,327 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:16:25,116 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:16:25,368 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:16:27,407 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:16:27,416 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:16:29,455 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:16:30,871 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:16:31,496 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:16:33,545 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:16:35,537 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:16:35,593 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:16:36,544 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:16:36,843 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:16:36,844 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:16:37,627 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:16:39,670 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:16:39,672 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:16:41,729 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:16:42,440 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:16:43,774 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:16:45,881 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:16:47,492 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:16:47,895 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:16:49,915 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:16:51,861 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:16:51,861 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:16:51,962 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:16:53,123 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:16:53,993 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:16:54,001 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:16:56,039 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:16:58,087 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:16:58,166 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:17:00,131 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:17:02,169 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:17:03,212 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:17:04,217 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:17:05,230 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:17:05,534 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:17:06,262 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:17:06,870 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:17:06,871 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:17:08,306 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:17:09,157 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:17:10,346 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:17:12,410 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:17:14,193 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:17:14,448 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:17:16,373 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 03:17:16,374 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 03:17:16,375 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 03:17:16,377 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 03:17:16,530 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 03:17:16,563 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:17:17,537 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:17:18,553 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:17:18,577 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:17:19,673 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:17:20,591 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:17:21,887 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:17:21,887 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:17:22,639 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:17:24,685 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:17:25,171 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:17:26,734 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:17:28,773 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:17:30,699 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:17:30,841 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:17:31,838 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:17:32,872 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:17:34,922 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:17:35,565 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:17:36,592 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:17:36,891 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:17:36,891 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:17:36,960 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:17:39,002 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:17:41,045 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:17:42,165 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:17:43,087 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:17:44,082 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:17:45,116 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:17:47,226 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:17:47,416 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:17:49,244 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:17:51,266 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:17:51,896 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:17:51,896 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:17:53,155 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:17:53,293 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:17:55,338 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:17:55,339 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:17:57,380 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:17:59,116 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:17:59,426 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:18:01,479 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:18:03,540 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:18:04,157 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:18:05,572 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:18:05,582 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:18:06,917 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:18:06,918 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:18:07,664 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:18:09,692 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:18:09,704 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:18:09,830 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:18:11,744 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:18:13,787 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:18:14,888 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:18:15,826 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:18:17,938 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:18:19,928 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:18:19,960 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:18:21,961 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:18:21,961 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:18:21,963 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:18:21,973 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:18:24,016 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:18:25,243 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:18:26,060 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:18:28,106 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:18:30,161 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:18:30,272 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:18:32,224 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:18:34,260 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:18:34,266 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:18:35,582 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:18:35,583 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:18:36,316 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:18:36,936 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:18:36,937 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:18:38,353 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:18:40,414 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:18:41,214 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:18:42,451 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:18:44,500 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:18:46,548 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:18:46,603 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:18:47,553 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:18:48,666 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:18:50,685 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:18:51,610 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:18:51,941 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:18:51,942 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:18:52,709 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:18:54,751 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:18:56,794 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:18:57,228 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:18:58,835 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:18:59,835 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:19:00,896 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:19:02,229 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:19:02,933 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:19:04,974 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:19:05,596 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:19:06,950 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:19:06,951 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:19:07,008 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:19:08,219 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:19:09,048 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:19:11,104 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:19:12,100 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:19:13,145 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:19:13,781 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:19:15,186 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:19:17,225 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:19:18,845 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:19:19,332 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:19:21,340 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:19:21,965 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:19:21,965 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:19:23,357 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:19:23,403 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 03:19:23,404 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 03:19:23,404 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 03:19:23,406 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 03:19:24,348 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:19:24,348 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 03:19:24,680 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:19:25,400 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:19:26,395 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:19:27,443 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:19:29,487 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:19:29,710 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:19:31,518 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:19:33,579 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:19:34,764 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:19:35,612 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:19:35,624 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:19:36,988 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:19:36,989 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:19:37,688 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:19:38,681 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:19:39,735 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:19:40,258 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:19:41,766 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:19:43,815 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:19:45,305 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:19:45,862 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:19:47,925 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:19:50,000 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:19:50,021 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:19:51,288 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:19:52,017 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:19:52,017 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:19:52,038 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:19:54,067 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:19:56,098 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:19:56,308 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:19:58,155 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:20:00,199 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:20:01,341 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:20:02,231 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:20:04,277 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:20:04,287 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:20:05,618 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:20:06,321 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:20:06,629 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:20:07,006 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:20:07,006 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:20:08,358 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:20:10,396 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:20:12,284 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:20:12,450 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:20:14,493 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:20:16,533 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:20:16,540 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:20:17,643 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:20:18,588 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:20:20,704 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:20:22,025 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:20:22,026 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:20:22,729 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:20:23,285 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:20:24,748 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:20:26,772 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:20:28,789 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:20:28,798 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:20:29,276 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:20:30,858 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:20:32,910 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:20:34,316 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:20:34,948 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:20:35,645 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:20:36,998 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:20:37,028 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:20:37,029 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:20:39,040 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:20:39,819 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:20:40,038 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:20:41,080 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:20:43,125 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:20:44,868 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:20:45,167 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:20:47,202 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:20:49,243 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:20:49,926 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:20:51,351 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:20:52,030 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:20:52,031 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:20:53,361 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:20:54,344 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:20:55,387 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:20:55,451 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:20:57,438 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:20:59,487 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:21:00,496 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:21:01,532 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:21:03,582 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:21:05,631 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:21:05,646 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:21:05,647 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:21:06,627 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:21:07,036 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:21:07,036 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:21:07,680 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:21:09,727 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:21:11,327 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:21:11,778 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:21:13,828 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:21:15,878 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:21:16,364 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:21:17,929 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:21:18,919 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:21:19,960 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:21:21,677 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:21:22,059 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:21:22,059 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:21:22,069 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:21:24,086 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:21:26,110 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:21:27,354 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:21:28,153 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:21:30,188 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:21:30,337 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 03:21:30,338 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 03:21:30,338 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 03:21:30,339 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 03:21:31,186 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 03:21:32,238 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:21:32,246 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:21:32,573 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:21:33,251 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:21:34,285 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:21:35,649 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:21:36,330 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:21:37,058 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:21:37,059 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:21:38,319 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:21:38,377 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:21:40,439 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:21:42,500 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:21:43,371 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:21:44,547 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:21:44,562 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:21:46,591 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:21:48,639 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:21:48,650 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:21:50,691 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:21:52,063 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:21:52,064 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:21:52,801 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:21:54,340 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:21:54,825 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:21:56,846 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:21:58,861 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:21:58,872 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:22:00,189 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:22:00,917 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:22:02,976 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:22:05,031 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:22:05,230 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:22:05,655 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:22:07,065 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:22:07,065 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:22:07,074 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:22:09,123 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:22:10,840 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:22:11,158 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:22:11,162 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:22:13,209 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:22:15,256 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:22:15,875 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:22:17,292 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:22:19,332 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:22:20,935 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:22:21,374 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:22:22,057 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:22:22,057 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:22:23,453 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:22:23,474 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:22:25,484 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:22:26,339 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:22:27,507 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:22:29,552 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:22:31,388 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:22:31,588 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:22:33,642 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:22:34,635 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:22:35,672 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:22:35,683 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:22:36,686 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:22:37,062 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:22:37,063 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:22:37,715 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:22:39,761 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:22:41,804 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:22:42,371 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:22:43,849 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:22:45,885 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:22:47,601 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:22:47,922 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:22:48,921 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:22:49,963 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:22:52,007 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:22:52,061 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:22:52,062 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:22:53,326 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:22:54,139 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:22:56,152 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:22:58,167 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:22:58,379 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:23:00,202 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:23:01,200 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:23:02,237 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:23:04,162 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:23:04,293 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:23:05,678 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:23:06,333 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:23:07,067 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:23:07,067 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:23:08,381 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:23:09,326 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:23:10,424 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:23:12,467 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:23:13,455 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:23:14,505 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:23:14,932 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:23:16,547 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:23:18,587 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:23:19,959 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:23:20,625 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:23:22,062 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:23:22,063 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:23:22,673 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:23:24,780 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:23:25,560 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:23:25,765 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:23:26,799 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:23:28,818 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:23:30,601 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:23:30,847 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:23:32,879 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:23:34,929 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:23:35,643 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:23:35,691 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:23:36,985 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:23:37,075 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:23:37,076 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:23:37,230 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 03:23:37,335 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 03:23:37,335 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 03:23:37,335 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 03:23:37,992 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 03:23:39,015 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:23:39,024 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:23:40,025 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:23:41,075 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:23:41,510 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:23:43,115 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:23:45,163 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:23:46,536 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:23:47,213 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:23:49,259 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:23:51,303 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:23:51,306 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:23:51,758 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:23:52,083 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:23:52,083 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:23:53,350 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:23:55,464 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:23:57,385 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:23:57,477 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:23:59,511 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:24:01,560 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:24:02,434 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:24:03,598 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:24:05,633 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:24:05,642 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:24:05,695 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:24:07,095 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:24:07,096 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:24:07,688 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:24:08,343 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:24:09,750 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:24:11,800 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:24:13,377 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:24:13,845 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:24:15,882 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:24:17,919 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:24:17,927 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:24:18,728 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:24:19,965 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:24:21,997 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:24:22,099 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:24:22,099 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:24:24,038 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:24:24,357 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:24:26,163 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:24:28,183 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:24:29,127 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:24:30,199 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:24:30,287 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:24:32,232 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:24:34,273 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:24:35,328 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:24:35,708 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:24:36,329 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:24:37,100 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:24:37,100 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:24:38,367 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:24:40,412 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:24:40,422 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:24:42,463 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:24:43,456 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:24:44,508 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:24:45,997 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:24:46,557 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:24:48,592 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:24:50,638 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:24:51,033 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:24:52,101 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:24:52,101 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:24:52,675 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:24:54,718 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:24:55,724 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:24:56,532 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:24:56,842 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:24:58,864 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:25:00,923 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:25:01,591 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:25:02,973 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:25:05,031 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:25:05,709 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:25:06,797 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:25:07,076 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:25:07,121 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:25:07,121 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:25:08,073 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:25:09,122 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:25:11,155 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:25:12,407 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:25:13,206 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:25:15,245 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:25:17,294 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:25:17,454 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:25:19,334 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:25:21,363 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:25:21,374 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:25:22,133 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:25:22,133 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:25:22,665 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:25:23,411 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:25:25,454 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:25:27,577 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:25:27,684 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:25:29,597 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:25:31,624 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:25:33,517 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:25:33,631 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:25:33,639 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:25:35,682 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:25:35,720 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:25:37,138 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:25:37,138 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:25:37,732 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:25:39,412 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:25:39,778 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:25:41,821 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:25:43,864 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:25:44,149 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 03:25:44,151 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 03:25:44,151 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 03:25:44,152 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 03:25:44,867 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 03:25:45,403 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:25:45,892 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:25:45,899 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:25:47,941 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:25:49,998 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:25:50,441 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:25:52,064 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:25:52,140 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:25:52,140 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:25:54,109 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:25:56,156 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:25:56,412 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:25:58,242 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:25:58,265 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:26:00,282 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:26:02,302 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:26:02,305 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:26:04,341 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:26:05,736 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:26:06,378 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:26:07,143 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:26:07,143 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:26:07,397 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:26:08,418 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:26:10,465 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:26:12,495 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:26:12,510 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:26:12,868 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:26:14,550 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:26:16,601 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:26:17,910 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:26:18,635 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:26:20,683 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:26:22,145 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:26:22,146 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:26:22,723 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:26:23,422 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:26:23,723 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:26:24,764 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:26:26,811 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:26:28,476 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:26:28,917 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:26:30,929 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:26:32,957 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:26:33,530 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:26:34,996 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:26:35,742 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:26:36,012 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:26:37,032 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:26:37,149 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:26:37,149 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:26:39,069 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:26:39,423 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:26:41,107 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:26:43,156 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:26:44,470 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:26:45,200 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:26:47,241 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:26:49,289 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:26:49,623 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:26:50,290 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:26:51,335 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:26:52,158 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:26:52,158 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:26:53,382 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:26:55,431 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:26:55,434 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:26:57,474 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:26:59,576 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:27:01,248 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:27:01,586 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:27:02,572 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:27:03,607 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:27:05,661 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:27:05,750 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:27:06,757 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:27:07,166 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:27:07,166 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:27:07,696 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:27:09,743 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:27:11,787 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:27:12,501 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:27:13,820 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:27:13,828 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:27:15,874 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:27:17,920 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:27:17,938 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:27:19,958 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:27:21,999 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:27:22,167 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:27:22,168 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:27:23,447 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:27:24,040 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:27:26,069 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:27:26,076 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:27:28,129 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:27:28,461 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:27:30,234 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:27:32,255 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:27:33,502 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:27:34,271 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:27:35,765 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:27:36,296 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:27:37,190 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:27:37,190 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:27:38,331 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:27:39,106 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:27:40,372 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:27:40,378 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:27:42,417 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:27:44,148 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:27:44,462 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:27:46,506 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:27:48,543 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:27:49,198 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:27:50,584 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:27:50,643 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 03:27:50,645 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 03:27:50,645 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 03:27:50,646 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 03:27:51,582 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 03:27:52,188 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:27:52,188 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:27:52,613 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:27:52,620 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:27:54,457 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:27:54,668 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:27:56,710 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:27:58,754 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:27:59,505 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:28:00,893 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:28:02,913 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:28:04,899 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:28:04,921 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:28:05,778 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:28:06,948 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:28:06,958 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:28:07,196 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:28:07,197 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:28:09,011 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:28:10,486 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:28:11,063 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:28:13,097 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:28:15,139 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:28:15,535 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:28:17,197 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:28:18,187 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:28:19,257 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:28:21,310 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:28:21,528 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:28:22,219 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:28:22,220 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:28:23,353 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:28:25,399 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:28:27,447 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:28:27,513 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:28:29,490 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:28:30,482 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:28:31,583 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:28:33,072 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:28:33,602 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:28:35,629 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:28:35,792 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:28:37,229 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:28:37,229 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:28:37,652 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:28:38,492 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:28:39,699 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:28:41,733 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:28:42,734 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:28:43,703 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:28:43,773 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:28:45,819 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:28:47,852 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:28:48,757 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:28:49,890 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:28:51,950 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:28:52,249 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:28:52,250 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:28:53,987 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:28:54,341 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:28:56,037 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:28:57,037 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:28:58,085 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:28:59,381 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:29:00,121 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:29:02,218 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:29:04,243 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:29:04,440 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:29:05,794 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:29:06,261 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:29:07,248 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:29:07,248 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:29:08,280 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:29:08,289 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:29:09,520 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:29:10,331 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:29:12,396 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:29:14,440 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:29:14,552 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:29:16,494 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:29:18,549 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:29:20,538 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:29:20,583 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:29:20,595 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:29:22,259 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:29:22,259 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:29:22,653 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:29:24,688 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:29:26,534 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:29:26,751 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:29:28,795 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:29:30,870 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:29:31,580 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:29:32,958 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:29:32,978 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:29:34,999 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:29:35,798 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:29:36,802 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:29:37,018 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:29:37,279 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:29:37,280 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:29:39,064 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:29:41,106 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:29:42,557 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:29:43,154 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:29:45,209 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:29:47,236 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:29:47,246 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:29:47,702 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:29:49,281 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:29:51,325 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:29:52,288 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:29:52,289 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:29:53,357 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:29:53,551 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:29:55,400 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:29:57,284 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 03:29:57,285 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 03:29:57,285 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 03:29:57,286 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 03:29:57,434 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 03:29:57,446 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:29:58,556 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:29:59,474 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:29:59,488 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:30:01,523 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:30:03,625 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:30:03,638 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:30:05,656 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:30:05,805 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:30:07,301 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:30:07,301 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:30:07,671 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:30:09,570 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:30:09,703 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:30:11,740 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:30:12,742 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:30:13,787 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:30:14,654 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:30:15,826 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:30:17,877 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:30:19,690 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:30:19,923 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:30:21,979 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:30:22,305 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:30:22,305 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:30:24,025 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:30:25,019 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:30:25,152 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:30:26,058 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:30:28,102 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:30:30,151 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:30:30,197 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:30:32,217 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:30:34,325 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:30:35,263 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:30:35,815 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:30:36,347 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:30:37,328 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:30:37,329 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:30:37,330 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:30:38,376 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:30:40,402 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:30:40,606 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:30:42,450 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:30:44,493 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:30:45,668 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:30:46,531 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:30:48,577 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:30:49,573 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:30:50,614 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:30:51,355 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:30:52,336 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:30:52,336 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:30:52,644 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:30:54,694 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:30:56,609 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:30:56,748 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:30:58,792 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:31:00,867 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:31:01,959 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:31:02,897 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:31:02,904 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:31:05,032 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:31:05,819 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:31:07,014 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:31:07,052 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:31:07,354 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:31:07,355 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:31:09,083 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:31:11,129 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:31:12,640 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:31:13,170 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:31:15,206 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:31:15,215 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:31:17,253 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:31:18,600 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:31:19,290 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:31:21,328 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:31:22,373 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:31:22,373 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:31:23,369 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:31:23,640 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:31:25,404 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:31:27,436 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:31:27,443 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:31:29,155 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:31:29,484 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:31:31,529 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:31:33,580 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:31:34,185 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:31:35,694 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:31:35,832 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:31:37,381 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:31:37,382 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:31:37,720 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:31:39,725 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:31:39,744 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:31:41,763 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:31:41,779 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:31:43,837 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:31:44,819 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:31:45,865 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:31:47,921 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:31:49,963 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:31:49,973 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:31:52,033 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:31:52,388 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:31:52,388 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:31:54,064 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:31:54,078 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:31:55,657 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:31:56,109 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:31:58,160 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:32:00,205 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:32:00,704 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:32:02,253 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:32:03,962 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 03:32:03,964 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 03:32:03,964 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 03:32:03,965 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 03:32:04,309 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 03:32:04,312 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:32:05,324 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:32:05,843 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:32:05,844 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:32:06,434 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:32:07,404 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:32:07,405 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:32:08,458 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:32:10,483 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:32:11,693 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:32:12,498 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:32:14,544 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:32:16,594 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:32:16,740 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:32:18,633 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:32:19,639 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:32:20,678 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:32:22,056 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:32:22,400 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:32:22,400 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:32:22,721 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:32:24,780 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:32:26,842 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:32:27,679 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:32:28,900 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:32:30,944 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:32:31,946 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:32:32,994 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:32:33,010 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:32:35,040 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:32:35,843 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:32:37,149 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:32:37,417 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:32:37,417 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:32:38,683 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:32:39,168 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:32:41,193 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:32:43,241 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:32:44,233 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:32:44,662 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:32:45,282 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:32:47,327 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:32:49,376 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:32:49,702 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:32:51,414 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:32:52,436 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:32:52,454 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:32:53,467 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:32:54,739 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:32:55,515 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:32:55,526 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:32:57,561 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:32:59,610 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:33:00,287 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:33:01,665 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:33:03,713 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:33:05,339 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:33:05,754 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:33:05,859 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:33:07,445 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:33:07,446 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:33:07,866 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:33:09,852 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:33:09,876 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:33:11,014 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:33:11,901 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:33:13,925 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:33:15,959 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:33:16,056 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:33:18,007 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:33:20,041 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:33:21,686 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:33:22,084 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:33:22,092 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:33:22,471 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:33:22,472 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:33:24,142 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:33:26,189 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:33:26,767 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:33:28,243 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:33:30,287 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:33:31,794 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:33:32,340 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:33:34,378 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:33:34,386 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:33:35,858 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:33:36,435 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:33:36,865 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:33:37,476 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:33:37,477 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:33:38,552 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:33:40,563 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:33:42,576 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:33:42,752 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:33:44,621 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:33:46,667 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:33:47,661 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:33:47,911 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:33:48,711 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:33:50,758 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:33:52,490 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:33:52,490 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:33:52,814 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:33:53,748 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:33:54,858 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:33:56,905 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:33:58,960 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:33:59,604 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:33:59,954 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:34:01,030 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:34:03,083 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:34:04,630 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:34:05,136 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:34:05,859 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:34:07,207 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:34:07,498 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:34:07,498 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:34:09,318 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:34:09,771 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:34:11,288 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 03:34:11,289 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 03:34:11,289 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 03:34:11,290 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 03:34:11,291 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 03:34:11,341 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:34:12,298 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:34:13,364 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:34:14,335 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:34:15,389 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:34:15,584 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:34:17,415 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:34:19,448 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:34:20,637 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:34:21,490 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:34:22,511 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:34:22,512 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:34:23,525 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:34:25,563 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:34:26,468 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:34:26,565 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:34:27,621 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:34:29,669 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:34:31,523 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:34:31,708 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:34:33,742 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:34:35,786 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:34:35,867 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:34:36,881 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:34:37,527 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:34:37,528 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:34:37,843 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:34:38,834 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:34:39,955 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:34:41,969 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:34:42,812 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:34:43,993 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:34:46,010 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:34:47,842 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:34:48,060 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:34:50,094 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:34:50,102 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:34:52,165 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:34:52,523 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:34:52,524 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:34:53,781 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:34:54,218 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:34:56,258 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:34:58,300 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:34:58,802 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:35:00,333 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:35:02,386 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:35:04,428 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:35:04,437 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:35:04,525 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:35:05,877 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:35:06,476 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:35:07,541 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:35:07,541 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:35:08,523 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:35:09,805 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:35:10,631 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:35:12,640 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:35:14,656 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:35:14,846 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:35:16,685 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:35:16,696 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:35:18,734 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:35:20,245 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:35:20,777 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:35:22,541 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:35:22,541 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:35:22,821 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:35:24,864 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:35:25,863 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:35:26,913 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:35:28,961 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:35:28,973 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:35:30,888 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:35:31,026 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:35:33,057 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:35:35,101 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:35:35,891 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:35:35,892 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:35:37,138 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:35:37,555 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:35:37,556 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:35:39,181 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:35:41,303 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:35:41,429 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:35:42,280 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:35:43,322 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:35:45,339 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:35:46,489 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:35:47,378 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:35:49,420 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:35:51,463 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:35:51,533 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:35:52,577 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:35:52,577 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:35:53,506 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:35:54,505 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:35:55,554 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:35:57,004 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:35:57,584 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:35:59,622 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:36:01,661 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:36:02,056 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:36:03,707 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:36:05,752 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:36:05,903 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:36:06,756 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:36:07,151 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:36:07,581 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:36:07,582 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:36:07,793 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:36:09,858 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:36:11,985 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:36:12,919 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:36:14,009 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:36:16,020 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:36:17,970 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:36:18,044 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:36:18,262 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 03:36:18,263 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 03:36:18,264 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 03:36:18,265 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 03:36:19,042 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 03:36:20,098 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:36:21,094 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:36:22,144 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:36:22,591 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:36:22,591 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:36:23,851 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:36:24,201 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:36:26,253 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:36:28,300 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:36:28,907 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:36:30,348 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:36:32,397 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:36:32,405 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:36:34,452 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:36:34,644 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:36:35,906 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:36:36,496 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:36:37,603 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:36:37,603 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:36:38,546 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:36:39,863 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:36:40,587 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:36:42,698 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:36:44,692 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:36:44,711 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:36:45,284 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:36:46,750 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:36:48,801 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:36:50,320 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:36:50,834 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:36:52,625 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:36:52,626 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:36:52,875 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:36:54,910 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:36:55,893 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:36:56,952 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:36:58,978 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:36:58,986 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:37:01,011 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:37:01,048 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:37:03,086 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:37:05,142 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:37:05,909 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:37:06,919 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:37:07,186 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:37:07,631 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:37:07,631 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:37:09,238 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:37:11,288 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:37:11,299 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:37:12,584 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:37:13,424 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:37:15,436 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:37:17,469 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:37:17,623 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:37:19,508 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:37:21,549 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:37:22,649 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:37:22,649 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:37:22,892 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:37:23,587 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:37:23,594 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:37:25,641 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:37:27,686 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:37:27,927 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:37:29,725 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:37:31,772 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:37:32,976 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:37:33,817 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:37:35,865 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:37:35,918 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:37:36,861 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:37:37,669 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:37:37,670 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:37:37,907 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:37:38,939 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:37:39,952 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:37:41,992 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:37:44,030 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:37:44,088 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:37:46,107 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:37:48,129 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:37:49,128 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:37:49,522 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:37:50,164 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:37:52,223 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:37:52,681 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:37:52,682 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:37:54,271 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:37:54,951 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:37:56,302 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:37:58,356 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:37:59,997 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:38:00,385 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:38:01,384 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:38:02,432 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:38:04,473 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:38:05,230 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:38:05,923 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:38:06,516 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:38:07,691 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:38:07,691 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:38:08,570 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:38:10,613 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:38:10,942 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:38:12,672 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:38:14,794 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:38:15,749 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:38:16,819 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:38:16,883 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:38:18,831 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:38:20,857 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:38:21,936 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:38:22,693 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:38:22,693 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:38:22,883 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:38:24,932 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:38:25,536 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 03:38:25,537 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 03:38:25,537 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 03:38:25,538 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 03:38:25,933 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 03:38:26,969 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:38:26,982 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:38:27,842 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:38:29,021 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:38:31,086 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:38:32,873 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:38:33,123 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:38:35,157 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:38:35,936 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:38:37,208 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:38:37,700 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:38:37,700 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:38:37,951 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:38:39,244 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:38:41,295 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:38:41,306 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:38:42,982 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:38:43,348 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:38:45,450 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:38:47,461 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:38:48,018 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:38:49,484 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:38:51,525 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:38:52,704 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:38:52,704 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:38:53,559 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:38:53,569 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:38:53,948 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:38:55,620 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:38:57,667 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:38:58,995 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:38:59,705 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:39:01,747 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:39:03,794 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:39:04,045 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:39:05,821 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:39:05,836 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:39:05,940 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:39:07,711 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:39:07,711 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:39:07,865 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:39:09,902 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:39:09,989 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:39:11,965 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:39:14,000 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:39:15,050 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:39:16,110 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:39:18,125 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:39:19,095 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:39:20,148 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:39:20,171 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:39:22,173 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:39:22,729 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:39:22,729 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:39:24,194 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:39:25,997 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:39:26,227 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:39:28,270 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:39:30,313 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:39:31,315 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:39:31,832 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:39:32,364 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:39:34,400 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:39:35,945 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:39:36,445 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:39:36,958 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:39:37,749 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:39:37,749 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:39:38,480 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:39:40,523 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:39:42,035 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:39:42,576 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:39:43,568 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:39:44,602 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:39:46,727 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:39:47,521 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:39:48,752 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:39:50,779 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:39:52,743 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:39:52,743 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:39:52,805 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:39:52,986 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:39:54,849 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:39:55,842 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:39:56,909 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:39:58,141 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:39:58,946 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:40:00,995 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:40:03,032 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:40:03,196 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:40:05,076 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:40:05,947 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:40:07,112 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:40:07,765 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:40:07,765 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:40:09,028 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:40:09,150 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:40:10,152 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:40:11,193 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:40:13,241 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:40:14,078 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:40:15,293 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:40:17,425 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:40:19,119 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:40:19,446 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:40:21,438 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:40:21,473 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:40:22,769 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:40:22,770 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:40:23,501 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:40:25,036 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:40:25,563 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:40:27,620 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:40:29,661 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:40:30,089 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:40:31,697 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:40:33,114 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 03:40:33,115 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 03:40:33,115 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 03:40:33,116 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 03:40:33,742 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:40:33,743 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 03:40:33,752 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:40:35,361 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:40:35,776 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:40:35,784 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:40:35,962 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:40:37,775 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:40:37,776 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:40:37,831 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:40:39,874 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:40:41,061 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:40:41,938 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:40:43,971 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:40:46,011 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:40:46,115 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:40:48,095 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:40:48,118 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:40:50,139 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:40:51,523 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:40:52,161 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:40:52,790 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:40:52,791 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:40:54,192 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:40:56,238 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:40:57,073 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:40:58,278 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:41:00,311 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:41:02,189 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:41:02,351 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:41:02,359 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:41:04,397 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:41:05,965 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:41:06,437 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:41:07,309 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:41:07,797 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:41:07,798 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:41:08,477 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:41:10,527 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:41:12,582 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:41:12,853 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:41:14,619 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:41:14,631 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:41:16,675 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:41:17,907 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:41:18,769 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:41:20,785 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:41:22,813 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:41:22,813 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:41:22,820 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:41:23,082 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:41:24,865 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:41:25,872 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:41:26,915 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:41:28,650 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:41:28,954 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:41:31,011 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:41:33,055 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:41:33,683 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:41:35,102 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:41:35,968 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:41:37,144 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:41:37,830 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:41:37,830 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:41:38,142 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:41:39,082 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:41:39,183 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:41:41,235 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:41:43,285 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:41:44,118 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:41:45,318 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:41:47,371 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:41:49,183 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:41:49,463 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:41:51,477 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:41:52,457 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:41:52,836 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:41:52,837 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:41:53,499 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:41:55,096 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:41:55,535 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:41:57,569 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:41:59,606 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:42:00,150 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:42:01,650 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:42:03,701 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:42:04,702 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:42:05,743 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:42:05,744 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:42:05,983 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:42:07,780 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:42:07,849 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:42:07,850 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:42:09,821 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:42:11,125 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:42:11,863 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:42:13,904 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:42:15,948 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:42:15,952 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:42:16,489 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:42:17,998 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:42:20,105 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:42:21,543 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:42:22,118 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:42:22,854 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:42:22,854 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:42:24,139 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:42:26,179 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:42:27,169 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:42:28,232 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:42:30,262 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:42:30,271 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:42:32,273 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:42:32,333 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:42:34,380 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:42:35,985 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:42:36,427 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:42:37,374 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:42:37,866 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:42:37,866 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:42:38,463 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:42:40,503 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:42:40,938 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 03:42:40,940 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 03:42:40,940 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 03:42:40,943 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 03:42:41,501 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 03:42:42,537 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:42:42,545 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:42:43,223 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:42:44,584 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:42:46,629 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:42:48,267 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:42:48,673 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:42:50,782 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:42:52,802 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:42:52,882 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:42:52,882 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:42:54,128 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:42:54,811 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:42:56,826 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:42:56,836 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:42:58,872 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:42:59,287 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:43:00,935 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:43:02,977 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:43:04,318 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:43:05,018 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:43:05,985 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:43:07,062 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:43:07,904 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:43:07,904 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:43:09,104 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:43:09,112 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:43:10,185 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:43:11,158 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:43:13,202 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:43:15,225 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:43:15,234 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:43:17,291 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:43:19,334 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:43:20,330 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:43:20,672 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:43:21,443 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:43:22,915 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:43:22,915 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:43:23,469 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:43:25,490 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:43:26,192 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:43:27,525 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:43:29,560 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:43:31,229 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:43:31,608 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:43:33,676 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:43:34,657 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:43:35,721 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:43:35,999 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:43:37,005 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:43:37,770 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:43:37,933 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:43:37,933 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:43:39,824 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:43:41,871 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:43:42,216 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:43:43,928 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:43:45,980 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:43:46,988 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:43:48,024 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:43:48,029 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:43:50,073 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:43:52,210 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:43:52,950 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:43:52,950 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:43:53,208 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:43:54,237 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:43:56,259 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:43:58,271 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:43:58,720 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:43:59,274 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:44:00,314 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:44:02,350 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:44:03,752 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:44:04,394 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:44:06,013 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:44:06,447 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:44:07,968 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:44:07,968 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:44:08,482 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:44:09,229 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:44:10,520 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:44:12,561 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:44:12,570 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:44:14,434 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:44:14,618 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:44:16,654 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:44:18,696 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:44:19,473 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:44:20,735 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:44:22,841 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:44:22,976 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:44:22,977 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:44:24,828 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:44:24,865 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:44:25,248 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:44:26,881 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:44:28,919 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:44:30,294 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:44:30,972 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:44:33,015 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:44:35,058 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:44:35,344 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:44:36,025 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:44:37,091 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:44:37,104 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:44:37,987 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:44:37,988 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:44:39,165 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:44:41,196 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:44:41,286 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:44:43,241 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:44:45,285 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:44:46,337 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:44:47,343 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:44:48,495 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 03:44:48,496 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 03:44:48,497 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 03:44:48,498 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 03:44:49,374 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 03:44:49,387 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:44:51,421 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:44:51,430 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:44:51,782 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:44:53,000 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:44:53,001 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:44:53,536 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:44:55,554 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:44:57,277 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:44:57,569 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:44:59,598 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:45:01,637 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:45:02,631 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:45:02,771 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:45:03,677 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:45:05,723 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:45:06,032 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:45:07,762 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:45:08,005 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:45:08,005 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:45:08,246 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:45:09,801 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:45:11,851 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:45:13,276 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:45:13,889 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:45:14,885 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:45:15,942 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:45:17,977 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:45:18,469 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:45:20,026 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:45:22,073 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:45:23,021 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:45:23,022 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:45:24,176 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:45:24,278 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:45:26,198 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:45:28,219 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:45:29,213 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:45:30,126 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:45:30,266 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:45:32,315 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:45:34,361 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:45:35,179 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:45:36,034 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:45:36,391 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:45:38,027 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:45:38,028 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:45:38,435 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:45:40,474 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:45:40,917 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:45:41,470 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:45:42,515 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:45:44,574 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:45:45,980 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:45:46,606 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:45:48,658 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:45:50,695 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:45:51,034 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:45:52,738 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:45:53,041 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:45:53,041 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:45:54,842 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:45:55,797 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:45:56,319 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:45:56,868 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:45:58,882 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:46:00,939 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:46:01,368 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:46:02,989 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:46:05,029 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:46:06,045 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:46:07,060 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:46:07,061 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:46:07,074 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:46:08,047 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:46:08,048 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:46:09,116 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:46:11,149 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:46:12,329 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:46:13,187 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:46:15,235 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:46:17,276 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:46:17,370 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:46:19,303 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:46:19,311 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:46:21,352 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:46:22,513 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:46:23,046 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:46:23,046 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:46:23,392 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:46:25,497 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:46:27,511 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:46:28,337 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:46:29,527 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:46:31,564 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:46:31,574 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:46:33,620 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:46:33,733 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:46:35,657 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:46:36,055 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:46:37,691 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:46:38,061 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:46:38,062 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:46:39,305 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:46:39,737 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:46:41,775 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:46:43,824 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:46:44,439 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:46:45,864 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:46:45,869 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:46:47,931 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:46:49,486 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:46:49,982 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:46:52,036 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:46:53,064 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:46:53,064 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:46:54,087 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:46:55,342 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:46:56,153 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 03:46:56,154 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 03:46:56,168 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 03:46:56,184 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 03:46:56,195 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:46:57,169 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:46:57,169 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 03:46:58,211 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:47:00,239 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:47:00,411 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:47:02,266 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:47:04,306 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:47:05,449 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:47:06,066 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:47:06,341 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:47:08,068 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:47:08,069 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:47:08,380 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:47:10,425 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:47:10,535 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:47:11,417 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:47:12,472 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:47:14,506 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:47:15,586 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:47:16,547 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:47:18,591 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:47:20,635 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:47:20,636 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:47:22,676 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:47:23,078 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:47:23,078 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:47:23,677 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:47:24,718 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:47:26,331 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:47:26,822 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:47:28,845 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:47:30,873 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:47:31,369 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:47:32,906 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:47:34,949 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:47:36,073 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:47:36,994 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:47:37,079 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:47:37,999 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:47:38,077 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:47:38,078 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:47:39,029 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:47:41,071 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:47:42,353 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:47:43,127 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:47:45,182 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:47:47,229 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:47:47,400 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:47:49,273 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:47:50,267 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:47:51,320 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:47:52,582 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:47:53,072 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:47:53,072 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:47:53,366 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:47:55,427 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:47:57,531 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:47:58,410 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:47:59,541 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:48:01,548 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:48:01,566 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:48:03,609 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:48:03,686 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:48:05,672 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:48:06,074 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:48:07,721 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:48:08,088 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:48:08,088 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:48:09,345 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:48:09,774 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:48:11,829 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:48:13,876 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:48:14,698 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:48:15,918 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:48:15,930 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:48:17,968 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:48:19,739 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:48:20,008 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:48:22,057 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:48:23,098 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:48:23,098 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:48:24,092 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:48:25,346 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:48:26,135 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:48:28,235 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:48:28,262 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:48:30,286 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:48:30,557 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:48:32,310 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:48:34,326 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:48:35,607 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:48:36,080 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:48:36,371 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:48:38,093 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:48:38,093 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:48:38,414 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:48:40,454 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:48:41,346 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:48:42,487 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:48:42,497 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:48:44,541 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:48:46,389 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:48:46,572 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:48:48,606 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:48:50,658 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:48:51,439 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:48:52,705 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:48:53,097 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:48:53,097 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:48:53,695 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:48:54,745 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:48:56,791 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:48:57,382 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:48:58,897 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:49:00,915 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:49:02,450 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:49:02,938 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:49:04,815 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 03:49:04,817 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 03:49:04,817 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 03:49:04,818 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 03:49:04,957 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 03:49:04,966 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:49:05,964 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:49:06,092 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:49:07,009 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:49:07,612 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:49:08,104 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:49:08,104 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:49:09,057 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:49:11,106 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:49:13,149 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:49:13,401 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:49:15,193 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:49:17,233 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:49:19,260 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:49:19,268 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:49:20,264 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:49:21,321 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:49:23,109 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:49:23,110 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:49:23,356 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:49:24,372 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:49:25,408 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:49:27,454 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:49:29,505 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:49:29,565 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:49:31,587 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:49:32,572 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:49:33,612 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:49:35,055 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:49:35,657 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:49:36,098 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:49:37,705 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:49:38,116 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:49:38,117 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:49:39,749 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:49:40,383 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:49:41,793 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:49:43,841 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:49:45,762 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:49:45,878 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:49:45,888 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:49:47,934 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:49:49,985 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:49:50,818 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:49:52,061 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:49:53,128 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:49:53,128 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:49:54,101 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:49:56,147 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:49:56,387 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:49:58,177 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:49:58,188 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:50:00,294 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:50:01,479 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:50:02,308 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:50:04,330 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:50:06,113 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:50:06,345 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:50:07,122 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:50:08,119 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:50:08,120 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:50:08,396 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:50:10,430 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:50:10,441 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:50:12,317 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:50:12,487 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:50:14,528 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:50:16,569 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:50:17,342 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:50:18,616 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:50:20,654 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:50:22,632 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:50:22,694 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:50:23,134 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:50:23,134 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:50:24,725 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:50:24,732 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:50:26,789 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:50:28,437 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:50:28,829 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:50:30,947 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:50:32,969 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:50:33,494 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:50:34,988 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:50:36,127 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:50:37,004 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:50:37,013 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:50:38,150 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:50:38,151 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:50:39,064 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:50:39,413 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:50:41,100 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:50:43,149 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:50:44,463 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:50:45,183 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:50:47,225 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:50:48,226 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:50:49,275 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:50:49,808 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:50:51,327 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:50:53,166 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:50:53,166 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:50:53,364 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:50:55,400 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:50:55,438 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:50:57,442 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:50:59,489 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:51:00,497 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:51:01,588 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:51:02,566 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:51:03,611 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:51:05,595 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:51:05,634 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:51:06,131 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:51:07,652 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:51:08,182 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:51:08,182 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:51:09,678 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:51:11,473 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:51:11,725 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:51:13,295 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 03:51:13,297 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 03:51:13,297 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 03:51:13,298 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 03:51:13,785 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 03:51:13,788 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:51:14,791 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:51:15,853 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:51:16,562 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:51:17,891 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:51:19,942 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:51:21,611 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:51:21,992 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:51:23,193 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:51:23,193 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:51:24,023 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:51:26,062 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:51:26,628 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:51:28,107 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:51:29,105 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:51:30,141 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:51:31,667 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:51:32,266 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:51:34,285 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:51:36,146 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:51:36,309 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:51:37,156 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:51:38,201 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:51:38,202 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:51:38,336 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:51:40,372 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:51:40,379 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:51:42,423 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:51:42,466 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:51:44,474 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:51:46,516 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:51:47,516 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:51:48,559 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:51:50,592 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:51:52,641 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:51:52,701 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:51:53,207 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:51:53,208 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:51:54,681 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:51:54,693 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:51:56,737 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:51:58,515 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:51:58,785 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:52:00,837 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:52:02,957 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:52:03,561 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:52:04,980 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:52:06,159 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:52:06,985 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:52:06,993 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:52:08,216 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:52:08,217 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:52:09,032 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:52:09,482 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:52:11,077 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:52:13,112 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:52:14,523 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:52:15,161 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:52:17,193 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:52:19,228 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:52:19,240 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:52:19,931 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:52:21,286 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:52:23,234 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:52:23,234 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:52:23,332 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:52:25,378 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:52:25,508 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:52:27,429 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:52:29,469 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:52:30,537 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:52:31,508 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:52:33,580 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:52:33,614 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:52:35,635 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:52:35,681 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:52:36,174 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:52:37,661 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:52:38,241 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:52:38,241 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:52:39,694 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:52:41,512 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:52:41,742 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:52:43,781 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:52:44,783 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:52:45,811 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:52:47,466 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:52:47,864 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:52:49,893 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:52:51,948 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:52:52,510 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:52:53,253 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:52:53,253 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:52:53,985 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:52:56,025 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:52:57,023 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:52:58,076 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:52:58,186 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:53:00,137 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:53:02,171 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:53:03,230 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:53:04,311 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:53:06,189 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:53:06,323 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:53:08,266 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:53:08,266 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:53:08,372 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:53:08,527 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:53:10,421 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:53:11,430 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:53:12,474 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:53:13,954 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:53:14,516 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:53:16,556 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:53:18,603 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:53:18,992 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:53:20,635 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:53:21,693 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 03:53:21,695 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 03:53:21,695 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 03:53:21,696 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 03:53:22,675 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:53:22,676 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 03:53:22,689 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:53:23,272 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:53:23,273 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:53:23,680 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:53:24,519 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:53:24,720 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:53:26,767 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:53:28,804 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:53:29,572 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:53:30,873 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:53:32,902 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:53:34,615 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:53:35,013 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:53:36,199 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:53:37,003 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:53:37,022 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:53:38,292 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:53:38,292 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:53:39,054 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:53:40,569 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:53:41,095 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:53:43,140 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:53:45,186 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:53:45,621 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:53:47,240 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:53:49,273 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:53:49,282 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:53:50,777 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:53:51,323 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:53:53,303 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:53:53,304 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:53:53,373 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:53:55,418 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:53:56,577 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:53:57,462 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:53:59,507 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:54:01,544 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:54:01,551 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:54:02,504 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:54:03,611 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:54:05,736 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:54:06,201 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:54:07,754 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:54:07,794 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:54:08,314 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:54:08,314 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:54:09,774 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:54:11,814 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:54:13,256 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:54:13,857 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:54:15,892 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:54:15,903 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:54:17,948 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:54:18,294 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:54:19,992 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:54:22,030 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:54:23,318 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:54:23,318 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:54:23,569 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:54:24,073 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:54:26,117 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:54:28,152 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:54:28,161 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:54:29,027 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:54:30,203 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:54:32,265 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:54:34,069 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:54:34,304 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:54:36,216 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:54:36,420 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:54:38,334 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:54:38,334 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:54:38,437 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:54:39,406 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:54:39,831 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:54:40,458 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:54:42,496 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:54:44,583 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:54:44,877 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:54:46,653 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:54:48,695 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:54:49,934 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:54:50,754 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:54:52,823 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:54:53,355 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:54:53,356 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:54:53,814 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:54:54,873 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:54:55,637 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:54:56,918 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:54:58,971 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:55:00,688 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:55:01,038 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:55:03,099 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:55:05,148 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:55:06,229 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:55:06,230 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:55:07,291 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:55:08,241 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:55:08,365 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:55:08,365 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:55:09,309 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:55:11,332 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:55:11,648 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:55:13,374 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:55:15,421 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:55:16,694 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:55:17,487 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:55:19,561 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:55:19,572 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:55:21,628 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:55:22,521 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:55:23,391 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:55:23,391 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:55:23,675 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:55:25,716 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:55:27,745 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:55:27,759 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:55:29,823 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:55:31,641 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 03:55:31,643 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 03:55:31,644 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 03:55:31,645 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 03:55:31,853 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 03:55:31,866 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:55:32,939 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:55:33,899 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:55:33,907 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:55:35,968 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:55:36,231 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:55:38,077 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:55:38,397 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:55:38,397 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:55:38,653 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:55:40,090 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:55:42,114 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:55:43,695 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:55:44,159 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:55:46,218 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:55:48,244 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:55:48,253 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:55:49,674 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:55:50,340 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:55:52,402 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:55:53,415 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:55:53,416 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:55:54,444 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:55:55,673 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:55:56,499 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:55:58,540 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:55:59,533 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:56:00,619 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:56:01,679 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:56:02,676 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:56:04,725 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:56:06,230 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:56:06,772 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:56:07,238 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:56:08,438 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:56:08,438 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:56:08,893 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:56:10,904 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:56:12,660 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:56:12,922 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:56:13,909 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:56:14,952 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:56:17,003 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:56:17,713 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:56:19,048 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:56:21,089 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:56:22,768 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:56:23,138 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:56:23,448 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:56:23,449 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:56:25,191 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:56:26,191 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:56:27,247 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:56:28,580 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:56:29,301 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:56:31,351 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:56:33,398 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:56:33,619 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:56:35,446 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:56:36,237 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:56:37,493 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:56:38,465 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:56:38,466 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:56:38,497 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:56:38,723 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:56:39,614 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:56:41,640 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:56:43,663 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:56:43,774 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:56:45,688 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:56:47,730 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:56:48,810 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:56:49,780 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:56:51,804 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:56:51,814 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:56:53,488 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:56:53,488 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:56:53,865 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:56:54,743 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:56:55,919 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:56:57,966 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:56:59,804 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:57:00,011 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:57:02,062 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:57:04,115 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:57:04,117 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:57:05,300 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:57:06,177 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:57:06,252 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:57:08,227 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:57:08,494 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:57:08,494 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:57:10,342 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:57:10,773 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:57:12,357 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:57:14,384 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:57:15,836 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:57:16,433 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:57:18,478 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:57:18,494 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:57:20,527 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:57:21,254 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:57:22,584 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:57:23,513 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:57:23,513 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:57:24,624 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:57:26,680 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:57:26,796 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:57:28,722 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:57:30,766 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:57:30,784 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:57:32,099 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:57:32,811 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:57:34,863 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:57:36,265 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:57:36,914 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:57:37,277 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:57:38,533 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:57:38,534 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:57:38,965 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:57:41,080 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:57:42,026 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 03:57:42,028 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 03:57:42,028 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 03:57:42,030 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 03:57:42,042 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 03:57:42,321 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:57:43,107 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:57:44,074 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:57:45,132 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:57:47,174 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:57:47,381 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:57:49,216 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:57:51,270 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:57:52,421 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:57:53,313 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:57:53,541 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:57:53,542 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:57:55,353 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:57:57,392 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:57:57,758 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:57:58,391 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:57:59,431 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:58:01,466 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:58:02,810 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:58:03,519 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:58:05,564 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:58:06,267 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:58:07,609 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:58:07,899 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:58:08,559 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:58:08,560 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:58:09,662 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:58:10,662 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:58:11,782 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:58:13,804 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:58:13,867 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:58:15,820 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:58:17,845 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:58:18,914 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:58:19,894 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:58:21,943 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:58:22,938 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:58:23,565 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:58:23,565 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:58:23,991 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:58:24,827 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:58:26,036 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:58:28,077 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:58:29,874 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:58:30,129 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:58:32,218 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:58:34,266 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:58:35,686 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:58:36,315 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:58:36,317 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:58:36,333 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:58:38,373 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:58:38,560 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:58:38,561 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:58:40,423 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:58:40,833 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:58:42,554 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:58:44,575 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:58:45,877 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:58:46,596 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:58:48,628 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:58:48,643 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:58:50,684 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:58:51,687 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:58:52,736 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:58:53,565 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:58:53,565 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:58:54,784 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:58:56,824 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:58:56,824 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:58:58,864 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:59:00,927 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:59:02,650 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:59:02,965 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:59:02,976 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:59:05,010 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:59:06,307 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:59:07,060 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:59:07,934 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:59:08,562 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:59:08,563 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:59:09,106 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:59:11,153 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:59:13,277 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:59:13,621 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:59:15,265 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:59:15,304 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:59:17,328 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:59:18,665 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:59:19,378 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:59:21,424 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:59:23,489 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:59:23,574 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:59:23,574 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:59:23,834 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:59:25,542 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:59:27,596 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:59:28,599 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:59:29,624 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:59:29,633 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:59:31,680 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:59:33,726 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:59:34,713 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:59:35,780 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:59:36,313 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 03:59:37,829 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:59:38,589 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:59:38,590 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:59:39,864 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:59:40,562 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:59:40,858 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:59:41,912 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:59:44,032 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:59:45,603 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:59:46,054 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:59:48,077 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:59:50,137 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:59:50,650 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:59:52,205 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:59:52,518 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 03:59:52,519 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 03:59:52,520 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 03:59:52,521 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 03:59:53,207 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:59:53,208 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 03:59:53,594 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 03:59:53,595 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 03:59:54,244 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:59:55,241 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 03:59:55,871 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 03:59:56,292 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 03:59:58,326 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:00:00,368 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:00:00,943 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:00:02,414 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:00:04,468 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:00:06,004 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:00:06,320 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:00:06,515 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:00:07,508 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:00:08,549 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:00:08,598 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:00:08,599 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:00:10,585 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:00:11,894 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:00:12,637 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:00:14,755 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:00:16,783 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:00:16,943 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:00:18,796 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:00:20,835 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:00:20,842 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:00:22,199 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:00:22,897 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:00:23,607 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:00:23,608 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:00:24,939 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:00:26,975 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:00:27,899 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:00:29,033 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:00:31,085 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:00:33,138 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:00:33,145 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:00:33,147 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:00:35,187 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:00:36,328 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:00:37,236 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:00:38,613 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:00:38,614 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:00:38,869 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:00:39,289 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:00:41,337 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:00:43,388 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:00:43,889 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:00:45,517 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:00:47,493 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:00:47,550 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:00:49,133 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:00:49,575 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:00:51,602 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:00:53,632 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:00:53,632 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:00:53,645 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:00:54,898 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:00:55,690 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:00:57,734 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:00:59,759 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:00:59,769 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:01:00,103 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:01:01,816 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:01:03,866 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:01:05,153 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:01:05,909 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:01:06,337 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:01:07,946 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:01:08,645 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:01:08,645 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:01:09,998 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:01:10,907 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:01:12,063 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:01:13,059 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:01:14,110 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:01:16,144 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:01:16,239 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:01:18,251 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:01:20,275 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:01:21,269 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:01:22,329 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:01:23,645 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:01:23,646 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:01:24,380 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:01:25,383 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:01:26,424 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:01:27,124 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:01:28,479 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:01:30,593 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:01:32,165 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:01:32,646 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:01:34,690 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:01:36,353 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:01:36,750 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:01:37,295 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:01:38,644 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:01:38,644 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:01:38,813 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:01:39,800 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:01:40,863 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:01:42,923 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:01:42,928 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:01:44,969 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:01:47,105 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:01:48,000 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:01:49,129 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:01:51,154 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:01:52,135 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:01:53,118 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:01:53,206 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:01:53,656 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:01:53,656 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:01:55,256 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:01:57,302 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:01:58,954 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:01:59,372 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:02:01,435 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:02:03,489 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:02:04,000 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:02:04,068 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 04:02:04,070 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 04:02:04,070 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 04:02:04,071 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 04:02:04,496 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 04:02:05,535 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:02:05,551 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:02:06,362 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:02:07,636 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:02:08,663 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:02:08,664 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:02:09,689 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:02:09,937 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:02:11,748 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:02:13,809 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:02:14,996 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:02:15,871 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:02:18,010 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:02:19,974 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:02:20,029 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:02:20,493 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:02:22,049 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:02:23,681 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:02:23,682 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:02:24,096 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:02:25,959 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:02:26,156 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:02:28,194 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:02:30,256 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:02:30,994 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:02:32,310 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:02:34,353 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:02:34,366 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:02:36,396 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:02:36,397 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:02:36,410 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:02:38,460 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:02:38,692 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:02:38,693 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:02:40,514 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:02:41,981 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:02:42,555 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:02:44,606 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:02:45,601 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:02:46,659 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:02:47,664 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:02:48,766 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:02:50,782 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:02:52,706 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:02:52,795 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:02:53,684 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:02:53,684 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:02:54,845 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:02:56,884 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:02:57,890 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:02:58,298 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:02:58,923 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:03:00,979 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:03:03,008 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:03:03,323 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:03:05,061 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:03:06,376 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:03:07,103 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:03:08,695 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:03:08,696 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:03:08,938 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:03:09,139 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:03:11,193 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:03:12,190 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:03:13,226 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:03:14,228 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:03:15,271 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:03:17,320 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:03:19,265 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:03:19,420 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:03:21,443 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:03:23,471 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:03:23,712 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:03:23,712 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:03:24,464 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:03:24,983 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:03:25,510 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:03:27,558 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:03:29,601 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:03:30,023 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:03:31,647 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:03:33,686 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:03:35,071 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:03:35,737 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:03:36,386 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:03:37,758 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:03:37,766 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:03:38,733 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:03:38,733 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:03:39,805 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:03:41,001 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:03:41,853 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:03:43,901 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:03:45,970 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:03:46,042 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:03:48,018 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:03:50,094 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:03:50,154 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:03:51,873 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:03:52,179 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:03:53,746 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:03:53,746 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:03:54,192 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:03:56,229 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:03:57,031 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:03:58,289 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:04:00,344 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:04:02,379 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:04:02,684 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:04:04,408 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:04:04,416 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:04:06,398 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:04:06,461 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:04:08,191 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:04:08,502 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:04:08,757 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:04:08,758 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:04:10,544 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:04:12,575 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:04:14,079 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:04:14,560 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 04:04:14,561 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 04:04:14,561 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 04:04:14,562 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 04:04:14,611 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 04:04:14,619 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:04:15,621 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:04:16,659 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:04:16,667 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:04:18,712 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:04:19,842 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:04:20,830 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:04:22,845 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:04:23,766 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:04:23,766 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:04:24,866 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:04:25,020 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:04:26,906 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:04:28,949 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:04:29,953 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:04:30,060 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:04:31,000 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:04:33,043 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:04:35,077 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:04:35,099 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:04:36,403 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:04:37,121 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:04:38,766 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:04:38,767 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:04:39,169 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:04:40,767 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:04:41,225 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:04:42,222 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:04:43,277 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:04:45,323 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:04:45,826 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:04:47,358 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:04:49,401 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:04:50,884 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:04:51,520 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:04:53,529 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:04:53,772 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:04:53,772 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:04:55,548 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:04:56,051 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:04:56,528 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:04:57,570 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:04:59,621 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:05:01,098 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:05:01,665 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:05:03,704 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:05:05,742 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:05:06,147 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:05:06,419 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:05:07,790 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:05:08,780 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:05:08,780 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:05:08,782 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:05:09,833 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:05:11,885 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:05:12,053 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:05:13,932 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:05:15,982 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:05:17,095 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:05:18,029 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:05:20,068 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:05:21,069 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:05:22,167 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:05:22,363 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:05:23,792 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:05:23,793 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:05:24,182 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:05:26,227 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:05:28,073 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:05:28,260 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:05:30,296 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:05:32,352 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:05:33,124 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:05:34,381 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:05:34,389 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:05:36,431 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:05:36,441 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:05:38,232 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:05:38,482 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:05:38,799 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:05:38,800 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:05:40,532 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:05:42,567 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:05:44,103 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:05:44,615 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:05:46,654 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:05:46,664 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:05:48,699 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:05:49,977 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:05:50,754 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:05:52,873 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:05:53,815 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:05:53,815 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:05:54,882 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:05:55,081 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:05:56,899 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:05:58,927 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:05:58,934 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:06:00,758 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:06:00,991 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:06:03,024 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:06:05,066 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:06:05,792 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:06:06,433 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:06:07,110 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:06:08,818 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:06:08,819 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:06:09,156 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:06:11,192 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:06:11,586 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:06:13,229 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:06:13,231 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:06:15,277 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:06:16,639 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:06:17,311 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:06:19,346 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:06:21,391 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:06:21,682 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:06:23,458 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 04:06:23,460 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 04:06:23,460 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 04:06:23,493 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 04:06:23,494 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 04:06:23,540 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:06:23,922 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:06:23,922 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:06:24,503 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:06:25,514 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:06:25,557 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:06:27,186 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:06:27,580 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:06:29,601 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:06:31,635 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:06:32,240 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:06:33,693 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:06:35,719 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:06:36,433 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:06:37,771 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:06:37,855 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:06:38,767 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:06:38,876 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:06:38,876 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:06:39,817 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:06:41,879 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:06:43,147 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:06:43,939 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:06:45,998 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:06:48,029 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:06:48,175 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:06:50,074 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:06:51,081 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:06:52,156 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:06:53,264 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:06:53,876 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:06:53,876 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:06:54,286 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:06:56,313 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:06:58,337 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:06:59,163 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:07:00,379 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:07:02,409 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:07:04,451 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:07:04,453 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:07:05,448 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:07:06,436 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:07:06,493 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:07:08,538 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:07:08,895 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:07:08,895 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:07:10,164 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:07:10,585 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:07:12,614 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:07:14,660 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:07:15,215 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:07:16,718 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:07:17,710 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:07:18,765 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:07:20,281 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:07:20,798 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:07:22,837 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:07:23,899 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:07:23,900 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:07:24,944 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:07:26,164 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:07:26,969 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:07:28,991 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:07:29,000 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:07:31,058 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:07:31,206 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:07:33,098 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:07:35,146 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:07:36,268 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:07:36,442 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:07:37,189 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:07:38,910 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:07:38,910 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:07:39,237 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:07:41,302 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:07:42,194 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:07:43,338 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:07:43,348 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:07:45,393 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:07:47,239 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:07:47,424 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:07:49,471 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:07:51,518 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:07:52,284 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:07:53,556 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:07:53,906 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:07:53,906 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:07:55,627 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:07:55,683 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:07:57,696 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:07:58,175 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:07:59,724 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:08:01,782 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:08:03,227 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:08:03,827 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:08:05,871 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:08:06,453 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:08:07,916 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:08:08,315 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:08:08,915 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:08:08,915 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:08:09,944 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:08:09,952 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:08:12,019 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:08:14,054 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:08:14,217 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:08:16,100 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:08:18,145 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:08:19,252 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:08:20,194 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:08:21,197 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:08:22,236 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:08:23,936 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:08:23,936 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:08:24,281 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:08:25,192 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:08:26,397 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:08:28,415 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:08:30,241 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:08:30,443 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:08:32,488 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:08:32,731 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 04:08:32,733 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 04:08:32,733 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 04:08:32,734 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 04:08:33,487 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:08:33,487 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 04:08:34,534 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:08:35,537 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:08:35,990 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:08:36,453 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:08:36,579 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:08:38,622 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:08:38,941 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:08:38,941 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:08:40,658 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:08:41,216 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:08:42,706 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:08:44,751 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:08:46,788 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:08:47,198 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:08:47,793 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:08:48,828 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:08:50,889 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:08:52,240 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:08:52,922 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:08:53,930 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:08:53,931 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:08:54,958 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:08:57,068 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:08:58,220 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:08:59,076 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:09:01,109 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:09:02,114 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:09:03,153 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:09:04,155 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:09:05,190 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:09:06,460 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:09:07,239 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:09:08,938 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:09:08,938 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:09:09,200 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:09:09,290 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:09:11,333 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:09:13,362 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:09:13,370 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:09:14,943 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:09:15,408 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:09:17,460 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:09:19,503 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:09:19,980 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:09:21,543 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:09:23,585 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:09:23,936 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:09:23,937 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:09:25,615 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:09:25,623 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:09:25,707 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:09:27,735 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:09:29,757 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:09:30,746 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:09:31,781 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:09:33,824 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:09:35,806 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:09:35,880 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:09:36,464 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:09:37,923 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:09:38,947 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:09:38,947 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:09:39,962 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:09:39,969 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:09:41,225 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:09:42,010 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:09:44,056 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:09:46,101 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:09:46,260 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:09:48,145 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:09:50,191 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:09:51,280 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:09:52,237 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:09:52,251 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:09:53,955 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:09:53,955 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:09:54,280 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:09:56,341 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:09:57,221 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:09:58,457 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:10:00,470 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:10:02,276 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:10:02,519 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:10:03,523 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:10:04,559 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:10:06,473 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:10:06,593 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:10:07,489 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:10:08,649 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:10:08,959 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:10:08,959 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:10:10,704 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:10:12,765 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:10:13,222 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:10:14,796 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:10:16,838 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:10:17,836 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:10:18,894 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:10:18,969 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:10:20,935 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:10:22,975 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:10:23,983 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:10:23,984 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:10:24,255 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:10:25,015 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:10:27,077 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:10:29,184 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:10:29,781 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:10:30,146 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:10:31,200 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:10:33,216 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:10:34,832 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:10:35,241 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:10:36,473 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:10:37,288 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:10:38,995 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:10:38,996 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:10:39,342 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:10:40,264 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:10:41,386 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:10:41,593 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 04:10:41,594 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 04:10:41,595 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 04:10:41,596 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 04:10:42,393 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 04:10:43,424 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:10:43,432 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:10:44,436 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:10:45,483 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:10:45,889 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:10:47,528 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:10:49,570 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:10:50,938 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:10:51,617 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:10:53,661 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:10:54,009 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:10:54,010 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:10:55,699 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:10:56,221 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:10:56,694 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:10:57,739 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:10:59,857 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:11:01,270 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:11:01,883 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:11:03,893 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:11:05,930 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:11:06,315 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:11:06,476 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:11:07,973 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:11:09,019 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:11:09,020 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:11:10,011 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:11:10,019 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:11:12,083 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:11:12,316 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:11:14,123 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:11:16,173 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:11:17,354 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:11:18,211 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:11:20,264 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:11:22,304 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:11:22,315 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:11:22,928 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:11:24,028 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:11:24,028 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:11:24,358 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:11:26,402 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:11:28,314 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:11:28,435 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:11:30,556 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:11:32,580 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:11:33,365 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:11:34,555 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:11:34,593 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:11:36,483 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:11:36,616 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:11:38,433 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:11:38,654 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:11:39,042 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:11:39,043 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:11:40,689 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:11:42,733 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:11:44,335 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:11:44,793 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:11:46,837 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:11:48,880 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:11:48,887 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:11:49,495 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:11:50,933 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:11:52,985 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:11:54,046 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:11:54,047 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:11:55,021 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:11:55,298 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:11:57,074 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:11:59,118 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:12:00,116 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:12:01,216 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:12:01,300 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:12:03,234 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:12:05,256 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:12:06,349 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:12:06,494 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:12:07,277 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:12:09,059 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:12:09,060 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:12:09,326 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:12:11,369 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:12:12,151 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:12:13,402 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:12:14,398 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:12:15,447 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:12:17,195 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:12:17,488 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:12:19,538 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:12:21,576 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:12:22,249 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:12:23,624 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:12:24,087 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:12:24,087 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:12:25,673 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:12:26,675 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:12:27,714 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:12:27,990 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:12:29,771 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:12:31,915 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:12:33,045 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:12:33,931 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:12:35,947 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:12:36,497 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:12:37,973 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:12:38,459 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:12:39,086 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:12:39,087 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:12:40,006 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:12:41,015 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:12:42,058 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:12:44,108 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:12:44,395 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:12:46,156 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:12:48,209 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:12:49,442 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:12:50,253 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:12:50,692 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 04:12:50,694 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 04:12:50,694 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 04:12:50,695 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 04:12:51,247 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 04:12:52,307 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:12:52,322 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:12:54,108 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:12:54,108 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:12:54,357 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:12:55,374 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:12:56,399 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:12:58,443 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:13:00,431 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:13:00,492 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:13:02,614 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:13:04,619 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:13:04,634 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:13:06,131 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:13:06,506 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:13:06,660 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:13:08,707 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:13:09,119 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:13:09,119 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:13:10,754 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:13:11,414 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:13:12,800 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:13:14,844 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:13:16,465 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:13:16,882 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:13:18,921 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:13:18,930 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:13:20,966 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:13:21,968 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:13:23,001 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:13:24,118 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:13:24,119 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:13:25,052 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:13:27,097 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:13:27,385 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:13:29,156 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:13:31,185 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:13:31,195 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:13:32,802 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:13:33,306 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:13:35,320 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:13:36,520 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:13:37,343 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:13:38,488 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:13:39,117 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:13:39,118 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:13:39,369 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:13:41,410 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:13:43,466 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:13:43,618 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:13:45,504 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:13:45,506 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:13:47,564 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:13:48,657 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:13:49,610 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:13:51,653 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:13:53,686 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:13:54,120 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:13:54,121 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:13:54,372 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:13:55,730 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:13:56,732 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:13:57,783 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:13:59,449 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:13:59,845 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:14:01,899 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:14:04,005 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:14:04,510 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:14:06,022 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:14:06,534 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:14:08,032 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:14:09,134 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:14:09,135 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:14:10,073 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:14:10,389 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:14:11,068 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:14:12,130 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:14:14,164 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:14:15,439 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:14:16,207 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:14:18,265 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:14:20,297 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:14:20,489 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:14:22,347 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:14:23,349 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:14:24,151 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:14:24,151 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:14:24,391 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:14:26,426 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:14:26,428 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:14:28,479 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:14:30,547 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:14:31,482 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:14:32,601 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:14:34,707 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:14:35,697 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:14:36,540 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:14:36,541 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:14:36,722 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:14:38,737 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:14:39,166 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:14:39,166 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:14:40,790 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:14:42,449 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:14:42,830 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:14:44,872 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:14:46,925 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:14:47,736 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:14:48,958 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:14:48,966 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:14:51,007 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:14:52,798 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:14:53,056 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:14:54,189 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:14:54,189 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:14:55,112 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:14:57,155 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:14:58,470 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:14:59,197 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:14:59,495 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 04:14:59,497 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 04:14:59,497 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 04:14:59,498 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 04:15:00,195 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 04:15:01,220 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:15:01,229 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:15:03,281 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:15:03,779 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:15:05,383 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:15:06,554 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:15:07,397 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:15:09,188 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:15:09,188 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:15:09,425 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:15:09,431 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:15:11,469 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:15:13,539 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:15:15,211 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:15:15,581 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:15:15,598 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:15:17,642 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:15:19,691 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:15:20,256 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:15:21,735 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:15:23,798 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:15:24,193 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:15:24,194 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:15:25,475 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:15:25,834 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:15:27,873 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:15:27,880 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:15:29,933 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:15:31,443 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:15:32,004 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:15:34,059 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:15:36,180 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:15:36,489 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:15:36,568 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:15:38,205 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:15:39,211 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:15:39,212 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:15:40,228 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:15:41,220 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:15:41,582 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:15:42,275 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:15:44,327 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:15:46,380 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:15:46,631 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:15:48,436 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:15:50,482 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:15:51,686 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:15:52,541 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:15:53,549 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:15:54,229 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:15:54,230 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:15:54,608 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:15:56,644 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:15:57,511 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:15:58,694 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:16:00,768 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:16:02,559 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:16:02,815 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:16:04,873 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:16:06,573 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:16:06,991 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:16:07,588 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:16:07,950 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:16:09,014 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:16:09,244 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:16:09,244 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:16:11,031 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:16:13,062 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:16:13,526 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:16:15,098 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:16:17,141 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:16:18,559 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:16:19,184 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:16:20,200 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:16:21,240 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:16:23,283 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:16:23,592 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:16:24,256 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:16:24,257 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:16:25,324 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:16:27,370 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:16:29,421 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:16:29,555 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:16:31,461 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:16:33,500 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:16:33,508 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:16:34,590 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:16:35,547 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:16:36,582 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:16:37,662 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:16:39,261 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:16:39,261 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:16:39,680 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:16:40,524 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:16:41,708 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:16:43,742 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:16:45,772 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:16:45,787 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:16:46,451 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:16:47,820 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:16:49,865 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:16:51,495 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:16:51,922 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:16:53,968 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:16:54,276 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:16:54,276 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:16:56,009 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:16:56,537 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:16:58,044 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:17:00,081 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:17:00,090 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:17:02,150 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:17:02,330 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:17:04,214 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:17:06,258 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:17:06,589 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:17:07,603 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:17:08,366 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:17:09,294 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:17:09,294 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:17:10,177 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 04:17:10,179 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 04:17:10,179 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 04:17:10,180 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 04:17:10,350 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 04:17:10,387 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:17:11,363 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:17:12,403 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:17:12,413 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:17:13,475 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:17:14,464 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:17:16,502 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:17:18,545 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:17:18,548 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:17:20,578 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:17:22,615 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:17:23,615 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:17:24,310 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:17:24,310 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:17:24,666 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:17:25,670 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:17:26,713 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:17:28,754 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:17:29,602 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:17:30,805 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:17:32,845 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:17:34,639 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:17:34,890 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:17:36,600 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:17:36,925 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:17:37,935 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:17:39,040 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:17:39,316 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:17:39,316 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:17:40,570 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:17:41,057 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:17:43,067 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:17:45,100 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:17:45,597 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:17:47,144 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:17:49,180 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:17:50,177 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:17:51,228 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:17:51,326 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:17:53,278 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:17:54,321 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:17:54,321 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:17:55,321 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:17:56,595 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:17:57,363 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:17:59,414 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:18:01,453 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:18:01,630 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:18:03,496 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:18:04,496 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:18:05,539 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:18:06,612 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:18:07,570 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:18:07,624 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:18:09,336 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:18:09,337 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:18:09,699 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:18:11,718 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:18:12,659 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:18:13,735 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:18:15,763 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:18:16,755 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:18:17,794 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:18:17,831 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:18:19,838 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:18:21,890 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:18:22,885 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:18:23,926 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:18:24,358 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:18:24,358 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:18:25,970 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:18:28,017 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:18:28,591 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:18:30,056 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:18:30,066 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:18:32,130 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:18:33,629 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:18:34,175 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:18:36,220 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:18:36,615 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:18:38,262 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:18:38,635 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:18:39,366 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:18:39,366 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:18:40,379 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:18:42,349 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:18:42,406 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:18:44,417 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:18:44,427 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:18:46,441 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:18:48,483 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:18:49,488 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:18:50,528 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:18:52,589 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:18:54,373 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:18:54,373 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:18:54,636 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:18:54,637 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:18:54,649 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:18:56,709 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:18:58,748 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:18:59,685 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:19:00,807 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:19:02,850 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:19:04,758 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:19:04,904 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:19:06,628 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:19:06,947 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:19:08,987 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:19:08,994 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:19:09,390 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:19:09,391 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:19:10,649 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:19:11,106 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:19:13,131 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:19:15,148 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:19:15,701 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:19:17,187 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:19:18,866 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 04:19:18,868 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 04:19:18,868 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 04:19:18,869 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 04:19:19,246 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 04:19:19,257 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:19:20,258 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:19:21,139 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:19:21,316 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:19:21,333 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:19:23,367 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:19:24,388 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:19:24,388 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:19:25,411 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:19:26,665 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:19:27,455 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:19:29,502 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:19:31,543 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:19:31,716 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:19:33,581 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:19:34,578 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:19:35,618 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:19:36,643 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:19:37,651 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:19:37,655 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:19:39,389 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:19:39,389 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:19:39,707 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:19:41,815 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:19:42,670 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:19:43,824 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:19:45,846 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:19:46,815 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:19:47,872 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:19:48,381 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:19:49,899 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:19:51,955 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:19:53,432 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:19:53,994 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:19:54,407 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:19:54,408 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:19:56,042 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:19:58,103 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:19:59,186 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:20:00,173 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:20:01,168 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:20:02,226 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:20:04,252 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:20:04,262 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:20:06,306 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:20:06,653 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:20:08,349 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:20:09,411 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:20:09,412 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:20:09,689 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:20:10,397 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:20:12,469 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:20:12,505 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:20:14,526 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:20:15,222 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:20:16,551 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:20:18,574 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:20:20,254 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:20:20,617 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:20:22,646 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:20:24,436 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:20:24,437 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:20:24,696 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:20:24,702 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:20:25,702 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:20:26,748 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:20:28,790 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:20:30,839 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:20:30,854 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:20:32,888 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:20:34,930 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:20:35,929 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:20:36,665 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:20:36,976 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:20:39,018 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:20:39,025 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:20:39,438 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:20:39,438 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:20:41,084 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:20:41,703 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:20:43,201 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:20:45,215 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:20:46,743 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:20:47,230 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:20:49,266 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:20:51,310 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:20:51,320 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:20:51,820 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:20:53,367 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:20:54,442 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:20:54,442 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:20:55,402 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:20:57,447 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:20:57,703 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:20:59,488 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:21:01,523 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:21:02,711 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:21:03,580 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:21:04,575 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:21:05,616 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:21:06,670 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:21:07,665 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:21:08,690 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:21:09,444 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:21:09,444 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:21:09,709 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:21:11,753 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:21:13,738 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:21:13,860 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:21:15,873 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:21:16,873 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:21:17,894 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:21:19,596 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:21:19,920 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:21:21,959 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:21:24,000 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:21:24,462 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:21:24,463 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:21:24,717 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:21:26,037 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:21:28,077 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:21:28,472 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 04:21:28,474 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 04:21:28,475 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 04:21:28,476 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 04:21:29,078 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 04:21:30,120 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:21:30,136 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:21:30,719 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:21:31,134 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:21:32,203 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:21:34,247 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:21:35,753 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:21:36,277 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:21:36,676 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:21:38,319 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:21:39,469 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:21:39,470 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:21:40,382 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:21:41,738 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:21:42,415 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:21:43,417 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:21:44,545 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:21:46,562 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:21:47,030 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:21:48,574 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:21:50,639 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:21:52,086 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:21:52,681 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:21:54,466 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:21:54,466 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:21:54,738 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:21:56,788 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:21:56,790 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:21:57,827 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:21:58,844 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:22:00,922 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:22:02,911 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:22:02,968 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:22:05,007 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:22:06,693 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:22:07,058 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:22:08,592 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:22:09,104 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:22:09,116 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:22:09,476 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:22:09,478 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:22:11,161 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:22:13,208 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:22:13,773 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:22:15,311 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:22:17,324 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:22:18,806 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:22:19,344 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:22:21,379 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:22:21,388 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:22:23,428 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:22:23,815 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:22:24,496 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:22:24,496 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:22:25,478 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:22:27,524 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:22:29,570 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:22:29,789 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:22:31,605 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:22:33,654 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:22:35,234 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:22:35,691 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:22:35,698 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:22:36,702 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:22:37,737 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:22:39,509 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:22:39,510 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:22:39,785 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:22:40,791 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:22:41,831 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:22:43,887 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:22:45,842 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:22:46,006 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:22:47,976 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:22:48,027 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:22:50,049 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:22:51,019 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:22:52,090 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:22:54,133 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:22:54,521 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:22:54,521 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:22:56,177 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:22:56,792 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:22:58,219 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:23:00,268 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:23:01,269 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:23:02,330 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:23:02,698 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:23:04,371 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:23:06,417 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:23:06,710 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:23:07,716 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:23:08,471 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:23:09,527 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:23:09,528 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:23:10,509 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:23:12,554 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:23:13,544 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:23:13,545 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:23:14,601 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:23:16,708 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:23:18,633 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:23:18,724 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:23:20,756 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:23:22,797 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:23:23,666 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:23:24,540 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:23:24,540 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:23:24,841 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:23:25,832 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:23:26,876 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:23:28,834 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:23:28,925 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:23:30,988 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:23:33,032 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:23:33,886 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:23:35,088 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:23:36,725 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:23:37,130 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:23:37,215 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 04:23:37,217 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 04:23:37,217 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 04:23:37,218 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 04:23:38,129 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 04:23:39,162 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:23:39,171 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:23:39,548 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:23:39,549 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:23:39,800 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:23:40,170 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:23:41,220 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:23:43,286 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:23:44,840 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:23:45,330 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:23:47,443 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:23:49,462 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:23:49,884 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:23:51,469 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:23:51,486 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:23:53,533 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:23:54,558 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:23:54,558 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:23:55,600 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:23:55,809 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:23:57,632 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:23:59,687 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:24:00,866 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:24:01,728 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:24:03,771 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:24:05,802 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:24:05,810 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:24:06,660 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:24:06,739 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:24:07,859 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:24:09,568 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:24:09,569 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:24:09,901 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:24:11,829 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:24:11,964 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:24:14,009 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:24:16,056 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:24:17,595 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:24:18,132 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:24:18,178 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:24:20,199 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:24:22,221 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:24:22,645 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:24:24,235 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:24:24,590 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:24:24,590 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:24:26,281 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:24:27,872 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:24:28,327 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:24:30,377 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:24:30,388 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:24:32,445 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:24:33,332 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:24:34,490 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:24:36,527 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:24:36,749 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:24:38,576 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:24:38,766 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:24:39,602 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:24:39,603 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:24:40,626 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:24:42,677 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:24:43,676 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:24:44,260 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:24:44,750 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:24:46,822 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:24:48,936 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:24:49,316 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:24:50,959 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:24:52,989 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:24:54,616 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:24:54,617 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:24:54,868 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:24:55,040 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:24:56,046 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:24:57,097 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:24:59,143 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:25:00,134 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:25:01,186 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:25:03,219 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:25:05,252 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:25:05,263 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:25:06,750 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:25:07,328 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:25:09,374 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:25:09,633 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:25:09,634 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:25:10,372 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:25:10,903 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:25:11,418 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:25:13,463 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:25:15,509 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:25:15,937 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:25:17,568 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:25:19,672 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:25:21,686 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:25:21,722 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:25:22,655 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:25:23,702 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:25:24,655 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:25:24,655 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:25:25,724 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:25:26,918 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:25:27,773 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:25:29,819 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:25:31,868 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:25:31,975 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:25:33,900 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:25:33,907 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:25:35,952 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:25:36,755 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:25:37,761 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:25:38,007 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:25:39,670 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:25:39,671 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:25:40,043 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:25:42,085 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:25:43,210 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:25:44,124 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:25:46,169 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:25:46,217 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 04:25:46,219 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 04:25:46,219 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 04:25:46,220 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 04:25:47,175 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:25:47,175 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 04:25:48,207 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:25:48,215 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:25:48,509 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:25:50,321 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:25:52,342 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:25:53,558 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:25:54,368 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:25:54,684 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:25:54,685 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:25:56,430 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:25:58,477 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:25:58,985 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:26:00,519 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:26:02,558 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:26:02,567 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:26:04,609 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:26:04,876 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:26:06,659 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:26:06,766 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:26:08,704 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:26:09,678 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:26:09,679 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:26:09,930 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:26:10,746 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:26:12,796 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:26:14,825 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:26:14,832 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:26:15,759 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:26:16,882 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:26:18,938 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:26:20,805 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:26:21,070 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:26:23,093 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:26:24,694 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:26:24,694 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:26:25,114 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:26:26,106 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:26:26,750 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:26:27,150 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:26:29,210 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:26:31,251 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:26:31,818 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:26:33,300 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:26:35,344 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:26:36,780 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:26:37,391 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:26:37,794 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:26:39,436 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:26:39,718 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:26:39,719 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:26:40,431 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:26:41,482 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:26:42,974 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:26:43,528 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:26:45,573 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:26:47,615 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:26:48,024 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:26:49,655 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:26:51,774 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:26:52,732 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:26:53,472 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:26:53,797 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:26:54,721 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:26:54,722 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:26:55,820 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:26:57,861 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:26:59,004 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:26:59,921 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:27:01,957 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:27:04,015 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:27:04,041 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:27:06,060 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:27:06,784 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:27:07,054 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:27:08,111 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:27:09,725 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:27:09,726 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:27:09,984 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:27:10,173 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:27:12,238 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:27:14,281 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:27:15,027 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:27:16,335 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:27:18,363 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:27:18,372 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:27:20,270 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:27:20,421 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:27:22,550 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:27:24,570 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:27:24,743 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:27:24,743 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:27:26,009 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:27:26,580 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:27:28,603 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:27:30,676 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:27:31,014 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:27:32,727 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:27:32,729 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:27:34,802 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:27:36,067 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:27:36,789 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:27:36,850 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:27:38,894 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:27:39,752 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:27:39,753 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:27:40,938 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:27:42,018 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:27:42,993 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:27:45,025 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:27:45,035 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:27:47,074 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:27:47,861 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:27:49,108 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:27:51,151 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:27:52,904 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:27:53,285 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:27:54,765 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:27:54,765 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:27:55,310 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:27:55,582 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 04:27:55,585 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 04:27:55,585 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 04:27:55,586 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 04:27:56,271 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 04:27:57,313 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:27:57,332 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:27:58,823 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:27:59,360 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:28:01,409 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:28:03,465 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:28:03,865 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:28:05,511 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:28:06,792 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:28:07,572 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:28:09,067 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:28:09,622 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:28:09,762 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:28:09,762 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:28:10,625 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:28:11,674 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:28:13,731 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:28:15,051 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:28:15,775 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:28:17,822 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:28:19,866 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:28:20,098 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:28:21,913 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:28:22,910 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:28:24,011 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:28:24,775 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:28:24,776 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:28:26,028 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:28:26,035 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:28:28,057 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:28:30,095 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:28:31,082 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:28:32,143 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:28:34,194 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:28:36,237 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:28:36,801 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:28:36,802 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:28:37,229 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:28:38,267 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:28:39,781 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:28:39,782 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:28:40,318 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:28:42,053 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:28:42,359 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:28:44,414 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:28:46,478 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:28:47,112 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:28:48,548 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:28:49,543 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:28:50,592 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:28:52,631 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:28:52,856 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:28:54,736 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:28:54,782 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:28:54,783 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:28:56,755 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:28:58,050 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:28:58,784 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:29:00,850 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:29:02,882 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:29:02,890 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:29:03,772 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:29:04,927 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:29:06,804 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:29:06,981 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:29:08,834 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:29:09,028 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:29:09,785 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:29:09,786 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:29:11,071 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:29:13,115 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:29:14,481 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:29:15,150 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:29:15,159 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:29:17,203 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:29:19,246 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:29:19,512 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:29:21,292 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:29:23,339 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:29:24,783 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:29:24,784 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:29:25,034 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:29:25,458 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:29:27,426 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:29:27,486 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:29:29,504 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:29:30,281 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:29:31,518 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:29:33,562 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:29:35,332 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:29:35,597 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:29:36,815 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:29:37,632 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:29:39,673 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:29:39,787 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:29:39,787 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:29:41,050 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:29:41,703 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:29:41,711 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:29:43,754 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:29:45,808 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:29:46,084 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:29:47,856 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:29:49,897 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:29:51,140 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:29:51,964 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:29:52,956 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:29:54,009 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:29:54,801 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:29:54,801 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:29:56,114 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:29:57,071 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:29:58,137 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:30:00,161 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:30:02,132 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:30:02,180 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:30:04,230 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:30:04,634 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 04:30:04,636 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 04:30:04,637 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 04:30:04,638 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 04:30:05,224 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:30:05,224 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 04:30:06,262 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:30:06,272 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:30:06,830 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:30:07,273 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:30:07,844 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:30:08,316 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:30:09,802 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:30:09,802 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:30:10,374 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:30:12,439 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:30:13,053 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:30:14,490 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:30:16,532 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:30:18,082 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:30:18,584 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:30:19,587 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:30:20,624 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:30:22,685 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:30:23,190 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:30:24,735 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:30:24,815 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:30:24,816 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:30:26,870 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:30:28,891 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:30:29,104 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:30:30,940 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:30:32,997 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:30:34,000 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:30:34,994 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:30:35,060 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:30:36,837 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:30:37,106 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:30:39,157 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:30:39,808 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:30:39,808 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:30:40,061 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:30:41,198 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:30:43,236 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:30:45,276 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:30:45,290 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:30:45,683 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:30:47,322 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:30:49,367 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:30:50,731 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:30:51,406 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:30:53,454 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:30:54,825 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:30:54,826 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:30:55,498 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:30:56,083 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:30:57,566 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:30:57,602 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:30:59,622 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:31:01,479 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:31:01,644 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:31:03,662 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:31:05,697 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:31:06,526 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:31:06,843 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:31:07,740 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:31:09,788 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:31:09,839 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:31:09,840 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:31:11,831 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:31:11,839 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:31:12,102 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:31:13,879 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:31:15,923 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:31:17,144 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:31:17,964 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:31:19,996 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:31:22,042 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:31:23,040 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:31:24,070 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:31:24,078 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:31:24,839 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:31:24,839 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:31:26,127 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:31:28,171 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:31:28,231 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:31:30,256 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:31:32,280 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:31:33,179 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:31:34,330 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:31:36,381 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:31:36,384 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:31:36,846 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:31:38,431 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:31:38,863 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:31:39,849 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:31:39,849 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:31:40,493 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:31:42,527 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:31:44,138 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:31:44,587 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:31:46,624 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:31:48,671 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:31:49,673 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:31:49,704 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:31:50,712 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:31:52,762 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:31:54,817 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:31:54,865 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:31:54,866 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:31:55,119 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:31:56,852 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:31:58,965 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:32:00,163 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:32:00,989 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:32:01,979 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:32:03,017 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:32:05,062 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:32:05,618 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:32:06,850 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:32:07,104 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:32:09,150 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:32:09,877 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:32:09,877 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:32:11,183 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:32:11,196 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:32:13,243 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:32:13,454 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 04:32:13,456 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 04:32:13,456 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 04:32:13,456 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 04:32:14,248 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:32:14,248 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 04:32:15,277 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:32:15,285 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:32:16,291 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:32:16,696 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:32:17,334 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:32:19,389 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:32:21,422 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:32:21,746 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:32:23,468 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:32:24,894 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:32:24,894 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:32:25,523 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:32:26,984 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:32:27,576 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:32:28,570 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:32:29,705 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:32:31,724 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:32:32,024 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:32:33,743 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:32:35,774 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:32:36,855 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:32:37,817 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:32:37,867 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:32:39,881 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:32:39,901 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:32:39,901 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:32:41,908 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:32:41,916 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:32:43,158 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:32:43,970 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:32:46,019 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:32:48,067 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:32:48,201 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:32:50,116 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:32:52,177 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:32:53,827 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:32:54,204 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:32:54,217 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:32:54,924 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:32:54,924 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:32:56,254 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:32:58,299 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:32:59,198 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:33:00,441 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:33:02,467 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:33:04,245 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:33:04,487 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:33:06,512 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:33:06,860 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:33:08,528 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:33:08,534 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:33:09,293 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:33:09,928 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:33:09,928 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:33:10,581 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:33:12,628 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:33:14,661 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:33:15,254 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:33:16,700 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:33:18,759 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:33:20,631 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:33:20,798 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:33:20,809 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:33:22,848 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:33:24,903 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:33:24,936 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:33:24,936 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:33:26,207 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:33:26,948 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:33:28,990 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:33:31,127 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:33:31,252 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:33:32,105 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:33:33,137 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:33:35,180 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:33:36,461 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:33:36,871 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:33:37,225 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:33:39,269 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:33:39,930 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:33:39,930 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:33:41,314 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:33:42,192 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:33:43,353 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:33:45,392 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:33:46,385 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:33:47,279 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:33:47,435 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:33:49,490 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:33:51,533 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:33:52,328 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:33:53,567 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:33:54,944 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:33:54,944 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:33:55,607 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:33:57,652 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:33:58,018 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:33:58,646 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:33:59,696 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:34:01,809 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:34:03,061 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:34:03,830 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:34:05,847 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:34:06,874 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:34:07,879 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:34:08,896 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:34:09,929 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:34:09,968 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:34:09,969 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:34:10,939 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:34:12,001 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:34:14,036 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:34:14,250 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:34:16,081 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:34:18,131 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:34:19,300 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:34:20,183 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:34:22,234 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:34:22,715 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 04:34:22,716 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 04:34:22,717 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 04:34:22,717 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 04:34:23,234 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 04:34:24,261 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:34:24,274 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:34:24,320 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:34:24,984 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:34:24,984 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:34:25,269 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:34:26,341 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:34:28,389 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:34:30,278 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:34:30,431 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:34:32,553 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:34:34,569 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:34:35,320 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:34:36,586 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:34:36,878 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:34:38,606 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:34:38,612 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:34:39,990 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:34:39,990 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:34:40,650 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:34:41,250 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:34:42,700 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:34:44,737 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:34:46,301 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:34:46,789 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:34:48,828 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:34:50,855 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:34:50,863 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:34:52,020 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:34:52,902 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:34:54,942 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:34:55,006 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:34:55,006 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:34:56,988 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:34:57,274 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:34:59,032 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:35:01,098 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:35:02,877 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:35:03,177 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:35:03,221 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:35:05,232 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:35:06,890 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:35:07,255 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:35:07,903 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:35:09,279 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:35:10,022 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:35:10,022 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:35:11,329 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:35:13,295 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:35:13,369 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:35:15,422 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:35:16,437 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:35:17,477 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:35:18,764 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:35:19,525 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:35:21,574 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:35:23,612 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:35:23,805 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:35:25,036 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:35:25,036 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:35:25,668 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:35:27,722 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:35:28,727 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:35:29,512 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:35:29,799 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:35:31,850 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:35:33,962 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:35:34,533 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:35:35,981 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:35:36,900 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:35:38,006 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:35:40,040 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:35:40,046 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:35:40,047 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:35:40,299 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:35:42,086 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:35:43,092 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:35:44,130 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:35:45,705 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:35:46,157 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:35:48,207 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:35:50,259 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:35:50,762 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:35:52,319 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:35:54,369 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:35:55,063 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:35:55,064 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:35:55,365 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:35:56,318 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:35:56,408 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:35:58,444 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:36:00,498 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:36:01,366 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:36:02,534 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:36:04,636 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:36:06,624 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:36:06,646 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:36:06,904 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:36:06,905 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:36:08,682 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:36:10,057 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:36:10,058 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:36:10,738 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:36:12,337 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:36:12,773 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:36:14,834 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:36:16,879 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:36:17,380 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:36:18,924 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:36:20,958 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:36:20,975 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:36:23,004 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:36:23,087 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:36:25,047 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:36:25,066 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:36:25,067 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:36:27,111 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:36:28,345 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:36:29,168 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:36:31,207 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:36:31,870 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 04:36:31,872 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 04:36:31,872 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 04:36:31,873 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 04:36:32,207 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 04:36:33,234 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:36:33,242 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:36:34,243 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:36:34,245 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:36:35,391 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:36:36,917 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:36:37,410 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:36:39,405 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:36:39,433 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:36:40,065 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:36:40,066 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:36:41,456 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:36:43,496 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:36:45,359 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:36:45,526 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:36:47,564 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:36:47,565 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:36:49,607 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:36:50,442 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:36:51,649 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:36:53,699 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:36:55,060 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:36:55,061 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:36:55,750 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:36:56,322 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:36:57,792 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:36:59,821 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:36:59,835 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:37:01,873 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:37:02,239 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:37:03,914 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:37:06,030 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:37:06,928 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:37:07,935 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:37:08,054 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:37:10,068 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:37:10,068 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:37:10,082 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:37:12,108 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:37:13,102 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:37:13,133 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:37:14,162 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:37:16,240 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:37:18,246 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:37:18,273 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:37:20,314 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:37:22,354 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:37:23,290 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:37:24,392 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:37:25,091 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:37:25,092 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:37:25,392 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:37:26,445 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:37:28,356 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:37:28,493 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:37:30,567 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:37:32,608 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:37:33,384 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:37:34,643 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:37:36,748 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:37:36,933 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:37:38,761 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:37:38,944 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:37:39,731 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:37:40,094 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:37:40,103 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:37:40,784 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:37:42,828 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:37:44,378 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:37:44,863 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:37:46,903 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:37:48,948 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:37:49,407 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:37:50,979 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:37:51,974 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:37:53,029 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:37:54,448 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:37:55,079 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:37:55,095 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:37:55,095 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:37:57,121 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:37:59,164 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:38:00,389 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:38:01,212 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:38:03,251 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:38:03,258 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:38:05,297 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:38:05,666 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:38:06,946 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:38:07,393 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:38:09,406 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:38:10,104 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:38:10,105 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:38:11,379 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:38:11,454 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:38:13,495 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:38:15,535 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:38:16,426 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:38:17,579 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:38:17,590 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:38:19,642 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:38:21,471 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:38:21,691 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:38:23,736 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:38:25,114 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:38:25,115 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:38:25,771 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:38:27,385 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:38:27,810 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:38:29,842 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:38:29,855 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:38:31,909 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:38:33,361 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:38:33,956 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:38:35,993 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:38:36,955 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:38:38,106 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:38:38,971 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:38:40,117 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:38:40,118 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:38:40,128 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:38:41,189 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 04:38:41,190 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 04:38:41,190 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 04:38:41,191 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 04:38:42,160 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:38:42,160 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 04:38:42,172 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:38:44,218 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:38:44,231 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:38:44,467 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:38:46,273 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:38:48,319 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:38:49,506 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:38:50,371 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:38:52,420 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:38:54,459 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:38:54,574 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:38:55,126 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:38:55,127 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:38:55,457 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:38:56,513 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:38:58,552 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:39:00,425 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:39:00,624 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:39:02,659 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:39:04,702 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:39:05,465 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:39:06,739 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:39:06,968 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:39:08,847 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:39:09,829 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:39:10,137 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:39:10,138 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:39:10,862 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:39:11,382 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:39:12,885 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:39:14,906 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:39:16,414 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:39:16,949 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:39:18,981 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:39:21,040 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:39:22,033 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:39:22,190 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:39:23,083 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:39:25,137 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:39:25,142 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:39:25,142 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:39:27,189 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:39:27,415 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:39:29,237 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:39:31,276 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:39:32,480 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:39:33,313 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:39:34,326 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:39:35,371 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:39:36,977 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:39:37,418 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:39:37,990 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:39:39,524 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:39:40,140 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:39:40,140 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:39:41,548 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:39:43,420 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:39:43,583 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:39:45,621 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:39:47,656 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:39:47,668 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:39:48,813 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:39:49,715 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:39:51,776 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:39:53,814 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:39:53,866 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:39:55,141 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:39:55,142 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:39:55,866 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:39:57,914 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:39:59,768 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:39:59,945 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:39:59,958 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:40:01,995 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:40:04,034 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:40:04,803 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:40:06,090 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:40:06,986 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:40:08,129 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:40:10,167 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:40:10,180 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:40:10,248 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:40:10,451 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:40:12,269 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:40:14,272 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:40:14,292 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:40:15,709 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:40:16,328 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:40:18,368 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:40:20,404 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:40:20,771 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:40:22,458 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:40:24,504 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:40:25,157 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:40:25,157 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:40:26,405 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:40:26,534 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:40:26,543 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:40:28,581 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:40:30,639 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:40:31,461 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:40:32,678 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:40:34,722 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:40:36,506 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:40:36,789 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:40:36,990 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:40:38,819 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:40:38,829 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:40:40,155 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:40:40,155 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:40:40,933 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:40:42,438 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:40:42,950 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:40:44,968 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:40:47,015 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:40:47,476 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:40:49,048 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:40:50,018 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 04:40:50,019 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 04:40:50,020 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 04:40:50,020 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 04:40:50,049 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 04:40:51,077 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:40:51,084 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:40:52,092 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:40:53,127 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:40:53,305 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:40:55,167 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:40:55,168 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:40:55,171 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:40:57,212 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:40:58,449 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:40:59,257 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:41:01,289 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:41:03,338 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:41:03,500 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:41:04,346 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:41:05,384 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:41:06,997 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:41:07,423 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:41:09,022 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:41:09,464 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:41:10,177 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:41:10,177 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:41:11,572 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:41:13,585 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:41:14,482 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:41:15,599 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:41:17,644 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:41:18,638 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:41:19,682 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:41:20,452 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:41:21,725 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:41:23,764 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:41:25,182 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:41:25,183 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:41:25,808 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:41:26,454 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:41:27,857 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:41:29,909 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:41:30,905 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:41:31,973 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:41:32,357 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:41:34,011 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:41:36,061 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:41:37,003 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:41:38,009 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:41:38,099 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:41:40,142 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:41:40,189 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:41:40,190 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:41:42,257 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:41:43,272 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:41:44,242 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:41:44,276 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:41:46,301 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:41:48,336 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:41:48,351 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:41:50,385 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:41:52,448 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:41:53,445 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:41:54,488 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:41:55,184 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:41:55,184 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:41:56,511 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:41:56,520 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:41:58,468 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:41:58,561 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:42:00,611 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:42:02,649 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:42:03,514 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:42:04,678 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:42:06,719 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:42:07,019 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:42:08,760 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:42:08,769 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:42:09,026 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:42:10,177 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:42:10,178 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:42:10,815 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:42:12,919 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:42:14,450 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:42:14,937 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:42:16,953 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:42:18,998 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:42:19,497 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:42:21,053 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:42:23,105 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:42:23,114 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:42:24,600 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:42:25,165 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:42:25,202 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:42:25,203 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:42:27,219 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:42:29,276 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:42:30,495 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:42:31,306 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:42:33,351 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:42:35,383 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:42:35,396 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:42:36,006 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:42:37,023 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:42:37,451 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:42:39,487 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:42:40,199 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:42:40,200 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:42:41,464 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:42:41,529 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:42:43,637 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:42:45,647 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:42:46,503 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:42:47,669 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:42:48,672 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:42:49,715 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:42:51,746 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:42:51,815 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:42:53,779 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:42:55,204 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:42:55,205 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:42:55,813 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:42:57,469 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:42:57,857 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:42:59,777 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 04:42:59,778 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 04:42:59,779 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 04:42:59,780 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 04:42:59,895 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 04:42:59,904 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:43:00,908 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:43:01,933 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:43:01,939 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:43:03,013 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:43:03,981 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:43:06,024 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:43:07,027 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:43:08,065 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:43:08,067 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:43:10,135 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:43:10,211 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:43:10,212 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:43:12,199 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:43:13,324 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:43:14,328 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:43:15,293 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:43:16,350 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:43:18,359 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:43:18,369 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:43:20,386 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:43:22,406 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:43:23,412 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:43:24,446 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:43:25,226 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:43:25,226 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:43:26,487 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:43:27,487 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:43:28,534 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:43:29,235 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:43:30,591 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:43:32,641 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:43:34,266 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:43:34,691 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:43:36,730 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:43:37,028 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:43:38,774 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:43:39,632 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:43:40,243 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:43:40,244 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:43:40,813 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:43:40,814 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:43:42,872 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:43:44,980 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:43:45,552 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:43:47,000 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:43:49,020 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:43:50,603 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:43:51,049 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:43:53,079 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:43:53,095 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:43:55,135 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:43:55,251 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:43:55,251 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:43:56,509 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:43:57,188 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:43:59,224 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:44:01,266 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:44:01,556 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:44:03,325 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:44:05,381 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:44:07,033 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:44:07,035 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:44:07,415 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:44:07,423 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:44:09,479 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:44:10,265 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:44:10,266 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:44:11,527 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:44:12,544 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:44:13,573 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:44:15,687 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:44:17,633 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:44:17,703 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:44:19,690 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:44:19,728 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:44:21,754 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:44:23,093 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:44:23,783 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:44:25,259 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:44:25,259 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:44:25,827 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:44:27,862 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:44:28,527 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:44:29,921 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:44:31,979 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:44:33,950 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:44:34,019 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:44:34,032 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:44:36,067 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:44:37,048 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:44:38,129 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:44:39,064 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:44:40,168 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:44:40,270 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:44:40,271 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:44:42,205 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:44:44,240 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:44:44,629 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:44:45,244 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:44:46,350 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:44:48,360 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:44:49,681 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:44:50,383 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:44:52,419 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:44:54,459 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:44:55,283 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:44:55,284 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:44:55,540 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:44:56,502 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:44:57,506 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:44:58,536 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:45:00,565 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:45:00,596 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:45:02,643 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:45:04,695 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:45:05,603 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:45:06,733 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:45:07,059 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:45:08,784 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:45:09,231 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 04:45:09,232 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 04:45:09,233 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 04:45:09,233 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 04:45:09,782 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:45:09,783 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 04:45:10,275 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:45:10,276 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:45:10,823 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:45:11,543 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:45:11,824 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:45:12,884 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:45:14,946 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:45:16,579 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:45:17,045 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:45:19,064 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:45:21,083 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:45:21,605 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:45:23,135 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:45:24,132 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:45:25,176 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:45:25,285 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:45:25,286 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:45:27,205 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:45:27,561 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:45:29,246 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:45:31,284 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:45:32,612 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:45:33,326 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:45:35,373 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:45:37,063 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:45:37,400 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:45:37,407 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:45:38,075 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:45:39,446 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:45:40,276 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:45:40,277 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:45:41,483 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:45:43,524 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:45:43,604 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:45:45,564 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:45:47,702 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:45:49,593 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:45:49,689 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:45:49,712 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:45:51,734 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:45:53,781 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:45:54,638 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:45:55,288 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:45:55,288 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:45:55,830 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:45:57,874 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:45:59,920 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:46:00,590 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:46:01,961 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:46:03,997 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:46:04,005 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:46:06,053 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:46:06,531 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:46:07,065 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:46:08,100 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:46:10,147 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:46:10,308 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:46:10,309 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:46:11,575 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:46:12,210 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:46:14,243 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:46:16,272 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:46:16,280 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:46:17,288 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:46:18,385 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:46:20,404 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:46:22,400 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:46:22,419 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:46:24,449 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:46:25,321 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:46:25,321 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:46:26,496 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:46:27,954 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:46:28,531 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:46:28,540 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:46:30,599 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:46:32,629 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:46:32,995 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:46:34,675 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:46:36,747 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:46:37,079 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:46:38,084 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:46:38,818 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:46:40,335 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:46:40,336 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:46:40,849 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:46:41,845 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:46:42,905 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:46:43,611 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:46:44,934 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:46:46,977 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:46:48,650 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:46:49,079 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:46:51,103 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:46:53,113 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:46:53,814 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:46:54,099 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:46:55,136 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:46:55,347 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:46:55,348 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:46:57,186 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:46:59,227 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:46:59,632 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:47:01,270 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:47:03,299 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:47:04,686 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:47:05,342 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:47:07,084 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:47:07,387 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:47:08,389 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:47:09,422 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:47:09,794 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:47:10,342 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:47:10,343 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:47:11,466 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:47:13,524 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:47:15,566 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:47:15,633 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:47:17,620 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:47:18,404 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 04:47:18,406 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 04:47:18,406 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 04:47:18,407 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 04:47:18,626 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 04:47:19,728 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:47:20,697 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:47:20,712 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:47:21,755 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:47:23,809 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:47:25,342 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:47:25,343 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:47:25,858 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:47:26,600 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:47:27,902 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:47:29,943 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:47:31,634 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:47:32,003 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:47:34,032 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:47:34,039 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:47:36,077 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:47:37,070 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:47:37,086 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:47:38,125 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:47:40,162 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:47:40,355 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:47:40,356 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:47:42,206 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:47:42,657 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:47:44,236 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:47:46,273 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:47:46,281 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:47:47,974 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:47:48,319 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:47:50,419 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:47:52,432 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:47:53,031 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:47:54,452 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:47:55,370 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:47:55,371 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:47:56,502 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:47:58,549 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:47:58,853 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:48:00,604 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:48:00,616 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:48:02,665 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:48:03,896 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:48:04,698 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:48:06,750 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:48:07,086 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:48:08,806 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:48:09,095 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:48:10,387 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:48:10,387 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:48:10,854 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:48:12,901 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:48:12,910 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:48:14,830 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:48:14,956 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:48:16,990 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:48:19,042 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:48:19,862 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:48:21,150 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:48:23,168 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:48:24,135 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:48:25,185 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:48:25,395 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:48:25,395 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:48:25,653 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:48:27,217 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:48:29,253 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:48:30,693 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:48:31,302 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:48:33,340 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:48:35,377 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:48:35,729 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:48:37,089 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:48:37,412 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:48:38,420 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:48:39,454 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:48:40,391 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:48:40,392 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:48:41,499 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:48:41,654 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:48:43,563 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:48:45,613 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:48:46,697 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:48:47,675 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:48:49,722 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:48:50,717 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:48:51,831 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:48:52,502 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:48:53,849 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:48:55,406 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:48:55,406 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:48:55,870 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:48:57,666 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:48:57,915 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:48:59,952 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:49:01,996 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:49:03,338 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:49:04,043 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:49:05,043 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:49:06,077 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:49:07,102 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:49:08,128 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:49:09,125 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:49:10,196 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:49:10,410 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:49:10,410 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:49:12,244 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:49:14,299 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:49:14,704 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:49:16,331 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:49:16,338 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:49:18,384 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:49:20,189 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:49:20,431 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:49:22,542 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:49:24,562 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:49:25,406 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:49:25,406 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:49:25,652 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:49:26,601 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:49:28,055 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 04:49:28,056 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 04:49:28,056 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 04:49:28,057 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 04:49:28,650 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 04:49:28,663 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:49:29,654 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:49:30,711 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:49:30,730 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:49:31,321 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:49:32,757 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:49:34,803 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:49:36,364 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:49:36,848 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:49:37,117 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:49:38,909 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:49:40,403 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:49:40,404 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:49:40,959 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:49:41,555 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:49:42,980 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:49:42,988 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:49:45,031 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:49:46,597 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:49:47,066 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:49:49,124 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:49:51,163 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:49:51,654 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:49:53,274 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:49:55,285 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:49:55,419 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:49:55,420 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:49:56,680 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:49:57,286 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:49:57,304 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:49:59,335 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:50:01,376 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:50:01,727 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:50:03,420 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:50:05,462 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:50:06,779 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:50:07,126 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:50:07,504 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:50:08,501 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:50:09,549 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:50:10,436 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:50:10,437 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:50:11,594 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:50:12,713 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:50:13,644 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:50:15,688 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:50:17,738 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:50:17,752 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:50:19,780 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:50:20,773 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:50:21,828 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:50:23,224 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:50:23,936 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:50:25,445 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:50:25,446 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:50:25,961 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:50:27,983 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:50:28,717 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:50:30,021 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:50:32,082 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:50:34,039 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:50:34,128 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:50:35,129 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:50:36,166 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:50:37,132 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:50:38,199 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:50:39,152 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:50:40,245 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:50:40,451 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:50:40,451 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:50:42,292 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:50:44,353 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:50:44,748 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:50:46,399 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:50:47,401 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:50:48,464 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:50:49,850 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:50:50,524 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:50:52,550 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:50:54,666 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:50:54,901 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:50:55,469 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:50:55,470 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:50:56,688 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:50:58,710 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:50:59,706 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:51:00,744 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:51:00,757 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:51:02,789 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:51:04,831 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:51:05,774 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:51:06,881 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:51:07,136 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:51:08,937 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:51:10,477 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:51:10,477 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:51:10,986 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:51:11,516 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:51:13,017 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:51:13,026 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:51:15,078 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:51:16,554 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:51:17,124 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:51:19,153 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:51:21,193 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:51:21,597 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:51:23,239 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:51:25,314 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:51:25,348 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:51:25,506 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:51:25,506 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:51:26,757 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:51:27,360 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:51:29,382 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:51:31,396 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:51:31,778 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:51:33,438 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:51:35,477 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:51:36,815 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:51:37,141 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:51:37,289 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 04:51:37,290 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 04:51:37,291 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 04:51:37,291 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 04:51:37,532 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 04:51:37,542 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:51:39,580 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:51:39,590 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:51:40,512 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:51:40,513 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:51:41,624 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:51:42,779 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:51:43,661 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:51:45,701 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:51:47,743 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:51:47,840 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:51:49,808 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:51:51,877 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:51:53,764 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:51:53,913 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:51:53,925 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:51:55,512 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:51:55,512 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:51:56,031 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:51:58,041 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:51:58,783 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:52:00,065 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:52:02,099 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:52:04,136 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:52:04,599 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:52:05,135 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:52:06,186 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:52:07,157 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:52:08,234 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:52:09,952 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:52:10,292 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:52:10,512 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:52:10,512 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:52:12,340 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:52:14,398 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:52:15,803 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:52:16,441 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:52:17,440 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:52:18,490 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:52:20,536 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:52:21,535 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:52:22,581 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:52:24,643 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:52:25,524 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:52:25,526 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:52:26,748 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:52:26,792 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:52:28,771 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:52:30,805 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:52:31,798 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:52:32,304 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:52:32,843 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:52:34,884 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:52:36,932 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:52:37,164 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:52:38,176 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:52:38,973 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:52:40,536 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:52:40,537 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:52:41,003 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:52:43,050 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:52:44,051 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:52:44,160 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:52:45,097 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:52:47,134 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:52:49,176 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:52:49,195 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:52:51,213 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:52:53,256 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:52:54,253 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:52:55,302 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:52:55,558 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:52:55,558 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:52:57,380 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:52:57,420 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:52:59,441 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:52:59,832 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:53:01,466 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:53:03,484 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:53:04,878 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:53:05,529 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:53:07,174 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:53:07,585 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:53:09,618 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:53:09,629 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:53:09,980 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:53:10,572 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:53:10,572 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:53:11,662 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:53:13,702 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:53:15,744 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:53:15,867 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:53:17,792 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:53:19,843 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:53:21,712 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:53:21,874 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:53:21,883 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:53:23,931 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:53:25,590 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:53:25,590 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:53:25,981 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:53:26,845 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:53:28,092 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:53:30,112 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:53:31,872 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:53:32,136 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:53:34,195 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:53:36,228 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:53:36,237 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:53:37,184 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:53:37,185 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:53:38,293 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:53:40,332 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:53:40,602 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:53:40,602 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:53:42,377 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:53:42,874 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:53:44,422 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:53:46,466 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 04:53:46,469 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 04:53:46,469 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 04:53:46,470 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 04:53:46,471 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 04:53:46,480 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:53:48,514 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:53:48,522 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:53:48,718 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:53:50,566 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:53:52,599 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:53:53,757 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:53:54,636 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:53:55,616 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:53:55,617 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:53:56,685 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:53:58,803 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:53:58,893 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:54:00,820 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:54:01,811 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:54:02,858 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:54:04,050 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:54:04,914 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:54:06,951 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:54:07,188 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:54:08,999 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:54:09,209 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:54:10,634 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:54:10,635 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:54:11,042 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:54:13,083 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:54:14,078 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:54:14,771 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:54:15,129 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:54:17,175 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:54:19,218 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:54:19,811 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:54:21,253 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:54:23,293 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:54:24,845 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:54:25,344 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:54:25,649 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:54:25,649 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:54:27,387 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:54:28,382 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:54:29,489 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:54:29,924 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:54:31,505 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:54:33,537 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:54:34,951 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:54:35,580 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:54:37,192 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:54:37,619 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:54:39,654 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:54:40,027 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:54:40,662 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:54:40,683 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:54:40,684 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:54:41,696 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:54:43,741 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:54:45,786 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:54:45,964 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:54:47,842 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:54:49,881 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:54:51,007 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:54:51,944 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:54:53,988 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:54:54,000 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:54:55,691 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:54:55,691 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:54:56,054 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:54:56,945 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:54:58,109 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:55:00,238 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:55:01,987 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:55:02,261 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:55:04,286 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:55:06,304 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:55:06,311 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:55:07,202 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:55:07,203 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:55:08,348 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:55:10,390 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:55:10,688 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:55:10,688 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:55:12,434 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:55:12,961 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:55:14,479 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:55:16,524 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:55:18,094 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:55:18,556 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:55:18,564 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:55:20,608 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:55:22,657 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:55:23,143 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:55:24,703 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:55:25,698 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:55:25,698 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:55:26,749 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:55:28,788 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:55:28,977 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:55:30,921 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:55:32,888 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:55:32,941 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:55:33,981 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:55:34,955 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:55:36,989 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:55:37,212 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:55:39,036 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:55:39,229 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:55:40,716 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:55:40,717 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:55:41,072 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:55:43,111 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:55:44,115 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:55:44,858 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:55:45,161 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:55:47,198 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:55:49,249 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:55:49,911 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:55:51,295 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:55:53,356 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:55:54,946 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:55:55,384 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:55:55,665 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 04:55:55,666 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 04:55:55,666 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 04:55:55,667 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 04:55:55,931 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:55:55,931 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:55:56,382 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:55:56,383 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 04:55:57,412 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:55:57,423 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:55:58,417 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:55:59,461 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:56:00,214 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:56:01,578 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:56:03,596 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:56:05,259 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:56:05,621 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:56:07,225 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:56:07,668 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:56:09,700 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:56:10,704 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:56:10,765 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:56:10,766 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:56:11,022 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:56:11,749 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:56:13,795 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:56:15,841 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:56:16,076 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:56:17,888 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:56:19,919 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:56:21,125 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:56:21,968 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:56:24,010 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:56:25,015 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:56:25,758 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:56:25,759 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:56:26,062 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:56:27,007 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:56:28,099 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:56:30,163 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:56:32,072 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:56:32,290 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:56:34,306 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:56:36,303 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:56:36,322 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:56:37,229 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:56:37,229 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:56:38,356 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:56:40,407 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:56:40,773 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:56:40,774 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:56:42,451 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:56:43,024 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:56:44,505 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:56:46,551 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:56:48,542 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:56:48,594 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:56:48,596 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:56:50,637 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:56:52,699 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:56:53,595 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:56:54,744 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:56:55,796 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:56:55,796 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:56:56,776 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:56:58,818 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:56:59,077 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:57:00,893 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:57:02,955 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:57:02,989 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:57:04,485 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:57:05,000 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:57:07,018 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:57:07,232 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:57:09,057 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:57:10,119 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:57:10,805 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:57:10,806 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:57:11,098 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:57:13,139 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:57:15,175 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:57:15,183 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:57:15,390 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:57:17,226 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:57:19,279 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:57:20,437 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:57:21,319 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:57:23,355 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:57:25,407 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:57:25,818 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:57:25,818 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:57:26,069 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:57:27,437 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:57:27,450 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:57:29,487 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:57:31,194 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:57:31,532 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:57:33,651 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:57:35,674 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:57:36,213 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:57:37,238 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:57:37,688 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:57:39,728 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:57:40,727 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:57:40,833 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:57:40,834 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:57:41,767 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:57:42,116 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:57:43,818 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:57:45,865 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:57:47,161 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:57:47,901 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:57:49,947 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:57:52,023 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:57:52,771 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:57:53,025 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:57:54,065 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:57:55,840 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:57:55,841 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:57:56,116 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:57:58,157 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:57:58,165 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:58:00,213 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:58:02,257 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:58:03,177 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:58:04,378 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:58:04,642 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 04:58:04,644 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 04:58:04,644 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 04:58:04,645 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 04:58:05,352 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:58:05,352 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 04:58:06,391 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:58:07,252 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:58:07,379 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:58:08,259 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:58:08,422 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:58:10,473 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:58:10,857 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:58:10,858 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:58:12,508 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:58:14,127 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:58:14,547 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:58:16,597 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:58:18,643 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:58:19,195 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:58:19,641 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:58:20,690 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:58:22,725 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:58:24,238 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:58:24,770 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:58:25,857 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:58:25,857 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:58:26,814 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:58:28,850 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:58:30,118 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:58:30,927 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:58:32,965 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:58:32,966 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:58:35,113 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:58:36,099 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:58:37,123 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:58:37,256 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:58:39,141 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:58:40,852 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:58:40,853 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:58:41,107 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:58:41,191 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:58:43,235 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:58:45,276 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:58:45,277 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:58:46,918 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:58:47,327 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:58:49,372 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:58:51,434 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:58:51,960 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:58:53,480 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:58:55,513 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:58:55,864 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:58:55,864 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:58:57,556 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:58:57,558 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:58:57,720 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:58:59,612 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:59:01,646 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:59:02,755 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:59:03,697 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:59:05,786 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:59:07,266 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:59:07,799 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:59:08,279 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:59:09,822 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:59:10,886 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:59:10,887 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:59:11,859 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:59:11,867 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:59:13,917 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:59:14,137 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:59:15,968 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:59:18,012 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:59:19,164 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:59:20,058 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:59:22,098 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:59:24,134 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:59:24,142 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:59:24,415 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:59:25,897 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:59:25,897 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:59:26,186 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:59:28,236 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:59:30,189 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:59:30,283 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:59:32,342 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:59:34,384 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:59:35,226 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:59:36,496 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:59:37,267 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 04:59:37,471 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:59:38,519 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:59:40,533 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:59:40,901 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:59:40,901 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:59:41,147 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:59:42,562 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:59:44,610 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:59:46,193 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:59:46,654 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:59:48,696 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:59:49,701 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 04:59:50,740 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:59:52,139 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:59:52,777 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:59:54,824 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:59:55,905 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 04:59:55,905 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 04:59:56,867 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 04:59:57,170 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 04:59:58,906 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:00:00,964 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:00:01,960 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:00:03,018 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:00:03,049 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:00:05,065 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:00:07,171 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:00:07,273 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:00:08,278 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:00:09,189 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:00:10,916 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:00:10,917 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:00:11,206 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:00:13,239 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:00:13,923 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:00:13,934 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 05:00:13,935 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 05:00:13,935 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 05:00:13,936 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 05:00:14,243 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 05:00:15,281 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:00:16,277 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:00:17,338 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:00:19,235 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:00:19,375 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:00:21,417 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:00:23,467 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:00:24,279 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:00:25,507 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:00:25,916 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:00:25,917 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:00:27,577 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:00:29,525 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:00:29,607 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:00:29,615 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:00:31,649 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:00:33,687 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:00:34,575 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:00:35,732 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:00:37,282 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:00:37,824 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:00:39,833 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:00:40,251 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:00:40,925 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:00:40,925 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:00:41,836 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:00:41,846 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:00:43,894 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:00:45,928 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:00:46,216 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:00:47,968 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:00:50,010 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:00:51,270 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:00:52,060 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:00:54,095 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:00:55,940 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:00:55,941 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:00:56,133 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:00:56,135 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:00:57,200 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:00:58,179 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:01:00,227 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:01:02,254 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:01:02,277 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:01:04,319 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:01:06,387 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:01:07,289 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:01:07,290 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:01:08,476 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:01:08,510 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:01:10,532 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:01:10,934 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:01:10,935 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:01:12,544 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:01:13,198 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:01:14,585 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:01:16,634 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:01:18,240 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:01:18,678 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:01:20,711 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:01:20,720 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:01:22,755 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:01:24,041 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:01:24,792 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:01:25,946 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:01:25,946 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:01:26,832 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:01:28,879 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:01:29,231 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:01:30,928 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:01:32,968 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:01:33,966 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:01:34,864 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:01:35,016 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:01:37,062 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:01:37,302 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:01:39,184 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:01:40,288 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:01:40,964 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:01:40,965 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:01:41,209 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:01:43,230 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:01:45,253 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:01:45,695 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:01:46,243 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:01:47,301 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:01:49,342 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:01:50,756 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:01:51,398 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:01:53,431 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:01:55,471 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:01:55,975 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:01:55,975 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:01:56,228 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:01:57,515 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:01:58,514 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:01:59,546 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:02:01,591 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:02:01,592 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:02:03,637 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:02:05,688 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:02:06,636 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:02:07,316 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:02:07,721 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:02:09,825 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:02:10,984 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:02:10,984 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:02:11,842 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:02:12,231 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:02:12,812 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:02:13,862 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:02:15,885 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:02:17,295 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:02:17,903 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:02:19,941 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:02:21,984 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:02:22,348 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:02:23,233 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 05:02:23,235 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 05:02:23,236 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 05:02:23,237 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 05:02:24,020 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 05:02:24,035 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:02:25,029 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:02:25,983 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:02:25,983 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:02:26,125 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:02:28,183 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:02:28,238 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:02:30,223 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:02:32,277 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:02:33,288 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:02:34,324 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:02:36,357 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:02:37,321 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:02:38,332 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:02:38,396 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:02:38,404 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:02:40,517 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:02:40,989 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:02:40,989 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:02:42,533 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:02:44,280 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:02:44,556 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:02:46,581 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:02:48,621 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:02:49,312 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:02:50,654 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:02:50,662 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:02:52,720 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:02:54,763 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:02:54,907 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:02:55,999 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:02:55,999 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:02:56,818 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:02:58,862 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:03:00,266 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:03:00,927 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:03:02,976 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:03:05,010 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:03:05,025 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:03:05,747 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:03:07,059 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:03:07,336 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:03:09,105 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:03:10,996 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:03:10,997 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:03:11,219 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:03:11,237 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:03:13,247 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:03:15,273 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:03:16,671 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:03:17,284 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:03:17,298 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:03:19,329 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:03:21,370 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:03:21,713 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:03:23,402 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:03:25,451 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:03:26,004 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:03:26,005 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:03:27,266 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:03:27,494 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:03:29,548 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:03:30,543 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:03:31,594 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:03:32,518 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:03:33,632 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:03:35,667 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:03:37,341 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:03:37,713 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:03:38,345 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:03:39,759 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:03:41,018 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:03:41,019 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:03:41,868 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:03:42,850 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:03:43,381 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:03:43,891 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:03:45,913 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:03:47,952 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:03:48,417 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:03:49,996 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:03:52,048 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:03:53,477 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:03:54,091 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:03:56,017 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:03:56,017 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:03:56,150 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:03:57,147 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:03:58,185 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:03:59,296 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:04:00,223 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:04:02,277 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:04:04,319 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:04:04,336 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:04:06,366 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:04:07,352 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:04:08,406 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:04:09,376 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:04:09,409 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:04:10,454 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:04:11,024 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:04:11,025 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:04:12,616 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:04:14,635 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:04:15,322 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:04:16,656 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:04:18,709 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:04:20,740 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:04:20,751 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:04:20,941 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:04:22,788 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:04:24,828 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:04:26,039 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:04:26,040 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:04:26,298 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:04:26,868 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:04:28,916 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:04:30,978 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:04:31,357 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:04:32,762 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 05:04:32,763 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 05:04:32,763 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 05:04:32,764 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 05:04:33,064 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 05:04:33,066 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:04:35,099 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:04:35,108 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:04:37,141 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:04:37,156 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:04:37,360 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:04:39,201 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:04:41,057 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:04:41,058 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:04:41,241 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:04:42,317 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:04:43,353 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:04:45,370 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:04:47,385 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:04:47,394 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:04:48,234 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:04:49,440 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:04:51,473 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:04:53,297 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:04:53,520 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:04:55,567 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:04:56,073 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:04:56,074 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:04:57,610 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:04:58,359 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:04:59,649 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:05:01,696 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:05:01,697 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:05:03,759 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:05:04,068 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:05:05,805 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:05:07,367 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:05:07,861 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:05:09,384 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:05:09,899 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:05:11,091 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:05:11,092 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:05:11,959 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:05:12,961 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:05:14,069 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:05:14,807 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:05:16,097 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:05:18,115 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:05:19,860 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:05:20,138 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:05:22,166 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:05:24,210 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:05:25,211 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:05:25,446 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:05:26,089 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:05:26,089 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:05:26,258 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:05:28,298 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:05:30,347 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:05:31,392 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:05:32,405 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:05:34,453 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:05:36,445 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:05:36,487 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:05:37,378 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:05:38,526 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:05:39,533 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:05:40,574 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:05:41,084 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:05:41,084 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:05:42,349 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:05:42,612 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:05:44,717 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:05:46,730 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:05:47,386 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:05:48,747 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:05:50,771 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:05:51,771 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:05:52,813 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:05:53,116 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:05:54,867 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:05:56,103 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:05:56,103 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:05:56,914 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:05:58,360 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:05:58,954 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:06:01,015 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:06:03,064 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:06:03,898 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:06:05,101 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:06:05,109 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:06:07,159 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:06:07,382 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:06:09,207 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:06:09,397 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:06:11,104 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:06:11,105 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:06:11,240 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:06:13,305 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:06:14,399 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:06:15,426 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:06:17,403 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:06:17,443 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:06:19,455 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:06:19,699 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:06:21,479 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:06:23,526 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:06:24,748 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:06:25,567 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:06:26,133 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:06:26,133 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:06:27,602 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:06:29,644 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:06:29,651 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:06:30,414 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:06:31,696 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:06:33,748 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:06:35,469 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:06:35,784 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:06:37,390 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:06:37,823 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:06:39,871 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:06:40,498 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:06:41,131 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:06:41,132 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:06:41,198 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 05:06:41,387 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 05:06:41,387 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 05:06:41,388 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 05:06:41,921 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 05:06:41,932 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:06:43,972 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:06:43,980 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:06:45,539 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:06:46,095 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:06:48,106 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:06:50,130 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:06:50,554 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:06:52,190 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:06:54,223 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:06:56,140 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:06:56,141 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:06:56,256 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:06:56,264 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:06:56,407 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:06:58,303 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:07:00,337 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:07:01,462 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:07:02,375 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:07:04,415 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:07:06,460 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:07:06,520 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:07:07,393 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:07:08,513 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:07:09,525 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:07:10,555 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:07:11,142 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:07:11,142 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:07:12,405 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:07:12,598 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:07:14,642 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:07:16,776 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:07:17,446 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:07:18,792 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:07:20,845 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:07:21,858 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:07:22,922 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:07:23,435 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:07:24,969 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:07:26,145 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:07:26,145 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:07:27,028 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:07:29,071 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:07:29,428 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:07:31,129 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:07:33,168 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:07:35,222 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:07:35,302 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:07:36,219 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:07:37,255 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:07:37,406 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:07:39,301 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:07:40,431 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:07:41,149 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:07:41,150 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:07:41,359 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:07:43,394 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:07:45,429 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:07:45,434 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:07:47,532 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:07:48,514 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:07:49,542 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:07:51,126 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:07:51,563 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:07:53,607 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:07:55,645 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:07:56,161 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:07:56,161 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:07:56,407 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:07:57,700 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:07:59,735 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:07:59,745 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:08:01,787 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:08:01,926 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:08:03,858 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:08:05,887 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:08:06,983 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:08:07,411 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:08:07,930 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:08:09,971 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:08:11,157 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:08:11,157 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:08:12,016 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:08:12,733 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:08:14,056 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:08:14,064 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:08:16,117 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:08:17,780 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:08:18,224 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:08:20,248 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:08:22,268 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:08:22,821 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:08:24,308 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:08:26,159 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:08:26,160 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:08:26,334 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:08:26,346 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:08:28,379 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:08:28,414 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:08:30,418 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:08:32,471 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:08:33,470 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:08:34,507 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:08:36,550 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:08:37,425 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:08:38,596 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:08:38,603 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:08:39,440 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:08:40,640 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:08:41,162 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:08:41,163 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:08:42,687 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:08:44,442 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:08:44,740 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:08:46,777 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:08:48,882 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:08:49,503 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:08:50,135 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 05:08:50,136 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 05:08:50,136 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 05:08:50,137 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 05:08:50,868 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 05:08:50,901 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:08:51,870 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:08:52,928 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:08:54,956 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:08:55,464 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:08:56,173 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:08:56,173 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:08:56,990 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:08:59,027 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:09:00,474 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:09:01,089 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:09:03,122 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:09:04,124 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:09:05,168 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:09:05,685 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:09:07,228 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:09:07,431 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:09:09,273 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:09:11,188 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:09:11,188 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:09:11,310 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:09:11,437 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:09:13,348 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:09:15,402 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:09:16,480 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:09:17,450 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:09:18,448 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:09:19,558 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:09:21,569 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:09:21,579 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:09:23,605 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:09:25,633 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:09:26,217 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:09:26,217 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:09:27,461 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:09:27,663 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:09:29,710 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:09:30,707 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:09:31,750 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:09:33,408 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:09:33,799 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:09:35,839 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:09:37,440 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:09:37,880 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:09:38,454 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:09:39,919 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:09:41,220 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:09:41,221 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:09:41,964 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:09:44,019 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:09:44,191 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:09:45,029 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:09:46,078 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:09:48,118 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:09:49,245 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:09:50,228 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:09:52,251 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:09:54,267 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:09:54,289 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:09:56,228 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:09:56,229 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:09:56,278 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:09:56,286 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:09:58,330 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:09:59,515 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:10:00,376 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:10:02,433 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:10:04,484 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:10:04,566 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:10:06,537 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:10:07,447 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:10:08,568 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:10:08,577 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:10:09,829 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:10:10,618 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:10:11,234 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:10:11,235 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:10:12,661 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:10:14,709 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:10:15,516 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:10:16,775 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:10:18,829 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:10:20,557 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:10:20,935 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:10:22,926 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:10:22,950 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:10:24,968 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:10:25,631 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:10:26,245 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:10:26,245 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:10:26,989 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:10:29,024 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:10:31,077 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:10:31,543 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:10:33,118 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:10:35,145 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:10:35,154 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:10:37,193 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:10:37,452 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:10:37,453 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:10:39,234 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:10:41,260 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:10:41,261 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:10:41,269 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:10:42,511 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:10:43,311 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:10:45,351 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:10:47,396 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:10:48,388 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:10:48,389 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:10:49,432 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:10:51,545 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:10:53,434 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:10:53,572 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:10:55,581 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:10:56,257 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:10:56,258 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:10:57,605 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:10:58,524 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:10:59,220 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 05:10:59,222 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 05:10:59,222 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 05:10:59,223 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 05:10:59,661 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 05:10:59,675 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:11:00,674 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:11:01,712 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:11:03,746 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:11:04,484 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:11:05,790 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:11:07,465 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:11:07,828 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:11:09,884 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:11:10,489 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:11:11,273 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:11:11,274 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:11:11,941 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:11:13,983 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:11:14,992 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:11:15,616 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:11:16,046 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:11:18,081 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:11:20,135 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:11:20,668 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:11:22,244 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:11:24,263 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:11:25,677 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:11:26,278 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:11:26,279 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:11:26,288 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:11:27,256 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:11:28,300 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:11:30,348 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:11:31,579 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:11:32,398 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:11:34,442 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:11:36,489 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:11:36,624 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:11:37,478 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:11:38,532 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:11:39,525 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:11:40,580 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:11:41,300 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:11:41,301 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:11:42,561 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:11:42,627 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:11:44,666 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:11:46,702 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:11:47,585 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:11:48,746 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:11:50,788 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:11:52,699 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:11:52,871 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:11:52,912 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:11:54,935 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:11:56,308 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:11:56,309 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:11:56,949 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:11:58,573 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:11:58,974 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:12:01,027 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:12:03,073 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:12:04,445 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:12:05,120 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:12:05,127 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:12:07,176 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:12:07,483 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:12:09,219 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:12:09,499 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:12:11,266 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:12:11,320 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:12:11,321 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:12:13,299 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:12:14,587 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:12:15,358 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:12:17,390 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:12:17,401 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:12:19,438 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:12:20,182 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:12:21,478 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:12:23,586 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:12:25,238 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:12:25,598 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:12:26,349 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:12:26,350 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:12:27,622 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:12:29,654 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:12:30,904 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:12:31,690 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:12:31,702 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:12:33,747 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:12:35,792 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:12:35,951 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:12:37,489 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:12:37,834 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:12:39,868 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:12:41,354 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:12:41,355 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:12:41,611 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:12:41,908 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:12:42,906 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:12:43,948 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:12:45,995 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:12:46,666 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:12:48,043 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:12:50,082 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:12:51,707 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:12:52,133 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:12:54,241 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:12:55,198 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:12:56,262 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:12:56,369 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:12:56,369 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:12:57,626 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:12:58,288 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:13:00,316 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:13:02,349 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:13:02,661 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:13:04,389 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:13:06,433 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:13:07,046 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 05:13:07,047 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 05:13:07,048 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 05:13:07,049 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 05:13:07,425 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 05:13:07,502 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:13:08,478 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:13:08,507 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:13:09,468 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:13:10,530 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:13:11,389 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:13:11,390 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:13:12,570 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:13:13,676 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:13:14,600 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:13:16,641 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:13:18,687 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:13:18,706 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:13:20,732 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:13:21,725 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:13:22,777 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:13:24,524 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:13:24,887 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:13:26,397 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:13:26,398 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:13:26,895 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:13:28,918 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:13:29,646 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:13:30,966 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:13:33,021 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:13:35,086 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:13:35,282 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:13:36,089 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:13:37,147 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:13:37,518 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:13:39,193 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:13:40,540 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:13:41,238 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:13:41,417 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:13:41,418 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:13:43,279 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:13:45,325 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:13:45,707 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:13:47,367 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:13:47,375 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:13:49,412 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:13:50,998 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:13:51,452 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:13:53,500 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:13:55,609 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:13:56,431 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:13:56,432 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:13:56,683 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:13:57,632 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:13:59,633 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:13:59,655 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:14:01,686 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:14:02,670 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:14:03,725 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:14:05,758 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:14:07,531 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:14:07,808 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:14:08,540 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:14:09,851 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:14:11,444 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:14:11,445 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:14:11,906 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:14:13,710 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:14:13,931 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:14:13,940 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:14:15,975 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:14:18,021 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:14:18,759 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:14:20,073 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:14:22,117 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:14:23,807 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:14:24,165 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:14:26,264 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:14:26,286 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:14:26,466 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:14:26,467 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:14:28,298 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:14:29,748 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:14:30,309 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:14:32,362 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:14:34,408 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:14:34,793 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:14:36,460 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:14:37,469 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:14:37,528 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:14:38,507 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:14:40,552 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:14:40,555 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:14:41,463 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:14:41,464 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:14:42,610 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:14:44,648 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:14:45,738 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:14:46,693 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:14:48,730 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:14:50,774 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:14:51,443 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:14:51,770 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:14:52,829 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:14:54,872 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:14:56,471 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:14:56,471 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:14:56,725 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:14:56,970 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:14:58,984 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:15:01,008 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:15:01,761 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:15:03,067 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:15:04,069 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:15:05,114 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:15:07,144 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:15:07,183 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:15:07,533 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:15:09,203 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:15:11,233 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:15:11,473 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:15:11,474 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:15:12,738 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:15:13,277 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:15:14,882 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 05:15:14,883 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 05:15:14,883 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 05:15:14,884 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 05:15:15,339 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:15:15,339 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 05:15:15,341 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:15:16,355 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:15:17,399 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:15:18,176 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:15:19,438 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:15:21,477 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:15:23,232 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:15:23,525 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:15:25,567 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:15:26,465 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:15:26,465 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:15:27,678 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:15:29,180 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:15:29,700 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:15:30,685 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:15:31,720 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:15:33,753 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:15:34,220 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:15:35,803 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:15:37,546 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:15:37,857 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:15:39,559 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:15:39,903 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:15:41,470 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:15:41,471 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:15:41,937 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:15:41,948 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:15:43,978 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:15:44,746 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:15:46,026 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:15:48,058 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:15:49,798 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:15:50,095 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:15:52,162 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:15:54,194 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:15:54,205 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:15:55,443 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:15:56,245 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:15:56,471 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:15:56,472 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:15:58,344 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:16:00,353 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:16:00,778 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:16:02,374 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:16:04,416 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:16:05,841 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:16:06,468 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:16:07,557 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:16:08,505 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:16:08,514 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:16:10,559 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:16:11,490 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:16:11,490 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:16:11,746 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:16:12,607 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:16:14,656 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:16:16,706 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:16:16,790 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:16:18,744 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:16:20,789 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:16:20,796 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:16:22,693 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:16:22,844 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:16:24,891 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:16:26,507 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:16:26,508 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:16:26,937 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:16:27,778 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:16:29,054 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:16:31,071 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:16:32,050 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:16:33,099 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:16:33,312 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:16:35,135 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:16:37,201 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:16:37,573 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:16:38,586 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:16:39,239 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:16:41,292 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:16:41,518 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:16:41,519 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:16:43,342 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:16:43,775 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:16:45,382 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:16:46,388 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:16:47,435 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:16:48,987 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:16:49,497 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:16:51,533 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:16:53,570 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:16:54,014 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:16:55,607 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:16:56,528 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:16:56,528 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:16:57,655 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:16:58,654 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:16:59,769 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:16:59,802 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:17:01,788 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:17:03,798 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:17:04,849 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:17:05,826 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:17:07,585 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:17:07,864 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:17:09,907 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:17:10,305 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:17:10,898 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:17:11,544 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:17:11,545 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:17:11,960 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:17:14,008 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:17:15,814 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:17:16,050 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:17:18,098 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:17:20,129 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:17:20,856 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:17:21,974 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 05:17:21,975 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 05:17:21,975 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 05:17:21,977 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 05:17:22,175 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 05:17:22,187 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:17:24,224 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:17:24,236 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:17:25,889 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:17:26,275 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:17:26,561 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:17:26,561 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:17:28,315 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:17:30,424 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:17:31,849 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:17:32,446 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:17:34,469 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:17:36,477 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:17:36,485 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:17:37,440 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:17:37,599 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:17:38,523 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:17:40,571 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:17:41,568 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:17:41,569 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:17:42,622 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:17:42,923 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:17:44,667 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:17:46,715 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:17:47,955 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:17:48,749 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:17:48,756 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:17:50,801 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:17:52,847 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:17:53,124 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:17:54,885 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:17:56,593 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:17:56,593 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:17:56,921 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:17:58,870 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:17:58,963 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:18:01,086 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:18:03,056 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:18:03,095 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:18:04,829 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:18:05,106 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:18:07,149 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:18:07,615 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:18:09,190 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:18:10,641 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:18:11,235 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:18:11,597 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:18:11,597 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:18:13,277 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:18:15,320 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:18:15,330 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:18:16,467 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:18:17,379 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:18:19,423 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:18:21,466 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:18:21,517 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:18:23,513 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:18:25,546 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:18:26,545 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:18:26,592 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:18:26,593 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:18:26,842 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:18:27,590 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:18:29,643 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:18:31,767 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:18:31,869 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:18:33,788 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:18:35,817 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:18:36,913 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:18:37,628 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:18:37,867 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:18:39,905 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:18:40,902 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:18:41,594 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:18:41,594 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:18:41,959 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:18:42,845 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:18:44,001 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:18:46,044 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:18:47,892 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:18:48,088 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:18:50,133 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:18:52,186 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:18:53,190 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:18:53,378 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:18:54,228 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:18:56,274 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:18:56,591 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:18:56,592 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:18:58,311 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:18:58,859 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:19:00,358 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:19:02,472 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:19:03,891 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:19:04,482 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:19:05,465 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:19:06,505 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:19:07,629 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:19:08,548 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:19:09,653 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:19:10,584 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:19:11,596 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:19:11,596 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:19:12,631 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:19:14,677 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:19:14,859 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:19:16,732 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:19:18,775 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:19:18,776 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:19:20,709 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:19:20,832 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:19:22,879 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:19:24,931 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:19:25,751 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:19:26,602 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:19:26,603 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:19:26,981 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:19:29,011 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:19:29,294 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 05:19:29,296 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 05:19:29,297 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 05:19:29,298 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 05:19:30,013 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 05:19:31,057 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:19:31,070 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:19:31,558 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:19:33,215 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:19:35,251 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:19:36,599 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:19:37,271 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:19:37,642 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:19:39,290 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:19:41,322 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:19:41,624 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:19:41,625 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:19:41,881 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:19:43,382 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:19:45,410 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:19:45,422 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:19:47,455 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:19:47,667 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:19:49,495 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:19:51,543 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:19:52,714 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:19:53,586 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:19:55,635 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:19:56,631 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:19:56,632 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:19:57,679 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:19:57,680 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:19:57,896 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:19:59,729 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:20:01,766 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:20:02,937 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:20:03,872 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:20:05,897 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:20:07,655 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:20:07,913 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:20:08,886 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:20:08,887 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:20:09,936 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:20:11,654 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:20:11,655 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:20:11,989 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:20:13,927 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:20:14,030 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:20:16,073 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:20:18,117 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:20:18,969 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:20:20,164 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:20:21,167 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:20:22,209 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:20:24,255 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:20:24,557 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:20:26,301 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:20:26,652 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:20:26,652 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:20:28,346 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:20:29,933 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:20:30,391 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:20:32,447 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:20:34,550 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:20:35,196 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:20:35,510 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:20:36,565 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:20:37,669 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:20:38,582 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:20:40,619 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:20:40,704 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:20:41,670 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:20:41,671 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:20:42,663 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:20:44,707 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:20:45,842 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:20:46,754 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:20:47,755 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:20:48,806 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:20:50,850 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:20:50,888 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:20:52,892 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:20:54,942 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:20:55,932 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:20:56,686 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:20:56,686 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:20:56,979 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:20:59,024 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:21:00,024 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:21:01,080 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:21:01,526 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:21:03,115 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:21:05,212 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:21:06,577 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:21:07,232 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:21:07,682 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:21:09,256 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:21:11,280 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:21:11,682 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:21:11,683 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:21:11,934 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:21:13,305 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:21:13,313 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:21:15,353 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:21:16,971 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:21:17,399 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:21:19,436 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:21:21,481 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:21:22,020 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:21:23,520 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:21:25,553 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:21:25,563 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:21:26,687 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:21:26,688 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:21:27,600 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:21:28,002 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:21:29,640 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:21:31,684 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:21:33,057 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:21:33,735 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:21:35,857 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:21:36,566 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 05:21:36,567 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 05:21:36,568 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 05:21:36,569 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 05:21:36,820 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 05:21:37,688 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:21:37,833 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:21:37,867 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:21:38,699 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:21:38,843 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:21:39,872 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:21:39,887 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:21:41,704 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:21:41,705 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:21:41,919 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:21:43,978 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:21:43,980 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:21:46,015 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:21:48,071 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:21:49,050 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:21:50,118 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:21:52,165 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:21:52,181 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:21:54,227 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:21:54,869 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:21:56,271 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:21:56,716 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:21:56,716 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:21:58,315 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:21:59,990 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:22:00,361 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:22:02,409 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:22:03,408 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:22:04,458 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:22:05,453 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:22:06,554 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:22:07,695 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:22:08,571 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:22:10,586 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:22:10,720 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:22:11,722 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:22:11,723 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:22:12,613 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:22:14,664 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:22:16,048 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:22:16,712 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:22:17,712 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:22:18,748 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:22:20,789 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:22:21,111 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:22:22,833 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:22:24,868 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:22:26,714 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:22:26,714 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:22:26,915 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:22:26,955 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:22:28,960 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:22:29,963 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:22:31,022 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:22:32,682 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:22:33,068 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:22:35,111 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:22:37,226 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:22:37,707 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:22:37,708 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:22:39,245 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:22:41,265 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:22:41,724 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:22:41,725 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:22:42,271 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:22:42,992 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:22:43,305 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:22:45,342 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:22:47,384 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:22:48,036 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:22:49,456 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:22:51,508 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:22:53,558 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:22:53,997 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:22:54,556 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:22:55,602 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:22:56,733 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:22:56,734 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:22:57,646 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:22:59,000 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:22:59,686 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:23:01,729 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:23:03,787 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:23:04,034 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:23:05,840 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:23:07,721 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:23:07,901 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:23:07,950 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:23:09,742 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:23:09,976 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:23:11,741 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:23:11,741 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:23:11,999 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:23:14,015 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:23:15,019 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:23:16,056 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:23:18,100 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:23:20,141 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:23:20,149 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:23:20,287 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:23:22,201 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:23:24,242 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:23:25,316 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:23:26,282 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:23:26,751 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:23:26,751 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:23:28,321 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:23:30,366 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:23:30,935 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:23:32,411 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:23:32,419 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:23:34,480 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:23:36,006 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:23:36,518 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:23:37,730 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:23:38,622 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:23:40,637 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:23:41,051 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:23:41,751 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:23:41,752 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:23:42,648 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:23:43,626 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 05:23:43,628 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 05:23:43,628 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 05:23:43,629 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 05:23:43,630 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 05:23:44,681 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:23:45,686 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:23:46,725 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:23:46,733 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:23:46,921 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:23:48,771 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:23:50,814 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:23:51,956 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:23:52,859 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:23:54,907 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:23:56,764 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:23:56,765 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:23:56,949 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:23:57,022 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:23:57,956 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:23:58,995 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:24:01,057 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:24:02,077 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:24:03,091 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:24:05,147 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:24:07,172 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:24:07,183 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:24:07,744 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:24:09,282 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:24:10,267 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:24:11,314 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:24:11,767 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:24:11,767 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:24:13,045 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:24:13,333 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:24:15,359 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:24:17,394 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:24:18,102 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:24:19,447 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:24:21,490 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:24:23,141 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:24:23,537 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:24:24,552 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:24:25,574 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:24:26,767 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:24:26,767 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:24:27,612 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:24:29,032 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:24:29,674 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:24:31,708 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:24:33,750 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:24:34,066 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:24:35,784 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:24:36,782 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:24:37,745 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:24:37,823 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:24:39,761 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:24:39,944 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:24:41,782 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:24:41,782 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:24:41,970 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:24:43,996 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:24:45,051 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:24:46,026 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:24:48,069 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:24:49,062 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:24:50,114 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:24:50,304 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:24:52,168 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:24:54,215 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:24:55,359 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:24:56,258 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:24:56,782 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:24:56,783 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:24:58,299 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:25:00,339 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:25:00,347 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:25:00,895 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:25:02,386 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:25:04,427 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:25:05,941 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:25:06,478 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:25:07,746 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:25:08,511 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:25:10,621 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:25:11,111 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:25:11,783 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:25:11,783 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:25:12,637 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:25:14,656 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:25:14,668 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:25:16,534 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:25:16,712 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:25:18,756 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:25:20,797 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:25:21,593 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:25:22,840 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:25:24,873 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:25:26,781 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:25:26,781 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:25:26,909 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:25:26,917 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:25:27,024 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:25:28,962 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:25:31,006 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:25:32,057 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:25:33,037 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:25:35,099 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:25:37,138 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:25:37,146 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:25:37,756 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:25:39,182 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:25:39,190 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:25:41,295 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:25:41,776 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:25:41,777 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:25:43,041 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:25:43,320 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:25:45,352 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:25:47,395 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:25:48,099 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:25:49,443 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:25:50,299 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 05:25:50,300 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 05:25:50,301 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 05:25:50,301 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 05:25:50,445 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 05:25:51,494 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:25:52,496 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:25:53,559 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:25:53,578 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:25:55,620 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:25:56,790 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:25:56,791 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:25:57,669 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:25:59,300 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:25:59,714 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:26:01,758 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:26:03,805 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:26:04,576 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:26:04,812 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:26:05,848 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:26:07,757 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:26:07,895 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:26:09,782 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:26:09,941 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:26:11,796 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:26:11,797 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:26:12,070 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:26:14,081 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:26:15,068 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:26:16,104 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:26:17,074 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:26:18,123 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:26:20,141 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:26:20,170 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:26:22,207 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:26:24,250 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:26:25,246 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:26:26,298 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:26:26,811 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:26:26,812 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:26:28,331 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:26:30,377 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:26:30,769 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:26:31,376 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:26:32,430 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:26:34,488 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:26:35,813 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:26:36,537 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:26:37,767 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:26:38,579 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:26:40,645 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:26:41,173 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:26:41,817 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:26:41,818 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:26:42,723 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:26:42,763 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:26:44,785 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:26:46,804 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:26:47,101 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:26:48,846 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:26:50,897 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:26:52,119 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:26:52,946 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:26:54,975 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:26:54,988 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:26:56,813 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:26:56,814 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:26:57,037 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:26:58,064 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:26:59,071 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:27:01,123 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:27:03,119 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:27:03,175 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:27:05,210 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:27:07,254 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:27:07,780 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:27:08,791 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:27:09,280 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:27:09,289 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:27:11,331 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:27:11,826 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:27:11,827 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:27:13,443 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:27:14,079 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:27:15,461 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:27:17,479 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:27:19,124 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:27:19,523 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:27:21,558 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:27:21,565 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:27:23,602 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:27:24,364 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:27:25,655 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:27:26,848 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:27:26,848 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:27:27,694 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:27:29,740 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:27:30,117 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:27:31,775 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:27:33,811 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:27:33,819 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:27:35,860 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:27:35,890 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:27:37,786 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:27:37,891 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:27:39,933 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:27:41,212 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:27:41,847 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:27:41,847 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:27:41,986 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:27:44,087 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:27:45,062 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:27:46,105 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:27:46,474 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:27:48,121 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:27:50,160 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:27:51,510 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:27:52,211 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:27:54,250 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:27:56,306 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:27:56,843 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:27:56,844 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:27:56,995 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 05:27:57,095 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 05:27:57,095 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:27:57,095 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 05:27:57,097 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 05:27:57,299 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 05:27:58,364 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:27:59,366 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:28:00,420 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:28:02,297 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:28:02,464 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:28:04,513 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:28:06,559 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:28:07,348 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:28:07,793 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:28:08,594 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:28:10,631 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:28:11,626 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:28:11,858 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:28:11,859 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:28:12,675 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:28:13,119 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:28:14,773 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:28:16,784 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:28:18,173 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:28:18,804 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:28:20,849 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:28:22,889 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:28:23,886 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:28:23,933 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:28:24,926 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:28:26,863 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:28:26,864 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:28:26,966 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:28:29,009 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:28:29,127 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:28:31,062 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:28:33,105 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:28:34,186 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:28:35,143 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:28:37,182 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:28:37,806 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:28:38,186 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:28:39,223 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:28:39,830 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:28:41,261 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:28:41,859 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:28:41,859 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:28:43,303 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:28:45,140 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:28:45,408 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:28:47,425 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:28:49,416 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:28:49,436 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:28:51,109 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:28:51,485 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:28:53,515 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:28:55,553 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:28:56,144 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:28:56,872 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:28:56,872 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:28:57,606 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:28:59,652 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:29:01,704 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:29:01,705 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:29:01,713 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:29:03,751 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:29:05,784 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:29:06,785 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:29:07,816 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:29:07,828 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:29:09,864 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:29:11,905 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:29:11,906 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:29:11,922 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:29:12,153 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:29:13,951 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:29:16,029 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:29:16,050 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:29:17,340 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:29:18,073 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:29:20,085 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:29:22,112 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:29:22,400 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:29:24,145 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:29:26,184 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:29:26,883 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:29:26,884 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:29:28,140 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:29:28,226 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:29:28,234 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:29:30,274 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:29:32,326 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:29:33,186 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:29:34,369 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:29:36,413 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:29:37,819 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:29:38,437 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:29:38,445 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:29:40,483 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:29:40,491 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:29:41,887 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:29:41,888 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:29:42,533 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:29:44,143 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:29:44,579 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:29:46,690 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:29:48,710 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:29:49,192 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:29:50,731 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:29:51,719 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:29:52,782 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:29:54,827 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:29:54,975 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:29:56,878 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:29:56,909 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:29:56,909 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:29:58,927 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:30:00,197 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:30:00,979 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:30:03,019 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:30:03,537 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 05:30:03,539 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 05:30:03,539 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 05:30:03,540 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 05:30:04,014 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 05:30:05,044 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:30:05,051 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:30:05,793 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:30:06,047 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:30:07,118 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:30:07,826 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:30:09,158 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:30:10,859 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:30:11,205 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:30:11,915 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:30:11,916 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:30:13,251 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:30:15,291 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:30:16,211 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:30:17,406 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:30:18,364 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:30:19,417 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:30:21,437 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:30:21,745 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:30:23,463 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:30:25,502 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:30:26,928 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:30:26,928 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:30:27,176 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:30:27,559 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:30:29,604 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:30:30,603 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:30:31,644 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:30:32,302 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:30:33,683 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:30:35,725 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:30:37,349 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:30:37,755 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:30:37,841 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:30:39,803 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:30:41,861 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:30:41,939 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:30:41,939 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:30:43,189 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:30:43,890 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:30:43,902 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:30:45,948 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:30:48,055 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:30:48,234 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:30:50,078 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:30:52,098 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:30:53,289 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:30:54,142 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:30:56,171 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:30:56,179 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:30:56,930 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:30:56,931 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:30:58,226 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:30:59,211 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:31:00,267 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:31:02,317 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:31:04,256 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:31:04,345 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:31:06,386 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:31:07,842 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:31:08,414 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:31:08,421 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:31:09,860 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:31:10,478 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:31:11,936 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:31:11,936 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:31:12,528 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:31:14,561 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:31:15,230 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:31:16,602 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:31:18,717 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:31:20,698 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:31:20,713 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:31:20,750 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:31:22,762 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:31:24,788 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:31:25,767 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:31:26,829 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:31:26,944 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:31:26,944 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:31:28,868 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:31:30,924 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:31:31,227 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:31:32,955 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:31:34,986 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:31:34,996 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:31:36,293 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:31:37,046 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:31:37,841 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:31:39,095 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:31:41,136 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:31:41,346 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:31:41,973 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:31:41,974 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:31:43,180 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:31:45,227 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:31:46,222 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:31:46,870 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:31:47,263 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:31:49,371 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:31:51,388 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:31:51,932 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:31:53,410 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:31:55,439 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:31:56,973 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:31:56,973 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:31:57,211 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:31:57,486 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:31:58,487 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:31:59,531 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:32:01,579 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:32:02,551 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:32:03,616 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:32:05,651 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:32:07,600 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:32:07,689 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:32:07,856 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:32:09,729 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:32:10,033 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 05:32:10,036 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 05:32:10,036 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 05:32:10,037 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 05:32:10,732 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 05:32:11,756 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:32:11,763 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:32:11,968 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:32:11,969 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:32:12,761 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:32:13,218 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:32:13,799 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:32:15,834 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:32:17,889 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:32:18,273 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:32:20,013 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:32:22,028 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:32:23,289 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:32:24,043 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:32:25,033 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:32:26,105 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:32:26,976 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:32:26,977 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:32:28,186 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:32:29,243 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:32:30,246 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:32:32,298 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:32:34,327 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:32:34,339 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:32:36,394 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:32:37,870 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:32:38,416 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:32:38,425 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:32:39,882 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:32:40,468 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:32:41,990 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:32:41,990 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:32:42,511 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:32:44,556 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:32:45,271 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:32:46,599 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:32:48,639 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:32:50,713 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:32:50,750 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:32:50,782 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:32:52,775 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:32:54,802 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:32:55,807 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:32:56,832 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:32:57,007 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:32:57,008 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:32:58,871 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:33:00,925 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:33:01,272 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:33:02,955 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:33:02,965 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:33:05,008 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:33:06,434 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:33:07,054 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:33:07,873 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:33:09,098 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:33:11,144 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:33:12,018 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:33:12,019 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:33:12,267 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:33:13,207 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:33:15,247 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:33:15,254 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:33:17,299 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:33:18,085 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:33:19,348 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:33:21,461 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:33:23,119 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:33:23,476 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:33:25,518 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:33:27,019 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:33:27,020 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:33:27,563 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:33:28,274 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:33:28,557 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:33:29,613 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:33:31,654 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:33:33,295 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:33:33,704 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:33:35,750 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:33:37,797 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:33:37,876 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:33:38,880 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:33:39,844 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:33:40,837 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:33:41,888 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:33:42,008 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:33:42,009 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:33:43,928 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:33:44,282 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:33:45,968 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:33:48,032 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:33:49,337 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:33:50,071 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:33:52,199 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:33:53,173 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:33:54,228 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:33:54,887 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:33:56,238 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:33:57,017 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:33:57,017 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:33:58,264 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:34:00,298 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:34:00,307 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:34:02,349 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:34:04,395 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:34:05,501 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:34:06,439 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:34:07,430 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:34:07,889 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:34:08,486 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:34:10,522 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:34:10,907 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:34:12,024 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:34:12,024 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:34:12,564 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:34:14,613 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:34:16,299 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:34:16,651 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:34:17,126 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 05:34:17,127 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 05:34:17,127 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 05:34:17,128 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 05:34:17,659 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 05:34:18,693 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:34:18,704 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:34:19,696 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:34:20,753 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:34:21,388 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:34:22,860 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:34:24,879 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:34:26,449 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:34:26,900 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:34:27,033 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:34:27,034 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:34:28,940 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:34:30,981 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:34:30,991 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:34:32,387 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:34:33,031 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:34:35,066 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:34:37,116 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:34:37,438 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:34:37,899 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:34:39,156 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:34:41,197 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:34:42,038 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:34:42,039 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:34:42,998 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:34:43,247 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:34:45,286 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:34:45,298 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:34:47,335 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:34:48,046 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:34:49,372 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:34:51,417 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:34:53,079 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:34:53,536 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:34:55,559 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:34:57,065 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:34:57,066 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:34:57,578 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:34:57,585 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:34:58,328 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:34:59,632 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:35:01,681 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:35:03,389 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:35:03,721 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:35:05,769 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:35:07,812 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:35:07,906 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:35:09,268 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:35:09,837 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:35:09,845 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:35:11,909 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:35:12,083 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:35:12,083 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:35:13,956 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:35:14,349 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:35:16,002 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:35:18,043 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:35:19,390 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:35:20,089 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:35:22,125 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:35:23,123 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:35:24,284 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:35:24,934 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:35:26,293 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:35:27,094 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:35:27,095 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:35:28,313 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:35:30,368 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:35:30,368 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:35:32,433 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:35:34,490 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:35:35,487 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:35:35,598 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:35:36,538 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:35:37,921 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:35:38,581 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:35:40,630 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:35:40,951 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:35:42,106 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:35:42,107 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:35:42,671 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:35:44,711 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:35:46,198 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:35:46,744 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:35:47,749 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:35:48,790 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:35:50,832 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:35:51,247 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:35:52,882 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:35:55,002 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:35:56,288 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:35:57,027 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:35:57,108 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:35:57,108 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:35:59,045 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:36:00,002 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:36:01,067 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:36:01,797 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:36:03,116 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:36:05,158 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:36:06,846 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:36:07,201 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:36:07,935 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:36:09,245 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:36:11,289 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:36:12,127 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:36:12,127 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:36:12,374 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:36:13,317 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:36:13,326 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:36:15,365 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:36:17,407 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:36:17,409 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:36:19,470 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:36:21,524 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:36:22,456 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:36:23,562 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:36:24,156 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 05:36:24,158 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 05:36:24,158 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 05:36:24,158 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 05:36:24,558 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:36:24,559 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 05:36:25,627 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:36:25,663 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:36:27,128 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:36:27,129 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:36:27,671 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:36:28,390 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:36:29,699 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:36:31,738 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:36:33,435 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:36:33,784 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:36:35,828 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:36:37,873 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:36:37,949 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:36:38,958 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:36:39,927 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:36:39,941 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:36:41,984 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:36:42,128 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:36:42,128 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:36:44,028 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:36:44,388 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:36:46,071 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:36:48,105 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:36:49,463 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:36:50,163 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:36:52,197 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:36:52,211 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:36:54,243 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:36:55,005 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:36:56,348 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:36:57,139 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:36:57,140 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:36:58,359 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:37:00,379 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:37:00,419 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:37:02,422 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:37:04,454 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:37:04,461 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:37:05,611 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:37:06,502 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:37:07,955 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:37:08,537 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:37:10,590 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:37:10,982 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:37:12,147 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:37:12,147 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:37:12,648 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:37:14,689 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:37:15,692 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:37:16,208 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:37:16,730 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:37:18,775 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:37:20,812 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:37:21,246 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:37:22,854 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:37:24,894 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:37:26,288 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:37:26,995 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:37:27,158 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:37:27,159 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:37:29,008 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:37:29,998 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:37:31,044 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:37:31,801 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:37:33,081 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:37:35,136 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:37:36,851 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:37:37,178 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:37:37,962 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:37:39,216 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:37:41,271 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:37:42,155 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:37:42,156 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:37:42,265 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:37:42,406 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:37:43,313 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:37:45,366 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:37:47,408 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:37:47,440 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:37:49,451 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:37:51,501 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:37:52,482 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:37:53,557 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:37:54,567 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:37:55,614 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:37:57,161 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:37:57,162 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:37:57,738 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:37:58,424 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:37:59,766 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:38:01,793 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:38:03,487 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:38:03,812 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:38:05,833 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:38:05,847 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:38:07,893 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:38:07,972 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:38:08,981 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:38:09,926 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:38:11,980 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:38:12,175 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:38:12,175 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:38:14,018 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:38:14,446 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:38:16,067 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:38:18,106 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:38:20,149 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:38:20,157 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:38:20,176 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:38:22,200 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:38:24,238 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:38:25,244 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:38:26,285 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:38:27,186 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:38:27,187 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:38:28,426 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:38:30,455 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:38:30,472 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:38:30,704 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 05:38:30,705 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 05:38:30,705 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 05:38:30,706 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 05:38:31,410 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:38:31,410 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 05:38:32,440 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:38:32,477 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:38:34,511 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:38:36,066 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:38:36,568 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:38:37,978 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:38:38,633 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:38:40,687 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:38:41,614 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:38:42,183 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:38:42,184 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:38:42,747 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:38:44,788 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:38:46,825 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:38:46,837 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:38:47,040 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:38:48,876 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:38:50,920 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:38:52,074 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:38:52,968 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:38:55,004 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:38:57,044 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:38:57,188 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:38:57,189 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:38:57,444 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:38:58,051 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:38:59,167 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:39:01,196 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:39:02,484 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:39:03,222 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:39:05,267 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:39:07,316 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:39:07,535 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:39:07,992 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:39:09,353 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:39:10,360 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:39:11,411 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:39:12,183 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:39:12,184 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:39:13,459 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:39:13,461 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:39:15,510 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:39:17,569 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:39:18,491 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:39:19,610 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:39:21,666 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:39:23,713 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:39:23,766 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:39:24,719 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:39:25,762 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:39:27,196 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:39:27,197 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:39:27,810 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:39:29,462 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:39:29,918 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:39:31,935 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:39:33,957 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:39:35,313 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:39:36,006 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:39:36,999 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:39:37,992 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:39:38,051 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:39:40,093 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:39:41,020 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:39:42,126 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:39:42,207 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:39:42,207 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:39:44,166 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:39:46,206 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:39:46,509 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:39:48,245 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:39:48,253 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:39:50,289 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:39:51,902 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:39:52,348 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:39:54,389 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:39:56,435 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:39:57,213 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:39:57,214 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:39:57,471 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:39:58,472 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:40:00,570 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:40:00,617 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:40:02,497 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:40:02,634 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:40:04,662 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:40:06,689 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:40:07,549 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:40:08,008 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:40:08,734 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:40:10,769 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:40:12,236 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:40:12,237 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:40:12,814 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:40:13,504 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:40:14,842 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:40:14,852 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:40:16,904 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:40:18,543 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:40:18,948 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:40:21,001 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:40:23,047 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:40:23,586 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:40:25,086 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:40:27,122 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:40:27,131 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:40:27,252 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:40:27,253 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:40:29,176 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:40:29,523 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:40:31,271 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:40:33,291 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:40:34,563 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:40:35,315 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:40:37,267 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 05:40:37,269 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 05:40:37,270 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 05:40:37,270 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 05:40:37,328 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 05:40:37,340 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:40:38,042 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:40:39,370 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:40:39,382 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:40:40,071 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:40:40,386 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:40:41,432 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:40:42,263 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:40:42,264 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:40:43,479 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:40:45,533 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:40:45,535 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:40:47,574 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:40:49,614 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:40:50,545 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:40:51,661 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:40:52,658 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:40:53,697 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:40:55,596 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:40:55,748 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:40:57,271 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:40:57,272 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:40:57,784 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:40:59,827 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:41:01,582 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:41:01,945 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:41:03,972 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:41:04,936 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:41:05,983 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:41:07,129 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:41:08,025 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:41:08,031 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:41:10,065 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:41:12,120 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:41:12,287 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:41:12,287 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:41:12,542 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:41:14,152 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:41:16,192 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:41:17,190 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:41:17,707 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:41:18,236 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:41:20,276 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:41:22,313 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:41:22,763 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:41:24,355 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:41:26,388 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:41:27,303 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:41:27,303 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:41:28,251 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:41:28,429 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:41:29,426 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:41:30,463 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:41:32,579 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:41:33,307 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:41:34,590 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:41:36,617 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:41:38,043 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:41:38,638 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:41:39,052 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:41:40,669 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:41:42,319 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:41:42,320 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:41:42,721 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:41:43,715 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:41:44,593 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:41:44,774 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:41:46,825 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:41:48,864 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:41:49,652 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:41:50,933 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:41:52,963 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:41:54,993 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:41:55,000 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:41:55,513 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:41:57,035 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:41:57,323 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:41:57,323 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:41:59,071 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:42:00,605 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:42:01,111 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:42:03,212 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:42:05,227 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:42:05,641 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:42:07,239 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:42:07,247 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:42:08,057 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:42:09,281 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:42:11,085 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:42:11,319 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:42:12,334 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:42:12,334 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:42:13,352 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:42:15,393 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:42:16,626 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:42:17,424 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:42:19,463 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:42:21,497 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:42:21,508 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:42:21,713 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:42:23,555 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:42:25,584 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:42:26,738 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:42:27,348 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:42:27,348 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:42:27,631 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:42:29,676 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:42:31,722 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:42:32,353 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:42:33,810 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:42:33,832 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:42:35,841 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:42:37,409 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:42:37,887 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:42:38,069 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:42:39,925 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:42:41,963 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:42:42,360 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:42:42,361 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:42:42,613 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:42:43,932 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 05:42:43,933 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 05:42:43,934 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 05:42:43,934 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 05:42:44,013 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 05:42:44,020 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:42:45,016 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:42:46,060 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:42:47,064 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:42:48,109 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:42:48,241 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:42:50,158 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:42:52,204 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:42:53,289 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:42:54,246 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:42:56,292 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:42:57,376 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:42:57,377 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:42:58,341 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:42:58,644 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:42:59,339 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:43:00,383 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:43:02,432 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:43:03,715 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:43:04,550 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:43:06,564 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:43:08,083 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:43:08,580 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:43:09,087 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:43:10,626 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:43:11,626 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:43:12,393 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:43:12,394 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:43:12,670 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:43:14,708 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:43:14,717 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:43:16,759 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:43:18,795 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:43:19,725 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:43:20,848 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:43:22,895 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:43:23,902 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:43:24,938 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:43:25,523 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:43:26,982 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:43:27,407 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:43:27,408 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:43:29,026 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:43:30,677 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:43:31,074 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:43:33,119 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:43:35,220 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:43:36,075 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:43:37,239 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:43:38,090 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:43:38,230 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:43:39,256 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:43:41,124 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:43:41,281 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:43:42,399 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:43:42,400 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:43:43,332 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:43:45,370 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:43:46,689 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:43:47,412 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:43:49,450 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:43:49,456 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:43:51,498 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:43:51,781 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:43:53,538 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:43:55,581 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:43:56,812 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:43:57,398 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:43:57,399 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:43:57,625 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:43:59,673 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:44:01,695 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:44:01,702 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:44:02,310 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:44:03,748 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:44:05,860 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:44:07,352 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:44:07,880 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:44:08,105 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:44:09,900 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:44:11,951 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:44:12,408 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:44:12,408 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:44:12,659 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:44:14,004 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:44:14,019 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:44:16,047 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:44:17,999 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:44:18,092 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:44:20,125 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:44:22,158 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:44:23,039 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:44:24,204 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:44:26,234 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:44:27,411 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:44:27,412 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:44:28,267 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:44:28,276 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:44:28,678 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:44:30,324 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:44:32,374 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:44:33,711 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:44:34,416 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:44:36,525 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:44:38,110 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:44:38,548 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:44:39,165 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:44:40,543 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:44:40,563 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:44:42,427 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:44:42,428 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:44:42,600 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:44:44,646 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:44:44,714 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:44:46,694 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:44:48,748 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:44:49,758 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:44:50,737 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 05:44:50,739 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 05:44:50,739 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 05:44:50,740 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 05:44:50,795 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 05:44:50,805 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:44:51,798 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:44:52,830 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:44:52,837 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:44:54,891 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:44:55,032 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:44:56,943 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:44:57,445 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:44:57,445 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:44:59,001 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:45:00,712 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:45:01,068 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:45:03,114 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:45:05,176 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:45:06,077 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:45:06,173 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:45:07,286 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:45:08,114 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:45:09,306 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:45:11,135 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:45:11,333 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:45:12,470 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:45:12,471 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:45:13,361 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:45:15,378 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:45:16,696 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:45:17,400 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:45:18,402 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:45:19,443 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:45:21,476 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:45:21,752 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:45:23,518 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:45:25,554 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:45:26,810 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:45:27,481 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:45:27,482 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:45:27,585 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:45:29,625 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:45:30,627 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:45:31,671 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:45:32,372 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:45:33,718 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:45:35,763 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:45:37,415 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:45:37,884 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:45:38,128 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:45:39,913 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:45:41,937 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:45:42,504 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:45:42,504 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:45:42,749 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:45:43,960 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:45:43,970 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:45:46,012 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:45:47,792 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:45:48,055 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:45:50,102 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:45:52,161 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:45:52,839 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:45:54,197 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:45:56,223 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:45:56,233 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:45:57,501 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:45:57,502 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:45:58,272 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:45:58,779 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:46:00,308 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:46:02,355 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:46:03,838 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:46:04,398 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:46:06,445 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:46:08,143 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:46:08,518 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:46:08,551 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:46:09,153 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:46:10,562 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:46:12,519 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:46:12,520 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:46:12,575 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:46:14,613 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:46:14,791 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:46:16,656 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:46:18,695 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:46:20,730 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:46:20,733 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:46:20,788 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:46:22,776 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:46:24,818 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:46:25,825 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:46:26,859 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:46:27,522 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:46:27,522 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:46:28,894 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:46:30,956 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:46:31,794 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:46:32,998 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:46:35,030 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:46:35,045 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:46:37,088 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:46:37,421 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:46:38,150 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:46:39,197 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:46:41,210 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:46:42,523 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:46:42,524 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:46:42,777 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:46:43,228 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:46:45,268 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:46:46,277 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:46:47,307 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:46:48,029 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:46:49,349 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:46:51,388 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:46:53,058 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:46:53,427 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:46:55,468 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:46:57,508 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:46:57,539 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:46:57,540 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:46:57,690 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 05:46:57,801 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 05:46:57,801 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 05:46:57,802 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 05:46:58,512 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:46:58,512 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 05:46:58,951 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:46:59,546 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:46:59,555 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:47:00,554 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:47:01,596 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:47:03,635 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:47:03,998 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:47:05,672 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:47:07,714 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:47:08,151 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:47:09,164 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:47:09,828 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:47:11,843 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:47:12,561 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:47:12,561 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:47:12,814 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:47:13,856 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:47:14,826 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:47:15,894 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:47:17,938 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:47:19,869 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:47:19,979 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:47:22,029 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:47:24,091 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:47:25,085 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:47:25,571 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:47:26,133 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:47:27,576 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:47:27,576 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:47:28,166 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:47:30,208 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:47:30,840 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:47:32,263 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:47:34,319 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:47:35,879 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:47:36,354 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:47:37,362 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:47:38,154 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:47:38,398 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:47:40,511 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:47:41,164 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:47:42,542 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:47:42,589 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:47:42,589 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:47:44,556 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:47:46,579 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:47:46,870 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:47:48,619 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:47:50,666 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:47:50,674 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:47:52,726 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:47:52,871 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:47:54,755 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:47:56,807 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:47:57,597 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:47:57,597 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:47:58,847 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:47:58,857 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:48:00,919 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:48:02,951 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:48:02,959 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:48:04,451 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:48:05,000 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:48:07,036 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:48:08,160 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:48:09,083 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:48:10,184 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:48:11,202 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:48:12,605 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:48:12,606 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:48:13,212 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:48:15,239 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:48:15,248 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:48:15,971 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:48:17,286 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:48:19,323 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:48:21,017 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:48:21,361 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:48:23,415 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:48:25,461 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:48:26,075 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:48:27,487 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:48:27,495 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:48:27,624 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:48:27,625 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:48:29,557 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:48:31,603 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:48:31,889 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:48:33,655 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:48:35,699 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:48:36,938 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:48:37,741 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:48:38,175 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:48:39,792 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:48:40,790 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:48:41,889 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:48:41,989 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:48:42,621 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:48:42,622 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:48:43,901 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:48:45,938 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:48:47,919 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:48:47,985 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:48:50,028 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:48:52,076 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:48:53,068 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:48:53,508 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:48:54,121 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:48:56,169 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:48:57,631 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:48:57,632 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:48:58,211 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:48:58,892 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:49:00,264 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:49:02,321 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:49:03,946 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:49:04,057 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 05:49:04,059 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 05:49:04,059 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 05:49:04,060 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 05:49:04,385 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 05:49:04,395 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:49:05,393 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:49:06,422 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:49:06,432 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:49:08,190 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:49:08,477 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:49:09,198 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:49:10,525 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:49:12,635 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:49:12,640 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:49:12,640 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:49:14,648 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:49:14,891 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:49:16,673 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:49:18,718 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:49:19,718 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:49:20,288 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:49:20,772 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:49:22,821 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:49:24,877 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:49:25,336 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:49:26,927 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:49:27,645 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:49:27,645 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:49:29,010 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:49:30,908 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:49:31,061 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:49:31,076 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:49:33,112 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:49:35,166 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:49:35,942 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:49:37,207 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:49:38,199 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:49:39,248 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:49:41,228 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:49:41,303 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:49:42,646 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:49:42,646 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:49:43,436 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:49:45,418 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:49:45,451 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:49:46,920 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:49:47,468 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:49:49,517 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:49:51,565 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:49:51,965 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:49:53,611 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:49:55,663 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:49:57,045 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:49:57,695 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:49:57,695 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:49:57,698 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:49:57,706 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:49:59,747 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:50:01,794 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:50:03,001 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:50:03,840 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:50:05,897 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:50:07,945 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:50:08,052 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:50:08,212 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:50:09,981 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:50:09,994 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:50:12,041 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:50:12,680 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:50:12,681 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:50:13,939 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:50:14,154 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:50:16,177 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:50:18,199 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:50:18,978 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:50:20,222 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:50:21,208 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:50:22,270 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:50:24,310 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:50:24,739 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:50:26,347 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:50:27,692 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:50:27,693 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:50:28,385 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:50:29,969 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:50:30,435 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:50:32,484 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:50:34,534 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:50:35,249 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:50:35,534 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:50:36,575 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:50:38,225 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:50:38,619 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:50:40,661 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:50:41,255 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:50:42,688 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:50:42,688 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:50:42,696 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:50:44,821 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:50:46,817 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:50:46,842 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:50:47,801 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:50:48,864 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:50:50,885 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:50:51,866 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:50:52,931 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:50:54,968 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:50:56,953 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:50:57,009 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:50:57,691 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:50:57,691 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:50:59,046 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:51:00,052 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:51:01,105 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:51:02,440 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:51:03,147 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:51:05,186 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:51:07,221 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:51:07,498 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:51:08,229 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:51:09,268 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:51:10,966 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 05:51:10,968 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 05:51:10,968 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 05:51:10,969 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 05:51:11,325 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 05:51:11,334 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:51:12,327 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:51:12,701 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:51:12,702 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:51:12,947 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:51:13,380 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:51:15,496 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:51:17,508 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:51:18,006 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:51:19,534 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:51:21,575 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:51:23,057 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:51:23,618 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:51:25,646 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:51:25,654 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:51:27,699 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:51:27,702 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:51:27,703 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:51:28,967 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:51:29,733 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:51:31,797 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:51:33,855 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:51:34,004 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:51:35,933 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:51:37,977 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:51:37,983 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:51:38,243 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:51:39,256 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:51:40,024 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:51:42,061 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:51:42,721 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:51:42,722 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:51:44,103 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:51:44,985 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:51:46,220 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:51:48,244 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:51:50,265 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:51:50,381 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:51:52,292 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:51:52,306 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:51:54,339 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:51:55,427 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:51:56,381 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:51:57,733 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:51:57,733 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:51:58,430 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:52:00,476 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:52:01,014 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:52:02,511 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:52:04,538 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:52:04,547 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:52:06,039 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:52:06,592 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:52:08,246 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:52:08,636 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:52:10,686 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:52:11,273 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:52:12,723 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:52:12,744 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:52:12,745 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:52:14,761 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:52:15,768 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:52:16,617 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:52:16,863 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:52:18,887 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:52:20,900 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:52:21,664 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:52:22,939 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:52:24,988 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:52:26,708 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:52:27,033 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:52:27,744 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:52:27,745 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:52:28,040 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:52:29,077 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:52:31,131 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:52:32,036 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:52:33,179 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:52:35,218 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:52:37,082 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:52:37,266 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:52:38,259 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:52:39,319 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:52:41,382 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:52:42,177 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:52:42,383 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:52:42,758 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:52:42,758 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:52:43,428 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:52:45,486 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:52:47,595 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:52:48,053 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:52:49,605 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:52:51,628 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:52:53,509 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:52:53,665 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:52:54,670 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:52:55,713 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:52:57,755 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:52:57,775 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:52:57,775 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:52:59,041 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:52:59,795 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:53:01,828 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:53:03,876 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:53:04,088 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:53:05,920 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:53:05,921 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:53:07,977 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:53:08,261 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:53:09,269 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:53:10,023 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:53:12,073 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:53:12,793 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:53:12,794 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:53:14,116 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:53:15,041 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:53:16,158 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:53:17,668 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 05:53:17,669 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 05:53:17,670 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 05:53:17,670 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 05:53:18,267 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 05:53:18,294 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:53:19,265 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:53:20,274 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:53:20,306 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:53:20,925 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:53:22,324 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:53:24,369 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:53:25,957 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:53:26,402 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:53:27,816 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:53:27,816 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:53:28,450 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:53:30,490 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:53:31,908 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:53:32,520 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:53:32,528 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:53:34,576 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:53:36,634 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:53:36,958 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:53:38,276 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:53:38,663 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:53:40,714 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:53:42,185 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:53:42,759 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:53:42,813 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:53:42,813 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:53:44,802 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:53:44,814 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:53:46,859 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:53:47,567 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:53:48,969 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:53:50,995 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:53:52,615 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:53:53,019 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:53:55,039 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:53:57,081 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:53:57,807 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:53:57,808 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:53:58,061 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:53:59,114 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:53:59,121 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:54:01,165 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:54:03,095 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:54:03,219 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:54:05,264 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:54:07,298 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:54:08,167 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:54:08,283 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:54:09,339 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:54:10,333 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:54:11,388 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:54:12,818 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:54:12,819 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:54:13,432 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:54:14,079 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:54:15,479 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:54:17,519 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:54:19,106 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:54:19,641 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:54:21,669 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:54:22,624 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:54:23,697 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:54:24,537 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:54:25,715 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:54:27,739 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:54:27,824 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:54:27,825 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:54:29,781 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:54:30,086 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:54:31,821 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:54:33,865 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:54:34,864 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:54:35,097 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:54:35,911 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:54:37,963 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:54:38,295 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:54:39,997 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:54:40,320 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:54:42,057 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:54:42,829 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:54:42,829 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:54:44,098 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:54:46,123 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:54:46,153 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:54:48,205 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:54:49,200 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:54:50,309 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:54:51,707 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:54:52,328 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:54:54,346 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:54:56,391 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:54:56,768 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:54:57,835 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:54:57,836 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:54:58,438 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:55:00,483 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:55:01,479 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:55:02,318 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:55:02,530 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:55:04,576 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:55:06,629 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:55:07,373 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:55:08,305 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:55:08,674 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:55:10,717 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:55:12,751 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:55:12,761 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:55:12,841 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:55:12,842 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:55:13,085 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:55:14,803 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:55:16,840 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:55:18,125 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:55:18,880 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:55:20,985 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:55:23,000 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:55:23,184 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:55:24,607 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 05:55:24,609 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 05:55:24,609 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 05:55:24,610 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 05:55:25,027 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 05:55:25,040 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:55:26,033 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:55:27,093 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:55:27,107 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:55:27,843 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:55:27,843 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:55:29,089 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:55:29,148 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:55:31,192 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:55:33,227 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:55:34,132 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:55:35,277 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:55:37,317 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:55:38,316 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:55:39,322 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:55:39,362 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:55:39,368 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:55:41,415 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:55:42,859 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:55:42,860 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:55:43,470 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:55:45,136 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:55:45,508 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:55:47,554 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:55:49,585 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:55:50,190 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:55:51,658 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:55:51,694 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:55:53,710 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:55:55,297 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:55:55,724 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:55:57,785 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:55:57,869 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:55:57,869 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:55:59,820 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:56:01,139 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:56:01,856 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:56:03,904 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:56:04,902 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:56:05,946 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:56:06,945 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:56:07,996 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:56:08,330 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:56:10,043 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:56:12,095 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:56:12,254 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:56:12,880 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:56:12,880 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:56:14,137 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:56:16,186 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:56:17,183 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:56:17,453 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:56:18,232 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:56:20,261 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:56:22,369 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:56:22,511 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:56:24,382 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:56:26,396 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:56:27,886 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:56:27,886 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:56:28,136 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:56:28,423 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:56:29,419 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:56:30,460 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:56:32,499 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:56:33,190 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:56:34,538 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:56:36,568 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:56:38,229 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:56:38,340 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:56:38,617 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:56:40,673 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:56:41,667 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:56:42,712 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:56:42,894 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:56:42,894 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:56:44,139 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:56:44,766 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:56:46,799 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:56:48,844 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:56:49,172 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:56:50,898 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:56:53,017 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:56:54,210 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:56:55,027 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:56:56,006 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:56:57,054 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:56:57,893 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:56:57,893 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:56:59,109 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:57:00,157 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:57:01,152 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:57:03,196 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:57:05,236 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:57:05,247 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:57:07,282 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:57:07,291 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:57:08,355 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:57:09,326 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:57:10,371 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:57:11,366 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:57:12,904 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:57:12,905 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:57:13,407 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:57:15,450 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:57:16,182 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:57:17,494 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:57:19,538 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:57:19,539 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:57:21,326 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:57:21,584 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:57:23,701 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:57:25,721 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:57:26,371 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:57:27,740 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:57:27,921 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:57:27,922 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:57:29,764 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:57:30,922 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 05:57:30,924 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 05:57:30,924 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 05:57:30,925 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 05:57:31,804 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:57:31,804 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 05:57:31,813 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:57:32,218 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:57:33,846 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:57:33,855 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:57:35,909 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:57:37,262 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:57:37,958 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:57:38,369 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:57:40,009 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:57:42,056 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:57:42,312 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:57:42,929 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:57:42,930 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:57:44,099 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:57:46,137 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:57:46,145 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:57:48,211 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:57:48,222 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:57:50,282 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:57:52,339 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:57:53,252 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:57:54,446 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:57:56,457 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:57:57,424 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:57:57,942 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:57:57,942 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:57:58,494 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:57:59,207 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:58:00,560 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:58:02,602 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:58:04,264 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:58:04,642 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:58:06,681 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:58:08,375 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:58:08,725 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:58:09,721 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:58:10,224 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:58:10,762 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:58:12,800 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:58:12,931 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:58:12,931 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:58:14,848 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:58:16,203 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:58:16,893 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:58:18,938 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:58:20,993 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:58:21,245 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:58:23,046 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:58:24,041 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:58:25,168 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:58:26,897 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:58:27,187 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:58:27,938 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:58:27,938 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:58:29,211 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:58:31,242 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:58:32,241 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:58:33,283 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:58:35,343 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:58:36,338 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:58:37,390 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:58:37,459 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:58:38,377 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:58:39,433 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:58:41,483 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:58:42,934 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:58:42,935 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:58:43,190 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:58:43,541 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:58:45,573 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:58:47,615 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:58:47,623 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:58:49,022 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:58:49,664 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:58:51,712 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:58:53,750 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:58:54,068 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:58:55,861 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:58:57,879 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:58:57,950 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:58:57,950 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:58:59,198 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:58:59,904 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:59:01,906 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:59:01,927 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:59:03,954 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:59:04,620 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:59:05,989 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:59:08,044 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:59:08,387 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:59:10,093 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:59:10,411 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:59:12,147 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:59:12,950 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:59:12,950 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:59:14,185 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:59:14,194 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:59:16,233 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:59:16,236 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:59:18,298 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:59:20,330 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:59:21,271 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:59:22,375 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:59:24,422 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:59:26,493 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:59:26,530 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:59:26,879 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:59:27,952 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: stop_status 2023-04-23 05:59:27,953 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: stop_status 2023-04-23 05:59:28,558 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:59:30,593 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:59:32,240 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:59:32,635 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:59:34,679 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:59:36,722 ERROR gpu :34348 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2023-04-23 05:59:37,290 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:59:37,547 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 05:59:37,548 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 05:59:37,548 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 05:59:37,549 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 05:59:37,717 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 05:59:38,156 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: partial_history 2023-04-23 05:59:38,158 DEBUG SenderThread:34348 [sender.py:send():375] send: metric 2023-04-23 05:59:38,159 DEBUG SenderThread:34348 [sender.py:send():375] send: metric 2023-04-23 05:59:38,159 DEBUG SenderThread:34348 [sender.py:send():375] send: metric 2023-04-23 05:59:38,159 DEBUG SenderThread:34348 [sender.py:send():375] send: metric 2023-04-23 05:59:38,159 DEBUG SenderThread:34348 [sender.py:send():375] send: metric 2023-04-23 05:59:38,159 DEBUG SenderThread:34348 [sender.py:send():375] send: history 2023-04-23 05:59:38,159 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: summary_record 2023-04-23 05:59:38,160 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 05:59:38,188 DEBUG SenderThread:34348 [sender.py:send():375] send: exit 2023-04-23 05:59:38,188 INFO SenderThread:34348 [sender.py:send_exit():598] handling exit code: 0 2023-04-23 05:59:38,188 INFO SenderThread:34348 [sender.py:send_exit():600] handling runtime: 12873 2023-04-23 05:59:38,188 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 05:59:38,189 INFO SenderThread:34348 [sender.py:send_exit():606] send defer 2023-04-23 05:59:38,191 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: defer 2023-04-23 05:59:38,191 INFO HandlerThread:34348 [handler.py:handle_request_defer():170] handle defer: 0 2023-04-23 05:59:38,191 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: defer 2023-04-23 05:59:38,191 INFO SenderThread:34348 [sender.py:send_request_defer():622] handle sender defer: 0 2023-04-23 05:59:38,191 INFO SenderThread:34348 [sender.py:transition_state():626] send defer: 1 2023-04-23 05:59:38,192 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: defer 2023-04-23 05:59:38,192 INFO HandlerThread:34348 [handler.py:handle_request_defer():170] handle defer: 1 2023-04-23 05:59:38,192 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: defer 2023-04-23 05:59:38,192 INFO SenderThread:34348 [sender.py:send_request_defer():622] handle sender defer: 1 2023-04-23 05:59:38,192 INFO SenderThread:34348 [sender.py:transition_state():626] send defer: 2 2023-04-23 05:59:38,192 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: defer 2023-04-23 05:59:38,192 INFO HandlerThread:34348 [handler.py:handle_request_defer():170] handle defer: 2 2023-04-23 05:59:38,193 INFO HandlerThread:34348 [system_monitor.py:finish():190] Stopping system monitor 2023-04-23 05:59:38,193 DEBUG SystemMonitor:34348 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-04-23 05:59:38,193 DEBUG SystemMonitor:34348 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-04-23 05:59:38,194 INFO HandlerThread:34348 [interfaces.py:finish():202] Joined cpu monitor 2023-04-23 05:59:38,194 INFO HandlerThread:34348 [interfaces.py:finish():202] Joined disk monitor 2023-04-23 05:59:38,256 INFO HandlerThread:34348 [interfaces.py:finish():202] Joined gpu monitor 2023-04-23 05:59:38,257 INFO HandlerThread:34348 [interfaces.py:finish():202] Joined memory monitor 2023-04-23 05:59:38,257 INFO HandlerThread:34348 [interfaces.py:finish():202] Joined network monitor 2023-04-23 05:59:38,257 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: defer 2023-04-23 05:59:38,257 INFO SenderThread:34348 [sender.py:send_request_defer():622] handle sender defer: 2 2023-04-23 05:59:38,257 INFO SenderThread:34348 [sender.py:transition_state():626] send defer: 3 2023-04-23 05:59:38,258 DEBUG SenderThread:34348 [sender.py:send():375] send: stats 2023-04-23 05:59:38,258 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: defer 2023-04-23 05:59:38,258 INFO HandlerThread:34348 [handler.py:handle_request_defer():170] handle defer: 3 2023-04-23 05:59:38,258 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: defer 2023-04-23 05:59:38,258 INFO SenderThread:34348 [sender.py:send_request_defer():622] handle sender defer: 3 2023-04-23 05:59:38,259 INFO SenderThread:34348 [sender.py:transition_state():626] send defer: 4 2023-04-23 05:59:38,259 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: defer 2023-04-23 05:59:38,259 INFO HandlerThread:34348 [handler.py:handle_request_defer():170] handle defer: 4 2023-04-23 05:59:38,259 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: defer 2023-04-23 05:59:38,259 INFO SenderThread:34348 [sender.py:send_request_defer():622] handle sender defer: 4 2023-04-23 05:59:38,259 INFO SenderThread:34348 [sender.py:transition_state():626] send defer: 5 2023-04-23 05:59:38,259 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: defer 2023-04-23 05:59:38,259 INFO HandlerThread:34348 [handler.py:handle_request_defer():170] handle defer: 5 2023-04-23 05:59:38,260 DEBUG SenderThread:34348 [sender.py:send():375] send: summary 2023-04-23 05:59:38,260 INFO SenderThread:34348 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end 2023-04-23 05:59:38,261 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: defer 2023-04-23 05:59:38,261 INFO SenderThread:34348 [sender.py:send_request_defer():622] handle sender defer: 5 2023-04-23 05:59:38,261 INFO SenderThread:34348 [sender.py:transition_state():626] send defer: 6 2023-04-23 05:59:38,261 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: defer 2023-04-23 05:59:38,261 INFO HandlerThread:34348 [handler.py:handle_request_defer():170] handle defer: 6 2023-04-23 05:59:38,262 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: defer 2023-04-23 05:59:38,262 INFO SenderThread:34348 [sender.py:send_request_defer():622] handle sender defer: 6 2023-04-23 05:59:38,265 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:59:38,727 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:59:38,727 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 05:59:38,907 INFO SenderThread:34348 [sender.py:transition_state():626] send defer: 7 2023-04-23 05:59:38,907 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: defer 2023-04-23 05:59:38,907 INFO HandlerThread:34348 [handler.py:handle_request_defer():170] handle defer: 7 2023-04-23 05:59:38,907 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: defer 2023-04-23 05:59:38,908 INFO SenderThread:34348 [sender.py:send_request_defer():622] handle sender defer: 7 2023-04-23 05:59:39,273 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: poll_exit 2023-04-23 05:59:39,730 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\config.yaml 2023-04-23 05:59:40,300 INFO SenderThread:34348 [sender.py:transition_state():626] send defer: 8 2023-04-23 05:59:40,301 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: poll_exit 2023-04-23 05:59:40,301 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: defer 2023-04-23 05:59:40,301 INFO HandlerThread:34348 [handler.py:handle_request_defer():170] handle defer: 8 2023-04-23 05:59:40,301 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: defer 2023-04-23 05:59:40,301 INFO SenderThread:34348 [sender.py:send_request_defer():622] handle sender defer: 8 2023-04-23 05:59:40,318 INFO SenderThread:34348 [sender.py:transition_state():626] send defer: 9 2023-04-23 05:59:40,319 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: defer 2023-04-23 05:59:40,319 DEBUG SenderThread:34348 [sender.py:send():375] send: artifact 2023-04-23 05:59:40,319 INFO HandlerThread:34348 [handler.py:handle_request_defer():170] handle defer: 9 2023-04-23 05:59:40,739 INFO Thread-16 :34348 [dir_watcher.py:_on_file_modified():295] file/dir modified: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:59:41,812 INFO wandb-upload_0:34348 [upload_job.py:push():92] Skipped uploading C:\Users\Lenovo\AppData\Local\wandb\wandb\artifacts\staging\tmp62h9wd9b 2023-04-23 05:59:42,657 INFO wandb-upload_1:34348 [upload_job.py:push():95] Uploaded file C:\Users\Lenovo\AppData\Local\wandb\wandb\artifacts\staging\tmpvi_xae6k 2023-04-23 05:59:44,317 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: keepalive 2023-04-23 05:59:44,727 INFO SenderThread:34348 [sender.py:send_artifact():1474] sent artifact job-https___github.com_THUDM_ChatGLM-6B_ptuning_main.py - {'id': 'QXJ0aWZhY3Q6NDMxMDA2NTAw', 'digest': 'd16240778017459800c83cd904d36ea4', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjYzNDcyMzc3', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6NDI5OTE5NTc5', 'versionIndex': 5}}, 'version': 'latest'} 2023-04-23 05:59:44,727 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: defer 2023-04-23 05:59:44,727 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: status_report 2023-04-23 05:59:44,727 INFO SenderThread:34348 [sender.py:send_request_defer():622] handle sender defer: 9 2023-04-23 05:59:44,727 INFO SenderThread:34348 [dir_watcher.py:finish():365] shutting down directory watcher 2023-04-23 05:59:44,771 INFO SenderThread:34348 [dir_watcher.py:finish():395] scan: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files 2023-04-23 05:59:44,771 INFO SenderThread:34348 [dir_watcher.py:finish():409] scan save: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\config.yaml config.yaml 2023-04-23 05:59:44,772 INFO SenderThread:34348 [dir_watcher.py:finish():409] scan save: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log output.log 2023-04-23 05:59:44,774 INFO SenderThread:34348 [dir_watcher.py:finish():409] scan save: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\requirements.txt requirements.txt 2023-04-23 05:59:44,777 INFO SenderThread:34348 [dir_watcher.py:finish():409] scan save: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-metadata.json wandb-metadata.json 2023-04-23 05:59:44,777 INFO SenderThread:34348 [dir_watcher.py:finish():409] scan save: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json wandb-summary.json 2023-04-23 05:59:44,780 INFO SenderThread:34348 [sender.py:transition_state():626] send defer: 10 2023-04-23 05:59:44,781 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: defer 2023-04-23 05:59:44,781 INFO HandlerThread:34348 [handler.py:handle_request_defer():170] handle defer: 10 2023-04-23 05:59:44,782 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: defer 2023-04-23 05:59:44,783 INFO SenderThread:34348 [sender.py:send_request_defer():622] handle sender defer: 10 2023-04-23 05:59:44,783 INFO SenderThread:34348 [file_pusher.py:finish():168] shutting down file pusher 2023-04-23 05:59:45,501 INFO wandb-upload_0:34348 [upload_job.py:push():137] Uploaded file E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\config.yaml 2023-04-23 05:59:46,044 INFO wandb-upload_3:34348 [upload_job.py:push():137] Uploaded file E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\wandb-summary.json 2023-04-23 05:59:46,136 INFO wandb-upload_2:34348 [upload_job.py:push():137] Uploaded file E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\requirements.txt 2023-04-23 05:59:46,313 INFO wandb-upload_1:34348 [upload_job.py:push():137] Uploaded file E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\files\output.log 2023-04-23 05:59:46,519 INFO Thread-15 :34348 [sender.py:transition_state():626] send defer: 11 2023-04-23 05:59:46,519 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: defer 2023-04-23 05:59:46,519 INFO HandlerThread:34348 [handler.py:handle_request_defer():170] handle defer: 11 2023-04-23 05:59:46,520 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: defer 2023-04-23 05:59:46,520 INFO SenderThread:34348 [sender.py:send_request_defer():622] handle sender defer: 11 2023-04-23 05:59:46,520 INFO SenderThread:34348 [file_pusher.py:join():173] waiting for file pusher 2023-04-23 05:59:46,520 INFO SenderThread:34348 [sender.py:transition_state():626] send defer: 12 2023-04-23 05:59:46,520 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: defer 2023-04-23 05:59:46,520 INFO HandlerThread:34348 [handler.py:handle_request_defer():170] handle defer: 12 2023-04-23 05:59:46,520 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: defer 2023-04-23 05:59:46,521 INFO SenderThread:34348 [sender.py:send_request_defer():622] handle sender defer: 12 2023-04-23 05:59:47,695 INFO SenderThread:34348 [sender.py:transition_state():626] send defer: 13 2023-04-23 05:59:47,697 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: defer 2023-04-23 05:59:47,697 INFO HandlerThread:34348 [handler.py:handle_request_defer():170] handle defer: 13 2023-04-23 05:59:47,697 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: defer 2023-04-23 05:59:47,697 INFO SenderThread:34348 [sender.py:send_request_defer():622] handle sender defer: 13 2023-04-23 05:59:47,697 INFO SenderThread:34348 [sender.py:transition_state():626] send defer: 14 2023-04-23 05:59:47,698 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: defer 2023-04-23 05:59:47,698 INFO HandlerThread:34348 [handler.py:handle_request_defer():170] handle defer: 14 2023-04-23 05:59:47,698 DEBUG SenderThread:34348 [sender.py:send():375] send: final 2023-04-23 05:59:47,698 DEBUG SenderThread:34348 [sender.py:send():375] send: footer 2023-04-23 05:59:47,698 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: defer 2023-04-23 05:59:47,698 INFO SenderThread:34348 [sender.py:send_request_defer():622] handle sender defer: 14 2023-04-23 05:59:47,699 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: poll_exit 2023-04-23 05:59:47,699 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: server_info 2023-04-23 05:59:47,699 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: poll_exit 2023-04-23 05:59:47,699 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: get_summary 2023-04-23 05:59:47,699 DEBUG SenderThread:34348 [sender.py:send_request():402] send_request: server_info 2023-04-23 05:59:47,700 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: sampled_history 2023-04-23 05:59:47,945 INFO MainThread:34348 [wandb_run.py:_footer_history_summary_info():3476] rendering history 2023-04-23 05:59:47,959 INFO MainThread:34348 [wandb_run.py:_footer_history_summary_info():3508] rendering summary 2023-04-23 05:59:47,964 INFO MainThread:34348 [wandb_run.py:_footer_sync_info():3434] logging synced files 2023-04-23 05:59:47,966 DEBUG HandlerThread:34348 [handler.py:handle_request():144] handle_request: shutdown 2023-04-23 05:59:47,966 INFO HandlerThread:34348 [handler.py:finish():845] shutting down handler 2023-04-23 05:59:48,713 INFO WriterThread:34348 [datastore.py:close():298] close: E:\Documents\Desktop\ChatGLM-6B\ptuning\wandb\run-20230423_022503-g3y4djvd\run-g3y4djvd.wandb 2023-04-23 05:59:48,950 INFO SenderThread:34348 [sender.py:finish():1550] shutting down sender 2023-04-23 05:59:48,950 INFO SenderThread:34348 [file_pusher.py:finish():168] shutting down file pusher 2023-04-23 05:59:48,950 INFO SenderThread:34348 [file_pusher.py:join():173] waiting for file pusher