2024-06-03 17:54:49,038 INFO StreamThr :322 [internal.py:wandb_internal():85] W&B internal server running at pid: 322, started at: 2024-06-03 17:54:49.038075 2024-06-03 17:54:49,040 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status 2024-06-03 17:54:49,383 INFO WriterThread:322 [datastore.py:open_for_write():87] open: /kaggle/working/wandb/run-20240603_175449-d191dh7n/run-d191dh7n.wandb 2024-06-03 17:54:49,383 DEBUG SenderThread:322 [sender.py:send():378] send: header 2024-06-03 17:54:49,387 DEBUG SenderThread:322 [sender.py:send():378] send: run 2024-06-03 17:54:49,680 INFO SenderThread:322 [dir_watcher.py:__init__():211] watching files in: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files 2024-06-03 17:54:49,680 INFO SenderThread:322 [sender.py:_start_run_threads():1123] run started: d191dh7n with start time 1717437289.03907 2024-06-03 17:54:49,689 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: check_version 2024-06-03 17:54:49,689 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: check_version 2024-06-03 17:54:49,778 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: run_start 2024-06-03 17:54:49,789 DEBUG HandlerThread:322 [system_info.py:__init__():26] System info init 2024-06-03 17:54:49,789 DEBUG HandlerThread:322 [system_info.py:__init__():41] System info init done 2024-06-03 17:54:49,790 INFO HandlerThread:322 [system_monitor.py:start():194] Starting system monitor 2024-06-03 17:54:49,790 INFO SystemMonitor:322 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-06-03 17:54:49,790 INFO HandlerThread:322 [system_monitor.py:probe():214] Collecting system info 2024-06-03 17:54:49,791 INFO SystemMonitor:322 [interfaces.py:start():188] Started cpu monitoring 2024-06-03 17:54:49,791 INFO SystemMonitor:322 [interfaces.py:start():188] Started disk monitoring 2024-06-03 17:54:49,792 INFO SystemMonitor:322 [interfaces.py:start():188] Started gpu monitoring 2024-06-03 17:54:49,792 INFO SystemMonitor:322 [interfaces.py:start():188] Started memory monitoring 2024-06-03 17:54:49,793 INFO SystemMonitor:322 [interfaces.py:start():188] Started network monitoring 2024-06-03 17:54:49,804 DEBUG HandlerThread:322 [system_info.py:probe():150] Probing system 2024-06-03 17:54:49,806 DEBUG HandlerThread:322 [gitlib.py:_init_repo():56] git repository is invalid 2024-06-03 17:54:49,806 DEBUG HandlerThread:322 [system_info.py:probe():198] Probing system done 2024-06-03 17:54:49,806 DEBUG HandlerThread:322 [system_monitor.py:probe():223] {'os': 'Linux-5.15.133+-x86_64-with-glibc2.31', 'python': '3.10.13', 'heartbeatAt': '2024-06-03T17:54:49.804208', 'startedAt': '2024-06-03T17:54:49.031804', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': 'kaggle.ipynb', 'codePathLocal': None, 'root': '/kaggle/working', 'host': 'f28ebe0d2526', 'username': 'root', 'executable': '/opt/conda/bin/python3.10', 'cpu_count': 2, 'cpu_count_logical': 4, 'cpu_freq': {'current': 2000.194, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2000.194, 'min': 0.0, 'max': 0.0}, {'current': 2000.194, 'min': 0.0, 'max': 0.0}, {'current': 2000.194, 'min': 0.0, 'max': 0.0}, {'current': 2000.194, 'min': 0.0, 'max': 0.0}], 'disk': {'/': {'total': 8062.387607574463, 'used': 5657.45686340332}}, 'gpu': 'Tesla T4', 'gpu_count': 2, 'gpu_devices': [{'name': 'Tesla T4', 'memory_total': 16106127360}, {'name': 'Tesla T4', 'memory_total': 16106127360}], 'memory': {'total': 31.357563018798828}} 2024-06-03 17:54:49,806 INFO HandlerThread:322 [system_monitor.py:probe():224] Finished collecting system info 2024-06-03 17:54:49,806 INFO HandlerThread:322 [system_monitor.py:probe():227] Publishing system info 2024-06-03 17:54:49,806 DEBUG HandlerThread:322 [system_info.py:_save_conda():207] Saving list of conda packages installed into the current environment 2024-06-03 17:54:50,682 INFO Thread-12 :322 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/conda-environment.yaml 2024-06-03 17:55:04,821 ERROR HandlerThread:322 [system_info.py:_save_conda():221] Error saving conda packages: Command '['conda', 'env', 'export']' timed out after 15 seconds Traceback (most recent call last): File "/opt/conda/lib/python3.10/site-packages/wandb/sdk/internal/system/system_info.py", line 214, in _save_conda subprocess.call( File "/opt/conda/lib/python3.10/subprocess.py", line 347, in call return p.wait(timeout=timeout) File "/opt/conda/lib/python3.10/subprocess.py", line 1209, in wait return self._wait(timeout=timeout) File "/opt/conda/lib/python3.10/subprocess.py", line 1951, in _wait raise TimeoutExpired(self.args, timeout) subprocess.TimeoutExpired: Command '['conda', 'env', 'export']' timed out after 15 seconds 2024-06-03 17:55:04,824 DEBUG HandlerThread:322 [system_info.py:_save_conda():222] Saving conda packages done 2024-06-03 17:55:04,824 INFO HandlerThread:322 [system_monitor.py:probe():229] Finished publishing system info 2024-06-03 17:55:04,832 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 17:55:04,833 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 17:55:04,833 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 17:55:04,833 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 17:55:04,833 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 17:55:04,833 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 17:55:04,833 DEBUG SenderThread:322 [sender.py:send():378] send: files 2024-06-03 17:55:04,834 INFO SenderThread:322 [sender.py:_save_file():1389] saving file wandb-metadata.json with policy now 2024-06-03 17:55:05,037 INFO wandb-upload_0:322 [upload_job.py:push():130] Uploaded file /tmp/tmp9gua4py3wandb/q833oiff-wandb-metadata.json 2024-06-03 17:55:05,685 INFO Thread-12 :322 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/wandb-metadata.json 2024-06-03 17:55:06,076 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: python_packages 2024-06-03 17:55:06,077 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: python_packages 2024-06-03 17:55:06,079 DEBUG SenderThread:322 [sender.py:send():378] send: telemetry 2024-06-03 17:55:06,080 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 17:55:06,080 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 17:55:06,204 DEBUG SenderThread:322 [sender.py:send():378] send: config 2024-06-03 17:55:06,205 DEBUG SenderThread:322 [sender.py:send():378] send: metric 2024-06-03 17:55:06,206 DEBUG SenderThread:322 [sender.py:send():378] send: telemetry 2024-06-03 17:55:06,206 DEBUG SenderThread:322 [sender.py:send():378] send: telemetry 2024-06-03 17:55:06,206 DEBUG SenderThread:322 [sender.py:send():378] send: metric 2024-06-03 17:55:06,206 WARNING SenderThread:322 [sender.py:send_metric():1340] Seen metric with glob (shouldn't happen) 2024-06-03 17:55:06,206 DEBUG SenderThread:322 [sender.py:send():378] send: telemetry 2024-06-03 17:55:06,686 INFO Thread-12 :322 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/requirements.txt 2024-06-03 17:55:07,353 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: pause 2024-06-03 17:55:07,354 INFO HandlerThread:322 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-06-03 17:55:07,354 INFO HandlerThread:322 [system_monitor.py:finish():203] Stopping system monitor 2024-06-03 17:55:07,354 DEBUG SystemMonitor:322 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-06-03 17:55:07,354 DEBUG SystemMonitor:322 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-06-03 17:55:07,354 INFO HandlerThread:322 [interfaces.py:finish():200] Joined cpu monitor 2024-06-03 17:55:07,355 INFO HandlerThread:322 [interfaces.py:finish():200] Joined disk monitor 2024-06-03 17:55:07,355 DEBUG SystemMonitor:322 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-06-03 17:55:07,368 INFO HandlerThread:322 [interfaces.py:finish():200] Joined gpu monitor 2024-06-03 17:55:07,368 INFO HandlerThread:322 [interfaces.py:finish():200] Joined memory monitor 2024-06-03 17:55:07,368 INFO HandlerThread:322 [interfaces.py:finish():200] Joined network monitor 2024-06-03 17:55:07,369 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 17:55:10,369 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 17:55:15,371 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 17:55:20,377 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 17:55:20,690 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/config.yaml 2024-06-03 17:55:21,077 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 17:55:21,078 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 17:55:26,188 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 17:55:31,190 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 17:55:36,077 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 17:55:36,078 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 17:55:37,179 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 17:55:42,181 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 17:55:47,181 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 17:55:51,077 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 17:55:51,078 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 17:55:52,184 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 17:55:57,185 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 17:56:02,186 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 17:56:06,078 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 17:56:06,078 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 17:56:07,202 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 17:56:12,204 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 17:56:17,204 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 17:56:21,078 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 17:56:21,078 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 17:56:23,202 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 17:56:28,203 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 17:56:33,204 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 17:56:36,078 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 17:56:36,079 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 17:56:39,151 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 17:56:44,152 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 17:56:49,153 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 17:56:51,078 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 17:56:51,079 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 17:56:55,152 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 17:56:56,275 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: resume 2024-06-03 17:56:56,276 INFO HandlerThread:322 [handler.py:handle_request_resume():715] starting system metrics thread 2024-06-03 17:56:56,276 INFO HandlerThread:322 [system_monitor.py:start():194] Starting system monitor 2024-06-03 17:56:56,276 INFO SystemMonitor:322 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-06-03 17:56:56,277 INFO SystemMonitor:322 [interfaces.py:start():188] Started cpu monitoring 2024-06-03 17:56:56,277 INFO SystemMonitor:322 [interfaces.py:start():188] Started disk monitoring 2024-06-03 17:56:56,278 INFO SystemMonitor:322 [interfaces.py:start():188] Started gpu monitoring 2024-06-03 17:56:56,279 INFO SystemMonitor:322 [interfaces.py:start():188] Started memory monitoring 2024-06-03 17:56:56,280 INFO SystemMonitor:322 [interfaces.py:start():188] Started network monitoring 2024-06-03 17:56:56,291 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: pause 2024-06-03 17:56:56,291 INFO HandlerThread:322 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-06-03 17:56:56,291 INFO HandlerThread:322 [system_monitor.py:finish():203] Stopping system monitor 2024-06-03 17:56:56,291 DEBUG SystemMonitor:322 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-06-03 17:56:56,292 INFO HandlerThread:322 [interfaces.py:finish():200] Joined cpu monitor 2024-06-03 17:56:56,292 DEBUG SystemMonitor:322 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-06-03 17:56:56,292 INFO HandlerThread:322 [interfaces.py:finish():200] Joined disk monitor 2024-06-03 17:56:56,292 DEBUG SystemMonitor:322 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-06-03 17:56:56,304 INFO HandlerThread:322 [interfaces.py:finish():200] Joined gpu monitor 2024-06-03 17:56:56,304 INFO HandlerThread:322 [interfaces.py:finish():200] Joined memory monitor 2024-06-03 17:56:56,305 INFO HandlerThread:322 [interfaces.py:finish():200] Joined network monitor 2024-06-03 17:56:56,305 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 17:56:59,515 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: resume 2024-06-03 17:56:59,515 INFO HandlerThread:322 [handler.py:handle_request_resume():715] starting system metrics thread 2024-06-03 17:56:59,515 INFO HandlerThread:322 [system_monitor.py:start():194] Starting system monitor 2024-06-03 17:56:59,516 INFO SystemMonitor:322 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-06-03 17:56:59,516 INFO SystemMonitor:322 [interfaces.py:start():188] Started cpu monitoring 2024-06-03 17:56:59,517 INFO SystemMonitor:322 [interfaces.py:start():188] Started disk monitoring 2024-06-03 17:56:59,518 INFO SystemMonitor:322 [interfaces.py:start():188] Started gpu monitoring 2024-06-03 17:56:59,519 INFO SystemMonitor:322 [interfaces.py:start():188] Started memory monitoring 2024-06-03 17:56:59,532 INFO SystemMonitor:322 [interfaces.py:start():188] Started network monitoring 2024-06-03 17:56:59,596 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: pause 2024-06-03 17:56:59,596 INFO HandlerThread:322 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-06-03 17:56:59,596 INFO HandlerThread:322 [system_monitor.py:finish():203] Stopping system monitor 2024-06-03 17:56:59,596 DEBUG SystemMonitor:322 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-06-03 17:56:59,596 DEBUG SystemMonitor:322 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-06-03 17:56:59,596 DEBUG SystemMonitor:322 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-06-03 17:56:59,597 INFO HandlerThread:322 [interfaces.py:finish():200] Joined cpu monitor 2024-06-03 17:56:59,598 INFO HandlerThread:322 [interfaces.py:finish():200] Joined disk monitor 2024-06-03 17:56:59,609 INFO HandlerThread:322 [interfaces.py:finish():200] Joined gpu monitor 2024-06-03 17:56:59,609 INFO HandlerThread:322 [interfaces.py:finish():200] Joined memory monitor 2024-06-03 17:56:59,609 INFO HandlerThread:322 [interfaces.py:finish():200] Joined network monitor 2024-06-03 17:56:59,610 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 17:57:00,611 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 17:57:05,612 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 17:57:06,079 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 17:57:06,080 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 17:57:06,215 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: resume 2024-06-03 17:57:06,215 INFO HandlerThread:322 [handler.py:handle_request_resume():715] starting system metrics thread 2024-06-03 17:57:06,215 INFO HandlerThread:322 [system_monitor.py:start():194] Starting system monitor 2024-06-03 17:57:06,215 INFO SystemMonitor:322 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-06-03 17:57:06,216 INFO SystemMonitor:322 [interfaces.py:start():188] Started cpu monitoring 2024-06-03 17:57:06,217 INFO SystemMonitor:322 [interfaces.py:start():188] Started disk monitoring 2024-06-03 17:57:06,217 INFO SystemMonitor:322 [interfaces.py:start():188] Started gpu monitoring 2024-06-03 17:57:06,218 INFO SystemMonitor:322 [interfaces.py:start():188] Started memory monitoring 2024-06-03 17:57:06,219 INFO SystemMonitor:322 [interfaces.py:start():188] Started network monitoring 2024-06-03 17:57:06,262 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: pause 2024-06-03 17:57:06,262 INFO HandlerThread:322 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-06-03 17:57:06,262 INFO HandlerThread:322 [system_monitor.py:finish():203] Stopping system monitor 2024-06-03 17:57:06,263 INFO HandlerThread:322 [interfaces.py:finish():200] Joined cpu monitor 2024-06-03 17:57:06,263 DEBUG SystemMonitor:322 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-06-03 17:57:06,263 DEBUG SystemMonitor:322 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-06-03 17:57:06,263 DEBUG SystemMonitor:322 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-06-03 17:57:06,264 INFO HandlerThread:322 [interfaces.py:finish():200] Joined disk monitor 2024-06-03 17:57:06,275 INFO HandlerThread:322 [interfaces.py:finish():200] Joined gpu monitor 2024-06-03 17:57:06,275 INFO HandlerThread:322 [interfaces.py:finish():200] Joined memory monitor 2024-06-03 17:57:06,275 INFO HandlerThread:322 [interfaces.py:finish():200] Joined network monitor 2024-06-03 17:57:06,275 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 17:57:11,276 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 17:57:16,278 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 17:57:21,078 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 17:57:21,079 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 17:57:22,174 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 17:57:27,176 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 17:57:32,177 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 17:57:36,079 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 17:57:36,079 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 17:57:38,123 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 17:57:43,124 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 17:57:48,125 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 17:57:51,079 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 17:57:51,079 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 17:57:53,198 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 17:57:58,199 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 17:58:03,201 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 17:58:06,079 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 17:58:06,080 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 17:58:08,966 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 17:58:13,967 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 17:58:18,968 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 17:58:21,079 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 17:58:21,080 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 17:58:24,170 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 17:58:29,171 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 17:58:34,172 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 17:58:36,079 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 17:58:36,080 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 17:58:39,199 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 17:58:44,200 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 17:58:49,202 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 17:58:51,080 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 17:58:51,080 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 17:58:54,214 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 17:58:59,215 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 17:59:04,216 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 17:59:06,080 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 17:59:06,080 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 17:59:10,146 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 17:59:15,147 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 17:59:20,148 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 17:59:21,080 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 17:59:21,081 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 17:59:26,117 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 17:59:31,118 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 17:59:36,080 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 17:59:36,081 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 17:59:36,177 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 17:59:41,178 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 17:59:46,179 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 17:59:51,080 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 17:59:51,081 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 17:59:52,127 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 17:59:57,128 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:00:02,129 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:00:06,081 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:00:06,081 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:00:07,151 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:00:12,152 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:00:17,153 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:00:21,081 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:00:21,081 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:00:22,207 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:00:27,209 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:00:32,210 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:00:36,081 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:00:36,082 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:00:38,139 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:00:43,140 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:00:48,141 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:00:51,081 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:00:51,082 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:00:53,175 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:00:58,176 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:01:03,178 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:01:06,081 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:01:06,082 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:01:08,181 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:01:13,182 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:01:18,182 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:01:21,081 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:01:21,082 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:01:23,211 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:01:28,212 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:01:33,213 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:01:36,081 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:01:36,082 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:01:39,202 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:01:44,203 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:01:49,204 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:01:51,082 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:01:51,082 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:01:54,219 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:01:57,365 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: resume 2024-06-03 18:01:57,365 INFO HandlerThread:322 [handler.py:handle_request_resume():715] starting system metrics thread 2024-06-03 18:01:57,365 INFO HandlerThread:322 [system_monitor.py:start():194] Starting system monitor 2024-06-03 18:01:57,365 INFO SystemMonitor:322 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-06-03 18:01:57,366 INFO SystemMonitor:322 [interfaces.py:start():188] Started cpu monitoring 2024-06-03 18:01:57,366 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: pause 2024-06-03 18:01:57,366 INFO HandlerThread:322 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-06-03 18:01:57,367 INFO HandlerThread:322 [system_monitor.py:finish():203] Stopping system monitor 2024-06-03 18:01:57,370 INFO HandlerThread:322 [interfaces.py:finish():200] Joined cpu monitor 2024-06-03 18:01:57,371 INFO SystemMonitor:322 [interfaces.py:start():188] Started disk monitoring 2024-06-03 18:01:57,372 DEBUG SystemMonitor:322 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-06-03 18:01:57,372 DEBUG SystemMonitor:322 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-06-03 18:01:57,372 DEBUG SystemMonitor:322 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-06-03 18:01:57,373 INFO HandlerThread:322 [interfaces.py:finish():200] Joined disk monitor 2024-06-03 18:01:57,374 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 18:01:59,375 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:02:04,376 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:02:06,082 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:02:06,082 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:02:10,166 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:02:15,167 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:02:16,908 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: resume 2024-06-03 18:02:16,908 INFO HandlerThread:322 [handler.py:handle_request_resume():715] starting system metrics thread 2024-06-03 18:02:16,908 INFO HandlerThread:322 [system_monitor.py:start():194] Starting system monitor 2024-06-03 18:02:16,909 INFO SystemMonitor:322 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-06-03 18:02:16,909 INFO SystemMonitor:322 [interfaces.py:start():188] Started cpu monitoring 2024-06-03 18:02:16,910 INFO SystemMonitor:322 [interfaces.py:start():188] Started disk monitoring 2024-06-03 18:02:16,912 INFO SystemMonitor:322 [interfaces.py:start():188] Started gpu monitoring 2024-06-03 18:02:16,912 INFO SystemMonitor:322 [interfaces.py:start():188] Started memory monitoring 2024-06-03 18:02:16,913 INFO SystemMonitor:322 [interfaces.py:start():188] Started network monitoring 2024-06-03 18:02:16,952 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: pause 2024-06-03 18:02:16,952 INFO HandlerThread:322 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-06-03 18:02:16,952 INFO HandlerThread:322 [system_monitor.py:finish():203] Stopping system monitor 2024-06-03 18:02:16,952 INFO HandlerThread:322 [interfaces.py:finish():200] Joined cpu monitor 2024-06-03 18:02:16,953 DEBUG SystemMonitor:322 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-06-03 18:02:16,953 INFO HandlerThread:322 [interfaces.py:finish():200] Joined disk monitor 2024-06-03 18:02:16,953 DEBUG SystemMonitor:322 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-06-03 18:02:16,954 DEBUG SystemMonitor:322 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-06-03 18:02:16,966 INFO HandlerThread:322 [interfaces.py:finish():200] Joined gpu monitor 2024-06-03 18:02:16,967 INFO HandlerThread:322 [interfaces.py:finish():200] Joined memory monitor 2024-06-03 18:02:16,967 INFO HandlerThread:322 [interfaces.py:finish():200] Joined network monitor 2024-06-03 18:02:16,967 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 18:02:20,968 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:02:21,082 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:02:21,083 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:02:26,127 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:02:31,128 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:02:36,082 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:02:36,083 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:02:36,165 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:02:41,167 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:02:46,168 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:02:46,251 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: resume 2024-06-03 18:02:46,251 INFO HandlerThread:322 [handler.py:handle_request_resume():715] starting system metrics thread 2024-06-03 18:02:46,251 INFO HandlerThread:322 [system_monitor.py:start():194] Starting system monitor 2024-06-03 18:02:46,251 INFO SystemMonitor:322 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-06-03 18:02:46,252 INFO SystemMonitor:322 [interfaces.py:start():188] Started cpu monitoring 2024-06-03 18:02:46,252 INFO SystemMonitor:322 [interfaces.py:start():188] Started disk monitoring 2024-06-03 18:02:46,253 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: pause 2024-06-03 18:02:46,253 INFO HandlerThread:322 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-06-03 18:02:46,253 INFO HandlerThread:322 [system_monitor.py:finish():203] Stopping system monitor 2024-06-03 18:02:46,254 INFO SystemMonitor:322 [interfaces.py:start():188] Started gpu monitoring 2024-06-03 18:02:46,254 DEBUG SystemMonitor:322 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-06-03 18:02:46,255 DEBUG SystemMonitor:322 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-06-03 18:02:46,255 DEBUG SystemMonitor:322 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-06-03 18:02:46,256 INFO HandlerThread:322 [interfaces.py:finish():200] Joined cpu monitor 2024-06-03 18:02:46,257 INFO HandlerThread:322 [interfaces.py:finish():200] Joined disk monitor 2024-06-03 18:02:46,257 INFO HandlerThread:322 [interfaces.py:finish():200] Joined gpu monitor 2024-06-03 18:02:46,258 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 18:02:51,082 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:02:51,083 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:02:51,208 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:02:56,209 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:03:01,210 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:03:06,082 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:03:06,083 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:03:07,150 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:03:12,151 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:03:17,153 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:03:21,083 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:03:21,083 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:03:22,167 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:03:27,168 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:03:32,169 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:03:36,083 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:03:36,084 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:03:37,206 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:03:42,207 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:03:47,209 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:03:47,944 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: resume 2024-06-03 18:03:47,944 INFO HandlerThread:322 [handler.py:handle_request_resume():715] starting system metrics thread 2024-06-03 18:03:47,944 INFO HandlerThread:322 [system_monitor.py:start():194] Starting system monitor 2024-06-03 18:03:47,944 INFO SystemMonitor:322 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-06-03 18:03:47,945 INFO SystemMonitor:322 [interfaces.py:start():188] Started cpu monitoring 2024-06-03 18:03:47,945 INFO SystemMonitor:322 [interfaces.py:start():188] Started disk monitoring 2024-06-03 18:03:47,946 INFO SystemMonitor:322 [interfaces.py:start():188] Started gpu monitoring 2024-06-03 18:03:47,946 INFO SystemMonitor:322 [interfaces.py:start():188] Started memory monitoring 2024-06-03 18:03:47,947 INFO SystemMonitor:322 [interfaces.py:start():188] Started network monitoring 2024-06-03 18:03:48,030 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: pause 2024-06-03 18:03:48,030 INFO HandlerThread:322 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-06-03 18:03:48,030 INFO HandlerThread:322 [system_monitor.py:finish():203] Stopping system monitor 2024-06-03 18:03:48,030 DEBUG SystemMonitor:322 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-06-03 18:03:48,030 DEBUG SystemMonitor:322 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-06-03 18:03:48,031 DEBUG SystemMonitor:322 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-06-03 18:03:48,031 INFO HandlerThread:322 [interfaces.py:finish():200] Joined cpu monitor 2024-06-03 18:03:48,032 INFO HandlerThread:322 [interfaces.py:finish():200] Joined disk monitor 2024-06-03 18:03:48,041 INFO HandlerThread:322 [interfaces.py:finish():200] Joined gpu monitor 2024-06-03 18:03:48,042 INFO HandlerThread:322 [interfaces.py:finish():200] Joined memory monitor 2024-06-03 18:03:48,042 INFO HandlerThread:322 [interfaces.py:finish():200] Joined network monitor 2024-06-03 18:03:48,042 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 18:03:51,083 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:03:51,084 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:03:53,177 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:03:58,178 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:04:03,178 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:04:06,083 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:04:06,084 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:04:08,215 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:04:13,216 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:04:13,707 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: resume 2024-06-03 18:04:13,707 INFO HandlerThread:322 [handler.py:handle_request_resume():715] starting system metrics thread 2024-06-03 18:04:13,707 INFO HandlerThread:322 [system_monitor.py:start():194] Starting system monitor 2024-06-03 18:04:13,707 INFO SystemMonitor:322 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-06-03 18:04:13,707 INFO SystemMonitor:322 [interfaces.py:start():188] Started cpu monitoring 2024-06-03 18:04:13,708 INFO SystemMonitor:322 [interfaces.py:start():188] Started disk monitoring 2024-06-03 18:04:13,709 INFO SystemMonitor:322 [interfaces.py:start():188] Started gpu monitoring 2024-06-03 18:04:13,710 INFO SystemMonitor:322 [interfaces.py:start():188] Started memory monitoring 2024-06-03 18:04:13,711 INFO SystemMonitor:322 [interfaces.py:start():188] Started network monitoring 2024-06-03 18:04:13,759 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: pause 2024-06-03 18:04:13,760 INFO HandlerThread:322 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-06-03 18:04:13,760 INFO HandlerThread:322 [system_monitor.py:finish():203] Stopping system monitor 2024-06-03 18:04:13,760 DEBUG SystemMonitor:322 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-06-03 18:04:13,760 DEBUG SystemMonitor:322 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-06-03 18:04:13,760 DEBUG SystemMonitor:322 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-06-03 18:04:13,761 INFO HandlerThread:322 [interfaces.py:finish():200] Joined cpu monitor 2024-06-03 18:04:13,761 INFO HandlerThread:322 [interfaces.py:finish():200] Joined disk monitor 2024-06-03 18:04:13,772 INFO HandlerThread:322 [interfaces.py:finish():200] Joined gpu monitor 2024-06-03 18:04:13,772 INFO HandlerThread:322 [interfaces.py:finish():200] Joined memory monitor 2024-06-03 18:04:13,772 INFO HandlerThread:322 [interfaces.py:finish():200] Joined network monitor 2024-06-03 18:04:13,773 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 18:04:18,774 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:04:21,083 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:04:21,084 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:04:24,199 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:04:26,492 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: resume 2024-06-03 18:04:26,492 INFO HandlerThread:322 [handler.py:handle_request_resume():715] starting system metrics thread 2024-06-03 18:04:26,492 INFO HandlerThread:322 [system_monitor.py:start():194] Starting system monitor 2024-06-03 18:04:26,492 INFO SystemMonitor:322 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-06-03 18:04:26,493 INFO SystemMonitor:322 [interfaces.py:start():188] Started cpu monitoring 2024-06-03 18:04:26,494 INFO SystemMonitor:322 [interfaces.py:start():188] Started disk monitoring 2024-06-03 18:04:26,494 INFO SystemMonitor:322 [interfaces.py:start():188] Started gpu monitoring 2024-06-03 18:04:26,494 INFO SystemMonitor:322 [interfaces.py:start():188] Started memory monitoring 2024-06-03 18:04:26,495 INFO SystemMonitor:322 [interfaces.py:start():188] Started network monitoring 2024-06-03 18:04:26,697 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: pause 2024-06-03 18:04:26,698 INFO HandlerThread:322 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-06-03 18:04:26,698 INFO HandlerThread:322 [system_monitor.py:finish():203] Stopping system monitor 2024-06-03 18:04:26,698 DEBUG SystemMonitor:322 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-06-03 18:04:26,698 DEBUG SystemMonitor:322 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-06-03 18:04:26,698 DEBUG SystemMonitor:322 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-06-03 18:04:26,699 INFO HandlerThread:322 [interfaces.py:finish():200] Joined cpu monitor 2024-06-03 18:04:26,700 INFO HandlerThread:322 [interfaces.py:finish():200] Joined disk monitor 2024-06-03 18:04:26,709 INFO HandlerThread:322 [interfaces.py:finish():200] Joined gpu monitor 2024-06-03 18:04:26,709 INFO HandlerThread:322 [interfaces.py:finish():200] Joined memory monitor 2024-06-03 18:04:26,710 INFO HandlerThread:322 [interfaces.py:finish():200] Joined network monitor 2024-06-03 18:04:26,710 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 18:04:29,711 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:04:34,327 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: resume 2024-06-03 18:04:34,328 INFO HandlerThread:322 [handler.py:handle_request_resume():715] starting system metrics thread 2024-06-03 18:04:34,328 INFO HandlerThread:322 [system_monitor.py:start():194] Starting system monitor 2024-06-03 18:04:34,328 INFO SystemMonitor:322 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-06-03 18:04:34,328 INFO SystemMonitor:322 [interfaces.py:start():188] Started cpu monitoring 2024-06-03 18:04:34,329 INFO SystemMonitor:322 [interfaces.py:start():188] Started disk monitoring 2024-06-03 18:04:34,333 INFO SystemMonitor:322 [interfaces.py:start():188] Started gpu monitoring 2024-06-03 18:04:34,333 INFO SystemMonitor:322 [interfaces.py:start():188] Started memory monitoring 2024-06-03 18:04:34,337 INFO SystemMonitor:322 [interfaces.py:start():188] Started network monitoring 2024-06-03 18:04:34,712 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:04:35,575 DEBUG SenderThread:322 [sender.py:send():378] send: config 2024-06-03 18:04:35,576 DEBUG SenderThread:322 [sender.py:send():378] send: metric 2024-06-03 18:04:35,576 DEBUG SenderThread:322 [sender.py:send():378] send: metric 2024-06-03 18:04:35,576 WARNING SenderThread:322 [sender.py:send_metric():1340] Seen metric with glob (shouldn't happen) 2024-06-03 18:04:35,888 INFO Thread-12 :322 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/output.log 2024-06-03 18:04:36,084 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:04:36,085 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:04:37,889 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/output.log 2024-06-03 18:04:40,220 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:04:40,890 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/config.yaml 2024-06-03 18:04:45,320 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:04:50,321 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:04:51,633 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:04:51,633 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:04:55,717 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:05:00,718 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:05:05,719 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:05:06,178 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:05:06,178 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:05:11,618 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:05:16,619 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:05:21,178 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:05:21,179 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:05:22,618 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:05:27,619 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:05:32,620 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:05:34,338 DEBUG SystemMonitor:322 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-06-03 18:05:34,339 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 18:05:36,179 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:05:36,179 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:05:38,618 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:05:43,619 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:05:48,620 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:05:51,178 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:05:51,179 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:05:54,618 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:05:59,619 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:06:04,341 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 18:06:05,343 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:06:06,179 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:06:06,179 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:06:10,618 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:06:15,619 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:06:20,620 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:06:21,179 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:06:21,179 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:06:26,619 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:06:31,620 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:06:34,341 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 18:06:36,179 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:06:36,180 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:06:37,619 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:06:42,620 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:06:47,621 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:06:51,179 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:06:51,180 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:06:53,619 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:06:58,620 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:07:03,621 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:07:04,342 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 18:07:06,179 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:07:06,180 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:07:09,619 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:07:14,620 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:07:19,621 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:07:21,594 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:07:21,595 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:07:24,682 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:07:29,683 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:07:34,343 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 18:07:35,344 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:07:36,594 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:07:36,594 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:07:40,717 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:07:45,718 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:07:50,719 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:07:51,594 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:07:51,595 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:07:56,692 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:08:01,693 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:08:04,344 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 18:08:06,594 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:08:06,595 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:08:07,662 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:08:12,663 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:08:15,487 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: partial_history 2024-06-03 18:08:15,490 DEBUG SenderThread:322 [sender.py:send():378] send: metric 2024-06-03 18:08:15,490 DEBUG SenderThread:322 [sender.py:send():378] send: metric 2024-06-03 18:08:15,490 DEBUG SenderThread:322 [sender.py:send():378] send: metric 2024-06-03 18:08:15,490 DEBUG SenderThread:322 [sender.py:send():378] send: metric 2024-06-03 18:08:15,490 DEBUG SenderThread:322 [sender.py:send():378] send: history 2024-06-03 18:08:15,490 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: summary_record 2024-06-03 18:08:15,491 INFO SenderThread:322 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-06-03 18:08:15,966 INFO Thread-12 :322 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/wandb-summary.json 2024-06-03 18:08:18,491 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:08:21,594 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:08:21,595 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:08:23,896 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:08:23,969 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/output.log 2024-06-03 18:08:24,969 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/config.yaml 2024-06-03 18:08:29,051 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:08:34,051 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:08:34,345 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 18:08:36,594 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:08:36,595 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:08:39,724 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:08:44,725 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:08:49,726 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:08:51,595 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:08:51,595 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:08:55,716 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:09:00,717 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:09:04,345 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 18:09:06,346 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:09:06,595 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:09:06,595 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:09:11,671 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:09:16,673 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:09:21,595 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:09:21,595 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:09:22,664 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:09:27,665 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:09:32,666 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:09:34,346 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 18:09:36,595 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:09:36,596 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:09:38,652 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:09:43,653 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:09:48,654 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:09:51,595 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:09:51,596 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:09:53,691 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:09:58,692 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:10:03,693 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:10:04,347 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 18:10:06,595 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:10:06,596 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:10:08,697 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:10:13,698 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:10:18,699 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:10:21,596 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:10:21,596 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:10:24,669 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:10:29,670 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:10:34,353 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 18:10:35,353 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:10:36,596 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:10:36,596 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:10:40,728 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:10:45,729 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:10:50,730 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:10:51,596 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:10:51,596 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:10:56,651 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:11:01,652 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:11:04,349 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 18:11:06,596 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:11:06,596 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:11:06,727 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:11:11,728 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:11:16,728 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:11:21,596 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:11:21,597 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:11:22,690 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:11:24,911 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: partial_history 2024-06-03 18:11:24,913 DEBUG SenderThread:322 [sender.py:send():378] send: history 2024-06-03 18:11:24,913 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: summary_record 2024-06-03 18:11:24,915 INFO SenderThread:322 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-06-03 18:11:25,035 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/wandb-summary.json 2024-06-03 18:11:27,916 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:11:32,917 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:11:34,350 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 18:11:36,596 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:11:36,597 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:11:38,644 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:11:43,645 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:11:48,645 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:11:51,596 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:11:51,597 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:11:52,047 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/output.log 2024-06-03 18:11:53,710 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:11:58,711 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:12:03,711 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:12:04,351 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 18:12:06,597 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:12:06,597 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:12:09,706 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:12:14,707 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:12:19,708 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:12:21,597 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:12:21,597 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:12:25,703 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:12:30,704 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:12:34,352 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 18:12:36,353 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:12:36,597 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:12:36,597 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:12:41,705 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:12:46,706 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:12:51,597 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:12:51,597 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:12:52,688 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:12:57,689 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:13:02,690 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:13:04,353 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 18:13:06,597 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:13:06,598 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:13:07,697 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:13:12,698 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:13:17,699 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:13:21,597 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:13:21,598 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:13:23,626 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:13:28,627 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:13:33,628 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:13:34,354 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 18:13:36,597 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:13:36,598 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:13:38,650 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:13:43,651 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:13:48,652 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:13:51,598 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:13:51,598 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:13:54,635 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:13:59,636 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:14:04,355 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 18:14:05,356 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:14:06,598 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:14:06,598 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:14:10,747 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:14:15,748 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:14:20,749 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:14:21,598 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:14:21,598 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:14:26,675 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:14:31,676 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:14:34,356 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 18:14:34,503 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: partial_history 2024-06-03 18:14:34,504 DEBUG SenderThread:322 [sender.py:send():378] send: history 2024-06-03 18:14:34,505 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: summary_record 2024-06-03 18:14:34,505 INFO SenderThread:322 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-06-03 18:14:35,107 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/wandb-summary.json 2024-06-03 18:14:36,598 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:14:36,599 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:14:37,647 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:14:42,647 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:14:47,648 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:14:51,598 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:14:51,599 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:14:52,666 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:14:57,667 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:15:02,668 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:15:04,357 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 18:15:06,598 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:15:06,599 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:15:07,733 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:15:12,734 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:15:17,735 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:15:21,598 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:15:21,599 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:15:23,125 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/output.log 2024-06-03 18:15:23,735 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:15:28,736 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:15:33,737 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:15:34,357 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 18:15:36,599 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:15:36,599 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:15:39,732 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:15:44,733 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:15:49,734 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:15:51,599 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:15:51,599 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:15:55,718 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:16:00,719 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:16:04,359 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 18:16:06,360 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:16:06,599 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:16:06,599 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:16:11,722 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:16:16,723 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:16:21,599 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:16:21,600 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:16:22,671 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:16:27,671 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:16:32,672 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:16:34,359 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 18:16:36,599 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:16:36,600 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:16:38,667 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:16:43,667 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:16:48,668 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:16:51,600 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:16:51,600 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:16:54,633 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:16:59,634 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:17:04,361 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 18:17:05,362 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:17:06,600 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:17:06,600 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:17:10,704 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:17:15,705 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:17:20,706 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:17:21,600 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:17:21,600 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:17:26,693 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:17:31,694 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:17:34,362 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 18:17:36,600 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:17:36,601 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:17:36,720 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:17:41,721 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:17:44,033 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: partial_history 2024-06-03 18:17:44,035 DEBUG SenderThread:322 [sender.py:send():378] send: history 2024-06-03 18:17:44,035 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: summary_record 2024-06-03 18:17:44,035 INFO SenderThread:322 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-06-03 18:17:44,178 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/wandb-summary.json 2024-06-03 18:17:47,036 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:17:51,600 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:17:51,601 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:17:52,706 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:17:57,707 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:18:02,707 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:18:04,362 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 18:18:06,600 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:18:06,601 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:18:07,715 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:18:12,716 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:18:17,717 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:18:21,600 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:18:21,601 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:18:23,681 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:18:28,681 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:18:33,682 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:18:34,363 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 18:18:36,601 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:18:36,601 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:18:39,683 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:18:44,684 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:18:49,685 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:18:50,560 DEBUG SenderThread:322 [sender.py:send():378] send: telemetry 2024-06-03 18:18:50,561 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: partial_history 2024-06-03 18:18:50,562 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: summary_record 2024-06-03 18:18:50,563 INFO SenderThread:322 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-06-03 18:18:50,564 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: summary_record 2024-06-03 18:18:50,564 INFO SenderThread:322 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-06-03 18:18:50,564 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: summary_record 2024-06-03 18:18:50,565 INFO SenderThread:322 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-06-03 18:18:50,565 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: summary_record 2024-06-03 18:18:50,566 INFO SenderThread:322 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-06-03 18:18:50,566 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: summary_record 2024-06-03 18:18:50,566 INFO SenderThread:322 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-06-03 18:18:50,567 DEBUG SenderThread:322 [sender.py:send():378] send: history 2024-06-03 18:18:50,567 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: summary_record 2024-06-03 18:18:50,567 INFO SenderThread:322 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-06-03 18:18:50,784 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: pause 2024-06-03 18:18:50,785 INFO HandlerThread:322 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-06-03 18:18:50,785 INFO HandlerThread:322 [system_monitor.py:finish():203] Stopping system monitor 2024-06-03 18:18:50,785 DEBUG SystemMonitor:322 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-06-03 18:18:50,785 INFO HandlerThread:322 [interfaces.py:finish():200] Joined cpu monitor 2024-06-03 18:18:50,785 DEBUG SystemMonitor:322 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-06-03 18:18:50,786 INFO HandlerThread:322 [interfaces.py:finish():200] Joined disk monitor 2024-06-03 18:18:50,797 INFO HandlerThread:322 [interfaces.py:finish():200] Joined gpu monitor 2024-06-03 18:18:50,797 INFO HandlerThread:322 [interfaces.py:finish():200] Joined memory monitor 2024-06-03 18:18:50,797 INFO HandlerThread:322 [interfaces.py:finish():200] Joined network monitor 2024-06-03 18:18:50,798 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 18:18:51,202 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/wandb-summary.json 2024-06-03 18:18:51,601 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:18:51,601 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:18:53,203 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/output.log 2024-06-03 18:18:55,646 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:18:56,204 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/config.yaml 2024-06-03 18:19:00,760 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:19:05,761 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:19:06,601 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:19:06,601 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:19:11,734 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:19:16,735 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:19:21,601 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:19:21,602 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:19:22,665 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:19:27,666 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:19:32,667 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:19:36,601 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:19:36,602 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:19:38,643 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:19:43,644 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:19:48,646 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:19:51,602 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:19:51,602 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:19:53,698 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:19:58,699 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:20:03,701 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:20:06,602 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:20:06,602 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:20:09,692 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:20:14,694 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:20:19,695 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:20:21,602 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:20:21,602 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:20:24,726 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:20:29,727 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:20:34,727 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:20:36,602 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:20:36,603 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:20:40,629 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:20:45,630 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:20:50,631 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:20:51,602 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:20:51,603 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:20:55,697 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:21:00,698 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:21:05,699 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:21:06,602 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:21:06,603 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:21:11,689 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:21:16,690 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:21:21,603 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:21:21,603 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:21:21,693 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:21:26,694 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:21:31,695 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:21:36,603 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:21:36,603 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:21:37,655 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:21:42,656 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:21:47,657 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:21:51,603 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:21:51,603 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:21:53,656 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:21:58,657 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:22:03,658 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:22:06,603 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:22:06,604 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:22:08,729 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:22:13,730 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:22:18,731 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:22:21,603 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:22:21,604 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:22:24,656 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:22:29,658 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:22:34,658 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:22:36,604 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:22:36,604 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:22:39,714 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:22:44,715 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:22:49,717 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:22:51,604 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:22:51,604 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:22:55,674 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:23:00,675 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:23:05,676 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:23:06,604 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:23:06,604 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:23:11,666 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:23:16,668 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:23:21,604 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:23:21,604 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:23:21,696 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:23:26,697 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:23:31,698 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:23:36,604 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:23:36,605 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:23:37,690 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:23:42,691 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:23:47,692 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:23:51,604 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:23:51,605 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:23:52,725 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:23:57,726 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:24:02,727 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:24:06,604 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:24:06,605 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:24:08,640 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:24:13,641 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:24:18,642 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:24:21,605 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:24:21,605 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:24:23,718 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:24:28,718 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:24:33,720 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:24:36,605 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:24:36,605 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:24:39,666 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:24:44,668 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:24:49,669 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:24:51,605 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:24:51,605 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:24:54,675 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:24:59,676 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:25:04,677 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:25:06,605 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:25:06,605 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:25:09,697 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:25:14,698 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:25:19,699 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:25:21,605 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:25:21,606 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:25:24,740 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:25:29,741 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:25:34,742 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:25:36,605 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:25:36,606 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:25:40,660 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:25:45,661 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:25:50,662 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:25:51,605 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:25:51,606 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:25:55,695 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:26:00,696 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:26:05,697 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:26:06,606 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:26:06,606 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:26:11,649 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:26:16,650 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:26:21,606 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:26:21,606 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:26:22,634 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:26:27,635 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:26:32,636 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:26:36,606 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:26:36,606 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:26:37,738 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:26:42,740 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:26:47,741 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:26:51,606 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:26:51,607 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:26:53,697 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:26:58,698 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:27:03,699 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:27:06,606 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:27:06,607 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:27:08,732 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:27:13,734 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:27:18,735 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:27:21,607 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:27:21,607 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:27:24,715 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:27:29,716 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:27:34,717 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:27:36,607 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:27:36,607 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:27:39,732 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:27:44,734 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:27:49,735 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:27:51,607 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:27:51,607 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:27:54,740 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:27:59,741 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:28:04,742 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:28:06,607 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:28:06,608 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:28:10,666 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:28:15,667 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:28:20,668 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:28:21,607 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:28:21,608 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:28:25,703 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:28:30,704 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:28:35,706 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:28:36,607 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:28:36,608 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:28:41,649 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:28:46,650 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:28:51,607 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:28:51,608 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:28:51,705 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:28:56,706 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:29:01,707 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:29:06,608 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:29:06,608 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:29:06,721 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:29:11,722 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:29:16,723 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:29:21,608 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:29:21,608 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:29:22,659 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:29:27,661 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:29:32,662 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:29:36,608 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:29:36,608 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:29:37,685 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:29:42,687 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:29:47,688 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:29:51,608 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:29:51,609 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:29:53,636 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:29:58,637 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:30:03,639 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:30:06,608 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:30:06,609 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:30:08,699 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:30:13,700 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:30:18,702 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:30:21,609 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:30:21,609 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:30:23,714 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:30:28,715 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:30:33,716 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:30:36,609 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:30:36,609 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:30:38,730 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:30:43,731 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:30:48,732 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:30:51,609 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:30:51,609 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:30:53,734 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:30:58,735 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:31:03,736 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:31:06,609 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:31:06,610 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:31:09,704 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:31:14,705 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:31:19,706 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:31:21,609 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:31:21,610 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:31:24,706 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:31:29,707 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:31:34,709 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:31:36,610 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:31:36,610 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:31:39,721 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:31:44,722 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:31:49,723 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:31:51,610 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:31:51,610 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:31:55,635 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:32:00,636 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:32:05,638 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:32:06,610 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:32:06,610 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:32:10,710 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:32:15,711 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:32:20,711 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:32:21,610 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:32:21,610 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:32:26,666 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:32:31,667 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:32:36,610 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:32:36,611 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:32:37,646 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:32:42,647 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:32:47,648 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:32:51,610 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:32:51,611 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:32:52,665 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:32:57,666 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:33:02,667 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:33:06,611 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:33:06,611 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:33:07,713 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:33:12,714 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:33:17,715 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:33:21,611 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:33:21,611 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:33:22,722 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:33:27,723 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:33:32,724 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:33:36,611 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:33:36,611 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:33:37,733 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:33:42,734 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:33:47,735 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:33:51,611 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:33:51,611 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:33:53,713 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:33:58,715 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:34:03,716 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:34:06,611 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:34:06,612 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:34:09,686 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:34:14,687 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:34:19,688 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:34:21,611 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:34:21,612 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:34:25,646 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:34:30,647 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:34:35,648 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:34:36,612 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:34:36,612 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:34:40,672 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:34:45,674 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:34:50,675 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:34:51,612 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:34:51,612 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:34:56,665 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:35:01,666 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:35:06,612 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:35:06,613 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:35:06,711 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:35:11,712 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:35:16,713 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:35:21,612 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:35:21,613 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:35:22,663 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:35:27,664 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:35:32,665 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:35:36,612 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:35:36,613 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:35:37,672 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:35:42,673 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:35:47,674 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:35:51,613 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:35:51,613 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:35:53,667 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:35:58,668 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:36:03,669 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:36:06,613 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:36:06,613 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:36:09,662 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:36:14,663 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:36:19,664 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:36:21,613 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:36:21,613 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:36:24,728 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:36:29,729 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:36:34,730 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:36:36,613 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:36:36,613 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:36:40,701 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:36:45,701 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:36:50,703 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:36:51,613 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:36:51,614 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:36:56,679 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:37:01,680 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:37:06,613 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:37:06,614 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:37:07,665 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:37:12,666 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:37:17,667 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:37:21,614 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:37:21,614 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:37:23,665 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:37:28,666 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:37:33,667 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:37:36,614 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:37:36,614 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:37:38,685 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:37:43,687 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:37:48,688 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:37:51,614 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:37:51,614 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:37:54,676 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:37:59,678 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:38:04,679 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:38:06,614 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:38:06,615 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:38:10,656 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:38:15,657 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:38:20,658 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:38:21,614 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:38:21,615 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:38:25,687 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:38:30,688 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:38:35,689 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:38:36,614 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:38:36,615 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:38:40,702 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:38:45,703 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:38:50,704 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:38:51,615 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:38:51,615 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:38:55,721 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:39:00,722 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:39:05,723 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:39:06,615 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:39:06,615 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:39:11,650 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:39:16,651 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:39:21,615 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:39:21,616 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:39:21,674 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:39:26,675 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:39:31,676 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:39:36,615 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:39:36,616 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:39:36,688 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:39:41,689 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:39:46,690 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:39:51,615 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:39:51,616 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:39:51,736 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:39:56,737 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:40:01,738 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:40:06,616 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:40:06,616 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:40:07,700 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:40:12,701 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:40:17,701 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:40:21,616 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:40:21,616 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:40:22,745 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:40:27,746 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:40:32,747 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:40:36,616 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:40:36,616 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:40:38,687 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:40:43,688 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:40:48,689 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:40:51,616 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:40:51,617 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:40:54,678 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:40:59,679 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:41:04,680 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:41:05,951 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: resume 2024-06-03 18:41:05,951 INFO HandlerThread:322 [handler.py:handle_request_resume():715] starting system metrics thread 2024-06-03 18:41:05,952 INFO HandlerThread:322 [system_monitor.py:start():194] Starting system monitor 2024-06-03 18:41:05,952 INFO SystemMonitor:322 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-06-03 18:41:05,952 INFO SystemMonitor:322 [interfaces.py:start():188] Started cpu monitoring 2024-06-03 18:41:05,953 INFO SystemMonitor:322 [interfaces.py:start():188] Started disk monitoring 2024-06-03 18:41:05,955 INFO SystemMonitor:322 [interfaces.py:start():188] Started gpu monitoring 2024-06-03 18:41:05,956 INFO SystemMonitor:322 [interfaces.py:start():188] Started memory monitoring 2024-06-03 18:41:05,957 INFO SystemMonitor:322 [interfaces.py:start():188] Started network monitoring 2024-06-03 18:41:05,992 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: pause 2024-06-03 18:41:05,992 INFO HandlerThread:322 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-06-03 18:41:05,992 INFO HandlerThread:322 [system_monitor.py:finish():203] Stopping system monitor 2024-06-03 18:41:05,993 INFO HandlerThread:322 [interfaces.py:finish():200] Joined cpu monitor 2024-06-03 18:41:05,993 INFO HandlerThread:322 [interfaces.py:finish():200] Joined disk monitor 2024-06-03 18:41:05,993 DEBUG SystemMonitor:322 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-06-03 18:41:05,994 DEBUG SystemMonitor:322 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-06-03 18:41:05,994 DEBUG SystemMonitor:322 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-06-03 18:41:06,005 INFO HandlerThread:322 [interfaces.py:finish():200] Joined gpu monitor 2024-06-03 18:41:06,005 INFO HandlerThread:322 [interfaces.py:finish():200] Joined memory monitor 2024-06-03 18:41:06,005 INFO HandlerThread:322 [interfaces.py:finish():200] Joined network monitor 2024-06-03 18:41:06,006 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 18:41:06,616 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:41:06,617 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:41:08,718 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/output.log 2024-06-03 18:41:10,666 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:41:15,667 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:41:20,668 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:41:21,616 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:41:21,617 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:41:26,666 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:41:31,667 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:41:36,617 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:41:36,617 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:41:36,743 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:41:41,744 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:41:46,745 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:41:51,617 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:41:51,617 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:41:52,743 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:41:57,744 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:42:02,745 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:42:06,617 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:42:06,617 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:42:08,707 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:42:13,708 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:42:18,709 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:42:21,617 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:42:21,618 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:42:24,663 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:42:29,664 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:42:34,666 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:42:36,617 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:42:36,618 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:42:39,737 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:42:44,738 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:42:49,739 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:42:51,617 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:42:51,618 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:42:55,685 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:43:00,686 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:43:05,687 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:43:06,618 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:43:06,618 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:43:10,719 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:43:15,720 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:43:20,721 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:43:21,618 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:43:21,618 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:43:25,735 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:43:30,736 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:43:35,737 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:43:36,618 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:43:36,618 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:43:41,707 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:43:46,708 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:43:51,618 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:43:51,619 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:43:52,690 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:43:57,691 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:44:02,693 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:44:06,618 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:44:06,619 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:44:07,734 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:44:12,735 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:44:17,736 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:44:21,618 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:44:21,619 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:44:23,728 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:44:28,729 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:44:33,730 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:44:36,619 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:44:36,619 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:44:39,666 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:44:44,668 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:44:49,669 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:44:51,619 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:44:51,619 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:44:55,646 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:45:00,647 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:45:05,648 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:45:06,619 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:45:06,620 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:45:10,747 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:45:15,748 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:45:20,749 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:45:21,619 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:45:21,619 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:45:26,729 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:45:28,893 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: resume 2024-06-03 18:45:28,893 INFO HandlerThread:322 [handler.py:handle_request_resume():715] starting system metrics thread 2024-06-03 18:45:28,893 INFO HandlerThread:322 [system_monitor.py:start():194] Starting system monitor 2024-06-03 18:45:28,893 INFO SystemMonitor:322 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-06-03 18:45:28,894 INFO SystemMonitor:322 [interfaces.py:start():188] Started cpu monitoring 2024-06-03 18:45:28,895 INFO SystemMonitor:322 [interfaces.py:start():188] Started disk monitoring 2024-06-03 18:45:28,895 INFO SystemMonitor:322 [interfaces.py:start():188] Started gpu monitoring 2024-06-03 18:45:28,896 INFO SystemMonitor:322 [interfaces.py:start():188] Started memory monitoring 2024-06-03 18:45:28,898 INFO SystemMonitor:322 [interfaces.py:start():188] Started network monitoring 2024-06-03 18:45:28,927 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: pause 2024-06-03 18:45:28,927 INFO HandlerThread:322 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-06-03 18:45:28,927 INFO HandlerThread:322 [system_monitor.py:finish():203] Stopping system monitor 2024-06-03 18:45:28,928 INFO HandlerThread:322 [interfaces.py:finish():200] Joined cpu monitor 2024-06-03 18:45:28,928 DEBUG SystemMonitor:322 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-06-03 18:45:28,928 DEBUG SystemMonitor:322 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-06-03 18:45:28,928 DEBUG SystemMonitor:322 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-06-03 18:45:28,929 INFO HandlerThread:322 [interfaces.py:finish():200] Joined disk monitor 2024-06-03 18:45:28,939 INFO HandlerThread:322 [interfaces.py:finish():200] Joined gpu monitor 2024-06-03 18:45:28,939 INFO HandlerThread:322 [interfaces.py:finish():200] Joined memory monitor 2024-06-03 18:45:28,940 INFO HandlerThread:322 [interfaces.py:finish():200] Joined network monitor 2024-06-03 18:45:28,940 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 18:45:30,229 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: resume 2024-06-03 18:45:30,229 INFO HandlerThread:322 [handler.py:handle_request_resume():715] starting system metrics thread 2024-06-03 18:45:30,229 INFO HandlerThread:322 [system_monitor.py:start():194] Starting system monitor 2024-06-03 18:45:30,229 INFO SystemMonitor:322 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-06-03 18:45:30,230 INFO SystemMonitor:322 [interfaces.py:start():188] Started cpu monitoring 2024-06-03 18:45:30,230 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: pause 2024-06-03 18:45:30,230 INFO HandlerThread:322 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-06-03 18:45:30,231 INFO HandlerThread:322 [system_monitor.py:finish():203] Stopping system monitor 2024-06-03 18:45:30,230 INFO SystemMonitor:322 [interfaces.py:start():188] Started disk monitoring 2024-06-03 18:45:30,231 INFO HandlerThread:322 [interfaces.py:finish():200] Joined cpu monitor 2024-06-03 18:45:30,231 DEBUG SystemMonitor:322 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-06-03 18:45:30,232 DEBUG SystemMonitor:322 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-06-03 18:45:30,232 DEBUG SystemMonitor:322 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-06-03 18:45:30,234 INFO HandlerThread:322 [interfaces.py:finish():200] Joined disk monitor 2024-06-03 18:45:30,235 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 18:45:31,256 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: resume 2024-06-03 18:45:31,256 INFO HandlerThread:322 [handler.py:handle_request_resume():715] starting system metrics thread 2024-06-03 18:45:31,256 INFO HandlerThread:322 [system_monitor.py:start():194] Starting system monitor 2024-06-03 18:45:31,256 INFO SystemMonitor:322 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-06-03 18:45:31,257 INFO SystemMonitor:322 [interfaces.py:start():188] Started cpu monitoring 2024-06-03 18:45:31,257 INFO SystemMonitor:322 [interfaces.py:start():188] Started disk monitoring 2024-06-03 18:45:31,258 INFO SystemMonitor:322 [interfaces.py:start():188] Started gpu monitoring 2024-06-03 18:45:31,259 INFO SystemMonitor:322 [interfaces.py:start():188] Started memory monitoring 2024-06-03 18:45:31,260 INFO SystemMonitor:322 [interfaces.py:start():188] Started network monitoring 2024-06-03 18:45:31,277 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: pause 2024-06-03 18:45:31,277 INFO HandlerThread:322 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-06-03 18:45:31,277 INFO HandlerThread:322 [system_monitor.py:finish():203] Stopping system monitor 2024-06-03 18:45:31,278 DEBUG SystemMonitor:322 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-06-03 18:45:31,278 DEBUG SystemMonitor:322 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-06-03 18:45:31,278 DEBUG SystemMonitor:322 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-06-03 18:45:31,278 INFO HandlerThread:322 [interfaces.py:finish():200] Joined cpu monitor 2024-06-03 18:45:31,279 INFO HandlerThread:322 [interfaces.py:finish():200] Joined disk monitor 2024-06-03 18:45:31,289 INFO HandlerThread:322 [interfaces.py:finish():200] Joined gpu monitor 2024-06-03 18:45:31,289 INFO HandlerThread:322 [interfaces.py:finish():200] Joined memory monitor 2024-06-03 18:45:31,289 INFO HandlerThread:322 [interfaces.py:finish():200] Joined network monitor 2024-06-03 18:45:31,290 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 18:45:32,291 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:45:33,122 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: resume 2024-06-03 18:45:33,122 INFO HandlerThread:322 [handler.py:handle_request_resume():715] starting system metrics thread 2024-06-03 18:45:33,122 INFO HandlerThread:322 [system_monitor.py:start():194] Starting system monitor 2024-06-03 18:45:33,123 INFO SystemMonitor:322 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-06-03 18:45:33,123 INFO SystemMonitor:322 [interfaces.py:start():188] Started cpu monitoring 2024-06-03 18:45:33,124 INFO SystemMonitor:322 [interfaces.py:start():188] Started disk monitoring 2024-06-03 18:45:33,124 INFO SystemMonitor:322 [interfaces.py:start():188] Started gpu monitoring 2024-06-03 18:45:33,126 INFO SystemMonitor:322 [interfaces.py:start():188] Started memory monitoring 2024-06-03 18:45:33,127 INFO SystemMonitor:322 [interfaces.py:start():188] Started network monitoring 2024-06-03 18:45:33,359 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: pause 2024-06-03 18:45:33,359 INFO HandlerThread:322 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-06-03 18:45:33,359 INFO HandlerThread:322 [system_monitor.py:finish():203] Stopping system monitor 2024-06-03 18:45:33,359 DEBUG SystemMonitor:322 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-06-03 18:45:33,359 DEBUG SystemMonitor:322 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-06-03 18:45:33,360 DEBUG SystemMonitor:322 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-06-03 18:45:33,360 INFO HandlerThread:322 [interfaces.py:finish():200] Joined cpu monitor 2024-06-03 18:45:33,361 INFO HandlerThread:322 [interfaces.py:finish():200] Joined disk monitor 2024-06-03 18:45:33,373 INFO HandlerThread:322 [interfaces.py:finish():200] Joined gpu monitor 2024-06-03 18:45:33,373 INFO HandlerThread:322 [interfaces.py:finish():200] Joined memory monitor 2024-06-03 18:45:33,373 INFO HandlerThread:322 [interfaces.py:finish():200] Joined network monitor 2024-06-03 18:45:33,374 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 18:45:36,415 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: resume 2024-06-03 18:45:36,416 INFO HandlerThread:322 [handler.py:handle_request_resume():715] starting system metrics thread 2024-06-03 18:45:36,416 INFO HandlerThread:322 [system_monitor.py:start():194] Starting system monitor 2024-06-03 18:45:36,416 INFO SystemMonitor:322 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-06-03 18:45:36,416 INFO SystemMonitor:322 [interfaces.py:start():188] Started cpu monitoring 2024-06-03 18:45:36,417 INFO SystemMonitor:322 [interfaces.py:start():188] Started disk monitoring 2024-06-03 18:45:36,418 INFO SystemMonitor:322 [interfaces.py:start():188] Started gpu monitoring 2024-06-03 18:45:36,418 INFO SystemMonitor:322 [interfaces.py:start():188] Started memory monitoring 2024-06-03 18:45:36,418 INFO SystemMonitor:322 [interfaces.py:start():188] Started network monitoring 2024-06-03 18:45:36,814 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:45:36,814 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:45:37,687 DEBUG SenderThread:322 [sender.py:send():378] send: config 2024-06-03 18:45:37,689 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:45:37,727 DEBUG SenderThread:322 [sender.py:send():378] send: metric 2024-06-03 18:45:37,727 DEBUG SenderThread:322 [sender.py:send():378] send: metric 2024-06-03 18:45:37,727 WARNING SenderThread:322 [sender.py:send_metric():1340] Seen metric with glob (shouldn't happen) 2024-06-03 18:45:38,821 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/output.log 2024-06-03 18:45:42,728 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:45:47,729 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:45:52,102 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:45:52,103 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:45:53,227 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:45:53,827 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/config.yaml 2024-06-03 18:45:58,335 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:46:03,336 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:46:07,092 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:46:07,092 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:46:09,170 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:46:14,171 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:46:19,172 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:46:22,092 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:46:22,092 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:46:25,165 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:46:30,166 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:46:35,167 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:46:36,419 DEBUG SystemMonitor:322 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-06-03 18:46:36,420 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 18:46:37,092 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:46:37,092 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:46:41,162 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:46:46,162 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:46:51,163 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:46:52,092 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:46:52,093 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:46:56,196 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:47:01,197 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:47:06,198 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:47:06,422 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 18:47:07,092 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:47:07,093 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:47:11,223 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:47:16,224 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:47:21,225 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:47:22,092 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:47:22,093 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:47:27,210 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:47:32,211 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:47:36,426 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 18:47:37,093 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:47:37,093 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:47:37,235 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:47:42,236 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:47:47,237 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:47:52,093 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:47:52,093 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:47:53,187 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:47:58,189 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:48:03,190 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:48:06,422 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 18:48:07,093 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:48:07,093 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:48:09,182 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:48:14,182 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:48:19,183 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:48:22,093 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:48:22,094 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:48:25,147 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:48:30,148 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:48:35,149 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:48:36,425 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 18:48:37,093 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:48:37,094 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:48:40,176 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:48:45,177 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:48:50,178 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:48:52,093 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:48:52,094 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:48:56,162 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:49:01,163 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:49:06,164 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:49:06,424 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 18:49:07,094 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:49:07,094 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:49:11,237 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:49:15,811 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: partial_history 2024-06-03 18:49:15,812 DEBUG SenderThread:322 [sender.py:send():378] send: history 2024-06-03 18:49:15,812 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: summary_record 2024-06-03 18:49:15,813 INFO SenderThread:322 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-06-03 18:49:15,902 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/wandb-summary.json 2024-06-03 18:49:16,813 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:49:21,815 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:49:22,139 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:49:22,139 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:49:22,904 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/output.log 2024-06-03 18:49:24,905 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/output.log 2024-06-03 18:49:27,275 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:49:32,276 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:49:36,425 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 18:49:37,099 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:49:37,100 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:49:38,126 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:49:43,127 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:49:48,128 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:49:52,099 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:49:52,100 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:49:53,234 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:49:58,235 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:50:03,236 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:50:06,426 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 18:50:07,100 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:50:07,100 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:50:09,170 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:50:14,171 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:50:19,171 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:50:22,100 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:50:22,100 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:50:24,189 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:50:29,190 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:50:34,191 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:50:36,427 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 18:50:37,100 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:50:37,100 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:50:40,132 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:50:45,133 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:50:50,134 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:50:52,100 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:50:52,100 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:50:55,203 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:51:00,204 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:51:05,205 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:51:06,428 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 18:51:07,100 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:51:07,101 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:51:11,174 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:51:16,175 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:51:21,176 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:51:22,100 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:51:22,101 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:51:26,192 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:51:31,193 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:51:36,194 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:51:36,429 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 18:51:37,100 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: stop_status 2024-06-03 18:51:37,101 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: stop_status 2024-06-03 18:51:43,102 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:51:48,103 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:51:53,104 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:51:58,105 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:51:58,352 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:52:03,106 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:52:03,353 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:52:06,430 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 18:52:08,107 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:52:08,431 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:52:13,108 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:52:13,432 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:52:18,109 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:52:18,433 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:52:23,110 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:52:23,434 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:52:25,402 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: partial_history 2024-06-03 18:52:25,404 DEBUG SenderThread:322 [sender.py:send():378] send: history 2024-06-03 18:52:25,405 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: summary_record 2024-06-03 18:52:25,405 INFO SenderThread:322 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-06-03 18:52:25,973 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/wandb-summary.json 2024-06-03 18:52:28,111 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:52:29,406 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:52:33,284 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:52:34,407 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:52:36,431 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 18:52:38,285 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:52:39,431 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:52:43,286 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:52:44,433 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:52:48,287 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:52:49,434 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:52:52,982 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/output.log 2024-06-03 18:52:53,288 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:52:54,957 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:52:58,289 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:52:59,957 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:53:03,290 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:53:04,958 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:53:06,431 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 18:53:08,291 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:53:10,433 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:53:13,293 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:53:15,434 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:53:18,294 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:53:20,435 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:53:23,295 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:53:25,435 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:53:28,296 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:53:30,436 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:53:33,297 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:53:35,437 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:53:36,432 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 18:53:38,298 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:53:41,434 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:53:43,299 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:53:46,435 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:53:48,300 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:53:51,435 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:53:53,301 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:53:56,436 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:53:58,303 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:54:01,437 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:54:03,304 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:54:06,433 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 18:54:07,434 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:54:08,305 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:54:12,435 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:54:13,306 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:54:17,436 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:54:18,307 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:54:22,437 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:54:23,546 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:54:27,438 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:54:28,547 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:54:32,439 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:54:33,548 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:54:36,434 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 18:54:38,435 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:54:38,549 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:54:43,436 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:54:43,550 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:54:48,437 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:54:48,551 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:54:53,438 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:54:53,552 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:54:58,439 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:54:58,553 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:55:03,440 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:55:03,671 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:55:06,435 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 18:55:08,673 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:55:09,436 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:55:13,673 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:55:14,437 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:55:18,674 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:55:19,438 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:55:23,675 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:55:24,439 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:55:28,676 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:55:29,440 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:55:33,677 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:55:34,441 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:55:34,972 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: partial_history 2024-06-03 18:55:34,973 DEBUG SenderThread:322 [sender.py:send():378] send: history 2024-06-03 18:55:34,974 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: summary_record 2024-06-03 18:55:34,974 INFO SenderThread:322 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-06-03 18:55:35,045 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/wandb-summary.json 2024-06-03 18:55:36,436 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 18:55:38,678 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:55:40,437 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:55:43,821 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:55:45,438 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:55:48,822 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:55:50,439 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:55:53,823 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:55:55,440 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:55:58,824 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:56:00,441 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:56:04,086 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:56:05,442 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:56:06,437 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 18:56:09,087 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:56:11,438 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:56:14,088 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:56:16,439 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:56:19,089 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:56:21,635 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:56:23,064 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/output.log 2024-06-03 18:56:24,090 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:56:26,658 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:56:29,509 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:56:31,659 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:56:34,510 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:56:36,438 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 18:56:37,439 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:56:39,511 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:56:42,439 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:56:44,512 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:56:47,440 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:56:49,930 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:56:52,441 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:56:54,931 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:56:57,442 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:56:59,932 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:57:02,443 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:57:04,933 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:57:06,438 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 18:57:08,440 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:57:10,276 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:57:13,441 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:57:15,277 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:57:18,442 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:57:20,278 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:57:23,443 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:57:25,280 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:57:28,443 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:57:30,281 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:57:33,444 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:57:35,282 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:57:36,439 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 18:57:39,440 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:57:40,283 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:57:44,441 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:57:45,284 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:57:49,443 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:57:50,286 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:57:54,443 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:57:55,287 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:57:59,444 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:58:00,288 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:58:04,445 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:58:05,289 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:58:06,440 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 18:58:10,391 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:58:10,391 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:58:10,441 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:58:15,392 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:58:15,443 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:58:20,393 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:58:20,444 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:58:25,395 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:58:25,444 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:58:30,396 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:58:30,445 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:58:35,397 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:58:35,446 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:58:36,441 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 18:58:40,398 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:58:41,442 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:58:44,385 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: partial_history 2024-06-03 18:58:44,386 DEBUG SenderThread:322 [sender.py:send():378] send: history 2024-06-03 18:58:44,387 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: summary_record 2024-06-03 18:58:44,387 INFO SenderThread:322 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-06-03 18:58:45,117 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/wandb-summary.json 2024-06-03 18:58:45,399 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:58:47,388 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:58:50,400 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:58:52,389 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:58:55,401 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:58:57,390 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:59:00,402 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:59:02,391 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:59:05,403 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:59:06,442 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 18:59:07,443 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:59:10,404 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:59:12,443 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:59:15,504 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:59:17,444 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:59:20,505 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:59:22,445 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:59:25,507 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:59:27,446 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:59:30,508 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:59:32,447 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:59:35,518 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:59:36,443 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 18:59:38,444 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:59:40,519 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:59:43,444 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:59:45,520 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:59:48,445 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:59:50,521 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:59:51,141 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/output.log 2024-06-03 18:59:53,142 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/output.log 2024-06-03 18:59:54,030 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 18:59:55,522 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 18:59:59,031 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:00:00,859 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:00:04,032 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:00:05,860 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:00:06,444 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:00:09,445 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:00:10,861 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:00:14,446 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:00:15,862 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:00:19,446 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:00:21,187 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:00:24,447 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:00:26,188 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:00:29,448 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:00:31,189 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:00:34,449 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:00:36,190 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:00:36,445 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:00:40,446 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:00:41,191 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:00:45,447 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:00:46,192 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:00:50,447 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:00:51,193 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:00:55,448 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:00:56,194 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:01:00,449 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:01:01,236 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:01:05,449 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:01:06,237 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:01:06,445 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:01:11,238 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:01:11,447 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:01:16,239 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:01:16,447 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:01:21,448 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:01:21,585 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:01:26,449 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:01:26,587 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:01:31,449 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:01:31,588 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:01:36,446 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:01:36,589 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:01:37,447 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:01:41,648 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:01:42,448 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:01:46,649 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:01:47,449 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:01:51,650 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:01:52,450 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:01:53,083 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: partial_history 2024-06-03 19:01:53,084 DEBUG SenderThread:322 [sender.py:send():378] send: history 2024-06-03 19:01:53,085 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: summary_record 2024-06-03 19:01:53,085 INFO SenderThread:322 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-06-03 19:01:53,186 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/wandb-summary.json 2024-06-03 19:01:56,651 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:01:58,086 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:02:01,935 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:02:03,087 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:02:06,447 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:02:06,936 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:02:08,448 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:02:11,937 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:02:13,449 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:02:16,938 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:02:18,450 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:02:22,112 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:02:23,451 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:02:27,372 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:02:28,452 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:02:32,373 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:02:33,453 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:02:36,448 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:02:37,375 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:02:39,449 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:02:42,729 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:02:44,450 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:02:47,730 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:02:49,451 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:02:52,731 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:02:54,452 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:02:57,732 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:02:59,453 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:03:02,732 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:03:04,454 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:03:06,453 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:03:07,734 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:03:10,454 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:03:12,735 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:03:15,455 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:03:17,736 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:03:20,541 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:03:22,219 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/output.log 2024-06-03 19:03:22,737 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:03:25,560 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:03:27,738 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:03:30,561 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:03:32,739 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:03:35,562 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:03:36,450 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:03:37,740 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:03:41,451 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:03:42,741 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:03:46,452 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:03:47,793 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:03:51,453 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:03:52,795 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:03:56,454 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:03:57,796 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:04:01,455 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:04:02,797 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:04:06,452 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:04:07,453 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:04:07,864 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:04:12,454 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:04:12,865 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:04:17,455 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:04:17,866 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:04:22,456 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:04:22,867 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:04:27,457 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:04:27,868 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:04:32,458 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:04:32,869 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:04:36,452 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:04:37,870 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:04:38,453 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:04:42,871 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:04:43,454 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:04:47,872 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:04:48,454 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:04:52,873 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:04:53,455 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:04:57,874 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:04:58,456 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:05:02,156 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: partial_history 2024-06-03 19:05:02,158 DEBUG SenderThread:322 [sender.py:send():378] send: history 2024-06-03 19:05:02,158 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: summary_record 2024-06-03 19:05:02,159 INFO SenderThread:322 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-06-03 19:05:02,255 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/wandb-summary.json 2024-06-03 19:05:02,875 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:05:04,159 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:05:06,452 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:05:07,876 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:05:09,453 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:05:12,960 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:05:14,455 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:05:17,961 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:05:19,456 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:05:22,962 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:05:24,457 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:05:27,963 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:05:29,457 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:05:33,170 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:05:34,459 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:05:36,453 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:05:38,171 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:05:40,455 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:05:43,172 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:05:45,455 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:05:48,174 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:05:50,456 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:05:53,510 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:05:55,458 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:05:58,511 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:06:00,458 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:06:03,512 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:06:05,459 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:06:06,454 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:06:08,513 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:06:11,455 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:06:13,760 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:06:16,456 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:06:18,761 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:06:21,457 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:06:23,762 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:06:26,459 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:06:28,763 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:06:31,459 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:06:33,915 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:06:36,455 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:06:37,456 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:06:38,916 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:06:42,457 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:06:43,917 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:06:47,458 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:06:48,918 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:06:52,295 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/output.log 2024-06-03 19:06:52,885 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:06:53,919 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:06:57,886 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:06:58,920 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:07:02,887 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:07:03,921 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:07:06,456 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:07:08,457 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:07:08,922 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:07:13,458 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:07:13,923 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:07:18,458 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:07:19,092 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:07:23,459 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:07:24,093 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:07:28,461 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:07:29,094 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:07:33,462 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:07:34,095 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:07:36,457 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:07:39,096 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:07:39,458 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:07:44,097 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:07:44,459 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:07:49,098 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:07:49,460 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:07:54,099 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:07:54,461 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:07:59,334 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:07:59,462 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:08:04,335 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:08:04,463 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:08:06,458 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:08:09,337 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:08:10,460 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:08:12,043 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: partial_history 2024-06-03 19:08:12,045 DEBUG SenderThread:322 [sender.py:send():378] send: history 2024-06-03 19:08:12,045 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: summary_record 2024-06-03 19:08:12,046 INFO SenderThread:322 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-06-03 19:08:12,324 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/wandb-summary.json 2024-06-03 19:08:14,337 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:08:16,047 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:08:19,338 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:08:21,047 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:08:24,339 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:08:26,048 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:08:29,340 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:08:31,049 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:08:34,341 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:08:36,050 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:08:36,459 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:08:39,342 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:08:41,460 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:08:44,343 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:08:46,461 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:08:49,344 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:08:51,462 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:08:54,345 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:08:56,463 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:08:59,345 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:09:01,464 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:09:04,346 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:09:06,460 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:09:07,461 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:09:09,347 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:09:12,461 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:09:14,348 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:09:17,462 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:09:19,369 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:09:22,464 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:09:24,370 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:09:27,464 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:09:29,371 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:09:32,465 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:09:34,372 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:09:36,461 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:09:38,462 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:09:39,575 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:09:43,463 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:09:44,577 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:09:48,463 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:09:49,578 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:09:53,464 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:09:54,579 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:09:58,465 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:09:59,914 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:10:03,466 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:10:04,915 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:10:06,462 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:10:09,463 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:10:09,916 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:10:14,463 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:10:14,917 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:10:19,918 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:10:20,325 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:10:20,370 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/output.log 2024-06-03 19:10:24,919 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:10:25,326 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:10:30,210 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:10:30,327 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:10:35,211 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:10:35,328 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:10:36,463 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:10:40,212 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:10:40,464 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:10:45,213 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:10:45,465 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:10:50,393 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:10:50,465 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:10:55,394 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:10:55,466 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:11:00,395 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:11:00,467 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:11:05,396 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:11:05,468 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:11:06,463 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:11:10,537 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:11:11,465 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:11:15,537 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:11:16,466 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:11:20,149 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: partial_history 2024-06-03 19:11:20,151 DEBUG SenderThread:322 [sender.py:send():378] send: history 2024-06-03 19:11:20,151 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: summary_record 2024-06-03 19:11:20,151 INFO SenderThread:322 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-06-03 19:11:20,392 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/wandb-summary.json 2024-06-03 19:11:20,538 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:11:22,152 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:11:25,539 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:11:27,153 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:11:30,540 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:11:32,154 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:11:35,541 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:11:36,465 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:11:37,465 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:11:40,542 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:11:42,466 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:11:45,542 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:11:47,467 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:11:50,751 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:11:52,468 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:11:55,752 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:11:57,469 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:12:00,754 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:12:02,470 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:12:05,755 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:12:06,466 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:12:08,467 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:12:10,945 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:12:13,467 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:12:15,946 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:12:18,468 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:12:20,947 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:12:23,469 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:12:25,948 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:12:28,470 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:12:30,949 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:12:33,471 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:12:35,950 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:12:36,467 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:12:39,468 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:12:40,951 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:12:44,468 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:12:45,952 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:12:49,469 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:12:50,953 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:12:54,470 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:12:55,954 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:12:59,471 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:13:00,955 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:13:04,472 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:13:05,957 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:13:06,467 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:13:10,469 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:13:10,958 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:13:15,469 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:13:15,959 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:13:20,470 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:13:20,960 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:13:25,471 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:13:25,961 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:13:30,472 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:13:31,372 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:13:35,472 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:13:36,373 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:13:36,468 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:13:41,374 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:13:41,470 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:13:46,375 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:13:46,470 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:13:50,448 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/output.log 2024-06-03 19:13:51,376 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:13:51,820 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:13:56,624 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:13:56,821 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:14:01,625 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:14:01,822 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:14:06,469 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:14:06,627 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:14:07,470 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:14:11,628 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:14:12,471 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:14:16,849 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:14:17,472 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:14:21,850 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:14:22,473 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:14:26,851 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:14:27,473 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:14:29,344 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: partial_history 2024-06-03 19:14:29,345 DEBUG SenderThread:322 [sender.py:send():378] send: history 2024-06-03 19:14:29,345 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: summary_record 2024-06-03 19:14:29,346 INFO SenderThread:322 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-06-03 19:14:29,463 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/wandb-summary.json 2024-06-03 19:14:31,853 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:14:33,347 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:14:36,470 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:14:37,119 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:14:38,471 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:14:42,174 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:14:43,472 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:14:47,176 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:14:48,472 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:14:52,177 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:14:53,473 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:14:57,345 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:14:58,474 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:15:02,346 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:15:03,475 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:15:06,471 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:15:07,347 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:15:09,472 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:15:12,348 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:15:14,473 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:15:17,349 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:15:19,474 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:15:22,350 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:15:24,475 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:15:27,351 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:15:29,476 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:15:32,354 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:15:34,477 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:15:36,472 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:15:37,355 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:15:40,474 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:15:42,356 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:15:45,475 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:15:47,358 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:15:50,475 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:15:52,359 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:15:55,476 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:15:57,770 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:16:00,478 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:16:02,771 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:16:05,478 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:16:06,473 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:16:07,772 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:16:11,475 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:16:12,773 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:16:16,476 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:16:17,774 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:16:21,477 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:16:22,775 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:16:26,477 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:16:27,776 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:16:31,478 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:16:32,777 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:16:36,474 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:16:37,475 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:16:37,778 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:16:42,476 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:16:42,778 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:16:47,477 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:16:47,779 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:16:52,478 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:16:52,780 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:16:57,479 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:16:57,781 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:17:02,480 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:17:02,782 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:17:06,475 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:17:07,783 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:17:08,476 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:17:12,784 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:17:13,477 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:17:17,785 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:17:18,526 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/output.log 2024-06-03 19:17:19,184 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:17:20,527 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/output.log 2024-06-03 19:17:22,786 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:17:24,185 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:17:27,985 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:17:29,186 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:17:32,986 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:17:34,187 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:17:36,480 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:17:37,987 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:17:38,187 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: partial_history 2024-06-03 19:17:38,188 DEBUG SenderThread:322 [sender.py:send():378] send: history 2024-06-03 19:17:38,188 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: summary_record 2024-06-03 19:17:38,189 INFO SenderThread:322 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-06-03 19:17:38,534 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/wandb-summary.json 2024-06-03 19:17:39,190 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:17:42,988 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:17:44,190 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:17:48,008 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:17:49,191 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:17:53,009 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:17:54,192 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:17:58,010 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:17:59,193 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:18:03,011 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:18:04,194 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:18:06,477 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:18:08,120 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:18:09,478 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:18:13,121 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:18:14,479 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:18:18,122 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:18:19,480 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:18:23,124 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:18:24,481 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:18:28,337 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:18:29,482 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:18:33,339 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:18:34,483 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:18:36,478 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:18:38,340 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:18:40,480 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:18:43,341 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:18:45,481 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:18:48,342 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:18:50,482 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:18:53,343 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:18:55,483 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:18:58,344 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:19:00,484 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:19:03,345 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:19:05,485 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:19:06,479 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:19:08,345 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:19:11,480 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:19:13,347 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:19:16,481 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:19:18,347 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:19:21,482 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:19:23,348 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:19:26,483 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:19:28,601 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:19:31,484 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:19:33,601 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:19:36,480 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:19:37,480 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:19:38,602 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:19:42,481 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:19:43,603 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:19:47,482 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:19:48,604 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:19:52,483 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:19:53,605 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:19:57,485 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:19:58,606 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:20:02,486 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:20:03,607 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:20:06,480 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:20:08,481 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:20:08,625 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:20:08,626 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:20:13,482 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:20:13,627 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:20:18,483 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:20:18,628 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:20:23,484 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:20:23,629 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:20:28,485 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:20:28,709 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:20:33,486 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:20:33,709 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:20:36,481 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:20:38,710 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:20:39,482 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:20:43,711 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:20:44,483 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:20:47,385 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: partial_history 2024-06-03 19:20:47,386 DEBUG SenderThread:322 [sender.py:send():378] send: history 2024-06-03 19:20:47,386 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: summary_record 2024-06-03 19:20:47,387 INFO SenderThread:322 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-06-03 19:20:47,602 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/wandb-summary.json 2024-06-03 19:20:48,602 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/output.log 2024-06-03 19:20:48,712 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:20:49,731 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:20:53,713 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:20:54,732 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:20:58,714 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:20:59,733 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:21:03,715 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:21:04,734 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:21:06,482 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:21:08,716 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:21:10,484 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:21:13,717 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:21:15,484 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:21:18,831 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:21:20,485 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:21:23,832 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:21:25,486 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:21:28,834 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:21:30,487 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:21:33,835 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:21:35,488 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:21:36,483 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:21:39,241 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:21:41,485 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:21:44,243 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:21:46,486 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:21:49,243 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:21:51,487 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:21:54,244 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:21:56,488 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:21:59,682 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:22:01,489 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:22:04,683 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:22:06,484 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:22:07,485 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:22:09,684 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:22:12,486 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:22:14,685 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:22:17,487 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:22:19,951 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:22:22,488 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:22:24,952 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:22:27,489 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:22:29,953 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:22:32,490 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:22:34,953 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:22:36,485 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:22:38,486 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:22:39,995 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:22:43,487 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:22:44,996 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:22:48,488 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:22:49,997 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:22:53,489 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:22:54,998 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:22:58,490 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:22:59,999 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:23:03,490 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:23:05,000 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:23:06,486 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:23:09,487 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:23:10,001 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:23:14,488 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:23:15,002 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:23:19,489 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:23:20,003 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:23:24,490 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:23:25,004 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:23:29,491 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:23:30,005 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:23:34,492 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:23:35,007 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:23:36,487 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:23:40,288 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:23:40,488 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:23:45,289 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:23:45,489 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:23:50,290 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:23:50,490 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:23:55,291 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:23:55,491 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:24:00,292 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:24:00,492 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:24:05,293 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:24:05,493 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:24:06,488 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:24:10,294 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:24:11,223 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: partial_history 2024-06-03 19:24:11,224 DEBUG SenderThread:322 [sender.py:send():378] send: history 2024-06-03 19:24:11,225 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: summary_record 2024-06-03 19:24:11,225 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:24:11,225 INFO SenderThread:322 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-06-03 19:24:11,676 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/wandb-summary.json 2024-06-03 19:24:15,295 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:24:16,226 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:24:18,679 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/output.log 2024-06-03 19:24:20,296 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:24:21,230 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:24:25,297 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:24:26,230 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:24:30,298 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:24:31,231 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:24:35,299 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:24:36,232 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:24:36,489 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:24:40,301 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:24:41,490 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:24:45,431 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:24:46,491 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:24:50,432 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:24:51,492 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:24:55,433 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:24:56,493 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:25:00,434 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:25:01,494 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:25:05,621 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:25:06,490 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:25:07,491 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:25:10,622 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:25:12,492 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:25:15,623 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:25:17,493 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:25:20,624 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:25:22,493 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:25:25,625 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:25:27,494 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:25:30,626 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:25:32,496 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:25:35,627 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:25:36,491 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:25:38,492 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:25:40,628 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:25:43,493 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:25:45,629 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:25:48,494 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:25:50,630 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:25:53,495 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:25:55,631 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:25:58,496 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:26:00,632 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:26:03,497 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:26:05,632 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:26:06,492 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:26:09,493 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:26:10,634 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:26:14,494 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:26:15,635 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:26:19,495 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:26:20,635 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:26:24,495 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:26:25,773 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:26:29,497 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:26:30,774 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:26:34,498 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:26:35,775 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:26:36,493 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:26:40,494 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:26:40,776 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:26:45,495 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:26:45,941 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:26:50,496 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:26:50,942 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:26:55,496 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:26:55,943 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:27:00,497 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:27:00,944 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:27:05,498 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:27:05,945 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:27:06,493 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:27:10,946 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:27:11,494 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:27:15,947 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:27:16,495 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:27:20,110 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: partial_history 2024-06-03 19:27:20,111 DEBUG SenderThread:322 [sender.py:send():378] send: history 2024-06-03 19:27:20,112 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: summary_record 2024-06-03 19:27:20,112 INFO SenderThread:322 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-06-03 19:27:20,746 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/wandb-summary.json 2024-06-03 19:27:20,948 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:27:22,113 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:27:25,949 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:27:27,114 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:27:30,988 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:27:32,114 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:27:35,992 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:27:36,494 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:27:37,495 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:27:40,993 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:27:42,496 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:27:45,994 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:27:47,895 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:27:49,756 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/output.log 2024-06-03 19:27:50,995 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:27:52,895 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:27:55,996 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:27:57,896 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:28:00,997 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:28:02,897 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:28:05,998 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:28:06,495 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:28:08,496 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:28:10,999 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:28:13,497 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:28:16,323 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:28:18,498 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:28:21,324 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:28:23,499 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:28:26,325 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:28:28,500 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:28:31,326 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:28:33,501 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:28:36,327 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:28:36,496 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:28:39,497 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:28:41,328 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:28:44,498 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:28:46,329 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:28:49,499 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:28:51,330 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:28:54,500 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:28:56,331 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:28:59,501 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:29:01,332 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:29:04,502 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:29:06,333 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:29:06,497 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:29:10,498 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:29:11,334 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:29:15,499 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:29:16,335 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:29:20,500 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:29:21,335 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:29:25,501 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:29:26,336 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:29:30,502 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:29:31,337 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:29:35,502 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:29:36,498 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:29:36,677 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:29:41,499 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:29:41,677 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:29:46,500 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:29:46,678 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:29:51,501 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:29:51,679 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:29:56,502 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:29:57,105 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:30:01,503 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:30:02,106 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:30:06,499 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:30:07,107 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:30:07,500 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:30:12,108 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:30:12,501 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:30:17,289 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:30:17,502 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:30:22,290 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:30:22,503 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:30:27,291 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:30:27,504 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:30:29,732 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: partial_history 2024-06-03 19:30:29,733 DEBUG SenderThread:322 [sender.py:send():378] send: history 2024-06-03 19:30:29,733 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: summary_record 2024-06-03 19:30:29,733 INFO SenderThread:322 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-06-03 19:30:29,814 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/wandb-summary.json 2024-06-03 19:30:32,292 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:30:32,734 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:30:36,500 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:30:37,554 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:30:38,501 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:30:42,555 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:30:43,502 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:30:47,556 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:30:48,503 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:30:52,558 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:30:53,504 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:30:57,559 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:30:58,505 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:31:02,559 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:31:03,506 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:31:06,501 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:31:07,561 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:31:09,502 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:31:12,562 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:31:14,502 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:31:17,563 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:31:17,832 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/output.log 2024-06-03 19:31:20,494 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:31:22,564 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:31:25,495 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:31:27,565 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:31:30,496 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:31:32,566 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:31:35,497 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:31:36,501 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:31:37,567 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:31:40,502 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:31:42,568 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:31:45,504 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:31:47,675 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:31:50,505 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:31:52,676 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:31:55,506 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:31:57,677 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:32:00,507 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:32:02,678 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:32:05,508 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:32:06,502 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:32:07,856 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:32:11,504 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:32:12,857 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:32:16,504 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:32:17,858 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:32:21,505 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:32:22,858 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:32:26,506 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:32:27,859 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:32:31,507 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:32:32,860 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:32:36,503 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:32:37,504 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:32:37,861 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:32:42,505 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:32:42,862 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:32:47,506 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:32:48,102 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:32:52,506 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:32:53,103 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:32:57,507 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:32:58,104 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:33:02,508 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:33:03,105 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:33:06,508 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:33:08,106 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:33:08,509 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:33:13,106 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:33:13,510 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:33:18,108 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:33:18,510 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:33:23,109 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:33:23,511 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:33:28,110 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:33:28,512 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:33:33,111 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:33:33,513 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:33:36,505 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:33:38,112 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:33:39,417 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: partial_history 2024-06-03 19:33:39,418 DEBUG SenderThread:322 [sender.py:send():378] send: history 2024-06-03 19:33:39,418 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: summary_record 2024-06-03 19:33:39,418 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:33:39,421 INFO SenderThread:322 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-06-03 19:33:39,882 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/wandb-summary.json 2024-06-03 19:33:43,113 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:33:44,422 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:33:48,291 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:33:49,422 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:33:53,292 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:33:54,424 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:33:58,293 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:33:59,425 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:34:03,294 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:34:04,426 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:34:06,506 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:34:08,408 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:34:09,507 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:34:13,409 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:34:14,508 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:34:18,410 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:34:19,509 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:34:23,411 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:34:24,510 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:34:28,655 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:34:29,510 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:34:33,656 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:34:34,511 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:34:36,508 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:34:38,657 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:34:40,509 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:34:43,658 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:34:45,510 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:34:47,907 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/output.log 2024-06-03 19:34:48,659 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:34:51,023 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:34:53,897 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:34:56,024 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:34:58,898 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:35:01,025 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:35:03,899 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:35:06,026 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:35:06,507 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:35:08,900 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:35:11,509 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:35:14,163 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:35:16,510 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:35:19,164 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:35:21,511 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:35:24,166 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:35:26,511 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:35:29,167 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:35:31,512 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:35:34,431 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:35:36,509 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:35:37,509 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:35:39,432 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:35:42,510 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:35:44,433 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:35:47,511 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:35:49,434 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:35:52,512 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:35:54,524 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:35:57,514 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:35:59,525 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:36:02,514 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:36:04,526 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:36:06,509 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:36:08,510 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:36:09,527 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:36:13,511 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:36:14,877 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:36:18,512 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:36:19,878 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:36:23,513 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:36:24,879 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:36:28,514 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:36:29,880 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:36:33,515 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:36:35,249 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:36:36,510 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:36:39,511 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:36:40,250 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:36:44,512 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:36:45,251 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:36:47,570 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: partial_history 2024-06-03 19:36:47,571 DEBUG SenderThread:322 [sender.py:send():378] send: history 2024-06-03 19:36:47,572 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: summary_record 2024-06-03 19:36:47,572 INFO SenderThread:322 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-06-03 19:36:47,951 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/wandb-summary.json 2024-06-03 19:36:49,572 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:36:50,252 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:36:54,573 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:36:55,435 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:36:59,574 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:37:00,436 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:37:04,575 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:37:05,437 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:37:06,511 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:37:10,438 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:37:10,512 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:37:15,439 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:37:15,513 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:37:20,440 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:37:20,514 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:37:25,441 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:37:25,515 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:37:30,442 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:37:30,516 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:37:35,443 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:37:35,517 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:37:36,512 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:37:40,444 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:37:41,513 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:37:45,445 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:37:46,514 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:37:50,446 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:37:51,515 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:37:55,554 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:37:56,516 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:38:00,556 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:38:01,517 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:38:05,557 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:38:06,513 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:38:07,514 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:38:10,558 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:38:12,514 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:38:15,558 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:38:17,560 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:38:17,982 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/output.log 2024-06-03 19:38:20,560 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:38:22,561 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:38:25,561 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:38:27,562 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:38:30,562 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:38:32,562 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:38:35,563 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:38:36,514 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:38:38,515 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:38:40,564 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:38:43,516 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:38:45,663 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:38:48,517 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:38:50,664 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:38:53,518 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:38:55,665 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:38:58,519 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:39:00,666 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:39:03,520 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:39:05,925 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:39:06,515 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:39:09,516 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:39:10,926 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:39:14,517 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:39:15,927 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:39:19,518 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:39:20,928 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:39:24,518 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:39:25,929 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:39:29,519 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:39:30,930 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:39:34,521 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:39:35,931 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:39:36,515 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:39:40,516 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:39:40,931 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:39:45,517 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:39:46,022 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:39:50,518 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:39:51,023 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:39:55,520 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:39:56,024 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:39:57,265 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: partial_history 2024-06-03 19:39:57,266 DEBUG SenderThread:322 [sender.py:send():378] send: history 2024-06-03 19:39:57,266 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: summary_record 2024-06-03 19:39:57,267 INFO SenderThread:322 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-06-03 19:39:58,020 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/wandb-summary.json 2024-06-03 19:40:01,025 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:40:01,268 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:40:06,105 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:40:06,268 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:40:06,516 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:40:11,106 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:40:11,517 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:40:16,107 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:40:16,518 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:40:21,108 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:40:21,520 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:40:26,521 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:40:26,522 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:40:31,521 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:40:31,523 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:40:36,517 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:40:36,523 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:40:37,518 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:40:41,524 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:40:42,519 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:40:46,611 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:40:47,520 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:40:51,612 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:40:52,521 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:40:56,613 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:40:57,522 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:41:01,614 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:41:02,523 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:41:06,518 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:41:07,044 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:41:08,519 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:41:12,045 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:41:13,520 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:41:17,046 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:41:18,521 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:41:22,047 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:41:23,522 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:41:27,486 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:41:28,523 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:41:32,487 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:41:33,524 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:41:36,519 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:41:37,488 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:41:39,520 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:41:42,489 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:41:44,521 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:41:46,059 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/output.log 2024-06-03 19:41:47,490 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:41:50,148 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:41:52,490 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:41:55,149 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:41:57,491 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:42:00,150 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:42:02,492 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:42:05,150 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:42:06,520 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:42:07,494 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:42:10,521 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:42:12,495 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:42:15,522 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:42:17,496 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:42:20,523 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:42:22,497 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:42:25,524 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:42:27,498 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:42:30,524 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:42:32,499 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:42:35,525 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:42:36,521 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:42:37,500 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:42:41,522 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:42:42,501 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:42:46,523 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:42:47,502 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:42:51,524 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:42:52,667 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:42:56,524 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:42:57,668 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:43:01,526 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:43:02,669 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:43:06,354 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: partial_history 2024-06-03 19:43:06,356 DEBUG SenderThread:322 [sender.py:send():378] send: history 2024-06-03 19:43:06,356 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: summary_record 2024-06-03 19:43:06,356 INFO SenderThread:322 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-06-03 19:43:06,522 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:43:07,088 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/wandb-summary.json 2024-06-03 19:43:07,522 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:43:07,670 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:43:12,524 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:43:12,671 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:43:17,524 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:43:17,672 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:43:22,525 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:43:22,673 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:43:27,526 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:43:27,674 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:43:32,527 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:43:32,675 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:43:36,522 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:43:37,676 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:43:38,523 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:43:42,677 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:43:43,524 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:43:47,678 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:43:48,525 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:43:52,679 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:43:53,526 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:43:57,761 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:43:58,527 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:44:02,762 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:44:03,528 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:44:06,523 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:44:07,763 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:44:09,524 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:44:12,764 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:44:14,525 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:44:17,951 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:44:19,526 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:44:22,952 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:44:24,527 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:44:27,953 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:44:29,528 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:44:32,954 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:44:34,529 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:44:36,524 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:44:38,157 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:44:40,525 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:44:43,159 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:44:45,526 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:44:48,160 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:44:50,527 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:44:53,160 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:44:55,528 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:44:58,502 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:45:00,529 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:45:03,503 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:45:05,530 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:45:06,525 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:45:08,504 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:45:11,526 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:45:13,506 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:45:16,136 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/output.log 2024-06-03 19:45:16,559 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:45:18,507 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:45:21,560 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:45:23,508 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:45:26,562 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:45:28,509 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:45:31,563 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:45:33,510 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:45:36,526 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:45:37,527 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:45:38,511 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:45:42,527 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:45:43,591 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:45:47,528 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:45:48,592 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:45:52,529 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:45:53,593 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:45:57,530 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:45:58,594 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:46:02,531 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:46:04,005 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:46:06,527 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:46:08,528 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:46:09,006 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:46:13,529 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:46:14,007 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:46:15,700 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: partial_history 2024-06-03 19:46:15,702 DEBUG SenderThread:322 [sender.py:send():378] send: history 2024-06-03 19:46:15,702 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: summary_record 2024-06-03 19:46:15,702 INFO SenderThread:322 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-06-03 19:46:16,158 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/wandb-summary.json 2024-06-03 19:46:18,703 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:46:19,008 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:46:23,704 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:46:24,009 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:46:28,705 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:46:29,010 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:46:33,706 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:46:34,011 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:46:36,528 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:46:39,012 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:46:39,529 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:46:44,013 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:46:44,530 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:46:49,014 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:46:49,531 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:46:54,015 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:46:54,531 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:46:59,016 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:46:59,532 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:47:04,032 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:47:04,533 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:47:06,529 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:47:09,033 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:47:10,530 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:47:14,034 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:47:15,531 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:47:19,035 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:47:20,532 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:47:24,036 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:47:25,532 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:47:29,037 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:47:30,533 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:47:34,038 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:47:35,534 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:47:36,530 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:47:39,039 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:47:41,531 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:47:44,055 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:47:46,532 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:47:49,057 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:47:51,533 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:47:54,058 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:47:56,534 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:47:59,059 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:48:01,535 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:48:04,419 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:48:06,535 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:48:06,536 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:48:09,420 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:48:11,536 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:48:14,421 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:48:16,537 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:48:19,422 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:48:21,538 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:48:24,774 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:48:26,538 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:48:29,775 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:48:31,539 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:48:34,776 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:48:36,531 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:48:37,532 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:48:39,777 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:48:42,533 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:48:44,778 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:48:46,214 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/output.log 2024-06-03 19:48:48,389 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:48:49,779 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:48:53,390 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:48:54,780 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:48:58,391 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:48:59,781 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:49:03,392 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:49:04,782 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:49:06,532 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:49:08,533 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:49:09,783 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:49:13,534 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:49:14,784 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:49:18,535 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:49:19,785 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:49:23,536 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:49:24,786 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:49:24,798 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: partial_history 2024-06-03 19:49:24,799 DEBUG SenderThread:322 [sender.py:send():378] send: history 2024-06-03 19:49:24,799 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: summary_record 2024-06-03 19:49:24,801 INFO SenderThread:322 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-06-03 19:49:25,228 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/wandb-summary.json 2024-06-03 19:49:28,802 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:49:29,787 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:49:33,803 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:49:34,789 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:49:36,533 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:49:39,534 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:49:39,790 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:49:44,535 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:49:44,791 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:49:49,536 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:49:49,791 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:49:54,537 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:49:54,907 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:49:59,538 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:50:00,191 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:50:04,538 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:50:05,192 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:50:06,535 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:50:10,193 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:50:10,536 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:50:15,437 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:50:15,538 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:50:20,438 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:50:20,539 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:50:25,439 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:50:25,540 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:50:30,440 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:50:30,540 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:50:35,541 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:50:35,558 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:50:36,535 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:50:40,559 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:50:41,536 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:50:45,560 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:50:46,537 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:50:50,561 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:50:51,538 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:50:55,762 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:50:56,539 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:51:00,763 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:51:01,540 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:51:05,764 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:51:06,536 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:51:07,536 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:51:10,765 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:51:12,537 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:51:16,137 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:51:17,539 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:51:21,139 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:51:22,540 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:51:26,139 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:51:27,541 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:51:31,141 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:51:32,541 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:51:36,142 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:51:36,536 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:51:38,537 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:51:41,143 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:51:43,538 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:51:46,144 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:51:48,539 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:51:51,145 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:51:53,540 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:51:56,179 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:51:56,179 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:51:58,541 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:52:01,180 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:52:03,542 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:52:06,181 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:52:06,537 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:52:09,538 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:52:11,182 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:52:14,694 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:52:15,289 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/output.log 2024-06-03 19:52:16,183 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:52:19,694 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:52:21,490 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:52:24,695 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:52:26,490 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:52:29,696 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:52:31,492 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:52:33,654 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: partial_history 2024-06-03 19:52:33,655 DEBUG SenderThread:322 [sender.py:send():378] send: history 2024-06-03 19:52:33,655 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: summary_record 2024-06-03 19:52:33,656 INFO SenderThread:322 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-06-03 19:52:34,296 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/wandb-summary.json 2024-06-03 19:52:35,656 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:52:36,493 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:52:36,538 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:52:41,518 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:52:41,539 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:52:46,519 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:52:46,540 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:52:51,520 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:52:51,541 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:52:56,521 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:52:56,542 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:53:01,543 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:53:01,929 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:53:06,539 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:53:06,930 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:53:07,540 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:53:11,931 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:53:12,541 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:53:16,932 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:53:17,542 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:53:21,933 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:53:22,543 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:53:26,935 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:53:27,544 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:53:31,936 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:53:32,545 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:53:36,540 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:53:36,937 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:53:38,541 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:53:42,002 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:53:43,542 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:53:47,003 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:53:48,543 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:53:52,004 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:53:53,544 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:53:57,005 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:53:58,544 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:54:02,188 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:54:03,545 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:54:06,541 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:54:07,190 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:54:09,542 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:54:12,191 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:54:14,543 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:54:17,192 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:54:19,544 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:54:22,550 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:54:24,545 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:54:27,551 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:54:29,546 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:54:32,552 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:54:34,547 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:54:36,542 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:54:37,553 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:54:40,543 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:54:42,582 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:54:45,544 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:54:47,583 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:54:50,545 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:54:52,585 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:54:55,546 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:54:57,586 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:55:00,547 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:55:02,587 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:55:05,547 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:55:06,543 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:55:07,588 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:55:11,544 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:55:12,588 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:55:16,545 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:55:17,589 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:55:21,546 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:55:22,590 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:55:26,546 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:55:27,591 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:55:31,547 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:55:32,592 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:55:36,543 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:55:37,544 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:55:37,593 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:55:42,545 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:55:42,594 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:55:42,928 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: partial_history 2024-06-03 19:55:42,929 DEBUG SenderThread:322 [sender.py:send():378] send: history 2024-06-03 19:55:42,929 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: summary_record 2024-06-03 19:55:42,929 INFO SenderThread:322 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-06-03 19:55:43,254 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: summary_record 2024-06-03 19:55:43,255 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: partial_history 2024-06-03 19:55:43,256 INFO SenderThread:322 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-06-03 19:55:43,257 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: summary_record 2024-06-03 19:55:43,257 INFO SenderThread:322 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-06-03 19:55:43,257 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: summary_record 2024-06-03 19:55:43,258 INFO SenderThread:322 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-06-03 19:55:43,258 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: summary_record 2024-06-03 19:55:43,258 INFO SenderThread:322 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-06-03 19:55:43,259 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: summary_record 2024-06-03 19:55:43,259 INFO SenderThread:322 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-06-03 19:55:43,259 DEBUG SenderThread:322 [sender.py:send():378] send: history 2024-06-03 19:55:43,259 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: summary_record 2024-06-03 19:55:43,260 INFO SenderThread:322 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-06-03 19:55:43,363 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/wandb-summary.json 2024-06-03 19:55:43,602 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: pause 2024-06-03 19:55:43,602 INFO HandlerThread:322 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-06-03 19:55:43,602 INFO HandlerThread:322 [system_monitor.py:finish():203] Stopping system monitor 2024-06-03 19:55:43,603 DEBUG SystemMonitor:322 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-06-03 19:55:43,603 INFO HandlerThread:322 [interfaces.py:finish():200] Joined cpu monitor 2024-06-03 19:55:43,603 DEBUG SystemMonitor:322 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-06-03 19:55:43,604 INFO HandlerThread:322 [interfaces.py:finish():200] Joined disk monitor 2024-06-03 19:55:43,616 INFO HandlerThread:322 [interfaces.py:finish():200] Joined gpu monitor 2024-06-03 19:55:43,616 INFO HandlerThread:322 [interfaces.py:finish():200] Joined memory monitor 2024-06-03 19:55:43,616 INFO HandlerThread:322 [interfaces.py:finish():200] Joined network monitor 2024-06-03 19:55:43,617 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:55:45,364 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/output.log 2024-06-03 19:55:47,595 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:55:47,617 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:55:52,596 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:55:52,618 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:55:57,598 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:55:57,619 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:56:02,599 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:56:02,620 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:56:07,600 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:56:07,621 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:56:12,601 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:56:12,622 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:56:17,602 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:56:17,623 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:56:22,603 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:56:22,624 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:56:27,604 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:56:27,625 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:56:32,605 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:56:32,626 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:56:37,606 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:56:37,627 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:56:42,607 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:56:42,627 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:56:47,608 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:56:47,628 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:56:52,610 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:56:52,629 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:56:53,517 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: resume 2024-06-03 19:56:53,517 INFO HandlerThread:322 [handler.py:handle_request_resume():715] starting system metrics thread 2024-06-03 19:56:53,517 INFO HandlerThread:322 [system_monitor.py:start():194] Starting system monitor 2024-06-03 19:56:53,517 INFO SystemMonitor:322 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-06-03 19:56:53,518 INFO SystemMonitor:322 [interfaces.py:start():188] Started cpu monitoring 2024-06-03 19:56:53,519 INFO SystemMonitor:322 [interfaces.py:start():188] Started disk monitoring 2024-06-03 19:56:53,519 INFO SystemMonitor:322 [interfaces.py:start():188] Started gpu monitoring 2024-06-03 19:56:53,520 INFO SystemMonitor:322 [interfaces.py:start():188] Started memory monitoring 2024-06-03 19:56:53,521 INFO SystemMonitor:322 [interfaces.py:start():188] Started network monitoring 2024-06-03 19:56:55,309 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: pause 2024-06-03 19:56:55,310 INFO HandlerThread:322 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-06-03 19:56:55,310 INFO HandlerThread:322 [system_monitor.py:finish():203] Stopping system monitor 2024-06-03 19:56:55,310 INFO HandlerThread:322 [interfaces.py:finish():200] Joined cpu monitor 2024-06-03 19:56:55,311 DEBUG SystemMonitor:322 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-06-03 19:56:55,311 DEBUG SystemMonitor:322 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-06-03 19:56:55,311 DEBUG SystemMonitor:322 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-06-03 19:56:55,312 INFO HandlerThread:322 [interfaces.py:finish():200] Joined disk monitor 2024-06-03 19:56:55,323 INFO HandlerThread:322 [interfaces.py:finish():200] Joined gpu monitor 2024-06-03 19:56:55,323 INFO HandlerThread:322 [interfaces.py:finish():200] Joined memory monitor 2024-06-03 19:56:55,323 INFO HandlerThread:322 [interfaces.py:finish():200] Joined network monitor 2024-06-03 19:56:55,324 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 19:56:55,390 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/output.log 2024-06-03 19:56:57,616 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:56:58,325 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:57:02,617 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:57:03,326 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:57:07,618 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:57:08,327 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:57:12,619 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:57:13,328 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:57:17,620 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:57:18,329 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:57:22,621 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:57:23,330 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:57:27,622 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:57:28,331 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:57:32,623 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:57:33,332 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:57:37,624 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:57:38,333 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:57:42,626 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:57:43,334 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:57:47,627 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:57:48,335 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:57:52,628 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:57:53,336 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:57:57,629 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:57:58,337 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:58:02,630 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:58:03,338 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:58:07,631 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:58:08,339 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:58:12,632 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:58:13,340 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:58:17,634 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:58:18,341 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:58:22,635 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:58:23,342 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:58:27,636 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:58:28,343 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:58:32,637 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:58:33,344 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:58:37,638 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:58:38,345 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:58:42,639 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:58:43,346 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:58:47,640 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:58:48,347 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:58:52,642 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:58:53,348 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:58:57,643 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:58:58,349 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:59:02,644 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:59:03,350 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:59:07,645 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:59:08,351 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:59:12,646 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:59:13,352 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:59:17,647 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:59:18,353 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:59:22,649 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:59:23,353 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:59:27,650 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:59:28,354 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:59:32,651 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:59:33,355 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:59:37,652 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:59:38,356 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:59:42,653 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:59:43,358 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:59:47,654 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:59:48,359 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:59:52,655 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:59:53,360 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 19:59:57,656 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 19:59:58,361 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:00:02,657 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:00:03,362 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:00:07,658 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:00:08,363 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:00:12,659 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:00:13,364 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:00:17,661 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:00:18,365 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:00:22,662 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:00:23,367 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:00:27,663 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:00:28,368 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:00:32,664 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:00:33,369 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:00:37,665 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:00:38,370 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:00:42,666 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:00:43,371 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:00:47,668 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:00:48,372 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:00:52,669 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:00:53,373 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:00:57,670 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:00:58,373 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:01:02,671 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:01:03,374 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:01:07,672 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:01:08,376 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:01:12,673 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:01:13,377 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:01:17,674 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:01:18,378 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:01:22,675 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:01:23,379 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:01:27,676 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:01:28,380 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:01:32,677 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:01:33,381 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:01:37,678 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:01:38,382 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:01:42,679 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:01:43,383 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:01:47,681 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:01:48,384 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:01:52,682 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:01:53,385 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:01:57,683 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:01:58,386 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:02:02,684 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:02:03,387 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:02:07,685 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:02:08,388 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:02:12,686 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:02:13,389 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:02:17,687 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:02:18,390 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:02:21,392 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: resume 2024-06-03 20:02:21,392 INFO HandlerThread:322 [handler.py:handle_request_resume():715] starting system metrics thread 2024-06-03 20:02:21,392 INFO HandlerThread:322 [system_monitor.py:start():194] Starting system monitor 2024-06-03 20:02:21,392 INFO SystemMonitor:322 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-06-03 20:02:21,393 INFO SystemMonitor:322 [interfaces.py:start():188] Started cpu monitoring 2024-06-03 20:02:21,394 INFO SystemMonitor:322 [interfaces.py:start():188] Started disk monitoring 2024-06-03 20:02:21,394 INFO SystemMonitor:322 [interfaces.py:start():188] Started gpu monitoring 2024-06-03 20:02:21,395 INFO SystemMonitor:322 [interfaces.py:start():188] Started memory monitoring 2024-06-03 20:02:21,396 INFO SystemMonitor:322 [interfaces.py:start():188] Started network monitoring 2024-06-03 20:02:22,164 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: pause 2024-06-03 20:02:22,165 INFO HandlerThread:322 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-06-03 20:02:22,165 INFO HandlerThread:322 [system_monitor.py:finish():203] Stopping system monitor 2024-06-03 20:02:22,165 DEBUG SystemMonitor:322 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-06-03 20:02:22,165 DEBUG SystemMonitor:322 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-06-03 20:02:22,166 INFO HandlerThread:322 [interfaces.py:finish():200] Joined cpu monitor 2024-06-03 20:02:22,166 INFO HandlerThread:322 [interfaces.py:finish():200] Joined disk monitor 2024-06-03 20:02:22,166 DEBUG SystemMonitor:322 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-06-03 20:02:22,178 INFO HandlerThread:322 [interfaces.py:finish():200] Joined gpu monitor 2024-06-03 20:02:22,178 INFO HandlerThread:322 [interfaces.py:finish():200] Joined memory monitor 2024-06-03 20:02:22,178 INFO HandlerThread:322 [interfaces.py:finish():200] Joined network monitor 2024-06-03 20:02:22,179 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 20:02:22,688 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:02:24,179 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:02:27,690 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:02:29,181 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:02:32,691 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:02:34,182 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:02:37,692 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:02:39,183 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:02:42,693 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:02:44,183 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:02:47,694 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:02:49,184 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:02:52,695 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:02:54,185 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:02:57,696 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:02:59,186 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:03:02,697 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:03:04,187 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:03:07,698 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:03:09,188 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:03:12,699 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:03:12,803 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: resume 2024-06-03 20:03:12,803 INFO HandlerThread:322 [handler.py:handle_request_resume():715] starting system metrics thread 2024-06-03 20:03:12,804 INFO HandlerThread:322 [system_monitor.py:start():194] Starting system monitor 2024-06-03 20:03:12,804 INFO SystemMonitor:322 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-06-03 20:03:12,804 INFO SystemMonitor:322 [interfaces.py:start():188] Started cpu monitoring 2024-06-03 20:03:12,805 INFO SystemMonitor:322 [interfaces.py:start():188] Started disk monitoring 2024-06-03 20:03:12,806 INFO SystemMonitor:322 [interfaces.py:start():188] Started gpu monitoring 2024-06-03 20:03:12,807 INFO SystemMonitor:322 [interfaces.py:start():188] Started memory monitoring 2024-06-03 20:03:12,808 INFO SystemMonitor:322 [interfaces.py:start():188] Started network monitoring 2024-06-03 20:03:12,828 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: pause 2024-06-03 20:03:12,828 INFO HandlerThread:322 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-06-03 20:03:12,828 INFO HandlerThread:322 [system_monitor.py:finish():203] Stopping system monitor 2024-06-03 20:03:12,828 DEBUG SystemMonitor:322 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-06-03 20:03:12,829 DEBUG SystemMonitor:322 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-06-03 20:03:12,829 DEBUG SystemMonitor:322 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-06-03 20:03:12,829 INFO HandlerThread:322 [interfaces.py:finish():200] Joined cpu monitor 2024-06-03 20:03:12,830 INFO HandlerThread:322 [interfaces.py:finish():200] Joined disk monitor 2024-06-03 20:03:12,840 INFO HandlerThread:322 [interfaces.py:finish():200] Joined gpu monitor 2024-06-03 20:03:12,841 INFO HandlerThread:322 [interfaces.py:finish():200] Joined memory monitor 2024-06-03 20:03:12,841 INFO HandlerThread:322 [interfaces.py:finish():200] Joined network monitor 2024-06-03 20:03:12,841 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 20:03:14,842 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:03:17,700 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:03:19,843 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:03:22,701 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:03:22,909 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: resume 2024-06-03 20:03:22,909 INFO HandlerThread:322 [handler.py:handle_request_resume():715] starting system metrics thread 2024-06-03 20:03:22,909 INFO HandlerThread:322 [system_monitor.py:start():194] Starting system monitor 2024-06-03 20:03:22,910 INFO SystemMonitor:322 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-06-03 20:03:22,910 INFO SystemMonitor:322 [interfaces.py:start():188] Started cpu monitoring 2024-06-03 20:03:22,911 INFO SystemMonitor:322 [interfaces.py:start():188] Started disk monitoring 2024-06-03 20:03:22,912 INFO SystemMonitor:322 [interfaces.py:start():188] Started gpu monitoring 2024-06-03 20:03:22,913 INFO SystemMonitor:322 [interfaces.py:start():188] Started memory monitoring 2024-06-03 20:03:22,914 INFO SystemMonitor:322 [interfaces.py:start():188] Started network monitoring 2024-06-03 20:03:23,546 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: pause 2024-06-03 20:03:23,546 INFO HandlerThread:322 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-06-03 20:03:23,546 INFO HandlerThread:322 [system_monitor.py:finish():203] Stopping system monitor 2024-06-03 20:03:23,547 INFO HandlerThread:322 [interfaces.py:finish():200] Joined cpu monitor 2024-06-03 20:03:23,547 DEBUG SystemMonitor:322 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-06-03 20:03:23,547 INFO HandlerThread:322 [interfaces.py:finish():200] Joined disk monitor 2024-06-03 20:03:23,547 DEBUG SystemMonitor:322 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-06-03 20:03:23,548 DEBUG SystemMonitor:322 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-06-03 20:03:23,560 INFO HandlerThread:322 [interfaces.py:finish():200] Joined gpu monitor 2024-06-03 20:03:23,560 INFO HandlerThread:322 [interfaces.py:finish():200] Joined memory monitor 2024-06-03 20:03:23,561 INFO HandlerThread:322 [interfaces.py:finish():200] Joined network monitor 2024-06-03 20:03:23,561 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 20:03:25,562 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:03:27,702 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:03:30,563 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:03:32,704 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:03:35,564 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:03:37,705 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:03:40,565 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:03:42,706 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:03:45,566 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:03:47,707 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:03:50,566 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:03:52,708 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:03:55,567 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:03:57,709 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:04:00,568 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:04:02,710 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:04:05,569 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:04:07,711 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:04:10,570 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:04:12,712 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:04:15,571 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:04:16,405 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: resume 2024-06-03 20:04:16,405 INFO HandlerThread:322 [handler.py:handle_request_resume():715] starting system metrics thread 2024-06-03 20:04:16,406 INFO HandlerThread:322 [system_monitor.py:start():194] Starting system monitor 2024-06-03 20:04:16,406 INFO SystemMonitor:322 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-06-03 20:04:16,406 INFO SystemMonitor:322 [interfaces.py:start():188] Started cpu monitoring 2024-06-03 20:04:16,407 INFO SystemMonitor:322 [interfaces.py:start():188] Started disk monitoring 2024-06-03 20:04:16,409 INFO SystemMonitor:322 [interfaces.py:start():188] Started gpu monitoring 2024-06-03 20:04:16,409 INFO SystemMonitor:322 [interfaces.py:start():188] Started memory monitoring 2024-06-03 20:04:16,410 INFO SystemMonitor:322 [interfaces.py:start():188] Started network monitoring 2024-06-03 20:04:16,447 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: pause 2024-06-03 20:04:16,447 INFO HandlerThread:322 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-06-03 20:04:16,448 INFO HandlerThread:322 [system_monitor.py:finish():203] Stopping system monitor 2024-06-03 20:04:16,448 DEBUG SystemMonitor:322 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-06-03 20:04:16,448 DEBUG SystemMonitor:322 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-06-03 20:04:16,448 DEBUG SystemMonitor:322 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-06-03 20:04:16,449 INFO HandlerThread:322 [interfaces.py:finish():200] Joined cpu monitor 2024-06-03 20:04:16,449 INFO HandlerThread:322 [interfaces.py:finish():200] Joined disk monitor 2024-06-03 20:04:16,461 INFO HandlerThread:322 [interfaces.py:finish():200] Joined gpu monitor 2024-06-03 20:04:16,462 INFO HandlerThread:322 [interfaces.py:finish():200] Joined memory monitor 2024-06-03 20:04:16,462 INFO HandlerThread:322 [interfaces.py:finish():200] Joined network monitor 2024-06-03 20:04:16,462 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 20:04:17,713 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:04:21,463 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:04:22,714 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:04:26,464 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:04:27,715 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:04:31,465 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:04:32,716 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:04:32,979 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: resume 2024-06-03 20:04:32,979 INFO HandlerThread:322 [handler.py:handle_request_resume():715] starting system metrics thread 2024-06-03 20:04:32,979 INFO HandlerThread:322 [system_monitor.py:start():194] Starting system monitor 2024-06-03 20:04:32,979 INFO SystemMonitor:322 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-06-03 20:04:32,980 INFO SystemMonitor:322 [interfaces.py:start():188] Started cpu monitoring 2024-06-03 20:04:32,980 INFO SystemMonitor:322 [interfaces.py:start():188] Started disk monitoring 2024-06-03 20:04:32,981 INFO SystemMonitor:322 [interfaces.py:start():188] Started gpu monitoring 2024-06-03 20:04:32,982 INFO SystemMonitor:322 [interfaces.py:start():188] Started memory monitoring 2024-06-03 20:04:32,988 INFO SystemMonitor:322 [interfaces.py:start():188] Started network monitoring 2024-06-03 20:04:33,028 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: pause 2024-06-03 20:04:33,028 INFO HandlerThread:322 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-06-03 20:04:33,028 INFO HandlerThread:322 [system_monitor.py:finish():203] Stopping system monitor 2024-06-03 20:04:33,028 INFO HandlerThread:322 [interfaces.py:finish():200] Joined cpu monitor 2024-06-03 20:04:33,029 INFO HandlerThread:322 [interfaces.py:finish():200] Joined disk monitor 2024-06-03 20:04:33,029 DEBUG SystemMonitor:322 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-06-03 20:04:33,029 DEBUG SystemMonitor:322 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-06-03 20:04:33,029 DEBUG SystemMonitor:322 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-06-03 20:04:33,040 INFO HandlerThread:322 [interfaces.py:finish():200] Joined gpu monitor 2024-06-03 20:04:33,040 INFO HandlerThread:322 [interfaces.py:finish():200] Joined memory monitor 2024-06-03 20:04:33,041 INFO HandlerThread:322 [interfaces.py:finish():200] Joined network monitor 2024-06-03 20:04:33,041 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 20:04:37,042 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:04:37,717 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:04:42,043 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:04:42,718 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:04:47,044 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:04:47,719 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:04:52,045 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:04:52,721 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:04:57,046 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:04:57,722 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:05:02,047 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:05:02,723 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:05:07,047 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:05:07,724 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:05:12,048 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:05:12,725 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:05:17,049 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:05:17,726 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:05:18,073 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: resume 2024-06-03 20:05:18,073 INFO HandlerThread:322 [handler.py:handle_request_resume():715] starting system metrics thread 2024-06-03 20:05:18,073 INFO HandlerThread:322 [system_monitor.py:start():194] Starting system monitor 2024-06-03 20:05:18,074 INFO SystemMonitor:322 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-06-03 20:05:18,074 INFO SystemMonitor:322 [interfaces.py:start():188] Started cpu monitoring 2024-06-03 20:05:18,075 INFO SystemMonitor:322 [interfaces.py:start():188] Started disk monitoring 2024-06-03 20:05:18,075 INFO SystemMonitor:322 [interfaces.py:start():188] Started gpu monitoring 2024-06-03 20:05:18,076 INFO SystemMonitor:322 [interfaces.py:start():188] Started memory monitoring 2024-06-03 20:05:18,076 INFO SystemMonitor:322 [interfaces.py:start():188] Started network monitoring 2024-06-03 20:05:18,119 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: pause 2024-06-03 20:05:18,119 INFO HandlerThread:322 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-06-03 20:05:18,119 INFO HandlerThread:322 [system_monitor.py:finish():203] Stopping system monitor 2024-06-03 20:05:18,119 DEBUG SystemMonitor:322 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-06-03 20:05:18,120 DEBUG SystemMonitor:322 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-06-03 20:05:18,120 DEBUG SystemMonitor:322 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-06-03 20:05:18,120 INFO HandlerThread:322 [interfaces.py:finish():200] Joined cpu monitor 2024-06-03 20:05:18,121 INFO HandlerThread:322 [interfaces.py:finish():200] Joined disk monitor 2024-06-03 20:05:18,131 INFO HandlerThread:322 [interfaces.py:finish():200] Joined gpu monitor 2024-06-03 20:05:18,131 INFO HandlerThread:322 [interfaces.py:finish():200] Joined memory monitor 2024-06-03 20:05:18,131 INFO HandlerThread:322 [interfaces.py:finish():200] Joined network monitor 2024-06-03 20:05:18,132 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 20:05:22,133 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:05:22,727 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:05:27,134 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:05:27,728 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:05:31,532 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: resume 2024-06-03 20:05:31,532 INFO HandlerThread:322 [handler.py:handle_request_resume():715] starting system metrics thread 2024-06-03 20:05:31,532 INFO HandlerThread:322 [system_monitor.py:start():194] Starting system monitor 2024-06-03 20:05:31,532 INFO SystemMonitor:322 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-06-03 20:05:31,533 INFO SystemMonitor:322 [interfaces.py:start():188] Started cpu monitoring 2024-06-03 20:05:31,534 INFO SystemMonitor:322 [interfaces.py:start():188] Started disk monitoring 2024-06-03 20:05:31,534 INFO SystemMonitor:322 [interfaces.py:start():188] Started gpu monitoring 2024-06-03 20:05:31,535 INFO SystemMonitor:322 [interfaces.py:start():188] Started memory monitoring 2024-06-03 20:05:31,535 INFO SystemMonitor:322 [interfaces.py:start():188] Started network monitoring 2024-06-03 20:05:31,581 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: pause 2024-06-03 20:05:31,581 INFO HandlerThread:322 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-06-03 20:05:31,581 INFO HandlerThread:322 [system_monitor.py:finish():203] Stopping system monitor 2024-06-03 20:05:31,581 INFO HandlerThread:322 [interfaces.py:finish():200] Joined cpu monitor 2024-06-03 20:05:31,581 DEBUG SystemMonitor:322 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-06-03 20:05:31,581 INFO HandlerThread:322 [interfaces.py:finish():200] Joined disk monitor 2024-06-03 20:05:31,582 DEBUG SystemMonitor:322 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-06-03 20:05:31,582 DEBUG SystemMonitor:322 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-06-03 20:05:31,593 INFO HandlerThread:322 [interfaces.py:finish():200] Joined gpu monitor 2024-06-03 20:05:31,593 INFO HandlerThread:322 [interfaces.py:finish():200] Joined memory monitor 2024-06-03 20:05:31,593 INFO HandlerThread:322 [interfaces.py:finish():200] Joined network monitor 2024-06-03 20:05:31,594 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 20:05:32,594 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:05:32,729 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:05:37,595 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:05:37,730 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:05:42,596 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:05:42,731 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:05:44,102 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: resume 2024-06-03 20:05:44,102 INFO HandlerThread:322 [handler.py:handle_request_resume():715] starting system metrics thread 2024-06-03 20:05:44,102 INFO HandlerThread:322 [system_monitor.py:start():194] Starting system monitor 2024-06-03 20:05:44,103 INFO SystemMonitor:322 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-06-03 20:05:44,103 INFO SystemMonitor:322 [interfaces.py:start():188] Started cpu monitoring 2024-06-03 20:05:44,104 INFO SystemMonitor:322 [interfaces.py:start():188] Started disk monitoring 2024-06-03 20:05:44,105 INFO SystemMonitor:322 [interfaces.py:start():188] Started gpu monitoring 2024-06-03 20:05:44,106 INFO SystemMonitor:322 [interfaces.py:start():188] Started memory monitoring 2024-06-03 20:05:44,107 INFO SystemMonitor:322 [interfaces.py:start():188] Started network monitoring 2024-06-03 20:05:44,781 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: pause 2024-06-03 20:05:44,781 INFO HandlerThread:322 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-06-03 20:05:44,781 INFO HandlerThread:322 [system_monitor.py:finish():203] Stopping system monitor 2024-06-03 20:05:44,781 DEBUG SystemMonitor:322 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-06-03 20:05:44,781 DEBUG SystemMonitor:322 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-06-03 20:05:44,781 DEBUG SystemMonitor:322 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-06-03 20:05:44,784 INFO HandlerThread:322 [interfaces.py:finish():200] Joined cpu monitor 2024-06-03 20:05:44,784 INFO HandlerThread:322 [interfaces.py:finish():200] Joined disk monitor 2024-06-03 20:05:44,794 INFO HandlerThread:322 [interfaces.py:finish():200] Joined gpu monitor 2024-06-03 20:05:44,794 INFO HandlerThread:322 [interfaces.py:finish():200] Joined memory monitor 2024-06-03 20:05:44,794 INFO HandlerThread:322 [interfaces.py:finish():200] Joined network monitor 2024-06-03 20:05:44,795 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 20:05:47,738 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:05:47,795 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:05:52,739 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:05:52,797 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:05:57,740 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:05:57,798 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:06:02,741 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:06:02,799 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:06:07,743 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:06:07,800 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:06:12,744 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:06:12,801 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:06:17,745 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:06:17,802 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:06:22,746 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:06:22,803 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:06:27,747 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:06:27,804 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:06:32,748 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:06:32,805 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:06:37,085 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: resume 2024-06-03 20:06:37,085 INFO HandlerThread:322 [handler.py:handle_request_resume():715] starting system metrics thread 2024-06-03 20:06:37,085 INFO HandlerThread:322 [system_monitor.py:start():194] Starting system monitor 2024-06-03 20:06:37,085 INFO SystemMonitor:322 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-06-03 20:06:37,086 INFO SystemMonitor:322 [interfaces.py:start():188] Started cpu monitoring 2024-06-03 20:06:37,086 INFO SystemMonitor:322 [interfaces.py:start():188] Started disk monitoring 2024-06-03 20:06:37,087 INFO SystemMonitor:322 [interfaces.py:start():188] Started gpu monitoring 2024-06-03 20:06:37,088 INFO SystemMonitor:322 [interfaces.py:start():188] Started memory monitoring 2024-06-03 20:06:37,089 INFO SystemMonitor:322 [interfaces.py:start():188] Started network monitoring 2024-06-03 20:06:37,754 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:06:37,806 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:06:37,831 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: pause 2024-06-03 20:06:37,832 INFO HandlerThread:322 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-06-03 20:06:37,832 INFO HandlerThread:322 [system_monitor.py:finish():203] Stopping system monitor 2024-06-03 20:06:37,832 INFO HandlerThread:322 [interfaces.py:finish():200] Joined cpu monitor 2024-06-03 20:06:37,833 DEBUG SystemMonitor:322 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-06-03 20:06:37,833 DEBUG SystemMonitor:322 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-06-03 20:06:37,833 DEBUG SystemMonitor:322 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-06-03 20:06:37,834 INFO HandlerThread:322 [interfaces.py:finish():200] Joined disk monitor 2024-06-03 20:06:37,845 INFO HandlerThread:322 [interfaces.py:finish():200] Joined gpu monitor 2024-06-03 20:06:37,845 INFO HandlerThread:322 [interfaces.py:finish():200] Joined memory monitor 2024-06-03 20:06:37,845 INFO HandlerThread:322 [interfaces.py:finish():200] Joined network monitor 2024-06-03 20:06:37,846 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 20:06:42,761 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:06:42,847 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:06:47,762 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:06:47,848 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:06:52,763 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:06:52,849 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:06:57,764 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:06:57,850 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:07:02,765 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:07:02,850 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:07:07,766 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:07:07,851 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:07:12,768 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:07:12,852 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:07:17,769 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:07:17,854 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:07:22,770 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:07:22,855 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:07:27,771 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:07:27,856 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:07:32,772 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:07:32,856 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:07:37,773 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:07:37,858 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:07:42,774 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:07:42,859 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:07:47,776 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:07:47,860 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:07:52,777 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:07:52,861 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:07:57,778 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:07:57,862 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:08:02,779 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:08:02,862 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:08:07,781 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:08:07,863 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:08:12,782 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:08:12,864 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:08:17,783 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:08:17,865 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:08:22,784 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:08:22,866 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:08:27,786 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:08:27,867 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:08:32,787 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:08:32,868 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:08:37,788 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:08:37,869 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:08:42,789 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:08:42,870 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:08:47,790 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:08:47,870 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:08:52,791 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:08:52,871 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:08:57,792 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:08:57,873 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:08:59,976 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: resume 2024-06-03 20:08:59,976 INFO HandlerThread:322 [handler.py:handle_request_resume():715] starting system metrics thread 2024-06-03 20:08:59,976 INFO HandlerThread:322 [system_monitor.py:start():194] Starting system monitor 2024-06-03 20:08:59,976 INFO SystemMonitor:322 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-06-03 20:08:59,977 INFO SystemMonitor:322 [interfaces.py:start():188] Started cpu monitoring 2024-06-03 20:08:59,977 INFO SystemMonitor:322 [interfaces.py:start():188] Started disk monitoring 2024-06-03 20:08:59,978 INFO SystemMonitor:322 [interfaces.py:start():188] Started gpu monitoring 2024-06-03 20:08:59,979 INFO SystemMonitor:322 [interfaces.py:start():188] Started memory monitoring 2024-06-03 20:08:59,979 INFO SystemMonitor:322 [interfaces.py:start():188] Started network monitoring 2024-06-03 20:09:00,011 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: pause 2024-06-03 20:09:00,011 INFO HandlerThread:322 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-06-03 20:09:00,011 INFO HandlerThread:322 [system_monitor.py:finish():203] Stopping system monitor 2024-06-03 20:09:00,012 INFO HandlerThread:322 [interfaces.py:finish():200] Joined cpu monitor 2024-06-03 20:09:00,012 DEBUG SystemMonitor:322 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-06-03 20:09:00,012 INFO HandlerThread:322 [interfaces.py:finish():200] Joined disk monitor 2024-06-03 20:09:00,012 DEBUG SystemMonitor:322 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-06-03 20:09:00,013 DEBUG SystemMonitor:322 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-06-03 20:09:00,023 INFO HandlerThread:322 [interfaces.py:finish():200] Joined gpu monitor 2024-06-03 20:09:00,023 INFO HandlerThread:322 [interfaces.py:finish():200] Joined memory monitor 2024-06-03 20:09:00,023 INFO HandlerThread:322 [interfaces.py:finish():200] Joined network monitor 2024-06-03 20:09:00,024 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 20:09:02,794 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:09:03,025 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:09:06,499 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: resume 2024-06-03 20:09:06,500 INFO HandlerThread:322 [handler.py:handle_request_resume():715] starting system metrics thread 2024-06-03 20:09:06,500 INFO HandlerThread:322 [system_monitor.py:start():194] Starting system monitor 2024-06-03 20:09:06,500 INFO SystemMonitor:322 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-06-03 20:09:06,501 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: pause 2024-06-03 20:09:06,501 INFO HandlerThread:322 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-06-03 20:09:06,501 INFO HandlerThread:322 [system_monitor.py:finish():203] Stopping system monitor 2024-06-03 20:09:06,501 WARNING HandlerThread:322 [interfaces.py:finish():205] Failed to finish cpu monitoring: cannot join thread before it is started 2024-06-03 20:09:06,501 INFO SystemMonitor:322 [interfaces.py:start():188] Started cpu monitoring 2024-06-03 20:09:06,501 DEBUG SystemMonitor:322 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-06-03 20:09:06,501 DEBUG SystemMonitor:322 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-06-03 20:09:06,501 DEBUG SystemMonitor:322 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-06-03 20:09:06,502 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 20:09:07,198 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: resume 2024-06-03 20:09:07,198 INFO HandlerThread:322 [handler.py:handle_request_resume():715] starting system metrics thread 2024-06-03 20:09:07,198 INFO HandlerThread:322 [system_monitor.py:start():194] Starting system monitor 2024-06-03 20:09:07,198 INFO SystemMonitor:322 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-06-03 20:09:07,199 INFO SystemMonitor:322 [interfaces.py:start():188] Started cpu monitoring 2024-06-03 20:09:07,200 INFO SystemMonitor:322 [interfaces.py:start():188] Started disk monitoring 2024-06-03 20:09:07,202 INFO SystemMonitor:322 [interfaces.py:start():188] Started gpu monitoring 2024-06-03 20:09:07,202 INFO SystemMonitor:322 [interfaces.py:start():188] Started memory monitoring 2024-06-03 20:09:07,204 INFO SystemMonitor:322 [interfaces.py:start():188] Started network monitoring 2024-06-03 20:09:07,218 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: pause 2024-06-03 20:09:07,218 INFO HandlerThread:322 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-06-03 20:09:07,218 INFO HandlerThread:322 [system_monitor.py:finish():203] Stopping system monitor 2024-06-03 20:09:07,219 DEBUG SystemMonitor:322 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-06-03 20:09:07,219 DEBUG SystemMonitor:322 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-06-03 20:09:07,219 DEBUG SystemMonitor:322 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-06-03 20:09:07,220 INFO HandlerThread:322 [interfaces.py:finish():200] Joined cpu monitor 2024-06-03 20:09:07,220 INFO HandlerThread:322 [interfaces.py:finish():200] Joined disk monitor 2024-06-03 20:09:07,230 INFO HandlerThread:322 [interfaces.py:finish():200] Joined gpu monitor 2024-06-03 20:09:07,231 INFO HandlerThread:322 [interfaces.py:finish():200] Joined memory monitor 2024-06-03 20:09:07,231 INFO HandlerThread:322 [interfaces.py:finish():200] Joined network monitor 2024-06-03 20:09:07,231 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 20:09:07,795 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:09:08,232 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:09:12,796 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:09:13,233 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:09:17,797 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:09:18,234 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:09:18,370 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: resume 2024-06-03 20:09:18,370 INFO HandlerThread:322 [handler.py:handle_request_resume():715] starting system metrics thread 2024-06-03 20:09:18,370 INFO HandlerThread:322 [system_monitor.py:start():194] Starting system monitor 2024-06-03 20:09:18,371 INFO SystemMonitor:322 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-06-03 20:09:18,371 INFO SystemMonitor:322 [interfaces.py:start():188] Started cpu monitoring 2024-06-03 20:09:18,372 INFO SystemMonitor:322 [interfaces.py:start():188] Started disk monitoring 2024-06-03 20:09:18,373 INFO SystemMonitor:322 [interfaces.py:start():188] Started gpu monitoring 2024-06-03 20:09:18,374 INFO SystemMonitor:322 [interfaces.py:start():188] Started memory monitoring 2024-06-03 20:09:18,375 INFO SystemMonitor:322 [interfaces.py:start():188] Started network monitoring 2024-06-03 20:09:19,120 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: pause 2024-06-03 20:09:19,120 INFO HandlerThread:322 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-06-03 20:09:19,120 INFO HandlerThread:322 [system_monitor.py:finish():203] Stopping system monitor 2024-06-03 20:09:19,121 INFO HandlerThread:322 [interfaces.py:finish():200] Joined cpu monitor 2024-06-03 20:09:19,121 DEBUG SystemMonitor:322 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-06-03 20:09:19,121 DEBUG SystemMonitor:322 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-06-03 20:09:19,121 DEBUG SystemMonitor:322 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-06-03 20:09:19,122 INFO HandlerThread:322 [interfaces.py:finish():200] Joined disk monitor 2024-06-03 20:09:19,133 INFO HandlerThread:322 [interfaces.py:finish():200] Joined gpu monitor 2024-06-03 20:09:19,133 INFO HandlerThread:322 [interfaces.py:finish():200] Joined memory monitor 2024-06-03 20:09:19,133 INFO HandlerThread:322 [interfaces.py:finish():200] Joined network monitor 2024-06-03 20:09:19,134 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 20:09:19,666 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/output.log 2024-06-03 20:09:22,798 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:09:24,135 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:09:27,799 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:09:29,136 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:09:32,801 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:09:34,137 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:09:37,802 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:09:39,138 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:09:42,803 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:09:44,139 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:09:47,804 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:09:49,140 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:09:52,805 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:09:54,141 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:09:57,806 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:09:59,142 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:10:02,807 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:10:04,142 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:10:07,809 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:10:09,143 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:10:12,810 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:10:14,144 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:10:17,811 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:10:19,145 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:10:22,812 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:10:24,146 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:10:27,813 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:10:29,147 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:10:32,814 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:10:34,147 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:10:37,816 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:10:39,148 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:10:42,817 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:10:44,149 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:10:47,818 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:10:49,150 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:10:52,819 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:10:54,151 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:10:57,820 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:10:59,152 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:11:02,821 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:11:04,153 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:11:07,822 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:11:09,154 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:11:12,823 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:11:14,154 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:11:17,824 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:11:19,155 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:11:22,825 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:11:24,156 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:11:27,826 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:11:29,157 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:11:32,828 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:11:34,158 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:11:37,829 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:11:39,159 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:11:42,830 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:11:44,160 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:11:47,831 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:11:49,161 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:11:52,561 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: resume 2024-06-03 20:11:52,561 INFO HandlerThread:322 [handler.py:handle_request_resume():715] starting system metrics thread 2024-06-03 20:11:52,561 INFO HandlerThread:322 [system_monitor.py:start():194] Starting system monitor 2024-06-03 20:11:52,562 INFO SystemMonitor:322 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-06-03 20:11:52,562 INFO SystemMonitor:322 [interfaces.py:start():188] Started cpu monitoring 2024-06-03 20:11:52,563 INFO SystemMonitor:322 [interfaces.py:start():188] Started disk monitoring 2024-06-03 20:11:52,564 INFO SystemMonitor:322 [interfaces.py:start():188] Started gpu monitoring 2024-06-03 20:11:52,566 INFO SystemMonitor:322 [interfaces.py:start():188] Started memory monitoring 2024-06-03 20:11:52,567 INFO SystemMonitor:322 [interfaces.py:start():188] Started network monitoring 2024-06-03 20:11:52,832 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:11:53,722 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/output.log 2024-06-03 20:11:54,590 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: pause 2024-06-03 20:11:54,590 INFO HandlerThread:322 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-06-03 20:11:54,590 INFO HandlerThread:322 [system_monitor.py:finish():203] Stopping system monitor 2024-06-03 20:11:54,590 DEBUG SystemMonitor:322 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-06-03 20:11:54,590 DEBUG SystemMonitor:322 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-06-03 20:11:54,590 DEBUG SystemMonitor:322 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-06-03 20:11:54,591 INFO HandlerThread:322 [interfaces.py:finish():200] Joined cpu monitor 2024-06-03 20:11:54,592 INFO HandlerThread:322 [interfaces.py:finish():200] Joined disk monitor 2024-06-03 20:11:54,603 INFO HandlerThread:322 [interfaces.py:finish():200] Joined gpu monitor 2024-06-03 20:11:54,603 INFO HandlerThread:322 [interfaces.py:finish():200] Joined memory monitor 2024-06-03 20:11:54,603 INFO HandlerThread:322 [interfaces.py:finish():200] Joined network monitor 2024-06-03 20:11:54,604 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 20:11:54,604 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:11:57,838 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:11:59,605 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:12:02,839 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:12:04,606 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:12:07,840 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:12:09,607 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:12:12,841 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:12:14,608 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:12:17,842 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:12:19,609 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:12:22,843 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:12:24,610 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:12:27,844 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:12:29,611 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:12:32,845 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:12:34,612 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:12:37,846 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:12:39,613 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:12:42,847 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:12:44,614 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:12:47,849 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:12:49,615 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:12:52,850 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:12:54,616 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:12:57,850 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:12:59,617 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:13:02,851 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:13:04,618 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:13:07,852 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:13:09,619 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:13:12,854 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:13:14,620 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:13:17,855 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:13:19,620 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:13:22,856 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:13:24,622 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:13:27,857 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:13:29,623 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:13:32,858 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:13:34,624 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:13:37,859 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:13:39,625 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:13:42,860 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:13:44,626 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:13:47,861 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:13:49,627 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:13:52,862 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:13:54,628 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:13:57,863 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:13:59,628 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:14:02,865 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:14:04,629 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:14:07,866 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:14:09,630 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:14:12,867 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:14:14,631 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:14:17,868 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:14:19,632 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:14:22,869 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:14:24,633 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:14:27,870 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:14:29,634 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:14:32,871 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:14:34,635 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:14:37,872 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:14:39,636 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:14:42,874 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:14:44,637 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:14:47,875 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:14:49,638 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:14:52,876 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:14:54,639 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:14:57,877 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:14:59,640 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:15:02,878 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:15:04,641 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:15:07,879 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:15:09,642 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:15:12,880 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:15:14,643 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:15:17,881 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:15:19,644 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:15:22,883 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:15:24,644 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:15:27,884 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:15:29,646 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:15:32,885 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:15:34,646 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:15:37,886 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:15:39,648 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:15:42,887 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:15:44,648 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:15:47,889 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:15:49,649 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:15:52,890 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:15:54,650 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:15:57,891 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:15:59,651 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:16:02,892 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:16:04,652 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:16:07,893 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:16:09,653 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:16:12,894 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:16:14,653 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:16:17,895 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:16:19,654 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:16:22,896 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:16:24,655 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:16:27,897 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:16:29,656 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:16:32,898 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:16:34,657 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:16:37,899 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:16:39,657 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:16:42,900 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:16:44,658 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:16:47,902 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:16:49,659 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:16:50,594 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: resume 2024-06-03 20:16:50,595 INFO HandlerThread:322 [handler.py:handle_request_resume():715] starting system metrics thread 2024-06-03 20:16:50,595 INFO HandlerThread:322 [system_monitor.py:start():194] Starting system monitor 2024-06-03 20:16:50,595 INFO SystemMonitor:322 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-06-03 20:16:50,595 INFO SystemMonitor:322 [interfaces.py:start():188] Started cpu monitoring 2024-06-03 20:16:50,596 INFO SystemMonitor:322 [interfaces.py:start():188] Started disk monitoring 2024-06-03 20:16:50,597 INFO SystemMonitor:322 [interfaces.py:start():188] Started gpu monitoring 2024-06-03 20:16:50,597 INFO SystemMonitor:322 [interfaces.py:start():188] Started memory monitoring 2024-06-03 20:16:50,599 INFO SystemMonitor:322 [interfaces.py:start():188] Started network monitoring 2024-06-03 20:16:52,908 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:16:54,660 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:16:57,909 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:16:59,660 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:17:02,910 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:17:04,661 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:17:07,911 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:17:09,662 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:17:12,912 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:17:14,663 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:17:17,913 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:17:19,664 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:17:22,914 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:17:24,665 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:17:27,915 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:17:29,665 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:17:32,848 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/output.log 2024-06-03 20:17:33,296 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:17:34,737 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:17:38,298 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:17:39,738 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:17:43,299 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:17:44,739 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:17:48,300 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:17:49,741 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:17:50,599 DEBUG SystemMonitor:322 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-06-03 20:17:50,602 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 20:17:53,302 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:17:55,604 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:17:58,303 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:18:00,605 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:18:03,304 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:18:05,607 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:18:08,306 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:18:10,608 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:18:13,307 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:18:15,609 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:18:18,308 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:18:20,607 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 20:18:21,608 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:18:23,310 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:18:26,610 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:18:28,311 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:18:31,611 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:18:33,312 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:18:36,612 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:18:38,320 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:18:41,613 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:18:43,321 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:18:46,614 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:18:48,323 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:18:50,613 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 20:18:52,614 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:18:53,324 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:18:57,616 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:18:58,325 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:19:02,617 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:19:03,327 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:19:04,531 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: pause 2024-06-03 20:19:04,531 INFO HandlerThread:322 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-06-03 20:19:04,531 INFO HandlerThread:322 [system_monitor.py:finish():203] Stopping system monitor 2024-06-03 20:19:04,532 INFO HandlerThread:322 [interfaces.py:finish():200] Joined cpu monitor 2024-06-03 20:19:04,532 INFO HandlerThread:322 [interfaces.py:finish():200] Joined disk monitor 2024-06-03 20:19:04,532 DEBUG SystemMonitor:322 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-06-03 20:19:04,532 DEBUG SystemMonitor:322 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-06-03 20:19:04,544 INFO HandlerThread:322 [interfaces.py:finish():200] Joined gpu monitor 2024-06-03 20:19:04,544 INFO HandlerThread:322 [interfaces.py:finish():200] Joined memory monitor 2024-06-03 20:19:04,545 INFO HandlerThread:322 [interfaces.py:finish():200] Joined network monitor 2024-06-03 20:19:04,545 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 20:19:06,907 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/output.log 2024-06-03 20:19:08,328 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:19:08,546 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:19:13,329 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:19:13,547 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:19:18,330 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:19:18,548 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:19:23,332 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:19:23,548 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:19:28,333 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:19:28,549 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:19:33,334 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:19:33,550 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:19:38,335 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:19:38,551 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:19:43,336 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:19:43,552 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:19:48,337 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:19:48,554 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:19:53,338 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:19:53,554 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:19:58,339 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:19:58,555 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:20:03,340 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:20:03,556 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:20:08,342 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:20:08,557 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:20:13,343 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:20:13,558 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:20:18,344 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:20:18,559 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:20:23,345 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:20:23,560 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:20:28,346 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:20:28,561 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:20:33,195 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: resume 2024-06-03 20:20:33,196 INFO HandlerThread:322 [handler.py:handle_request_resume():715] starting system metrics thread 2024-06-03 20:20:33,196 INFO HandlerThread:322 [system_monitor.py:start():194] Starting system monitor 2024-06-03 20:20:33,196 INFO SystemMonitor:322 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-06-03 20:20:33,196 INFO SystemMonitor:322 [interfaces.py:start():188] Started cpu monitoring 2024-06-03 20:20:33,197 INFO SystemMonitor:322 [interfaces.py:start():188] Started disk monitoring 2024-06-03 20:20:33,198 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: pause 2024-06-03 20:20:33,198 INFO HandlerThread:322 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-06-03 20:20:33,198 INFO HandlerThread:322 [system_monitor.py:finish():203] Stopping system monitor 2024-06-03 20:20:33,198 INFO SystemMonitor:322 [interfaces.py:start():188] Started gpu monitoring 2024-06-03 20:20:33,198 DEBUG SystemMonitor:322 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-06-03 20:20:33,198 DEBUG SystemMonitor:322 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-06-03 20:20:33,198 DEBUG SystemMonitor:322 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-06-03 20:20:33,200 INFO HandlerThread:322 [interfaces.py:finish():200] Joined cpu monitor 2024-06-03 20:20:33,202 INFO HandlerThread:322 [interfaces.py:finish():200] Joined disk monitor 2024-06-03 20:20:33,221 INFO HandlerThread:322 [interfaces.py:finish():200] Joined gpu monitor 2024-06-03 20:20:33,222 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 20:20:33,347 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:20:34,222 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:20:38,348 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:20:39,223 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:20:43,349 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:20:44,224 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:20:48,350 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:20:49,225 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:20:53,351 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:20:53,791 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: resume 2024-06-03 20:20:53,791 INFO HandlerThread:322 [handler.py:handle_request_resume():715] starting system metrics thread 2024-06-03 20:20:53,791 INFO HandlerThread:322 [system_monitor.py:start():194] Starting system monitor 2024-06-03 20:20:53,791 INFO SystemMonitor:322 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-06-03 20:20:53,792 INFO SystemMonitor:322 [interfaces.py:start():188] Started cpu monitoring 2024-06-03 20:20:53,792 INFO SystemMonitor:322 [interfaces.py:start():188] Started disk monitoring 2024-06-03 20:20:53,793 INFO SystemMonitor:322 [interfaces.py:start():188] Started gpu monitoring 2024-06-03 20:20:53,794 INFO SystemMonitor:322 [interfaces.py:start():188] Started memory monitoring 2024-06-03 20:20:53,796 INFO SystemMonitor:322 [interfaces.py:start():188] Started network monitoring 2024-06-03 20:20:54,802 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:20:54,949 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/output.log 2024-06-03 20:20:58,353 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:20:59,803 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:21:03,354 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:21:04,804 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:21:08,356 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:21:09,809 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:21:13,357 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:21:14,810 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:21:18,358 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:21:19,811 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:21:23,360 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:21:24,812 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:21:28,361 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:21:29,814 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:21:33,363 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:21:34,815 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:21:38,364 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:21:39,817 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:21:43,366 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:21:44,818 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:21:48,367 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:21:49,819 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:21:53,368 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:21:53,796 DEBUG SystemMonitor:322 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-06-03 20:21:53,799 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 20:21:55,800 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:21:58,370 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:22:00,801 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:22:03,371 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:22:05,803 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:22:08,373 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:22:10,805 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:22:13,374 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:22:15,806 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:22:18,376 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:22:20,807 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:22:23,378 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:22:23,801 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 20:22:26,805 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:22:28,379 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:22:31,806 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:22:33,381 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:22:36,808 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:22:38,382 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:22:41,809 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:22:43,384 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:22:46,811 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:22:48,386 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:22:51,812 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:22:53,387 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:22:53,803 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 20:22:57,805 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:22:58,388 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:23:02,806 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:23:03,389 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:23:07,808 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:23:08,391 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:23:12,809 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:23:13,392 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:23:17,810 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:23:18,394 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:23:22,813 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:23:23,396 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:23:23,804 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 20:23:28,397 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:23:28,806 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:23:33,399 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:23:33,808 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:23:38,400 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:23:38,809 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:23:43,402 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:23:43,810 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:23:48,404 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:23:48,811 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:23:53,405 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:23:53,806 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 20:23:54,807 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:23:58,406 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:23:59,809 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:24:03,407 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:24:04,810 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:24:08,409 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:24:09,811 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:24:13,411 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:24:14,812 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:24:18,412 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:24:19,813 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:24:20,236 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: pause 2024-06-03 20:24:20,236 INFO HandlerThread:322 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-06-03 20:24:20,236 INFO HandlerThread:322 [system_monitor.py:finish():203] Stopping system monitor 2024-06-03 20:24:20,237 DEBUG SystemMonitor:322 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-06-03 20:24:20,237 DEBUG SystemMonitor:322 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-06-03 20:24:20,238 INFO HandlerThread:322 [interfaces.py:finish():200] Joined cpu monitor 2024-06-03 20:24:20,238 INFO HandlerThread:322 [interfaces.py:finish():200] Joined disk monitor 2024-06-03 20:24:20,250 INFO HandlerThread:322 [interfaces.py:finish():200] Joined gpu monitor 2024-06-03 20:24:20,251 INFO HandlerThread:322 [interfaces.py:finish():200] Joined memory monitor 2024-06-03 20:24:20,251 INFO HandlerThread:322 [interfaces.py:finish():200] Joined network monitor 2024-06-03 20:24:20,251 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 20:24:23,083 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/output.log 2024-06-03 20:24:23,413 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:24:25,252 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:24:28,414 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:24:30,253 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:24:33,415 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:24:35,254 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:24:38,416 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:24:40,255 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:24:43,418 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:24:45,256 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:24:48,419 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:24:50,257 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:24:53,420 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:24:55,258 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:24:58,421 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:25:00,259 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:25:03,422 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:25:05,260 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:25:08,423 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:25:10,261 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:25:13,425 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:25:15,262 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:25:18,426 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:25:20,263 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:25:23,427 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:25:25,264 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:25:28,428 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:25:30,265 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:25:33,429 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:25:35,266 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:25:38,430 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:25:40,267 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:25:43,431 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:25:45,268 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:25:48,432 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:25:50,269 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:25:53,433 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:25:55,270 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:25:58,434 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:26:00,271 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:26:03,435 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:26:05,271 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:26:08,436 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:26:10,272 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:26:13,438 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:26:15,273 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:26:18,439 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:26:20,274 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:26:23,440 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:26:25,275 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:26:28,441 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:26:30,276 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:26:33,442 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:26:35,277 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:26:38,443 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:26:40,278 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:26:43,445 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:26:45,279 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:26:48,446 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:26:50,280 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:26:53,447 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:26:55,281 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:26:58,448 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:27:00,282 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:27:03,449 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:27:05,283 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:27:08,450 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:27:10,284 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:27:13,451 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:27:15,284 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:27:18,452 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:27:20,285 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:27:23,453 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:27:25,286 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:27:28,454 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:27:30,287 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:27:33,455 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:27:35,289 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:27:38,457 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:27:40,290 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:27:43,458 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:27:45,291 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:27:48,459 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:27:50,291 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:27:53,460 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:27:55,292 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:27:58,461 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:28:00,293 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:28:03,462 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:28:05,294 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:28:08,463 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:28:10,295 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:28:13,464 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:28:15,296 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:28:18,465 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:28:20,297 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:28:23,466 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:28:25,298 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:28:28,467 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:28:30,299 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:28:33,469 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:28:35,300 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:28:38,470 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:28:40,301 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:28:43,470 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:28:45,302 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:28:48,471 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:28:50,303 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:28:53,472 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:28:55,304 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:28:58,473 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:29:00,305 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:29:03,474 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:29:05,306 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:29:08,475 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:29:10,307 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:29:13,476 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:29:15,308 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:29:18,477 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:29:20,310 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:29:23,479 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:29:25,311 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:29:28,480 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:29:30,312 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:29:33,481 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:29:35,313 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:29:38,482 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:29:40,314 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:29:43,483 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:29:45,315 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:29:48,483 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:29:50,315 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:29:53,484 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:29:55,316 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:29:58,485 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:30:00,317 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:30:03,486 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:30:05,318 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:30:08,487 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:30:10,319 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:30:13,488 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:30:15,320 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:30:18,489 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:30:20,321 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:30:23,490 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:30:25,322 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:30:28,491 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:30:30,324 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:30:33,492 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:30:35,324 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:30:38,494 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:30:40,325 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:30:43,495 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:30:45,326 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:30:48,496 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:30:50,327 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:30:53,497 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:30:55,328 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:30:58,498 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:31:00,329 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:31:03,499 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:31:05,330 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:31:08,501 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:31:10,330 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:31:13,502 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:31:15,331 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:31:18,503 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:31:20,332 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:31:23,504 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:31:25,333 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:31:28,505 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:31:30,334 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:31:33,506 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:31:35,335 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:31:38,507 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:31:40,335 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:31:43,508 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:31:45,336 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:31:48,509 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:31:50,337 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:31:53,510 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:31:55,339 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:31:58,511 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:32:00,340 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:32:03,512 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:32:05,340 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:32:08,513 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:32:10,342 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:32:13,514 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:32:15,342 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:32:18,515 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:32:20,343 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:32:23,516 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:32:25,344 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:32:28,518 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:32:30,345 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:32:33,519 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:32:35,346 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:32:38,520 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:32:40,347 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:32:43,521 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:32:45,348 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:32:45,841 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: resume 2024-06-03 20:32:45,841 INFO HandlerThread:322 [handler.py:handle_request_resume():715] starting system metrics thread 2024-06-03 20:32:45,841 INFO HandlerThread:322 [system_monitor.py:start():194] Starting system monitor 2024-06-03 20:32:45,841 INFO SystemMonitor:322 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-06-03 20:32:45,842 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: pause 2024-06-03 20:32:45,842 INFO SystemMonitor:322 [interfaces.py:start():188] Started cpu monitoring 2024-06-03 20:32:45,842 INFO HandlerThread:322 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-06-03 20:32:45,843 INFO HandlerThread:322 [system_monitor.py:finish():203] Stopping system monitor 2024-06-03 20:32:45,843 INFO SystemMonitor:322 [interfaces.py:start():188] Started disk monitoring 2024-06-03 20:32:45,844 DEBUG SystemMonitor:322 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-06-03 20:32:45,844 DEBUG SystemMonitor:322 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-06-03 20:32:45,844 INFO HandlerThread:322 [interfaces.py:finish():200] Joined cpu monitor 2024-06-03 20:32:45,846 DEBUG SystemMonitor:322 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-06-03 20:32:45,846 INFO HandlerThread:322 [interfaces.py:finish():200] Joined disk monitor 2024-06-03 20:32:45,847 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 20:32:48,522 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:32:50,848 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:32:53,523 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:32:55,849 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:32:58,524 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:33:00,850 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:33:03,525 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:33:05,851 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:33:08,526 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:33:10,852 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:33:13,528 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:33:15,853 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:33:18,528 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:33:20,854 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:33:23,529 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:33:25,855 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:33:28,530 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:33:30,856 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:33:33,531 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:33:35,857 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:33:38,532 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:33:40,858 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:33:43,533 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:33:45,859 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:33:48,534 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:33:50,860 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:33:53,535 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:33:55,861 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:33:58,537 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:34:00,862 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:34:03,538 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:34:05,863 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:34:08,539 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:34:10,864 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:34:13,540 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:34:15,865 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:34:18,541 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:34:19,719 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: resume 2024-06-03 20:34:19,719 INFO HandlerThread:322 [handler.py:handle_request_resume():715] starting system metrics thread 2024-06-03 20:34:19,719 INFO HandlerThread:322 [system_monitor.py:start():194] Starting system monitor 2024-06-03 20:34:19,719 INFO SystemMonitor:322 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-06-03 20:34:19,720 INFO SystemMonitor:322 [interfaces.py:start():188] Started cpu monitoring 2024-06-03 20:34:19,721 INFO SystemMonitor:322 [interfaces.py:start():188] Started disk monitoring 2024-06-03 20:34:19,721 INFO SystemMonitor:322 [interfaces.py:start():188] Started gpu monitoring 2024-06-03 20:34:19,723 INFO SystemMonitor:322 [interfaces.py:start():188] Started memory monitoring 2024-06-03 20:34:19,724 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: pause 2024-06-03 20:34:19,724 INFO HandlerThread:322 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-06-03 20:34:19,724 INFO HandlerThread:322 [system_monitor.py:finish():203] Stopping system monitor 2024-06-03 20:34:19,724 INFO HandlerThread:322 [interfaces.py:finish():200] Joined cpu monitor 2024-06-03 20:34:19,724 INFO SystemMonitor:322 [interfaces.py:start():188] Started network monitoring 2024-06-03 20:34:19,725 DEBUG SystemMonitor:322 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-06-03 20:34:19,725 DEBUG SystemMonitor:322 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-06-03 20:34:19,725 DEBUG SystemMonitor:322 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-06-03 20:34:19,727 INFO HandlerThread:322 [interfaces.py:finish():200] Joined disk monitor 2024-06-03 20:34:19,748 INFO HandlerThread:322 [interfaces.py:finish():200] Joined gpu monitor 2024-06-03 20:34:19,748 INFO HandlerThread:322 [interfaces.py:finish():200] Joined memory monitor 2024-06-03 20:34:19,748 INFO HandlerThread:322 [interfaces.py:finish():200] Joined network monitor 2024-06-03 20:34:19,748 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 20:34:21,602 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: resume 2024-06-03 20:34:21,602 INFO HandlerThread:322 [handler.py:handle_request_resume():715] starting system metrics thread 2024-06-03 20:34:21,602 INFO HandlerThread:322 [system_monitor.py:start():194] Starting system monitor 2024-06-03 20:34:21,602 INFO SystemMonitor:322 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-06-03 20:34:21,603 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: pause 2024-06-03 20:34:21,603 INFO HandlerThread:322 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-06-03 20:34:21,603 INFO HandlerThread:322 [system_monitor.py:finish():203] Stopping system monitor 2024-06-03 20:34:21,603 INFO SystemMonitor:322 [interfaces.py:start():188] Started cpu monitoring 2024-06-03 20:34:21,603 DEBUG SystemMonitor:322 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-06-03 20:34:21,604 DEBUG SystemMonitor:322 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-06-03 20:34:21,604 DEBUG SystemMonitor:322 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-06-03 20:34:21,605 INFO HandlerThread:322 [interfaces.py:finish():200] Joined cpu monitor 2024-06-03 20:34:21,606 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 20:34:21,606 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:34:23,188 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: resume 2024-06-03 20:34:23,188 INFO HandlerThread:322 [handler.py:handle_request_resume():715] starting system metrics thread 2024-06-03 20:34:23,188 INFO HandlerThread:322 [system_monitor.py:start():194] Starting system monitor 2024-06-03 20:34:23,188 INFO SystemMonitor:322 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-06-03 20:34:23,189 INFO SystemMonitor:322 [interfaces.py:start():188] Started cpu monitoring 2024-06-03 20:34:23,189 INFO SystemMonitor:322 [interfaces.py:start():188] Started disk monitoring 2024-06-03 20:34:23,190 INFO SystemMonitor:322 [interfaces.py:start():188] Started gpu monitoring 2024-06-03 20:34:23,192 INFO SystemMonitor:322 [interfaces.py:start():188] Started memory monitoring 2024-06-03 20:34:23,194 INFO SystemMonitor:322 [interfaces.py:start():188] Started network monitoring 2024-06-03 20:34:23,542 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:34:25,310 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/output.log 2024-06-03 20:34:27,194 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:34:28,544 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:34:32,195 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:34:33,545 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:34:37,197 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:34:38,546 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:34:42,198 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:34:43,548 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:34:47,199 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:34:48,549 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:34:52,201 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:34:53,551 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:34:57,202 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:34:58,552 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:35:02,205 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:35:03,554 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:35:07,211 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:35:08,555 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:35:12,213 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:35:13,557 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:35:17,214 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:35:18,558 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:35:22,215 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:35:23,194 DEBUG SystemMonitor:322 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-06-03 20:35:23,196 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 20:35:23,559 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:35:28,200 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:35:28,561 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:35:33,201 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:35:33,562 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:35:38,202 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:35:38,563 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:35:43,203 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:35:43,564 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:35:48,205 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:35:48,566 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:35:53,202 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 20:35:53,568 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:35:54,206 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:35:58,569 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:35:59,204 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:36:03,570 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:36:04,205 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:36:08,573 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:36:09,207 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:36:13,574 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:36:14,208 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:36:18,576 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:36:19,210 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:36:23,200 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 20:36:23,577 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:36:25,202 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:36:28,579 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:36:30,203 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:36:33,580 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:36:35,204 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:36:38,581 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:36:40,206 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:36:43,583 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:36:45,207 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:36:48,584 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:36:50,208 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:36:53,202 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 20:36:53,585 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:36:56,204 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:36:58,587 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:37:01,205 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:37:03,588 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:37:06,206 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:37:08,589 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:37:11,207 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:37:13,591 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:37:16,209 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:37:18,592 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:37:21,210 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:37:23,203 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 20:37:23,593 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:37:27,205 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:37:28,595 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:37:32,206 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:37:33,596 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:37:37,208 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:37:38,597 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:37:42,209 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:37:43,599 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:37:47,210 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:37:48,398 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: pause 2024-06-03 20:37:48,398 INFO HandlerThread:322 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-06-03 20:37:48,398 INFO HandlerThread:322 [system_monitor.py:finish():203] Stopping system monitor 2024-06-03 20:37:48,398 DEBUG SystemMonitor:322 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-06-03 20:37:48,401 DEBUG SystemMonitor:322 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-06-03 20:37:48,403 INFO HandlerThread:322 [interfaces.py:finish():200] Joined cpu monitor 2024-06-03 20:37:48,403 INFO HandlerThread:322 [interfaces.py:finish():200] Joined disk monitor 2024-06-03 20:37:48,413 INFO HandlerThread:322 [interfaces.py:finish():200] Joined gpu monitor 2024-06-03 20:37:48,414 INFO HandlerThread:322 [interfaces.py:finish():200] Joined memory monitor 2024-06-03 20:37:48,414 INFO HandlerThread:322 [interfaces.py:finish():200] Joined network monitor 2024-06-03 20:37:48,414 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 20:37:48,600 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:37:50,444 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/output.log 2024-06-03 20:37:52,415 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:37:53,601 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:37:57,416 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:37:58,602 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:38:02,417 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:38:03,603 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:38:07,418 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:38:08,604 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:38:12,419 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:38:13,605 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:38:17,420 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:38:18,606 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:38:22,421 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:38:23,607 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:38:27,422 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:38:28,608 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:38:32,423 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:38:33,503 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: resume 2024-06-03 20:38:33,503 INFO HandlerThread:322 [handler.py:handle_request_resume():715] starting system metrics thread 2024-06-03 20:38:33,503 INFO HandlerThread:322 [system_monitor.py:start():194] Starting system monitor 2024-06-03 20:38:33,503 INFO SystemMonitor:322 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-06-03 20:38:33,504 INFO SystemMonitor:322 [interfaces.py:start():188] Started cpu monitoring 2024-06-03 20:38:33,505 INFO SystemMonitor:322 [interfaces.py:start():188] Started disk monitoring 2024-06-03 20:38:33,505 INFO SystemMonitor:322 [interfaces.py:start():188] Started gpu monitoring 2024-06-03 20:38:33,508 INFO SystemMonitor:322 [interfaces.py:start():188] Started memory monitoring 2024-06-03 20:38:33,508 INFO SystemMonitor:322 [interfaces.py:start():188] Started network monitoring 2024-06-03 20:38:33,610 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:38:35,461 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/output.log 2024-06-03 20:38:37,511 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:38:38,611 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:38:42,512 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:38:43,612 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:38:47,513 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:38:48,614 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:38:52,514 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:38:53,616 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:38:57,516 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:38:58,617 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:39:02,517 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:39:03,619 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:39:07,518 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:39:08,620 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:39:12,519 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:39:13,622 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:39:17,520 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:39:18,623 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:39:22,522 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:39:23,624 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:39:27,523 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:39:28,626 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:39:32,524 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:39:33,509 DEBUG SystemMonitor:322 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-06-03 20:39:33,512 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 20:39:33,627 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:39:38,515 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:39:38,629 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:39:43,516 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:39:43,630 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:39:48,517 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:39:48,631 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:39:53,518 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:39:53,633 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:39:58,519 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:39:58,634 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:40:03,514 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 20:40:03,635 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:40:04,515 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:40:08,637 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:40:09,516 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:40:13,638 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:40:14,518 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:40:18,640 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:40:19,519 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:40:23,641 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:40:24,520 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:40:28,642 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:40:29,521 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:40:33,515 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 20:40:33,644 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:40:35,516 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:40:38,645 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:40:40,517 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:40:43,646 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:40:45,519 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:40:48,648 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:40:50,520 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:40:53,649 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:40:55,521 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:40:58,651 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:41:00,522 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:41:03,517 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 20:41:03,652 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:41:06,520 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:41:08,654 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:41:11,521 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:41:13,655 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:41:16,522 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:41:18,657 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:41:21,523 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:41:23,658 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:41:26,524 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:41:28,659 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:41:31,526 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:41:33,518 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 20:41:33,661 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:41:37,519 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:41:38,662 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:41:42,521 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:41:43,663 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:41:47,522 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:41:48,665 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:41:52,523 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:41:53,666 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:41:57,524 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:41:58,667 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:41:58,863 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: pause 2024-06-03 20:41:58,863 INFO HandlerThread:322 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-06-03 20:41:58,863 INFO HandlerThread:322 [system_monitor.py:finish():203] Stopping system monitor 2024-06-03 20:41:58,864 DEBUG SystemMonitor:322 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-06-03 20:41:58,864 DEBUG SystemMonitor:322 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-06-03 20:41:58,865 INFO HandlerThread:322 [interfaces.py:finish():200] Joined cpu monitor 2024-06-03 20:41:58,865 INFO HandlerThread:322 [interfaces.py:finish():200] Joined disk monitor 2024-06-03 20:41:58,876 INFO HandlerThread:322 [interfaces.py:finish():200] Joined gpu monitor 2024-06-03 20:41:58,876 INFO HandlerThread:322 [interfaces.py:finish():200] Joined memory monitor 2024-06-03 20:41:58,877 INFO HandlerThread:322 [interfaces.py:finish():200] Joined network monitor 2024-06-03 20:41:58,877 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 20:42:00,607 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/output.log 2024-06-03 20:42:02,878 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:42:03,668 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:42:07,879 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:42:08,669 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:42:12,880 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:42:13,670 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:42:17,881 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:42:18,671 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:42:22,882 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:42:23,672 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:42:27,883 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:42:28,673 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:42:32,884 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:42:33,675 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:42:37,885 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:42:38,676 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:42:42,886 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:42:43,677 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:42:47,887 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:42:48,678 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:42:52,888 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:42:53,679 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:42:57,889 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:42:58,680 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:43:02,890 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:43:03,681 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:43:07,891 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:43:08,682 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:43:12,892 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:43:13,683 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:43:17,893 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:43:18,684 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:43:22,894 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:43:23,685 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:43:27,894 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:43:28,686 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:43:32,895 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:43:33,687 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:43:37,896 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:43:38,688 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:43:42,897 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:43:43,689 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:43:47,898 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:43:48,690 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:43:51,168 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: resume 2024-06-03 20:43:51,168 INFO HandlerThread:322 [handler.py:handle_request_resume():715] starting system metrics thread 2024-06-03 20:43:51,168 INFO HandlerThread:322 [system_monitor.py:start():194] Starting system monitor 2024-06-03 20:43:51,169 INFO SystemMonitor:322 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-06-03 20:43:51,169 INFO SystemMonitor:322 [interfaces.py:start():188] Started cpu monitoring 2024-06-03 20:43:51,170 INFO SystemMonitor:322 [interfaces.py:start():188] Started disk monitoring 2024-06-03 20:43:51,171 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: pause 2024-06-03 20:43:51,171 INFO HandlerThread:322 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-06-03 20:43:51,171 INFO HandlerThread:322 [system_monitor.py:finish():203] Stopping system monitor 2024-06-03 20:43:51,172 INFO SystemMonitor:322 [interfaces.py:start():188] Started gpu monitoring 2024-06-03 20:43:51,172 INFO HandlerThread:322 [interfaces.py:finish():200] Joined cpu monitor 2024-06-03 20:43:51,172 DEBUG SystemMonitor:322 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-06-03 20:43:51,173 DEBUG SystemMonitor:322 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-06-03 20:43:51,173 DEBUG SystemMonitor:322 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-06-03 20:43:51,175 INFO HandlerThread:322 [interfaces.py:finish():200] Joined disk monitor 2024-06-03 20:43:51,193 INFO HandlerThread:322 [interfaces.py:finish():200] Joined gpu monitor 2024-06-03 20:43:51,194 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 20:43:53,195 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:43:53,691 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:43:53,896 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: resume 2024-06-03 20:43:53,896 INFO HandlerThread:322 [handler.py:handle_request_resume():715] starting system metrics thread 2024-06-03 20:43:53,896 INFO HandlerThread:322 [system_monitor.py:start():194] Starting system monitor 2024-06-03 20:43:53,896 INFO SystemMonitor:322 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-06-03 20:43:53,897 INFO SystemMonitor:322 [interfaces.py:start():188] Started cpu monitoring 2024-06-03 20:43:53,897 INFO SystemMonitor:322 [interfaces.py:start():188] Started disk monitoring 2024-06-03 20:43:53,898 INFO SystemMonitor:322 [interfaces.py:start():188] Started gpu monitoring 2024-06-03 20:43:53,899 INFO SystemMonitor:322 [interfaces.py:start():188] Started memory monitoring 2024-06-03 20:43:53,900 INFO SystemMonitor:322 [interfaces.py:start():188] Started network monitoring 2024-06-03 20:43:56,651 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/output.log 2024-06-03 20:43:58,692 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:43:58,904 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:44:03,694 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:44:03,905 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:44:08,695 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:44:08,906 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:44:13,696 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:44:13,908 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:44:18,697 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:44:18,909 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:44:23,699 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:44:23,910 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:44:28,700 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:44:28,911 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:44:33,702 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:44:33,912 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:44:38,703 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:44:38,914 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:44:43,704 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:44:43,915 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:44:48,706 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:44:48,916 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:44:53,707 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:44:53,901 DEBUG SystemMonitor:322 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-06-03 20:44:53,903 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 20:44:54,905 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:44:58,708 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:44:59,906 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:45:03,710 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:45:04,908 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:45:08,711 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:45:09,911 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:45:13,712 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:45:14,913 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:45:18,713 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:45:19,915 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:45:23,715 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:45:23,904 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 20:45:25,908 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:45:28,716 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:45:30,910 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:45:33,717 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:45:35,911 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:45:38,719 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:45:40,912 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:45:43,720 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:45:45,913 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:45:48,722 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:45:50,915 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:45:53,723 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:45:53,906 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 20:45:56,908 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:45:58,725 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:46:01,909 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:46:03,726 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:46:06,910 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:46:08,728 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:46:11,911 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:46:13,729 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:46:16,912 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:46:18,730 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:46:21,914 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:46:23,732 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:46:23,908 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 20:46:27,910 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:46:28,734 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:46:32,911 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:46:33,735 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:46:37,912 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:46:38,736 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:46:42,913 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:46:43,738 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:46:47,915 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:46:48,739 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:46:52,916 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:46:53,741 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:46:53,910 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 20:46:58,742 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:46:58,912 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:47:03,743 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:47:03,913 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:47:08,745 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:47:08,914 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:47:13,746 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:47:13,915 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:47:18,747 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:47:18,919 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:47:23,749 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:47:23,911 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 20:47:24,913 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:47:25,896 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: pause 2024-06-03 20:47:25,896 INFO HandlerThread:322 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-06-03 20:47:25,896 INFO HandlerThread:322 [system_monitor.py:finish():203] Stopping system monitor 2024-06-03 20:47:25,896 DEBUG SystemMonitor:322 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-06-03 20:47:25,896 DEBUG SystemMonitor:322 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-06-03 20:47:25,898 INFO HandlerThread:322 [interfaces.py:finish():200] Joined cpu monitor 2024-06-03 20:47:25,898 INFO HandlerThread:322 [interfaces.py:finish():200] Joined disk monitor 2024-06-03 20:47:25,911 INFO HandlerThread:322 [interfaces.py:finish():200] Joined gpu monitor 2024-06-03 20:47:25,911 INFO HandlerThread:322 [interfaces.py:finish():200] Joined memory monitor 2024-06-03 20:47:25,911 INFO HandlerThread:322 [interfaces.py:finish():200] Joined network monitor 2024-06-03 20:47:25,911 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 20:47:28,750 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:47:28,793 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/output.log 2024-06-03 20:47:30,912 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:47:33,751 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:47:35,914 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:47:38,752 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:47:40,915 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:47:43,753 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:47:45,916 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:47:48,755 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:47:50,917 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:47:53,755 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:47:55,918 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:47:58,757 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:48:00,919 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:48:03,758 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:48:05,920 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:48:08,759 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:48:10,921 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:48:13,760 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:48:15,922 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:48:18,761 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:48:20,923 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:48:23,763 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:48:25,924 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:48:28,764 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:48:30,925 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:48:33,765 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:48:35,925 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:48:38,766 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:48:40,926 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:48:43,767 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:48:45,927 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:48:48,768 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:48:50,928 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:48:53,769 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:48:55,929 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:48:58,770 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:49:00,929 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:49:03,771 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:49:05,930 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:49:08,772 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:49:10,931 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:49:13,774 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:49:15,932 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:49:18,775 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:49:20,933 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:49:23,776 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:49:25,934 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:49:28,777 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:49:30,935 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:49:33,778 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:49:35,936 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:49:38,780 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:49:40,937 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:49:43,781 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:49:45,938 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:49:48,782 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:49:50,939 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:49:53,783 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:49:55,940 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:49:58,784 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:50:00,941 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:50:03,785 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:50:05,943 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:50:07,262 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: resume 2024-06-03 20:50:07,263 INFO HandlerThread:322 [handler.py:handle_request_resume():715] starting system metrics thread 2024-06-03 20:50:07,263 INFO HandlerThread:322 [system_monitor.py:start():194] Starting system monitor 2024-06-03 20:50:07,263 INFO SystemMonitor:322 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-06-03 20:50:07,264 INFO SystemMonitor:322 [interfaces.py:start():188] Started cpu monitoring 2024-06-03 20:50:07,264 INFO SystemMonitor:322 [interfaces.py:start():188] Started disk monitoring 2024-06-03 20:50:07,265 INFO SystemMonitor:322 [interfaces.py:start():188] Started gpu monitoring 2024-06-03 20:50:07,266 INFO SystemMonitor:322 [interfaces.py:start():188] Started memory monitoring 2024-06-03 20:50:07,268 INFO SystemMonitor:322 [interfaces.py:start():188] Started network monitoring 2024-06-03 20:50:07,304 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: pause 2024-06-03 20:50:07,304 INFO HandlerThread:322 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-06-03 20:50:07,304 INFO HandlerThread:322 [system_monitor.py:finish():203] Stopping system monitor 2024-06-03 20:50:07,304 DEBUG SystemMonitor:322 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-06-03 20:50:07,304 DEBUG SystemMonitor:322 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-06-03 20:50:07,304 DEBUG SystemMonitor:322 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-06-03 20:50:07,305 INFO HandlerThread:322 [interfaces.py:finish():200] Joined cpu monitor 2024-06-03 20:50:07,305 INFO HandlerThread:322 [interfaces.py:finish():200] Joined disk monitor 2024-06-03 20:50:07,315 INFO HandlerThread:322 [interfaces.py:finish():200] Joined gpu monitor 2024-06-03 20:50:07,315 INFO HandlerThread:322 [interfaces.py:finish():200] Joined memory monitor 2024-06-03 20:50:07,316 INFO HandlerThread:322 [interfaces.py:finish():200] Joined network monitor 2024-06-03 20:50:07,319 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 20:50:08,786 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:50:09,916 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: resume 2024-06-03 20:50:09,916 INFO HandlerThread:322 [handler.py:handle_request_resume():715] starting system metrics thread 2024-06-03 20:50:09,916 INFO HandlerThread:322 [system_monitor.py:start():194] Starting system monitor 2024-06-03 20:50:09,917 INFO SystemMonitor:322 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-06-03 20:50:09,917 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: pause 2024-06-03 20:50:09,918 INFO HandlerThread:322 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-06-03 20:50:09,918 INFO HandlerThread:322 [system_monitor.py:finish():203] Stopping system monitor 2024-06-03 20:50:09,918 INFO SystemMonitor:322 [interfaces.py:start():188] Started cpu monitoring 2024-06-03 20:50:09,918 DEBUG SystemMonitor:322 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-06-03 20:50:09,918 DEBUG SystemMonitor:322 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-06-03 20:50:09,918 DEBUG SystemMonitor:322 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-06-03 20:50:09,919 INFO HandlerThread:322 [interfaces.py:finish():200] Joined cpu monitor 2024-06-03 20:50:09,920 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 20:50:10,464 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: resume 2024-06-03 20:50:10,465 INFO HandlerThread:322 [handler.py:handle_request_resume():715] starting system metrics thread 2024-06-03 20:50:10,465 INFO HandlerThread:322 [system_monitor.py:start():194] Starting system monitor 2024-06-03 20:50:10,465 INFO SystemMonitor:322 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-06-03 20:50:10,465 INFO SystemMonitor:322 [interfaces.py:start():188] Started cpu monitoring 2024-06-03 20:50:10,466 INFO SystemMonitor:322 [interfaces.py:start():188] Started disk monitoring 2024-06-03 20:50:10,467 INFO SystemMonitor:322 [interfaces.py:start():188] Started gpu monitoring 2024-06-03 20:50:10,469 INFO SystemMonitor:322 [interfaces.py:start():188] Started memory monitoring 2024-06-03 20:50:10,469 INFO SystemMonitor:322 [interfaces.py:start():188] Started network monitoring 2024-06-03 20:50:10,485 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: pause 2024-06-03 20:50:10,485 INFO HandlerThread:322 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-06-03 20:50:10,485 INFO HandlerThread:322 [system_monitor.py:finish():203] Stopping system monitor 2024-06-03 20:50:10,485 INFO HandlerThread:322 [interfaces.py:finish():200] Joined cpu monitor 2024-06-03 20:50:10,486 DEBUG SystemMonitor:322 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-06-03 20:50:10,486 INFO HandlerThread:322 [interfaces.py:finish():200] Joined disk monitor 2024-06-03 20:50:10,486 DEBUG SystemMonitor:322 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-06-03 20:50:10,486 DEBUG SystemMonitor:322 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-06-03 20:50:10,501 INFO HandlerThread:322 [interfaces.py:finish():200] Joined gpu monitor 2024-06-03 20:50:10,502 INFO HandlerThread:322 [interfaces.py:finish():200] Joined memory monitor 2024-06-03 20:50:10,502 INFO HandlerThread:322 [interfaces.py:finish():200] Joined network monitor 2024-06-03 20:50:10,502 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 20:50:11,503 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:50:13,787 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:50:13,976 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: resume 2024-06-03 20:50:13,976 INFO HandlerThread:322 [handler.py:handle_request_resume():715] starting system metrics thread 2024-06-03 20:50:13,976 INFO HandlerThread:322 [system_monitor.py:start():194] Starting system monitor 2024-06-03 20:50:13,976 INFO SystemMonitor:322 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-06-03 20:50:13,977 INFO SystemMonitor:322 [interfaces.py:start():188] Started cpu monitoring 2024-06-03 20:50:13,977 INFO SystemMonitor:322 [interfaces.py:start():188] Started disk monitoring 2024-06-03 20:50:13,978 INFO SystemMonitor:322 [interfaces.py:start():188] Started gpu monitoring 2024-06-03 20:50:13,978 INFO SystemMonitor:322 [interfaces.py:start():188] Started memory monitoring 2024-06-03 20:50:13,980 INFO SystemMonitor:322 [interfaces.py:start():188] Started network monitoring 2024-06-03 20:50:14,119 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: pause 2024-06-03 20:50:14,119 INFO HandlerThread:322 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-06-03 20:50:14,120 INFO HandlerThread:322 [system_monitor.py:finish():203] Stopping system monitor 2024-06-03 20:50:14,120 DEBUG SystemMonitor:322 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-06-03 20:50:14,120 DEBUG SystemMonitor:322 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-06-03 20:50:14,120 DEBUG SystemMonitor:322 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-06-03 20:50:14,121 INFO HandlerThread:322 [interfaces.py:finish():200] Joined cpu monitor 2024-06-03 20:50:14,121 INFO HandlerThread:322 [interfaces.py:finish():200] Joined disk monitor 2024-06-03 20:50:14,132 INFO HandlerThread:322 [interfaces.py:finish():200] Joined gpu monitor 2024-06-03 20:50:14,132 INFO HandlerThread:322 [interfaces.py:finish():200] Joined memory monitor 2024-06-03 20:50:14,132 INFO HandlerThread:322 [interfaces.py:finish():200] Joined network monitor 2024-06-03 20:50:14,133 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 20:50:15,413 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: resume 2024-06-03 20:50:15,413 INFO HandlerThread:322 [handler.py:handle_request_resume():715] starting system metrics thread 2024-06-03 20:50:15,413 INFO HandlerThread:322 [system_monitor.py:start():194] Starting system monitor 2024-06-03 20:50:15,413 INFO SystemMonitor:322 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-06-03 20:50:15,414 INFO SystemMonitor:322 [interfaces.py:start():188] Started cpu monitoring 2024-06-03 20:50:15,414 INFO SystemMonitor:322 [interfaces.py:start():188] Started disk monitoring 2024-06-03 20:50:15,415 INFO SystemMonitor:322 [interfaces.py:start():188] Started gpu monitoring 2024-06-03 20:50:15,416 INFO SystemMonitor:322 [interfaces.py:start():188] Started memory monitoring 2024-06-03 20:50:15,416 INFO SystemMonitor:322 [interfaces.py:start():188] Started network monitoring 2024-06-03 20:50:16,877 DEBUG SenderThread:322 [sender.py:send():378] send: config 2024-06-03 20:50:16,878 DEBUG SenderThread:322 [sender.py:send():378] send: metric 2024-06-03 20:50:16,879 DEBUG SenderThread:322 [sender.py:send():378] send: metric 2024-06-03 20:50:16,879 WARNING SenderThread:322 [sender.py:send_metric():1340] Seen metric with glob (shouldn't happen) 2024-06-03 20:50:16,879 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:50:18,858 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/output.log 2024-06-03 20:50:18,876 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:50:21,921 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:50:23,860 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/config.yaml 2024-06-03 20:50:23,877 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:50:28,261 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:50:28,878 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:50:33,262 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:50:33,904 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:50:38,263 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:50:39,341 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:50:39,342 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:50:43,264 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:50:44,343 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:50:48,265 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:50:49,344 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:50:53,266 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:50:54,345 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:50:58,267 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:50:59,758 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:51:03,268 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:51:04,760 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:51:08,269 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:51:09,761 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:51:13,270 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:51:14,762 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:51:15,417 DEBUG SystemMonitor:322 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-06-03 20:51:15,418 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 20:51:18,419 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:51:19,854 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:51:23,420 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:51:24,855 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:51:28,421 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:51:29,856 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:51:33,422 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:51:34,857 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:51:38,423 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:51:40,183 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:51:40,184 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:51:43,424 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:51:45,185 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:51:45,421 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 20:51:49,422 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:51:50,186 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:51:54,423 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:51:55,187 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:51:59,424 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:52:00,188 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:52:04,425 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:52:05,189 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:52:09,426 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:52:10,190 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:52:14,427 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:52:15,191 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:52:15,420 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 20:52:20,192 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:52:20,421 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:52:25,413 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:52:25,422 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:52:30,414 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:52:30,422 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:52:35,415 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:52:35,423 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:52:40,416 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:52:40,424 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:52:45,421 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 20:52:45,828 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:52:46,422 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:52:50,828 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:52:51,422 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:52:55,830 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:52:56,423 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:53:00,831 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:53:01,424 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:53:05,908 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:53:06,425 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:53:10,909 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:53:11,426 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:53:15,422 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 20:53:15,910 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:53:17,423 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:53:20,911 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:53:22,423 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:53:25,951 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:53:27,424 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:53:31,233 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:53:32,425 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:53:36,234 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:53:37,426 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:53:41,235 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:53:42,427 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:53:45,422 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 20:53:46,606 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:53:48,423 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:53:51,607 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:53:53,424 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:53:55,930 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: partial_history 2024-06-03 20:53:55,931 DEBUG SenderThread:322 [sender.py:send():378] send: history 2024-06-03 20:53:55,932 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: summary_record 2024-06-03 20:53:55,932 INFO SenderThread:322 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-06-03 20:53:55,941 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/wandb-summary.json 2024-06-03 20:53:56,608 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:53:58,933 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:54:01,609 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:54:02,943 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/output.log 2024-06-03 20:54:04,353 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:54:06,610 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:54:09,354 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:54:11,611 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:54:14,354 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:54:15,423 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 20:54:16,611 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:54:19,424 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:54:21,612 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:54:24,425 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:54:26,614 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:54:29,426 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:54:31,615 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:54:34,427 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:54:36,616 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:54:39,428 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:54:41,617 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:54:44,429 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:54:45,424 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 20:54:46,618 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:54:50,425 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:54:51,987 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:54:55,426 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:54:56,988 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:55:00,427 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:55:01,990 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:55:05,428 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:55:06,991 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:55:10,429 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:55:12,262 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:55:12,263 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:55:15,425 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 20:55:16,426 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:55:17,263 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:55:21,427 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:55:22,264 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:55:26,428 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:55:27,265 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:55:31,429 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:55:32,508 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:55:36,429 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:55:37,509 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:55:41,430 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:55:42,510 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:55:45,426 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 20:55:47,427 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:55:47,511 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:55:52,428 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:55:52,512 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:55:57,428 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:55:57,513 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:56:02,429 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:56:02,514 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:56:07,430 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:56:07,515 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:56:12,431 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:56:12,516 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:56:15,427 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 20:56:17,517 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:56:18,428 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:56:22,518 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:56:23,429 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:56:27,520 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:56:28,430 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:56:32,520 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:56:33,431 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:56:37,522 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:56:38,431 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:56:42,523 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:56:43,432 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:56:45,427 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 20:56:47,524 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:56:49,429 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:56:52,525 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:56:54,430 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:56:57,526 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:56:59,431 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:57:02,527 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:57:04,432 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:57:05,665 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: partial_history 2024-06-03 20:57:05,666 DEBUG SenderThread:322 [sender.py:send():378] send: history 2024-06-03 20:57:05,667 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: summary_record 2024-06-03 20:57:05,667 INFO SenderThread:322 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-06-03 20:57:06,010 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/wandb-summary.json 2024-06-03 20:57:07,528 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:57:09,668 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:57:12,529 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:57:14,669 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:57:15,428 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 20:57:17,530 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:57:20,430 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:57:22,531 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:57:25,430 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:57:27,532 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:57:30,431 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:57:32,533 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:57:33,020 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/output.log 2024-06-03 20:57:36,192 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:57:37,534 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:57:41,193 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:57:42,535 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:57:45,429 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 20:57:46,430 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:57:47,536 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:57:51,431 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:57:52,539 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:57:56,432 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:57:57,540 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:58:01,433 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:58:02,541 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:58:06,434 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:58:07,542 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:58:11,435 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:58:12,543 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:58:15,430 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 20:58:17,431 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:58:17,544 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:58:22,432 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:58:22,545 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:58:27,433 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:58:27,546 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:58:32,434 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:58:32,547 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:58:37,435 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:58:37,548 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:58:42,435 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:58:42,567 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:58:45,431 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 20:58:47,568 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:58:48,432 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:58:52,569 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:58:53,433 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:58:57,570 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:58:58,434 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:59:02,913 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:59:03,435 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:59:07,914 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:59:08,436 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:59:12,915 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:59:13,437 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:59:15,432 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 20:59:17,916 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:59:19,433 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:59:22,917 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:59:24,434 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:59:27,919 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:59:29,435 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:59:32,919 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:59:34,436 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:59:37,920 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:59:39,436 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:59:42,921 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:59:44,437 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:59:45,432 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 20:59:47,922 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:59:50,434 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:59:52,923 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 20:59:55,435 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 20:59:57,924 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:00:00,436 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:00:02,925 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:00:05,436 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:00:07,926 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:00:10,437 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:00:12,927 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:00:15,213 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: partial_history 2024-06-03 21:00:15,215 DEBUG SenderThread:322 [sender.py:send():378] send: history 2024-06-03 21:00:15,215 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: summary_record 2024-06-03 21:00:15,215 INFO SenderThread:322 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-06-03 21:00:15,433 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:00:16,082 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/wandb-summary.json 2024-06-03 21:00:16,434 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:00:17,928 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:00:21,435 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:00:22,954 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:00:26,435 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:00:28,061 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:00:31,436 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:00:33,062 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:00:36,437 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:00:38,063 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:00:41,438 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:00:43,283 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:00:45,434 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:00:47,435 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:00:48,284 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:00:52,436 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:00:53,285 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:00:57,437 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:00:58,286 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:01:02,698 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:01:03,099 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/output.log 2024-06-03 21:01:03,287 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:01:07,699 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:01:08,531 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:01:12,700 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:01:13,532 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:01:15,439 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:01:18,440 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:01:18,533 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:01:23,441 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:01:23,534 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:01:28,442 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:01:28,535 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:01:33,443 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:01:33,536 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:01:38,444 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:01:38,537 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:01:43,445 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:01:43,537 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:01:45,436 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:01:48,538 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:01:49,437 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:01:53,539 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:01:54,438 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:01:58,540 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:01:59,439 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:02:03,541 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:02:04,439 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:02:08,542 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:02:09,441 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:02:13,544 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:02:14,442 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:02:15,438 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:02:18,545 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:02:20,440 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:02:23,546 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:02:25,441 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:02:28,547 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:02:30,442 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:02:33,548 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:02:35,443 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:02:38,549 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:02:40,443 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:02:43,550 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:02:45,438 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:02:46,439 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:02:48,551 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:02:51,440 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:02:53,641 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:02:56,440 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:02:58,643 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:03:01,441 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:03:03,644 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:03:06,442 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:03:08,645 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:03:11,443 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:03:13,646 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:03:15,439 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:03:17,440 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:03:18,647 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:03:22,441 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:03:23,647 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:03:24,543 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: partial_history 2024-06-03 21:03:24,545 DEBUG SenderThread:322 [sender.py:send():378] send: history 2024-06-03 21:03:24,545 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: summary_record 2024-06-03 21:03:24,545 INFO SenderThread:322 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-06-03 21:03:25,151 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/wandb-summary.json 2024-06-03 21:03:27,546 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:03:28,648 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:03:32,547 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:03:33,649 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:03:37,548 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:03:38,650 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:03:42,549 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:03:43,651 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:03:45,440 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:03:48,441 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:03:48,652 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:03:53,442 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:03:53,653 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:03:58,443 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:03:58,654 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:04:03,444 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:04:03,655 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:04:08,444 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:04:08,656 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:04:13,446 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:04:13,660 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:04:15,440 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:04:18,661 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:04:19,442 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:04:23,662 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:04:24,442 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:04:28,663 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:04:29,443 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:04:33,176 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/output.log 2024-06-03 21:04:33,664 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:04:35,231 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:04:39,087 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:04:40,232 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:04:44,089 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:04:45,233 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:04:45,441 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:04:49,090 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:04:50,442 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:04:54,091 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:04:55,443 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:04:59,356 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:05:00,444 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:05:04,358 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:05:05,445 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:05:09,359 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:05:10,446 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:05:14,360 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:05:15,442 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:05:16,443 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:05:19,361 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:05:21,444 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:05:24,362 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:05:26,445 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:05:29,363 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:05:31,445 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:05:34,364 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:05:36,446 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:05:39,498 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:05:39,498 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:05:41,447 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:05:44,499 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:05:45,443 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:05:47,444 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:05:49,500 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:05:52,445 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:05:54,501 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:05:57,446 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:05:59,788 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:06:02,447 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:06:04,790 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:06:07,448 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:06:09,790 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:06:12,449 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:06:14,791 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:06:15,444 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:06:18,445 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:06:19,909 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:06:23,446 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:06:24,910 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:06:28,447 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:06:29,912 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:06:33,268 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: partial_history 2024-06-03 21:06:33,269 DEBUG SenderThread:322 [sender.py:send():378] send: history 2024-06-03 21:06:33,269 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: summary_record 2024-06-03 21:06:33,270 INFO SenderThread:322 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-06-03 21:06:34,221 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/wandb-summary.json 2024-06-03 21:06:34,270 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:06:34,914 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:06:39,271 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:06:40,116 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:06:40,117 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:06:44,272 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:06:45,118 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:06:45,445 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:06:49,446 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:06:50,119 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:06:54,447 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:06:55,120 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:06:59,447 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:07:00,121 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:07:04,449 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:07:05,122 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:07:09,449 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:07:10,123 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:07:14,450 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:07:15,124 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:07:15,446 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:07:20,125 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:07:20,447 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:07:25,126 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:07:25,448 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:07:30,127 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:07:30,448 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:07:35,128 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:07:35,449 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:07:40,129 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:07:40,450 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:07:45,446 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:07:45,526 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:07:46,447 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:07:50,527 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:07:51,448 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:07:55,528 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:07:56,449 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:08:00,529 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:08:01,932 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:08:02,253 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/output.log 2024-06-03 21:08:05,530 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:08:06,933 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:08:10,793 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:08:11,934 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:08:15,448 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:08:15,794 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:08:17,449 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:08:20,795 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:08:22,450 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:08:25,797 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:08:27,451 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:08:31,203 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:08:32,452 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:08:36,204 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:08:37,453 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:08:41,205 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:08:42,453 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:08:45,449 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:08:46,206 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:08:48,450 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:08:51,284 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:08:53,450 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:08:56,285 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:08:58,451 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:09:01,286 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:09:03,452 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:09:06,287 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:09:08,453 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:09:11,288 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:09:13,454 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:09:15,450 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:09:16,289 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:09:19,451 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:09:21,290 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:09:24,452 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:09:26,292 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:09:29,452 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:09:31,293 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:09:34,453 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:09:36,294 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:09:39,454 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:09:41,295 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:09:42,530 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: partial_history 2024-06-03 21:09:42,531 DEBUG SenderThread:322 [sender.py:send():378] send: history 2024-06-03 21:09:42,531 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: summary_record 2024-06-03 21:09:42,532 INFO SenderThread:322 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-06-03 21:09:43,289 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/wandb-summary.json 2024-06-03 21:09:44,532 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:09:45,451 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:09:46,296 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:09:50,452 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:09:51,338 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:09:55,453 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:09:56,339 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:10:00,453 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:10:01,340 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:10:05,454 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:10:06,341 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:10:10,455 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:10:11,502 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:10:11,502 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:10:15,451 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:10:16,452 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:10:16,503 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:10:21,453 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:10:21,504 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:10:26,454 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:10:26,505 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:10:31,455 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:10:31,902 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:10:36,456 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:10:36,903 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:10:41,457 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:10:41,904 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:10:45,453 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:10:46,905 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:10:47,453 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:10:52,158 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:10:52,454 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:10:57,159 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:10:57,456 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:11:02,160 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:11:02,456 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:11:07,160 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:11:07,457 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:11:12,304 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:11:12,458 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:11:15,453 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:11:17,305 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:11:18,455 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:11:22,306 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:11:23,455 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:11:27,307 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:11:28,456 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:11:32,308 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:11:32,329 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/output.log 2024-06-03 21:11:34,282 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:11:37,309 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:11:39,283 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:11:42,310 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:11:44,284 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:11:45,454 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:11:47,311 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:11:49,455 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:11:52,312 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:11:54,457 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:11:57,534 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:11:59,458 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:12:02,535 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:12:04,459 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:12:07,536 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:12:09,460 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:12:12,537 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:12:14,460 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:12:15,455 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:12:17,538 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:12:20,456 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:12:22,539 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:12:25,457 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:12:27,540 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:12:30,458 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:12:32,541 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:12:35,459 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:12:37,542 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:12:40,459 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:12:42,861 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:12:45,456 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:12:46,457 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:12:47,862 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:12:51,458 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:12:52,546 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: partial_history 2024-06-03 21:12:52,547 DEBUG SenderThread:322 [sender.py:send():378] send: history 2024-06-03 21:12:52,547 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: summary_record 2024-06-03 21:12:52,548 INFO SenderThread:322 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-06-03 21:12:52,863 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:12:53,359 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/wandb-summary.json 2024-06-03 21:12:56,549 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:12:57,864 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:13:01,550 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:13:02,865 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:13:06,551 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:13:07,866 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:13:11,551 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:13:12,867 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:13:15,457 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:13:17,458 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:13:17,868 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:13:22,459 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:13:22,869 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:13:27,459 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:13:27,871 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:13:32,460 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:13:32,872 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:13:37,461 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:13:37,872 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:13:42,462 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:13:42,873 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:13:45,458 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:13:47,874 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:13:48,459 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:13:52,875 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:13:53,460 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:13:57,876 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:13:58,461 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:14:02,877 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:14:03,462 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:14:07,878 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:14:08,462 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:14:12,879 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:14:13,463 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:14:15,459 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:14:17,881 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:14:19,460 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:14:23,093 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:14:24,461 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:14:28,133 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:14:29,462 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:14:33,134 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:14:34,463 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:14:38,135 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:14:39,464 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:14:43,551 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:14:44,464 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:14:45,460 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:14:48,552 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:14:50,461 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:14:53,552 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:14:55,462 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:14:58,554 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:15:00,817 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:15:02,405 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/output.log 2024-06-03 21:15:03,554 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:15:05,818 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:15:08,649 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:15:10,819 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:15:13,650 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:15:15,461 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:15:16,462 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:15:18,651 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:15:21,463 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:15:23,652 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:15:26,463 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:15:28,914 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:15:31,464 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:15:33,915 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:15:36,465 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:15:38,916 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:15:41,466 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:15:43,917 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:15:45,462 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:15:47,463 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:15:49,064 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:15:52,464 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:15:54,066 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:15:57,464 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:15:59,066 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:16:00,672 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: partial_history 2024-06-03 21:16:00,673 DEBUG SenderThread:322 [sender.py:send():378] send: history 2024-06-03 21:16:00,673 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: summary_record 2024-06-03 21:16:00,674 INFO SenderThread:322 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-06-03 21:16:01,426 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/wandb-summary.json 2024-06-03 21:16:02,674 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:16:04,067 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:16:07,676 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:16:09,069 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:16:12,676 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:16:14,070 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:16:15,462 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:16:18,463 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:16:19,070 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:16:23,464 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:16:24,071 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:16:28,465 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:16:29,294 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:16:33,467 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:16:34,294 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:16:38,468 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:16:39,295 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:16:43,469 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:16:44,296 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:16:45,467 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:16:49,468 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:16:49,511 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:16:54,469 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:16:54,512 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:16:59,470 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:16:59,513 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:17:04,471 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:17:04,514 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:17:09,472 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:17:09,515 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:17:14,473 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:17:14,516 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:17:15,464 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:17:19,516 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:17:20,465 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:17:24,517 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:17:25,466 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:17:29,518 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:17:30,467 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:17:34,520 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:17:35,467 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:17:39,521 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:17:40,468 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:17:44,522 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:17:45,466 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:17:46,467 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:17:49,523 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:17:51,469 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:17:54,524 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:17:56,469 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:17:59,525 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:18:01,470 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:18:04,526 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:18:06,471 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:18:09,958 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:18:11,472 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:18:14,960 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:18:15,466 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:18:17,467 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:18:19,961 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:18:22,468 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:18:24,962 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:18:27,469 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:18:29,963 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:18:30,484 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/output.log 2024-06-03 21:18:33,190 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:18:34,964 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:18:38,191 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:18:40,012 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:18:43,192 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:18:45,013 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:18:45,467 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:18:48,468 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:18:50,014 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:18:53,468 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:18:55,015 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:18:58,469 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:19:00,264 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:19:03,470 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:19:05,265 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:19:08,471 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:19:09,758 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: partial_history 2024-06-03 21:19:09,759 DEBUG SenderThread:322 [sender.py:send():378] send: history 2024-06-03 21:19:09,760 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: summary_record 2024-06-03 21:19:09,760 INFO SenderThread:322 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-06-03 21:19:10,266 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:19:10,498 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/wandb-summary.json 2024-06-03 21:19:13,761 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:19:15,267 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:19:15,468 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:19:19,469 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:19:20,613 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:19:24,470 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:19:25,614 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:19:29,471 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:19:30,615 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:19:34,471 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:19:35,616 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:19:39,472 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:19:40,678 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:19:44,473 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:19:45,469 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:19:45,679 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:19:50,470 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:19:50,680 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:19:55,471 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:19:55,681 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:20:00,472 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:20:00,682 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:20:05,473 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:20:05,683 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:20:10,474 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:20:10,684 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:20:15,470 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:20:15,685 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:20:16,470 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:20:20,686 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:20:21,472 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:20:25,687 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:20:26,473 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:20:30,688 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:20:31,474 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:20:35,689 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:20:36,474 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:20:40,690 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:20:41,475 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:20:45,471 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:20:45,691 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:20:47,471 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:20:50,692 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:20:52,473 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:20:55,693 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:20:57,474 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:21:00,851 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:21:02,475 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:21:05,852 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:21:07,475 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:21:10,853 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:21:12,477 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:21:15,472 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:21:15,854 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:21:18,473 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:21:21,156 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:21:23,474 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:21:26,157 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:21:28,475 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:21:31,158 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:21:33,476 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:21:36,159 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:21:38,477 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:21:41,301 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:21:41,301 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:21:43,478 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:21:45,473 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:21:46,302 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:21:49,474 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:21:51,303 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:21:54,475 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:21:56,304 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:21:59,786 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:22:00,560 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/output.log 2024-06-03 21:22:01,305 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:22:04,788 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:22:06,614 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:22:09,789 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:22:11,615 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:22:14,790 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:22:15,473 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:22:16,616 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:22:18,796 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: partial_history 2024-06-03 21:22:18,797 DEBUG SenderThread:322 [sender.py:send():378] send: history 2024-06-03 21:22:18,797 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: summary_record 2024-06-03 21:22:18,797 INFO SenderThread:322 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-06-03 21:22:19,567 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/wandb-summary.json 2024-06-03 21:22:19,798 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:22:21,617 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:22:24,799 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:22:26,618 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:22:29,800 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:22:31,619 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:22:34,801 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:22:36,620 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:22:39,802 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:22:41,621 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:22:44,803 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:22:45,474 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:22:46,746 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:22:50,476 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:22:51,747 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:22:55,476 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:22:56,748 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:23:00,477 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:23:01,749 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:23:05,478 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:23:06,935 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:23:10,480 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:23:11,936 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:23:15,475 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:23:16,476 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:23:16,937 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:23:21,477 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:23:21,938 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:23:26,478 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:23:26,939 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:23:31,479 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:23:31,940 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:23:36,479 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:23:36,941 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:23:41,481 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:23:41,942 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:23:45,476 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:23:46,943 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:23:47,477 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:23:51,944 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:23:52,478 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:23:56,945 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:23:57,479 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:24:01,946 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:24:02,480 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:24:07,306 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:24:07,482 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:24:12,308 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:24:12,483 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:24:15,477 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:24:17,308 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:24:18,478 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:24:22,309 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:24:23,479 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:24:27,310 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:24:28,480 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:24:32,311 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:24:33,481 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:24:37,312 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:24:38,481 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:24:42,313 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:24:43,482 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:24:45,478 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:24:47,314 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:24:49,479 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:24:52,315 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:24:54,480 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:24:57,316 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:24:59,481 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:25:02,317 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:25:04,482 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:25:07,318 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:25:09,483 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:25:12,424 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:25:12,424 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:25:14,483 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:25:15,479 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:25:17,425 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:25:20,480 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:25:22,426 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:25:25,481 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:25:27,427 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:25:28,060 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: partial_history 2024-06-03 21:25:28,061 DEBUG SenderThread:322 [sender.py:send():378] send: history 2024-06-03 21:25:28,061 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: summary_record 2024-06-03 21:25:28,061 INFO SenderThread:322 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-06-03 21:25:28,637 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/wandb-summary.json 2024-06-03 21:25:28,637 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/output.log 2024-06-03 21:25:30,638 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/output.log 2024-06-03 21:25:31,349 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:25:32,428 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:25:36,350 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:25:37,430 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:25:41,351 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:25:42,430 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:25:45,480 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:25:46,481 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:25:47,431 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:25:51,482 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:25:52,432 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:25:56,483 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:25:57,433 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:26:01,484 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:26:02,434 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:26:06,484 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:26:07,435 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:26:11,485 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:26:12,436 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:26:15,481 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:26:17,482 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:26:17,856 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:26:22,483 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:26:22,857 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:26:27,484 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:26:27,858 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:26:32,485 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:26:32,859 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:26:37,486 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:26:37,860 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:26:42,487 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:26:42,861 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:26:45,482 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:26:47,862 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:26:48,483 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:26:52,863 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:26:53,484 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:26:57,864 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:26:58,484 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:27:02,865 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:27:03,485 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:27:07,866 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:27:08,486 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:27:12,867 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:27:13,487 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:27:15,483 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:27:17,868 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:27:19,484 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:27:22,868 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:27:24,485 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:27:27,869 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:27:29,486 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:27:32,871 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:27:34,486 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:27:38,222 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:27:39,487 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:27:43,223 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:27:44,488 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:27:45,484 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:27:48,224 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:27:50,485 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:27:53,225 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:27:55,486 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:27:58,630 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:28:00,487 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:28:03,631 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:28:05,488 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:28:08,632 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:28:10,489 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:28:13,633 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:28:15,485 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:28:16,485 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:28:18,999 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:28:21,486 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:28:24,000 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:28:26,487 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:28:29,001 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:28:31,488 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:28:34,002 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:28:36,489 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:28:39,003 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:28:41,490 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:28:44,004 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:28:45,485 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:28:47,486 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:28:49,005 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:28:52,059 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: partial_history 2024-06-03 21:28:52,060 DEBUG SenderThread:322 [sender.py:send():378] send: history 2024-06-03 21:28:52,061 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: summary_record 2024-06-03 21:28:52,061 INFO SenderThread:322 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-06-03 21:28:52,713 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/wandb-summary.json 2024-06-03 21:28:53,062 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:28:54,120 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:28:58,715 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/output.log 2024-06-03 21:28:59,031 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:28:59,121 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:29:04,033 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:29:04,122 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:29:09,033 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:29:09,123 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:29:14,034 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:29:14,124 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:29:15,486 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:29:19,126 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:29:19,487 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:29:24,127 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:29:24,488 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:29:29,256 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:29:29,489 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:29:34,257 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:29:34,490 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:29:39,258 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:29:39,490 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:29:44,259 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:29:44,491 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:29:45,487 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:29:49,448 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:29:50,489 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:29:54,449 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:29:55,489 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:29:59,450 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:30:00,490 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:30:04,451 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:30:05,491 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:30:09,452 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:30:10,492 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:30:14,453 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:30:15,488 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:30:16,489 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:30:19,454 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:30:21,490 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:30:24,455 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:30:26,491 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:30:29,456 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:30:31,491 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:30:34,457 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:30:36,492 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:30:39,458 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:30:41,493 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:30:44,459 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:30:45,489 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:30:47,490 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:30:49,559 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:30:52,491 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:30:54,560 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:30:57,492 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:30:59,561 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:31:02,493 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:31:04,562 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:31:07,494 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:31:09,591 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:31:12,495 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:31:14,826 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:31:15,490 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:31:18,491 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:31:19,827 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:31:23,492 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:31:24,828 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:31:28,493 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:31:29,866 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:31:33,494 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:31:34,867 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:31:38,495 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:31:39,868 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:31:43,496 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:31:44,870 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:31:45,495 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:31:48,496 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:31:49,871 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:31:53,497 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:31:54,872 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:31:58,498 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:31:59,873 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:32:01,092 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: partial_history 2024-06-03 21:32:01,093 DEBUG SenderThread:322 [sender.py:send():378] send: history 2024-06-03 21:32:01,093 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: summary_record 2024-06-03 21:32:01,094 INFO SenderThread:322 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-06-03 21:32:01,783 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/wandb-summary.json 2024-06-03 21:32:04,095 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:32:04,874 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:32:09,096 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:32:10,010 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:32:14,097 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:32:15,011 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:32:15,492 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:32:19,493 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:32:20,012 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:32:24,494 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:32:25,013 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:32:29,794 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/output.log 2024-06-03 21:32:29,860 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:32:30,017 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:32:34,861 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:32:35,018 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:32:39,862 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:32:40,019 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:32:44,863 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:32:45,020 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:32:45,493 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:32:50,021 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:32:50,494 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:32:55,225 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:32:55,495 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:33:00,226 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:33:00,496 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:33:05,227 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:33:05,497 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:33:10,228 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:33:10,497 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:33:15,229 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:33:15,493 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:33:16,494 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:33:20,230 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:33:21,495 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:33:25,231 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:33:26,496 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:33:30,232 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:33:31,496 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:33:35,233 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:33:36,498 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:33:40,234 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:33:41,498 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:33:45,234 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:33:45,494 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:33:47,495 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:33:50,236 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:33:52,496 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:33:55,237 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:33:57,497 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:34:00,388 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:34:02,498 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:34:05,390 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:34:07,499 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:34:10,391 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:34:12,500 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:34:15,393 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:34:15,495 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:34:18,496 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:34:20,703 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:34:23,497 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:34:25,704 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:34:28,498 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:34:30,705 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:34:33,499 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:34:35,706 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:34:38,499 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:34:40,707 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:34:43,500 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:34:45,496 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:34:45,708 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:34:49,497 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:34:50,709 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:34:54,499 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:34:55,710 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:34:59,500 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:35:00,711 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:35:04,501 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:35:05,712 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:35:09,502 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:35:10,713 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:35:10,853 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: partial_history 2024-06-03 21:35:10,855 DEBUG SenderThread:322 [sender.py:send():378] send: history 2024-06-03 21:35:10,855 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: summary_record 2024-06-03 21:35:10,856 INFO SenderThread:322 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-06-03 21:35:11,852 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/wandb-summary.json 2024-06-03 21:35:14,856 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:35:15,497 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:35:15,713 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:35:20,498 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:35:20,728 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:35:25,500 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:35:25,729 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:35:30,501 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:35:30,730 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:35:35,501 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:35:35,731 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:35:40,502 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:35:41,107 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:35:45,498 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:35:46,108 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:35:46,499 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:35:51,109 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:35:51,500 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:35:56,110 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:35:56,500 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:35:59,869 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/output.log 2024-06-03 21:36:01,111 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:36:01,510 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:36:06,342 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:36:06,511 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:36:11,344 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:36:11,512 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:36:15,499 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:36:16,345 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:36:17,500 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:36:21,346 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:36:22,501 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:36:26,613 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:36:26,613 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:36:27,502 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:36:31,614 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:36:32,503 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:36:36,615 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:36:37,504 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:36:41,616 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:36:42,505 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:36:45,500 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:36:46,872 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:36:48,501 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:36:51,874 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:36:53,502 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:36:56,875 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:36:58,502 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:37:01,876 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:37:03,503 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:37:06,877 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:37:08,504 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:37:11,878 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:37:13,505 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:37:15,501 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:37:16,879 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:37:19,502 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:37:21,880 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:37:24,503 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:37:27,174 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:37:29,504 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:37:32,175 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:37:34,505 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:37:37,176 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:37:39,506 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:37:42,177 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:37:44,507 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:37:45,501 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:37:47,178 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:37:50,503 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:37:52,179 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:37:55,504 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:37:57,180 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:38:00,505 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:38:02,181 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:38:05,506 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:38:07,182 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:38:10,507 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:38:12,183 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:38:15,502 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:38:16,503 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:38:17,184 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:38:20,502 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: partial_history 2024-06-03 21:38:20,503 DEBUG SenderThread:322 [sender.py:send():378] send: history 2024-06-03 21:38:20,503 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: summary_record 2024-06-03 21:38:20,504 INFO SenderThread:322 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-06-03 21:38:20,919 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/wandb-summary.json 2024-06-03 21:38:21,504 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:38:22,185 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:38:26,505 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:38:27,321 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:38:31,506 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:38:32,322 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:38:36,507 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:38:37,323 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:38:41,508 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:38:42,324 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:38:45,503 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:38:47,504 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:38:47,643 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:38:52,505 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:38:52,644 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:38:57,507 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:38:57,645 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:39:02,507 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:39:02,646 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:39:07,508 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:39:07,879 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:39:12,509 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:39:12,880 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:39:15,504 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:39:17,881 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:39:18,505 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:39:22,882 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:39:23,506 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:39:27,883 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:39:27,942 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/output.log 2024-06-03 21:39:29,246 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:39:29,943 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/output.log 2024-06-03 21:39:32,884 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:39:34,247 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:39:38,101 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:39:39,248 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:39:43,102 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:39:44,249 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:39:45,505 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:39:48,103 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:39:49,506 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:39:53,104 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:39:54,507 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:39:58,409 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:39:59,508 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:40:03,410 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:40:04,509 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:40:08,411 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:40:09,510 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:40:13,412 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:40:14,511 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:40:15,506 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:40:18,692 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:40:20,507 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:40:23,693 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:40:25,508 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:40:28,694 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:40:30,509 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:40:33,695 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:40:35,510 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:40:38,855 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:40:40,510 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:40:43,856 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:40:45,507 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:40:46,507 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:40:48,857 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:40:51,509 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:40:53,858 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:40:56,510 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:40:59,193 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:41:01,510 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:41:04,194 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:41:06,511 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:41:09,195 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:41:11,512 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:41:14,196 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:41:15,508 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:41:17,509 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:41:19,610 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:41:22,509 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:41:24,611 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:41:27,510 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:41:28,912 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: partial_history 2024-06-03 21:41:28,913 DEBUG SenderThread:322 [sender.py:send():378] send: history 2024-06-03 21:41:28,913 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: summary_record 2024-06-03 21:41:28,914 INFO SenderThread:322 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-06-03 21:41:28,984 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/wandb-summary.json 2024-06-03 21:41:29,612 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:41:32,915 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:41:34,613 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:41:37,916 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:41:39,803 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:41:42,917 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:41:44,804 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:41:45,509 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:41:48,509 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:41:49,805 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:41:53,510 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:41:54,806 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:41:58,511 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:42:00,126 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:42:03,512 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:42:05,127 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:42:08,513 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:42:10,128 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:42:13,513 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:42:15,129 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:42:15,510 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:42:19,511 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:42:20,310 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:42:24,512 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:42:25,311 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:42:29,513 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:42:30,312 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:42:34,514 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:42:35,313 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:42:39,514 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:42:40,314 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:42:44,516 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:42:45,315 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:42:45,511 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:42:50,316 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:42:50,512 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:42:55,317 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:42:55,513 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:42:58,018 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/output.log 2024-06-03 21:43:00,318 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:43:00,835 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:43:05,675 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:43:05,836 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:43:10,676 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:43:10,837 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:43:15,512 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:43:15,677 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:43:16,512 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:43:20,678 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:43:21,513 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:43:25,873 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:43:25,874 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:43:26,514 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:43:30,875 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:43:31,515 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:43:35,876 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:43:36,516 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:43:40,877 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:43:41,517 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:43:45,512 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:43:46,184 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:43:47,513 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:43:51,185 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:43:52,514 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:43:56,186 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:43:57,515 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:44:01,187 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:44:02,516 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:44:06,188 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:44:07,516 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:44:11,189 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:44:12,517 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:44:15,513 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:44:16,190 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:44:18,514 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:44:21,191 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:44:23,516 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:44:26,358 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:44:28,516 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:44:31,359 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:44:33,517 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:44:36,360 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:44:38,518 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:44:38,525 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: partial_history 2024-06-03 21:44:38,527 DEBUG SenderThread:322 [sender.py:send():378] send: history 2024-06-03 21:44:38,527 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: summary_record 2024-06-03 21:44:38,527 INFO SenderThread:322 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-06-03 21:44:39,053 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/wandb-summary.json 2024-06-03 21:44:41,361 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:44:43,528 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:44:45,514 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:44:46,362 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:44:49,515 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:44:51,363 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:44:54,516 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:44:56,364 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:44:59,517 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:45:01,365 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:45:04,518 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:45:06,719 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:45:09,519 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:45:11,720 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:45:14,520 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:45:15,515 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:45:16,721 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:45:20,516 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:45:21,722 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:45:25,517 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:45:26,905 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:45:26,905 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:45:30,518 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:45:31,906 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:45:35,519 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:45:36,907 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:45:40,520 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:45:41,908 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:45:45,516 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:45:46,517 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:45:47,236 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:45:51,518 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:45:52,237 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:45:56,519 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:45:57,238 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:46:01,520 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:46:02,239 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:46:06,521 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:46:07,240 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:46:11,522 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:46:12,241 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:46:15,517 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:46:17,242 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:46:17,518 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:46:22,243 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:46:22,519 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:46:27,249 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:46:28,093 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/output.log 2024-06-03 21:46:28,432 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:46:32,250 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:46:33,433 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:46:37,251 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:46:38,433 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:46:42,252 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:46:43,434 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:46:45,518 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:46:47,253 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:46:48,519 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:46:52,254 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:46:53,520 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:46:57,358 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:46:58,521 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:47:02,359 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:47:03,522 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:47:07,360 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:47:08,523 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:47:12,361 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:47:13,523 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:47:15,519 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:47:17,450 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:47:19,520 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:47:22,451 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:47:24,521 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:47:27,452 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:47:29,522 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:47:32,453 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:47:34,523 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:47:37,782 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:47:39,524 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:47:42,783 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:47:44,525 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:47:45,520 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:47:47,477 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: partial_history 2024-06-03 21:47:47,478 DEBUG SenderThread:322 [sender.py:send():378] send: history 2024-06-03 21:47:47,479 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: summary_record 2024-06-03 21:47:47,479 INFO SenderThread:322 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-06-03 21:47:47,784 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:47:48,121 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/wandb-summary.json 2024-06-03 21:47:50,480 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:47:52,785 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:47:55,481 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:47:57,786 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:48:00,482 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:48:02,787 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:48:05,483 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:48:07,788 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:48:10,484 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:48:12,789 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:48:15,485 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:48:15,520 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:48:17,790 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:48:20,522 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:48:22,791 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:48:25,523 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:48:27,792 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:48:30,524 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:48:32,793 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:48:35,525 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:48:37,883 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:48:40,526 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:48:42,884 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:48:45,521 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:48:46,522 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:48:47,885 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:48:51,523 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:48:52,886 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:48:56,524 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:48:58,154 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:48:58,155 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:49:01,525 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:49:03,156 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:49:06,526 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:49:08,156 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:49:11,527 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:49:13,157 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:49:15,522 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:49:17,523 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:49:18,357 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:49:22,524 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:49:23,358 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:49:27,525 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:49:28,359 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:49:32,526 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:49:33,360 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:49:37,527 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:49:38,659 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:49:42,528 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:49:43,660 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:49:45,523 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:49:48,524 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:49:48,661 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:49:53,525 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:49:53,741 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:49:56,168 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/output.log 2024-06-03 21:49:58,735 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:49:58,742 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:50:03,736 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:50:03,743 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:50:08,737 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:50:08,744 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:50:13,738 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:50:13,745 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:50:15,524 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:50:18,746 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:50:19,525 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:50:23,877 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:50:24,526 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:50:28,888 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:50:29,527 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:50:33,889 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:50:34,528 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:50:38,890 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:50:39,529 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:50:44,165 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:50:44,530 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:50:45,525 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:50:49,166 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:50:50,526 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:50:54,167 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:50:55,527 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:50:56,872 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: partial_history 2024-06-03 21:50:56,874 DEBUG SenderThread:322 [sender.py:send():378] send: history 2024-06-03 21:50:56,874 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: summary_record 2024-06-03 21:50:56,874 INFO SenderThread:322 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-06-03 21:50:57,189 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/wandb-summary.json 2024-06-03 21:50:59,167 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:51:00,875 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:51:04,168 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:51:05,876 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:51:09,169 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:51:10,877 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:51:14,170 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:51:15,526 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:51:16,527 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:51:19,171 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:51:21,528 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:51:24,172 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:51:26,529 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:51:29,173 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:51:31,529 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:51:34,174 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:51:36,530 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:51:39,175 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:51:41,531 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:51:44,203 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:51:45,527 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:51:47,528 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:51:49,204 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:51:52,529 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:51:54,205 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:51:57,530 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:51:59,206 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:52:02,531 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:52:04,207 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:52:07,532 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:52:09,208 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:52:12,532 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:52:14,209 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:52:15,528 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:52:18,529 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:52:19,210 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:52:23,530 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:52:24,211 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:52:28,531 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:52:29,212 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:52:33,532 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:52:34,213 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:52:38,533 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:52:39,214 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:52:43,533 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:52:44,215 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:52:45,529 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:52:49,530 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:52:49,623 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:52:54,531 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:52:54,624 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:52:59,532 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:52:59,625 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:53:04,533 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:53:04,627 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:53:09,534 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:53:09,879 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:53:14,535 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:53:14,956 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:53:15,530 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:53:19,957 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:53:20,531 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:53:24,958 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:53:26,507 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:53:27,244 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/output.log 2024-06-03 21:53:29,959 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:53:31,508 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:53:35,328 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:53:36,509 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:53:40,329 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:53:41,510 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:53:45,330 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:53:45,531 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:53:46,531 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:53:50,331 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:53:51,532 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:53:55,731 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:53:56,533 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:54:00,732 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:54:01,534 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:54:05,733 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:54:05,942 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: partial_history 2024-06-03 21:54:05,944 DEBUG SenderThread:322 [sender.py:send():378] send: history 2024-06-03 21:54:05,944 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: summary_record 2024-06-03 21:54:05,944 INFO SenderThread:322 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-06-03 21:54:06,257 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/wandb-summary.json 2024-06-03 21:54:06,945 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:54:10,734 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:54:11,946 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:54:15,531 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:54:15,803 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:54:17,532 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:54:20,804 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:54:22,534 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:54:25,806 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:54:27,534 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:54:30,807 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:54:32,536 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:54:35,808 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:54:37,537 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:54:40,809 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:54:42,538 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:54:45,533 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:54:45,810 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:54:48,534 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:54:50,811 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:54:53,535 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:54:55,812 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:54:58,535 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:55:00,813 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:55:03,537 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:55:05,814 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:55:08,538 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:55:10,815 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:55:13,539 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:55:15,533 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:55:15,877 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:55:19,535 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:55:20,878 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:55:24,536 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:55:25,879 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:55:29,537 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:55:30,881 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:55:34,538 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:55:36,091 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:55:39,539 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:55:41,092 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:55:44,540 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:55:45,535 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:55:46,093 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:55:50,536 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:55:51,094 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:55:55,537 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:55:56,531 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:56:00,538 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:56:01,533 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:56:05,539 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:56:06,533 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:56:10,539 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:56:11,534 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:56:15,536 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:56:16,537 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:56:16,550 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:56:21,538 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:56:21,551 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:56:26,538 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:56:26,552 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:56:31,540 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:56:31,554 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:56:36,540 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:56:36,670 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:56:41,541 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:56:41,671 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:56:45,537 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:56:46,672 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:56:47,538 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:56:51,673 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:56:52,538 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:56:56,674 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:56:57,340 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/output.log 2024-06-03 21:56:58,120 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:57:01,936 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:57:03,121 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:57:06,937 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:57:08,122 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:57:11,938 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:57:13,122 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:57:15,116 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: partial_history 2024-06-03 21:57:15,117 DEBUG SenderThread:322 [sender.py:send():378] send: history 2024-06-03 21:57:15,117 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: summary_record 2024-06-03 21:57:15,118 INFO SenderThread:322 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-06-03 21:57:15,348 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/wandb-summary.json 2024-06-03 21:57:15,537 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:57:16,939 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:57:18,538 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:57:21,948 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:57:23,539 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:57:26,950 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:57:28,540 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:57:31,951 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:57:33,541 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:57:36,952 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:57:38,542 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:57:42,384 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:57:43,543 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:57:45,538 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:57:47,385 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:57:49,540 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:57:52,386 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:57:54,541 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:57:57,387 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:57:59,542 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:58:02,388 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:58:04,543 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:58:07,388 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:58:09,543 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:58:12,390 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:58:14,545 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:58:15,540 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:58:17,391 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:58:20,541 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:58:22,392 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:58:25,542 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:58:27,569 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:58:30,543 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:58:32,570 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:58:35,544 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:58:37,571 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:58:40,544 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:58:42,571 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:58:45,540 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:58:46,541 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:58:47,649 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:58:51,542 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:58:52,650 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:58:56,543 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:58:57,651 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:59:01,544 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:59:02,652 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:59:06,544 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:59:08,095 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:59:11,545 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:59:13,096 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:59:15,541 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:59:17,542 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:59:18,097 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:59:22,543 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:59:23,098 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:59:27,544 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:59:28,195 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:59:28,195 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:59:32,545 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:59:33,196 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:59:37,546 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:59:38,197 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:59:42,547 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:59:43,198 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:59:45,542 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 21:59:48,199 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:59:48,543 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:59:53,200 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:59:53,544 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 21:59:58,204 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 21:59:58,545 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 22:00:03,206 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 22:00:03,546 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 22:00:08,207 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 22:00:08,547 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 22:00:13,208 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 22:00:13,547 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 22:00:15,543 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 22:00:18,209 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 22:00:19,544 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 22:00:23,210 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 22:00:24,545 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 22:00:24,561 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: partial_history 2024-06-03 22:00:24,563 DEBUG SenderThread:322 [sender.py:send():378] send: history 2024-06-03 22:00:24,563 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: summary_record 2024-06-03 22:00:24,563 INFO SenderThread:322 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-06-03 22:00:24,922 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: summary_record 2024-06-03 22:00:24,924 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: partial_history 2024-06-03 22:00:24,925 INFO SenderThread:322 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-06-03 22:00:24,925 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: summary_record 2024-06-03 22:00:24,934 INFO SenderThread:322 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-06-03 22:00:24,935 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: summary_record 2024-06-03 22:00:24,935 INFO SenderThread:322 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-06-03 22:00:24,935 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: summary_record 2024-06-03 22:00:24,936 INFO SenderThread:322 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-06-03 22:00:24,936 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: summary_record 2024-06-03 22:00:24,937 INFO SenderThread:322 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-06-03 22:00:24,937 DEBUG SenderThread:322 [sender.py:send():378] send: history 2024-06-03 22:00:24,937 DEBUG SenderThread:322 [sender.py:send_request():405] send_request: summary_record 2024-06-03 22:00:24,937 INFO SenderThread:322 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-06-03 22:00:25,443 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/wandb-summary.json 2024-06-03 22:00:27,444 INFO Thread-12 :322 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240603_175449-d191dh7n/files/output.log 2024-06-03 22:00:27,456 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: pause 2024-06-03 22:00:27,456 INFO HandlerThread:322 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-06-03 22:00:27,456 INFO HandlerThread:322 [system_monitor.py:finish():203] Stopping system monitor 2024-06-03 22:00:27,456 DEBUG SystemMonitor:322 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-06-03 22:00:27,456 DEBUG SystemMonitor:322 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-06-03 22:00:27,457 INFO HandlerThread:322 [interfaces.py:finish():200] Joined cpu monitor 2024-06-03 22:00:27,458 INFO HandlerThread:322 [interfaces.py:finish():200] Joined disk monitor 2024-06-03 22:00:27,468 INFO HandlerThread:322 [interfaces.py:finish():200] Joined gpu monitor 2024-06-03 22:00:27,468 INFO HandlerThread:322 [interfaces.py:finish():200] Joined memory monitor 2024-06-03 22:00:27,469 INFO HandlerThread:322 [interfaces.py:finish():200] Joined network monitor 2024-06-03 22:00:27,469 DEBUG SenderThread:322 [sender.py:send():378] send: stats 2024-06-03 22:00:28,211 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 22:00:30,470 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 22:00:33,212 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 22:00:35,471 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 22:00:38,213 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 22:00:40,472 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 22:00:43,214 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 22:00:45,473 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 22:00:48,215 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 22:00:50,474 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 22:00:53,216 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 22:00:55,474 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 22:00:58,217 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 22:01:00,475 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 22:01:03,218 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 22:01:05,476 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 22:01:08,219 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 22:01:10,477 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 22:01:13,220 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 22:01:15,478 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 22:01:18,222 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 22:01:20,479 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 22:01:23,223 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 22:01:25,480 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 22:01:28,223 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 22:01:30,481 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 22:01:33,225 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 22:01:35,482 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 22:01:38,226 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 22:01:40,483 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 22:01:43,227 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 22:01:45,484 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: status_report 2024-06-03 22:01:48,228 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: keepalive 2024-06-03 22:01:48,662 DEBUG HandlerThread:322 [handler.py:handle_request():158] handle_request: resume 2024-06-03 22:01:48,662 INFO HandlerThread:322 [handler.py:handle_request_resume():715] starting system metrics thread 2024-06-03 22:01:48,662 INFO HandlerThread:322 [system_monitor.py:start():194] Starting system monitor 2024-06-03 22:01:48,662 INFO SystemMonitor:322 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-06-03 22:01:48,663 INFO SystemMonitor:322 [interfaces.py:start():188] Started cpu monitoring 2024-06-03 22:01:48,663 INFO SystemMonitor:322 [interfaces.py:start():188] Started disk monitoring 2024-06-03 22:01:48,664 INFO SystemMonitor:322 [interfaces.py:start():188] Started gpu monitoring 2024-06-03 22:01:48,666 INFO SystemMonitor:322 [interfaces.py:start():188] Started memory monitoring 2024-06-03 22:01:48,667 INFO SystemMonitor:322 [interfaces.py:start():188] Started network monitoring